2024-03-26T18:34:43.335 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T18:34:43.335 DEBUG:teuthology.run:Teuthology command: teuthology --name lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi --verbose --owner scheduled_lflores@teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --archive /home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819 -- /home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819/orig.config.yaml 2024-03-26T18:34:43.390 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T18:34:43.479 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819 branch: wip-batrick-testing-20240320.145742 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: null first_in_suite: false job_id: '7623819' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.0 ' name: lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240320.145742 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - \(PG_DEGRADED - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 360516069d9393362c4cc6eb9371680fe16d66ab ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 360516069d9393362c4cc6eb9371680fe16d66ab kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-tracker-65018 sha1: 72e2437271dcafb9826f2f49bb1001426fa6de81 owner: scheduled_lflores@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4067 sha1: 360516069d9393362c4cc6eb9371680fe16d66ab sleep_before_teardown: 0 subset: 502/512 suite: fs suite_branch: wip-tracker-65018 suite_path: /home/teuthworker/src/github.com_ljflores_ceph_72e2437271dcafb9826f2f49bb1001426fa6de81/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 72e2437271dcafb9826f2f49bb1001426fa6de81 targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZpbodomQLvthcoICNQcf6ibB9ilx0/GNeph2m0RDf7GpIp5VcACBcLxaaQvoespXI1c/rn5zInyCJ3nJZ5s2k= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLf6khNxzXZdL3tBloMh3uw9/F6YAfmhLce1rGzKsbtkP/pBLC3kxF6TIpePJZGs1ZVsUdXgTIeg8vFfa9brxd8= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.0 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.0 roleless: true - print: '**** done end installing v18.2.0 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/lflores/src/github.com_ljflores_ceph_wip-tracker-65018/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_17:10:07 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-26T18:34:43.479 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_72e2437271dcafb9826f2f49bb1001426fa6de81/qa; will attempt to use it 2024-03-26T18:34:43.480 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_72e2437271dcafb9826f2f49bb1001426fa6de81/qa/tasks 2024-03-26T18:34:43.480 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T18:34:43.482 INFO:teuthology.task.internal:Checking packages... 2024-03-26T18:34:43.504 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '360516069d9393362c4cc6eb9371680fe16d66ab' 2024-03-26T18:34:43.505 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T18:34:43.505 INFO:teuthology.packaging:ref: None 2024-03-26T18:34:43.505 INFO:teuthology.packaging:tag: None 2024-03-26T18:34:43.505 INFO:teuthology.packaging:branch: wip-batrick-testing-20240320.145742 2024-03-26T18:34:43.505 INFO:teuthology.packaging:sha1: 360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:34:43.505 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240320.145742 2024-03-26T18:34:43.725 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2311.g36051606 2024-03-26T18:34:43.727 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T18:34:43.734 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T18:34:43.734 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T18:34:43.780 INFO:teuthology.task.internal:Saving configuration 2024-03-26T18:34:43.799 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T18:34:43.807 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T18:34:43.834 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 18:28:57.764628', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZpbodomQLvthcoICNQcf6ibB9ilx0/GNeph2m0RDf7GpIp5VcACBcLxaaQvoespXI1c/rn5zInyCJ3nJZ5s2k='} 2024-03-26T18:34:43.860 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 18:28:57.766374', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLf6khNxzXZdL3tBloMh3uw9/F6YAfmhLce1rGzKsbtkP/pBLC3kxF6TIpePJZGs1ZVsUdXgTIeg8vFfa9brxd8='} 2024-03-26T18:34:43.860 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T18:34:43.868 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-26T18:34:43.868 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-26T18:34:43.868 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T18:34:43.943 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb4ff70a160>, signals=[15]) 2024-03-26T18:34:43.944 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T18:34:43.950 INFO:teuthology.task.internal:Opening connections... 2024-03-26T18:34:43.951 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T18:34:43.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T18:34:44.044 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-03-26T18:34:44.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T18:34:44.131 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T18:34:44.138 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-03-26T18:34:44.155 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-03-26T18:34:44.155 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-03-26T18:34:44.211 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T18:34:44.212 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T18:34:44.213 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-03-26T18:34:44.239 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-03-26T18:34:44.256 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-03-26T18:34:44.257 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-03-26T18:34:44.312 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T18:34:44.313 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T18:34:44.314 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T18:34:44.314 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-03-26T18:34:44.339 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T18:34:44.350 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T18:34:44.358 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T18:34:44.359 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-03-26T18:34:44.361 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-03-26T18:34:44.376 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T18:34:44.399 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T18:34:44.399 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-03-26T18:34:44.417 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-03-26T18:34:44.528 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T18:34:44.575 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T18:34:44.618 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T18:34:44.618 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T18:34:44.619 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T18:34:44.619 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T18:34:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T18:34:44.635 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-26T18:34:44.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T18:34:44.637 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-03-26T18:34:44.693 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-361.el9.x86_64 2024-03-26T18:34:44.693 INFO:teuthology.task.kernel:Running kernel on smithi102: 5.14.0-361.el9.x86_64 2024-03-26T18:34:44.693 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2024-03-26T18:34:44.694 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-361.el9.x86_64 2024-03-26T18:34:44.694 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.14.0-361.el9.x86_64 2024-03-26T18:34:44.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-03-26T18:34:46.964 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-26T18:34:47.138 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 8.7 MB/s | 6.7 MB 00:00 2024-03-26T18:34:48.431 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-26T18:34:48.813 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-26T18:34:50.555 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.0 MB 00:01 2024-03-26T18:34:51.421 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.0 MB 00:02 2024-03-26T18:34:52.348 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-26T18:34:53.405 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-26T18:34:56.573 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.1 MB 00:00 2024-03-26T18:34:57.506 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.1 MB 00:00 2024-03-26T18:34:59.057 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-03-26T18:35:00.065 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-03-26T18:35:01.250 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-26T18:35:02.608 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 6.0 MB/s | 21 MB 00:03 2024-03-26T18:35:06.275 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-26T18:35:07.466 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-26T18:35:08.448 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T18:35:08.448 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T18:35:08.545 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:35:08.556 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:35:08.557 INFO:teuthology.orchestra.run.smithi102.stdout:Install 4 Packages 2024-03-26T18:35:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:35:08.558 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 76 M 2024-03-26T18:35:08.558 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 123 M 2024-03-26T18:35:08.558 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:35:08.751 INFO:teuthology.orchestra.run.smithi102.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 484 kB/s | 66 kB 00:00 2024-03-26T18:35:09.260 INFO:teuthology.orchestra.run.smithi102.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-26T18:35:09.649 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T18:35:09.649 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T18:35:09.738 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:35:09.745 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-03-26T18:35:09.746 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:35:09.747 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 76 M 2024-03-26T18:35:09.747 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 123 M 2024-03-26T18:35:09.747 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:35:09.901 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 564 kB/s | 66 kB 00:00 2024-03-26T18:35:10.818 INFO:teuthology.orchestra.run.smithi102.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 15 MB/s | 33 MB 00:02 2024-03-26T18:35:11.068 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 26 MB/s | 33 MB 00:01 2024-03-26T18:35:11.243 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 10 MB/s | 15 MB 00:01 2024-03-26T18:35:12.144 INFO:teuthology.orchestra.run.smithi102.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 8.1 MB/s | 27 MB 00:03 2024-03-26T18:35:12.144 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:35:12.144 INFO:teuthology.orchestra.run.smithi102.stdout:Total 21 MB/s | 76 MB 00:03 2024-03-26T18:35:12.744 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 9.7 MB/s | 27 MB 00:02 2024-03-26T18:35:12.744 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:35:12.744 INFO:teuthology.orchestra.run.smithi190.stdout:Total 25 MB/s | 76 MB 00:02 2024-03-26T18:35:12.813 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:35:13.188 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:35:13.189 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:35:13.407 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:35:13.773 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:35:13.774 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:35:14.170 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:35:14.170 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:35:14.768 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:35:14.769 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:35:16.647 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:35:17.246 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:35:17.283 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T18:35:17.369 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T18:35:17.861 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T18:35:17.952 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T18:35:20.113 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T18:35:20.249 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T18:35:20.696 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T18:35:20.830 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T18:35:24.954 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T18:35:25.066 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:35:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T18:35:25.629 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:35:29.819 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:35:30.348 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:35:59.233 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:35:59.234 INFO:teuthology.orchestra.run.smithi102.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T18:35:59.234 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:35:59.278 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:35:59.631 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:35:59.632 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T18:35:59.632 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T18:35:59.632 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T18:35:59.659 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:35:59.659 INFO:teuthology.orchestra.run.smithi190.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T18:35:59.659 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:35:59.704 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:36:00.044 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:36:00.045 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:36:00.045 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:36:00.045 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T18:36:00.045 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T18:36:00.045 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T18:36:00.045 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T18:36:00.045 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:36:00.045 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:36:00.095 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:36:00.095 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T18:36:00.095 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T18:36:00.095 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T18:36:00.372 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T18:36:00.464 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T18:36:00.465 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T18:36:00.465 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T18:36:00.465 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T18:36:00.465 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T18:36:00.465 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-26T18:36:00.465 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T18:36:00.465 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T18:36:00.506 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T18:36:00.506 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:36:00.506 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:36:00.506 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T18:36:00.506 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T18:36:00.506 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T18:36:00.506 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T18:36:00.506 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:36:00.506 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:36:00.544 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T18:36:00.544 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T18:36:00.544 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2024-03-26T18:36:00.578 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2024-03-26T18:36:00.578 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2024-03-26T18:36:00.578 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.06 2024-03-26T18:36:00.578 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 68.el9 2024-03-26T18:36:00.578 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2024-03-26T18:36:00.578 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T18:36:00.578 INFO:teuthology.orchestra.run.smithi102.stdout:Group : Unspecified 2024-03-26T18:36:00.578 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 8264254 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : builder@centos.org 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2024-03-26T18:36:00.579 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:36:00.580 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T18:36:00.580 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T18:36:00.580 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 5.14.0-432.el9.x86_64 2024-03-26T18:36:00.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T18:36:00.839 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T18:36:00.940 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T18:36:00.940 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T18:36:00.940 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T18:36:00.940 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T18:36:00.940 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T18:36:00.940 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-26T18:36:00.940 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T18:36:00.940 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T18:36:01.020 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T18:36:01.020 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T18:36:01.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2024-03-26T18:36:01.053 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2024-03-26T18:36:01.054 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2024-03-26T18:36:01.054 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.06 2024-03-26T18:36:01.054 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 68.el9 2024-03-26T18:36:01.054 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2024-03-26T18:36:01.054 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T18:36:01.054 INFO:teuthology.orchestra.run.smithi190.stdout:Group : Unspecified 2024-03-26T18:36:01.054 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 8264254 2024-03-26T18:36:01.054 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2024-03-26T18:36:01.054 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : builder@centos.org 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2024-03-26T18:36:01.055 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:36:01.056 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T18:36:01.056 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T18:36:01.056 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 5.14.0-432.el9.x86_64 2024-03-26T18:36:01.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T18:36:01.196 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-03-26T18:36:01.744 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-03-26T18:36:02.611 INFO:teuthology.orchestra.run.smithi102.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T18:36:02.634 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-03-26T18:36:02.646 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-26T18:36:02.661 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.zOvU4ugiri 2024-03-26T18:36:02.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zOvU4ugiri 2024-03-26T18:36:02.731 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.zOvU4ugiri 2024-03-26T18:36:02.867 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.zOvU4ugiri is 6KB 2024-03-26T18:36:02.917 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.zOvU4ugiri 2024-03-26T18:36:02.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T18:36:02.997 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T18:36:02.997 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T18:36:02.997 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T18:36:02.997 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T18:36:02.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T18:36:03.073 INFO:teuthology.orchestra.run.smithi190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T18:36:03.107 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-03-26T18:36:03.109 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-03-26T18:36:03.125 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.DiKLI3F9Lb 2024-03-26T18:36:03.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DiKLI3F9Lb 2024-03-26T18:36:03.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2024-03-26T18:36:03.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.DiKLI3F9Lb 2024-03-26T18:36:03.336 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.DiKLI3F9Lb is 6KB 2024-03-26T18:36:03.384 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.DiKLI3F9Lb 2024-03-26T18:36:03.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T18:36:03.463 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T18:36:03.463 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T18:36:03.463 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T18:36:03.463 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T18:36:03.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T18:36:03.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-03-26T18:36:03.719 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-26T18:36:33.188 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-26T18:36:33.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-26T18:36:33.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T18:36:33.614 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-26T18:36:33.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-26T18:36:33.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T18:36:51.615 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-26T18:36:52.126 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-03-26T18:37:00.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-26T18:37:00.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T18:37:01.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-26T18:37:01.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T18:37:03.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-26T18:37:04.194 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-03-26T18:37:15.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-26T18:37:15.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T18:37:16.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-26T18:37:16.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T18:37:34.111 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-26T18:37:47.986 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-03-26T18:37:48.393 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-26T18:37:48.393 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T18:37:48.393 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-03-26T18:37:48.407 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-432.el9.x86_64 2024-03-26T18:37:48.408 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T18:37:48.408 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T18:37:48.408 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T18:37:49.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-26T18:37:49.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T18:37:49.409 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-26T18:37:49.410 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T18:37:49.470 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-03-26T18:37:49.511 DEBUG:teuthology.parallel:result is None 2024-03-26T18:37:49.519 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-26T18:37:49.932 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-26T18:37:49.932 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T18:37:49.933 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-26T18:37:49.946 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-432.el9.x86_64 2024-03-26T18:37:49.947 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T18:37:49.947 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T18:37:49.947 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T18:37:50.948 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-26T18:37:50.948 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T18:37:51.038 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2024-03-26T18:37:51.087 DEBUG:teuthology.parallel:result is None 2024-03-26T18:37:51.087 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T18:37:51.094 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T18:37:51.095 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T18:37:51.097 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T18:37:51.112 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T18:37:51.140 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T18:37:51.149 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T18:37:51.150 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T18:37:51.155 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T18:37:51.248 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T18:37:51.256 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T18:37:51.256 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T18:37:51.258 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T18:37:51.284 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T18:37:51.294 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T18:37:51.297 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T18:37:51.307 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T18:37:51.308 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T18:37:51.317 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T18:37:51.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T18:37:51.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T18:37:51.375 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T18:37:51.386 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T18:37:51.386 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T18:37:51.418 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T18:37:51.433 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T18:37:51.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T18:37:51.568 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T18:37:51.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T18:37:51.697 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:37:51.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T18:37:51.767 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T18:37:51.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T18:37:51.857 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T18:37:51.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T18:37:51.983 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:37:51.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T18:37:52.049 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-03-26T18:37:52.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-03-26T18:37:52.132 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T18:37:52.154 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T18:37:52.658 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T18:37:52.666 INFO:teuthology.task.internal:Starting timer... 2024-03-26T18:37:52.666 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T18:37:52.696 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T18:37:52.705 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-26T18:37:52.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-03-26T18:37:52.790 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-03-26T18:37:52.795 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-03-26T18:37:52.863 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-03-26T18:37:52.865 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T18:37:52.865 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-03-26T18:37:52.904 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-03-26T18:37:52.904 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-03-26T18:37:52.932 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-03-26T18:37:52.932 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-03-26T18:37:52.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T18:37:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T18:37:52.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T18:37:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T18:37:53.027 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T18:37:53.027 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-03-26T18:37:53.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-03-26T18:37:53.125 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T18:37:53.134 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T18:37:53.151 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T18:37:53.152 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T18:42:23.518 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-03-26T18:42:23.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-26T18:42:23.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T18:42:23.621 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-26T18:42:23.697 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-26T18:42:23.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-26T18:42:23.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T18:42:23.787 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-03-26T18:42:23.870 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-26T18:42:23.871 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T18:42:23.884 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T18:42:23.885 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T18:42:23.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T18:42:23.887 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T18:42:23.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T18:42:23.917 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T18:42:23.934 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T18:42:23.942 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T18:42:23.959 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T18:42:23.963 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-03-26T18:42:23.976 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-03-26T18:42:23.988 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-03-26T18:42:23.992 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T18:42:24.002 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-03-26T18:42:24.009 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T18:42:24.018 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T18:42:24.034 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T18:42:24.060 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-26T18:42:24.063 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T18:42:24.063 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-26T18:42:24.064 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T18:42:24.064 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T18:42:24.064 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T18:42:24.064 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T18:42:24.098 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-03-26T18:42:24.101 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T18:42:24.102 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-03-26T18:42:24.102 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T18:42:24.102 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T18:42:24.102 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T18:42:24.102 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T18:42:24.103 INFO:teuthology.run_tasks:Running task install... 2024-03-26T18:42:24.112 DEBUG:teuthology.task.install:project ceph 2024-03-26T18:42:24.112 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '360516069d9393362c4cc6eb9371680fe16d66ab'}} 2024-03-26T18:42:24.112 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.0', 'flavor': 'default', 'sha1': '360516069d9393362c4cc6eb9371680fe16d66ab'} 2024-03-26T18:42:24.112 INFO:teuthology.task.install:Using flavor: default 2024-03-26T18:42:24.120 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T18:42:24.120 INFO:teuthology.task.install:extra packages: [] 2024-03-26T18:42:24.121 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '360516069d9393362c4cc6eb9371680fe16d66ab', 'tag': 'v18.2.0', 'wait_for_package': False} 2024-03-26T18:42:24.121 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-26T18:42:24.121 INFO:teuthology.packaging:ref: None 2024-03-26T18:42:24.121 INFO:teuthology.packaging:tag: v18.2.0 2024-03-26T18:42:24.121 INFO:teuthology.packaging:branch: None 2024-03-26T18:42:24.121 INFO:teuthology.packaging:sha1: 360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:42:24.498 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v18.2.0^{} -> None 2024-03-26T18:42:24.498 INFO:teuthology.packaging:Tag 'v18.2.0' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-03-26T18:42:26.479 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.0^{} -> 5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-03-26T18:42:26.480 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-03-26T18:42:26.482 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '360516069d9393362c4cc6eb9371680fe16d66ab', 'tag': 'v18.2.0', 'wait_for_package': False} 2024-03-26T18:42:26.483 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-26T18:42:26.483 INFO:teuthology.packaging:ref: None 2024-03-26T18:42:26.483 INFO:teuthology.packaging:tag: v18.2.0 2024-03-26T18:42:26.483 INFO:teuthology.packaging:branch: None 2024-03-26T18:42:26.483 INFO:teuthology.packaging:sha1: 360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:42:26.483 INFO:teuthology.packaging:Tag 'v18.2.0' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-03-26T18:42:26.483 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-03-26T18:42:26.700 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/ 2024-03-26T18:42:26.700 INFO:teuthology.task.install.rpm:Package version is 18.2.0-0 2024-03-26T18:42:26.715 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/ 2024-03-26T18:42:26.715 INFO:teuthology.task.install.rpm:Package version is 18.2.0-0 2024-03-26T18:42:26.746 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T18:42:26.747 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:42:26.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T18:42:26.760 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T18:42:26.760 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:42:26.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T18:42:26.779 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T18:42:26.779 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-26T18:42:26.779 INFO:teuthology.packaging:ref: None 2024-03-26T18:42:26.779 INFO:teuthology.packaging:tag: v18.2.0 2024-03-26T18:42:26.779 INFO:teuthology.packaging:branch: None 2024-03-26T18:42:26.780 INFO:teuthology.packaging:sha1: 360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:42:26.780 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T18:42:26.794 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T18:42:26.794 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-26T18:42:26.794 INFO:teuthology.packaging:ref: None 2024-03-26T18:42:26.794 INFO:teuthology.packaging:tag: v18.2.0 2024-03-26T18:42:26.794 INFO:teuthology.packaging:branch: None 2024-03-26T18:42:26.794 INFO:teuthology.packaging:sha1: 360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:42:26.794 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T18:42:26.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T18:42:26.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T18:42:26.931 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T18:42:26.949 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T18:42:26.999 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2024-03-26T18:42:27.000 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2024-03-26T18:42:27.017 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-03-26T18:42:27.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-03-26T18:42:27.256 INFO:teuthology.orchestra.run.smithi102.stdout:74 files removed 2024-03-26T18:42:27.269 INFO:teuthology.orchestra.run.smithi190.stdout:74 files removed 2024-03-26T18:42:27.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw 2024-03-26T18:42:27.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw 2024-03-26T18:42:27.764 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 412 kB/s | 76 kB 00:00 2024-03-26T18:42:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 457 kB/s | 76 kB 00:00 2024-03-26T18:42:27.981 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 55 kB/s | 9.3 kB 00:00 2024-03-26T18:42:28.143 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 30 kB/s | 9.3 kB 00:00 2024-03-26T18:42:28.182 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 13 kB/s | 2.2 kB 00:00 2024-03-26T18:42:28.327 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 15 kB/s | 2.2 kB 00:00 2024-03-26T18:42:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 19 MB/s | 6.7 MB 00:00 2024-03-26T18:42:28.812 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-26T18:42:29.944 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-26T18:42:30.237 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-26T18:42:33.828 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-03-26T18:42:34.209 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:03 2024-03-26T18:42:36.202 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-26T18:42:45.419 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 1.1 MB/s | 6.1 MB 00:05 2024-03-26T18:42:47.044 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-03-26T18:42:48.474 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-26T18:42:53.371 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-26T18:42:55.849 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:42:55.875 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:42:55.875 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-26T18:42:55.875 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:42:55.876 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:42:55.876 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:18.2.0-0.el9 ceph 7.6 M 2024-03-26T18:42:55.876 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-03-26T18:42:55.876 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:18.2.0-0.el9 ceph 3.3 M 2024-03-26T18:42:55.876 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-03-26T18:42:55.876 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-26T18:42:55.876 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:18.2.0-0.el9 ceph 5.2 M 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:18.2.0-0.el9 ceph 18 M 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:18.2.0-0.el9 ceph 24 k 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:18.2.0-0.el9 ceph 653 k 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:18.2.0-0.el9 ceph 474 k 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T18:42:55.877 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:18.2.0-0.el9 ceph 45 k 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:18.2.0-0.el9 ceph 119 k 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:18.2.0-0.el9 ceph 155 k 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:18.2.0-0.el9 ceph 321 k 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:18.2.0-0.el9 ceph 297 k 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:18.2.0-0.el9 ceph 99 k 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:42:55.878 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:42:55.879 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:42:55.879 INFO:teuthology.orchestra.run.smithi102.stdout:Install 28 Packages 2024-03-26T18:42:55.879 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 2 Packages 2024-03-26T18:42:55.879 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:42:55.879 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 53 M 2024-03-26T18:42:55.879 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:42:56.473 INFO:teuthology.orchestra.run.smithi102.stdout:(1/30): ceph-radosgw-18.2.0-0.el9.x86_64.rpm 14 MB/s | 7.6 MB 00:00 2024-03-26T18:42:57.220 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 866 kB/s | 19 MB 00:22 2024-03-26T18:42:57.221 INFO:teuthology.orchestra.run.smithi102.stdout:(2/30): ceph-selinux-18.2.0-0.el9.x86_64.rpm 483 kB/s | 24 kB 00:00 2024-03-26T18:42:57.221 INFO:teuthology.orchestra.run.smithi102.stdout:(3/30): ceph-common-18.2.0-0.el9.x86_64.rpm 23 MB/s | 18 MB 00:00 2024-03-26T18:42:57.221 INFO:teuthology.orchestra.run.smithi102.stdout:(4/30): libcephfs2-18.2.0-0.el9.x86_64.rpm 2.6 MB/s | 653 kB 00:00 2024-03-26T18:42:57.221 INFO:teuthology.orchestra.run.smithi102.stdout:(5/30): ceph-base-18.2.0-0.el9.x86_64.rpm 5.7 MB/s | 5.2 MB 00:00 2024-03-26T18:42:57.221 INFO:teuthology.orchestra.run.smithi102.stdout:(6/30): libradosstriper1-18.2.0-0.el9.x86_64.rp 3.5 MB/s | 474 kB 00:00 2024-03-26T18:42:57.221 INFO:teuthology.orchestra.run.smithi102.stdout:(7/30): librgw2-18.2.0-0.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-03-26T18:42:57.221 INFO:teuthology.orchestra.run.smithi102.stdout:(8/30): python3-ceph-argparse-18.2.0-0.el9.x86_ 256 kB/s | 45 kB 00:00 2024-03-26T18:42:57.221 INFO:teuthology.orchestra.run.smithi102.stdout:(9/30): python3-ceph-common-18.2.0-0.el9.x86_64 682 kB/s | 119 kB 00:00 2024-03-26T18:42:57.221 INFO:teuthology.orchestra.run.smithi102.stdout:(10/30): python3-cephfs-18.2.0-0.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-26T18:42:57.222 INFO:teuthology.orchestra.run.smithi102.stdout:(11/30): python3-rados-18.2.0-0.el9.x86_64.rpm 3.4 MB/s | 321 kB 00:00 2024-03-26T18:42:57.222 INFO:teuthology.orchestra.run.smithi102.stdout:(12/30): python3-rbd-18.2.0-0.el9.x86_64.rpm 2.2 MB/s | 297 kB 00:00 2024-03-26T18:42:57.226 INFO:teuthology.orchestra.run.smithi102.stdout:(13/30): python3-rgw-18.2.0-0.el9.x86_64.rpm 625 kB/s | 99 kB 00:00 2024-03-26T18:42:57.260 INFO:teuthology.orchestra.run.smithi102.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 702 kB/s | 106 kB 00:00 2024-03-26T18:42:57.302 INFO:teuthology.orchestra.run.smithi102.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-26T18:42:57.320 INFO:teuthology.orchestra.run.smithi102.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 481 kB/s | 44 kB 00:00 2024-03-26T18:42:57.343 INFO:teuthology.orchestra.run.smithi102.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-26T18:42:57.360 INFO:teuthology.orchestra.run.smithi102.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 41 kB 00:00 2024-03-26T18:42:57.385 INFO:teuthology.orchestra.run.smithi102.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-26T18:42:57.844 INFO:teuthology.orchestra.run.smithi102.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 221 kB/s | 111 kB 00:00 2024-03-26T18:42:57.961 INFO:teuthology.orchestra.run.smithi102.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-03-26T18:42:57.987 INFO:teuthology.orchestra.run.smithi102.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 494 kB/s | 309 kB 00:00 2024-03-26T18:42:58.096 INFO:teuthology.orchestra.run.smithi102.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 363 kB/s | 49 kB 00:00 2024-03-26T18:42:58.120 INFO:teuthology.orchestra.run.smithi102.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 505 kB/s | 67 kB 00:00 2024-03-26T18:42:58.271 INFO:teuthology.orchestra.run.smithi102.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-26T18:42:58.354 INFO:teuthology.orchestra.run.smithi102.stdout:(26/30): libarrow-9.0.0-10.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-03-26T18:42:58.505 INFO:teuthology.orchestra.run.smithi102.stdout:(27/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-03-26T18:42:58.588 INFO:teuthology.orchestra.run.smithi102.stdout:(28/30): librados2-18.2.0-0.el9.x86_64.rpm 14 MB/s | 3.3 MB 00:00 2024-03-26T18:42:58.630 INFO:teuthology.orchestra.run.smithi102.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2024-03-26T18:42:58.680 INFO:teuthology.orchestra.run.smithi102.stdout:(30/30): librbd1-18.2.0-0.el9.x86_64.rpm 17 MB/s | 3.0 MB 00:00 2024-03-26T18:42:58.683 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:42:58.683 INFO:teuthology.orchestra.run.smithi102.stdout:Total 19 MB/s | 53 MB 00:02 2024-03-26T18:42:58.684 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:42:58.718 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:42:58.719 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:42:59.166 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:42:59.167 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:43:00.536 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:43:00.624 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-26T18:43:00.660 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-26T18:43:00.711 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-26T18:43:01.104 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-26T18:43:01.142 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:18.2.0-0.el9.x86_64 5/32 2024-03-26T18:43:01.238 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:18.2.0-0.el9.x86_64 5/32 2024-03-26T18:43:01.316 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:18.2.0-0.el9.x86_64 6/32 2024-03-26T18:43:01.369 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-03-26T18:43:01.415 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-03-26T18:43:01.444 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-26T18:43:01.489 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 9/32 2024-03-26T18:43:01.554 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-03-26T18:43:01.580 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-03-26T18:43:01.645 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-03-26T18:43:01.754 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-26T18:43:01.787 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-26T18:43:01.828 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-26T18:43:01.860 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-26T18:43:01.904 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-26T18:43:01.939 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-26T18:43:01.972 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-26T18:43:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-26T18:43:02.302 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-03-26T18:43:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-03-26T18:43:02.432 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:18.2.0-0.el9.x86_64 21/32 2024-03-26T18:43:02.472 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-26T18:43:02.866 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-26T18:43:02.888 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-03-26T18:43:02.891 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.1 MB 00:02 2024-03-26T18:43:02.959 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-03-26T18:43:03.017 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:18.2.0-0.el9.x86_64 25/32 2024-03-26T18:43:03.041 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:18.2.0-0.el9.x86_64 26/32 2024-03-26T18:43:04.336 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-03-26T18:43:05.030 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-03-26T18:43:05.079 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-03-26T18:43:05.620 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-03-26T18:43:05.657 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-03-26T18:43:05.751 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-03-26T18:43:05.752 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T18:43:05.752 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T18:43:05.752 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:05.781 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-26T18:43:05.791 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-03-26T18:43:10.762 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-26T18:43:13.241 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:43:13.292 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:43:13.292 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-26T18:43:13.292 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:43:13.293 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:43:13.293 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:18.2.0-0.el9 ceph 7.6 M 2024-03-26T18:43:13.293 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-03-26T18:43:13.293 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:18.2.0-0.el9 ceph 3.3 M 2024-03-26T18:43:13.293 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-03-26T18:43:13.293 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-26T18:43:13.293 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T18:43:13.293 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:18.2.0-0.el9 ceph 5.2 M 2024-03-26T18:43:13.294 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:18.2.0-0.el9 ceph 18 M 2024-03-26T18:43:13.294 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:18.2.0-0.el9 ceph 24 k 2024-03-26T18:43:13.294 INFO:teuthology.orchestra.run.smithi190.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-26T18:43:13.294 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T18:43:13.294 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:18.2.0-0.el9 ceph 653 k 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:18.2.0-0.el9 ceph 474 k 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:18.2.0-0.el9 ceph 45 k 2024-03-26T18:43:13.295 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:18.2.0-0.el9 ceph 119 k 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:18.2.0-0.el9 ceph 155 k 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:18.2.0-0.el9 ceph 321 k 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:18.2.0-0.el9 ceph 297 k 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:18.2.0-0.el9 ceph 99 k 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout:Install 28 Packages 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2024-03-26T18:43:13.296 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:13.297 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 53 M 2024-03-26T18:43:13.297 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:43:13.547 INFO:teuthology.orchestra.run.smithi190.stdout:(1/30): ceph-base-18.2.0-0.el9.x86_64.rpm 30 MB/s | 5.2 MB 00:00 2024-03-26T18:43:13.572 INFO:teuthology.orchestra.run.smithi190.stdout:(2/30): ceph-selinux-18.2.0-0.el9.x86_64.rpm 970 kB/s | 24 kB 00:00 2024-03-26T18:43:13.623 INFO:teuthology.orchestra.run.smithi190.stdout:(3/30): libcephfs2-18.2.0-0.el9.x86_64.rpm 13 MB/s | 653 kB 00:00 2024-03-26T18:43:13.656 INFO:teuthology.orchestra.run.smithi190.stdout:(4/30): libradosstriper1-18.2.0-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-03-26T18:43:13.789 INFO:teuthology.orchestra.run.smithi190.stdout:(5/30): ceph-radosgw-18.2.0-0.el9.x86_64.rpm 18 MB/s | 7.6 MB 00:00 2024-03-26T18:43:13.815 INFO:teuthology.orchestra.run.smithi190.stdout:(6/30): python3-ceph-argparse-18.2.0-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-03-26T18:43:13.873 INFO:teuthology.orchestra.run.smithi190.stdout:(7/30): librgw2-18.2.0-0.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-03-26T18:43:13.898 INFO:teuthology.orchestra.run.smithi190.stdout:(8/30): python3-ceph-common-18.2.0-0.el9.x86_64 1.4 MB/s | 119 kB 00:00 2024-03-26T18:43:13.924 INFO:teuthology.orchestra.run.smithi190.stdout:(9/30): python3-cephfs-18.2.0-0.el9.x86_64.rpm 6.1 MB/s | 155 kB 00:00 2024-03-26T18:43:13.949 INFO:teuthology.orchestra.run.smithi190.stdout:(10/30): python3-rados-18.2.0-0.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-26T18:43:13.974 INFO:teuthology.orchestra.run.smithi190.stdout:(11/30): python3-rbd-18.2.0-0.el9.x86_64.rpm 5.8 MB/s | 297 kB 00:00 2024-03-26T18:43:13.999 INFO:teuthology.orchestra.run.smithi190.stdout:(12/30): python3-rgw-18.2.0-0.el9.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-03-26T18:43:14.208 INFO:teuthology.orchestra.run.smithi190.stdout:(13/30): ceph-common-18.2.0-0.el9.x86_64.rpm 22 MB/s | 18 MB 00:00 2024-03-26T18:43:14.467 INFO:teuthology.orchestra.run.smithi190.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 215 kB/s | 106 kB 00:00 2024-03-26T18:43:14.492 INFO:teuthology.orchestra.run.smithi190.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 188 kB/s | 44 kB 00:00 2024-03-26T18:43:14.517 INFO:teuthology.orchestra.run.smithi190.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 307 kB/s | 159 kB 00:00 2024-03-26T18:43:14.584 INFO:teuthology.orchestra.run.smithi190.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 661 kB 00:00 2024-03-26T18:43:14.609 INFO:teuthology.orchestra.run.smithi190.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2024-03-26T18:43:14.643 INFO:teuthology.orchestra.run.smithi190.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 329 kB/s | 41 kB 00:00 2024-03-26T18:43:14.818 INFO:teuthology.orchestra.run.smithi190.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 474 kB/s | 111 kB 00:00 2024-03-26T18:43:14.860 INFO:teuthology.orchestra.run.smithi190.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-03-26T18:43:14.902 INFO:teuthology.orchestra.run.smithi190.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-26T18:43:14.936 INFO:teuthology.orchestra.run.smithi190.stdout:(23/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-26T18:43:14.961 INFO:teuthology.orchestra.run.smithi190.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-26T18:43:15.061 INFO:teuthology.orchestra.run.smithi190.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-26T18:43:15.368 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-03-26T18:43:15.368 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2024-03-26T18:43:15.368 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2024-03-26T18:43:15.368 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2024-03-26T18:43:15.368 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2024-03-26T18:43:15.368 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2024-03-26T18:43:15.368 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /root 2024-03-26T18:43:15.368 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2024-03-26T18:43:15.368 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:16.113 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-03-26T18:43:16.138 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-03-26T18:43:16.138 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:43:16.138 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T18:43:16.138 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T18:43:16.138 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T18:43:16.139 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:16.140 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-26T18:43:16.249 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-26T18:43:16.249 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T18:43:16.428 INFO:teuthology.orchestra.run.smithi190.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 562 kB/s | 838 kB 00:01 2024-03-26T18:43:16.503 INFO:teuthology.orchestra.run.smithi190.stdout:(27/30): librados2-18.2.0-0.el9.x86_64.rpm 44 MB/s | 3.3 MB 00:00 2024-03-26T18:43:16.579 INFO:teuthology.orchestra.run.smithi190.stdout:(28/30): librbd1-18.2.0-0.el9.x86_64.rpm 40 MB/s | 3.0 MB 00:00 2024-03-26T18:43:16.679 INFO:teuthology.orchestra.run.smithi190.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 1.0 MB/s | 1.6 MB 00:01 2024-03-26T18:43:16.845 INFO:teuthology.orchestra.run.smithi190.stdout:(30/30): libarrow-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 4.4 MB 00:02 2024-03-26T18:43:16.850 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:43:16.850 INFO:teuthology.orchestra.run.smithi190.stdout:Total 15 MB/s | 53 MB 00:03 2024-03-26T18:43:16.851 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:43:16.887 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:43:16.887 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:43:17.320 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T18:43:17.320 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:18.2.0-0.el9.x86_64 1/32 2024-03-26T18:43:17.320 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:18.2.0-0.el9.x86_64 2/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:18.2.0-0.el9.x86_64 3/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:18.2.0-0.el9.x86_64 4/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:18.2.0-0.el9.x86_64 5/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:18.2.0-0.el9.x86_64 6/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:18.2.0-0.el9.x86_64 7/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 8/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:18.2.0-0.el9.x86_64 9/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:18.2.0-0.el9.x86_64 11/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:18.2.0-0.el9.x86_64 12/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:18.2.0-0.el9.x86_64 13/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-26T18:43:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-26T18:43:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:18.2.0-0.el9.x86_64 29/32 2024-03-26T18:43:17.323 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-26T18:43:17.323 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:18.2.0-0.el9.x86_64 31/32 2024-03-26T18:43:17.344 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:43:17.345 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:43:18.258 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T18:43:18.258 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:18.258 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-03-26T18:43:18.258 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:18.2.0-0.el9.x86_64 librbd1-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.258 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:43:18.258 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T18:43:18.258 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.258 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.258 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.258 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.258 INFO:teuthology.orchestra.run.smithi102.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-26T18:43:18.259 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T18:43:18.259 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T18:43:18.259 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T18:43:18.259 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.259 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T18:43:18.259 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T18:43:18.259 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T18:43:18.259 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.259 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T18:43:18.259 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.259 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:18.260 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:43:18.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-test 2024-03-26T18:43:18.740 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:43:18.816 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-26T18:43:18.843 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-26T18:43:18.910 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-26T18:43:19.137 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:26 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:43:19.224 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-26T18:43:19.296 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:43:19.297 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:19.297 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-26T18:43:19.297 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:19.297 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:43:19.297 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:18.2.0-0.el9 ceph 40 M 2024-03-26T18:43:19.297 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-26T18:43:19.297 INFO:teuthology.orchestra.run.smithi102.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T18:43:19.297 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:18.2.0-0.el9 ceph 161 k 2024-03-26T18:43:19.297 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T18:43:19.297 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T18:43:19.297 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T18:43:19.298 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:19.298 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:43:19.298 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:19.298 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2024-03-26T18:43:19.298 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:19.298 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 40 M 2024-03-26T18:43:19.298 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 167 M 2024-03-26T18:43:19.299 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:43:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:18.2.0-0.el9.x86_64 5/32 2024-03-26T18:43:19.400 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:18.2.0-0.el9.x86_64 5/32 2024-03-26T18:43:19.480 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:18.2.0-0.el9.x86_64 6/32 2024-03-26T18:43:19.506 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-26T18:43:19.509 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-03-26T18:43:19.564 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-26T18:43:19.571 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-03-26T18:43:19.598 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-26T18:43:19.600 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-26T18:43:19.632 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): libcephsqlite-18.2.0-0.el9.x86_64.rpm 544 kB/s | 161 kB 00:00 2024-03-26T18:43:19.648 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 9/32 2024-03-26T18:43:19.710 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-03-26T18:43:19.736 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-03-26T18:43:19.785 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-03-26T18:43:19.831 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 805 kB/s | 187 kB 00:00 2024-03-26T18:43:19.892 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-26T18:43:19.919 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-26T18:43:19.959 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-26T18:43:19.999 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-26T18:43:20.026 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-26T18:43:20.062 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-26T18:43:20.095 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-26T18:43:20.403 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-26T18:43:20.433 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-03-26T18:43:20.507 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-03-26T18:43:20.533 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): ceph-test-18.2.0-0.el9.x86_64.rpm 33 MB/s | 40 MB 00:01 2024-03-26T18:43:20.535 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:43:20.536 INFO:teuthology.orchestra.run.smithi102.stdout:Total 33 MB/s | 40 MB 00:01 2024-03-26T18:43:20.538 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:18.2.0-0.el9.x86_64 21/32 2024-03-26T18:43:20.549 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:43:20.569 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:43:20.569 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:43:20.584 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-26T18:43:20.912 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:43:20.912 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:43:20.978 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-26T18:43:21.011 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-03-26T18:43:21.065 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-03-26T18:43:21.132 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:18.2.0-0.el9.x86_64 25/32 2024-03-26T18:43:21.163 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:18.2.0-0.el9.x86_64 26/32 2024-03-26T18:43:21.450 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:43:21.531 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T18:43:21.581 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T18:43:21.614 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T18:43:21.658 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T18:43:21.685 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:18.2.0-0.el9.x86_64 5/6 2024-03-26T18:43:23.283 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-03-26T18:43:23.319 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-03-26T18:43:23.873 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-03-26T18:43:23.938 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-03-26T18:43:24.031 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-03-26T18:43:24.031 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T18:43:24.032 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T18:43:24.032 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:24.064 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-03-26T18:43:25.407 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:18.2.0-0.el9.x86_64 5/6 2024-03-26T18:43:25.494 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:18.2.0-0.el9.x86_64 6/6 2024-03-26T18:43:27.600 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-test-2:18.2.0-0.el9.x86_64 6/6 2024-03-26T18:43:27.600 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:18.2.0-0.el9.x86_64 1/6 2024-03-26T18:43:27.600 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:18.2.0-0.el9.x86_64 2/6 2024-03-26T18:43:27.600 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T18:43:27.600 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T18:43:27.600 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T18:43:27.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T18:43:27.975 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:27.975 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:43:27.975 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:18.2.0-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-03-26T18:43:27.976 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:18.2.0-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T18:43:27.976 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T18:43:27.976 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:27.976 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:43:28.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph 2024-03-26T18:43:28.858 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:35 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:43:29.014 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:43:29.017 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:29.017 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-26T18:43:29.017 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:29.017 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:43:29.017 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:18.2.0-0.el9 ceph 6.4 k 2024-03-26T18:43:29.017 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-26T18:43:29.017 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:18.2.0-0.el9 ceph 2.1 M 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:18.2.0-0.el9 ceph 1.4 M 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:18.2.0-0.el9 ceph-noarch 240 k 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:18.2.0-0.el9 ceph 18 M 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T18:43:29.018 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T18:43:29.019 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T18:43:29.020 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T18:43:29.020 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T18:43:29.020 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T18:43:29.020 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T18:43:29.020 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T18:43:29.020 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:29.020 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:43:29.020 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:29.020 INFO:teuthology.orchestra.run.smithi102.stdout:Install 33 Packages 2024-03-26T18:43:29.020 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:29.021 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 30 M 2024-03-26T18:43:29.021 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 109 M 2024-03-26T18:43:29.021 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:43:29.245 INFO:teuthology.orchestra.run.smithi102.stdout:(1/33): ceph-18.2.0-0.el9.x86_64.rpm 53 kB/s | 6.4 kB 00:00 2024-03-26T18:43:29.320 INFO:teuthology.orchestra.run.smithi102.stdout:(2/33): ceph-mds-18.2.0-0.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2024-03-26T18:43:29.471 INFO:teuthology.orchestra.run.smithi102.stdout:(3/33): ceph-mon-18.2.0-0.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-03-26T18:43:29.504 INFO:teuthology.orchestra.run.smithi102.stdout:(4/33): ceph-mgr-18.2.0-0.el9.x86_64.rpm 3.8 MB/s | 1.4 MB 00:00 2024-03-26T18:43:29.529 INFO:teuthology.orchestra.run.smithi102.stdout:(5/33): ceph-mgr-modules-core-18.2.0-0.el9.noar 4.0 MB/s | 240 kB 00:00 2024-03-26T18:43:29.730 INFO:teuthology.orchestra.run.smithi102.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 158 kB/s | 32 kB 00:00 2024-03-26T18:43:29.764 INFO:teuthology.orchestra.run.smithi102.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 557 kB/s | 144 kB 00:00 2024-03-26T18:43:29.797 INFO:teuthology.orchestra.run.smithi102.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 708 kB/s | 47 kB 00:00 2024-03-26T18:43:29.822 INFO:teuthology.orchestra.run.smithi102.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 661 kB/s | 39 kB 00:00 2024-03-26T18:43:30.006 INFO:teuthology.orchestra.run.smithi102.stdout:(10/33): ceph-osd-18.2.0-0.el9.x86_64.rpm 26 MB/s | 18 MB 00:00 2024-03-26T18:43:30.031 INFO:teuthology.orchestra.run.smithi102.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 241 kB 00:00 2024-03-26T18:43:30.190 INFO:teuthology.orchestra.run.smithi102.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 3.1 MB/s | 1.1 MB 00:00 2024-03-26T18:43:30.215 INFO:teuthology.orchestra.run.smithi102.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 495 kB/s | 103 kB 00:00 2024-03-26T18:43:30.240 INFO:teuthology.orchestra.run.smithi102.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 594 kB/s | 124 kB 00:00 2024-03-26T18:43:30.266 INFO:teuthology.orchestra.run.smithi102.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-03-26T18:43:30.641 INFO:teuthology.orchestra.run.smithi102.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 109 kB/s | 43 kB 00:00 2024-03-26T18:43:30.841 INFO:teuthology.orchestra.run.smithi102.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 299 kB/s | 172 kB 00:00 2024-03-26T18:43:30.967 INFO:teuthology.orchestra.run.smithi102.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-03-26T18:43:31.075 INFO:teuthology.orchestra.run.smithi102.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 827 kB/s | 358 kB 00:00 2024-03-26T18:43:31.101 INFO:teuthology.orchestra.run.smithi102.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2024-03-26T18:43:31.184 INFO:teuthology.orchestra.run.smithi102.stdout:(21/33): python3-requests-2.25.1-8.el9.noarch.r 129 kB/s | 125 kB 00:00 2024-03-26T18:43:31.209 INFO:teuthology.orchestra.run.smithi102.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 173 kB/s | 23 kB 00:00 2024-03-26T18:43:31.235 INFO:teuthology.orchestra.run.smithi102.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 145 kB/s | 19 kB 00:00 2024-03-26T18:43:31.318 INFO:teuthology.orchestra.run.smithi102.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2024-03-26T18:43:31.369 INFO:teuthology.orchestra.run.smithi102.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 591 kB/s | 79 kB 00:00 2024-03-26T18:43:31.444 INFO:teuthology.orchestra.run.smithi102.stdout:(26/33): python3-jaraco-text-3.2.0-6.el9.noarch 76 kB/s | 20 kB 00:00 2024-03-26T18:43:31.469 INFO:teuthology.orchestra.run.smithi102.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-26T18:43:31.494 INFO:teuthology.orchestra.run.smithi102.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-03-26T18:43:31.570 INFO:teuthology.orchestra.run.smithi102.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-03-26T18:43:31.678 INFO:teuthology.orchestra.run.smithi102.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-26T18:43:31.720 INFO:teuthology.orchestra.run.smithi102.stdout:(31/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 326 kB/s | 90 kB 00:00 2024-03-26T18:43:31.745 INFO:teuthology.orchestra.run.smithi102.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-03-26T18:43:31.779 INFO:teuthology.orchestra.run.smithi102.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-26T18:43:31.783 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:43:31.783 INFO:teuthology.orchestra.run.smithi102.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-26T18:43:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:43:31.837 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:43:31.837 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:43:32.204 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:43:32.204 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:43:33.433 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:43:33.506 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-26T18:43:33.539 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-26T18:43:33.572 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-26T18:43:33.664 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-26T18:43:33.670 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-03-26T18:43:33.670 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-03-26T18:43:33.670 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-03-26T18:43:33.670 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-03-26T18:43:33.670 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-03-26T18:43:33.671 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-03-26T18:43:33.671 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-03-26T18:43:33.671 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-03-26T18:43:33.671 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:33.750 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-26T18:43:33.790 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-26T18:43:33.862 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-26T18:43:33.917 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-26T18:43:33.963 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-26T18:43:34.012 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-26T18:43:34.075 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-26T18:43:34.267 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-26T18:43:34.326 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-26T18:43:34.373 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-26T18:43:34.422 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-26T18:43:34.429 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-03-26T18:43:34.454 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-26T18:43:34.454 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-03-26T18:43:34.454 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:43:34.454 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T18:43:34.455 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T18:43:34.455 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T18:43:34.455 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:34.456 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-26T18:43:34.530 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-26T18:43:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-26T18:43:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T18:43:34.564 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-26T18:43:34.598 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-26T18:43:34.638 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-26T18:43:34.673 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-26T18:43:34.707 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-26T18:43:34.796 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-26T18:43:34.838 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-26T18:43:34.911 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-26T18:43:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-26T18:43:35.172 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-26T18:43:35.328 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 28/33 2024-03-26T18:43:35.351 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-03-26T18:43:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-03-26T18:43:35.377 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:43:35.377 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T18:43:35.377 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T18:43:35.377 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T18:43:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T18:43:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:18.2.0-0.el9.x86_64 1/32 2024-03-26T18:43:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:18.2.0-0.el9.x86_64 2/32 2024-03-26T18:43:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:18.2.0-0.el9.x86_64 3/32 2024-03-26T18:43:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:18.2.0-0.el9.x86_64 4/32 2024-03-26T18:43:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:18.2.0-0.el9.x86_64 5/32 2024-03-26T18:43:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:18.2.0-0.el9.x86_64 6/32 2024-03-26T18:43:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:18.2.0-0.el9.x86_64 7/32 2024-03-26T18:43:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 8/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:18.2.0-0.el9.x86_64 9/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:18.2.0-0.el9.x86_64 11/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:18.2.0-0.el9.x86_64 12/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:18.2.0-0.el9.x86_64 13/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-26T18:43:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-26T18:43:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-26T18:43:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-26T18:43:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-26T18:43:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-26T18:43:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-26T18:43:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:18.2.0-0.el9.x86_64 29/32 2024-03-26T18:43:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-26T18:43:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:18.2.0-0.el9.x86_64 31/32 2024-03-26T18:43:36.817 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T18:43:36.817 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:18.2.0-0.el9.x86_64 librbd1-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T18:43:36.818 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.819 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T18:43:36.820 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.820 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.820 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:36.820 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T18:43:36.820 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T18:43:36.820 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:36.820 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:43:36.945 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-03-26T18:43:36.974 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-03-26T18:43:36.974 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:43:36.974 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T18:43:36.974 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T18:43:36.974 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T18:43:36.974 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:37.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test 2024-03-26T18:43:37.429 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-03-26T18:43:37.454 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-03-26T18:43:37.454 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:43:37.454 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T18:43:37.454 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T18:43:37.454 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T18:43:37.454 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:37.655 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-03-26T18:43:37.678 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-03-26T18:43:37.678 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:43:37.678 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T18:43:37.679 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T18:43:37.679 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T18:43:37.679 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:37.720 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:27 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:43:37.744 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:18.2.0-0.el9.x86_64 33/33 2024-03-26T18:43:37.868 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:43:37.869 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:43:37.869 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-26T18:43:37.869 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:43:37.869 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:18.2.0-0.el9 ceph 40 M 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:18.2.0-0.el9 ceph 161 k 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-03-26T18:43:37.870 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:37.871 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 40 M 2024-03-26T18:43:37.871 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 167 M 2024-03-26T18:43:37.871 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:43:38.034 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): libcephsqlite-18.2.0-0.el9.x86_64.rpm 1.5 MB/s | 161 kB 00:00 2024-03-26T18:43:38.143 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 300 kB 00:00 2024-03-26T18:43:38.169 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 902 kB/s | 217 kB 00:00 2024-03-26T18:43:38.194 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-26T18:43:38.577 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): ceph-test-18.2.0-0.el9.x86_64.rpm 61 MB/s | 40 MB 00:00 2024-03-26T18:43:38.927 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): jq-1.6-16.el9.x86_64.rpm 247 kB/s | 187 kB 00:00 2024-03-26T18:43:38.930 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:43:38.930 INFO:teuthology.orchestra.run.smithi190.stdout:Total 38 MB/s | 40 MB 00:01 2024-03-26T18:43:38.945 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:43:38.965 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:43:38.966 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:43:39.320 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:43:39.320 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:43:39.456 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-2:18.2.0-0.el9.x86_64 33/33 2024-03-26T18:43:39.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:18.2.0-0.el9.x86_64 1/33 2024-03-26T18:43:39.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:18.2.0-0.el9.x86_64 2/33 2024-03-26T18:43:39.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:18.2.0-0.el9.x86_64 3/33 2024-03-26T18:43:39.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:18.2.0-0.el9.x86_64 4/33 2024-03-26T18:43:39.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:18.2.0-0.el9.x86_64 5/33 2024-03-26T18:43:39.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 6/33 2024-03-26T18:43:39.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-26T18:43:39.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-26T18:43:39.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-26T18:43:39.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-26T18:43:39.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-26T18:43:39.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-26T18:43:39.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-26T18:43:39.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-26T18:43:39.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-26T18:43:39.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-26T18:43:39.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-26T18:43:39.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-26T18:43:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:43:39.902 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T18:43:39.948 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T18:43:39.996 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T18:43:40.044 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T18:43:40.072 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:18.2.0-0.el9.x86_64 5/6 2024-03-26T18:43:40.381 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-26T18:43:40.381 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:40.381 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:43:40.381 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:40.381 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:40.381 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:40.381 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 2024-03-26T18:43:40.382 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:40.382 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:40.382 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T18:43:40.382 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T18:43:40.382 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T18:43:40.382 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T18:43:40.382 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T18:43:40.382 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T18:43:40.382 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T18:43:40.382 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T18:43:40.382 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T18:43:40.383 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T18:43:40.384 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T18:43:40.384 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T18:43:40.384 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T18:43:40.384 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:40.384 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:43:40.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-base 2024-03-26T18:43:41.283 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:48 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:43:41.380 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-base-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:43:41.431 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:43:41.432 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-26T18:43:41.432 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:43:41.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install cephadm 2024-03-26T18:43:42.058 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:49 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:43:42.205 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:43:42.206 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:42.206 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-26T18:43:42.206 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:42.206 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:43:42.206 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:18.2.0-0.el9 ceph-noarch 209 k 2024-03-26T18:43:42.206 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:42.206 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:43:42.206 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:42.206 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-26T18:43:42.206 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:42.207 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 209 k 2024-03-26T18:43:42.207 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 200 k 2024-03-26T18:43:42.207 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:43:42.295 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm-18.2.0-0.el9.noarch.rpm 2.3 MB/s | 209 kB 00:00 2024-03-26T18:43:42.296 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:43:42.296 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.3 MB/s | 209 kB 00:00 2024-03-26T18:43:42.296 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:43:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:43:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:43:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:43:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:43:42.375 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:43:42.788 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-03-26T18:43:42.875 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-03-26T18:43:43.262 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-03-26T18:43:43.549 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-03-26T18:43:43.549 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:43.549 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:43:43.549 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:18.2.0-0.el9.noarch 2024-03-26T18:43:43.549 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:43.549 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:43:43.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T18:43:43.801 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:18.2.0-0.el9.x86_64 5/6 2024-03-26T18:43:43.887 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:18.2.0-0.el9.x86_64 6/6 2024-03-26T18:43:44.342 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:51 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:43:44.495 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:43:44.495 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:44.495 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-26T18:43:44.495 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:44.496 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:43:44.496 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-0.el9 ceph 142 k 2024-03-26T18:43:44.496 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:44.496 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:43:44.496 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:44.496 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-26T18:43:44.496 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:44.496 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 142 k 2024-03-26T18:43:44.496 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 399 k 2024-03-26T18:43:44.496 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:43:44.586 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-immutable-object-cache-18.2.0-0.el9.x86_64 1.6 MB/s | 142 kB 00:00 2024-03-26T18:43:44.587 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:43:44.587 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.5 MB/s | 142 kB 00:00 2024-03-26T18:43:44.587 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:43:44.594 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:43:44.594 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:43:44.646 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:43:44.647 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:43:44.783 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:43:44.845 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:43:44.869 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:43:44.870 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:43:44.870 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T18:43:44.870 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:45.857 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:43:45.857 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:45.857 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:43:45.857 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 2024-03-26T18:43:45.857 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:45.858 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:43:45.996 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-2:18.2.0-0.el9.x86_64 6/6 2024-03-26T18:43:45.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:18.2.0-0.el9.x86_64 1/6 2024-03-26T18:43:45.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:18.2.0-0.el9.x86_64 2/6 2024-03-26T18:43:45.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T18:43:45.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T18:43:45.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T18:43:46.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr 2024-03-26T18:43:46.403 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T18:43:46.403 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:46.404 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:43:46.404 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:18.2.0-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-03-26T18:43:46.404 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:18.2.0-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T18:43:46.404 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T18:43:46.404 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:46.404 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:43:46.684 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:53 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:43:46.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph 2024-03-26T18:43:46.782 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-mgr-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:43:46.833 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:43:46.834 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-26T18:43:46.834 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:43:46.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T18:43:47.276 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:37 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:43:47.430 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:43:47.432 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:43:47.432 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:18.2.0-0.el9 ceph 6.4 k 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:18.2.0-0.el9 ceph 2.1 M 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:18.2.0-0.el9 ceph 1.4 M 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:18.2.0-0.el9 ceph-noarch 240 k 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:18.2.0-0.el9 ceph 18 M 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T18:43:47.433 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T18:43:47.434 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:47.435 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:43:47.436 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:43:47.436 INFO:teuthology.orchestra.run.smithi190.stdout:Install 33 Packages 2024-03-26T18:43:47.436 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:47.436 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 30 M 2024-03-26T18:43:47.436 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 109 M 2024-03-26T18:43:47.436 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:43:47.466 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:54 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:43:47.615 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:43:47.616 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-26T18:43:47.616 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:47.616 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:43:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:18.2.0-0.el9 ceph-noarch 1.7 M 2024-03-26T18:43:47.616 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-26T18:43:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:18.2.0-0.el9 ceph-noarch 23 k 2024-03-26T18:43:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:18.2.0-0.el9 ceph-noarch 15 k 2024-03-26T18:43:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-26T18:43:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T18:43:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T18:43:47.617 INFO:teuthology.orchestra.run.smithi102.stdout:Installing weak dependencies: 2024-03-26T18:43:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-26T18:43:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:47.617 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:43:47.617 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:47.617 INFO:teuthology.orchestra.run.smithi102.stdout:Install 7 Packages 2024-03-26T18:43:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:47.618 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 2.0 M 2024-03-26T18:43:47.618 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 8.1 M 2024-03-26T18:43:47.618 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:43:47.815 INFO:teuthology.orchestra.run.smithi102.stdout:(1/7): ceph-grafana-dashboards-18.2.0-0.el9.noa 192 kB/s | 23 kB 00:00 2024-03-26T18:43:47.874 INFO:teuthology.orchestra.run.smithi102.stdout:(2/7): ceph-mgr-dashboard-18.2.0-0.el9.noarch.r 9.5 MB/s | 1.7 MB 00:00 2024-03-26T18:43:47.889 INFO:teuthology.orchestra.run.smithi190.stdout:(1/33): ceph-18.2.0-0.el9.x86_64.rpm 46 kB/s | 6.4 kB 00:00 2024-03-26T18:43:47.890 INFO:teuthology.orchestra.run.smithi102.stdout:(3/7): ceph-prometheus-alerts-18.2.0-0.el9.noar 75 kB/s | 15 kB 00:00 2024-03-26T18:43:47.931 INFO:teuthology.orchestra.run.smithi190.stdout:(2/33): ceph-mgr-18.2.0-0.el9.x86_64.rpm 7.9 MB/s | 1.4 MB 00:00 2024-03-26T18:43:47.981 INFO:teuthology.orchestra.run.smithi190.stdout:(3/33): ceph-mds-18.2.0-0.el9.x86_64.rpm 9.0 MB/s | 2.1 MB 00:00 2024-03-26T18:43:48.007 INFO:teuthology.orchestra.run.smithi190.stdout:(4/33): ceph-mgr-modules-core-18.2.0-0.el9.noar 9.4 MB/s | 240 kB 00:00 2024-03-26T18:43:48.082 INFO:teuthology.orchestra.run.smithi190.stdout:(5/33): ceph-mon-18.2.0-0.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-03-26T18:43:48.166 INFO:teuthology.orchestra.run.smithi102.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 26 kB/s | 9.0 kB 00:00 2024-03-26T18:43:48.208 INFO:teuthology.orchestra.run.smithi102.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 97 kB/s | 31 kB 00:00 2024-03-26T18:43:48.224 INFO:teuthology.orchestra.run.smithi102.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 117 kB/s | 41 kB 00:00 2024-03-26T18:43:48.340 INFO:teuthology.orchestra.run.smithi190.stdout:(6/33): ceph-osd-18.2.0-0.el9.x86_64.rpm 43 MB/s | 18 MB 00:00 2024-03-26T18:43:48.450 INFO:teuthology.orchestra.run.smithi190.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 92 kB/s | 32 kB 00:00 2024-03-26T18:43:48.500 INFO:teuthology.orchestra.run.smithi190.stdout:(8/33): python3-mako-1.1.4-6.el9.noarch.rpm 293 kB/s | 144 kB 00:00 2024-03-26T18:43:48.542 INFO:teuthology.orchestra.run.smithi102.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 502 kB/s | 188 kB 00:00 2024-03-26T18:43:48.547 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:43:48.547 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-26T18:43:48.547 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:43:48.554 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:43:48.555 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:43:48.558 INFO:teuthology.orchestra.run.smithi190.stdout:(9/33): python3-pytz-2021.1-5.el9.noarch.rpm 219 kB/s | 47 kB 00:00 2024-03-26T18:43:48.602 INFO:teuthology.orchestra.run.smithi190.stdout:(10/33): python3-toml-0.10.2-6.el9.noarch.rpm 257 kB/s | 39 kB 00:00 2024-03-26T18:43:48.617 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:43:48.617 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:43:48.851 INFO:teuthology.orchestra.run.smithi190.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 687 kB/s | 241 kB 00:00 2024-03-26T18:43:48.944 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:43:49.001 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-26T18:43:49.036 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-26T18:43:49.090 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-26T18:43:49.122 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-26T18:43:49.159 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 5/7 2024-03-26T18:43:49.387 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 6/7 2024-03-26T18:43:49.412 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-03-26T18:43:49.575 INFO:teuthology.orchestra.run.smithi190.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 105 kB/s | 103 kB 00:00 2024-03-26T18:43:49.601 INFO:teuthology.orchestra.run.smithi190.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 165 kB/s | 124 kB 00:00 2024-03-26T18:43:49.651 INFO:teuthology.orchestra.run.smithi190.stdout:(14/33): python3-cryptography-36.0.1-4.el9.x86_ 1.0 MB/s | 1.1 MB 00:01 2024-03-26T18:43:49.676 INFO:teuthology.orchestra.run.smithi190.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.8 MB/s | 187 kB 00:00 2024-03-26T18:43:49.785 INFO:teuthology.orchestra.run.smithi190.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 681 kB/s | 125 kB 00:00 2024-03-26T18:43:49.810 INFO:teuthology.orchestra.run.smithi190.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 274 kB/s | 43 kB 00:00 2024-03-26T18:43:49.910 INFO:teuthology.orchestra.run.smithi190.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 825 kB/s | 172 kB 00:00 2024-03-26T18:43:49.944 INFO:teuthology.orchestra.run.smithi190.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-03-26T18:43:49.969 INFO:teuthology.orchestra.run.smithi190.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 67 kB/s | 11 kB 00:00 2024-03-26T18:43:49.987 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-03-26T18:43:49.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 1/7 2024-03-26T18:43:49.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2/7 2024-03-26T18:43:49.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 3/7 2024-03-26T18:43:49.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-26T18:43:49.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-26T18:43:49.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-26T18:43:50.011 INFO:teuthology.orchestra.run.smithi190.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 427 kB/s | 18 kB 00:00 2024-03-26T18:43:50.036 INFO:teuthology.orchestra.run.smithi190.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 348 kB/s | 23 kB 00:00 2024-03-26T18:43:50.061 INFO:teuthology.orchestra.run.smithi190.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-26T18:43:50.270 INFO:teuthology.orchestra.run.smithi190.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 76 kB/s | 20 kB 00:00 2024-03-26T18:43:50.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-26T18:43:50.290 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:50.290 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:43:50.290 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 2024-03-26T18:43:50.290 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2024-03-26T18:43:50.290 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 2024-03-26T18:43:50.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-26T18:43:50.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-26T18:43:50.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T18:43:50.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T18:43:50.290 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:50.291 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:43:50.295 INFO:teuthology.orchestra.run.smithi190.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 198 kB/s | 46 kB 00:00 2024-03-26T18:43:50.320 INFO:teuthology.orchestra.run.smithi190.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 305 kB/s | 79 kB 00:00 2024-03-26T18:43:50.346 INFO:teuthology.orchestra.run.smithi190.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-26T18:43:50.371 INFO:teuthology.orchestra.run.smithi190.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 331 kB/s | 16 kB 00:00 2024-03-26T18:43:50.396 INFO:teuthology.orchestra.run.smithi190.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-26T18:43:50.421 INFO:teuthology.orchestra.run.smithi190.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 476 kB/s | 36 kB 00:00 2024-03-26T18:43:50.447 INFO:teuthology.orchestra.run.smithi190.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-26T18:43:50.510 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T18:43:51.065 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:58 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:43:51.212 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:43:51.213 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-0.el9 ceph-noarch 7.4 M 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T18:43:51.214 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T18:43:51.215 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T18:43:51.215 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T18:43:51.215 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T18:43:51.215 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T18:43:51.215 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:51.215 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:43:51.215 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:43:51.215 INFO:teuthology.orchestra.run.smithi102.stdout:Install 12 Packages 2024-03-26T18:43:51.215 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:43:51.216 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 37 M 2024-03-26T18:43:51.216 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 209 M 2024-03-26T18:43:51.216 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:43:51.557 INFO:teuthology.orchestra.run.smithi102.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 193 kB/s | 32 kB 00:00 2024-03-26T18:43:51.599 INFO:teuthology.orchestra.run.smithi102.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 414 kB/s | 17 kB 00:00 2024-03-26T18:43:51.657 INFO:teuthology.orchestra.run.smithi102.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 589 kB/s | 34 kB 00:00 2024-03-26T18:43:51.774 INFO:teuthology.orchestra.run.smithi102.stdout:(4/12): ceph-mgr-diskprediction-local-18.2.0-0. 19 MB/s | 7.4 MB 00:00 2024-03-26T18:43:51.841 INFO:teuthology.orchestra.run.smithi102.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.7 MB/s | 3.0 MB 00:00 2024-03-26T18:43:52.013 INFO:teuthology.orchestra.run.smithi190.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 268 kB/s | 427 kB 00:01 2024-03-26T18:43:52.038 INFO:teuthology.orchestra.run.smithi190.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 12 kB/s | 20 kB 00:01 2024-03-26T18:43:52.041 INFO:teuthology.orchestra.run.smithi102.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.1 MB/s | 217 kB 00:00 2024-03-26T18:43:52.045 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:43:52.045 INFO:teuthology.orchestra.run.smithi190.stdout:Total 6.5 MB/s | 30 MB 00:04 2024-03-26T18:43:52.058 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:43:52.099 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:43:52.100 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:43:52.324 INFO:teuthology.orchestra.run.smithi102.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.7 MB/s | 4.8 MB 00:00 2024-03-26T18:43:52.384 INFO:teuthology.orchestra.run.smithi102.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 6.5 MB/s | 4.7 MB 00:00 2024-03-26T18:43:52.451 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:43:52.451 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:43:52.859 INFO:teuthology.orchestra.run.smithi102.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-03-26T18:43:52.934 INFO:teuthology.orchestra.run.smithi102.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 357 kB/s | 195 kB 00:00 2024-03-26T18:43:53.084 INFO:teuthology.orchestra.run.smithi102.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-26T18:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 846 kB/s | 803 kB 00:00 2024-03-26T18:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:43:53.280 INFO:teuthology.orchestra.run.smithi102.stdout:Total 18 MB/s | 37 MB 00:02 2024-03-26T18:43:53.321 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:43:53.345 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:43:53.345 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:43:53.709 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:43:53.769 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:43:53.770 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:43:53.791 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-26T18:43:53.823 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-26T18:43:53.856 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-26T18:43:53.946 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-26T18:43:54.025 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-26T18:43:54.067 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-26T18:43:54.138 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-26T18:43:54.192 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-26T18:43:54.231 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-26T18:43:54.279 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-26T18:43:54.345 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-26T18:43:54.482 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:43:54.531 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-26T18:43:54.558 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T18:43:54.589 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T18:43:54.593 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-26T18:43:54.624 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-26T18:43:54.658 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-26T18:43:54.684 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T18:43:54.688 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-26T18:43:54.717 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T18:43:54.764 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-26T18:43:54.799 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-26T18:43:54.834 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-26T18:43:54.872 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-26T18:43:54.887 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T18:43:54.907 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-26T18:43:54.916 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T18:43:54.934 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-26T18:43:54.993 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T18:43:55.025 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-26T18:43:55.065 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-26T18:43:55.136 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-26T18:43:55.271 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-26T18:43:55.387 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-26T18:43:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 28/33 2024-03-26T18:43:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-03-26T18:43:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-03-26T18:43:55.586 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:43:55.587 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T18:43:55.587 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T18:43:55.587 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T18:43:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:55.661 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T18:43:55.778 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T18:43:57.035 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T18:43:57.130 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-03-26T18:43:57.157 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-03-26T18:43:57.157 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:43:57.157 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T18:43:57.158 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T18:43:57.158 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T18:43:57.158 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:57.606 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-03-26T18:43:57.629 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-03-26T18:43:57.629 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:43:57.629 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T18:43:57.630 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T18:43:57.630 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T18:43:57.630 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:57.831 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-03-26T18:43:57.854 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-03-26T18:43:57.854 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:43:57.854 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T18:43:57.854 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T18:43:57.854 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T18:43:57.855 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:43:57.920 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:18.2.0-0.el9.x86_64 33/33 2024-03-26T18:43:58.066 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T18:43:58.091 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-03-26T18:43:59.654 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-2:18.2.0-0.el9.x86_64 33/33 2024-03-26T18:43:59.654 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:18.2.0-0.el9.x86_64 1/33 2024-03-26T18:43:59.654 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:18.2.0-0.el9.x86_64 2/33 2024-03-26T18:43:59.655 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:18.2.0-0.el9.x86_64 3/33 2024-03-26T18:43:59.655 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:18.2.0-0.el9.x86_64 4/33 2024-03-26T18:43:59.655 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:18.2.0-0.el9.x86_64 5/33 2024-03-26T18:43:59.655 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 6/33 2024-03-26T18:43:59.655 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-26T18:43:59.655 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-26T18:43:59.655 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-26T18:43:59.655 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-26T18:43:59.655 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-26T18:43:59.656 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-26T18:43:59.657 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-26T18:43:59.657 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-26T18:43:59.657 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-26T18:43:59.657 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-26T18:43:59.657 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-26T18:43:59.657 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-26T18:43:59.657 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-26T18:44:00.523 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-03-26T18:44:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 1/12 2024-03-26T18:44:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T18:44:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T18:44:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T18:44:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T18:44:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T18:44:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T18:44:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T18:44:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T18:44:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T18:44:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T18:44:00.590 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-26T18:44:00.590 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T18:44:00.591 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T18:44:00.592 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T18:44:00.593 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T18:44:00.593 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T18:44:00.593 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T18:44:00.593 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T18:44:00.593 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T18:44:00.593 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T18:44:00.593 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T18:44:00.593 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:00.593 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:00.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base 2024-03-26T18:44:01.015 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T18:44:01.015 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarch 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T18:44:01.016 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T18:44:01.017 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:01.017 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:01.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-rook 2024-03-26T18:44:01.501 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:51 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:01.597 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:01.649 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:01.650 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-26T18:44:01.650 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:01.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install cephadm 2024-03-26T18:44:01.968 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:08 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:02.119 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:02.121 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:02.121 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-26T18:44:02.121 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:02.121 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:44:02.121 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:18.2.0-0.el9 ceph-noarch 47 k 2024-03-26T18:44:02.121 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-26T18:44:02.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T18:44:02.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T18:44:02.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T18:44:02.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T18:44:02.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T18:44:02.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T18:44:02.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T18:44:02.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T18:44:02.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T18:44:02.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T18:44:02.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T18:44:02.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T18:44:02.122 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:02.122 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:44:02.122 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:02.122 INFO:teuthology.orchestra.run.smithi102.stdout:Install 13 Packages 2024-03-26T18:44:02.123 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:02.123 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 2.0 M 2024-03-26T18:44:02.123 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 26 M 2024-03-26T18:44:02.123 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:44:02.282 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:52 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:02.283 INFO:teuthology.orchestra.run.smithi102.stdout:(1/13): ceph-mgr-rook-18.2.0-0.el9.noarch.rpm 420 kB/s | 47 kB 00:00 2024-03-26T18:44:02.299 INFO:teuthology.orchestra.run.smithi102.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 188 kB/s | 24 kB 00:00 2024-03-26T18:44:02.316 INFO:teuthology.orchestra.run.smithi102.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-26T18:44:02.375 INFO:teuthology.orchestra.run.smithi102.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-03-26T18:44:02.392 INFO:teuthology.orchestra.run.smithi102.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-26T18:44:02.417 INFO:teuthology.orchestra.run.smithi102.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.0 MB/s | 211 kB 00:00 2024-03-26T18:44:02.430 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:02.431 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:02.431 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:02.431 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:02.431 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:02.431 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:18.2.0-0.el9 ceph-noarch 209 k 2024-03-26T18:44:02.431 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:02.431 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:02.431 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:02.431 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-26T18:44:02.431 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:02.431 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 209 k 2024-03-26T18:44:02.432 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 200 k 2024-03-26T18:44:02.432 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:02.434 INFO:teuthology.orchestra.run.smithi102.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 730 kB/s | 43 kB 00:00 2024-03-26T18:44:02.497 INFO:teuthology.orchestra.run.smithi190.stdout:cephadm-18.2.0-0.el9.noarch.rpm 3.1 MB/s | 209 kB 00:00 2024-03-26T18:44:02.497 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:02.498 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.1 MB/s | 209 kB 00:00 2024-03-26T18:44:02.498 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:02.503 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:02.503 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:02.505 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:02.506 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:02.575 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:02.626 INFO:teuthology.orchestra.run.smithi102.stdout:(8/13): python3-google-auth-2.28.2-1.el9.noarch 1.0 MB/s | 203 kB 00:00 2024-03-26T18:44:02.710 INFO:teuthology.orchestra.run.smithi102.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2024-03-26T18:44:02.726 INFO:teuthology.orchestra.run.smithi102.stdout:(10/13): python3-certifi-2021.10.8-2.el9.noarch 48 kB/s | 15 kB 00:00 2024-03-26T18:44:02.843 INFO:teuthology.orchestra.run.smithi102.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-03-26T18:44:02.877 INFO:teuthology.orchestra.run.smithi102.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 707 kB/s | 90 kB 00:00 2024-03-26T18:44:02.911 INFO:teuthology.orchestra.run.smithi102.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2024-03-26T18:44:02.913 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:02.914 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-26T18:44:02.914 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:44:02.926 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:44:02.926 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:44:02.973 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-03-26T18:44:03.015 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:44:03.015 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:44:03.059 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-03-26T18:44:03.388 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-03-26T18:44:03.541 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:44:03.651 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-03-26T18:44:03.651 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:03.651 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:03.651 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:18.2.0-0.el9.noarch 2024-03-26T18:44:03.652 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:03.652 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:03.716 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T18:44:03.762 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T18:44:03.817 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T18:44:03.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T18:44:03.854 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T18:44:03.896 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T18:44:03.989 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T18:44:04.099 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T18:44:04.155 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T18:44:04.405 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:54 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:04.560 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:04.561 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:04.561 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-26T18:44:04.561 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:04.561 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:04.561 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-0.el9 ceph 142 k 2024-03-26T18:44:04.561 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:04.561 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:04.561 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:04.561 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-26T18:44:04.562 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:04.562 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 142 k 2024-03-26T18:44:04.562 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 399 k 2024-03-26T18:44:04.562 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:04.622 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-18.2.0-0.el9.x86_64 2.3 MB/s | 142 kB 00:00 2024-03-26T18:44:04.623 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:04.623 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.2 MB/s | 142 kB 00:00 2024-03-26T18:44:04.623 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:04.631 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:04.631 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:04.682 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:04.682 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:04.823 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:04.871 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T18:44:04.882 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:04.906 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:04.906 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:44:04.906 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T18:44:04.906 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:04.919 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T18:44:04.953 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T18:44:04.999 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T18:44:05.023 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-03-26T18:44:05.863 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:05.863 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:05.863 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:05.863 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:05.863 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:05.863 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:05.867 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-03-26T18:44:05.867 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:18.2.0-0.el9.noarch 1/13 2024-03-26T18:44:05.867 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T18:44:05.867 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T18:44:05.868 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T18:44:05.868 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T18:44:05.868 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T18:44:05.868 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T18:44:05.868 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T18:44:05.868 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T18:44:05.868 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T18:44:05.868 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T18:44:05.868 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T18:44:06.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr 2024-03-26T18:44:06.241 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T18:44:06.241 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:18.2.0-0.el9.noarch 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T18:44:06.242 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T18:44:06.243 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T18:44:06.243 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T18:44:06.243 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:06.243 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:06.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T18:44:06.707 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:56 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:06.805 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-mgr-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:06.860 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:06.861 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-26T18:44:06.861 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:06.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T18:44:07.090 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:14 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:07.241 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:07.242 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:07.242 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-26T18:44:07.242 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:07.242 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:44:07.242 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:18.2.0-0.el9 ceph-noarch 127 k 2024-03-26T18:44:07.242 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-26T18:44:07.242 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T18:44:07.243 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T18:44:07.243 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T18:44:07.243 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T18:44:07.243 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T18:44:07.243 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:07.243 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:44:07.244 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:07.244 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2024-03-26T18:44:07.244 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:07.244 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 6.8 M 2024-03-26T18:44:07.244 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 33 M 2024-03-26T18:44:07.244 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:44:07.478 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:57 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:07.626 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:07.627 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:07.627 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-26T18:44:07.627 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:07.627 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:18.2.0-0.el9 ceph-noarch 1.7 M 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:18.2.0-0.el9 ceph-noarch 23 k 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:18.2.0-0.el9 ceph-noarch 15 k 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout:Install 7 Packages 2024-03-26T18:44:07.628 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:07.629 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 2.0 M 2024-03-26T18:44:07.629 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 8.1 M 2024-03-26T18:44:07.629 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:07.780 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): ceph-mgr-cephadm-18.2.0-0.el9.noarch.rpm 1.4 MB/s | 127 kB 00:00 2024-03-26T18:44:07.819 INFO:teuthology.orchestra.run.smithi190.stdout:(1/7): ceph-grafana-dashboards-18.2.0-0.el9.noa 196 kB/s | 23 kB 00:00 2024-03-26T18:44:07.830 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-03-26T18:44:07.843 INFO:teuthology.orchestra.run.smithi190.stdout:(2/7): ceph-prometheus-alerts-18.2.0-0.el9.noar 102 kB/s | 15 kB 00:00 2024-03-26T18:44:07.894 INFO:teuthology.orchestra.run.smithi190.stdout:(3/7): ceph-mgr-dashboard-18.2.0-0.el9.noarch.r 8.8 MB/s | 1.7 MB 00:00 2024-03-26T18:44:07.978 INFO:teuthology.orchestra.run.smithi190.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 57 kB/s | 9.0 kB 00:00 2024-03-26T18:44:08.005 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-26T18:44:08.019 INFO:teuthology.orchestra.run.smithi190.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 245 kB/s | 31 kB 00:00 2024-03-26T18:44:08.053 INFO:teuthology.orchestra.run.smithi190.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 196 kB/s | 41 kB 00:00 2024-03-26T18:44:08.145 INFO:teuthology.orchestra.run.smithi190.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2024-03-26T18:44:08.146 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:08.147 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.8 MB/s | 2.0 MB 00:00 2024-03-26T18:44:08.147 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:08.154 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:08.155 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:08.172 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 169 kB/s | 58 kB 00:00 2024-03-26T18:44:08.212 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:08.213 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:08.339 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 153 kB/s | 51 kB 00:00 2024-03-26T18:44:08.373 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 925 kB/s | 548 kB 00:00 2024-03-26T18:44:08.374 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:08.374 INFO:teuthology.orchestra.run.smithi102.stdout:Total 6.0 MB/s | 6.8 MB 00:01 2024-03-26T18:44:08.387 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:44:08.398 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:44:08.398 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:44:08.485 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:44:08.486 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:44:08.508 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:08.571 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-26T18:44:08.605 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-26T18:44:08.659 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-26T18:44:08.692 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-26T18:44:08.720 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 5/7 2024-03-26T18:44:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:44:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T18:44:08.949 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 6/7 2024-03-26T18:44:08.982 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-03-26T18:44:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T18:44:09.478 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-03-26T18:44:09.478 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 1/7 2024-03-26T18:44:09.478 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2/7 2024-03-26T18:44:09.478 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 3/7 2024-03-26T18:44:09.479 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-26T18:44:09.479 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-26T18:44:09.479 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-26T18:44:09.502 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T18:44:09.585 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T18:44:09.660 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T18:44:09.686 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-03-26T18:44:09.775 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-26T18:44:09.775 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:09.775 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:09.775 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 2024-03-26T18:44:09.775 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2024-03-26T18:44:09.775 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 2024-03-26T18:44:09.775 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-26T18:44:09.776 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-26T18:44:09.776 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T18:44:09.776 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T18:44:09.776 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:09.776 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:09.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T18:44:10.522 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-03-26T18:44:10.522 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 1/6 2024-03-26T18:44:10.522 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T18:44:10.522 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T18:44:10.522 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T18:44:10.522 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T18:44:10.533 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:10.682 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:10.683 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:10.683 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-26T18:44:10.683 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:10.683 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:10.683 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-0.el9 ceph-noarch 7.4 M 2024-03-26T18:44:10.683 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-26T18:44:10.683 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T18:44:10.683 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T18:44:10.683 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T18:44:10.683 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout:Install 12 Packages 2024-03-26T18:44:10.684 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:10.685 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 37 M 2024-03-26T18:44:10.685 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 209 M 2024-03-26T18:44:10.685 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:10.836 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T18:44:10.836 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:10.836 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:44:10.836 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 2024-03-26T18:44:10.837 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T18:44:10.837 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T18:44:10.837 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T18:44:10.837 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T18:44:10.837 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T18:44:10.837 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:10.837 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:10.894 INFO:teuthology.orchestra.run.smithi190.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2024-03-26T18:44:10.986 INFO:teuthology.orchestra.run.smithi190.stdout:(2/12): ceph-mgr-diskprediction-local-18.2.0-0. 33 MB/s | 7.4 MB 00:00 2024-03-26T18:44:11.011 INFO:teuthology.orchestra.run.smithi190.stdout:(3/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 147 kB/s | 17 kB 00:00 2024-03-26T18:44:11.036 INFO:teuthology.orchestra.run.smithi190.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-26T18:44:11.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-fuse 2024-03-26T18:44:11.228 INFO:teuthology.orchestra.run.smithi190.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-26T18:44:11.295 INFO:teuthology.orchestra.run.smithi190.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-26T18:44:11.320 INFO:teuthology.orchestra.run.smithi190.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-26T18:44:11.429 INFO:teuthology.orchestra.run.smithi190.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.5 MB/s | 3.0 MB 00:00 2024-03-26T18:44:11.546 INFO:teuthology.orchestra.run.smithi190.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-03-26T18:44:11.661 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:18 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:11.696 INFO:teuthology.orchestra.run.smithi190.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-26T18:44:11.721 INFO:teuthology.orchestra.run.smithi190.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 674 kB/s | 195 kB 00:00 2024-03-26T18:44:11.810 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:11.810 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:11.810 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:11.811 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:11.811 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:44:11.811 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:18.2.0-0.el9 ceph 835 k 2024-03-26T18:44:11.811 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-26T18:44:11.811 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T18:44:11.811 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:11.814 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:44:11.814 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:11.814 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2024-03-26T18:44:11.814 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:11.814 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 913 k 2024-03-26T18:44:11.814 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 2.7 M 2024-03-26T18:44:11.814 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:44:11.924 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): ceph-fuse-18.2.0-0.el9.x86_64.rpm 7.2 MB/s | 835 kB 00:00 2024-03-26T18:44:11.949 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 570 kB/s | 79 kB 00:00 2024-03-26T18:44:11.950 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:11.951 INFO:teuthology.orchestra.run.smithi102.stdout:Total 6.4 MB/s | 913 kB 00:00 2024-03-26T18:44:11.951 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:44:11.970 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:44:11.971 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:44:12.045 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:44:12.046 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:44:12.223 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:44:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T18:44:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-03-26T18:44:12.613 INFO:teuthology.orchestra.run.smithi190.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 230 kB/s | 245 kB 00:01 2024-03-26T18:44:12.617 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:12.617 INFO:teuthology.orchestra.run.smithi190.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-26T18:44:12.660 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:12.685 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:12.686 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:13.109 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:13.109 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-03-26T18:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:18.2.0-0.el9.x86_64 1/2 2024-03-26T18:44:13.470 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T18:44:13.470 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:13.471 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:44:13.471 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:18.2.0-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T18:44:13.471 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:13.471 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:13.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados-devel 2024-03-26T18:44:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T18:44:13.908 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T18:44:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T18:44:14.079 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T18:44:14.257 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T18:44:14.295 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T18:44:14.331 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:21 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:14.373 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T18:44:14.486 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:18.2.0-0.el9 ceph 126 k 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 126 k 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 456 k 2024-03-26T18:44:14.487 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:44:14.565 INFO:teuthology.orchestra.run.smithi102.stdout:librados-devel-18.2.0-0.el9.x86_64.rpm 1.6 MB/s | 126 kB 00:00 2024-03-26T18:44:14.565 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:14.565 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.6 MB/s | 126 kB 00:00 2024-03-26T18:44:14.566 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:44:14.569 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:44:14.569 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:44:14.630 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:44:14.631 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:44:14.769 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:44:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:15.044 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T18:44:15.148 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T18:44:15.282 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:15.637 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:44:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:15.637 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:15.953 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs2 2024-03-26T18:44:16.405 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T18:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:23 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:16.635 INFO:teuthology.orchestra.run.smithi102.stdout:Package libcephfs2-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:16.690 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:16.691 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-26T18:44:16.691 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:16.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs-devel 2024-03-26T18:44:17.314 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:24 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:17.441 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T18:44:17.463 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:17.463 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:17.463 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:17.464 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:17.464 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:44:17.464 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:18.2.0-0.el9 ceph 30 k 2024-03-26T18:44:17.464 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:17.464 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:44:17.464 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:17.464 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-26T18:44:17.464 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:17.464 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 30 k 2024-03-26T18:44:17.464 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 137 k 2024-03-26T18:44:17.464 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:44:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-03-26T18:44:17.542 INFO:teuthology.orchestra.run.smithi102.stdout:libcephfs-devel-18.2.0-0.el9.x86_64.rpm 394 kB/s | 30 kB 00:00 2024-03-26T18:44:17.542 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:17.543 INFO:teuthology.orchestra.run.smithi102.stdout:Total 387 kB/s | 30 kB 00:00 2024-03-26T18:44:17.543 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:44:17.546 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:44:17.546 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:44:17.572 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:44:17.572 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:44:17.685 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:44:17.789 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:18.192 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:18.452 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:18.452 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:18.452 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:44:18.452 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:18.452 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:18.452 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:18.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados2 2024-03-26T18:44:19.200 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:26 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:19.295 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:19.348 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:19.349 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-26T18:44:19.349 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:19.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librbd1 2024-03-26T18:44:19.973 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:26 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:20.069 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:20.123 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:20.124 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-26T18:44:20.124 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:20.149 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-03-26T18:44:20.149 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 1/12 2024-03-26T18:44:20.149 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T18:44:20.149 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T18:44:20.149 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T18:44:20.149 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T18:44:20.149 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T18:44:20.149 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T18:44:20.149 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T18:44:20.149 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T18:44:20.150 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T18:44:20.150 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T18:44:20.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rados 2024-03-26T18:44:20.536 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T18:44:20.536 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:20.536 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:20.536 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarch 2024-03-26T18:44:20.536 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T18:44:20.536 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T18:44:20.536 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T18:44:20.536 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T18:44:20.536 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T18:44:20.536 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T18:44:20.536 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T18:44:20.537 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T18:44:20.537 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T18:44:20.537 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T18:44:20.537 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T18:44:20.537 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:20.537 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:20.755 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:27 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:20.850 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rados-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:20.898 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-rook 2024-03-26T18:44:20.902 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:20.904 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-26T18:44:20.904 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:20.973 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rgw 2024-03-26T18:44:21.469 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:11 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:21.532 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:28 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:21.622 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:21.624 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-26T18:44:21.624 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:21.624 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:18.2.0-0.el9 ceph-noarch 47 k 2024-03-26T18:44:21.624 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-26T18:44:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T18:44:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T18:44:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T18:44:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T18:44:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T18:44:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T18:44:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T18:44:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T18:44:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T18:44:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T18:44:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T18:44:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T18:44:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:21.626 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:21.626 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:21.626 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2024-03-26T18:44:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:21.627 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 2.0 M 2024-03-26T18:44:21.627 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 26 M 2024-03-26T18:44:21.627 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:21.628 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rgw-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:21.681 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:21.682 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-26T18:44:21.682 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:21.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-cephfs 2024-03-26T18:44:21.905 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): ceph-mgr-rook-18.2.0-0.el9.noarch.rpm 356 kB/s | 47 kB 00:00 2024-03-26T18:44:21.937 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 148 kB/s | 24 kB 00:00 2024-03-26T18:44:21.962 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 95 kB/s | 18 kB 00:00 2024-03-26T18:44:21.987 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-26T18:44:22.013 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-26T18:44:22.038 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-26T18:44:22.063 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 569 kB/s | 43 kB 00:00 2024-03-26T18:44:22.180 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 193 kB/s | 32 kB 00:00 2024-03-26T18:44:22.214 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 85 kB/s | 15 kB 00:00 2024-03-26T18:44:22.281 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 889 kB/s | 59 kB 00:00 2024-03-26T18:44:22.310 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:29 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:22.339 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 884 kB/s | 203 kB 00:00 2024-03-26T18:44:22.365 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-26T18:44:22.398 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-03-26T18:44:22.402 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:22.402 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-26T18:44:22.403 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:22.406 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-cephfs-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:22.415 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:22.415 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:22.459 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:22.460 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-26T18:44:22.460 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:22.505 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:22.505 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:22.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rbd 2024-03-26T18:44:23.061 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:23.084 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:23.181 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rbd-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:23.222 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T18:44:23.233 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:23.234 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-26T18:44:23.234 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:23.274 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T18:44:23.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-fuse 2024-03-26T18:44:23.330 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T18:44:23.367 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T18:44:23.400 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T18:44:23.486 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T18:44:23.604 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T18:44:23.651 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T18:44:23.862 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:24.011 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:24.011 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:24.011 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:44:24.012 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:18.2.0-0.el9 ceph 86 k 2024-03-26T18:44:24.012 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:44:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-26T18:44:24.012 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 86 k 2024-03-26T18:44:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 230 k 2024-03-26T18:44:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:44:24.104 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-fuse-18.2.0-0.el9.x86_64.rpm 939 kB/s | 86 kB 00:00 2024-03-26T18:44:24.104 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:24.104 INFO:teuthology.orchestra.run.smithi102.stdout:Total 927 kB/s | 86 kB 00:00 2024-03-26T18:44:24.105 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:44:24.111 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:44:24.111 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:44:24.165 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:44:24.165 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:44:24.313 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:44:24.395 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T18:44:24.409 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:24.440 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T18:44:24.475 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T18:44:24.520 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T18:44:24.544 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-03-26T18:44:24.774 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:25.069 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:25.069 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:25.069 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:44:25.069 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:25.069 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:25.069 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:25.328 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-mirror 2024-03-26T18:44:25.384 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-03-26T18:44:25.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:18.2.0-0.el9.noarch 1/13 2024-03-26T18:44:25.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T18:44:25.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T18:44:25.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T18:44:25.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T18:44:25.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T18:44:25.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T18:44:25.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T18:44:25.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T18:44:25.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T18:44:25.385 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T18:44:25.385 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T18:44:25.754 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T18:44:25.754 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:25.754 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:25.754 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:18.2.0-0.el9.noarch 2024-03-26T18:44:25.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T18:44:25.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T18:44:25.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T18:44:25.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T18:44:25.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T18:44:25.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T18:44:25.755 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T18:44:25.755 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T18:44:25.755 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T18:44:25.755 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T18:44:25.755 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T18:44:25.755 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T18:44:25.755 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:25.755 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:25.893 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:32 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:25.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T18:44:26.048 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 3.0 M 2024-03-26T18:44:26.050 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 12 M 2024-03-26T18:44:26.051 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:44:26.207 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-mirror-18.2.0-0.el9.x86_64.rpm 19 MB/s | 3.0 MB 00:00 2024-03-26T18:44:26.208 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:26.208 INFO:teuthology.orchestra.run.smithi102.stdout:Total 19 MB/s | 3.0 MB 00:00 2024-03-26T18:44:26.208 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:44:26.217 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:44:26.217 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:44:26.288 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:44:26.288 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:44:26.558 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:16 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:26.692 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:44:26.707 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:18.2.0-0.el9 ceph-noarch 127 k 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:26.708 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:26.709 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:26.709 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-03-26T18:44:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:26.709 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 6.8 M 2024-03-26T18:44:26.709 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 33 M 2024-03-26T18:44:26.709 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:26.741 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:26.765 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:26.766 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:44:26.766 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T18:44:26.766 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T18:44:26.766 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T18:44:26.766 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:26.876 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): ceph-mgr-cephadm-18.2.0-0.el9.noarch.rpm 1.7 MB/s | 127 kB 00:00 2024-03-26T18:44:26.993 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-26T18:44:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.1 MB/s | 5.8 MB 00:00 2024-03-26T18:44:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 130 kB/s | 58 kB 00:00 2024-03-26T18:44:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 789 kB/s | 548 kB 00:00 2024-03-26T18:44:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 175 kB/s | 51 kB 00:00 2024-03-26T18:44:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:Total 6.6 MB/s | 6.8 MB 00:01 2024-03-26T18:44:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:27.753 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:27.753 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:27.840 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:27.841 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:27.859 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:27.859 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:27.859 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:44:27.859 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:27.859 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:27.859 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:28.122 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-nbd 2024-03-26T18:44:28.148 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T18:44:28.370 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T18:44:28.702 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:35 ago on Tue 26 Mar 2024 06:42:53 PM UTC. 2024-03-26T18:44:28.851 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-26T18:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-26T18:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:18.2.0-0.el9 ceph 169 k 2024-03-26T18:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-26T18:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-26T18:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-26T18:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 169 k 2024-03-26T18:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 490 k 2024-03-26T18:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-26T18:44:28.882 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T18:44:28.930 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-nbd-18.2.0-0.el9.x86_64.rpm 2.2 MB/s | 169 kB 00:00 2024-03-26T18:44:28.931 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:28.931 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.1 MB/s | 169 kB 00:00 2024-03-26T18:44:28.931 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-26T18:44:28.938 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-26T18:44:28.938 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-26T18:44:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T18:44:28.995 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-26T18:44:28.995 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-26T18:44:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T18:44:29.041 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-03-26T18:44:29.145 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-26T18:44:29.244 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:29.684 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-03-26T18:44:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 1/6 2024-03-26T18:44:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T18:44:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T18:44:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T18:44:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T18:44:29.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:29.985 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:29.985 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-26T18:44:29.985 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:29.985 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:44:29.985 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-26T18:44:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T18:44:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:30.056 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 2024-03-26T18:44:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T18:44:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T18:44:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T18:44:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T18:44:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T18:44:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:30.056 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:30.222 DEBUG:teuthology.parallel:result is None 2024-03-26T18:44:30.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse 2024-03-26T18:44:30.872 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:20 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:31.021 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:31.021 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:31.021 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:31.021 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:31.022 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:31.022 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:18.2.0-0.el9 ceph 835 k 2024-03-26T18:44:31.022 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-26T18:44:31.022 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T18:44:31.022 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:31.022 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:31.022 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:31.022 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-03-26T18:44:31.022 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:31.023 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 913 k 2024-03-26T18:44:31.023 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 2.7 M 2024-03-26T18:44:31.023 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:31.129 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): ceph-fuse-18.2.0-0.el9.x86_64.rpm 7.7 MB/s | 835 kB 00:00 2024-03-26T18:44:31.179 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 505 kB/s | 79 kB 00:00 2024-03-26T18:44:31.180 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:31.180 INFO:teuthology.orchestra.run.smithi190.stdout:Total 5.6 MB/s | 913 kB 00:00 2024-03-26T18:44:31.180 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:31.201 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:31.201 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:31.276 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:31.276 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:31.465 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:31.601 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T18:44:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-03-26T18:44:32.412 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-03-26T18:44:32.412 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:18.2.0-0.el9.x86_64 1/2 2024-03-26T18:44:32.720 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T18:44:32.720 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:32.720 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:32.720 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:18.2.0-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T18:44:32.720 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:32.720 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:33.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel 2024-03-26T18:44:33.577 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:23 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:33.729 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:33.730 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:33.730 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:33.730 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:33.730 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:33.730 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:18.2.0-0.el9 ceph 126 k 2024-03-26T18:44:33.730 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:33.731 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:33.731 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:33.731 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-26T18:44:33.731 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:33.731 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 126 k 2024-03-26T18:44:33.731 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 456 k 2024-03-26T18:44:33.731 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:33.792 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-18.2.0-0.el9.x86_64.rpm 2.0 MB/s | 126 kB 00:00 2024-03-26T18:44:33.793 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:33.793 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.0 MB/s | 126 kB 00:00 2024-03-26T18:44:33.793 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:33.799 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:33.799 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:33.853 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:33.853 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:33.993 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:34.090 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:34.803 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:34.803 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:34.803 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:34.803 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:34.803 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:34.803 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:35.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2 2024-03-26T18:44:35.660 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:25 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:35.763 INFO:teuthology.orchestra.run.smithi190.stdout:Package libcephfs2-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:35.817 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:35.818 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-26T18:44:35.818 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:35.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs-devel 2024-03-26T18:44:36.461 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:26 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:36.612 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:36.612 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:18.2.0-0.el9 ceph 30 k 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 30 k 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 137 k 2024-03-26T18:44:36.613 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:36.689 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs-devel-18.2.0-0.el9.x86_64.rpm 407 kB/s | 30 kB 00:00 2024-03-26T18:44:36.689 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:36.689 INFO:teuthology.orchestra.run.smithi190.stdout:Total 400 kB/s | 30 kB 00:00 2024-03-26T18:44:36.689 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:36.693 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:36.693 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:36.718 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:36.718 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:36.810 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:36.919 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:37.285 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:37.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:37.535 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:37.535 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:37.535 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:37.535 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:37.535 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:37.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2 2024-03-26T18:44:38.260 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:28 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:38.357 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:38.409 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:38.410 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-26T18:44:38.410 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:38.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1 2024-03-26T18:44:39.033 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:29 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:39.129 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:39.182 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:39.183 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-26T18:44:39.184 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:39.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados 2024-03-26T18:44:39.806 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:29 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:39.903 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rados-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:39.957 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:39.958 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-26T18:44:39.958 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:40.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw 2024-03-26T18:44:40.576 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:40.673 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rgw-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:40.725 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:40.726 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-26T18:44:40.726 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:40.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs 2024-03-26T18:44:41.346 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:31 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:41.443 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-cephfs-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:41.497 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:41.498 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-26T18:44:41.498 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:41.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd 2024-03-26T18:44:42.126 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:32 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:42.224 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rbd-2:18.2.0-0.el9.x86_64 is already installed. 2024-03-26T18:44:42.276 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:42.276 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-26T18:44:42.277 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:42.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse 2024-03-26T18:44:42.903 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:32 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:43.053 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:43.054 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:43.054 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:43.054 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:43.054 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:43.054 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:18.2.0-0.el9 ceph 86 k 2024-03-26T18:44:43.054 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:43.054 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:43.054 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:43.054 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-26T18:44:43.054 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:43.055 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 86 k 2024-03-26T18:44:43.055 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 230 k 2024-03-26T18:44:43.055 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:43.133 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-18.2.0-0.el9.x86_64.rpm 1.1 MB/s | 86 kB 00:00 2024-03-26T18:44:43.133 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:43.134 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.1 MB/s | 86 kB 00:00 2024-03-26T18:44:43.134 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:43.141 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:43.141 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:43.196 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:43.197 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:43.337 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:43.440 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:43.839 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:44.127 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:44.127 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:44.128 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:44.128 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:44.128 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:44.128 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:44.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2024-03-26T18:44:44.950 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:34 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:45.103 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:45.103 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:45.104 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:45.104 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-03-26T18:44:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:45.104 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:45.104 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:45.104 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-26T18:44:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:45.104 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.0 M 2024-03-26T18:44:45.104 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 12 M 2024-03-26T18:44:45.104 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:45.214 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-18.2.0-0.el9.x86_64.rpm 27 MB/s | 3.0 MB 00:00 2024-03-26T18:44:45.214 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:45.215 INFO:teuthology.orchestra.run.smithi190.stdout:Total 27 MB/s | 3.0 MB 00:00 2024-03-26T18:44:45.215 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:45.224 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:45.224 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:45.295 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:45.296 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:45.757 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:45.781 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:45.782 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T18:44:45.782 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T18:44:45.782 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T18:44:45.782 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T18:44:45.782 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:46.910 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:46.910 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:46.910 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:46.910 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:46.910 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:46.910 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:47.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd 2024-03-26T18:44:47.720 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:37 ago on Tue 26 Mar 2024 06:43:10 PM UTC. 2024-03-26T18:44:47.869 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-26T18:44:47.870 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:18.2.0-0.el9 ceph 169 k 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 169 k 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 490 k 2024-03-26T18:44:47.871 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-26T18:44:47.934 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-18.2.0-0.el9.x86_64.rpm 2.6 MB/s | 169 kB 00:00 2024-03-26T18:44:47.935 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:44:47.935 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.5 MB/s | 169 kB 00:00 2024-03-26T18:44:47.935 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-26T18:44:47.943 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-26T18:44:47.943 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-26T18:44:47.998 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-26T18:44:47.998 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-26T18:44:48.156 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-26T18:44:48.257 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:48.663 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:48.948 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-03-26T18:44:48.948 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:48.948 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-26T18:44:48.948 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:18.2.0-0.el9.x86_64 2024-03-26T18:44:48.948 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:44:48.948 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-26T18:44:49.172 DEBUG:teuthology.parallel:result is None 2024-03-26T18:44:49.173 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-26T18:44:49.173 INFO:teuthology.packaging:ref: None 2024-03-26T18:44:49.173 INFO:teuthology.packaging:tag: v18.2.0 2024-03-26T18:44:49.173 INFO:teuthology.packaging:branch: None 2024-03-26T18:44:49.173 INFO:teuthology.packaging:sha1: 360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:44:49.173 INFO:teuthology.packaging:Tag 'v18.2.0' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-03-26T18:44:49.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-03-26T18:44:49.395 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T18:44:49.420 INFO:teuthology.orchestra.run.smithi102.stdout:18.2.0-0.el9 2024-03-26T18:44:49.420 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-0.el9 2024-03-26T18:44:49.420 INFO:teuthology.task.install:The correct ceph version 18.2.0-0 is installed. 2024-03-26T18:44:49.422 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-03-26T18:44:49.422 INFO:teuthology.packaging:ref: None 2024-03-26T18:44:49.423 INFO:teuthology.packaging:tag: v18.2.0 2024-03-26T18:44:49.423 INFO:teuthology.packaging:branch: None 2024-03-26T18:44:49.423 INFO:teuthology.packaging:sha1: 360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:44:49.423 INFO:teuthology.packaging:Tag 'v18.2.0' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-03-26T18:44:49.423 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-03-26T18:44:49.563 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T18:44:49.586 INFO:teuthology.orchestra.run.smithi190.stdout:18.2.0-0.el9 2024-03-26T18:44:49.586 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-0.el9 2024-03-26T18:44:49.586 INFO:teuthology.task.install:The correct ceph version 18.2.0-0 is installed. 2024-03-26T18:44:49.588 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T18:44:49.589 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:44:49.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T18:44:49.620 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:44:49.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T18:44:49.655 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T18:44:49.655 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:44:49.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T18:44:49.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T18:44:49.758 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:44:49.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T18:44:49.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T18:44:49.854 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T18:44:49.855 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:44:49.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T18:44:49.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T18:44:49.956 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:44:49.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T18:44:49.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T18:44:50.051 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T18:44:50.052 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:44:50.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T18:44:50.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T18:44:50.147 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:44:50.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T18:44:50.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T18:44:50.242 INFO:teuthology.run_tasks:Running task print... 2024-03-26T18:44:50.251 INFO:teuthology.task.print:**** done install task... 2024-03-26T18:44:50.251 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T18:44:50.366 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.0', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', '\\(PG_DEGRADED', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '360516069d9393362c4cc6eb9371680fe16d66ab'} 2024-03-26T18:44:50.366 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.0 2024-03-26T18:44:50.367 INFO:tasks.cephadm:Cluster fsid is ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:44:50.367 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T18:44:50.367 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-26T18:44:50.367 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi102': '172.21.15.102', 'mon.smithi190': '172.21.15.190'} 2024-03-26T18:44:50.367 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T18:44:50.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname $(hostname -s) 2024-03-26T18:44:50.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo hostname $(hostname -s) 2024-03-26T18:44:50.423 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-03-26T18:44:50.423 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:44:50.640 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '360516069d9393362c4cc6eb9371680fe16d66ab', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77808/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2311-g36051606', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2311.g36051606'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240320.145742/360516069d9393362c4cc6eb9371680fe16d66ab/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-20 17:53:08.349574', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240320.145742', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240320.145742/360516069d9393362c4cc6eb9371680fe16d66ab/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-26T18:44:50.984 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref reef, sha1 d83b58fabace7c0b0c2d45a3ac2899524ef89e97 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-03-26T18:44:50.985 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/x86_64/flavors/default/cephadm 2024-03-26T18:44:50.985 DEBUG:teuthology.orchestra.run.smithi102:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T18:44:51.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r--. 1 ubuntu ubuntu 215331 Mar 26 18:44 /home/ubuntu/cephtest/cephadm 2024-03-26T18:44:51.412 DEBUG:teuthology.orchestra.run.smithi190:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T18:44:52.056 INFO:teuthology.orchestra.run.smithi190.stdout:-rw-r--r--. 1 ubuntu ubuntu 215331 Mar 26 18:44 /home/ubuntu/cephtest/cephadm 2024-03-26T18:44:52.057 DEBUG:teuthology.orchestra.run.smithi102:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T18:44:52.075 DEBUG:teuthology.orchestra.run.smithi190:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T18:44:52.105 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.0 on all hosts... 2024-03-26T18:44:52.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull 2024-03-26T18:44:52.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull 2024-03-26T18:44:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-03-26T18:44:52.937 INFO:teuthology.orchestra.run.smithi190.stderr:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-03-26T18:45:14.203 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:45:14.203 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-03-26T18:45:14.203 INFO:teuthology.orchestra.run.smithi102.stdout: "image_id": "dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946", 2024-03-26T18:45:14.204 INFO:teuthology.orchestra.run.smithi102.stdout: "repo_digests": [ 2024-03-26T18:45:14.204 INFO:teuthology.orchestra.run.smithi102.stdout: "quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007", 2024-03-26T18:45:14.204 INFO:teuthology.orchestra.run.smithi102.stdout: "quay.io/ceph/ceph@sha256:d58cf65589d0abf9d5261cd46fb62be3bfb29098febc78c0fcdf116a15274d27" 2024-03-26T18:45:14.204 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-03-26T18:45:14.204 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:45:15.930 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-26T18:45:15.930 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-03-26T18:45:15.930 INFO:teuthology.orchestra.run.smithi190.stdout: "image_id": "dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946", 2024-03-26T18:45:15.930 INFO:teuthology.orchestra.run.smithi190.stdout: "repo_digests": [ 2024-03-26T18:45:15.930 INFO:teuthology.orchestra.run.smithi190.stdout: "quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007", 2024-03-26T18:45:15.930 INFO:teuthology.orchestra.run.smithi190.stdout: "quay.io/ceph/ceph@sha256:d58cf65589d0abf9d5261cd46fb62be3bfb29098febc78c0fcdf116a15274d27" 2024-03-26T18:45:15.930 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-26T18:45:15.930 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-26T18:45:15.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph 2024-03-26T18:45:15.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph 2024-03-26T18:45:16.012 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /etc/ceph 2024-03-26T18:45:16.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /etc/ceph 2024-03-26T18:45:16.080 INFO:tasks.cephadm:Writing seed config... 2024-03-26T18:45:16.081 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-26T18:45:16.081 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-26T18:45:16.081 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-26T18:45:16.081 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-26T18:45:16.081 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-26T18:45:16.081 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-26T18:45:16.082 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-26T18:45:16.083 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-26T18:45:16.084 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-26T18:45:16.084 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T18:45:16.084 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T18:45:16.084 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T18:45:16.084 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T18:45:16.084 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T18:45:16.084 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-26T18:45:16.084 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-26T18:45:16.085 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:45:16.085 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T18:45:16.110 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ed4e027c-eba0-11ee-95d0-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-26T18:45:16.111 DEBUG:teuthology.orchestra.run.smithi102:mon.smithi102> sudo journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi102.service 2024-03-26T18:45:16.154 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T18:45:16.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 -v bootstrap --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.102 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T18:45:16.321 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-26T18:45:16.321 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.0', '-v', 'bootstrap', '--fsid', 'ed4e027c-eba0-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.102', '--skip-admin-label'] 2024-03-26T18:45:16.346 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T18:45:16.346 INFO:teuthology.orchestra.run.smithi102.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T18:45:16.346 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying podman|docker is present... 2024-03-26T18:45:16.369 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T18:45:16.369 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying lvm2 is present... 2024-03-26T18:45:16.369 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying time synchronization is in place... 2024-03-26T18:45:16.376 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T18:45:16.377 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T18:45:16.382 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T18:45:16.383 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-26T18:45:16.388 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-03-26T18:45:16.394 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-03-26T18:45:16.394 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-03-26T18:45:16.395 INFO:teuthology.orchestra.run.smithi102.stdout:Repeating the final host check... 2024-03-26T18:45:16.417 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T18:45:16.417 INFO:teuthology.orchestra.run.smithi102.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T18:45:16.417 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl is present 2024-03-26T18:45:16.417 INFO:teuthology.orchestra.run.smithi102.stdout:lvcreate is present 2024-03-26T18:45:16.423 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T18:45:16.423 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T18:45:16.429 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T18:45:16.429 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-26T18:45:16.435 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-03-26T18:45:16.441 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-03-26T18:45:16.441 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-03-26T18:45:16.441 INFO:teuthology.orchestra.run.smithi102.stdout:Host looks OK 2024-03-26T18:45:16.441 INFO:teuthology.orchestra.run.smithi102.stdout:Cluster fsid: ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:45:16.441 INFO:teuthology.orchestra.run.smithi102.stdout:Acquiring lock 139664795294400 on /run/cephadm/ed4e027c-eba0-11ee-95d0-87774f69a715.lock 2024-03-26T18:45:16.441 INFO:teuthology.orchestra.run.smithi102.stdout:Lock 139664795294400 acquired on /run/cephadm/ed4e027c-eba0-11ee-95d0-87774f69a715.lock 2024-03-26T18:45:16.441 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 3300 ... 2024-03-26T18:45:16.442 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 6789 ... 2024-03-26T18:45:16.442 INFO:teuthology.orchestra.run.smithi102.stdout:Base mon IP(s) is [172.21.15.102:3300, 172.21.15.102:6789], mon addrv is [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-03-26T18:45:16.444 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.102 metric 100 2024-03-26T18:45:16.444 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.102 metric 100 2024-03-26T18:45:16.447 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T18:45:16.447 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-26T18:45:16.447 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-03-26T18:45:16.449 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T18:45:16.449 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T18:45:16.449 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T18:45:16.449 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-26T18:45:16.450 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-26T18:45:16.450 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T18:45:16.450 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-03-26T18:45:16.450 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-03-26T18:45:16.450 INFO:teuthology.orchestra.run.smithi102.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T18:45:16.451 INFO:teuthology.orchestra.run.smithi102.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T18:45:16.451 INFO:teuthology.orchestra.run.smithi102.stdout:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-03-26T18:45:17.267 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 2024-03-26T18:45:17.267 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.0... 2024-03-26T18:45:17.267 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T18:45:17.267 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:3bd20aeff60302f668275dc2005d10679ae56492967a3a5a54fd3dde85333aec 2024-03-26T18:45:17.267 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:46af8f5390d4e94fc57efb422ccb97bb53dfe5b948546bfc191b46557eb2dbd9 2024-03-26T18:45:17.267 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying config sha256:dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 2024-03-26T18:45:17.267 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T18:45:18.135 INFO:teuthology.orchestra.run.smithi102.stdout:ceph: stdout ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 2024-03-26T18:45:18.135 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 2024-03-26T18:45:18.135 INFO:teuthology.orchestra.run.smithi102.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T18:45:19.040 INFO:teuthology.orchestra.run.smithi102.stdout:stat: stdout 167 167 2024-03-26T18:45:19.041 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial keys... 2024-03-26T18:45:20.012 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQC/FwNm2bfkFRAAHaljO6L4nTiCzxu4JASA2A== 2024-03-26T18:45:20.902 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQDAFwNmAuc5ExAAPDL00jxwHPyyIv2oojIbow== 2024-03-26T18:45:22.150 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQDBFwNmF/SgDxAAKL9srlyLkajq/xLMx64Yqw== 2024-03-26T18:45:22.151 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial monmap... 2024-03-26T18:45:23.080 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T18:45:23.081 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-03-26T18:45:23.081 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:45:23.081 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T18:45:23.081 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool for smithi102 [v2:172.21.15.102:3300,v1:172.21.15.102:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T18:45:23.081 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = pacific 2024-03-26T18:45:23.081 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: set fsid to ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:45:23.081 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T18:45:23.081 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:45:23.081 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mon... 2024-03-26T18:45:24.036 INFO:teuthology.orchestra.run.smithi102.stdout:create mon.smithi102 on 2024-03-26T18:45:24.249 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T18:45:24.393 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T18:45:24.594 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ed4e027c-eba0-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-ed4e027c-eba0-11ee-95d0-87774f69a715.target. 2024-03-26T18:45:24.594 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ed4e027c-eba0-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-ed4e027c-eba0-11ee-95d0-87774f69a715.target. 2024-03-26T18:45:24.815 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi102 2024-03-26T18:45:24.815 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi102.service: Unit ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi102.service not loaded. 2024-03-26T18:45:24.969 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ed4e027c-eba0-11ee-95d0-87774f69a715.target.wants/ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi102.service → /etc/systemd/system/ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@.service. 2024-03-26T18:45:25.444 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T18:45:25.444 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-26T18:45:25.451 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T18:45:25.451 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-26T18:45:25.451 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-26T18:45:25.451 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T18:45:25.451 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon to start... 2024-03-26T18:45:25.451 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon... 2024-03-26T18:45:25.556 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:25 smithi102 ceph-mon[28634]: mon.smithi102 is new leader, mons smithi102 in quorum (ranks 0) 2024-03-26T18:45:26.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout id: ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout services: 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi102 (age 0.54726s) 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout data: 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-26T18:45:26.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.915+0000 7fa58bc36700 1 Processor -- start 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.915+0000 7fa58bc36700 1 -- start start 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.915+0000 7fa58bc36700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58407a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.915+0000 7fa58bc36700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58407a850 con 0x7fa58407be10 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.915+0000 7fa5899d2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58407a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.916+0000 7fa5899d2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58407a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50202/0 (socket says 172.21.15.102:50202) 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.916+0000 7fa5899d2700 1 -- 172.21.15.102:0/3741964746 learned_addr learned my addr 172.21.15.102:0/3741964746 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.916+0000 7fa5899d2700 1 -- 172.21.15.102:0/3741964746 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58407a990 con 0x7fa58407be10 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.916+0000 7fa5899d2700 1 --2- 172.21.15.102:0/3741964746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58407a310 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa5800098a0 tx=0x7fa580009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1aaa0dd2ba760e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.916+0000 7fa5889d0700 1 -- 172.21.15.102:0/3741964746 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa580004180 con 0x7fa58407be10 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.916+0000 7fa5889d0700 1 -- 172.21.15.102:0/3741964746 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5800042e0 con 0x7fa58407be10 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.916+0000 7fa5889d0700 1 -- 172.21.15.102:0/3741964746 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5800036a0 con 0x7fa58407be10 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.917+0000 7fa58bc36700 1 -- 172.21.15.102:0/3741964746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 msgr2=0x7fa58407a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.917+0000 7fa58bc36700 1 --2- 172.21.15.102:0/3741964746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58407a310 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa5800098a0 tx=0x7fa580009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:26.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.917+0000 7fa58bc36700 1 -- 172.21.15.102:0/3741964746 shutdown_connections 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.917+0000 7fa58bc36700 1 --2- 172.21.15.102:0/3741964746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58407a310 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.917+0000 7fa58bc36700 1 -- 172.21.15.102:0/3741964746 >> 172.21.15.102:0/3741964746 conn(0x7fa584101180 msgr2=0x7fa584103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.917+0000 7fa58bc36700 1 -- 172.21.15.102:0/3741964746 shutdown_connections 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.917+0000 7fa58bc36700 1 -- 172.21.15.102:0/3741964746 wait complete. 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.917+0000 7fa58bc36700 1 Processor -- start 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.917+0000 7fa58bc36700 1 -- start start 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa58bc36700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58419b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa58bc36700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58419b770 con 0x7fa58407be10 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa5899d2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58419b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa5899d2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58419b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50204/0 (socket says 172.21.15.102:50204) 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa5899d2700 1 -- 172.21.15.102:0/2694122212 learned_addr learned my addr 172.21.15.102:0/2694122212 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa5899d2700 1 -- 172.21.15.102:0/2694122212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa580009550 con 0x7fa58407be10 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa5899d2700 1 --2- 172.21.15.102:0/2694122212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58419b230 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa580004000 tx=0x7fa580004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa57affd700 1 -- 172.21.15.102:0/2694122212 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa580003ce0 con 0x7fa58407be10 2024-03-26T18:45:26.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa57affd700 1 -- 172.21.15.102:0/2694122212 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa580020070 con 0x7fa58407be10 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa57affd700 1 -- 172.21.15.102:0/2694122212 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa580011630 con 0x7fa58407be10 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa58bc36700 1 -- 172.21.15.102:0/2694122212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58419b970 con 0x7fa58407be10 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa58bc36700 1 -- 172.21.15.102:0/2694122212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58419bd90 con 0x7fa58407be10 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa57affd700 1 -- 172.21.15.102:0/2694122212 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa580022030 con 0x7fa58407be10 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.918+0000 7fa57affd700 1 -- 172.21.15.102:0/2694122212 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa580025e20 con 0x7fa58407be10 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.919+0000 7fa58bc36700 1 -- 172.21.15.102:0/2694122212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa584195060 con 0x7fa58407be10 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.921+0000 7fa57affd700 1 -- 172.21.15.102:0/2694122212 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fa580025480 con 0x7fa58407be10 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.991+0000 7fa58bc36700 1 -- 172.21.15.102:0/2694122212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fa584062380 con 0x7fa58407be10 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.991+0000 7fa57affd700 1 -- 172.21.15.102:0/2694122212 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+324 (secure 0 0 0) 0x7fa580011c20 con 0x7fa58407be10 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.992+0000 7fa58bc36700 1 -- 172.21.15.102:0/2694122212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 msgr2=0x7fa58419b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.992+0000 7fa58bc36700 1 --2- 172.21.15.102:0/2694122212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58419b230 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa580004000 tx=0x7fa580004750 comp rx=0 tx=0).stop 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.992+0000 7fa58bc36700 1 -- 172.21.15.102:0/2694122212 shutdown_connections 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.992+0000 7fa58bc36700 1 --2- 172.21.15.102:0/2694122212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa58407be10 0x7fa58419b230 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:26.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.992+0000 7fa58bc36700 1 -- 172.21.15.102:0/2694122212 >> 172.21.15.102:0/2694122212 conn(0x7fa584101180 msgr2=0x7fa584102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:26.532 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.992+0000 7fa58bc36700 1 -- 172.21.15.102:0/2694122212 shutdown_connections 2024-03-26T18:45:26.532 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:25.992+0000 7fa58bc36700 1 -- 172.21.15.102:0/2694122212 wait complete. 2024-03-26T18:45:26.532 INFO:teuthology.orchestra.run.smithi102.stdout:mon is available 2024-03-26T18:45:26.532 INFO:teuthology.orchestra.run.smithi102.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T18:45:26.645 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:26 smithi102 ceph-mon[28634]: mon.smithi102 is new leader, mons smithi102 in quorum (ranks 0) 2024-03-26T18:45:26.645 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:26 smithi102 ceph-mon[28634]: monmap e1: 1 mons at {smithi102=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2024-03-26T18:45:26.646 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:26 smithi102 ceph-mon[28634]: fsmap 2024-03-26T18:45:26.646 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:26 smithi102 ceph-mon[28634]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T18:45:26.646 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:26 smithi102 ceph-mon[28634]: mgrmap e1: no daemons active 2024-03-26T18:45:26.646 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:26 smithi102 ceph-mon[28634]: from='client.? 172.21.15.102:0/2694122212' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T18:45:27.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-26T18:45:27.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T18:45:27.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:45:27.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-03-26T18:45:27.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T18:45:27.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T18:45:27.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T18:45:27.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T18:45:27.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-26T18:45:27.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T18:45:27.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T18:45:27.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-26T18:45:27.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T18:45:27.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-26T18:45:27.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-26T18:45:27.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-26T18:45:27.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T18:45:27.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T18:45:27.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.928+0000 7f233b808700 1 Processor -- start 2024-03-26T18:45:27.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.929+0000 7f233b808700 1 -- start start 2024-03-26T18:45:27.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.929+0000 7f233b808700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.929+0000 7f233b808700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2334106460 con 0x7f2334105b10 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.929+0000 7f23395a4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.929+0000 7f23395a4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50212/0 (socket says 172.21.15.102:50212) 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.929+0000 7f23395a4700 1 -- 172.21.15.102:0/2524796494 learned_addr learned my addr 172.21.15.102:0/2524796494 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.929+0000 7f23395a4700 1 -- 172.21.15.102:0/2524796494 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23341065a0 con 0x7f2334105b10 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.929+0000 7f23395a4700 1 --2- 172.21.15.102:0/2524796494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334105f20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f232c0098a0 tx=0x7f232c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59aa7063a2c90b7b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.929+0000 7f232bfff700 1 -- 172.21.15.102:0/2524796494 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f232c004180 con 0x7f2334105b10 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.929+0000 7f232bfff700 1 -- 172.21.15.102:0/2524796494 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c0042e0 con 0x7f2334105b10 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.930+0000 7f232bfff700 1 -- 172.21.15.102:0/2524796494 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f232c0036a0 con 0x7f2334105b10 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.930+0000 7f233b808700 1 -- 172.21.15.102:0/2524796494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 msgr2=0x7f2334105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.930+0000 7f233b808700 1 --2- 172.21.15.102:0/2524796494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334105f20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f232c0098a0 tx=0x7f232c009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.930+0000 7f233b808700 1 -- 172.21.15.102:0/2524796494 shutdown_connections 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.930+0000 7f233b808700 1 --2- 172.21.15.102:0/2524796494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334105f20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:27.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.930+0000 7f233b808700 1 -- 172.21.15.102:0/2524796494 >> 172.21.15.102:0/2524796494 conn(0x7f2334101180 msgr2=0x7f2334103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.930+0000 7f233b808700 1 -- 172.21.15.102:0/2524796494 shutdown_connections 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.930+0000 7f233b808700 1 -- 172.21.15.102:0/2524796494 wait complete. 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f233b808700 1 Processor -- start 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f233b808700 1 -- start start 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f233b808700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f233b808700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2334199550 con 0x7f2334105b10 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f23395a4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f23395a4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334199010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50220/0 (socket says 172.21.15.102:50220) 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f23395a4700 1 -- 172.21.15.102:0/3848566326 learned_addr learned my addr 172.21.15.102:0/3848566326 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f23395a4700 1 -- 172.21.15.102:0/3848566326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c009550 con 0x7f2334105b10 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f23395a4700 1 --2- 172.21.15.102:0/3848566326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334199010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2334106840 tx=0x7f232c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f232a7fc700 1 -- 172.21.15.102:0/3848566326 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f232c003ce0 con 0x7f2334105b10 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f232a7fc700 1 -- 172.21.15.102:0/3848566326 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c020070 con 0x7f2334105b10 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f233b808700 1 -- 172.21.15.102:0/3848566326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2334199750 con 0x7f2334105b10 2024-03-26T18:45:27.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.931+0000 7f232a7fc700 1 -- 172.21.15.102:0/3848566326 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f232c011630 con 0x7f2334105b10 2024-03-26T18:45:27.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.932+0000 7f233b808700 1 -- 172.21.15.102:0/3848566326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2334199b70 con 0x7f2334105b10 2024-03-26T18:45:27.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.932+0000 7f232a7fc700 1 -- 172.21.15.102:0/3848566326 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f232c022030 con 0x7f2334105b10 2024-03-26T18:45:27.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.932+0000 7f232a7fc700 1 -- 172.21.15.102:0/3848566326 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f232c025d40 con 0x7f2334105b10 2024-03-26T18:45:27.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.932+0000 7f233b808700 1 -- 172.21.15.102:0/3848566326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23341927b0 con 0x7f2334105b10 2024-03-26T18:45:27.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:26.935+0000 7f232a7fc700 1 -- 172.21.15.102:0/3848566326 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f232c025480 con 0x7f2334105b10 2024-03-26T18:45:27.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.002+0000 7f233b808700 1 -- 172.21.15.102:0/3848566326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f233402ce60 con 0x7f2334105b10 2024-03-26T18:45:27.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.006+0000 7f232a7fc700 1 -- 172.21.15.102:0/3848566326 <== mon.0 v2:172.21.15.102:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f232c011c20 con 0x7f2334105b10 2024-03-26T18:45:27.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.006+0000 7f232a7fc700 1 -- 172.21.15.102:0/3848566326 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f232c037590 con 0x7f2334105b10 2024-03-26T18:45:27.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.007+0000 7f233b808700 1 -- 172.21.15.102:0/3848566326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 msgr2=0x7f2334199010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:27.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.007+0000 7f233b808700 1 --2- 172.21.15.102:0/3848566326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334199010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2334106840 tx=0x7f232c004750 comp rx=0 tx=0).stop 2024-03-26T18:45:27.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.007+0000 7f233b808700 1 -- 172.21.15.102:0/3848566326 shutdown_connections 2024-03-26T18:45:27.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.007+0000 7f233b808700 1 --2- 172.21.15.102:0/3848566326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2334105b10 0x7f2334199010 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:27.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.007+0000 7f233b808700 1 -- 172.21.15.102:0/3848566326 >> 172.21.15.102:0/3848566326 conn(0x7f2334101180 msgr2=0x7f233418d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:27.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.007+0000 7f233b808700 1 -- 172.21.15.102:0/3848566326 shutdown_connections 2024-03-26T18:45:27.531 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.007+0000 7f233b808700 1 -- 172.21.15.102:0/3848566326 wait complete. 2024-03-26T18:45:27.531 INFO:teuthology.orchestra.run.smithi102.stdout:Generating new minimal ceph.conf... 2024-03-26T18:45:27.724 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:27 smithi102 ceph-mon[28634]: from='client.? 172.21.15.102:0/3848566326' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T18:45:27.724 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:27 smithi102 ceph-mon[28634]: from='client.? 172.21.15.102:0/3848566326' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T18:45:28.555 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.917+0000 7f7b8b1bb700 1 Processor -- start 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.917+0000 7f7b8b1bb700 1 -- start start 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.917+0000 7f7b8b1bb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8407a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.917+0000 7f7b8b1bb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8407a830 con 0x7f7b8407bdf0 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.917+0000 7f7b88f57700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8407a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.917+0000 7f7b88f57700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8407a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50234/0 (socket says 172.21.15.102:50234) 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.918+0000 7f7b88f57700 1 -- 172.21.15.102:0/3612221397 learned_addr learned my addr 172.21.15.102:0/3612221397 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.918+0000 7f7b88f57700 1 -- 172.21.15.102:0/3612221397 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8407a970 con 0x7f7b8407bdf0 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.918+0000 7f7b88f57700 1 --2- 172.21.15.102:0/3612221397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8407a2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7b800098a0 tx=0x7f7b80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d87acec16b73dd server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.918+0000 7f7b7b7fe700 1 -- 172.21.15.102:0/3612221397 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b80004180 con 0x7f7b8407bdf0 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.918+0000 7f7b7b7fe700 1 -- 172.21.15.102:0/3612221397 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7b800042e0 con 0x7f7b8407bdf0 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.918+0000 7f7b7b7fe700 1 -- 172.21.15.102:0/3612221397 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b80011a50 con 0x7f7b8407bdf0 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.919+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/3612221397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 msgr2=0x7f7b8407a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:28.556 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.919+0000 7f7b8b1bb700 1 --2- 172.21.15.102:0/3612221397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8407a2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7b800098a0 tx=0x7f7b80009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.919+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/3612221397 shutdown_connections 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.919+0000 7f7b8b1bb700 1 --2- 172.21.15.102:0/3612221397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8407a2f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.919+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/3612221397 >> 172.21.15.102:0/3612221397 conn(0x7f7b84101180 msgr2=0x7f7b84103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.919+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/3612221397 shutdown_connections 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.919+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/3612221397 wait complete. 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.919+0000 7f7b8b1bb700 1 Processor -- start 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.919+0000 7f7b8b1bb700 1 -- start start 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.919+0000 7f7b8b1bb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8419ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.919+0000 7f7b8b1bb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8419bfb0 con 0x7f7b8407bdf0 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.920+0000 7f7b88f57700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8419ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.920+0000 7f7b88f57700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8419ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50238/0 (socket says 172.21.15.102:50238) 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.920+0000 7f7b88f57700 1 -- 172.21.15.102:0/2594846340 learned_addr learned my addr 172.21.15.102:0/2594846340 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.920+0000 7f7b88f57700 1 -- 172.21.15.102:0/2594846340 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b80009550 con 0x7f7b8407bdf0 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.920+0000 7f7b88f57700 1 --2- 172.21.15.102:0/2594846340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8419ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7b80000c00 tx=0x7f7b80011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:28.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.920+0000 7f7b79ffb700 1 -- 172.21.15.102:0/2594846340 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b80003b90 con 0x7f7b8407bdf0 2024-03-26T18:45:28.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.920+0000 7f7b79ffb700 1 -- 172.21.15.102:0/2594846340 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7b800248b0 con 0x7f7b8407bdf0 2024-03-26T18:45:28.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.920+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/2594846340 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8419c1b0 con 0x7f7b8407bdf0 2024-03-26T18:45:28.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.920+0000 7f7b79ffb700 1 -- 172.21.15.102:0/2594846340 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b8001a8e0 con 0x7f7b8407bdf0 2024-03-26T18:45:28.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.920+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/2594846340 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8419c5d0 con 0x7f7b8407bdf0 2024-03-26T18:45:28.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.921+0000 7f7b79ffb700 1 -- 172.21.15.102:0/2594846340 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7b8002d3f0 con 0x7f7b8407bdf0 2024-03-26T18:45:28.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.921+0000 7f7b79ffb700 1 -- 172.21.15.102:0/2594846340 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7b80019400 con 0x7f7b8407bdf0 2024-03-26T18:45:28.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.921+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/2594846340 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b84062380 con 0x7f7b8407bdf0 2024-03-26T18:45:28.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.923+0000 7f7b79ffb700 1 -- 172.21.15.102:0/2594846340 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f7b80024a20 con 0x7f7b8407bdf0 2024-03-26T18:45:28.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.991+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/2594846340 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f7b8419f100 con 0x7f7b8407bdf0 2024-03-26T18:45:28.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.991+0000 7f7b79ffb700 1 -- 172.21.15.102:0/2594846340 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f7b80035090 con 0x7f7b8407bdf0 2024-03-26T18:45:28.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.992+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/2594846340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 msgr2=0x7f7b8419ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:28.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.992+0000 7f7b8b1bb700 1 --2- 172.21.15.102:0/2594846340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8419ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7b80000c00 tx=0x7f7b80011ab0 comp rx=0 tx=0).stop 2024-03-26T18:45:28.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.992+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/2594846340 shutdown_connections 2024-03-26T18:45:28.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.992+0000 7f7b8b1bb700 1 --2- 172.21.15.102:0/2594846340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8407bdf0 0x7f7b8419ba70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:28.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.992+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/2594846340 >> 172.21.15.102:0/2594846340 conn(0x7f7b84101180 msgr2=0x7f7b84102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:28.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.992+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/2594846340 shutdown_connections 2024-03-26T18:45:28.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:27.993+0000 7f7b8b1bb700 1 -- 172.21.15.102:0/2594846340 wait complete. 2024-03-26T18:45:28.559 INFO:teuthology.orchestra.run.smithi102.stdout:Restarting the monitor... 2024-03-26T18:45:28.819 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:28 smithi102 ceph-mon[28634]: from='client.? 172.21.15.102:0/2594846340' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:45:28.819 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:28 smithi102 systemd[1]: Stopping Ceph mon.smithi102 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:45:29.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:28 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102[28630]: 2024-03-26T18:45:28.828+0000 7fece2960700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi102 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T18:45:29.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:28 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102[28630]: 2024-03-26T18:45:28.828+0000 7fece2960700 -1 mon.smithi102@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T18:45:29.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:28 smithi102 podman[28877]: 2024-03-26 18:45:28.941215556 +0000 UTC m=+0.218404260 container died 5f00ad357a2531e082fab5c7364c73d16130057a973881faf5231843a2b722c9 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, org.label-schema.build-date=20231212, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-03-26T18:45:29.454 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:29 smithi102 podman[28877]: 2024-03-26 18:45:29.319814674 +0000 UTC m=+0.597003409 container cleanup 5f00ad357a2531e082fab5c7364c73d16130057a973881faf5231843a2b722c9 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T18:45:29.455 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:29 smithi102 bash[28877]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102 2024-03-26T18:45:29.861 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:29 smithi102 podman[28891]: 2024-03-26 18:45:29.56226383 +0000 UTC m=+0.612063461 container remove 5f00ad357a2531e082fab5c7364c73d16130057a973881faf5231843a2b722c9 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T18:45:29.861 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:29 smithi102 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi102.service: Deactivated successfully. 2024-03-26T18:45:29.861 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:29 smithi102 systemd[1]: Stopped Ceph mon.smithi102 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:45:29.861 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:29 smithi102 systemd[1]: Starting Ceph mon.smithi102 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:45:30.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:29 smithi102 podman[28984]: 2024-03-26 18:45:29.871366346 +0000 UTC m=+0.111053557 container create b5abd46c397361e721a12662f865e86dc7c3adeed8817d539e58ad91c23b5d42 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T18:45:30.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:29 smithi102 podman[28984]: 2024-03-26 18:45:29.777933461 +0000 UTC m=+0.017620650 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph:v18.2.0 2024-03-26T18:45:30.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 podman[28984]: 2024-03-26 18:45:30.065609818 +0000 UTC m=+0.305296983 container init b5abd46c397361e721a12662f865e86dc7c3adeed8817d539e58ad91c23b5d42 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, ceph=True) 2024-03-26T18:45:30.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 podman[28984]: 2024-03-26 18:45:30.068560629 +0000 UTC m=+0.308247795 container start b5abd46c397361e721a12662f865e86dc7c3adeed8817d539e58ad91c23b5d42 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable), process ceph-mon, pid 2 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: pidfile_write: ignore empty --pid-file 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: load: jerasure load: lrc 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: RocksDB version: 7.9.2 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Git sha 0 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Compile date 2023-08-03 19:21:13 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: DB SUMMARY 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: DB Session ID: ZA64HXHW90L012OUHSKX 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: CURRENT file: CURRENT 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T18:45:30.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi102/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T18:45:30.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi102/store.db: 000009.log size: 98292 ; 2024-03-26T18:45:30.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.error_if_exists: 0 2024-03-26T18:45:30.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.create_if_missing: 0 2024-03-26T18:45:30.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.paranoid_checks: 1 2024-03-26T18:45:30.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.env: 0x55fbd64ee720 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.info_log: 0x55fbd7ed32c0 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.statistics: (nil) 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.use_fsync: 0 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_log_file_size: 0 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T18:45:30.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.allow_fallocate: 1 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.use_direct_reads: 0 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.db_log_dir: 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.wal_dir: 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T18:45:30.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.write_buffer_manager: 0x55fbd71443c0 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.unordered_write: 0 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T18:45:30.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.row_cache: None 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.wal_filter: None 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.two_write_queues: 0 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.wal_compression: 0 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.atomic_flush: 0 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.log_readahead_size: 0 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T18:45:30.131 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_background_jobs: 2 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_background_compactions: -1 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_subcompactions: 1 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T18:45:30.132 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T18:45:30.133 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_open_files: -1 2024-03-26T18:45:30.133 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T18:45:30.133 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T18:45:30.133 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T18:45:30.133 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T18:45:30.133 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_background_flushes: -1 2024-03-26T18:45:30.133 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Compression algorithms supported: 2024-03-26T18:45:30.133 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T18:45:30.133 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: kZSTD supported: 0 2024-03-26T18:45:30.133 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: kXpressCompression supported: 0 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: kZlibCompression supported: 1 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: kSnappyCompression supported: 1 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: kLZ4Compression supported: 1 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: kBZip2Compression supported: 0 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi102/store.db/MANIFEST-000010 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.merge_operator: 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_filter: None 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T18:45:30.134 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fbd7ed33c0) 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: index_type: 0 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: data_block_index_type: 0 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: index_shortening: 1 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: checksum: 4 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: no_block_cache: 0 2024-03-26T18:45:30.135 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_cache: 0x55fbd71e51f0 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_cache_options: 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: capacity : 536870912 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: num_shard_bits : 4 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: strict_capacity_limit : 0 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_cache_compressed: (nil) 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: persistent_cache: (nil) 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_size: 4096 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_size_deviation: 10 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_restart_interval: 16 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: index_block_restart_interval: 1 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: metadata_block_size: 4096 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: partition_filters: 0 2024-03-26T18:45:30.136 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: use_delta_encoding: 1 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: filter_policy: bloomfilter 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: whole_key_filtering: 1 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: verify_compression: 0 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: format_version: 5 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: enable_index_compression: 1 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_align: 0 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: max_auto_readahead_size: 262144 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: prepopulate_block_cache: 0 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: initial_auto_readahead_size: 8192 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T18:45:30.137 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compression: NoCompression 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.num_levels: 7 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T18:45:30.138 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T18:45:30.139 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T18:45:30.140 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T18:45:30.141 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T18:45:30.141 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T18:45:30.141 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T18:45:30.141 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T18:45:30.141 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T18:45:30.141 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T18:45:30.141 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T18:45:30.141 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T18:45:30.141 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T18:45:30.141 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T18:45:30.141 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.table_properties_collectors: 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.inplace_update_support: 0 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.bloom_locality: 0 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.max_successive_merges: 0 2024-03-26T18:45:30.142 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T18:45:30.143 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T18:45:30.143 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T18:45:30.143 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T18:45:30.143 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.ttl: 2592000 2024-03-26T18:45:30.143 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T18:45:30.143 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T18:45:30.143 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T18:45:30.143 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.enable_blob_files: false 2024-03-26T18:45:30.143 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.min_blob_size: 0 2024-03-26T18:45:30.143 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T18:45:30.144 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T18:45:30.144 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T18:45:30.144 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T18:45:30.144 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T18:45:30.144 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T18:45:30.145 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T18:45:30.145 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T18:45:30.145 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi102/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T18:45:30.145 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T18:45:30.145 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 286a59b9-b308-4ae2-b340-ff4d305fe399 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711478730096356, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711478730097384, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93543, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91565, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84174, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711478730, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "286a59b9-b308-4ae2-b340-ff4d305fe399", "db_session_id": "ZA64HXHW90L012OUHSKX", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711478730097466, "job": 1, "event": "recovery_finished"} 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi102/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fbd7282000 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: DB pointer 0x55fbd726e000 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: ** DB Stats ** 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T18:45:30.146 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: L0 2/0 93.22 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 129.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Sum 2/0 93.22 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 129.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 129.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-26T18:45:30.147 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 129.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Cumulative compaction: 0.00 GB write, 29.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T18:45:30.148 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Interval compaction: 0.00 GB write, 29.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T18:45:30.149 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T18:45:30.149 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Block cache BinnedLRUCache@0x55fbd71e51f0#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T18:45:30.149 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-26T18:45:30.149 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:45:30.149 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T18:45:30.149 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: starting mon.smithi102 rank 0 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi102 fsid ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:45:30.149 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: mon.smithi102@-1(???) e1 preinit fsid ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:45:30.149 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: mon.smithi102@-1(???).mds e1 new map 2024-03-26T18:45:30.149 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: mon.smithi102@-1(???).mds e1 print_map 2024-03-26T18:45:30.150 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: e1 2024-03-26T18:45:30.150 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:45:30.150 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:45:30.150 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: legacy client fscid: -1 2024-03-26T18:45:30.150 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:45:30.150 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: No filesystems configured 2024-03-26T18:45:30.150 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: mon.smithi102@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T18:45:30.150 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: mon.smithi102@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T18:45:30.150 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: mon.smithi102@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T18:45:30.150 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: mon.smithi102@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T18:45:30.150 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: mon.smithi102@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta expand map: {default=false} 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta from 'false' to 'false' 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta expanded map: {default=false} 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta expand map: {default=info} 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta from 'info' to 'info' 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta expanded map: {default=info} 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta expand map: {default=daemon} 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta from 'daemon' to 'daemon' 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta expanded map: {default=daemon} 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta expand map: {default=debug} 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta from 'debug' to 'debug' 2024-03-26T18:45:30.151 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: expand_channel_meta expanded map: {default=debug} 2024-03-26T18:45:30.152 INFO:teuthology.orchestra.run.smithi102.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T18:45:30.450 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 bash[28984]: b5abd46c397361e721a12662f865e86dc7c3adeed8817d539e58ad91c23b5d42 2024-03-26T18:45:30.451 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 systemd[1]: Started Ceph mon.smithi102 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:45:30.451 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: mon.smithi102 is new leader, mons smithi102 in quorum (ranks 0) 2024-03-26T18:45:30.451 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: monmap e1: 1 mons at {smithi102=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2024-03-26T18:45:30.451 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: fsmap 2024-03-26T18:45:30.451 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T18:45:30.451 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:30 smithi102 ceph-mon[29002]: mgrmap e1: no daemons active 2024-03-26T18:45:31.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.561+0000 7f687bcab700 1 Processor -- start 2024-03-26T18:45:31.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.562+0000 7f687bcab700 1 -- start start 2024-03-26T18:45:31.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.562+0000 7f687bcab700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:31.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.562+0000 7f687bcab700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6874106460 con 0x7f6874105b10 2024-03-26T18:45:31.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.562+0000 7f6879a47700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:31.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.562+0000 7f6879a47700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33148/0 (socket says 172.21.15.102:33148) 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.562+0000 7f6879a47700 1 -- 172.21.15.102:0/3042149348 learned_addr learned my addr 172.21.15.102:0/3042149348 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.562+0000 7f6879a47700 1 -- 172.21.15.102:0/3042149348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68741065a0 con 0x7f6874105b10 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.563+0000 7f6879a47700 1 --2- 172.21.15.102:0/3042149348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874105f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f686c0098a0 tx=0x7f686c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48605fba3d02326 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.563+0000 7f6878a45700 1 -- 172.21.15.102:0/3042149348 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f686c004180 con 0x7f6874105b10 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.563+0000 7f6878a45700 1 -- 172.21.15.102:0/3042149348 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f686c0042e0 con 0x7f6874105b10 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.563+0000 7f6878a45700 1 -- 172.21.15.102:0/3042149348 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f686c011a50 con 0x7f6874105b10 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.563+0000 7f687bcab700 1 -- 172.21.15.102:0/3042149348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 msgr2=0x7f6874105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.563+0000 7f687bcab700 1 --2- 172.21.15.102:0/3042149348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874105f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f686c0098a0 tx=0x7f686c009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.563+0000 7f687bcab700 1 -- 172.21.15.102:0/3042149348 shutdown_connections 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.563+0000 7f687bcab700 1 --2- 172.21.15.102:0/3042149348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874105f20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.563+0000 7f687bcab700 1 -- 172.21.15.102:0/3042149348 >> 172.21.15.102:0/3042149348 conn(0x7f6874101180 msgr2=0x7f6874103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.563+0000 7f687bcab700 1 -- 172.21.15.102:0/3042149348 shutdown_connections 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.563+0000 7f687bcab700 1 -- 172.21.15.102:0/3042149348 wait complete. 2024-03-26T18:45:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.564+0000 7f687bcab700 1 Processor -- start 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.564+0000 7f687bcab700 1 -- start start 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.564+0000 7f687bcab700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.564+0000 7f687bcab700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6874199d90 con 0x7f6874105b10 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.564+0000 7f6879a47700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.564+0000 7f6879a47700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33152/0 (socket says 172.21.15.102:33152) 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.564+0000 7f6879a47700 1 -- 172.21.15.102:0/4282370790 learned_addr learned my addr 172.21.15.102:0/4282370790 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.565+0000 7f6879a47700 1 -- 172.21.15.102:0/4282370790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c009550 con 0x7f6874105b10 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.565+0000 7f6879a47700 1 --2- 172.21.15.102:0/4282370790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874199850 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f686c0119e0 tx=0x7f686c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.565+0000 7f686affd700 1 -- 172.21.15.102:0/4282370790 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f686c003b90 con 0x7f6874105b10 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.565+0000 7f686affd700 1 -- 172.21.15.102:0/4282370790 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f686c01a8a0 con 0x7f6874105b10 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.565+0000 7f687bcab700 1 -- 172.21.15.102:0/4282370790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6874199f90 con 0x7f6874105b10 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.565+0000 7f686affd700 1 -- 172.21.15.102:0/4282370790 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f686c019930 con 0x7f6874105b10 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.565+0000 7f687bcab700 1 -- 172.21.15.102:0/4282370790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f687419a3b0 con 0x7f6874105b10 2024-03-26T18:45:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.565+0000 7f686affd700 1 -- 172.21.15.102:0/4282370790 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f686c02d440 con 0x7f6874105b10 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.565+0000 7f686affd700 1 -- 172.21.15.102:0/4282370790 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f686c024860 con 0x7f6874105b10 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.566+0000 7f687bcab700 1 -- 172.21.15.102:0/4282370790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6874062380 con 0x7f6874105b10 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.568+0000 7f686affd700 1 -- 172.21.15.102:0/4282370790 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f686c01aa10 con 0x7f6874105b10 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.635+0000 7f687bcab700 1 -- 172.21.15.102:0/4282370790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f6874107980 con 0x7f6874105b10 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.640+0000 7f686affd700 1 -- 172.21.15.102:0/4282370790 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f686c01abb0 con 0x7f6874105b10 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.641+0000 7f687bcab700 1 -- 172.21.15.102:0/4282370790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 msgr2=0x7f6874199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.641+0000 7f687bcab700 1 --2- 172.21.15.102:0/4282370790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874199850 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f686c0119e0 tx=0x7f686c011ac0 comp rx=0 tx=0).stop 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.641+0000 7f687bcab700 1 -- 172.21.15.102:0/4282370790 shutdown_connections 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.641+0000 7f687bcab700 1 --2- 172.21.15.102:0/4282370790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6874105b10 0x7f6874199850 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.641+0000 7f687bcab700 1 -- 172.21.15.102:0/4282370790 >> 172.21.15.102:0/4282370790 conn(0x7f6874101180 msgr2=0x7f6874101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.641+0000 7f687bcab700 1 -- 172.21.15.102:0/4282370790 shutdown_connections 2024-03-26T18:45:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:30.641+0000 7f687bcab700 1 -- 172.21.15.102:0/4282370790 wait complete. 2024-03-26T18:45:31.155 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T18:45:31.169 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T18:45:31.170 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mgr... 2024-03-26T18:45:31.170 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T18:45:31.170 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T18:45:31.170 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:8443 ... 2024-03-26T18:45:31.413 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mgr.smithi102.rsfpnv 2024-03-26T18:45:31.414 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mgr.smithi102.rsfpnv.service: Unit ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mgr.smithi102.rsfpnv.service not loaded. 2024-03-26T18:45:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ed4e027c-eba0-11ee-95d0-87774f69a715.target.wants/ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mgr.smithi102.rsfpnv.service → /etc/systemd/system/ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@.service. 2024-03-26T18:45:31.874 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:31 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/4282370790' entity='client.admin' 2024-03-26T18:45:34.918 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T18:45:34.918 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-26T18:45:34.926 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T18:45:34.926 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-26T18:45:34.926 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-26T18:45:34.926 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T18:45:34.936 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T18:45:34.936 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-26T18:45:34.945 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T18:45:34.945 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-26T18:45:34.945 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-26T18:45:34.945 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-03-26T18:45:34.945 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr to start... 2024-03-26T18:45:34.945 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr... 2024-03-26T18:45:38.256 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:38 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2047593282' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T18:45:41.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: Activating manager daemon smithi102.rsfpnv 2024-03-26T18:45:41.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: mgrmap e2: smithi102.rsfpnv(active, starting, since 0.00286591s) 2024-03-26T18:45:41.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: from='mgr.14100 172.21.15.102:0/1832622471' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T18:45:41.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: from='mgr.14100 172.21.15.102:0/1832622471' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T18:45:41.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: from='mgr.14100 172.21.15.102:0/1832622471' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T18:45:41.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: from='mgr.14100 172.21.15.102:0/1832622471' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:45:41.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: from='mgr.14100 172.21.15.102:0/1832622471' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi102.rsfpnv", "id": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:45:41.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: Manager daemon smithi102.rsfpnv is now available 2024-03-26T18:45:41.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: from='mgr.14100 172.21.15.102:0/1832622471' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:45:41.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: from='mgr.14100 172.21.15.102:0/1832622471' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/trash_purge_schedule"}]: dispatch 2024-03-26T18:45:41.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: from='mgr.14100 172.21.15.102:0/1832622471' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:45:41.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: from='mgr.14100 172.21.15.102:0/1832622471' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:45:41.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:41 smithi102 ceph-mon[29002]: from='mgr.14100 172.21.15.102:0/1832622471' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:45:42.873 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:42 smithi102 ceph-mon[29002]: mgrmap e3: smithi102.rsfpnv(active, since 1.00722s) 2024-03-26T18:45:43.951 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-26T18:45:43.951 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-26T18:45:43.951 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "ed4e027c-eba0-11ee-95d0-87774f69a715", 2024-03-26T18:45:43.951 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T18:45:43.951 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T18:45:43.951 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T18:45:43.951 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T18:45:43.952 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:43.952 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T18:45:43.952 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T18:45:43.952 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-03-26T18:45:43.952 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-26T18:45:43.952 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T18:45:43.952 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "smithi102" 2024-03-26T18:45:43.952 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-26T18:45:43.952 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-26T18:45:43.952 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T18:45:43.952 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:43.953 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T18:45:43.954 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:43.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T18:45:25.446878+0000", 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.041+0000 7f317e427700 1 Processor -- start 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.041+0000 7f317e427700 1 -- start start 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.042+0000 7f317e427700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.042+0000 7f317e427700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3178105350 con 0x7f3178104a00 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.042+0000 7f3177fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.042+0000 7f3177fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33184/0 (socket says 172.21.15.102:33184) 2024-03-26T18:45:43.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.042+0000 7f3177fff700 1 -- 172.21.15.102:0/1674260308 learned_addr learned my addr 172.21.15.102:0/1674260308 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.042+0000 7f3177fff700 1 -- 172.21.15.102:0/1674260308 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3178105490 con 0x7f3178104a00 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.042+0000 7f3177fff700 1 --2- 172.21.15.102:0/1674260308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178104e10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f31680098a0 tx=0x7f3168009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd324818ab92ef4 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.042+0000 7f3176ffd700 1 -- 172.21.15.102:0/1674260308 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3168004180 con 0x7f3178104a00 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.043+0000 7f3176ffd700 1 -- 172.21.15.102:0/1674260308 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f31680042e0 con 0x7f3178104a00 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.043+0000 7f3176ffd700 1 -- 172.21.15.102:0/1674260308 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3168011a50 con 0x7f3178104a00 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.043+0000 7f317e427700 1 -- 172.21.15.102:0/1674260308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 msgr2=0x7f3178104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.043+0000 7f317e427700 1 --2- 172.21.15.102:0/1674260308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178104e10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f31680098a0 tx=0x7f3168009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.043+0000 7f317e427700 1 -- 172.21.15.102:0/1674260308 shutdown_connections 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.043+0000 7f317e427700 1 --2- 172.21.15.102:0/1674260308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178104e10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.043+0000 7f317e427700 1 -- 172.21.15.102:0/1674260308 >> 172.21.15.102:0/1674260308 conn(0x7f3178100070 msgr2=0x7f3178102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.043+0000 7f317e427700 1 -- 172.21.15.102:0/1674260308 shutdown_connections 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.043+0000 7f317e427700 1 -- 172.21.15.102:0/1674260308 wait complete. 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.044+0000 7f317e427700 1 Processor -- start 2024-03-26T18:45:43.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.044+0000 7f317e427700 1 -- start start 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.044+0000 7f317e427700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.044+0000 7f317e427700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3178199d70 con 0x7f3178104a00 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.044+0000 7f3177fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.044+0000 7f3177fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33200/0 (socket says 172.21.15.102:33200) 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.044+0000 7f3177fff700 1 -- 172.21.15.102:0/2047593282 learned_addr learned my addr 172.21.15.102:0/2047593282 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.044+0000 7f3177fff700 1 -- 172.21.15.102:0/2047593282 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3168009550 con 0x7f3178104a00 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.045+0000 7f3177fff700 1 --2- 172.21.15.102:0/2047593282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178199830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3168000c00 tx=0x7f3168011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.045+0000 7f31757fa700 1 -- 172.21.15.102:0/2047593282 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3168003b90 con 0x7f3178104a00 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.045+0000 7f31757fa700 1 -- 172.21.15.102:0/2047593282 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f31680248b0 con 0x7f3178104a00 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.045+0000 7f31757fa700 1 -- 172.21.15.102:0/2047593282 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f316801a8e0 con 0x7f3178104a00 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.045+0000 7f317e427700 1 -- 172.21.15.102:0/2047593282 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3178199f70 con 0x7f3178104a00 2024-03-26T18:45:43.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.045+0000 7f317e427700 1 -- 172.21.15.102:0/2047593282 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317819a390 con 0x7f3178104a00 2024-03-26T18:45:43.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.045+0000 7f31757fa700 1 -- 172.21.15.102:0/2047593282 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f316802d3f0 con 0x7f3178104a00 2024-03-26T18:45:43.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.045+0000 7f31757fa700 1 -- 172.21.15.102:0/2047593282 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f31680194a0 con 0x7f3178104a00 2024-03-26T18:45:43.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.046+0000 7f317e427700 1 -- 172.21.15.102:0/2047593282 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3178062380 con 0x7f3178104a00 2024-03-26T18:45:43.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.048+0000 7f31757fa700 1 -- 172.21.15.102:0/2047593282 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f3168024db0 con 0x7f3178104a00 2024-03-26T18:45:43.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.121+0000 7f317e427700 1 -- 172.21.15.102:0/2047593282 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3178106870 con 0x7f3178104a00 2024-03-26T18:45:43.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.122+0000 7f31757fa700 1 -- 172.21.15.102:0/2047593282 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f3168024ad0 con 0x7f3178104a00 2024-03-26T18:45:43.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.123+0000 7f317e427700 1 -- 172.21.15.102:0/2047593282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 msgr2=0x7f3178199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:43.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.123+0000 7f317e427700 1 --2- 172.21.15.102:0/2047593282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178199830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3168000c00 tx=0x7f3168011ab0 comp rx=0 tx=0).stop 2024-03-26T18:45:43.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.123+0000 7f317e427700 1 -- 172.21.15.102:0/2047593282 shutdown_connections 2024-03-26T18:45:43.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.123+0000 7f317e427700 1 --2- 172.21.15.102:0/2047593282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3178104a00 0x7f3178199830 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:43.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.123+0000 7f317e427700 1 -- 172.21.15.102:0/2047593282 >> 172.21.15.102:0/2047593282 conn(0x7f3178100070 msgr2=0x7f3178074070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:43.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.123+0000 7f317e427700 1 -- 172.21.15.102:0/2047593282 shutdown_connections 2024-03-26T18:45:43.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:38.123+0000 7f317e427700 1 -- 172.21.15.102:0/2047593282 wait complete. 2024-03-26T18:45:43.960 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (1/15)... 2024-03-26T18:45:44.874 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:44 smithi102 ceph-mon[29002]: mgrmap e4: smithi102.rsfpnv(active, since 3s) 2024-03-26T18:45:46.874 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:46 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/408236629' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T18:45:47.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-26T18:45:47.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-26T18:45:47.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "ed4e027c-eba0-11ee-95d0-87774f69a715", 2024-03-26T18:45:47.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T18:45:47.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T18:45:47.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "smithi102" 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-26T18:45:47.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T18:45:47.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T18:45:47.161 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T18:45:25.446878+0000", 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-26T18:45:47.162 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.337+0000 7f721a6b1700 1 Processor -- start 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.337+0000 7f721a6b1700 1 -- start start 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.337+0000 7f721a6b1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214108a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.337+0000 7f721a6b1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7214079d30 con 0x7f7214106660 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.337+0000 7f7213fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214108a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.337+0000 7f7213fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214108a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58516/0 (socket says 172.21.15.102:58516) 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.337+0000 7f7213fff700 1 -- 172.21.15.102:0/1234827449 learned_addr learned my addr 172.21.15.102:0/1234827449 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.338+0000 7f7213fff700 1 -- 172.21.15.102:0/1234827449 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7214079e70 con 0x7f7214106660 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.338+0000 7f7213fff700 1 --2- 172.21.15.102:0/1234827449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214108a40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f72040098a0 tx=0x7f7204009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2916df1994c73751 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.338+0000 7f7212ffd700 1 -- 172.21.15.102:0/1234827449 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7204004180 con 0x7f7214106660 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.338+0000 7f7212ffd700 1 -- 172.21.15.102:0/1234827449 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f72040042e0 con 0x7f7214106660 2024-03-26T18:45:47.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.338+0000 7f7212ffd700 1 -- 172.21.15.102:0/1234827449 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7204011a50 con 0x7f7214106660 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.339+0000 7f721a6b1700 1 -- 172.21.15.102:0/1234827449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 msgr2=0x7f7214108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.339+0000 7f721a6b1700 1 --2- 172.21.15.102:0/1234827449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214108a40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f72040098a0 tx=0x7f7204009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.339+0000 7f721a6b1700 1 -- 172.21.15.102:0/1234827449 shutdown_connections 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.339+0000 7f721a6b1700 1 --2- 172.21.15.102:0/1234827449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214108a40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.339+0000 7f721a6b1700 1 -- 172.21.15.102:0/1234827449 >> 172.21.15.102:0/1234827449 conn(0x7f7214100060 msgr2=0x7f7214102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.339+0000 7f721a6b1700 1 -- 172.21.15.102:0/1234827449 shutdown_connections 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.339+0000 7f721a6b1700 1 -- 172.21.15.102:0/1234827449 wait complete. 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.339+0000 7f721a6b1700 1 Processor -- start 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.339+0000 7f721a6b1700 1 -- start start 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.340+0000 7f721a6b1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.340+0000 7f721a6b1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7214199d70 con 0x7f7214106660 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.340+0000 7f7213fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.340+0000 7f7213fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58518/0 (socket says 172.21.15.102:58518) 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.340+0000 7f7213fff700 1 -- 172.21.15.102:0/408236629 learned_addr learned my addr 172.21.15.102:0/408236629 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:47.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.340+0000 7f7213fff700 1 -- 172.21.15.102:0/408236629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7204009550 con 0x7f7214106660 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.340+0000 7f7213fff700 1 --2- 172.21.15.102:0/408236629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214199830 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7204006af0 tx=0x7f7204011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.340+0000 7f72117fa700 1 -- 172.21.15.102:0/408236629 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7204003b90 con 0x7f7214106660 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.341+0000 7f72117fa700 1 -- 172.21.15.102:0/408236629 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f720401a8a0 con 0x7f7214106660 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.341+0000 7f721a6b1700 1 -- 172.21.15.102:0/408236629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7214199fd0 con 0x7f7214106660 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.341+0000 7f72117fa700 1 -- 172.21.15.102:0/408236629 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7204019880 con 0x7f7214106660 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.341+0000 7f721a6b1700 1 -- 172.21.15.102:0/408236629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721419a450 con 0x7f7214106660 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.341+0000 7f72117fa700 1 -- 172.21.15.102:0/408236629 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f7204019b40 con 0x7f7214106660 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.342+0000 7f72117fa700 1 --2- 172.21.15.102:0/408236629 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f71fc038360 0x7f71fc03a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.342+0000 7f72117fa700 1 -- 172.21.15.102:0/408236629 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f720404c6e0 con 0x7f7214106660 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.342+0000 7f72137fe700 1 --2- 172.21.15.102:0/408236629 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f71fc038360 0x7f71fc03a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.342+0000 7f721a6b1700 1 -- 172.21.15.102:0/408236629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7214109b40 con 0x7f7214106660 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.342+0000 7f72137fe700 1 --2- 172.21.15.102:0/408236629 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f71fc038360 0x7f71fc03a810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7208006fd0 tx=0x7f7208006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.347+0000 7f72117fa700 1 -- 172.21.15.102:0/408236629 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f720402c3a0 con 0x7f7214106660 2024-03-26T18:45:47.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.613+0000 7f721a6b1700 1 -- 172.21.15.102:0/408236629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7214109d60 con 0x7f7214106660 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.614+0000 7f72117fa700 1 -- 172.21.15.102:0/408236629 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f720404a020 con 0x7f7214106660 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.616+0000 7f721a6b1700 1 -- 172.21.15.102:0/408236629 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f71fc038360 msgr2=0x7f71fc03a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.616+0000 7f721a6b1700 1 --2- 172.21.15.102:0/408236629 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f71fc038360 0x7f71fc03a810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7208006fd0 tx=0x7f7208006e40 comp rx=0 tx=0).stop 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.616+0000 7f721a6b1700 1 -- 172.21.15.102:0/408236629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 msgr2=0x7f7214199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.616+0000 7f721a6b1700 1 --2- 172.21.15.102:0/408236629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214199830 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7204006af0 tx=0x7f7204011a90 comp rx=0 tx=0).stop 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.616+0000 7f721a6b1700 1 -- 172.21.15.102:0/408236629 shutdown_connections 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.616+0000 7f721a6b1700 1 --2- 172.21.15.102:0/408236629 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f71fc038360 0x7f71fc03a810 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.616+0000 7f721a6b1700 1 --2- 172.21.15.102:0/408236629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7214106660 0x7f7214199830 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.616+0000 7f721a6b1700 1 -- 172.21.15.102:0/408236629 >> 172.21.15.102:0/408236629 conn(0x7f7214100060 msgr2=0x7f72141017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.616+0000 7f721a6b1700 1 -- 172.21.15.102:0/408236629 shutdown_connections 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:46.616+0000 7f721a6b1700 1 -- 172.21.15.102:0/408236629 wait complete. 2024-03-26T18:45:47.166 INFO:teuthology.orchestra.run.smithi102.stdout:mgr is available 2024-03-26T18:45:48.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:47 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/625288266' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T18:45:48.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-26T18:45:48.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T18:45:48.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:45:48.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T18:45:48.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T18:45:48.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T18:45:48.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T18:45:48.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-26T18:45:48.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.580+0000 7f0275de0700 1 Processor -- start 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.580+0000 7f0275de0700 1 -- start start 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.580+0000 7f0275de0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.580+0000 7f0275de0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0270106460 con 0x7f0270105b10 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.580+0000 7f026f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.580+0000 7f026f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58528/0 (socket says 172.21.15.102:58528) 2024-03-26T18:45:48.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.580+0000 7f026f7fe700 1 -- 172.21.15.102:0/3299041370 learned_addr learned my addr 172.21.15.102:0/3299041370 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.581+0000 7f026f7fe700 1 -- 172.21.15.102:0/3299041370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02701065a0 con 0x7f0270105b10 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.581+0000 7f026f7fe700 1 --2- 172.21.15.102:0/3299041370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270105f20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02600098a0 tx=0x7f0260009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87046215c577b0c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.581+0000 7f026e7fc700 1 -- 172.21.15.102:0/3299041370 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0260004180 con 0x7f0270105b10 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.581+0000 7f026e7fc700 1 -- 172.21.15.102:0/3299041370 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f02600042e0 con 0x7f0270105b10 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.581+0000 7f026e7fc700 1 -- 172.21.15.102:0/3299041370 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0260011a50 con 0x7f0270105b10 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.582+0000 7f0275de0700 1 -- 172.21.15.102:0/3299041370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 msgr2=0x7f0270105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.582+0000 7f0275de0700 1 --2- 172.21.15.102:0/3299041370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270105f20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02600098a0 tx=0x7f0260009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.582+0000 7f0275de0700 1 -- 172.21.15.102:0/3299041370 shutdown_connections 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.582+0000 7f0275de0700 1 --2- 172.21.15.102:0/3299041370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270105f20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.582+0000 7f0275de0700 1 -- 172.21.15.102:0/3299041370 >> 172.21.15.102:0/3299041370 conn(0x7f0270101180 msgr2=0x7f0270103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.582+0000 7f0275de0700 1 -- 172.21.15.102:0/3299041370 shutdown_connections 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.582+0000 7f0275de0700 1 -- 172.21.15.102:0/3299041370 wait complete. 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.582+0000 7f0275de0700 1 Processor -- start 2024-03-26T18:45:48.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.582+0000 7f0275de0700 1 -- start start 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f0275de0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f0275de0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0270199d50 con 0x7f0270105b10 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f026f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f026f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58536/0 (socket says 172.21.15.102:58536) 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f026f7fe700 1 -- 172.21.15.102:0/625288266 learned_addr learned my addr 172.21.15.102:0/625288266 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f026f7fe700 1 -- 172.21.15.102:0/625288266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0260009550 con 0x7f0270105b10 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f026f7fe700 1 --2- 172.21.15.102:0/625288266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270199810 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f02600119e0 tx=0x7f0260011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f026cff9700 1 -- 172.21.15.102:0/625288266 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0260003b90 con 0x7f0270105b10 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f026cff9700 1 -- 172.21.15.102:0/625288266 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f026001a8a0 con 0x7f0270105b10 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f026cff9700 1 -- 172.21.15.102:0/625288266 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0260019930 con 0x7f0270105b10 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f0275de0700 1 -- 172.21.15.102:0/625288266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0270199f50 con 0x7f0270105b10 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.583+0000 7f0275de0700 1 -- 172.21.15.102:0/625288266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027019a370 con 0x7f0270105b10 2024-03-26T18:45:48.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.584+0000 7f026cff9700 1 -- 172.21.15.102:0/625288266 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f0260019bf0 con 0x7f0270105b10 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.584+0000 7f026cff9700 1 --2- 172.21.15.102:0/625288266 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f0258038360 0x7f025803a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.584+0000 7f026cff9700 1 -- 172.21.15.102:0/625288266 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f026004c6c0 con 0x7f0270105b10 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.584+0000 7f0275de0700 1 -- 172.21.15.102:0/625288266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0270062380 con 0x7f0270105b10 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.584+0000 7f026effd700 1 --2- 172.21.15.102:0/625288266 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f0258038360 0x7f025803a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.585+0000 7f026effd700 1 --2- 172.21.15.102:0/625288266 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f0258038360 0x7f025803a810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0264006fd0 tx=0x7f0264006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.589+0000 7f026cff9700 1 -- 172.21.15.102:0/625288266 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f026001a430 con 0x7f0270105b10 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.781+0000 7f0275de0700 1 -- 172.21.15.102:0/625288266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f027019e5c0 con 0x7f0270105b10 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.782+0000 7f026cff9700 1 -- 172.21.15.102:0/625288266 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f0260018350 con 0x7f0270105b10 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.784+0000 7f0275de0700 1 -- 172.21.15.102:0/625288266 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f0258038360 msgr2=0x7f025803a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.784+0000 7f0275de0700 1 --2- 172.21.15.102:0/625288266 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f0258038360 0x7f025803a810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0264006fd0 tx=0x7f0264006e40 comp rx=0 tx=0).stop 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.784+0000 7f0275de0700 1 -- 172.21.15.102:0/625288266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 msgr2=0x7f0270199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.784+0000 7f0275de0700 1 --2- 172.21.15.102:0/625288266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270199810 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f02600119e0 tx=0x7f0260011ac0 comp rx=0 tx=0).stop 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.784+0000 7f0275de0700 1 -- 172.21.15.102:0/625288266 shutdown_connections 2024-03-26T18:45:48.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.784+0000 7f0275de0700 1 --2- 172.21.15.102:0/625288266 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f0258038360 0x7f025803a810 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:48.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.784+0000 7f0275de0700 1 --2- 172.21.15.102:0/625288266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0270105b10 0x7f0270199810 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:48.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.784+0000 7f0275de0700 1 -- 172.21.15.102:0/625288266 >> 172.21.15.102:0/625288266 conn(0x7f0270101180 msgr2=0x7f0270101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:48.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.785+0000 7f0275de0700 1 -- 172.21.15.102:0/625288266 shutdown_connections 2024-03-26T18:45:48.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:47.785+0000 7f0275de0700 1 -- 172.21.15.102:0/625288266 wait complete. 2024-03-26T18:45:48.540 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling cephadm module... 2024-03-26T18:45:49.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:49 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3583544109' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T18:45:50.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.969+0000 7f35eaac7700 1 Processor -- start 2024-03-26T18:45:50.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.969+0000 7f35eaac7700 1 -- start start 2024-03-26T18:45:50.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.969+0000 7f35eaac7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e407a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:50.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.969+0000 7f35eaac7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e407a830 con 0x7f35e407bdf0 2024-03-26T18:45:50.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.969+0000 7f35e8863700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e407a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:50.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.969+0000 7f35e8863700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e407a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37628/0 (socket says 172.21.15.102:37628) 2024-03-26T18:45:50.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.969+0000 7f35e8863700 1 -- 172.21.15.102:0/461645761 learned_addr learned my addr 172.21.15.102:0/461645761 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:50.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.969+0000 7f35e8863700 1 -- 172.21.15.102:0/461645761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e407a970 con 0x7f35e407bdf0 2024-03-26T18:45:50.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.970+0000 7f35e8863700 1 --2- 172.21.15.102:0/461645761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e407a2f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f35d80098a0 tx=0x7f35d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd62b744e268297 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:50.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.970+0000 7f35e37fe700 1 -- 172.21.15.102:0/461645761 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35d8004180 con 0x7f35e407bdf0 2024-03-26T18:45:50.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.970+0000 7f35e37fe700 1 -- 172.21.15.102:0/461645761 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f35d80042e0 con 0x7f35e407bdf0 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.970+0000 7f35e37fe700 1 -- 172.21.15.102:0/461645761 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35d8011a50 con 0x7f35e407bdf0 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.970+0000 7f35eaac7700 1 -- 172.21.15.102:0/461645761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 msgr2=0x7f35e407a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.970+0000 7f35eaac7700 1 --2- 172.21.15.102:0/461645761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e407a2f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f35d80098a0 tx=0x7f35d8009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.970+0000 7f35eaac7700 1 -- 172.21.15.102:0/461645761 shutdown_connections 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.970+0000 7f35eaac7700 1 --2- 172.21.15.102:0/461645761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e407a2f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.970+0000 7f35eaac7700 1 -- 172.21.15.102:0/461645761 >> 172.21.15.102:0/461645761 conn(0x7f35e4101180 msgr2=0x7f35e4103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.970+0000 7f35eaac7700 1 -- 172.21.15.102:0/461645761 shutdown_connections 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.970+0000 7f35eaac7700 1 -- 172.21.15.102:0/461645761 wait complete. 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.971+0000 7f35eaac7700 1 Processor -- start 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.971+0000 7f35eaac7700 1 -- start start 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.971+0000 7f35eaac7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e419ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.971+0000 7f35eaac7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e419bfd0 con 0x7f35e407bdf0 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.971+0000 7f35e8863700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e419ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.971+0000 7f35e8863700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e419ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37630/0 (socket says 172.21.15.102:37630) 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.971+0000 7f35e8863700 1 -- 172.21.15.102:0/3583544109 learned_addr learned my addr 172.21.15.102:0/3583544109 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:50.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.972+0000 7f35e8863700 1 -- 172.21.15.102:0/3583544109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d8009550 con 0x7f35e407bdf0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.972+0000 7f35e8863700 1 --2- 172.21.15.102:0/3583544109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e419ba90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f35d8006af0 tx=0x7f35d8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.972+0000 7f35e1ffb700 1 -- 172.21.15.102:0/3583544109 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35d8003b90 con 0x7f35e407bdf0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.972+0000 7f35e1ffb700 1 -- 172.21.15.102:0/3583544109 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f35d801a8a0 con 0x7f35e407bdf0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.972+0000 7f35eaac7700 1 -- 172.21.15.102:0/3583544109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e419c1d0 con 0x7f35e407bdf0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.972+0000 7f35e1ffb700 1 -- 172.21.15.102:0/3583544109 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35d8019880 con 0x7f35e407bdf0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.972+0000 7f35eaac7700 1 -- 172.21.15.102:0/3583544109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e419c5f0 con 0x7f35e407bdf0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.973+0000 7f35e1ffb700 1 -- 172.21.15.102:0/3583544109 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f35d8019af0 con 0x7f35e407bdf0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.973+0000 7f35e1ffb700 1 --2- 172.21.15.102:0/3583544109 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f35d40383b0 0x7f35d403a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.973+0000 7f35eaac7700 1 -- 172.21.15.102:0/3583544109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e4062380 con 0x7f35e407bdf0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.973+0000 7f35e1ffb700 1 -- 172.21.15.102:0/3583544109 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f35d804c8f0 con 0x7f35e407bdf0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.973+0000 7f35e3fff700 1 --2- 172.21.15.102:0/3583544109 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f35d40383b0 0x7f35d403a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.974+0000 7f35e3fff700 1 --2- 172.21.15.102:0/3583544109 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f35d40383b0 0x7f35d403a860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35d0006fd0 tx=0x7f35d0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:48.978+0000 7f35e1ffb700 1 -- 172.21.15.102:0/3583544109 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f35d802c3a0 con 0x7f35e407bdf0 2024-03-26T18:45:50.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.217+0000 7f35eaac7700 1 -- 172.21.15.102:0/3583544109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f35e419f030 con 0x7f35e407bdf0 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.270+0000 7f35e1ffb700 1 -- 172.21.15.102:0/3583544109 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f35d800fda0 con 0x7f35e407bdf0 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.272+0000 7f35e1ffb700 1 -- 172.21.15.102:0/3583544109 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f35d8023550 con 0x7f35e407bdf0 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.274+0000 7f35eaac7700 1 -- 172.21.15.102:0/3583544109 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f35d40383b0 msgr2=0x7f35d403a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.274+0000 7f35eaac7700 1 --2- 172.21.15.102:0/3583544109 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f35d40383b0 0x7f35d403a860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35d0006fd0 tx=0x7f35d0006e40 comp rx=0 tx=0).stop 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.274+0000 7f35eaac7700 1 -- 172.21.15.102:0/3583544109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 msgr2=0x7f35e419ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.274+0000 7f35eaac7700 1 --2- 172.21.15.102:0/3583544109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e419ba90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f35d8006af0 tx=0x7f35d8011a90 comp rx=0 tx=0).stop 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.274+0000 7f35eaac7700 1 -- 172.21.15.102:0/3583544109 shutdown_connections 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.274+0000 7f35eaac7700 1 --2- 172.21.15.102:0/3583544109 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f35d40383b0 0x7f35d403a860 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.274+0000 7f35eaac7700 1 --2- 172.21.15.102:0/3583544109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35e407bdf0 0x7f35e419ba90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.274+0000 7f35eaac7700 1 -- 172.21.15.102:0/3583544109 >> 172.21.15.102:0/3583544109 conn(0x7f35e4101180 msgr2=0x7f35e41027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.275+0000 7f35eaac7700 1 -- 172.21.15.102:0/3583544109 shutdown_connections 2024-03-26T18:45:50.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:49.275+0000 7f35eaac7700 1 -- 172.21.15.102:0/3583544109 wait complete. 2024-03-26T18:45:50.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:50 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3583544109' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T18:45:50.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:50 smithi102 ceph-mon[29002]: mgrmap e5: smithi102.rsfpnv(active, since 8s) 2024-03-26T18:45:51.579 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:51 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1712863225' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T18:45:51.591 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-26T18:45:51.591 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T18:45:51.592 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T18:45:51.592 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "smithi102.rsfpnv", 2024-03-26T18:45:51.592 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T18:45:51.592 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-26T18:45:51.592 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.761+0000 7f3cc148a700 1 Processor -- start 2024-03-26T18:45:51.592 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.761+0000 7f3cc148a700 1 -- start start 2024-03-26T18:45:51.592 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.761+0000 7f3cc148a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:51.593 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.761+0000 7f3cc148a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc106460 con 0x7f3cbc105b10 2024-03-26T18:45:51.593 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.761+0000 7f3cbaffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:51.593 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.762+0000 7f3cbaffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37654/0 (socket says 172.21.15.102:37654) 2024-03-26T18:45:51.593 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.762+0000 7f3cbaffd700 1 -- 172.21.15.102:0/1634623758 learned_addr learned my addr 172.21.15.102:0/1634623758 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:51.593 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.762+0000 7f3cbaffd700 1 -- 172.21.15.102:0/1634623758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc1065a0 con 0x7f3cbc105b10 2024-03-26T18:45:51.593 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.762+0000 7f3cbaffd700 1 --2- 172.21.15.102:0/1634623758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc105f20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00098a0 tx=0x7f3cb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbd6860fb2c9dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:51.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.762+0000 7f3cb9ffb700 1 -- 172.21.15.102:0/1634623758 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cb0004180 con 0x7f3cbc105b10 2024-03-26T18:45:51.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.762+0000 7f3cb9ffb700 1 -- 172.21.15.102:0/1634623758 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3cb00042e0 con 0x7f3cbc105b10 2024-03-26T18:45:51.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.762+0000 7f3cb9ffb700 1 -- 172.21.15.102:0/1634623758 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cb0011a50 con 0x7f3cbc105b10 2024-03-26T18:45:51.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.763+0000 7f3cc148a700 1 -- 172.21.15.102:0/1634623758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 msgr2=0x7f3cbc105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:51.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.763+0000 7f3cc148a700 1 --2- 172.21.15.102:0/1634623758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc105f20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00098a0 tx=0x7f3cb0009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:51.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.763+0000 7f3cc148a700 1 -- 172.21.15.102:0/1634623758 shutdown_connections 2024-03-26T18:45:51.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.763+0000 7f3cc148a700 1 --2- 172.21.15.102:0/1634623758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc105f20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:51.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.763+0000 7f3cc148a700 1 -- 172.21.15.102:0/1634623758 >> 172.21.15.102:0/1634623758 conn(0x7f3cbc101180 msgr2=0x7f3cbc103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:51.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.763+0000 7f3cc148a700 1 -- 172.21.15.102:0/1634623758 shutdown_connections 2024-03-26T18:45:51.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.763+0000 7f3cc148a700 1 -- 172.21.15.102:0/1634623758 wait complete. 2024-03-26T18:45:51.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.763+0000 7f3cc148a700 1 Processor -- start 2024-03-26T18:45:51.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3cc148a700 1 -- start start 2024-03-26T18:45:51.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3cc148a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc07bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:51.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3cc148a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc07c180 con 0x7f3cbc105b10 2024-03-26T18:45:51.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3cbaffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc07bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:51.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3cbaffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc07bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37670/0 (socket says 172.21.15.102:37670) 2024-03-26T18:45:51.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3cbaffd700 1 -- 172.21.15.102:0/1712863225 learned_addr learned my addr 172.21.15.102:0/1712863225 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:51.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3cbaffd700 1 -- 172.21.15.102:0/1712863225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb0009550 con 0x7f3cbc105b10 2024-03-26T18:45:51.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3cbaffd700 1 --2- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc07bc40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00119e0 tx=0x7f3cb0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:51.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3ca3fff700 1 -- 172.21.15.102:0/1712863225 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cb0003b90 con 0x7f3cbc105b10 2024-03-26T18:45:51.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3ca3fff700 1 -- 172.21.15.102:0/1712863225 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3cb001a8a0 con 0x7f3cbc105b10 2024-03-26T18:45:51.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3ca3fff700 1 -- 172.21.15.102:0/1712863225 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cb0019930 con 0x7f3cbc105b10 2024-03-26T18:45:51.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3cc148a700 1 -- 172.21.15.102:0/1712863225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cbc07a290 con 0x7f3cbc105b10 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.764+0000 7f3cc148a700 1 -- 172.21.15.102:0/1712863225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cbc07a6b0 con 0x7f3cbc105b10 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.765+0000 7f3ca3fff700 1 -- 172.21.15.102:0/1712863225 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f3cb0019bf0 con 0x7f3cbc105b10 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.765+0000 7f3cc148a700 1 -- 172.21.15.102:0/1712863225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cbc062380 con 0x7f3cbc105b10 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.765+0000 7f3ca3fff700 1 --2- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f3ca40383b0 0x7f3ca403a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.766+0000 7f3ca3fff700 1 -- 172.21.15.102:0/1712863225 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3cb004c8e0 con 0x7f3cbc105b10 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.766+0000 7f3cba7fc700 1 -- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f3ca40383b0 msgr2=0x7f3ca403a860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/308643739 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.766+0000 7f3cba7fc700 1 --2- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f3ca40383b0 0x7f3ca403a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.770+0000 7f3ca3fff700 1 -- 172.21.15.102:0/1712863225 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3cb002d440 con 0x7f3cbc105b10 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.967+0000 7f3cba7fc700 1 -- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f3ca40383b0 msgr2=0x7f3ca403a860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/308643739 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:50.967+0000 7f3cba7fc700 1 --2- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f3ca40383b0 0x7f3ca403a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.024+0000 7f3cc148a700 1 -- 172.21.15.102:0/1712863225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3cbc1079d0 con 0x7f3cbc105b10 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.024+0000 7f3ca3fff700 1 -- 172.21.15.102:0/1712863225 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f3cb002b340 con 0x7f3cbc105b10 2024-03-26T18:45:51.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.027+0000 7f3cc148a700 1 -- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f3ca40383b0 msgr2=0x7f3ca403a860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T18:45:51.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.027+0000 7f3cc148a700 1 --2- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f3ca40383b0 0x7f3ca403a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:51.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.027+0000 7f3cc148a700 1 -- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 msgr2=0x7f3cbc07bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:51.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.027+0000 7f3cc148a700 1 --2- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc07bc40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00119e0 tx=0x7f3cb0011ac0 comp rx=0 tx=0).stop 2024-03-26T18:45:51.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.027+0000 7f3cc148a700 1 -- 172.21.15.102:0/1712863225 shutdown_connections 2024-03-26T18:45:51.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.027+0000 7f3cc148a700 1 --2- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7f3ca40383b0 0x7f3ca403a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:51.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.027+0000 7f3cc148a700 1 --2- 172.21.15.102:0/1712863225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cbc105b10 0x7f3cbc07bc40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:51.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.027+0000 7f3cc148a700 1 -- 172.21.15.102:0/1712863225 >> 172.21.15.102:0/1712863225 conn(0x7f3cbc101180 msgr2=0x7f3cbc101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:51.599 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.027+0000 7f3cc148a700 1 -- 172.21.15.102:0/1712863225 shutdown_connections 2024-03-26T18:45:51.599 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.027+0000 7f3cc148a700 1 -- 172.21.15.102:0/1712863225 wait complete. 2024-03-26T18:45:51.599 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-03-26T18:45:51.599 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 5... 2024-03-26T18:45:56.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: Active manager daemon smithi102.rsfpnv restarted 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: Activating manager daemon smithi102.rsfpnv 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: mgrmap e6: smithi102.rsfpnv(active, starting, since 0.00331177s) 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi102.rsfpnv", "id": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: Manager daemon smithi102.rsfpnv is now available 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:45:56.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:45:56.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:45:56.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:56 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/trash_purge_schedule"}]: dispatch 2024-03-26T18:45:57.243 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:57 smithi102 ceph-mon[29002]: Found migration_current of "None". Setting to last migration. 2024-03-26T18:45:57.243 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:57 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:45:57.243 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:57 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:45:57.243 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:57 smithi102 ceph-mon[29002]: mgrmap e7: smithi102.rsfpnv(active, since 1.00513s) 2024-03-26T18:45:57.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.981+0000 7fce28f91700 1 Processor -- start 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.981+0000 7fce28f91700 1 -- start start 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.982+0000 7fce28f91700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.982+0000 7fce28f91700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce24105350 con 0x7fce24104a00 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.982+0000 7fce2259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.982+0000 7fce2259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37682/0 (socket says 172.21.15.102:37682) 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.982+0000 7fce2259c700 1 -- 172.21.15.102:0/468359429 learned_addr learned my addr 172.21.15.102:0/468359429 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.982+0000 7fce2259c700 1 -- 172.21.15.102:0/468359429 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce24105490 con 0x7fce24104a00 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.982+0000 7fce2259c700 1 --2- 172.21.15.102:0/468359429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24104e10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fce140098a0 tx=0x7fce14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e76612e21f3a650 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:57.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.982+0000 7fce2159a700 1 -- 172.21.15.102:0/468359429 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce14004180 con 0x7fce24104a00 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.983+0000 7fce2159a700 1 -- 172.21.15.102:0/468359429 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fce140042e0 con 0x7fce24104a00 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.983+0000 7fce2159a700 1 -- 172.21.15.102:0/468359429 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce14011a50 con 0x7fce24104a00 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.983+0000 7fce28f91700 1 -- 172.21.15.102:0/468359429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 msgr2=0x7fce24104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.983+0000 7fce28f91700 1 --2- 172.21.15.102:0/468359429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24104e10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fce140098a0 tx=0x7fce14009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.983+0000 7fce28f91700 1 -- 172.21.15.102:0/468359429 shutdown_connections 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.983+0000 7fce28f91700 1 --2- 172.21.15.102:0/468359429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24104e10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.983+0000 7fce28f91700 1 -- 172.21.15.102:0/468359429 >> 172.21.15.102:0/468359429 conn(0x7fce24100070 msgr2=0x7fce24102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.983+0000 7fce28f91700 1 -- 172.21.15.102:0/468359429 shutdown_connections 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.983+0000 7fce28f91700 1 -- 172.21.15.102:0/468359429 wait complete. 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.984+0000 7fce28f91700 1 Processor -- start 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.984+0000 7fce28f91700 1 -- start start 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.984+0000 7fce28f91700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:57.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.984+0000 7fce28f91700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce24198c80 con 0x7fce24104a00 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.984+0000 7fce2259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.984+0000 7fce2259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37684/0 (socket says 172.21.15.102:37684) 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.984+0000 7fce2259c700 1 -- 172.21.15.102:0/3643363998 learned_addr learned my addr 172.21.15.102:0/3643363998 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.985+0000 7fce2259c700 1 -- 172.21.15.102:0/3643363998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce14009550 con 0x7fce24104a00 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.985+0000 7fce2259c700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24198740 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fce14000c00 tx=0x7fce14011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.985+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce14003b90 con 0x7fce24104a00 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.985+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fce140248b0 con 0x7fce24104a00 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.985+0000 7fce28f91700 1 -- 172.21.15.102:0/3643363998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce24198e80 con 0x7fce24104a00 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.985+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce1401a8e0 con 0x7fce24104a00 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.985+0000 7fce28f91700 1 -- 172.21.15.102:0/3643363998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce241992a0 con 0x7fce24104a00 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.986+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fce1401ab50 con 0x7fce24104a00 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.986+0000 7fce137fe700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 0x7fce0c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.986+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 --> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce0c03b000 con 0x7fce0c038440 2024-03-26T18:45:57.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.986+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fce1404c070 con 0x7fce24104a00 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.986+0000 7fce21d9b700 1 -- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 msgr2=0x7fce0c03a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/308643739 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:51.986+0000 7fce21d9b700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 0x7fce0c03a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:52.187+0000 7fce21d9b700 1 -- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 msgr2=0x7fce0c03a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/308643739 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:52.187+0000 7fce21d9b700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 0x7fce0c03a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:52.588+0000 7fce21d9b700 1 -- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 msgr2=0x7fce0c03a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/308643739 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:52.588+0000 7fce21d9b700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 0x7fce0c03a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:53.390+0000 7fce21d9b700 1 -- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 msgr2=0x7fce0c03a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/308643739 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:53.390+0000 7fce21d9b700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 0x7fce0c03a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:54.990+0000 7fce21d9b700 1 -- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 msgr2=0x7fce0c03a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/308643739 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:54.990+0000 7fce21d9b700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 0x7fce0c03a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:55.926+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mgrmap(e 6) v1 ==== 44851+0+0 (secure 0 0 0) 0x7fce1402d3f0 con 0x7fce24104a00 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:55.926+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 msgr2=0x7fce0c03a8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:55.926+0000 7fce137fe700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 0x7fce0c03a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.927+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 7) v1 ==== 44978+0+0 (secure 0 0 0) 0x7fce1404d990 con 0x7fce24104a00 2024-03-26T18:45:57.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.927+0000 7fce137fe700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fce0c03bd10 0x7fce0c03e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.927+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce0c03b000 con 0x7fce0c03bd10 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.927+0000 7fce21d9b700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fce0c03bd10 0x7fce0c03e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.928+0000 7fce21d9b700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fce0c03bd10 0x7fce0c03e0f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fce18003a10 tx=0x7fce180092b0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.928+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fce0c03b000 con 0x7fce0c03bd10 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.935+0000 7fce28f91700 1 -- 172.21.15.102:0/3643363998 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fce24199a80 con 0x7fce0c03bd10 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce137fe700 1 -- 172.21.15.102:0/3643363998 <== mgr.14116 v2:172.21.15.102:6800/1874485118 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fce24199a80 con 0x7fce0c03bd10 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce28f91700 1 -- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fce0c03bd10 msgr2=0x7fce0c03e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce28f91700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fce0c03bd10 0x7fce0c03e0f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fce18003a10 tx=0x7fce180092b0 comp rx=0 tx=0).stop 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce28f91700 1 -- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 msgr2=0x7fce24198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce28f91700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24198740 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fce14000c00 tx=0x7fce14011ab0 comp rx=0 tx=0).stop 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce28f91700 1 -- 172.21.15.102:0/3643363998 shutdown_connections 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce28f91700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fce0c03bd10 0x7fce0c03e0f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce28f91700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce24104a00 0x7fce24198740 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce28f91700 1 --2- 172.21.15.102:0/3643363998 >> [v2:172.21.15.102:6800/308643739,v1:172.21.15.102:6801/308643739] conn(0x7fce0c038440 0x7fce0c03a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:57.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce28f91700 1 -- 172.21.15.102:0/3643363998 >> 172.21.15.102:0/3643363998 conn(0x7fce24100070 msgr2=0x7fce24100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:57.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce28f91700 1 -- 172.21.15.102:0/3643363998 shutdown_connections 2024-03-26T18:45:57.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:56.936+0000 7fce28f91700 1 -- 172.21.15.102:0/3643363998 wait complete. 2024-03-26T18:45:57.542 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 5 is available 2024-03-26T18:45:57.542 INFO:teuthology.orchestra.run.smithi102.stdout:Setting orchestrator backend to cephadm... 2024-03-26T18:45:58.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:58 smithi102 ceph-mon[29002]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T18:45:58.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:58 smithi102 ceph-mon[29002]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T18:45:58.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:58 smithi102 ceph-mon[29002]: [26/Mar/2024:18:45:57] ENGINE Bus STARTING 2024-03-26T18:45:58.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:58 smithi102 ceph-mon[29002]: [26/Mar/2024:18:45:57] ENGINE Serving on http://172.21.15.102:8765 2024-03-26T18:45:58.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:58 smithi102 ceph-mon[29002]: [26/Mar/2024:18:45:57] ENGINE Serving on https://172.21.15.102:7150 2024-03-26T18:45:58.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:58 smithi102 ceph-mon[29002]: [26/Mar/2024:18:45:57] ENGINE Bus STARTED 2024-03-26T18:45:58.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:58 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:45:58.675 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.897+0000 7f144dbdb700 1 Processor -- start 2024-03-26T18:45:58.675 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.897+0000 7f144dbdb700 1 -- start start 2024-03-26T18:45:58.675 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.897+0000 7f144dbdb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448104e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:58.675 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.897+0000 7f144dbdb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1448105360 con 0x7f1448104a10 2024-03-26T18:45:58.675 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.897+0000 7f14477fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:58.675 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.897+0000 7f14477fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37766/0 (socket says 172.21.15.102:37766) 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.897+0000 7f14477fe700 1 -- 172.21.15.102:0/778108073 learned_addr learned my addr 172.21.15.102:0/778108073 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.897+0000 7f14477fe700 1 -- 172.21.15.102:0/778108073 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14481054a0 con 0x7f1448104a10 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.897+0000 7f14477fe700 1 --2- 172.21.15.102:0/778108073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448104e20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f14380098a0 tx=0x7f1438009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1c494305aad2cd server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.898+0000 7f14467fc700 1 -- 172.21.15.102:0/778108073 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1438004180 con 0x7f1448104a10 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.898+0000 7f14467fc700 1 -- 172.21.15.102:0/778108073 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f14380042e0 con 0x7f1448104a10 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.898+0000 7f14467fc700 1 -- 172.21.15.102:0/778108073 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1438011a50 con 0x7f1448104a10 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.898+0000 7f144dbdb700 1 -- 172.21.15.102:0/778108073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 msgr2=0x7f1448104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.898+0000 7f144dbdb700 1 --2- 172.21.15.102:0/778108073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448104e20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f14380098a0 tx=0x7f1438009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.898+0000 7f144dbdb700 1 -- 172.21.15.102:0/778108073 shutdown_connections 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.898+0000 7f144dbdb700 1 --2- 172.21.15.102:0/778108073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448104e20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.898+0000 7f144dbdb700 1 -- 172.21.15.102:0/778108073 >> 172.21.15.102:0/778108073 conn(0x7f1448100060 msgr2=0x7f1448102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:58.676 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.898+0000 7f144dbdb700 1 -- 172.21.15.102:0/778108073 shutdown_connections 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.898+0000 7f144dbdb700 1 -- 172.21.15.102:0/778108073 wait complete. 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.899+0000 7f144dbdb700 1 Processor -- start 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.899+0000 7f144dbdb700 1 -- start start 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.899+0000 7f144dbdb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448079b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.899+0000 7f144dbdb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144819c560 con 0x7f1448104a10 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.899+0000 7f14477fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448079b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.899+0000 7f14477fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448079b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37778/0 (socket says 172.21.15.102:37778) 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.899+0000 7f14477fe700 1 -- 172.21.15.102:0/1434711289 learned_addr learned my addr 172.21.15.102:0/1434711289 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.899+0000 7f14477fe700 1 -- 172.21.15.102:0/1434711289 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1438009550 con 0x7f1448104a10 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.900+0000 7f14477fe700 1 --2- 172.21.15.102:0/1434711289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448079b40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1438000c00 tx=0x7f1438011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.900+0000 7f1444ff9700 1 -- 172.21.15.102:0/1434711289 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1438003b90 con 0x7f1448104a10 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.900+0000 7f1444ff9700 1 -- 172.21.15.102:0/1434711289 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f14380248b0 con 0x7f1448104a10 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.900+0000 7f144dbdb700 1 -- 172.21.15.102:0/1434711289 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144819c6a0 con 0x7f1448104a10 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.900+0000 7f1444ff9700 1 -- 172.21.15.102:0/1434711289 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f143801a8e0 con 0x7f1448104a10 2024-03-26T18:45:58.677 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.900+0000 7f144dbdb700 1 -- 172.21.15.102:0/1434711289 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144819cac0 con 0x7f1448104a10 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.900+0000 7f1444ff9700 1 -- 172.21.15.102:0/1434711289 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 7) v1 ==== 44978+0+0 (secure 0 0 0) 0x7f143801ab50 con 0x7f1448104a10 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.901+0000 7f1444ff9700 1 --2- 172.21.15.102:0/1434711289 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1430038320 0x7f143003a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.901+0000 7f1444ff9700 1 -- 172.21.15.102:0/1434711289 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f143804c310 con 0x7f1448104a10 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.901+0000 7f144dbdb700 1 -- 172.21.15.102:0/1434711289 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1448062380 con 0x7f1448104a10 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.901+0000 7f1446ffd700 1 --2- 172.21.15.102:0/1434711289 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1430038320 0x7f143003a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.901+0000 7f1446ffd700 1 --2- 172.21.15.102:0/1434711289 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1430038320 0x7f143003a7d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f143c006fd0 tx=0x7f143c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:57.906+0000 7f1444ff9700 1 -- 172.21.15.102:0/1434711289 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f143801ae00 con 0x7f1448104a10 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.107+0000 7f144dbdb700 1 -- 172.21.15.102:0/1434711289 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f1448101560 con 0x7f1430038320 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.109+0000 7f1444ff9700 1 -- 172.21.15.102:0/1434711289 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f143802dc90 con 0x7f1448104a10 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.116+0000 7f1444ff9700 1 -- 172.21.15.102:0/1434711289 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1448101560 con 0x7f1430038320 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.118+0000 7f144dbdb700 1 -- 172.21.15.102:0/1434711289 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1430038320 msgr2=0x7f143003a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.118+0000 7f144dbdb700 1 --2- 172.21.15.102:0/1434711289 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1430038320 0x7f143003a7d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f143c006fd0 tx=0x7f143c006e40 comp rx=0 tx=0).stop 2024-03-26T18:45:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.118+0000 7f144dbdb700 1 -- 172.21.15.102:0/1434711289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 msgr2=0x7f1448079b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:58.679 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.118+0000 7f144dbdb700 1 --2- 172.21.15.102:0/1434711289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448079b40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1438000c00 tx=0x7f1438011ab0 comp rx=0 tx=0).stop 2024-03-26T18:45:58.679 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.118+0000 7f144dbdb700 1 -- 172.21.15.102:0/1434711289 shutdown_connections 2024-03-26T18:45:58.679 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.118+0000 7f144dbdb700 1 --2- 172.21.15.102:0/1434711289 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1430038320 0x7f143003a7d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:58.679 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.118+0000 7f144dbdb700 1 --2- 172.21.15.102:0/1434711289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1448104a10 0x7f1448079b40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:58.679 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.118+0000 7f144dbdb700 1 -- 172.21.15.102:0/1434711289 >> 172.21.15.102:0/1434711289 conn(0x7f1448100060 msgr2=0x7f1448100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:58.679 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.118+0000 7f144dbdb700 1 -- 172.21.15.102:0/1434711289 shutdown_connections 2024-03-26T18:45:58.679 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:58.118+0000 7f144dbdb700 1 -- 172.21.15.102:0/1434711289 wait complete. 2024-03-26T18:45:59.263 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:59 smithi102 ceph-mon[29002]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:45:59.263 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:59 smithi102 ceph-mon[29002]: mgrmap e8: smithi102.rsfpnv(active, since 2s) 2024-03-26T18:45:59.263 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:59 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:45:59.263 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:45:59 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:45:59.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T18:45:59.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.072+0000 7f9fcd673700 1 Processor -- start 2024-03-26T18:45:59.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.072+0000 7f9fcd673700 1 -- start start 2024-03-26T18:45:59.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.072+0000 7f9fcd673700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:59.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.072+0000 7f9fcd673700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc8105350 con 0x7f9fc8104a00 2024-03-26T18:45:59.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.073+0000 7f9fc6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.073+0000 7f9fc6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44236/0 (socket says 172.21.15.102:44236) 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.073+0000 7f9fc6ffd700 1 -- 172.21.15.102:0/299524972 learned_addr learned my addr 172.21.15.102:0/299524972 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.073+0000 7f9fc6ffd700 1 -- 172.21.15.102:0/299524972 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc8105490 con 0x7f9fc8104a00 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.073+0000 7f9fc6ffd700 1 --2- 172.21.15.102:0/299524972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8104e10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9fb80098a0 tx=0x7f9fb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92ebea77316a4a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.073+0000 7f9fc5ffb700 1 -- 172.21.15.102:0/299524972 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fb8004180 con 0x7f9fc8104a00 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.073+0000 7f9fc5ffb700 1 -- 172.21.15.102:0/299524972 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9fb80042e0 con 0x7f9fc8104a00 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.073+0000 7f9fc5ffb700 1 -- 172.21.15.102:0/299524972 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fb8011a50 con 0x7f9fc8104a00 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.074+0000 7f9fcd673700 1 -- 172.21.15.102:0/299524972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 msgr2=0x7f9fc8104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.074+0000 7f9fcd673700 1 --2- 172.21.15.102:0/299524972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8104e10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9fb80098a0 tx=0x7f9fb8009bb0 comp rx=0 tx=0).stop 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.074+0000 7f9fcd673700 1 -- 172.21.15.102:0/299524972 shutdown_connections 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.074+0000 7f9fcd673700 1 --2- 172.21.15.102:0/299524972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8104e10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.074+0000 7f9fcd673700 1 -- 172.21.15.102:0/299524972 >> 172.21.15.102:0/299524972 conn(0x7f9fc8100070 msgr2=0x7f9fc8102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.074+0000 7f9fcd673700 1 -- 172.21.15.102:0/299524972 shutdown_connections 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.074+0000 7f9fcd673700 1 -- 172.21.15.102:0/299524972 wait complete. 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.075+0000 7f9fcd673700 1 Processor -- start 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.075+0000 7f9fcd673700 1 -- start start 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.075+0000 7f9fcd673700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.075+0000 7f9fcd673700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc8199d70 con 0x7f9fc8104a00 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.075+0000 7f9fc6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.075+0000 7f9fc6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44250/0 (socket says 172.21.15.102:44250) 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.075+0000 7f9fc6ffd700 1 -- 172.21.15.102:0/992798145 learned_addr learned my addr 172.21.15.102:0/992798145 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.075+0000 7f9fc6ffd700 1 -- 172.21.15.102:0/992798145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb8009550 con 0x7f9fc8104a00 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.075+0000 7f9fc6ffd700 1 --2- 172.21.15.102:0/992798145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8199830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9fb80119e0 tx=0x7f9fb8011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.076+0000 7f9faffff700 1 -- 172.21.15.102:0/992798145 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fb8003b90 con 0x7f9fc8104a00 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.076+0000 7f9faffff700 1 -- 172.21.15.102:0/992798145 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9fb801a8a0 con 0x7f9fc8104a00 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.076+0000 7f9fcd673700 1 -- 172.21.15.102:0/992798145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc8199f70 con 0x7f9fc8104a00 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.076+0000 7f9faffff700 1 -- 172.21.15.102:0/992798145 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fb8019930 con 0x7f9fc8104a00 2024-03-26T18:45:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.076+0000 7f9fcd673700 1 -- 172.21.15.102:0/992798145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc819a390 con 0x7f9fc8104a00 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.076+0000 7f9faffff700 1 -- 172.21.15.102:0/992798145 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f9fb8019bf0 con 0x7f9fc8104a00 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.076+0000 7f9fcd673700 1 -- 172.21.15.102:0/992798145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc8062380 con 0x7f9fc8104a00 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.077+0000 7f9faffff700 1 --2- 172.21.15.102:0/992798145 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f9fb00383b0 0x7f9fb003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.077+0000 7f9faffff700 1 -- 172.21.15.102:0/992798145 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9fb804c700 con 0x7f9fc8104a00 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.077+0000 7f9fc67fc700 1 --2- 172.21.15.102:0/992798145 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f9fb00383b0 0x7f9fb003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.077+0000 7f9fc67fc700 1 --2- 172.21.15.102:0/992798145 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f9fb00383b0 0x7f9fb003a860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc006fd0 tx=0x7f9fbc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.081+0000 7f9faffff700 1 -- 172.21.15.102:0/992798145 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9fb802d440 con 0x7f9fc8104a00 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.272+0000 7f9fcd673700 1 -- 172.21.15.102:0/992798145 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fc8102210 con 0x7f9fb00383b0 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.272+0000 7f9faffff700 1 -- 172.21.15.102:0/992798145 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f9fc8102210 con 0x7f9fb00383b0 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.275+0000 7f9fcd673700 1 -- 172.21.15.102:0/992798145 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f9fb00383b0 msgr2=0x7f9fb003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.275+0000 7f9fcd673700 1 --2- 172.21.15.102:0/992798145 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f9fb00383b0 0x7f9fb003a860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc006fd0 tx=0x7f9fbc006e40 comp rx=0 tx=0).stop 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.275+0000 7f9fcd673700 1 -- 172.21.15.102:0/992798145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 msgr2=0x7f9fc8199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.275+0000 7f9fcd673700 1 --2- 172.21.15.102:0/992798145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8199830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9fb80119e0 tx=0x7f9fb8011ac0 comp rx=0 tx=0).stop 2024-03-26T18:45:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.275+0000 7f9fcd673700 1 -- 172.21.15.102:0/992798145 shutdown_connections 2024-03-26T18:45:59.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.275+0000 7f9fcd673700 1 --2- 172.21.15.102:0/992798145 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f9fb00383b0 0x7f9fb003a860 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:59.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.275+0000 7f9fcd673700 1 --2- 172.21.15.102:0/992798145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fc8104a00 0x7f9fc8199830 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:45:59.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.275+0000 7f9fcd673700 1 -- 172.21.15.102:0/992798145 >> 172.21.15.102:0/992798145 conn(0x7f9fc8100070 msgr2=0x7f9fc8074070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:45:59.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.275+0000 7f9fcd673700 1 -- 172.21.15.102:0/992798145 shutdown_connections 2024-03-26T18:45:59.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:45:59.275+0000 7f9fcd673700 1 -- 172.21.15.102:0/992798145 wait complete. 2024-03-26T18:45:59.821 INFO:teuthology.orchestra.run.smithi102.stdout:Generating ssh key... 2024-03-26T18:46:00.188 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:00 smithi102 ceph-mon[29002]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:01.680 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.294+0000 7f66884ab700 1 Processor -- start 2024-03-26T18:46:01.680 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.294+0000 7f66884ab700 1 -- start start 2024-03-26T18:46:01.680 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.294+0000 7f66884ab700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f66801053f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.294+0000 7f66884ab700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6680105930 con 0x7f6680104fe0 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.294+0000 7f6686247700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f66801053f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.294+0000 7f6686247700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f66801053f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44258/0 (socket says 172.21.15.102:44258) 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.294+0000 7f6686247700 1 -- 172.21.15.102:0/1155830769 learned_addr learned my addr 172.21.15.102:0/1155830769 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.294+0000 7f6686247700 1 -- 172.21.15.102:0/1155830769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6680105a70 con 0x7f6680104fe0 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.294+0000 7f6686247700 1 --2- 172.21.15.102:0/1155830769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f66801053f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f667c0098a0 tx=0x7f667c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10be9b26607ddf75 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.295+0000 7f6685245700 1 -- 172.21.15.102:0/1155830769 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f667c004180 con 0x7f6680104fe0 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.295+0000 7f6685245700 1 -- 172.21.15.102:0/1155830769 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f667c0042e0 con 0x7f6680104fe0 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.295+0000 7f6685245700 1 -- 172.21.15.102:0/1155830769 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f667c011a50 con 0x7f6680104fe0 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.295+0000 7f66884ab700 1 -- 172.21.15.102:0/1155830769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 msgr2=0x7f66801053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.295+0000 7f66884ab700 1 --2- 172.21.15.102:0/1155830769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f66801053f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f667c0098a0 tx=0x7f667c009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:01.681 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.295+0000 7f66884ab700 1 -- 172.21.15.102:0/1155830769 shutdown_connections 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.295+0000 7f66884ab700 1 --2- 172.21.15.102:0/1155830769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f66801053f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.295+0000 7f66884ab700 1 -- 172.21.15.102:0/1155830769 >> 172.21.15.102:0/1155830769 conn(0x7f66800ff7e0 msgr2=0x7f6680101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.295+0000 7f66884ab700 1 -- 172.21.15.102:0/1155830769 shutdown_connections 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.295+0000 7f66884ab700 1 -- 172.21.15.102:0/1155830769 wait complete. 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.296+0000 7f66884ab700 1 Processor -- start 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.296+0000 7f66884ab700 1 -- start start 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.296+0000 7f66884ab700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f6680198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.296+0000 7f66884ab700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6680198c60 con 0x7f6680104fe0 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.296+0000 7f6686247700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f6680198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.296+0000 7f6686247700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f6680198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44270/0 (socket says 172.21.15.102:44270) 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.296+0000 7f6686247700 1 -- 172.21.15.102:0/122674578 learned_addr learned my addr 172.21.15.102:0/122674578 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.296+0000 7f6686247700 1 -- 172.21.15.102:0/122674578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c009550 con 0x7f6680104fe0 2024-03-26T18:46:01.682 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.297+0000 7f6686247700 1 --2- 172.21.15.102:0/122674578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f6680198720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f667c0119e0 tx=0x7f667c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.297+0000 7f66777fe700 1 -- 172.21.15.102:0/122674578 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f667c003b90 con 0x7f6680104fe0 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.297+0000 7f66777fe700 1 -- 172.21.15.102:0/122674578 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f667c01a8a0 con 0x7f6680104fe0 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.297+0000 7f66777fe700 1 -- 172.21.15.102:0/122674578 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f667c019930 con 0x7f6680104fe0 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.297+0000 7f66884ab700 1 -- 172.21.15.102:0/122674578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6680198e60 con 0x7f6680104fe0 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.297+0000 7f66884ab700 1 -- 172.21.15.102:0/122674578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6680199280 con 0x7f6680104fe0 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.298+0000 7f66777fe700 1 -- 172.21.15.102:0/122674578 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f667c019ba0 con 0x7f6680104fe0 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.298+0000 7f66884ab700 1 -- 172.21.15.102:0/122674578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6680062380 con 0x7f6680104fe0 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.298+0000 7f66777fe700 1 --2- 172.21.15.102:0/122674578 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f666c038400 0x7f666c03a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.298+0000 7f66777fe700 1 -- 172.21.15.102:0/122674578 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f667c04c6a0 con 0x7f6680104fe0 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.298+0000 7f6685a46700 1 --2- 172.21.15.102:0/122674578 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f666c038400 0x7f666c03a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.298+0000 7f6685a46700 1 --2- 172.21.15.102:0/122674578 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f666c038400 0x7f666c03a8b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6670006fd0 tx=0x7f6670006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:01.683 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.303+0000 7f66777fe700 1 -- 172.21.15.102:0/122674578 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f667c01a430 con 0x7f6680104fe0 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:00.491+0000 7f66884ab700 1 -- 172.21.15.102:0/122674578 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6680100c60 con 0x7f666c038400 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:01.155+0000 7f66777fe700 1 -- 172.21.15.102:0/122674578 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6680100c60 con 0x7f666c038400 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:01.157+0000 7f66884ab700 1 -- 172.21.15.102:0/122674578 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f666c038400 msgr2=0x7f666c03a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:01.157+0000 7f66884ab700 1 --2- 172.21.15.102:0/122674578 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f666c038400 0x7f666c03a8b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6670006fd0 tx=0x7f6670006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:01.157+0000 7f66884ab700 1 -- 172.21.15.102:0/122674578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 msgr2=0x7f6680198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:01.157+0000 7f66884ab700 1 --2- 172.21.15.102:0/122674578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f6680198720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f667c0119e0 tx=0x7f667c011ac0 comp rx=0 tx=0).stop 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:01.157+0000 7f66884ab700 1 -- 172.21.15.102:0/122674578 shutdown_connections 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:01.157+0000 7f66884ab700 1 --2- 172.21.15.102:0/122674578 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f666c038400 0x7f666c03a8b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:01.157+0000 7f66884ab700 1 --2- 172.21.15.102:0/122674578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6680104fe0 0x7f6680198720 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:01.157+0000 7f66884ab700 1 -- 172.21.15.102:0/122674578 >> 172.21.15.102:0/122674578 conn(0x7f66800ff7e0 msgr2=0x7f6680100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:01.157+0000 7f66884ab700 1 -- 172.21.15.102:0/122674578 shutdown_connections 2024-03-26T18:46:01.684 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:01.157+0000 7f66884ab700 1 -- 172.21.15.102:0/122674578 wait complete. 2024-03-26T18:46:02.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:02 smithi102 ceph-mon[29002]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:02.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:02 smithi102 ceph-mon[29002]: Generating ssh key... 2024-03-26T18:46:02.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:02 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:02.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:02 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:02.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFWS+C1LDgGRKRV/YScaaQjFO9evkVm3e3Q4iW4Kd25KNaQKLwbLyJZQlMnVQPjxhsLGr0N2TMSPwKL9fj4h4jmfY014cWJECEUOqOmHb4WM8Zkmi23faeWwJrjV7poDk8SZaFBIJEFSM/glrUUa29GtG6sSWUNtD84C5FDbVY/nniLdjjqYRaitGX+kPGFjTN2+VEnuNoS/QgTZg33/5O+eBwzZFPq4ZRZENXMfKz2CYiYU2yhaRhwweTkGF8QJia+/CJJAMNpuKFvSdg2l82ty9ejFbTPqgfcvX2VQkPNapy3oJiLnsGj/RZyFJdbQHIFDb9G1c8ktxxLtj7VYiyO2ICpFnqFv6xxyQG6zEYpLlPTEKUOtSfkgI3q3L4JUAQuo6oMs4nbyM4CaXXpr78nyBVApphEN/ajmik3VBUlE+Tm6Hba4qGK9ukvpuYBBWu3MP85hgcds4IlE+e6jDmxO8qocpFPcD2tdZuOBCCijzUSW0qPG/T+cMLIT7YeCk= ceph-ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:46:02.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.085+0000 7fe4b0fe0700 1 Processor -- start 2024-03-26T18:46:02.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.086+0000 7fe4b0fe0700 1 -- start start 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.086+0000 7fe4b0fe0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac104e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.086+0000 7fe4b0fe0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac105360 con 0x7fe4ac104a10 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.086+0000 7fe4aa59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.086+0000 7fe4aa59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44286/0 (socket says 172.21.15.102:44286) 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.086+0000 7fe4aa59c700 1 -- 172.21.15.102:0/3692626208 learned_addr learned my addr 172.21.15.102:0/3692626208 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.086+0000 7fe4aa59c700 1 -- 172.21.15.102:0/3692626208 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac1054a0 con 0x7fe4ac104a10 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.086+0000 7fe4aa59c700 1 --2- 172.21.15.102:0/3692626208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac104e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00098a0 tx=0x7fe4a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8461cf88517ae8a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.087+0000 7fe4a959a700 1 -- 172.21.15.102:0/3692626208 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4a0004180 con 0x7fe4ac104a10 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.087+0000 7fe4a959a700 1 -- 172.21.15.102:0/3692626208 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a00042e0 con 0x7fe4ac104a10 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.087+0000 7fe4a959a700 1 -- 172.21.15.102:0/3692626208 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4a0011a50 con 0x7fe4ac104a10 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.087+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/3692626208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 msgr2=0x7fe4ac104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.087+0000 7fe4b0fe0700 1 --2- 172.21.15.102:0/3692626208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac104e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00098a0 tx=0x7fe4a0009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:02.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.087+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/3692626208 shutdown_connections 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.087+0000 7fe4b0fe0700 1 --2- 172.21.15.102:0/3692626208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac104e20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.087+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/3692626208 >> 172.21.15.102:0/3692626208 conn(0x7fe4ac100060 msgr2=0x7fe4ac102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.087+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/3692626208 shutdown_connections 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.087+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/3692626208 wait complete. 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.088+0000 7fe4b0fe0700 1 Processor -- start 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.088+0000 7fe4b0fe0700 1 -- start start 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.088+0000 7fe4b0fe0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.088+0000 7fe4b0fe0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac198c70 con 0x7fe4ac104a10 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.088+0000 7fe4aa59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.088+0000 7fe4aa59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44294/0 (socket says 172.21.15.102:44294) 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.088+0000 7fe4aa59c700 1 -- 172.21.15.102:0/1185365081 learned_addr learned my addr 172.21.15.102:0/1185365081 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.089+0000 7fe4aa59c700 1 -- 172.21.15.102:0/1185365081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a0009550 con 0x7fe4ac104a10 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.089+0000 7fe4aa59c700 1 --2- 172.21.15.102:0/1185365081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac198730 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0000c00 tx=0x7fe4a0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:02.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.089+0000 7fe49b7fe700 1 -- 172.21.15.102:0/1185365081 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4a0003b90 con 0x7fe4ac104a10 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.089+0000 7fe49b7fe700 1 -- 172.21.15.102:0/1185365081 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a00248b0 con 0x7fe4ac104a10 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.089+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/1185365081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ac198e70 con 0x7fe4ac104a10 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.089+0000 7fe49b7fe700 1 -- 172.21.15.102:0/1185365081 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4a001a8e0 con 0x7fe4ac104a10 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.089+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/1185365081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ac199290 con 0x7fe4ac104a10 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.090+0000 7fe49b7fe700 1 -- 172.21.15.102:0/1185365081 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fe4a0024a20 con 0x7fe4ac104a10 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.090+0000 7fe49b7fe700 1 --2- 172.21.15.102:0/1185365081 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fe494038440 0x7fe49403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.090+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/1185365081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ac062380 con 0x7fe4ac104a10 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.090+0000 7fe49b7fe700 1 -- 172.21.15.102:0/1185365081 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe4a004c4e0 con 0x7fe4ac104a10 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.090+0000 7fe4a9d9b700 1 --2- 172.21.15.102:0/1185365081 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fe494038440 0x7fe49403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.091+0000 7fe4a9d9b700 1 --2- 172.21.15.102:0/1185365081 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fe494038440 0x7fe49403a8f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe49c006fd0 tx=0x7fe49c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.095+0000 7fe49b7fe700 1 -- 172.21.15.102:0/1185365081 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe4a0024db0 con 0x7fe4ac104a10 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.285+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/1185365081 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4ac101520 con 0x7fe494038440 2024-03-26T18:46:02.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.286+0000 7fe49b7fe700 1 -- 172.21.15.102:0/1185365081 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fe4ac101520 con 0x7fe494038440 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.288+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/1185365081 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fe494038440 msgr2=0x7fe49403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.288+0000 7fe4b0fe0700 1 --2- 172.21.15.102:0/1185365081 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fe494038440 0x7fe49403a8f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe49c006fd0 tx=0x7fe49c006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.288+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/1185365081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 msgr2=0x7fe4ac198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.288+0000 7fe4b0fe0700 1 --2- 172.21.15.102:0/1185365081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac198730 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0000c00 tx=0x7fe4a0011ab0 comp rx=0 tx=0).stop 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.288+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/1185365081 shutdown_connections 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.288+0000 7fe4b0fe0700 1 --2- 172.21.15.102:0/1185365081 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fe494038440 0x7fe49403a8f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.288+0000 7fe4b0fe0700 1 --2- 172.21.15.102:0/1185365081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac104a10 0x7fe4ac198730 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.288+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/1185365081 >> 172.21.15.102:0/1185365081 conn(0x7fe4ac100060 msgr2=0x7fe4ac100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.288+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/1185365081 shutdown_connections 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:02.288+0000 7fe4b0fe0700 1 -- 172.21.15.102:0/1185365081 wait complete. 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T18:46:02.815 INFO:teuthology.orchestra.run.smithi102.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T18:46:02.816 INFO:teuthology.orchestra.run.smithi102.stdout:Adding host smithi102... 2024-03-26T18:46:03.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:03 smithi102 ceph-mon[29002]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:04.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:04 smithi102 ceph-mon[29002]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "addr": "172.21.15.102", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:04.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:04 smithi102 ceph-mon[29002]: Deploying cephadm binary to smithi102 2024-03-26T18:46:06.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Added host 'smithi102' with addr '172.21.15.102' 2024-03-26T18:46:06.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.209+0000 7f6afad63700 1 Processor -- start 2024-03-26T18:46:06.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.209+0000 7f6afad63700 1 -- start start 2024-03-26T18:46:06.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.210+0000 7f6afad63700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.210+0000 7f6afad63700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af4106460 con 0x7f6af4105b10 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.210+0000 7f6af8aff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.210+0000 7f6af8aff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44308/0 (socket says 172.21.15.102:44308) 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.210+0000 7f6af8aff700 1 -- 172.21.15.102:0/2021205471 learned_addr learned my addr 172.21.15.102:0/2021205471 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.210+0000 7f6af8aff700 1 -- 172.21.15.102:0/2021205471 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af41065a0 con 0x7f6af4105b10 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.210+0000 7f6af8aff700 1 --2- 172.21.15.102:0/2021205471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4105f20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80098a0 tx=0x7f6ae8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c2469a324bf0cf server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.210+0000 7f6af37fe700 1 -- 172.21.15.102:0/2021205471 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ae8004180 con 0x7f6af4105b10 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.210+0000 7f6af37fe700 1 -- 172.21.15.102:0/2021205471 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6ae80042e0 con 0x7f6af4105b10 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.210+0000 7f6af37fe700 1 -- 172.21.15.102:0/2021205471 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ae8011a50 con 0x7f6af4105b10 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.211+0000 7f6afad63700 1 -- 172.21.15.102:0/2021205471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 msgr2=0x7f6af4105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.211+0000 7f6afad63700 1 --2- 172.21.15.102:0/2021205471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4105f20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80098a0 tx=0x7f6ae8009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.211+0000 7f6afad63700 1 -- 172.21.15.102:0/2021205471 shutdown_connections 2024-03-26T18:46:06.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.211+0000 7f6afad63700 1 --2- 172.21.15.102:0/2021205471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4105f20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:06.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.211+0000 7f6afad63700 1 -- 172.21.15.102:0/2021205471 >> 172.21.15.102:0/2021205471 conn(0x7f6af4101180 msgr2=0x7f6af4103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:06.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.211+0000 7f6afad63700 1 -- 172.21.15.102:0/2021205471 shutdown_connections 2024-03-26T18:46:06.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.211+0000 7f6afad63700 1 -- 172.21.15.102:0/2021205471 wait complete. 2024-03-26T18:46:06.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.211+0000 7f6afad63700 1 Processor -- start 2024-03-26T18:46:06.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.211+0000 7f6afad63700 1 -- start start 2024-03-26T18:46:06.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6afad63700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:06.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6afad63700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af4199d90 con 0x7f6af4105b10 2024-03-26T18:46:06.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6af8aff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:06.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6af8aff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44310/0 (socket says 172.21.15.102:44310) 2024-03-26T18:46:06.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6af8aff700 1 -- 172.21.15.102:0/2064800890 learned_addr learned my addr 172.21.15.102:0/2064800890 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:06.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6af8aff700 1 -- 172.21.15.102:0/2064800890 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae8009550 con 0x7f6af4105b10 2024-03-26T18:46:06.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6af8aff700 1 --2- 172.21.15.102:0/2064800890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4199850 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80119e0 tx=0x7f6ae8011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:06.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6af1ffb700 1 -- 172.21.15.102:0/2064800890 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ae8003b90 con 0x7f6af4105b10 2024-03-26T18:46:06.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6af1ffb700 1 -- 172.21.15.102:0/2064800890 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6ae801a8a0 con 0x7f6af4105b10 2024-03-26T18:46:06.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6af1ffb700 1 -- 172.21.15.102:0/2064800890 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ae8019930 con 0x7f6af4105b10 2024-03-26T18:46:06.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6afad63700 1 -- 172.21.15.102:0/2064800890 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af4199f90 con 0x7f6af4105b10 2024-03-26T18:46:06.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.212+0000 7f6afad63700 1 -- 172.21.15.102:0/2064800890 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af419a3b0 con 0x7f6af4105b10 2024-03-26T18:46:06.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.213+0000 7f6af1ffb700 1 -- 172.21.15.102:0/2064800890 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f6ae801aa10 con 0x7f6af4105b10 2024-03-26T18:46:06.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.213+0000 7f6af1ffb700 1 --2- 172.21.15.102:0/2064800890 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f6ae4038400 0x7f6ae403a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:06.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.213+0000 7f6af1ffb700 1 -- 172.21.15.102:0/2064800890 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6ae804c720 con 0x7f6af4105b10 2024-03-26T18:46:06.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.213+0000 7f6afad63700 1 -- 172.21.15.102:0/2064800890 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af4062380 con 0x7f6af4105b10 2024-03-26T18:46:06.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.214+0000 7f6af3fff700 1 --2- 172.21.15.102:0/2064800890 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f6ae4038400 0x7f6ae403a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:06.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.214+0000 7f6af3fff700 1 --2- 172.21.15.102:0/2064800890 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f6ae4038400 0x7f6ae403a8b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0006fd0 tx=0x7f6ae0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:06.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.218+0000 7f6af1ffb700 1 -- 172.21.15.102:0/2064800890 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6ae8018530 con 0x7f6af4105b10 2024-03-26T18:46:06.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:03.409+0000 7f6afad63700 1 -- 172.21.15.102:0/2064800890 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi102", "addr": "172.21.15.102", "target": ["mon-mgr", ""]}) v1 -- 0x7f6af4102620 con 0x7f6ae4038400 2024-03-26T18:46:06.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:05.872+0000 7f6af1ffb700 1 -- 172.21.15.102:0/2064800890 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f6af4102620 con 0x7f6ae4038400 2024-03-26T18:46:06.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:05.874+0000 7f6afad63700 1 -- 172.21.15.102:0/2064800890 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f6ae4038400 msgr2=0x7f6ae403a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:06.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:05.874+0000 7f6afad63700 1 --2- 172.21.15.102:0/2064800890 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f6ae4038400 0x7f6ae403a8b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0006fd0 tx=0x7f6ae0006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:06.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:05.874+0000 7f6afad63700 1 -- 172.21.15.102:0/2064800890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 msgr2=0x7f6af4199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:06.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:05.874+0000 7f6afad63700 1 --2- 172.21.15.102:0/2064800890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4199850 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80119e0 tx=0x7f6ae8011ac0 comp rx=0 tx=0).stop 2024-03-26T18:46:06.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:05.875+0000 7f6afad63700 1 -- 172.21.15.102:0/2064800890 shutdown_connections 2024-03-26T18:46:06.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:05.875+0000 7f6afad63700 1 --2- 172.21.15.102:0/2064800890 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f6ae4038400 0x7f6ae403a8b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:06.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:05.875+0000 7f6afad63700 1 --2- 172.21.15.102:0/2064800890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6af4105b10 0x7f6af4199850 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:06.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:05.875+0000 7f6afad63700 1 -- 172.21.15.102:0/2064800890 >> 172.21.15.102:0/2064800890 conn(0x7f6af4101180 msgr2=0x7f6af4101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:06.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:05.875+0000 7f6afad63700 1 -- 172.21.15.102:0/2064800890 shutdown_connections 2024-03-26T18:46:06.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:05.875+0000 7f6afad63700 1 -- 172.21.15.102:0/2064800890 wait complete. 2024-03-26T18:46:06.408 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying mon service with default placement... 2024-03-26T18:46:07.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:06 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:07.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:06 smithi102 ceph-mon[29002]: Added host smithi102 2024-03-26T18:46:07.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:06 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:46:08.351 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:08 smithi102 ceph-mon[29002]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:08.351 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:08 smithi102 ceph-mon[29002]: Saving service mon spec with placement count:5 2024-03-26T18:46:08.351 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:08 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:08.351 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:08 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:08.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T18:46:08.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.792+0000 7f4146b44700 1 Processor -- start 2024-03-26T18:46:08.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.792+0000 7f4146b44700 1 -- start start 2024-03-26T18:46:08.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.792+0000 7f4146b44700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f4140104e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:08.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.792+0000 7f4146b44700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4140105360 con 0x7f4140104a10 2024-03-26T18:46:08.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.792+0000 7f41448e0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f4140104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:08.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.792+0000 7f41448e0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f4140104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44318/0 (socket says 172.21.15.102:44318) 2024-03-26T18:46:08.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.792+0000 7f41448e0700 1 -- 172.21.15.102:0/4192435543 learned_addr learned my addr 172.21.15.102:0/4192435543 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:08.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.792+0000 7f41448e0700 1 -- 172.21.15.102:0/4192435543 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41401054a0 con 0x7f4140104a10 2024-03-26T18:46:08.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.792+0000 7f41448e0700 1 --2- 172.21.15.102:0/4192435543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f4140104e20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f413800d180 tx=0x7f413800d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50c6a81101982c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.793+0000 7f413f7fe700 1 -- 172.21.15.102:0/4192435543 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4138010070 con 0x7f4140104a10 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.793+0000 7f413f7fe700 1 -- 172.21.15.102:0/4192435543 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f413800dcf0 con 0x7f4140104a10 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.793+0000 7f413f7fe700 1 -- 172.21.15.102:0/4192435543 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4138009950 con 0x7f4140104a10 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.793+0000 7f4146b44700 1 -- 172.21.15.102:0/4192435543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 msgr2=0x7f4140104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.793+0000 7f4146b44700 1 --2- 172.21.15.102:0/4192435543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f4140104e20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f413800d180 tx=0x7f413800d490 comp rx=0 tx=0).stop 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.793+0000 7f4146b44700 1 -- 172.21.15.102:0/4192435543 shutdown_connections 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.793+0000 7f4146b44700 1 --2- 172.21.15.102:0/4192435543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f4140104e20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.793+0000 7f4146b44700 1 -- 172.21.15.102:0/4192435543 >> 172.21.15.102:0/4192435543 conn(0x7f4140100060 msgr2=0x7f4140102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.794+0000 7f4146b44700 1 -- 172.21.15.102:0/4192435543 shutdown_connections 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.794+0000 7f4146b44700 1 -- 172.21.15.102:0/4192435543 wait complete. 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.794+0000 7f4146b44700 1 Processor -- start 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.794+0000 7f4146b44700 1 -- start start 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.794+0000 7f4146b44700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f41401975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:08.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.794+0000 7f4146b44700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4140197af0 con 0x7f4140104a10 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.794+0000 7f41448e0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f41401975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.794+0000 7f41448e0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f41401975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44322/0 (socket says 172.21.15.102:44322) 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.794+0000 7f41448e0700 1 -- 172.21.15.102:0/489632270 learned_addr learned my addr 172.21.15.102:0/489632270 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.795+0000 7f41448e0700 1 -- 172.21.15.102:0/489632270 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413800bd40 con 0x7f4140104a10 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.795+0000 7f41448e0700 1 --2- 172.21.15.102:0/489632270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f41401975b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41380072a0 tx=0x7f413800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.795+0000 7f413dffb700 1 -- 172.21.15.102:0/489632270 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4138010050 con 0x7f4140104a10 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.795+0000 7f413dffb700 1 -- 172.21.15.102:0/489632270 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4138004630 con 0x7f4140104a10 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.795+0000 7f4146b44700 1 -- 172.21.15.102:0/489632270 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4140197cf0 con 0x7f4140104a10 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.795+0000 7f413dffb700 1 -- 172.21.15.102:0/489632270 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f413801fbd0 con 0x7f4140104a10 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.795+0000 7f4146b44700 1 -- 172.21.15.102:0/489632270 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4140198110 con 0x7f4140104a10 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.796+0000 7f413dffb700 1 -- 172.21.15.102:0/489632270 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f413801b070 con 0x7f4140104a10 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.796+0000 7f4146b44700 1 -- 172.21.15.102:0/489632270 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4140191340 con 0x7f4140104a10 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.796+0000 7f413dffb700 1 --2- 172.21.15.102:0/489632270 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f4128038400 0x7f412803a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.796+0000 7f413dffb700 1 -- 172.21.15.102:0/489632270 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f413804e710 con 0x7f4140104a10 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.796+0000 7f413ffff700 1 --2- 172.21.15.102:0/489632270 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f4128038400 0x7f412803a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.797+0000 7f413ffff700 1 --2- 172.21.15.102:0/489632270 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f4128038400 0x7f412803a8b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4134006fd0 tx=0x7f4134006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:06.801+0000 7f413dffb700 1 -- 172.21.15.102:0/489632270 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f413801c790 con 0x7f4140104a10 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.004+0000 7f4146b44700 1 -- 172.21.15.102:0/489632270 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f4140061190 con 0x7f4128038400 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.012+0000 7f413dffb700 1 -- 172.21.15.102:0/489632270 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4140061190 con 0x7f4128038400 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.014+0000 7f4146b44700 1 -- 172.21.15.102:0/489632270 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f4128038400 msgr2=0x7f412803a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.014+0000 7f4146b44700 1 --2- 172.21.15.102:0/489632270 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f4128038400 0x7f412803a8b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4134006fd0 tx=0x7f4134006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.014+0000 7f4146b44700 1 -- 172.21.15.102:0/489632270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 msgr2=0x7f41401975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.014+0000 7f4146b44700 1 --2- 172.21.15.102:0/489632270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f41401975b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41380072a0 tx=0x7f413800b3b0 comp rx=0 tx=0).stop 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.014+0000 7f4146b44700 1 -- 172.21.15.102:0/489632270 shutdown_connections 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.014+0000 7f4146b44700 1 --2- 172.21.15.102:0/489632270 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f4128038400 0x7f412803a8b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.014+0000 7f4146b44700 1 --2- 172.21.15.102:0/489632270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4140104a10 0x7f41401975b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:08.363 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.014+0000 7f4146b44700 1 -- 172.21.15.102:0/489632270 >> 172.21.15.102:0/489632270 conn(0x7f4140100060 msgr2=0x7f414018e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:08.363 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.014+0000 7f4146b44700 1 -- 172.21.15.102:0/489632270 shutdown_connections 2024-03-26T18:46:08.363 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:07.014+0000 7f4146b44700 1 -- 172.21.15.102:0/489632270 wait complete. 2024-03-26T18:46:08.363 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying mgr service with default placement... 2024-03-26T18:46:09.820 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:09 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:09.821 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:09 smithi102 ceph-mon[29002]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:09.821 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:09 smithi102 ceph-mon[29002]: Saving service mgr spec with placement count:2 2024-03-26T18:46:09.821 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:09 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:10.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T18:46:10.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.814+0000 7f1abc406700 1 Processor -- start 2024-03-26T18:46:10.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.814+0000 7f1abc406700 1 -- start start 2024-03-26T18:46:10.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.814+0000 7f1abc406700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:10.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.814+0000 7f1abc406700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab4105350 con 0x7f1ab4104a00 2024-03-26T18:46:10.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.814+0000 7f1aba1a2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:10.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.814+0000 7f1aba1a2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49690/0 (socket says 172.21.15.102:49690) 2024-03-26T18:46:10.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.814+0000 7f1aba1a2700 1 -- 172.21.15.102:0/1437088529 learned_addr learned my addr 172.21.15.102:0/1437088529 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:10.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.815+0000 7f1aba1a2700 1 -- 172.21.15.102:0/1437088529 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4105490 con 0x7f1ab4104a00 2024-03-26T18:46:10.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.815+0000 7f1aba1a2700 1 --2- 172.21.15.102:0/1437088529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4104e10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00098a0 tx=0x7f1ab0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12488f686a5fd94 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:10.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.815+0000 7f1ab91a0700 1 -- 172.21.15.102:0/1437088529 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ab0004180 con 0x7f1ab4104a00 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.815+0000 7f1ab91a0700 1 -- 172.21.15.102:0/1437088529 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1ab00042e0 con 0x7f1ab4104a00 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.815+0000 7f1ab91a0700 1 -- 172.21.15.102:0/1437088529 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ab0011af0 con 0x7f1ab4104a00 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.815+0000 7f1abc406700 1 -- 172.21.15.102:0/1437088529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 msgr2=0x7f1ab4104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.816+0000 7f1abc406700 1 --2- 172.21.15.102:0/1437088529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4104e10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00098a0 tx=0x7f1ab0009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.816+0000 7f1abc406700 1 -- 172.21.15.102:0/1437088529 shutdown_connections 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.816+0000 7f1abc406700 1 --2- 172.21.15.102:0/1437088529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4104e10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.816+0000 7f1abc406700 1 -- 172.21.15.102:0/1437088529 >> 172.21.15.102:0/1437088529 conn(0x7f1ab4100070 msgr2=0x7f1ab4102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.816+0000 7f1abc406700 1 -- 172.21.15.102:0/1437088529 shutdown_connections 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.816+0000 7f1abc406700 1 -- 172.21.15.102:0/1437088529 wait complete. 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.816+0000 7f1abc406700 1 Processor -- start 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.816+0000 7f1abc406700 1 -- start start 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.816+0000 7f1abc406700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.816+0000 7f1abc406700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab4199db0 con 0x7f1ab4104a00 2024-03-26T18:46:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.816+0000 7f1aba1a2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.817+0000 7f1aba1a2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49692/0 (socket says 172.21.15.102:49692) 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.817+0000 7f1aba1a2700 1 -- 172.21.15.102:0/1430595217 learned_addr learned my addr 172.21.15.102:0/1430595217 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.817+0000 7f1aba1a2700 1 -- 172.21.15.102:0/1430595217 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab0009550 con 0x7f1ab4104a00 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.817+0000 7f1aba1a2700 1 --2- 172.21.15.102:0/1430595217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4199870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0006ba0 tx=0x7f1ab0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.817+0000 7f1aab7fe700 1 -- 172.21.15.102:0/1430595217 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ab0003b00 con 0x7f1ab4104a00 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.817+0000 7f1aab7fe700 1 -- 172.21.15.102:0/1430595217 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1ab001a8e0 con 0x7f1ab4104a00 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.817+0000 7f1abc406700 1 -- 172.21.15.102:0/1430595217 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab4199fb0 con 0x7f1ab4104a00 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.817+0000 7f1aab7fe700 1 -- 172.21.15.102:0/1430595217 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ab00199c0 con 0x7f1ab4104a00 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.817+0000 7f1abc406700 1 -- 172.21.15.102:0/1430595217 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab419a3d0 con 0x7f1ab4104a00 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.818+0000 7f1aab7fe700 1 -- 172.21.15.102:0/1430595217 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f1ab0019c80 con 0x7f1ab4104a00 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.818+0000 7f1aab7fe700 1 --2- 172.21.15.102:0/1430595217 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1aa00383a0 0x7f1aa003a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.818+0000 7f1aab7fe700 1 -- 172.21.15.102:0/1430595217 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1ab004c630 con 0x7f1ab4104a00 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.818+0000 7f1abc406700 1 -- 172.21.15.102:0/1430595217 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab4062380 con 0x7f1ab4104a00 2024-03-26T18:46:10.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.818+0000 7f1ab99a1700 1 --2- 172.21.15.102:0/1430595217 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1aa00383a0 0x7f1aa003a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:10.089 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.819+0000 7f1ab99a1700 1 --2- 172.21.15.102:0/1430595217 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1aa00383a0 0x7f1aa003a850 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4006fd0 tx=0x7f1aa4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:10.089 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:08.827+0000 7f1aab7fe700 1 -- 172.21.15.102:0/1430595217 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1ab0047020 con 0x7f1ab4104a00 2024-03-26T18:46:10.089 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.041+0000 7f1abc406700 1 -- 172.21.15.102:0/1430595217 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ab4102210 con 0x7f1aa00383a0 2024-03-26T18:46:10.089 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.045+0000 7f1aab7fe700 1 -- 172.21.15.102:0/1430595217 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1ab4102210 con 0x7f1aa00383a0 2024-03-26T18:46:10.089 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.047+0000 7f1abc406700 1 -- 172.21.15.102:0/1430595217 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1aa00383a0 msgr2=0x7f1aa003a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:10.089 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.047+0000 7f1abc406700 1 --2- 172.21.15.102:0/1430595217 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1aa00383a0 0x7f1aa003a850 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4006fd0 tx=0x7f1aa4006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:10.089 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.047+0000 7f1abc406700 1 -- 172.21.15.102:0/1430595217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 msgr2=0x7f1ab4199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:10.089 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.047+0000 7f1abc406700 1 --2- 172.21.15.102:0/1430595217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4199870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0006ba0 tx=0x7f1ab0011b50 comp rx=0 tx=0).stop 2024-03-26T18:46:10.089 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.047+0000 7f1abc406700 1 -- 172.21.15.102:0/1430595217 shutdown_connections 2024-03-26T18:46:10.089 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.047+0000 7f1abc406700 1 --2- 172.21.15.102:0/1430595217 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f1aa00383a0 0x7f1aa003a850 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:10.090 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.047+0000 7f1abc406700 1 --2- 172.21.15.102:0/1430595217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ab4104a00 0x7f1ab4199870 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:10.090 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.047+0000 7f1abc406700 1 -- 172.21.15.102:0/1430595217 >> 172.21.15.102:0/1430595217 conn(0x7f1ab4100070 msgr2=0x7f1ab4074070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:10.090 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.047+0000 7f1abc406700 1 -- 172.21.15.102:0/1430595217 shutdown_connections 2024-03-26T18:46:10.090 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:09.048+0000 7f1abc406700 1 -- 172.21.15.102:0/1430595217 wait complete. 2024-03-26T18:46:10.090 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying crash service with default placement... 2024-03-26T18:46:11.117 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:10 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:11.117 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:10 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:11.117 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:10 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:11.218 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-26T18:46:11.218 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.478+0000 7f61bd8fe700 1 Processor -- start 2024-03-26T18:46:11.218 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.478+0000 7f61bd8fe700 1 -- start start 2024-03-26T18:46:11.218 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.478+0000 7f61bd8fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:11.218 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.478+0000 7f61bd8fe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b8106460 con 0x7f61b8105b10 2024-03-26T18:46:11.218 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.478+0000 7f61b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:11.218 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.478+0000 7f61b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49704/0 (socket says 172.21.15.102:49704) 2024-03-26T18:46:11.218 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.478+0000 7f61b6ffd700 1 -- 172.21.15.102:0/4068018087 learned_addr learned my addr 172.21.15.102:0/4068018087 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:11.218 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.479+0000 7f61b6ffd700 1 -- 172.21.15.102:0/4068018087 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b81065a0 con 0x7f61b8105b10 2024-03-26T18:46:11.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.479+0000 7f61b6ffd700 1 --2- 172.21.15.102:0/4068018087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8105f20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f61ac0098a0 tx=0x7f61ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc6bc85adee32a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:11.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.479+0000 7f61b5ffb700 1 -- 172.21.15.102:0/4068018087 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61ac004180 con 0x7f61b8105b10 2024-03-26T18:46:11.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.479+0000 7f61b5ffb700 1 -- 172.21.15.102:0/4068018087 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f61ac0042e0 con 0x7f61b8105b10 2024-03-26T18:46:11.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.479+0000 7f61b5ffb700 1 -- 172.21.15.102:0/4068018087 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61ac011af0 con 0x7f61b8105b10 2024-03-26T18:46:11.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.479+0000 7f61bd8fe700 1 -- 172.21.15.102:0/4068018087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 msgr2=0x7f61b8105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:11.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.479+0000 7f61bd8fe700 1 --2- 172.21.15.102:0/4068018087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8105f20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f61ac0098a0 tx=0x7f61ac009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:11.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.480+0000 7f61bd8fe700 1 -- 172.21.15.102:0/4068018087 shutdown_connections 2024-03-26T18:46:11.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.480+0000 7f61bd8fe700 1 --2- 172.21.15.102:0/4068018087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8105f20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:11.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.480+0000 7f61bd8fe700 1 -- 172.21.15.102:0/4068018087 >> 172.21.15.102:0/4068018087 conn(0x7f61b8101180 msgr2=0x7f61b8103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:11.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.480+0000 7f61bd8fe700 1 -- 172.21.15.102:0/4068018087 shutdown_connections 2024-03-26T18:46:11.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.480+0000 7f61bd8fe700 1 -- 172.21.15.102:0/4068018087 wait complete. 2024-03-26T18:46:11.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.480+0000 7f61bd8fe700 1 Processor -- start 2024-03-26T18:46:11.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.480+0000 7f61bd8fe700 1 -- start start 2024-03-26T18:46:11.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.480+0000 7f61bd8fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:11.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.481+0000 7f61bd8fe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b8199dd0 con 0x7f61b8105b10 2024-03-26T18:46:11.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.481+0000 7f61b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:11.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.481+0000 7f61b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49706/0 (socket says 172.21.15.102:49706) 2024-03-26T18:46:11.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.481+0000 7f61b6ffd700 1 -- 172.21.15.102:0/72199378 learned_addr learned my addr 172.21.15.102:0/72199378 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:11.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.481+0000 7f61b6ffd700 1 -- 172.21.15.102:0/72199378 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac009550 con 0x7f61b8105b10 2024-03-26T18:46:11.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.481+0000 7f61b6ffd700 1 --2- 172.21.15.102:0/72199378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8199890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f61ac006ba0 tx=0x7f61ac011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:11.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.481+0000 7f61bc8fc700 1 -- 172.21.15.102:0/72199378 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61ac003b00 con 0x7f61b8105b10 2024-03-26T18:46:11.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.481+0000 7f61bd8fe700 1 -- 172.21.15.102:0/72199378 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b8199fd0 con 0x7f61b8105b10 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.481+0000 7f61bc8fc700 1 -- 172.21.15.102:0/72199378 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f61ac01a8e0 con 0x7f61b8105b10 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.481+0000 7f61bc8fc700 1 -- 172.21.15.102:0/72199378 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61ac0199c0 con 0x7f61b8105b10 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.481+0000 7f61bd8fe700 1 -- 172.21.15.102:0/72199378 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b819a3f0 con 0x7f61b8105b10 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.482+0000 7f61bc8fc700 1 -- 172.21.15.102:0/72199378 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f61ac019c80 con 0x7f61b8105b10 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.482+0000 7f61bc8fc700 1 --2- 172.21.15.102:0/72199378 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f61a00383a0 0x7f61a003a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.482+0000 7f61bc8fc700 1 -- 172.21.15.102:0/72199378 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f61ac04d640 con 0x7f61b8105b10 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.482+0000 7f61b67fc700 1 --2- 172.21.15.102:0/72199378 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f61a00383a0 0x7f61a003a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.483+0000 7f61bd8fe700 1 -- 172.21.15.102:0/72199378 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b8062380 con 0x7f61b8105b10 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.483+0000 7f61b67fc700 1 --2- 172.21.15.102:0/72199378 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f61a00383a0 0x7f61a003a850 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f61a8006fd0 tx=0x7f61a8006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.487+0000 7f61bc8fc700 1 -- 172.21.15.102:0/72199378 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f61ac02b930 con 0x7f61b8105b10 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.699+0000 7f61bd8fe700 1 -- 172.21.15.102:0/72199378 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f61b8062380 con 0x7f61a00383a0 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.703+0000 7f61bc8fc700 1 -- 172.21.15.102:0/72199378 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f61b8062380 con 0x7f61a00383a0 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.705+0000 7f61bd8fe700 1 -- 172.21.15.102:0/72199378 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f61a00383a0 msgr2=0x7f61a003a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:11.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.705+0000 7f61bd8fe700 1 --2- 172.21.15.102:0/72199378 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f61a00383a0 0x7f61a003a850 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f61a8006fd0 tx=0x7f61a8006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:11.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.705+0000 7f61bd8fe700 1 -- 172.21.15.102:0/72199378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 msgr2=0x7f61b8199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:11.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.705+0000 7f61bd8fe700 1 --2- 172.21.15.102:0/72199378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8199890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f61ac006ba0 tx=0x7f61ac011b50 comp rx=0 tx=0).stop 2024-03-26T18:46:11.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.706+0000 7f61bd8fe700 1 -- 172.21.15.102:0/72199378 shutdown_connections 2024-03-26T18:46:11.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.706+0000 7f61bd8fe700 1 --2- 172.21.15.102:0/72199378 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f61a00383a0 0x7f61a003a850 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:11.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.706+0000 7f61bd8fe700 1 --2- 172.21.15.102:0/72199378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b8105b10 0x7f61b8199890 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:11.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.706+0000 7f61bd8fe700 1 -- 172.21.15.102:0/72199378 >> 172.21.15.102:0/72199378 conn(0x7f61b8101180 msgr2=0x7f61b8101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:11.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.706+0000 7f61bd8fe700 1 -- 172.21.15.102:0/72199378 shutdown_connections 2024-03-26T18:46:11.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:10.706+0000 7f61bd8fe700 1 -- 172.21.15.102:0/72199378 wait complete. 2024-03-26T18:46:11.222 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying ceph-exporter service with default placement... 2024-03-26T18:46:12.117 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:11 smithi102 ceph-mon[29002]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:12.117 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:11 smithi102 ceph-mon[29002]: Saving service crash spec with placement * 2024-03-26T18:46:12.402 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-03-26T18:46:12.402 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.581+0000 7f1325958700 1 Processor -- start 2024-03-26T18:46:12.402 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.581+0000 7f1325958700 1 -- start start 2024-03-26T18:46:12.402 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.581+0000 7f1325958700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f1320105320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.581+0000 7f1325958700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1320105860 con 0x7f1320104f10 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.581+0000 7f131effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f1320105320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.581+0000 7f131effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f1320105320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49716/0 (socket says 172.21.15.102:49716) 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.581+0000 7f131effd700 1 -- 172.21.15.102:0/2325227028 learned_addr learned my addr 172.21.15.102:0/2325227028 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.582+0000 7f131effd700 1 -- 172.21.15.102:0/2325227028 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13201059a0 con 0x7f1320104f10 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.582+0000 7f131effd700 1 --2- 172.21.15.102:0/2325227028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f1320105320 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f13100098a0 tx=0x7f1310009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f679ae018695f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.582+0000 7f131dffb700 1 -- 172.21.15.102:0/2325227028 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1310004180 con 0x7f1320104f10 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.582+0000 7f131dffb700 1 -- 172.21.15.102:0/2325227028 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f13100042e0 con 0x7f1320104f10 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.582+0000 7f131dffb700 1 -- 172.21.15.102:0/2325227028 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1310011af0 con 0x7f1320104f10 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.583+0000 7f1325958700 1 -- 172.21.15.102:0/2325227028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 msgr2=0x7f1320105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.583+0000 7f1325958700 1 --2- 172.21.15.102:0/2325227028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f1320105320 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f13100098a0 tx=0x7f1310009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.583+0000 7f1325958700 1 -- 172.21.15.102:0/2325227028 shutdown_connections 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.583+0000 7f1325958700 1 --2- 172.21.15.102:0/2325227028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f1320105320 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.583+0000 7f1325958700 1 -- 172.21.15.102:0/2325227028 >> 172.21.15.102:0/2325227028 conn(0x7f13200ff7e0 msgr2=0x7f1320101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.583+0000 7f1325958700 1 -- 172.21.15.102:0/2325227028 shutdown_connections 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.583+0000 7f1325958700 1 -- 172.21.15.102:0/2325227028 wait complete. 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.583+0000 7f1325958700 1 Processor -- start 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.584+0000 7f1325958700 1 -- start start 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.584+0000 7f1325958700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f132018fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.584+0000 7f1325958700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1320190120 con 0x7f1320104f10 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.584+0000 7f131effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f132018fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.584+0000 7f131effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f132018fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49730/0 (socket says 172.21.15.102:49730) 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.584+0000 7f131effd700 1 -- 172.21.15.102:0/3728023620 learned_addr learned my addr 172.21.15.102:0/3728023620 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.584+0000 7f131effd700 1 -- 172.21.15.102:0/3728023620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1310009550 con 0x7f1320104f10 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.584+0000 7f131effd700 1 --2- 172.21.15.102:0/3728023620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f132018fbe0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1310006ba0 tx=0x7f1310011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.585+0000 7f1324956700 1 -- 172.21.15.102:0/3728023620 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1310003b00 con 0x7f1320104f10 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.585+0000 7f1324956700 1 -- 172.21.15.102:0/3728023620 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f131001a8e0 con 0x7f1320104f10 2024-03-26T18:46:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.585+0000 7f1325958700 1 -- 172.21.15.102:0/3728023620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f132018e230 con 0x7f1320104f10 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.585+0000 7f1324956700 1 -- 172.21.15.102:0/3728023620 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1310019990 con 0x7f1320104f10 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.585+0000 7f1325958700 1 -- 172.21.15.102:0/3728023620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f132018e650 con 0x7f1320104f10 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.586+0000 7f1324956700 1 -- 172.21.15.102:0/3728023620 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f1310019c00 con 0x7f1320104f10 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.586+0000 7f1325958700 1 -- 172.21.15.102:0/3728023620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1320062380 con 0x7f1320104f10 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.586+0000 7f1324956700 1 --2- 172.21.15.102:0/3728023620 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f13080383f0 0x7f130803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.586+0000 7f1324956700 1 -- 172.21.15.102:0/3728023620 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f131004c5a0 con 0x7f1320104f10 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.586+0000 7f131e7fc700 1 --2- 172.21.15.102:0/3728023620 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f13080383f0 0x7f130803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.587+0000 7f131e7fc700 1 --2- 172.21.15.102:0/3728023620 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f13080383f0 0x7f130803a8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1314006fd0 tx=0x7f1314006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.593+0000 7f1324956700 1 -- 172.21.15.102:0/3728023620 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f131002d440 con 0x7f1320104f10 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.804+0000 7f1325958700 1 -- 172.21.15.102:0/3728023620 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f13201019b0 con 0x7f13080383f0 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.810+0000 7f1324956700 1 -- 172.21.15.102:0/3728023620 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f13201019b0 con 0x7f13080383f0 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.812+0000 7f1325958700 1 -- 172.21.15.102:0/3728023620 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f13080383f0 msgr2=0x7f130803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.812+0000 7f1325958700 1 --2- 172.21.15.102:0/3728023620 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f13080383f0 0x7f130803a8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1314006fd0 tx=0x7f1314006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:12.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.812+0000 7f1325958700 1 -- 172.21.15.102:0/3728023620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 msgr2=0x7f132018fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:12.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.812+0000 7f1325958700 1 --2- 172.21.15.102:0/3728023620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f132018fbe0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1310006ba0 tx=0x7f1310011b50 comp rx=0 tx=0).stop 2024-03-26T18:46:12.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.812+0000 7f1325958700 1 -- 172.21.15.102:0/3728023620 shutdown_connections 2024-03-26T18:46:12.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.812+0000 7f1325958700 1 --2- 172.21.15.102:0/3728023620 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f13080383f0 0x7f130803a8a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:12.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.812+0000 7f1325958700 1 --2- 172.21.15.102:0/3728023620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1320104f10 0x7f132018fbe0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:12.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.812+0000 7f1325958700 1 -- 172.21.15.102:0/3728023620 >> 172.21.15.102:0/3728023620 conn(0x7f13200ff7e0 msgr2=0x7f13201043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:12.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.813+0000 7f1325958700 1 -- 172.21.15.102:0/3728023620 shutdown_connections 2024-03-26T18:46:12.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:11.813+0000 7f1325958700 1 -- 172.21.15.102:0/3728023620 wait complete. 2024-03-26T18:46:12.406 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying prometheus service with default placement... 2024-03-26T18:46:13.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:12 smithi102 ceph-mon[29002]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:13.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:12 smithi102 ceph-mon[29002]: Saving service ceph-exporter spec with placement * 2024-03-26T18:46:13.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:12 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:13.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-26T18:46:13.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.809+0000 7fcdfe63b700 1 Processor -- start 2024-03-26T18:46:13.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.809+0000 7fcdfe63b700 1 -- start start 2024-03-26T18:46:13.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.809+0000 7fcdfe63b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:13.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.809+0000 7fcdfe63b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf8106460 con 0x7fcdf8105b10 2024-03-26T18:46:13.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.809+0000 7fcdf7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:13.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.809+0000 7fcdf7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49742/0 (socket says 172.21.15.102:49742) 2024-03-26T18:46:13.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.809+0000 7fcdf7fff700 1 -- 172.21.15.102:0/804888384 learned_addr learned my addr 172.21.15.102:0/804888384 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:13.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.809+0000 7fcdf7fff700 1 -- 172.21.15.102:0/804888384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf81065a0 con 0x7fcdf8105b10 2024-03-26T18:46:13.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.809+0000 7fcdf7fff700 1 --2- 172.21.15.102:0/804888384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8105f20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcde80098a0 tx=0x7fcde8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a647441c88926de server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.811+0000 7fcdf6ffd700 1 -- 172.21.15.102:0/804888384 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde8004180 con 0x7fcdf8105b10 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.811+0000 7fcdf6ffd700 1 -- 172.21.15.102:0/804888384 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcde80042e0 con 0x7fcdf8105b10 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.811+0000 7fcdf6ffd700 1 -- 172.21.15.102:0/804888384 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde8011af0 con 0x7fcdf8105b10 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.812+0000 7fcdfe63b700 1 -- 172.21.15.102:0/804888384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 msgr2=0x7fcdf8105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.812+0000 7fcdfe63b700 1 --2- 172.21.15.102:0/804888384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8105f20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcde80098a0 tx=0x7fcde8009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.812+0000 7fcdfe63b700 1 -- 172.21.15.102:0/804888384 shutdown_connections 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.812+0000 7fcdfe63b700 1 --2- 172.21.15.102:0/804888384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8105f20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.812+0000 7fcdfe63b700 1 -- 172.21.15.102:0/804888384 >> 172.21.15.102:0/804888384 conn(0x7fcdf8101180 msgr2=0x7fcdf8103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.812+0000 7fcdfe63b700 1 -- 172.21.15.102:0/804888384 shutdown_connections 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.812+0000 7fcdfe63b700 1 -- 172.21.15.102:0/804888384 wait complete. 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.812+0000 7fcdfe63b700 1 Processor -- start 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.813+0000 7fcdfe63b700 1 -- start start 2024-03-26T18:46:13.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.813+0000 7fcdfe63b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.813+0000 7fcdfe63b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf8199d90 con 0x7fcdf8105b10 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.813+0000 7fcdf7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.813+0000 7fcdf7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49746/0 (socket says 172.21.15.102:49746) 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.813+0000 7fcdf7fff700 1 -- 172.21.15.102:0/311138431 learned_addr learned my addr 172.21.15.102:0/311138431 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.813+0000 7fcdf7fff700 1 -- 172.21.15.102:0/311138431 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde8009550 con 0x7fcdf8105b10 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.813+0000 7fcdf7fff700 1 --2- 172.21.15.102:0/311138431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8199850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcde8006ba0 tx=0x7fcde8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.813+0000 7fcdf57fa700 1 -- 172.21.15.102:0/311138431 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde8003b00 con 0x7fcdf8105b10 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.814+0000 7fcdf57fa700 1 -- 172.21.15.102:0/311138431 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcde801a8e0 con 0x7fcdf8105b10 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.814+0000 7fcdfe63b700 1 -- 172.21.15.102:0/311138431 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf8199f90 con 0x7fcdf8105b10 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.814+0000 7fcdf57fa700 1 -- 172.21.15.102:0/311138431 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde80199c0 con 0x7fcdf8105b10 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.814+0000 7fcdfe63b700 1 -- 172.21.15.102:0/311138431 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf819a3b0 con 0x7fcdf8105b10 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.814+0000 7fcdf57fa700 1 -- 172.21.15.102:0/311138431 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fcde8019c80 con 0x7fcdf8105b10 2024-03-26T18:46:13.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.814+0000 7fcdf57fa700 1 --2- 172.21.15.102:0/311138431 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fcde00383a0 0x7fcde003a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.815+0000 7fcdf57fa700 1 -- 172.21.15.102:0/311138431 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcde804c640 con 0x7fcdf8105b10 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.815+0000 7fcdfe63b700 1 -- 172.21.15.102:0/311138431 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf8062380 con 0x7fcdf8105b10 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.815+0000 7fcdf77fe700 1 --2- 172.21.15.102:0/311138431 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fcde00383a0 0x7fcde003a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.815+0000 7fcdf77fe700 1 --2- 172.21.15.102:0/311138431 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fcde00383a0 0x7fcde003a850 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcdec006fd0 tx=0x7fcdec006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:12.819+0000 7fcdf57fa700 1 -- 172.21.15.102:0/311138431 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcde802b930 con 0x7fcdf8105b10 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.025+0000 7fcdfe63b700 1 -- 172.21.15.102:0/311138431 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdf8062380 con 0x7fcde00383a0 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.032+0000 7fcdf57fa700 1 -- 172.21.15.102:0/311138431 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fcdf8062380 con 0x7fcde00383a0 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.034+0000 7fcdfe63b700 1 -- 172.21.15.102:0/311138431 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fcde00383a0 msgr2=0x7fcde003a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.034+0000 7fcdfe63b700 1 --2- 172.21.15.102:0/311138431 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fcde00383a0 0x7fcde003a850 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcdec006fd0 tx=0x7fcdec006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.034+0000 7fcdfe63b700 1 -- 172.21.15.102:0/311138431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 msgr2=0x7fcdf8199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.034+0000 7fcdfe63b700 1 --2- 172.21.15.102:0/311138431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8199850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcde8006ba0 tx=0x7fcde8011b50 comp rx=0 tx=0).stop 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.034+0000 7fcdfe63b700 1 -- 172.21.15.102:0/311138431 shutdown_connections 2024-03-26T18:46:13.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.034+0000 7fcdfe63b700 1 --2- 172.21.15.102:0/311138431 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fcde00383a0 0x7fcde003a850 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:13.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.034+0000 7fcdfe63b700 1 --2- 172.21.15.102:0/311138431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf8105b10 0x7fcdf8199850 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:13.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.034+0000 7fcdfe63b700 1 -- 172.21.15.102:0/311138431 >> 172.21.15.102:0/311138431 conn(0x7fcdf8101180 msgr2=0x7fcdf8101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:13.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.034+0000 7fcdfe63b700 1 -- 172.21.15.102:0/311138431 shutdown_connections 2024-03-26T18:46:13.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:13.034+0000 7fcdfe63b700 1 -- 172.21.15.102:0/311138431 wait complete. 2024-03-26T18:46:13.776 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying grafana service with default placement... 2024-03-26T18:46:14.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:14 smithi102 ceph-mon[29002]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:14.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:14 smithi102 ceph-mon[29002]: Saving service prometheus spec with placement count:1 2024-03-26T18:46:14.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:14 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:15.236 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-26T18:46:15.236 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.467+0000 7fb80ad56700 1 Processor -- start 2024-03-26T18:46:15.236 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.467+0000 7fb80ad56700 1 -- start start 2024-03-26T18:46:15.236 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.467+0000 7fb80ad56700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:15.236 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.467+0000 7fb80ad56700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb804106460 con 0x7fb804105b10 2024-03-26T18:46:15.236 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.467+0000 7fb808af2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:15.236 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.468+0000 7fb808af2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49762/0 (socket says 172.21.15.102:49762) 2024-03-26T18:46:15.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.468+0000 7fb808af2700 1 -- 172.21.15.102:0/1361723222 learned_addr learned my addr 172.21.15.102:0/1361723222 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:15.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.468+0000 7fb808af2700 1 -- 172.21.15.102:0/1361723222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8041065a0 con 0x7fb804105b10 2024-03-26T18:46:15.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.468+0000 7fb808af2700 1 --2- 172.21.15.102:0/1361723222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804105f20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb8000098a0 tx=0x7fb800009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf25656163eaa9b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:15.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.468+0000 7fb7fb7fe700 1 -- 172.21.15.102:0/1361723222 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb800004180 con 0x7fb804105b10 2024-03-26T18:46:15.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.468+0000 7fb7fb7fe700 1 -- 172.21.15.102:0/1361723222 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb8000042e0 con 0x7fb804105b10 2024-03-26T18:46:15.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.468+0000 7fb7fb7fe700 1 -- 172.21.15.102:0/1361723222 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb800011af0 con 0x7fb804105b10 2024-03-26T18:46:15.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.469+0000 7fb80ad56700 1 -- 172.21.15.102:0/1361723222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 msgr2=0x7fb804105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:15.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.469+0000 7fb80ad56700 1 --2- 172.21.15.102:0/1361723222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804105f20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb8000098a0 tx=0x7fb800009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:15.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.469+0000 7fb80ad56700 1 -- 172.21.15.102:0/1361723222 shutdown_connections 2024-03-26T18:46:15.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.469+0000 7fb80ad56700 1 --2- 172.21.15.102:0/1361723222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804105f20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:15.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.469+0000 7fb80ad56700 1 -- 172.21.15.102:0/1361723222 >> 172.21.15.102:0/1361723222 conn(0x7fb804101180 msgr2=0x7fb804103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:15.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.469+0000 7fb80ad56700 1 -- 172.21.15.102:0/1361723222 shutdown_connections 2024-03-26T18:46:15.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.469+0000 7fb80ad56700 1 -- 172.21.15.102:0/1361723222 wait complete. 2024-03-26T18:46:15.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.469+0000 7fb80ad56700 1 Processor -- start 2024-03-26T18:46:15.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.469+0000 7fb80ad56700 1 -- start start 2024-03-26T18:46:15.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.469+0000 7fb80ad56700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:15.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.470+0000 7fb80ad56700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb804199d90 con 0x7fb804105b10 2024-03-26T18:46:15.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.470+0000 7fb808af2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:15.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.470+0000 7fb808af2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49764/0 (socket says 172.21.15.102:49764) 2024-03-26T18:46:15.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.470+0000 7fb808af2700 1 -- 172.21.15.102:0/1262957077 learned_addr learned my addr 172.21.15.102:0/1262957077 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:15.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.470+0000 7fb808af2700 1 -- 172.21.15.102:0/1262957077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb800009550 con 0x7fb804105b10 2024-03-26T18:46:15.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.470+0000 7fb808af2700 1 --2- 172.21.15.102:0/1262957077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804199850 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb800006ba0 tx=0x7fb800011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:15.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.470+0000 7fb7f9ffb700 1 -- 172.21.15.102:0/1262957077 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb800003b00 con 0x7fb804105b10 2024-03-26T18:46:15.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.470+0000 7fb7f9ffb700 1 -- 172.21.15.102:0/1262957077 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb80001a8e0 con 0x7fb804105b10 2024-03-26T18:46:15.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.470+0000 7fb80ad56700 1 -- 172.21.15.102:0/1262957077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb804199f90 con 0x7fb804105b10 2024-03-26T18:46:15.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.470+0000 7fb7f9ffb700 1 -- 172.21.15.102:0/1262957077 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8000199c0 con 0x7fb804105b10 2024-03-26T18:46:15.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.470+0000 7fb80ad56700 1 -- 172.21.15.102:0/1262957077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80419a3b0 con 0x7fb804105b10 2024-03-26T18:46:15.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.471+0000 7fb7f9ffb700 1 -- 172.21.15.102:0/1262957077 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fb800019c80 con 0x7fb804105b10 2024-03-26T18:46:15.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.471+0000 7fb80ad56700 1 -- 172.21.15.102:0/1262957077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb804062380 con 0x7fb804105b10 2024-03-26T18:46:15.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.471+0000 7fb7f9ffb700 1 --2- 172.21.15.102:0/1262957077 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fb7ec0383a0 0x7fb7ec03a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.471+0000 7fb7f9ffb700 1 -- 172.21.15.102:0/1262957077 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb80004c880 con 0x7fb804105b10 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.471+0000 7fb7fbfff700 1 --2- 172.21.15.102:0/1262957077 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fb7ec0383a0 0x7fb7ec03a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.472+0000 7fb7fbfff700 1 --2- 172.21.15.102:0/1262957077 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fb7ec0383a0 0x7fb7ec03a850 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4006fd0 tx=0x7fb7f4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.476+0000 7fb7f9ffb700 1 -- 172.21.15.102:0/1262957077 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb80002b700 con 0x7fb804105b10 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.674+0000 7fb80ad56700 1 -- 172.21.15.102:0/1262957077 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fb804102610 con 0x7fb7ec0383a0 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.679+0000 7fb7f9ffb700 1 -- 172.21.15.102:0/1262957077 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fb804102610 con 0x7fb7ec0383a0 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.681+0000 7fb80ad56700 1 -- 172.21.15.102:0/1262957077 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fb7ec0383a0 msgr2=0x7fb7ec03a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.681+0000 7fb80ad56700 1 --2- 172.21.15.102:0/1262957077 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fb7ec0383a0 0x7fb7ec03a850 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4006fd0 tx=0x7fb7f4006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.681+0000 7fb80ad56700 1 -- 172.21.15.102:0/1262957077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 msgr2=0x7fb804199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.681+0000 7fb80ad56700 1 --2- 172.21.15.102:0/1262957077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804199850 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb800006ba0 tx=0x7fb800011b50 comp rx=0 tx=0).stop 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.681+0000 7fb80ad56700 1 -- 172.21.15.102:0/1262957077 shutdown_connections 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.681+0000 7fb80ad56700 1 --2- 172.21.15.102:0/1262957077 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fb7ec0383a0 0x7fb7ec03a850 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.681+0000 7fb80ad56700 1 --2- 172.21.15.102:0/1262957077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb804105b10 0x7fb804199850 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.681+0000 7fb80ad56700 1 -- 172.21.15.102:0/1262957077 >> 172.21.15.102:0/1262957077 conn(0x7fb804101180 msgr2=0x7fb804101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:15.240 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.681+0000 7fb80ad56700 1 -- 172.21.15.102:0/1262957077 shutdown_connections 2024-03-26T18:46:15.241 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:14.681+0000 7fb80ad56700 1 -- 172.21.15.102:0/1262957077 wait complete. 2024-03-26T18:46:15.241 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying node-exporter service with default placement... 2024-03-26T18:46:15.864 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:15 smithi102 ceph-mon[29002]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:15.864 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:15 smithi102 ceph-mon[29002]: Saving service grafana spec with placement count:1 2024-03-26T18:46:15.864 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:15 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:16.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-26T18:46:16.456 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.664+0000 7f80c100f700 1 Processor -- start 2024-03-26T18:46:16.456 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.664+0000 7f80c100f700 1 -- start start 2024-03-26T18:46:16.456 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.664+0000 7f80c100f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:16.456 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.664+0000 7f80c100f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc106460 con 0x7f80bc105b10 2024-03-26T18:46:16.456 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.664+0000 7f80bad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:16.457 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.664+0000 7f80bad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49776/0 (socket says 172.21.15.102:49776) 2024-03-26T18:46:16.457 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.664+0000 7f80bad9d700 1 -- 172.21.15.102:0/548870459 learned_addr learned my addr 172.21.15.102:0/548870459 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:16.457 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.664+0000 7f80bad9d700 1 -- 172.21.15.102:0/548870459 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc1065a0 con 0x7f80bc105b10 2024-03-26T18:46:16.457 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.665+0000 7f80bad9d700 1 --2- 172.21.15.102:0/548870459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc105f20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0098a0 tx=0x7f80ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ced7cd483bf2a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:16.457 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.665+0000 7f80b9d9b700 1 -- 172.21.15.102:0/548870459 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80ac004180 con 0x7f80bc105b10 2024-03-26T18:46:16.457 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.665+0000 7f80b9d9b700 1 -- 172.21.15.102:0/548870459 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f80ac0042e0 con 0x7f80bc105b10 2024-03-26T18:46:16.458 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.665+0000 7f80b9d9b700 1 -- 172.21.15.102:0/548870459 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80ac011af0 con 0x7f80bc105b10 2024-03-26T18:46:16.458 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.665+0000 7f80c100f700 1 -- 172.21.15.102:0/548870459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 msgr2=0x7f80bc105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:16.458 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.665+0000 7f80c100f700 1 --2- 172.21.15.102:0/548870459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc105f20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0098a0 tx=0x7f80ac009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:16.458 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.665+0000 7f80c100f700 1 -- 172.21.15.102:0/548870459 shutdown_connections 2024-03-26T18:46:16.458 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.665+0000 7f80c100f700 1 --2- 172.21.15.102:0/548870459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc105f20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:16.459 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.665+0000 7f80c100f700 1 -- 172.21.15.102:0/548870459 >> 172.21.15.102:0/548870459 conn(0x7f80bc101180 msgr2=0x7f80bc103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:16.459 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.665+0000 7f80c100f700 1 -- 172.21.15.102:0/548870459 shutdown_connections 2024-03-26T18:46:16.459 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.665+0000 7f80c100f700 1 -- 172.21.15.102:0/548870459 wait complete. 2024-03-26T18:46:16.459 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.666+0000 7f80c100f700 1 Processor -- start 2024-03-26T18:46:16.459 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.666+0000 7f80c100f700 1 -- start start 2024-03-26T18:46:16.459 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.666+0000 7f80c100f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:16.460 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.666+0000 7f80c100f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc199d90 con 0x7f80bc105b10 2024-03-26T18:46:16.460 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.667+0000 7f80bad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:16.460 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.667+0000 7f80bad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49780/0 (socket says 172.21.15.102:49780) 2024-03-26T18:46:16.460 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.667+0000 7f80bad9d700 1 -- 172.21.15.102:0/1084092989 learned_addr learned my addr 172.21.15.102:0/1084092989 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:16.460 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.667+0000 7f80bad9d700 1 -- 172.21.15.102:0/1084092989 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac009550 con 0x7f80bc105b10 2024-03-26T18:46:16.461 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.667+0000 7f80bad9d700 1 --2- 172.21.15.102:0/1084092989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc199850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f80ac006ba0 tx=0x7f80ac011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:16.461 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.667+0000 7f80a3fff700 1 -- 172.21.15.102:0/1084092989 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80ac003b00 con 0x7f80bc105b10 2024-03-26T18:46:16.461 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.667+0000 7f80a3fff700 1 -- 172.21.15.102:0/1084092989 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f80ac01a8e0 con 0x7f80bc105b10 2024-03-26T18:46:16.461 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.667+0000 7f80c100f700 1 -- 172.21.15.102:0/1084092989 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80bc199f90 con 0x7f80bc105b10 2024-03-26T18:46:16.461 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.667+0000 7f80a3fff700 1 -- 172.21.15.102:0/1084092989 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80ac0199c0 con 0x7f80bc105b10 2024-03-26T18:46:16.461 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.667+0000 7f80c100f700 1 -- 172.21.15.102:0/1084092989 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80bc19a3b0 con 0x7f80bc105b10 2024-03-26T18:46:16.462 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.668+0000 7f80a3fff700 1 -- 172.21.15.102:0/1084092989 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f80ac019c30 con 0x7f80bc105b10 2024-03-26T18:46:16.462 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.668+0000 7f80a3fff700 1 --2- 172.21.15.102:0/1084092989 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f80a40383f0 0x7f80a403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:16.462 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.668+0000 7f80a3fff700 1 -- 172.21.15.102:0/1084092989 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f80ac04d610 con 0x7f80bc105b10 2024-03-26T18:46:16.462 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.668+0000 7f80ba59c700 1 --2- 172.21.15.102:0/1084092989 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f80a40383f0 0x7f80a403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:16.462 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.668+0000 7f80c100f700 1 -- 172.21.15.102:0/1084092989 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80bc062380 con 0x7f80bc105b10 2024-03-26T18:46:16.463 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.669+0000 7f80ba59c700 1 --2- 172.21.15.102:0/1084092989 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f80a40383f0 0x7f80a403a8a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f80b0006fd0 tx=0x7f80b0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:16.463 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.673+0000 7f80a3fff700 1 -- 172.21.15.102:0/1084092989 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f80ac02d440 con 0x7f80bc105b10 2024-03-26T18:46:16.463 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.873+0000 7f80c100f700 1 -- 172.21.15.102:0/1084092989 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f80bc102610 con 0x7f80a40383f0 2024-03-26T18:46:16.463 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.878+0000 7f80a3fff700 1 -- 172.21.15.102:0/1084092989 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f80bc102610 con 0x7f80a40383f0 2024-03-26T18:46:16.463 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.880+0000 7f80c100f700 1 -- 172.21.15.102:0/1084092989 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f80a40383f0 msgr2=0x7f80a403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:16.463 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.880+0000 7f80c100f700 1 --2- 172.21.15.102:0/1084092989 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f80a40383f0 0x7f80a403a8a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f80b0006fd0 tx=0x7f80b0006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:16.464 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.880+0000 7f80c100f700 1 -- 172.21.15.102:0/1084092989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 msgr2=0x7f80bc199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:16.464 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.880+0000 7f80c100f700 1 --2- 172.21.15.102:0/1084092989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc199850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f80ac006ba0 tx=0x7f80ac011b50 comp rx=0 tx=0).stop 2024-03-26T18:46:16.464 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.880+0000 7f80c100f700 1 -- 172.21.15.102:0/1084092989 shutdown_connections 2024-03-26T18:46:16.464 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.880+0000 7f80c100f700 1 --2- 172.21.15.102:0/1084092989 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f80a40383f0 0x7f80a403a8a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:16.464 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.880+0000 7f80c100f700 1 --2- 172.21.15.102:0/1084092989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80bc105b10 0x7f80bc199850 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:16.465 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.880+0000 7f80c100f700 1 -- 172.21.15.102:0/1084092989 >> 172.21.15.102:0/1084092989 conn(0x7f80bc101180 msgr2=0x7f80bc101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:16.465 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.880+0000 7f80c100f700 1 -- 172.21.15.102:0/1084092989 shutdown_connections 2024-03-26T18:46:16.465 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:15.880+0000 7f80c100f700 1 -- 172.21.15.102:0/1084092989 wait complete. 2024-03-26T18:46:16.465 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying alertmanager service with default placement... 2024-03-26T18:46:17.070 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:16 smithi102 ceph-mon[29002]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:17.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:16 smithi102 ceph-mon[29002]: Saving service node-exporter spec with placement * 2024-03-26T18:46:17.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:16 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:17.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:16 smithi102 ceph-mon[29002]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:46:17.574 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.853+0000 7f857acda700 1 Processor -- start 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.853+0000 7f857acda700 1 -- start start 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.853+0000 7f857acda700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f8574105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.853+0000 7f857acda700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8574106460 con 0x7f8574105b10 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.854+0000 7f8578a76700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f8574105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.854+0000 7f8578a76700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f8574105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49794/0 (socket says 172.21.15.102:49794) 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.854+0000 7f8578a76700 1 -- 172.21.15.102:0/96950193 learned_addr learned my addr 172.21.15.102:0/96950193 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.854+0000 7f8578a76700 1 -- 172.21.15.102:0/96950193 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85741065a0 con 0x7f8574105b10 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.854+0000 7f8578a76700 1 --2- 172.21.15.102:0/96950193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f8574105f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8570009e80 tx=0x7f857000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf078c56b502395a server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.854+0000 7f856b7fe700 1 -- 172.21.15.102:0/96950193 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f857000e070 con 0x7f8574105b10 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.854+0000 7f856b7fe700 1 -- 172.21.15.102:0/96950193 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f857000b8c0 con 0x7f8574105b10 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.854+0000 7f856b7fe700 1 -- 172.21.15.102:0/96950193 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85700044d0 con 0x7f8574105b10 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.855+0000 7f857acda700 1 -- 172.21.15.102:0/96950193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 msgr2=0x7f8574105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.855+0000 7f857acda700 1 --2- 172.21.15.102:0/96950193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f8574105f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8570009e80 tx=0x7f857000b330 comp rx=0 tx=0).stop 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.855+0000 7f857acda700 1 -- 172.21.15.102:0/96950193 shutdown_connections 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.855+0000 7f857acda700 1 --2- 172.21.15.102:0/96950193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f8574105f20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.855+0000 7f857acda700 1 -- 172.21.15.102:0/96950193 >> 172.21.15.102:0/96950193 conn(0x7f8574101180 msgr2=0x7f8574103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.855+0000 7f857acda700 1 -- 172.21.15.102:0/96950193 shutdown_connections 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.855+0000 7f857acda700 1 -- 172.21.15.102:0/96950193 wait complete. 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.855+0000 7f857acda700 1 Processor -- start 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.855+0000 7f857acda700 1 -- start start 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f857acda700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f857407bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f857acda700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857407c180 con 0x7f8574105b10 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f8578a76700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f857407bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f8578a76700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f857407bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49810/0 (socket says 172.21.15.102:49810) 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f8578a76700 1 -- 172.21.15.102:0/660190184 learned_addr learned my addr 172.21.15.102:0/660190184 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f8578a76700 1 -- 172.21.15.102:0/660190184 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8570009b30 con 0x7f8574105b10 2024-03-26T18:46:17.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f8578a76700 1 --2- 172.21.15.102:0/660190184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f857407bc40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f85700072a0 tx=0x7f85700049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f8569ffb700 1 -- 172.21.15.102:0/660190184 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f857000e040 con 0x7f8574105b10 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f8569ffb700 1 -- 172.21.15.102:0/660190184 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f857000bd80 con 0x7f8574105b10 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f8569ffb700 1 -- 172.21.15.102:0/660190184 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85700126c0 con 0x7f8574105b10 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f857acda700 1 -- 172.21.15.102:0/660190184 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857407a290 con 0x7f8574105b10 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.856+0000 7f857acda700 1 -- 172.21.15.102:0/660190184 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857407a710 con 0x7f8574105b10 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.857+0000 7f8569ffb700 1 -- 172.21.15.102:0/660190184 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f8570019040 con 0x7f8574105b10 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.857+0000 7f857acda700 1 -- 172.21.15.102:0/660190184 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8574062380 con 0x7f8574105b10 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.858+0000 7f8569ffb700 1 --2- 172.21.15.102:0/660190184 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f855c038400 0x7f855c03a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.858+0000 7f8569ffb700 1 -- 172.21.15.102:0/660190184 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f857004bdc0 con 0x7f8574105b10 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.858+0000 7f856bfff700 1 --2- 172.21.15.102:0/660190184 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f855c038400 0x7f855c03a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.858+0000 7f856bfff700 1 --2- 172.21.15.102:0/660190184 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f855c038400 0x7f855c03a8b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8564006fd0 tx=0x7f8564006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:16.862+0000 7f8569ffb700 1 -- 172.21.15.102:0/660190184 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f857001a570 con 0x7f8574105b10 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.061+0000 7f857acda700 1 -- 172.21.15.102:0/660190184 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f8574102650 con 0x7f855c038400 2024-03-26T18:46:17.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.066+0000 7f8569ffb700 1 -- 172.21.15.102:0/660190184 <== mgr.14116 v2:172.21.15.102:6800/1874485118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f8574102650 con 0x7f855c038400 2024-03-26T18:46:17.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.068+0000 7f857acda700 1 -- 172.21.15.102:0/660190184 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f855c038400 msgr2=0x7f855c03a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:17.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.068+0000 7f857acda700 1 --2- 172.21.15.102:0/660190184 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f855c038400 0x7f855c03a8b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8564006fd0 tx=0x7f8564006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:17.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.068+0000 7f857acda700 1 -- 172.21.15.102:0/660190184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 msgr2=0x7f857407bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:17.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.068+0000 7f857acda700 1 --2- 172.21.15.102:0/660190184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f857407bc40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f85700072a0 tx=0x7f85700049f0 comp rx=0 tx=0).stop 2024-03-26T18:46:17.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.069+0000 7f857acda700 1 -- 172.21.15.102:0/660190184 shutdown_connections 2024-03-26T18:46:17.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.069+0000 7f857acda700 1 --2- 172.21.15.102:0/660190184 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f855c038400 0x7f855c03a8b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:17.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.069+0000 7f857acda700 1 --2- 172.21.15.102:0/660190184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8574105b10 0x7f857407bc40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:17.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.069+0000 7f857acda700 1 -- 172.21.15.102:0/660190184 >> 172.21.15.102:0/660190184 conn(0x7f8574101180 msgr2=0x7f8574101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:17.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.069+0000 7f857acda700 1 -- 172.21.15.102:0/660190184 shutdown_connections 2024-03-26T18:46:17.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:17.069+0000 7f857acda700 1 -- 172.21.15.102:0/660190184 wait complete. 2024-03-26T18:46:18.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:18 smithi102 ceph-mon[29002]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:18.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:18 smithi102 ceph-mon[29002]: Saving service alertmanager spec with placement count:1 2024-03-26T18:46:18.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:18 smithi102 ceph-mon[29002]: from='mgr.14116 172.21.15.102:0/1064126794' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:18.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.006+0000 7f0d9a220700 1 Processor -- start 2024-03-26T18:46:18.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.006+0000 7f0d9a220700 1 -- start start 2024-03-26T18:46:18.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.006+0000 7f0d9a220700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d941064d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:18.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.006+0000 7f0d9a220700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d94106a10 con 0x7f0d941060c0 2024-03-26T18:46:18.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.007+0000 7f0d937fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d941064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.007+0000 7f0d937fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d941064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49824/0 (socket says 172.21.15.102:49824) 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.007+0000 7f0d937fe700 1 -- 172.21.15.102:0/2815218822 learned_addr learned my addr 172.21.15.102:0/2815218822 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.007+0000 7f0d937fe700 1 -- 172.21.15.102:0/2815218822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d94106b50 con 0x7f0d941060c0 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.007+0000 7f0d937fe700 1 --2- 172.21.15.102:0/2815218822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d941064d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d84009e80 tx=0x7f0d8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77dd0f66c4f5429 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.007+0000 7f0d92ffd700 1 -- 172.21.15.102:0/2815218822 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d8400e070 con 0x7f0d941060c0 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.007+0000 7f0d92ffd700 1 -- 172.21.15.102:0/2815218822 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0d8400b8c0 con 0x7f0d941060c0 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.007+0000 7f0d92ffd700 1 -- 172.21.15.102:0/2815218822 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d840044d0 con 0x7f0d941060c0 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.008+0000 7f0d9a220700 1 -- 172.21.15.102:0/2815218822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 msgr2=0x7f0d941064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.008+0000 7f0d9a220700 1 --2- 172.21.15.102:0/2815218822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d941064d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d84009e80 tx=0x7f0d8400b330 comp rx=0 tx=0).stop 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.008+0000 7f0d9a220700 1 -- 172.21.15.102:0/2815218822 shutdown_connections 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.008+0000 7f0d9a220700 1 --2- 172.21.15.102:0/2815218822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d941064d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.008+0000 7f0d9a220700 1 -- 172.21.15.102:0/2815218822 >> 172.21.15.102:0/2815218822 conn(0x7f0d940ff7e0 msgr2=0x7f0d94101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:18.773 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.008+0000 7f0d9a220700 1 -- 172.21.15.102:0/2815218822 shutdown_connections 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.008+0000 7f0d9a220700 1 -- 172.21.15.102:0/2815218822 wait complete. 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.008+0000 7f0d9a220700 1 Processor -- start 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.008+0000 7f0d9a220700 1 -- start start 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d9a220700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d9419b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d9a220700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9419bf20 con 0x7f0d941060c0 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d937fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d9419b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d937fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d9419b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49838/0 (socket says 172.21.15.102:49838) 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d937fe700 1 -- 172.21.15.102:0/3726256409 learned_addr learned my addr 172.21.15.102:0/3726256409 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d937fe700 1 -- 172.21.15.102:0/3726256409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d84009b30 con 0x7f0d941060c0 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d937fe700 1 --2- 172.21.15.102:0/3726256409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d9419b9e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d840072a0 tx=0x7f0d840049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d917fa700 1 -- 172.21.15.102:0/3726256409 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d8400e040 con 0x7f0d941060c0 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d917fa700 1 -- 172.21.15.102:0/3726256409 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0d8400bd80 con 0x7f0d941060c0 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d9a220700 1 -- 172.21.15.102:0/3726256409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9419c120 con 0x7f0d941060c0 2024-03-26T18:46:18.774 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d917fa700 1 -- 172.21.15.102:0/3726256409 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d840126c0 con 0x7f0d941060c0 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.009+0000 7f0d9a220700 1 -- 172.21.15.102:0/3726256409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9419c5a0 con 0x7f0d941060c0 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.010+0000 7f0d917fa700 1 -- 172.21.15.102:0/3726256409 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f0d84019040 con 0x7f0d941060c0 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.010+0000 7f0d9a220700 1 -- 172.21.15.102:0/3726256409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d94062380 con 0x7f0d941060c0 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.010+0000 7f0d917fa700 1 --2- 172.21.15.102:0/3726256409 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f0d80038400 0x7f0d8003a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.010+0000 7f0d917fa700 1 -- 172.21.15.102:0/3726256409 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0d8404c010 con 0x7f0d941060c0 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.011+0000 7f0d8affd700 1 --2- 172.21.15.102:0/3726256409 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f0d80038400 0x7f0d8003a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.011+0000 7f0d8affd700 1 --2- 172.21.15.102:0/3726256409 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f0d80038400 0x7f0d8003a8b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c006fd0 tx=0x7f0d7c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.015+0000 7f0d917fa700 1 -- 172.21.15.102:0/3726256409 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0d8407e720 con 0x7f0d941060c0 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.205+0000 7f0d9a220700 1 -- 172.21.15.102:0/3726256409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f0d9419f430 con 0x7f0d941060c0 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.209+0000 7f0d917fa700 1 -- 172.21.15.102:0/3726256409 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0d8401f070 con 0x7f0d941060c0 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.212+0000 7f0d9a220700 1 -- 172.21.15.102:0/3726256409 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f0d80038400 msgr2=0x7f0d8003a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.212+0000 7f0d9a220700 1 --2- 172.21.15.102:0/3726256409 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f0d80038400 0x7f0d8003a8b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c006fd0 tx=0x7f0d7c006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.212+0000 7f0d9a220700 1 -- 172.21.15.102:0/3726256409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 msgr2=0x7f0d9419b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.212+0000 7f0d9a220700 1 --2- 172.21.15.102:0/3726256409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d9419b9e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d840072a0 tx=0x7f0d840049f0 comp rx=0 tx=0).stop 2024-03-26T18:46:18.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.212+0000 7f0d9a220700 1 -- 172.21.15.102:0/3726256409 shutdown_connections 2024-03-26T18:46:18.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.212+0000 7f0d9a220700 1 --2- 172.21.15.102:0/3726256409 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f0d80038400 0x7f0d8003a8b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:18.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.212+0000 7f0d9a220700 1 --2- 172.21.15.102:0/3726256409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d941060c0 0x7f0d9419b9e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:18.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.212+0000 7f0d9a220700 1 -- 172.21.15.102:0/3726256409 >> 172.21.15.102:0/3726256409 conn(0x7f0d940ff7e0 msgr2=0x7f0d94100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:18.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.212+0000 7f0d9a220700 1 -- 172.21.15.102:0/3726256409 shutdown_connections 2024-03-26T18:46:18.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:18.212+0000 7f0d9a220700 1 -- 172.21.15.102:0/3726256409 wait complete. 2024-03-26T18:46:19.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:19 smithi102 ceph-mon[29002]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:46:19.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:19 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3726256409' entity='client.admin' 2024-03-26T18:46:19.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.177+0000 7fa4b3dbd700 1 Processor -- start 2024-03-26T18:46:19.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.177+0000 7fa4b3dbd700 1 -- start start 2024-03-26T18:46:19.955 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.177+0000 7fa4b3dbd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac105f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.177+0000 7fa4b3dbd700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac1064b0 con 0x7fa4ac105b60 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.178+0000 7fa4b1b59700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac105f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.178+0000 7fa4b1b59700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac105f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46818/0 (socket says 172.21.15.102:46818) 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.178+0000 7fa4b1b59700 1 -- 172.21.15.102:0/2053669334 learned_addr learned my addr 172.21.15.102:0/2053669334 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.178+0000 7fa4b1b59700 1 -- 172.21.15.102:0/2053669334 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac1065f0 con 0x7fa4ac105b60 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.178+0000 7fa4b1b59700 1 --2- 172.21.15.102:0/2053669334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac105f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009e80 tx=0x7fa4a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845e8959239e920c server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.178+0000 7fa4b0b57700 1 -- 172.21.15.102:0/2053669334 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a800e070 con 0x7fa4ac105b60 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.178+0000 7fa4b0b57700 1 -- 172.21.15.102:0/2053669334 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa4a800b8c0 con 0x7fa4ac105b60 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.178+0000 7fa4b0b57700 1 -- 172.21.15.102:0/2053669334 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a80044d0 con 0x7fa4ac105b60 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.179+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2053669334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 msgr2=0x7fa4ac105f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.179+0000 7fa4b3dbd700 1 --2- 172.21.15.102:0/2053669334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac105f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009e80 tx=0x7fa4a800b330 comp rx=0 tx=0).stop 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.179+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2053669334 shutdown_connections 2024-03-26T18:46:19.956 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.179+0000 7fa4b3dbd700 1 --2- 172.21.15.102:0/2053669334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac105f70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.179+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2053669334 >> 172.21.15.102:0/2053669334 conn(0x7fa4ac1011b0 msgr2=0x7fa4ac1035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.179+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2053669334 shutdown_connections 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.179+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2053669334 wait complete. 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.179+0000 7fa4b3dbd700 1 Processor -- start 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.179+0000 7fa4b3dbd700 1 -- start start 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4b3dbd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4b3dbd700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac199d70 con 0x7fa4ac105b60 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4b1b59700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4b1b59700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46834/0 (socket says 172.21.15.102:46834) 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4b1b59700 1 -- 172.21.15.102:0/2717634923 learned_addr learned my addr 172.21.15.102:0/2717634923 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4b1b59700 1 -- 172.21.15.102:0/2717634923 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a8009b30 con 0x7fa4ac105b60 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4b1b59700 1 --2- 172.21.15.102:0/2717634923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac199830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80072a0 tx=0x7fa4a80049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:19.957 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4a2ffd700 1 -- 172.21.15.102:0/2717634923 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a800e040 con 0x7fa4ac105b60 2024-03-26T18:46:19.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4a2ffd700 1 -- 172.21.15.102:0/2717634923 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa4a800bd80 con 0x7fa4ac105b60 2024-03-26T18:46:19.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4a2ffd700 1 -- 172.21.15.102:0/2717634923 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a80126c0 con 0x7fa4ac105b60 2024-03-26T18:46:19.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2717634923 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac199f70 con 0x7fa4ac105b60 2024-03-26T18:46:19.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.180+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2717634923 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac19a3f0 con 0x7fa4ac105b60 2024-03-26T18:46:19.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.181+0000 7fa4a2ffd700 1 -- 172.21.15.102:0/2717634923 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fa4a8019040 con 0x7fa4ac105b60 2024-03-26T18:46:19.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.181+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2717634923 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac062380 con 0x7fa4ac105b60 2024-03-26T18:46:19.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.181+0000 7fa4a2ffd700 1 --2- 172.21.15.102:0/2717634923 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fa4980383b0 0x7fa49803a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:19.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.181+0000 7fa4a2ffd700 1 -- 172.21.15.102:0/2717634923 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa4a804bf90 con 0x7fa4ac105b60 2024-03-26T18:46:19.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.181+0000 7fa4b1358700 1 --2- 172.21.15.102:0/2717634923 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fa4980383b0 0x7fa49803a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:19.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.182+0000 7fa4b1358700 1 --2- 172.21.15.102:0/2717634923 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fa4980383b0 0x7fa49803a860 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa49c006fd0 tx=0x7fa49c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:19.958 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.186+0000 7fa4a2ffd700 1 -- 172.21.15.102:0/2717634923 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa4a807e720 con 0x7fa4ac105b60 2024-03-26T18:46:19.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.378+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2717634923 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fa4ac1079a0 con 0x7fa4ac105b60 2024-03-26T18:46:19.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.382+0000 7fa4a2ffd700 1 -- 172.21.15.102:0/2717634923 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fa4a801f020 con 0x7fa4ac105b60 2024-03-26T18:46:19.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.384+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2717634923 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fa4980383b0 msgr2=0x7fa49803a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:19.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.384+0000 7fa4b3dbd700 1 --2- 172.21.15.102:0/2717634923 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fa4980383b0 0x7fa49803a860 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa49c006fd0 tx=0x7fa49c006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:19.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.384+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2717634923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 msgr2=0x7fa4ac199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:19.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.384+0000 7fa4b3dbd700 1 --2- 172.21.15.102:0/2717634923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac199830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80072a0 tx=0x7fa4a80049f0 comp rx=0 tx=0).stop 2024-03-26T18:46:19.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.385+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2717634923 shutdown_connections 2024-03-26T18:46:19.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.385+0000 7fa4b3dbd700 1 --2- 172.21.15.102:0/2717634923 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7fa4980383b0 0x7fa49803a860 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:19.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.385+0000 7fa4b3dbd700 1 --2- 172.21.15.102:0/2717634923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4ac105b60 0x7fa4ac199830 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:19.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.385+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2717634923 >> 172.21.15.102:0/2717634923 conn(0x7fa4ac1011b0 msgr2=0x7fa4ac101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:19.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.385+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2717634923 shutdown_connections 2024-03-26T18:46:19.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:19.385+0000 7fa4b3dbd700 1 -- 172.21.15.102:0/2717634923 wait complete. 2024-03-26T18:46:19.960 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling the dashboard module... 2024-03-26T18:46:20.588 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:20 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2717634923' entity='client.admin' 2024-03-26T18:46:20.588 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:20 smithi102 ceph-mon[29002]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:46:21.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:21 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/4117796369' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T18:46:21.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.334+0000 7ff2cef84700 1 Processor -- start 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.334+0000 7ff2cef84700 1 -- start start 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.334+0000 7ff2cef84700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c81089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.334+0000 7ff2cef84700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8074510 con 0x7ff2c8106610 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.334+0000 7ff2ccd20700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c81089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.334+0000 7ff2ccd20700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c81089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46848/0 (socket says 172.21.15.102:46848) 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.334+0000 7ff2ccd20700 1 -- 172.21.15.102:0/3713917328 learned_addr learned my addr 172.21.15.102:0/3713917328 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.334+0000 7ff2ccd20700 1 -- 172.21.15.102:0/3713917328 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c8074650 con 0x7ff2c8106610 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.335+0000 7ff2ccd20700 1 --2- 172.21.15.102:0/3713917328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c81089f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0098a0 tx=0x7ff2bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e052a0f7205d014 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.335+0000 7ff2c77fe700 1 -- 172.21.15.102:0/3713917328 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2bc004180 con 0x7ff2c8106610 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.335+0000 7ff2c77fe700 1 -- 172.21.15.102:0/3713917328 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff2bc0042e0 con 0x7ff2c8106610 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.335+0000 7ff2c77fe700 1 -- 172.21.15.102:0/3713917328 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2bc011af0 con 0x7ff2c8106610 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.335+0000 7ff2cef84700 1 -- 172.21.15.102:0/3713917328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 msgr2=0x7ff2c81089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.335+0000 7ff2cef84700 1 --2- 172.21.15.102:0/3713917328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c81089f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0098a0 tx=0x7ff2bc009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:21.961 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.335+0000 7ff2cef84700 1 -- 172.21.15.102:0/3713917328 shutdown_connections 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.335+0000 7ff2cef84700 1 --2- 172.21.15.102:0/3713917328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c81089f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.335+0000 7ff2cef84700 1 -- 172.21.15.102:0/3713917328 >> 172.21.15.102:0/3713917328 conn(0x7ff2c8100070 msgr2=0x7ff2c8102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.335+0000 7ff2cef84700 1 -- 172.21.15.102:0/3713917328 shutdown_connections 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.335+0000 7ff2cef84700 1 -- 172.21.15.102:0/3713917328 wait complete. 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.336+0000 7ff2cef84700 1 Processor -- start 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.336+0000 7ff2cef84700 1 -- start start 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.336+0000 7ff2cef84700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c8199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.336+0000 7ff2cef84700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8199db0 con 0x7ff2c8106610 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.336+0000 7ff2ccd20700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c8199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.336+0000 7ff2ccd20700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c8199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46858/0 (socket says 172.21.15.102:46858) 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.336+0000 7ff2ccd20700 1 -- 172.21.15.102:0/4117796369 learned_addr learned my addr 172.21.15.102:0/4117796369 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.337+0000 7ff2ccd20700 1 -- 172.21.15.102:0/4117796369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc009550 con 0x7ff2c8106610 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.337+0000 7ff2ccd20700 1 --2- 172.21.15.102:0/4117796369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c8199870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc000c00 tx=0x7ff2bc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:21.962 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.337+0000 7ff2c5ffb700 1 -- 172.21.15.102:0/4117796369 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2bc003a70 con 0x7ff2c8106610 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.337+0000 7ff2cef84700 1 -- 172.21.15.102:0/4117796369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c8199fb0 con 0x7ff2c8106610 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.337+0000 7ff2c5ffb700 1 -- 172.21.15.102:0/4117796369 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff2bc003bd0 con 0x7ff2c8106610 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.337+0000 7ff2c5ffb700 1 -- 172.21.15.102:0/4117796369 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2bc0238c0 con 0x7ff2c8106610 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.337+0000 7ff2cef84700 1 -- 172.21.15.102:0/4117796369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c819a3d0 con 0x7ff2c8106610 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.338+0000 7ff2cef84700 1 -- 172.21.15.102:0/4117796369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c8109bf0 con 0x7ff2c8106610 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.338+0000 7ff2c5ffb700 1 -- 172.21.15.102:0/4117796369 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7ff2bc023b30 con 0x7ff2c8106610 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.338+0000 7ff2c5ffb700 1 --2- 172.21.15.102:0/4117796369 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7ff2b8038440 0x7ff2b803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.338+0000 7ff2c5ffb700 1 -- 172.21.15.102:0/4117796369 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff2bc04c3b0 con 0x7ff2c8106610 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.339+0000 7ff2c7fff700 1 --2- 172.21.15.102:0/4117796369 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7ff2b8038440 0x7ff2b803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.339+0000 7ff2c7fff700 1 --2- 172.21.15.102:0/4117796369 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7ff2b8038440 0x7ff2b803a8f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4006fd0 tx=0x7ff2b4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.343+0000 7ff2c5ffb700 1 -- 172.21.15.102:0/4117796369 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff2bc010c20 con 0x7ff2c8106610 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:20.598+0000 7ff2cef84700 1 -- 172.21.15.102:0/4117796369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ff2c8109de0 con 0x7ff2c8106610 2024-03-26T18:46:21.963 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.392+0000 7ff2c5ffb700 1 -- 172.21.15.102:0/4117796369 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7ff2bc019930 con 0x7ff2c8106610 2024-03-26T18:46:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.393+0000 7ff2c5ffb700 1 -- 172.21.15.102:0/4117796369 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7ff2bc01ad00 con 0x7ff2c8106610 2024-03-26T18:46:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.395+0000 7ff2cef84700 1 -- 172.21.15.102:0/4117796369 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7ff2b8038440 msgr2=0x7ff2b803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.395+0000 7ff2cef84700 1 --2- 172.21.15.102:0/4117796369 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7ff2b8038440 0x7ff2b803a8f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4006fd0 tx=0x7ff2b4006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.395+0000 7ff2cef84700 1 -- 172.21.15.102:0/4117796369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 msgr2=0x7ff2c8199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.395+0000 7ff2cef84700 1 --2- 172.21.15.102:0/4117796369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c8199870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc000c00 tx=0x7ff2bc011b50 comp rx=0 tx=0).stop 2024-03-26T18:46:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.395+0000 7ff2cef84700 1 -- 172.21.15.102:0/4117796369 shutdown_connections 2024-03-26T18:46:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.395+0000 7ff2cef84700 1 --2- 172.21.15.102:0/4117796369 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7ff2b8038440 0x7ff2b803a8f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.395+0000 7ff2cef84700 1 --2- 172.21.15.102:0/4117796369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c8106610 0x7ff2c8199870 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.395+0000 7ff2cef84700 1 -- 172.21.15.102:0/4117796369 >> 172.21.15.102:0/4117796369 conn(0x7ff2c8100070 msgr2=0x7ff2c8101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.396+0000 7ff2cef84700 1 -- 172.21.15.102:0/4117796369 shutdown_connections 2024-03-26T18:46:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:21.396+0000 7ff2cef84700 1 -- 172.21.15.102:0/4117796369 wait complete. 2024-03-26T18:46:22.594 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:22 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/4117796369' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T18:46:22.594 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:22 smithi102 ceph-mon[29002]: mgrmap e9: smithi102.rsfpnv(active, since 25s) 2024-03-26T18:46:23.326 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-26T18:46:23.326 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T18:46:23.326 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T18:46:23.326 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "smithi102.rsfpnv", 2024-03-26T18:46:23.326 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T18:46:23.326 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.397+0000 7f47f3c40700 1 Processor -- start 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.398+0000 7f47f3c40700 1 -- start start 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.398+0000 7f47f3c40700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec079cf0 0x7f47ec10aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.398+0000 7f47f3c40700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec1064c0 con 0x7f47ec079cf0 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.398+0000 7f47f19dc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec079cf0 0x7f47ec10aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.398+0000 7f47f19dc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec079cf0 0x7f47ec10aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46878/0 (socket says 172.21.15.102:46878) 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.398+0000 7f47f19dc700 1 -- 172.21.15.102:0/1132889240 learned_addr learned my addr 172.21.15.102:0/1132889240 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.398+0000 7f47f19dc700 1 -- 172.21.15.102:0/1132889240 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec10b010 con 0x7f47ec079cf0 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.398+0000 7f47f19dc700 1 --2- 172.21.15.102:0/1132889240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec079cf0 0x7f47ec10aad0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f47e80098a0 tx=0x7f47e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e549035e23d7d07f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.399+0000 7f47f09da700 1 -- 172.21.15.102:0/1132889240 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e8004180 con 0x7f47ec079cf0 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.399+0000 7f47f09da700 1 -- 172.21.15.102:0/1132889240 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f47e80042e0 con 0x7f47ec079cf0 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.399+0000 7f47f09da700 1 -- 172.21.15.102:0/1132889240 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e8011af0 con 0x7f47ec079cf0 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.399+0000 7f47f3c40700 1 -- 172.21.15.102:0/1132889240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec079cf0 msgr2=0x7f47ec10aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:23.327 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.399+0000 7f47f3c40700 1 --2- 172.21.15.102:0/1132889240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec079cf0 0x7f47ec10aad0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f47e80098a0 tx=0x7f47e8009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.399+0000 7f47f3c40700 1 -- 172.21.15.102:0/1132889240 shutdown_connections 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.399+0000 7f47f3c40700 1 --2- 172.21.15.102:0/1132889240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec079cf0 0x7f47ec10aad0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.399+0000 7f47f3c40700 1 -- 172.21.15.102:0/1132889240 >> 172.21.15.102:0/1132889240 conn(0x7f47ec100060 msgr2=0x7f47ec1024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.399+0000 7f47f3c40700 1 -- 172.21.15.102:0/1132889240 shutdown_connections 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.399+0000 7f47f3c40700 1 -- 172.21.15.102:0/1132889240 wait complete. 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.400+0000 7f47f3c40700 1 Processor -- start 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.400+0000 7f47f3c40700 1 -- start start 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.400+0000 7f47f3c40700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec19baf0 0x7f47ec19bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.400+0000 7f47f3c40700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec19c440 con 0x7f47ec19baf0 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.400+0000 7f47f19dc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec19baf0 0x7f47ec19bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.400+0000 7f47f19dc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec19baf0 0x7f47ec19bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46882/0 (socket says 172.21.15.102:46882) 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.400+0000 7f47f19dc700 1 -- 172.21.15.102:0/3008707113 learned_addr learned my addr 172.21.15.102:0/3008707113 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.400+0000 7f47f19dc700 1 -- 172.21.15.102:0/3008707113 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8009550 con 0x7f47ec19baf0 2024-03-26T18:46:23.328 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.401+0000 7f47f19dc700 1 --2- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec19baf0 0x7f47ec19bf00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f47e8011670 tx=0x7f47e8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.401+0000 7f47e2ffd700 1 -- 172.21.15.102:0/3008707113 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e80039c0 con 0x7f47ec19baf0 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.401+0000 7f47f3c40700 1 -- 172.21.15.102:0/3008707113 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec19c640 con 0x7f47ec19baf0 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.401+0000 7f47e2ffd700 1 -- 172.21.15.102:0/3008707113 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f47e8003b20 con 0x7f47ec19baf0 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.401+0000 7f47e2ffd700 1 -- 172.21.15.102:0/3008707113 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e80247f0 con 0x7f47ec19baf0 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.401+0000 7f47f3c40700 1 -- 172.21.15.102:0/3008707113 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ec19f2a0 con 0x7f47ec19baf0 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.402+0000 7f47e2ffd700 1 -- 172.21.15.102:0/3008707113 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f47e8024ab0 con 0x7f47ec19baf0 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.402+0000 7f47e2ffd700 1 --2- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f47d8038400 0x7f47d803a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.402+0000 7f47e2ffd700 1 -- 172.21.15.102:0/3008707113 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f47e804c360 con 0x7f47ec19baf0 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.402+0000 7f47f11db700 1 -- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f47d8038400 msgr2=0x7f47d803a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1874485118 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.402+0000 7f47f11db700 1 --2- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f47d8038400 0x7f47d803a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.402+0000 7f47f3c40700 1 -- 172.21.15.102:0/3008707113 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ec062380 con 0x7f47ec19baf0 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.407+0000 7f47e2ffd700 1 -- 172.21.15.102:0/3008707113 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f47e8022310 con 0x7f47ec19baf0 2024-03-26T18:46:23.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.603+0000 7f47f11db700 1 -- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f47d8038400 msgr2=0x7f47d803a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1874485118 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.603+0000 7f47f11db700 1 --2- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f47d8038400 0x7f47d803a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.664+0000 7f47f3c40700 1 -- 172.21.15.102:0/3008707113 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f47ec079f20 con 0x7f47ec19baf0 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.664+0000 7f47e2ffd700 1 -- 172.21.15.102:0/3008707113 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f47e802c350 con 0x7f47ec19baf0 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.667+0000 7f47f3c40700 1 -- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f47d8038400 msgr2=0x7f47d803a8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.667+0000 7f47f3c40700 1 --2- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f47d8038400 0x7f47d803a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.667+0000 7f47f3c40700 1 -- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec19baf0 msgr2=0x7f47ec19bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.667+0000 7f47f3c40700 1 --2- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec19baf0 0x7f47ec19bf00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f47e8011670 tx=0x7f47e8011aa0 comp rx=0 tx=0).stop 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.667+0000 7f47f3c40700 1 -- 172.21.15.102:0/3008707113 shutdown_connections 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.667+0000 7f47f3c40700 1 --2- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f47d8038400 0x7f47d803a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.667+0000 7f47f3c40700 1 --2- 172.21.15.102:0/3008707113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47ec19baf0 0x7f47ec19bf00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.667+0000 7f47f3c40700 1 -- 172.21.15.102:0/3008707113 >> 172.21.15.102:0/3008707113 conn(0x7f47ec100060 msgr2=0x7f47ec1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.667+0000 7f47f3c40700 1 -- 172.21.15.102:0/3008707113 shutdown_connections 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:22.667+0000 7f47f3c40700 1 -- 172.21.15.102:0/3008707113 wait complete. 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-03-26T18:46:23.330 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 9... 2024-03-26T18:46:23.577 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:23 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3008707113' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T18:46:28.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: Active manager daemon smithi102.rsfpnv restarted 2024-03-26T18:46:28.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: Activating manager daemon smithi102.rsfpnv 2024-03-26T18:46:28.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T18:46:28.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: mgrmap e10: smithi102.rsfpnv(active, starting, since 0.00359523s) 2024-03-26T18:46:28.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:46:28.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi102.rsfpnv", "id": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:46:28.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T18:46:28.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T18:46:28.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T18:46:28.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: Manager daemon smithi102.rsfpnv is now available 2024-03-26T18:46:28.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:46:28.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:28 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:46:29.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:29 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:46:29.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:29 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/trash_purge_schedule"}]: dispatch 2024-03-26T18:46:29.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:29 smithi102 ceph-mon[29002]: mgrmap e11: smithi102.rsfpnv(active, since 1.00534s) 2024-03-26T18:46:29.755 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-26T18:46:29.755 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T18:46:29.755 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T18:46:29.755 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-26T18:46:29.755 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.780+0000 7f1599796700 1 Processor -- start 2024-03-26T18:46:29.755 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.780+0000 7f1599796700 1 -- start start 2024-03-26T18:46:29.755 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.780+0000 7f1599796700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f1594104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:29.755 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.780+0000 7f1599796700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1594105350 con 0x7f1594104a00 2024-03-26T18:46:29.755 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.780+0000 7f1592ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f1594104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:29.755 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.780+0000 7f1592ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f1594104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46886/0 (socket says 172.21.15.102:46886) 2024-03-26T18:46:29.755 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.780+0000 7f1592ffd700 1 -- 172.21.15.102:0/1026969495 learned_addr learned my addr 172.21.15.102:0/1026969495 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.781+0000 7f1592ffd700 1 -- 172.21.15.102:0/1026969495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1594105490 con 0x7f1594104a00 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.781+0000 7f1592ffd700 1 --2- 172.21.15.102:0/1026969495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f1594104e10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f15840098a0 tx=0x7f1584009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b583f8bf8072ad3c server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.781+0000 7f1591ffb700 1 -- 172.21.15.102:0/1026969495 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1584004180 con 0x7f1594104a00 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.781+0000 7f1591ffb700 1 -- 172.21.15.102:0/1026969495 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f15840042e0 con 0x7f1594104a00 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.781+0000 7f1591ffb700 1 -- 172.21.15.102:0/1026969495 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1584011af0 con 0x7f1594104a00 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.782+0000 7f1599796700 1 -- 172.21.15.102:0/1026969495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 msgr2=0x7f1594104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.782+0000 7f1599796700 1 --2- 172.21.15.102:0/1026969495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f1594104e10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f15840098a0 tx=0x7f1584009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.782+0000 7f1599796700 1 -- 172.21.15.102:0/1026969495 shutdown_connections 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.782+0000 7f1599796700 1 --2- 172.21.15.102:0/1026969495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f1594104e10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.782+0000 7f1599796700 1 -- 172.21.15.102:0/1026969495 >> 172.21.15.102:0/1026969495 conn(0x7f1594100070 msgr2=0x7f1594102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.782+0000 7f1599796700 1 -- 172.21.15.102:0/1026969495 shutdown_connections 2024-03-26T18:46:29.756 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.782+0000 7f1599796700 1 -- 172.21.15.102:0/1026969495 wait complete. 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.782+0000 7f1599796700 1 Processor -- start 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.782+0000 7f1599796700 1 -- start start 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.783+0000 7f1599796700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f159418fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.783+0000 7f1599796700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1594190120 con 0x7f1594104a00 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.783+0000 7f1592ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f159418fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.783+0000 7f1592ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f159418fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46902/0 (socket says 172.21.15.102:46902) 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.783+0000 7f1592ffd700 1 -- 172.21.15.102:0/3383298332 learned_addr learned my addr 172.21.15.102:0/3383298332 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.783+0000 7f1592ffd700 1 -- 172.21.15.102:0/3383298332 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1584009550 con 0x7f1594104a00 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.783+0000 7f1592ffd700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f159418fbe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1584006ba0 tx=0x7f1584011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.783+0000 7f157bfff700 1 -- 172.21.15.102:0/3383298332 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1584003b00 con 0x7f1594104a00 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.783+0000 7f157bfff700 1 -- 172.21.15.102:0/3383298332 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f158401a8e0 con 0x7f1594104a00 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.783+0000 7f157bfff700 1 -- 172.21.15.102:0/3383298332 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15840199c0 con 0x7f1594104a00 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.783+0000 7f1599796700 1 -- 172.21.15.102:0/3383298332 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159418e230 con 0x7f1594104a00 2024-03-26T18:46:29.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.784+0000 7f1599796700 1 -- 172.21.15.102:0/3383298332 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159418e650 con 0x7f1594104a00 2024-03-26T18:46:29.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.784+0000 7f157bfff700 1 -- 172.21.15.102:0/3383298332 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f1584019c80 con 0x7f1594104a00 2024-03-26T18:46:29.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.784+0000 7f157bfff700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 0x7f157c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:29.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.784+0000 7f1599796700 1 -- 172.21.15.102:0/3383298332 --> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1594062380 con 0x7f157c0383f0 2024-03-26T18:46:29.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.785+0000 7f157bfff700 1 -- 172.21.15.102:0/3383298332 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f158404c710 con 0x7f1594104a00 2024-03-26T18:46:29.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.785+0000 7f15927fc700 1 -- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 msgr2=0x7f157c03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1874485118 2024-03-26T18:46:29.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.785+0000 7f15927fc700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 0x7f157c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T18:46:29.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.986+0000 7f15927fc700 1 -- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 msgr2=0x7f157c03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1874485118 2024-03-26T18:46:29.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:23.986+0000 7f15927fc700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 0x7f157c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T18:46:29.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:24.387+0000 7f15927fc700 1 -- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 msgr2=0x7f157c03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1874485118 2024-03-26T18:46:29.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:24.387+0000 7f15927fc700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 0x7f157c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T18:46:29.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:25.188+0000 7f15927fc700 1 -- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 msgr2=0x7f157c03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1874485118 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:25.188+0000 7f15927fc700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 0x7f157c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:26.789+0000 7f15927fc700 1 -- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 msgr2=0x7f157c03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1874485118 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:26.789+0000 7f15927fc700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 0x7f157c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:28.036+0000 7f157bfff700 1 -- 172.21.15.102:0/3383298332 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mgrmap(e 10) v1 ==== 44864+0+0 (secure 0 0 0) 0x7f1584018c70 con 0x7f1594104a00 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:28.036+0000 7f157bfff700 1 -- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 msgr2=0x7f157c03a8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:28.036+0000 7f157bfff700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 0x7f157c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.037+0000 7f157bfff700 1 -- 172.21.15.102:0/3383298332 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 11) v1 ==== 44991+0+0 (secure 0 0 0) 0x7f158402c3a0 con 0x7f1594104a00 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.037+0000 7f157bfff700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f157c03bca0 0x7f157c03e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.037+0000 7f157bfff700 1 -- 172.21.15.102:0/3383298332 --> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1594062380 con 0x7f157c03bca0 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.037+0000 7f15927fc700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f157c03bca0 0x7f157c03e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.038+0000 7f15927fc700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f157c03bca0 0x7f157c03e080 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1588003a10 tx=0x7f15880092b0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.038+0000 7f157bfff700 1 -- 172.21.15.102:0/3383298332 <== mgr.14160 v2:172.21.15.102:6800/3517043895 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f1594062380 con 0x7f157c03bca0 2024-03-26T18:46:29.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 -- 172.21.15.102:0/3383298332 --> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f159418f780 con 0x7f157c03bca0 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f157bfff700 1 -- 172.21.15.102:0/3383298332 <== mgr.14160 v2:172.21.15.102:6800/3517043895 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f159418f780 con 0x7f157c03bca0 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 -- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f157c03bca0 msgr2=0x7f157c03e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f157c03bca0 0x7f157c03e080 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1588003a10 tx=0x7f15880092b0 comp rx=0 tx=0).stop 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 -- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 msgr2=0x7f159418fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f159418fbe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1584006ba0 tx=0x7f1584011b50 comp rx=0 tx=0).stop 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 -- 172.21.15.102:0/3383298332 shutdown_connections 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f157c03bca0 0x7f157c03e080 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1594104a00 0x7f159418fbe0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 --2- 172.21.15.102:0/3383298332 >> [v2:172.21.15.102:6800/1874485118,v1:172.21.15.102:6801/1874485118] conn(0x7f157c0383f0 0x7f157c03a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 -- 172.21.15.102:0/3383298332 >> 172.21.15.102:0/3383298332 conn(0x7f1594100070 msgr2=0x7f15940740b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 -- 172.21.15.102:0/3383298332 shutdown_connections 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:29.045+0000 7f1599796700 1 -- 172.21.15.102:0/3383298332 wait complete. 2024-03-26T18:46:29.760 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 9 is available 2024-03-26T18:46:29.761 INFO:teuthology.orchestra.run.smithi102.stdout:Generating a dashboard self-signed certificate... 2024-03-26T18:46:30.391 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:30 smithi102 ceph-mon[29002]: [26/Mar/2024:18:46:29] ENGINE Bus STARTING 2024-03-26T18:46:30.391 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:30 smithi102 ceph-mon[29002]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T18:46:30.391 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:30 smithi102 ceph-mon[29002]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T18:46:30.392 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:30 smithi102 ceph-mon[29002]: [26/Mar/2024:18:46:29] ENGINE Serving on https://172.21.15.102:7150 2024-03-26T18:46:30.392 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:30 smithi102 ceph-mon[29002]: [26/Mar/2024:18:46:29] ENGINE Serving on http://172.21.15.102:8765 2024-03-26T18:46:30.392 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:30 smithi102 ceph-mon[29002]: [26/Mar/2024:18:46:29] ENGINE Bus STARTED 2024-03-26T18:46:30.392 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:30 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:30.392 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:30 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:31.083 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T18:46:31.083 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.184+0000 7f77969d5700 1 Processor -- start 2024-03-26T18:46:31.083 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.184+0000 7f77969d5700 1 -- start start 2024-03-26T18:46:31.083 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.184+0000 7f77969d5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.184+0000 7f77969d5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7790105350 con 0x7f7790104a00 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.184+0000 7f778ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.184+0000 7f778ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42444/0 (socket says 172.21.15.102:42444) 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.184+0000 7f778ffff700 1 -- 172.21.15.102:0/37318316 learned_addr learned my addr 172.21.15.102:0/37318316 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.185+0000 7f778ffff700 1 -- 172.21.15.102:0/37318316 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7790105490 con 0x7f7790104a00 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.185+0000 7f778ffff700 1 --2- 172.21.15.102:0/37318316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790104e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77800098a0 tx=0x7f7780009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490988f6fdd94dea server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.185+0000 7f778effd700 1 -- 172.21.15.102:0/37318316 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7780004180 con 0x7f7790104a00 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.185+0000 7f778effd700 1 -- 172.21.15.102:0/37318316 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f77800042e0 con 0x7f7790104a00 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.185+0000 7f778effd700 1 -- 172.21.15.102:0/37318316 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7780011af0 con 0x7f7790104a00 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.185+0000 7f77969d5700 1 -- 172.21.15.102:0/37318316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 msgr2=0x7f7790104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.185+0000 7f77969d5700 1 --2- 172.21.15.102:0/37318316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790104e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77800098a0 tx=0x7f7780009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.186+0000 7f77969d5700 1 -- 172.21.15.102:0/37318316 shutdown_connections 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.186+0000 7f77969d5700 1 --2- 172.21.15.102:0/37318316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790104e10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.186+0000 7f77969d5700 1 -- 172.21.15.102:0/37318316 >> 172.21.15.102:0/37318316 conn(0x7f7790100070 msgr2=0x7f7790102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:31.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.186+0000 7f77969d5700 1 -- 172.21.15.102:0/37318316 shutdown_connections 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.186+0000 7f77969d5700 1 -- 172.21.15.102:0/37318316 wait complete. 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.186+0000 7f77969d5700 1 Processor -- start 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.186+0000 7f77969d5700 1 -- start start 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.186+0000 7f77969d5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.186+0000 7f77969d5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7790197b50 con 0x7f7790104a00 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.187+0000 7f778ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.187+0000 7f778ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42454/0 (socket says 172.21.15.102:42454) 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.187+0000 7f778ffff700 1 -- 172.21.15.102:0/1446857285 learned_addr learned my addr 172.21.15.102:0/1446857285 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.187+0000 7f778ffff700 1 -- 172.21.15.102:0/1446857285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7780009550 con 0x7f7790104a00 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.187+0000 7f778ffff700 1 --2- 172.21.15.102:0/1446857285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790197610 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7780011ad0 tx=0x7f7780011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.187+0000 7f778d7fa700 1 -- 172.21.15.102:0/1446857285 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7780003ab0 con 0x7f7790104a00 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.187+0000 7f77969d5700 1 -- 172.21.15.102:0/1446857285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7790197d50 con 0x7f7790104a00 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.187+0000 7f778d7fa700 1 -- 172.21.15.102:0/1446857285 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7780003c10 con 0x7f7790104a00 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.187+0000 7f778d7fa700 1 -- 172.21.15.102:0/1446857285 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77800198e0 con 0x7f7790104a00 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.187+0000 7f77969d5700 1 -- 172.21.15.102:0/1446857285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7790198170 con 0x7f7790104a00 2024-03-26T18:46:31.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.188+0000 7f778d7fa700 1 -- 172.21.15.102:0/1446857285 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 11) v1 ==== 44991+0+0 (secure 0 0 0) 0x7f7780019b60 con 0x7f7790104a00 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.188+0000 7f778d7fa700 1 --2- 172.21.15.102:0/1446857285 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f777803a580 0x7f777803ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.188+0000 7f778d7fa700 1 -- 172.21.15.102:0/1446857285 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f778004d140 con 0x7f7790104a00 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.188+0000 7f778f7fe700 1 --2- 172.21.15.102:0/1446857285 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f777803a580 0x7f777803ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.188+0000 7f77969d5700 1 -- 172.21.15.102:0/1446857285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7790191330 con 0x7f7790104a00 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.189+0000 7f778f7fe700 1 --2- 172.21.15.102:0/1446857285 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f777803a580 0x7f777803ca30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7784006fd0 tx=0x7f7784006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.193+0000 7f778d7fa700 1 -- 172.21.15.102:0/1446857285 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f778002b300 con 0x7f7790104a00 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.400+0000 7f778d7fa700 1 -- 172.21.15.102:0/1446857285 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f778000bc80 con 0x7f7790104a00 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.402+0000 7f77969d5700 1 -- 172.21.15.102:0/1446857285 --> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f7790106a30 con 0x7f777803a580 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.526+0000 7f778d7fa700 1 -- 172.21.15.102:0/1446857285 <== mgr.14160 v2:172.21.15.102:6800/3517043895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f7784009e20 con 0x7f777803a580 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.528+0000 7f77969d5700 1 -- 172.21.15.102:0/1446857285 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f777803a580 msgr2=0x7f777803ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.528+0000 7f77969d5700 1 --2- 172.21.15.102:0/1446857285 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f777803a580 0x7f777803ca30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7784006fd0 tx=0x7f7784006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:31.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.528+0000 7f77969d5700 1 -- 172.21.15.102:0/1446857285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 msgr2=0x7f7790197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:31.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.528+0000 7f77969d5700 1 --2- 172.21.15.102:0/1446857285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790197610 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7780011ad0 tx=0x7f7780011bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:31.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.528+0000 7f77969d5700 1 -- 172.21.15.102:0/1446857285 shutdown_connections 2024-03-26T18:46:31.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.528+0000 7f77969d5700 1 --2- 172.21.15.102:0/1446857285 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f777803a580 0x7f777803ca30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:31.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.528+0000 7f77969d5700 1 --2- 172.21.15.102:0/1446857285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7790104a00 0x7f7790197610 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:31.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.528+0000 7f77969d5700 1 -- 172.21.15.102:0/1446857285 >> 172.21.15.102:0/1446857285 conn(0x7f7790100070 msgr2=0x7f779018e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:31.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.528+0000 7f77969d5700 1 -- 172.21.15.102:0/1446857285 shutdown_connections 2024-03-26T18:46:31.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:30.528+0000 7f77969d5700 1 -- 172.21.15.102:0/1446857285 wait complete. 2024-03-26T18:46:31.087 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial admin user... 2024-03-26T18:46:31.521 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:31 smithi102 ceph-mon[29002]: mgrmap e12: smithi102.rsfpnv(active, since 2s) 2024-03-26T18:46:31.521 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:31 smithi102 ceph-mon[29002]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:31.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:31 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:31.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:31 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:32.661 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$oK/PAGy9XnXNL2LhPQyABeKCJ5tSQtKxf1FLQYaTxPOhfFu1N8DM.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711478792, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T18:46:32.661 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.558+0000 7f3c029cb700 1 Processor -- start 2024-03-26T18:46:32.661 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.558+0000 7f3c029cb700 1 -- start start 2024-03-26T18:46:32.661 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.558+0000 7f3c029cb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc108a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:32.661 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.558+0000 7f3c029cb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc079d70 con 0x7f3bfc106640 2024-03-26T18:46:32.661 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.559+0000 7f3bfbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc108a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:32.661 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.559+0000 7f3bfbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc108a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42460/0 (socket says 172.21.15.102:42460) 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.559+0000 7f3bfbfff700 1 -- 172.21.15.102:0/4270870491 learned_addr learned my addr 172.21.15.102:0/4270870491 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.559+0000 7f3bfbfff700 1 -- 172.21.15.102:0/4270870491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc079eb0 con 0x7f3bfc106640 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.559+0000 7f3bfbfff700 1 --2- 172.21.15.102:0/4270870491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc108a20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0098a0 tx=0x7f3bec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a42216fae3ab96f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.559+0000 7f3bfaffd700 1 -- 172.21.15.102:0/4270870491 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bec004180 con 0x7f3bfc106640 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.559+0000 7f3bfaffd700 1 -- 172.21.15.102:0/4270870491 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3bec0042e0 con 0x7f3bfc106640 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.560+0000 7f3bfaffd700 1 -- 172.21.15.102:0/4270870491 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bec011af0 con 0x7f3bfc106640 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.560+0000 7f3c029cb700 1 -- 172.21.15.102:0/4270870491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 msgr2=0x7f3bfc108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.560+0000 7f3c029cb700 1 --2- 172.21.15.102:0/4270870491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc108a20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0098a0 tx=0x7f3bec009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.560+0000 7f3c029cb700 1 -- 172.21.15.102:0/4270870491 shutdown_connections 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.560+0000 7f3c029cb700 1 --2- 172.21.15.102:0/4270870491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc108a20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.560+0000 7f3c029cb700 1 -- 172.21.15.102:0/4270870491 >> 172.21.15.102:0/4270870491 conn(0x7f3bfc100060 msgr2=0x7f3bfc1024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.560+0000 7f3c029cb700 1 -- 172.21.15.102:0/4270870491 shutdown_connections 2024-03-26T18:46:32.662 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.561+0000 7f3c029cb700 1 -- 172.21.15.102:0/4270870491 wait complete. 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.561+0000 7f3c029cb700 1 Processor -- start 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.561+0000 7f3c029cb700 1 -- start start 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.561+0000 7f3c029cb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.561+0000 7f3c029cb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc199e70 con 0x7f3bfc106640 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.562+0000 7f3bfbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.562+0000 7f3bfbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42464/0 (socket says 172.21.15.102:42464) 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.562+0000 7f3bfbfff700 1 -- 172.21.15.102:0/2969741119 learned_addr learned my addr 172.21.15.102:0/2969741119 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.562+0000 7f3bfbfff700 1 -- 172.21.15.102:0/2969741119 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec009550 con 0x7f3bfc106640 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.562+0000 7f3bfbfff700 1 --2- 172.21.15.102:0/2969741119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc199930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3bec011670 tx=0x7f3bec011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.562+0000 7f3bf97fa700 1 -- 172.21.15.102:0/2969741119 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bec0039f0 con 0x7f3bfc106640 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.562+0000 7f3bf97fa700 1 -- 172.21.15.102:0/2969741119 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3bec003b50 con 0x7f3bfc106640 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.562+0000 7f3c029cb700 1 -- 172.21.15.102:0/2969741119 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc19a070 con 0x7f3bfc106640 2024-03-26T18:46:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.562+0000 7f3bf97fa700 1 -- 172.21.15.102:0/2969741119 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bec0247c0 con 0x7f3bfc106640 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.562+0000 7f3c029cb700 1 -- 172.21.15.102:0/2969741119 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc19a490 con 0x7f3bfc106640 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.563+0000 7f3c029cb700 1 -- 172.21.15.102:0/2969741119 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc193950 con 0x7f3bfc106640 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.564+0000 7f3bf97fa700 1 -- 172.21.15.102:0/2969741119 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f3bec01a430 con 0x7f3bfc106640 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.564+0000 7f3bf97fa700 1 --2- 172.21.15.102:0/2969741119 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3be4038490 0x7f3be403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.564+0000 7f3bf97fa700 1 -- 172.21.15.102:0/2969741119 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3bec04c5c0 con 0x7f3bfc106640 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.564+0000 7f3bfb7fe700 1 --2- 172.21.15.102:0/2969741119 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3be4038490 0x7f3be403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.564+0000 7f3bfb7fe700 1 --2- 172.21.15.102:0/2969741119 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3be4038490 0x7f3be403a940 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0006fd0 tx=0x7f3bf0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.569+0000 7f3bf97fa700 1 -- 172.21.15.102:0/2969741119 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3bec01ac00 con 0x7f3bfc106640 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:31.784+0000 7f3c029cb700 1 -- 172.21.15.102:0/2969741119 --> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3bfc02cf30 con 0x7f3be4038490 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:32.007+0000 7f3bf97fa700 1 -- 172.21.15.102:0/2969741119 <== mgr.14160 v2:172.21.15.102:6800/3517043895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f3bfc02cf30 con 0x7f3be4038490 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:32.009+0000 7f3c029cb700 1 -- 172.21.15.102:0/2969741119 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3be4038490 msgr2=0x7f3be403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:32.009+0000 7f3c029cb700 1 --2- 172.21.15.102:0/2969741119 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3be4038490 0x7f3be403a940 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0006fd0 tx=0x7f3bf0006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:32.009+0000 7f3c029cb700 1 -- 172.21.15.102:0/2969741119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 msgr2=0x7f3bfc199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:32.009+0000 7f3c029cb700 1 --2- 172.21.15.102:0/2969741119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc199930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3bec011670 tx=0x7f3bec011b50 comp rx=0 tx=0).stop 2024-03-26T18:46:32.665 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:32.009+0000 7f3c029cb700 1 -- 172.21.15.102:0/2969741119 shutdown_connections 2024-03-26T18:46:32.665 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:32.009+0000 7f3c029cb700 1 --2- 172.21.15.102:0/2969741119 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3be4038490 0x7f3be403a940 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:32.665 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:32.009+0000 7f3c029cb700 1 --2- 172.21.15.102:0/2969741119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bfc106640 0x7f3bfc199930 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:32.665 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:32.009+0000 7f3c029cb700 1 -- 172.21.15.102:0/2969741119 >> 172.21.15.102:0/2969741119 conn(0x7f3bfc100060 msgr2=0x7f3bfc1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:32.665 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:32.009+0000 7f3c029cb700 1 -- 172.21.15.102:0/2969741119 shutdown_connections 2024-03-26T18:46:32.665 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:32.009+0000 7f3c029cb700 1 -- 172.21.15.102:0/2969741119 wait complete. 2024-03-26T18:46:32.665 INFO:teuthology.orchestra.run.smithi102.stdout:Fetching dashboard port number... 2024-03-26T18:46:33.259 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:33 smithi102 ceph-mon[29002]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:33.259 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:33 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:34.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T18:46:34.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.062+0000 7f4af27da700 1 Processor -- start 2024-03-26T18:46:34.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.062+0000 7f4af27da700 1 -- start start 2024-03-26T18:46:34.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.062+0000 7f4af27da700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:34.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.062+0000 7f4af27da700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec106460 con 0x7f4aec105b10 2024-03-26T18:46:34.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.062+0000 7f4aebfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:34.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.062+0000 7f4aebfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42478/0 (socket says 172.21.15.102:42478) 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.062+0000 7f4aebfff700 1 -- 172.21.15.102:0/3153215204 learned_addr learned my addr 172.21.15.102:0/3153215204 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.063+0000 7f4aebfff700 1 -- 172.21.15.102:0/3153215204 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec1065a0 con 0x7f4aec105b10 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.063+0000 7f4aebfff700 1 --2- 172.21.15.102:0/3153215204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec105f20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4adc0098a0 tx=0x7f4adc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608f6472c9db4be3 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.063+0000 7f4aeaffd700 1 -- 172.21.15.102:0/3153215204 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4adc004180 con 0x7f4aec105b10 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.063+0000 7f4aeaffd700 1 -- 172.21.15.102:0/3153215204 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4adc0042e0 con 0x7f4aec105b10 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.063+0000 7f4aeaffd700 1 -- 172.21.15.102:0/3153215204 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4adc011af0 con 0x7f4aec105b10 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.064+0000 7f4af27da700 1 -- 172.21.15.102:0/3153215204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 msgr2=0x7f4aec105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.064+0000 7f4af27da700 1 --2- 172.21.15.102:0/3153215204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec105f20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4adc0098a0 tx=0x7f4adc009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.064+0000 7f4af27da700 1 -- 172.21.15.102:0/3153215204 shutdown_connections 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.064+0000 7f4af27da700 1 --2- 172.21.15.102:0/3153215204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec105f20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.064+0000 7f4af27da700 1 -- 172.21.15.102:0/3153215204 >> 172.21.15.102:0/3153215204 conn(0x7f4aec101180 msgr2=0x7f4aec103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.064+0000 7f4af27da700 1 -- 172.21.15.102:0/3153215204 shutdown_connections 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.064+0000 7f4af27da700 1 -- 172.21.15.102:0/3153215204 wait complete. 2024-03-26T18:46:34.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.064+0000 7f4af27da700 1 Processor -- start 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.064+0000 7f4af27da700 1 -- start start 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4af27da700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4af27da700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec199dd0 con 0x7f4aec105b10 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4aebfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4aebfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42482/0 (socket says 172.21.15.102:42482) 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4aebfff700 1 -- 172.21.15.102:0/1267494067 learned_addr learned my addr 172.21.15.102:0/1267494067 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4aebfff700 1 -- 172.21.15.102:0/1267494067 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc009550 con 0x7f4aec105b10 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4aebfff700 1 --2- 172.21.15.102:0/1267494067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec199890 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4adc011ad0 tx=0x7f4adc011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4ae97fa700 1 -- 172.21.15.102:0/1267494067 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4adc003ab0 con 0x7f4aec105b10 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4ae97fa700 1 -- 172.21.15.102:0/1267494067 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4adc003c10 con 0x7f4aec105b10 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4af27da700 1 -- 172.21.15.102:0/1267494067 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aec199fd0 con 0x7f4aec105b10 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4ae97fa700 1 -- 172.21.15.102:0/1267494067 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4adc0198e0 con 0x7f4aec105b10 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.065+0000 7f4af27da700 1 -- 172.21.15.102:0/1267494067 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aec19a3f0 con 0x7f4aec105b10 2024-03-26T18:46:34.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.066+0000 7f4ae97fa700 1 -- 172.21.15.102:0/1267494067 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f4adc01a430 con 0x7f4aec105b10 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.066+0000 7f4ae97fa700 1 --2- 172.21.15.102:0/1267494067 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ad4038490 0x7f4ad403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.066+0000 7f4ae97fa700 1 -- 172.21.15.102:0/1267494067 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4adc04c250 con 0x7f4aec105b10 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.066+0000 7f4af27da700 1 -- 172.21.15.102:0/1267494067 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aec062380 con 0x7f4aec105b10 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.066+0000 7f4aeb7fe700 1 --2- 172.21.15.102:0/1267494067 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ad4038490 0x7f4ad403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.067+0000 7f4aeb7fe700 1 --2- 172.21.15.102:0/1267494067 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ad4038490 0x7f4ad403a940 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0006fd0 tx=0x7f4ae0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.071+0000 7f4ae97fa700 1 -- 172.21.15.102:0/1267494067 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4adc018ca0 con 0x7f4aec105b10 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.267+0000 7f4af27da700 1 -- 172.21.15.102:0/1267494067 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f4aec19d250 con 0x7f4aec105b10 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.268+0000 7f4ae97fa700 1 -- 172.21.15.102:0/1267494067 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f4adc010380 con 0x7f4aec105b10 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.270+0000 7f4af27da700 1 -- 172.21.15.102:0/1267494067 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ad4038490 msgr2=0x7f4ad403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.270+0000 7f4af27da700 1 --2- 172.21.15.102:0/1267494067 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ad4038490 0x7f4ad403a940 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0006fd0 tx=0x7f4ae0006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.270+0000 7f4af27da700 1 -- 172.21.15.102:0/1267494067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 msgr2=0x7f4aec199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.270+0000 7f4af27da700 1 --2- 172.21.15.102:0/1267494067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec199890 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4adc011ad0 tx=0x7f4adc011bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.270+0000 7f4af27da700 1 -- 172.21.15.102:0/1267494067 shutdown_connections 2024-03-26T18:46:34.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.270+0000 7f4af27da700 1 --2- 172.21.15.102:0/1267494067 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ad4038490 0x7f4ad403a940 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:34.133 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.270+0000 7f4af27da700 1 --2- 172.21.15.102:0/1267494067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4aec105b10 0x7f4aec199890 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:34.133 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.270+0000 7f4af27da700 1 -- 172.21.15.102:0/1267494067 >> 172.21.15.102:0/1267494067 conn(0x7f4aec101180 msgr2=0x7f4aec101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:34.133 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.270+0000 7f4af27da700 1 -- 172.21.15.102:0/1267494067 shutdown_connections 2024-03-26T18:46:34.133 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:33.270+0000 7f4af27da700 1 -- 172.21.15.102:0/1267494067 wait complete. 2024-03-26T18:46:34.136 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T18:46:34.136 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-26T18:46:34.143 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T18:46:34.143 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-26T18:46:34.143 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-26T18:46:34.143 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T18:46:34.145 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph Dashboard is now available at: 2024-03-26T18:46:34.145 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:46:34.145 INFO:teuthology.orchestra.run.smithi102.stdout: URL: https://smithi102.front.sepia.ceph.com:8443/ 2024-03-26T18:46:34.145 INFO:teuthology.orchestra.run.smithi102.stdout: User: admin 2024-03-26T18:46:34.145 INFO:teuthology.orchestra.run.smithi102.stdout: Password: mf6gtt6807 2024-03-26T18:46:34.145 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:46:34.145 INFO:teuthology.orchestra.run.smithi102.stdout:Saving cluster configuration to /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config directory 2024-03-26T18:46:34.146 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling autotune for osd_memory_target 2024-03-26T18:46:34.255 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:34 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1267494067' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T18:46:35.293 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.527+0000 7fac0e6f2700 1 Processor -- start 2024-03-26T18:46:35.293 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.527+0000 7fac0e6f2700 1 -- start start 2024-03-26T18:46:35.293 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.527+0000 7fac0e6f2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac08104880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:35.294 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.527+0000 7fac0e6f2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac08079d10 con 0x7fac08104470 2024-03-26T18:46:35.294 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.527+0000 7fac07fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac08104880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:35.294 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.527+0000 7fac07fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac08104880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42486/0 (socket says 172.21.15.102:42486) 2024-03-26T18:46:35.294 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.527+0000 7fac07fff700 1 -- 172.21.15.102:0/3530361962 learned_addr learned my addr 172.21.15.102:0/3530361962 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:35.294 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.528+0000 7fac07fff700 1 -- 172.21.15.102:0/3530361962 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac08079e50 con 0x7fac08104470 2024-03-26T18:46:35.294 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.528+0000 7fac07fff700 1 --2- 172.21.15.102:0/3530361962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac08104880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0098a0 tx=0x7fabfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76106ce108785d63 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:35.294 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.528+0000 7fac06ffd700 1 -- 172.21.15.102:0/3530361962 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabfc004180 con 0x7fac08104470 2024-03-26T18:46:35.294 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.528+0000 7fac06ffd700 1 -- 172.21.15.102:0/3530361962 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fabfc0042e0 con 0x7fac08104470 2024-03-26T18:46:35.294 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.528+0000 7fac06ffd700 1 -- 172.21.15.102:0/3530361962 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabfc011af0 con 0x7fac08104470 2024-03-26T18:46:35.294 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.529+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3530361962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 msgr2=0x7fac08104880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:35.294 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.529+0000 7fac0e6f2700 1 --2- 172.21.15.102:0/3530361962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac08104880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0098a0 tx=0x7fabfc009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.529+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3530361962 shutdown_connections 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.529+0000 7fac0e6f2700 1 --2- 172.21.15.102:0/3530361962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac08104880 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.529+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3530361962 >> 172.21.15.102:0/3530361962 conn(0x7fac08100060 msgr2=0x7fac08102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.529+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3530361962 shutdown_connections 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.529+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3530361962 wait complete. 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.529+0000 7fac0e6f2700 1 Processor -- start 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.530+0000 7fac0e6f2700 1 -- start start 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.530+0000 7fac0e6f2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac0819ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.530+0000 7fac0e6f2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0819bfd0 con 0x7fac08104470 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.530+0000 7fac07fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac0819ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.530+0000 7fac07fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac0819ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42494/0 (socket says 172.21.15.102:42494) 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.530+0000 7fac07fff700 1 -- 172.21.15.102:0/3687781486 learned_addr learned my addr 172.21.15.102:0/3687781486 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:35.295 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.530+0000 7fac07fff700 1 -- 172.21.15.102:0/3687781486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc009550 con 0x7fac08104470 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.530+0000 7fac07fff700 1 --2- 172.21.15.102:0/3687781486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac0819ba90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fabfc006ba0 tx=0x7fabfc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.530+0000 7fac057fa700 1 -- 172.21.15.102:0/3687781486 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabfc003b00 con 0x7fac08104470 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.531+0000 7fac057fa700 1 -- 172.21.15.102:0/3687781486 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fabfc01a8e0 con 0x7fac08104470 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.531+0000 7fac057fa700 1 -- 172.21.15.102:0/3687781486 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabfc019990 con 0x7fac08104470 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.531+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3687781486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0819c1d0 con 0x7fac08104470 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.531+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3687781486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0819c5f0 con 0x7fac08104470 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.532+0000 7fac057fa700 1 -- 172.21.15.102:0/3687781486 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7fabfc019c00 con 0x7fac08104470 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.532+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3687781486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac081952d0 con 0x7fac08104470 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.532+0000 7fac057fa700 1 --2- 172.21.15.102:0/3687781486 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fabf0038440 0x7fabf003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.532+0000 7fac057fa700 1 -- 172.21.15.102:0/3687781486 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fabfc04c6b0 con 0x7fac08104470 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.532+0000 7fac077fe700 1 --2- 172.21.15.102:0/3687781486 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fabf0038440 0x7fabf003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.532+0000 7fac077fe700 1 --2- 172.21.15.102:0/3687781486 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fabf0038440 0x7fabf003a8f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fabf8006fd0 tx=0x7fabf8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.537+0000 7fac057fa700 1 -- 172.21.15.102:0/3687781486 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fabfc01a430 con 0x7fac08104470 2024-03-26T18:46:35.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.727+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3687781486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fac08062380 con 0x7fac08104470 2024-03-26T18:46:35.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.727+0000 7fac057fa700 1 -- 172.21.15.102:0/3687781486 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fabfc018b40 con 0x7fac08104470 2024-03-26T18:46:35.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.729+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3687781486 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fabf0038440 msgr2=0x7fabf003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:35.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.729+0000 7fac0e6f2700 1 --2- 172.21.15.102:0/3687781486 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fabf0038440 0x7fabf003a8f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fabf8006fd0 tx=0x7fabf8006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:35.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.729+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3687781486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 msgr2=0x7fac0819ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:35.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.729+0000 7fac0e6f2700 1 --2- 172.21.15.102:0/3687781486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac0819ba90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fabfc006ba0 tx=0x7fabfc011b50 comp rx=0 tx=0).stop 2024-03-26T18:46:35.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.729+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3687781486 shutdown_connections 2024-03-26T18:46:35.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.729+0000 7fac0e6f2700 1 --2- 172.21.15.102:0/3687781486 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fabf0038440 0x7fabf003a8f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:35.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.729+0000 7fac0e6f2700 1 --2- 172.21.15.102:0/3687781486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac08104470 0x7fac0819ba90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:35.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.729+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3687781486 >> 172.21.15.102:0/3687781486 conn(0x7fac08100060 msgr2=0x7fac08190030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:35.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.729+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3687781486 shutdown_connections 2024-03-26T18:46:35.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:34.729+0000 7fac0e6f2700 1 -- 172.21.15.102:0/3687781486 wait complete. 2024-03-26T18:46:36.261 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:36 smithi102 ceph-mon[29002]: mgrmap e13: smithi102.rsfpnv(active, since 6s) 2024-03-26T18:46:36.261 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:36 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3597496830' entity='client.admin' 2024-03-26T18:46:36.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.672+0000 7f8b0860a700 1 Processor -- start 2024-03-26T18:46:36.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.672+0000 7f8b0860a700 1 -- start start 2024-03-26T18:46:36.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.672+0000 7f8b0860a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:36.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.672+0000 7f8b0860a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b00106460 con 0x7f8b00105b10 2024-03-26T18:46:36.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.672+0000 7f8b063a6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:36.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.672+0000 7f8b063a6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42508/0 (socket says 172.21.15.102:42508) 2024-03-26T18:46:36.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.672+0000 7f8b063a6700 1 -- 172.21.15.102:0/313556496 learned_addr learned my addr 172.21.15.102:0/313556496 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:36.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.673+0000 7f8b063a6700 1 -- 172.21.15.102:0/313556496 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b001065a0 con 0x7f8b00105b10 2024-03-26T18:46:36.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.673+0000 7f8b063a6700 1 --2- 172.21.15.102:0/313556496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00105f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0098a0 tx=0x7f8afc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d58ae7c5cec853e server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.673+0000 7f8b053a4700 1 -- 172.21.15.102:0/313556496 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8afc004180 con 0x7f8b00105b10 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.673+0000 7f8b053a4700 1 -- 172.21.15.102:0/313556496 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8afc0042e0 con 0x7f8b00105b10 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.673+0000 7f8b053a4700 1 -- 172.21.15.102:0/313556496 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8afc011af0 con 0x7f8b00105b10 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.673+0000 7f8b0860a700 1 -- 172.21.15.102:0/313556496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 msgr2=0x7f8b00105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.673+0000 7f8b0860a700 1 --2- 172.21.15.102:0/313556496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00105f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0098a0 tx=0x7f8afc009bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.673+0000 7f8b0860a700 1 -- 172.21.15.102:0/313556496 shutdown_connections 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.673+0000 7f8b0860a700 1 --2- 172.21.15.102:0/313556496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00105f20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.674+0000 7f8b0860a700 1 -- 172.21.15.102:0/313556496 >> 172.21.15.102:0/313556496 conn(0x7f8b00101180 msgr2=0x7f8b00103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.674+0000 7f8b0860a700 1 -- 172.21.15.102:0/313556496 shutdown_connections 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.674+0000 7f8b0860a700 1 -- 172.21.15.102:0/313556496 wait complete. 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.674+0000 7f8b0860a700 1 Processor -- start 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.674+0000 7f8b0860a700 1 -- start start 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.674+0000 7f8b0860a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.674+0000 7f8b0860a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b00199dd0 con 0x7f8b00105b10 2024-03-26T18:46:36.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.674+0000 7f8b063a6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.674+0000 7f8b063a6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42512/0 (socket says 172.21.15.102:42512) 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.674+0000 7f8b063a6700 1 -- 172.21.15.102:0/3597496830 learned_addr learned my addr 172.21.15.102:0/3597496830 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.675+0000 7f8b063a6700 1 -- 172.21.15.102:0/3597496830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc009550 con 0x7f8b00105b10 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.675+0000 7f8b063a6700 1 --2- 172.21.15.102:0/3597496830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00199890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8afc011ad0 tx=0x7f8afc011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.675+0000 7f8af77fe700 1 -- 172.21.15.102:0/3597496830 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8afc003ab0 con 0x7f8b00105b10 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.675+0000 7f8af77fe700 1 -- 172.21.15.102:0/3597496830 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8afc003c10 con 0x7f8b00105b10 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.675+0000 7f8b0860a700 1 -- 172.21.15.102:0/3597496830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b00199fd0 con 0x7f8b00105b10 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.675+0000 7f8af77fe700 1 -- 172.21.15.102:0/3597496830 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8afc0198e0 con 0x7f8b00105b10 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.675+0000 7f8b0860a700 1 -- 172.21.15.102:0/3597496830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0019a3f0 con 0x7f8b00105b10 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.676+0000 7f8af77fe700 1 -- 172.21.15.102:0/3597496830 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f8afc019bc0 con 0x7f8b00105b10 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.676+0000 7f8b0860a700 1 -- 172.21.15.102:0/3597496830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b00062380 con 0x7f8b00105b10 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.676+0000 7f8af77fe700 1 --2- 172.21.15.102:0/3597496830 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f8aec0384e0 0x7f8aec03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.676+0000 7f8af77fe700 1 -- 172.21.15.102:0/3597496830 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8afc04c3a0 con 0x7f8b00105b10 2024-03-26T18:46:36.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.676+0000 7f8b05ba5700 1 --2- 172.21.15.102:0/3597496830 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f8aec0384e0 0x7f8aec03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.676+0000 7f8b05ba5700 1 --2- 172.21.15.102:0/3597496830 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f8aec0384e0 0x7f8aec03a990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8af0006fd0 tx=0x7f8af0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.681+0000 7f8af77fe700 1 -- 172.21.15.102:0/3597496830 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8afc010380 con 0x7f8b00105b10 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.978+0000 7f8b0860a700 1 -- 172.21.15.102:0/3597496830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f8b0019d250 con 0x7f8b00105b10 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.981+0000 7f8af77fe700 1 -- 172.21.15.102:0/3597496830 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f8afc02c7b0 con 0x7f8b00105b10 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.983+0000 7f8b0860a700 1 -- 172.21.15.102:0/3597496830 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f8aec0384e0 msgr2=0x7f8aec03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.983+0000 7f8b0860a700 1 --2- 172.21.15.102:0/3597496830 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f8aec0384e0 0x7f8aec03a990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8af0006fd0 tx=0x7f8af0006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.983+0000 7f8b0860a700 1 -- 172.21.15.102:0/3597496830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 msgr2=0x7f8b00199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.983+0000 7f8b0860a700 1 --2- 172.21.15.102:0/3597496830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00199890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8afc011ad0 tx=0x7f8afc011bb0 comp rx=0 tx=0).stop 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.983+0000 7f8b0860a700 1 -- 172.21.15.102:0/3597496830 shutdown_connections 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.983+0000 7f8b0860a700 1 --2- 172.21.15.102:0/3597496830 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f8aec0384e0 0x7f8aec03a990 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.983+0000 7f8b0860a700 1 --2- 172.21.15.102:0/3597496830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00105b10 0x7f8b00199890 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.983+0000 7f8b0860a700 1 -- 172.21.15.102:0/3597496830 >> 172.21.15.102:0/3597496830 conn(0x7f8b00101180 msgr2=0x7f8b00101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.983+0000 7f8b0860a700 1 -- 172.21.15.102:0/3597496830 shutdown_connections 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-26T18:46:35.983+0000 7f8b0860a700 1 -- 172.21.15.102:0/3597496830 wait complete. 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T18:46:36.539 INFO:teuthology.orchestra.run.smithi102.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: ceph telemetry on 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout:For more information see: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:46:36.540 INFO:teuthology.orchestra.run.smithi102.stdout:Bootstrap complete. 2024-03-26T18:46:36.572 INFO:tasks.cephadm:Fetching config... 2024-03-26T18:46:36.572 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:46:36.572 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T18:46:36.591 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T18:46:36.592 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:46:36.592 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T18:46:36.646 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T18:46:36.647 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:46:36.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/keyring of=/dev/stdout 2024-03-26T18:46:36.714 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T18:46:36.714 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:46:36.714 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T18:46:36.770 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T18:46:36.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFWS+C1LDgGRKRV/YScaaQjFO9evkVm3e3Q4iW4Kd25KNaQKLwbLyJZQlMnVQPjxhsLGr0N2TMSPwKL9fj4h4jmfY014cWJECEUOqOmHb4WM8Zkmi23faeWwJrjV7poDk8SZaFBIJEFSM/glrUUa29GtG6sSWUNtD84C5FDbVY/nniLdjjqYRaitGX+kPGFjTN2+VEnuNoS/QgTZg33/5O+eBwzZFPq4ZRZENXMfKz2CYiYU2yhaRhwweTkGF8QJia+/CJJAMNpuKFvSdg2l82ty9ejFbTPqgfcvX2VQkPNapy3oJiLnsGj/RZyFJdbQHIFDb9G1c8ktxxLtj7VYiyO2ICpFnqFv6xxyQG6zEYpLlPTEKUOtSfkgI3q3L4JUAQuo6oMs4nbyM4CaXXpr78nyBVApphEN/ajmik3VBUlE+Tm6Hba4qGK9ukvpuYBBWu3MP85hgcds4IlE+e6jDmxO8qocpFPcD2tdZuOBCCijzUSW0qPG/T+cMLIT7YeCk= ceph-ed4e027c-eba0-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T18:46:36.869 INFO:teuthology.orchestra.run.smithi102.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFWS+C1LDgGRKRV/YScaaQjFO9evkVm3e3Q4iW4Kd25KNaQKLwbLyJZQlMnVQPjxhsLGr0N2TMSPwKL9fj4h4jmfY014cWJECEUOqOmHb4WM8Zkmi23faeWwJrjV7poDk8SZaFBIJEFSM/glrUUa29GtG6sSWUNtD84C5FDbVY/nniLdjjqYRaitGX+kPGFjTN2+VEnuNoS/QgTZg33/5O+eBwzZFPq4ZRZENXMfKz2CYiYU2yhaRhwweTkGF8QJia+/CJJAMNpuKFvSdg2l82ty9ejFbTPqgfcvX2VQkPNapy3oJiLnsGj/RZyFJdbQHIFDb9G1c8ktxxLtj7VYiyO2ICpFnqFv6xxyQG6zEYpLlPTEKUOtSfkgI3q3L4JUAQuo6oMs4nbyM4CaXXpr78nyBVApphEN/ajmik3VBUlE+Tm6Hba4qGK9ukvpuYBBWu3MP85hgcds4IlE+e6jDmxO8qocpFPcD2tdZuOBCCijzUSW0qPG/T+cMLIT7YeCk= ceph-ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:46:36.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFWS+C1LDgGRKRV/YScaaQjFO9evkVm3e3Q4iW4Kd25KNaQKLwbLyJZQlMnVQPjxhsLGr0N2TMSPwKL9fj4h4jmfY014cWJECEUOqOmHb4WM8Zkmi23faeWwJrjV7poDk8SZaFBIJEFSM/glrUUa29GtG6sSWUNtD84C5FDbVY/nniLdjjqYRaitGX+kPGFjTN2+VEnuNoS/QgTZg33/5O+eBwzZFPq4ZRZENXMfKz2CYiYU2yhaRhwweTkGF8QJia+/CJJAMNpuKFvSdg2l82ty9ejFbTPqgfcvX2VQkPNapy3oJiLnsGj/RZyFJdbQHIFDb9G1c8ktxxLtj7VYiyO2ICpFnqFv6xxyQG6zEYpLlPTEKUOtSfkgI3q3L4JUAQuo6oMs4nbyM4CaXXpr78nyBVApphEN/ajmik3VBUlE+Tm6Hba4qGK9ukvpuYBBWu3MP85hgcds4IlE+e6jDmxO8qocpFPcD2tdZuOBCCijzUSW0qPG/T+cMLIT7YeCk= ceph-ed4e027c-eba0-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T18:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFWS+C1LDgGRKRV/YScaaQjFO9evkVm3e3Q4iW4Kd25KNaQKLwbLyJZQlMnVQPjxhsLGr0N2TMSPwKL9fj4h4jmfY014cWJECEUOqOmHb4WM8Zkmi23faeWwJrjV7poDk8SZaFBIJEFSM/glrUUa29GtG6sSWUNtD84C5FDbVY/nniLdjjqYRaitGX+kPGFjTN2+VEnuNoS/QgTZg33/5O+eBwzZFPq4ZRZENXMfKz2CYiYU2yhaRhwweTkGF8QJia+/CJJAMNpuKFvSdg2l82ty9ejFbTPqgfcvX2VQkPNapy3oJiLnsGj/RZyFJdbQHIFDb9G1c8ktxxLtj7VYiyO2ICpFnqFv6xxyQG6zEYpLlPTEKUOtSfkgI3q3L4JUAQuo6oMs4nbyM4CaXXpr78nyBVApphEN/ajmik3VBUlE+Tm6Hba4qGK9ukvpuYBBWu3MP85hgcds4IlE+e6jDmxO8qocpFPcD2tdZuOBCCijzUSW0qPG/T+cMLIT7YeCk= ceph-ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:46:36.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T18:46:37.145 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:46:39.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.495+0000 7f5b6a59d700 1 -- 172.21.15.102:0/3840010261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b640ffc70 msgr2=0x7f5b64100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.495+0000 7f5b6a59d700 1 --2- 172.21.15.102:0/3840010261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b640ffc70 0x7f5b64100080 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009910 tx=0x7f5b60009c20 comp rx=0 tx=0).stop 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.495+0000 7f5b6a59d700 1 -- 172.21.15.102:0/3840010261 shutdown_connections 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.495+0000 7f5b6a59d700 1 --2- 172.21.15.102:0/3840010261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b640ffc70 0x7f5b64100080 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.495+0000 7f5b6a59d700 1 -- 172.21.15.102:0/3840010261 >> 172.21.15.102:0/3840010261 conn(0x7f5b640fb260 msgr2=0x7f5b640fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.496+0000 7f5b6a59d700 1 -- 172.21.15.102:0/3840010261 shutdown_connections 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.496+0000 7f5b6a59d700 1 -- 172.21.15.102:0/3840010261 wait complete. 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.496+0000 7f5b6a59d700 1 Processor -- start 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.496+0000 7f5b6a59d700 1 -- start start 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.496+0000 7f5b6a59d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b640ffc70 0x7f5b64074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.496+0000 7f5b6a59d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b64075030 con 0x7f5b640ffc70 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.496+0000 7f5b6959b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b640ffc70 0x7f5b64074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.496+0000 7f5b6959b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b640ffc70 0x7f5b64074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39466/0 (socket says 172.21.15.102:39466) 2024-03-26T18:46:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.496+0000 7f5b6959b700 1 -- 172.21.15.102:0/394020497 learned_addr learned my addr 172.21.15.102:0/394020497 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:39.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.497+0000 7f5b6959b700 1 -- 172.21.15.102:0/394020497 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b600095f0 con 0x7f5b640ffc70 2024-03-26T18:46:39.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.497+0000 7f5b6959b700 1 --2- 172.21.15.102:0/394020497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b640ffc70 0x7f5b64074af0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b60005b40 tx=0x7f5b60004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:39.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.497+0000 7f5b5a7fc700 1 -- 172.21.15.102:0/394020497 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b6001d070 con 0x7f5b640ffc70 2024-03-26T18:46:39.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.497+0000 7f5b6a59d700 1 -- 172.21.15.102:0/394020497 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b64073140 con 0x7f5b640ffc70 2024-03-26T18:46:39.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.497+0000 7f5b5a7fc700 1 -- 172.21.15.102:0/394020497 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5b6000be10 con 0x7f5b640ffc70 2024-03-26T18:46:39.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.497+0000 7f5b5a7fc700 1 -- 172.21.15.102:0/394020497 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b6000f460 con 0x7f5b640ffc70 2024-03-26T18:46:39.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.497+0000 7f5b6a59d700 1 -- 172.21.15.102:0/394020497 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b640735e0 con 0x7f5b640ffc70 2024-03-26T18:46:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.498+0000 7f5b5a7fc700 1 -- 172.21.15.102:0/394020497 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f5b6000f6d0 con 0x7f5b640ffc70 2024-03-26T18:46:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.498+0000 7f5b5a7fc700 1 --2- 172.21.15.102:0/394020497 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5b50038490 0x7f5b5003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.498+0000 7f5b5a7fc700 1 -- 172.21.15.102:0/394020497 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5b6004c6b0 con 0x7f5b640ffc70 2024-03-26T18:46:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.498+0000 7f5b6a59d700 1 -- 172.21.15.102:0/394020497 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b64062380 con 0x7f5b640ffc70 2024-03-26T18:46:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.498+0000 7f5b68d9a700 1 --2- 172.21.15.102:0/394020497 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5b50038490 0x7f5b5003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.499+0000 7f5b68d9a700 1 --2- 172.21.15.102:0/394020497 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5b50038490 0x7f5b5003a940 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5b54006fd0 tx=0x7f5b54006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.503+0000 7f5b5a7fc700 1 -- 172.21.15.102:0/394020497 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5b60027070 con 0x7f5b640ffc70 2024-03-26T18:46:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.699+0000 7f5b6a59d700 1 -- 172.21.15.102:0/394020497 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f5b6418e700 con 0x7f5b640ffc70 2024-03-26T18:46:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.702+0000 7f5b5a7fc700 1 -- 172.21.15.102:0/394020497 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f5b6002c440 con 0x7f5b640ffc70 2024-03-26T18:46:39.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.706+0000 7f5b6a59d700 1 -- 172.21.15.102:0/394020497 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5b50038490 msgr2=0x7f5b5003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:39.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.706+0000 7f5b6a59d700 1 --2- 172.21.15.102:0/394020497 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5b50038490 0x7f5b5003a940 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5b54006fd0 tx=0x7f5b54006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.706+0000 7f5b6a59d700 1 -- 172.21.15.102:0/394020497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b640ffc70 msgr2=0x7f5b64074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.706+0000 7f5b6a59d700 1 --2- 172.21.15.102:0/394020497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b640ffc70 0x7f5b64074af0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b60005b40 tx=0x7f5b60004740 comp rx=0 tx=0).stop 2024-03-26T18:46:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.706+0000 7f5b6a59d700 1 -- 172.21.15.102:0/394020497 shutdown_connections 2024-03-26T18:46:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.706+0000 7f5b6a59d700 1 --2- 172.21.15.102:0/394020497 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5b50038490 0x7f5b5003a940 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.706+0000 7f5b6a59d700 1 --2- 172.21.15.102:0/394020497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b640ffc70 0x7f5b64074af0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.706+0000 7f5b6a59d700 1 -- 172.21.15.102:0/394020497 >> 172.21.15.102:0/394020497 conn(0x7f5b640fb260 msgr2=0x7f5b640fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.706+0000 7f5b6a59d700 1 -- 172.21.15.102:0/394020497 shutdown_connections 2024-03-26T18:46:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:39.706+0000 7f5b6a59d700 1 -- 172.21.15.102:0/394020497 wait complete. 2024-03-26T18:46:40.318 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T18:46:40.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T18:46:40.874 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:40 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/394020497' entity='client.admin' 2024-03-26T18:46:40.928 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:46:42.099 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:41 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:42.099 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:41 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:42.099 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:41 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:46:42.099 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:41 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:42.099 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:41 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:46:42.099 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:41 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-26T18:46:42.099 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:41 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:46:42.099 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:41 smithi102 ceph-mon[29002]: Deploying daemon ceph-exporter.smithi102 on smithi102 2024-03-26T18:46:43.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.025+0000 7f67f3b87700 1 -- 172.21.15.102:0/2074374527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ec0fed30 msgr2=0x7f67ec0ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:43.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.025+0000 7f67f3b87700 1 --2- 172.21.15.102:0/2074374527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ec0fed30 0x7f67ec0ff140 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f67e8009910 tx=0x7f67e8009c20 comp rx=0 tx=0).stop 2024-03-26T18:46:43.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.025+0000 7f67f3b87700 1 -- 172.21.15.102:0/2074374527 shutdown_connections 2024-03-26T18:46:43.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.025+0000 7f67f3b87700 1 --2- 172.21.15.102:0/2074374527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ec0fed30 0x7f67ec0ff140 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.025+0000 7f67f3b87700 1 -- 172.21.15.102:0/2074374527 >> 172.21.15.102:0/2074374527 conn(0x7f67ec0fa380 msgr2=0x7f67ec0fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.026+0000 7f67f3b87700 1 -- 172.21.15.102:0/2074374527 shutdown_connections 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.026+0000 7f67f3b87700 1 -- 172.21.15.102:0/2074374527 wait complete. 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.026+0000 7f67f3b87700 1 Processor -- start 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.026+0000 7f67f3b87700 1 -- start start 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.026+0000 7f67f3b87700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ec0fed30 0x7f67ec194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.026+0000 7f67f3b87700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec1954b0 con 0x7f67ec0fed30 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.026+0000 7f67f1923700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ec0fed30 0x7f67ec194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.026+0000 7f67f1923700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ec0fed30 0x7f67ec194f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39488/0 (socket says 172.21.15.102:39488) 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.026+0000 7f67f1923700 1 -- 172.21.15.102:0/2679040870 learned_addr learned my addr 172.21.15.102:0/2679040870 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.027+0000 7f67f1923700 1 -- 172.21.15.102:0/2679040870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e80095f0 con 0x7f67ec0fed30 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.027+0000 7f67f1923700 1 --2- 172.21.15.102:0/2679040870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ec0fed30 0x7f67ec194f70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f67e8005b40 tx=0x7f67e8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.027+0000 7f67e2ffd700 1 -- 172.21.15.102:0/2679040870 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e801d070 con 0x7f67ec0fed30 2024-03-26T18:46:43.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.027+0000 7f67e2ffd700 1 -- 172.21.15.102:0/2679040870 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f67e800be10 con 0x7f67ec0fed30 2024-03-26T18:46:43.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.027+0000 7f67e2ffd700 1 -- 172.21.15.102:0/2679040870 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e800f460 con 0x7f67ec0fed30 2024-03-26T18:46:43.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.027+0000 7f67f3b87700 1 -- 172.21.15.102:0/2679040870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ec1956b0 con 0x7f67ec0fed30 2024-03-26T18:46:43.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.027+0000 7f67f3b87700 1 -- 172.21.15.102:0/2679040870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ec195b50 con 0x7f67ec0fed30 2024-03-26T18:46:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.028+0000 7f67e2ffd700 1 -- 172.21.15.102:0/2679040870 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f67e80227e0 con 0x7f67ec0fed30 2024-03-26T18:46:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.028+0000 7f67e2ffd700 1 --2- 172.21.15.102:0/2679040870 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f67d80384e0 0x7f67d803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.028+0000 7f67f3b87700 1 -- 172.21.15.102:0/2679040870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ec18e9a0 con 0x7f67ec0fed30 2024-03-26T18:46:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.028+0000 7f67e2ffd700 1 -- 172.21.15.102:0/2679040870 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f67e804c450 con 0x7f67ec0fed30 2024-03-26T18:46:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.028+0000 7f67f1122700 1 --2- 172.21.15.102:0/2679040870 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f67d80384e0 0x7f67d803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.029+0000 7f67f1122700 1 --2- 172.21.15.102:0/2679040870 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f67d80384e0 0x7f67d803a990 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f67dc006fd0 tx=0x7f67dc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:43.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.033+0000 7f67e2ffd700 1 -- 172.21.15.102:0/2679040870 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f67e802ba00 con 0x7f67ec0fed30 2024-03-26T18:46:43.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.230+0000 7f67f3b87700 1 -- 172.21.15.102:0/2679040870 --> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f67ec061220 con 0x7f67d80384e0 2024-03-26T18:46:43.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.234+0000 7f67e2ffd700 1 -- 172.21.15.102:0/2679040870 <== mgr.14160 v2:172.21.15.102:6800/3517043895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f67ec061220 con 0x7f67d80384e0 2024-03-26T18:46:43.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.236+0000 7f67f3b87700 1 -- 172.21.15.102:0/2679040870 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f67d80384e0 msgr2=0x7f67d803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:43.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.236+0000 7f67f3b87700 1 --2- 172.21.15.102:0/2679040870 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f67d80384e0 0x7f67d803a990 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f67dc006fd0 tx=0x7f67dc006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:43.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.236+0000 7f67f3b87700 1 -- 172.21.15.102:0/2679040870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ec0fed30 msgr2=0x7f67ec194f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:43.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.236+0000 7f67f3b87700 1 --2- 172.21.15.102:0/2679040870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ec0fed30 0x7f67ec194f70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f67e8005b40 tx=0x7f67e8004740 comp rx=0 tx=0).stop 2024-03-26T18:46:43.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.236+0000 7f67f3b87700 1 -- 172.21.15.102:0/2679040870 shutdown_connections 2024-03-26T18:46:43.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.236+0000 7f67f3b87700 1 --2- 172.21.15.102:0/2679040870 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f67d80384e0 0x7f67d803a990 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:43.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.236+0000 7f67f3b87700 1 --2- 172.21.15.102:0/2679040870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ec0fed30 0x7f67ec194f70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:43.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.236+0000 7f67f3b87700 1 -- 172.21.15.102:0/2679040870 >> 172.21.15.102:0/2679040870 conn(0x7f67ec0fa380 msgr2=0x7f67ec069100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:43.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.236+0000 7f67f3b87700 1 -- 172.21.15.102:0/2679040870 shutdown_connections 2024-03-26T18:46:43.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:43.236+0000 7f67f3b87700 1 -- 172.21.15.102:0/2679040870 wait complete. 2024-03-26T18:46:44.071 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi190 2024-03-26T18:46:44.071 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:46:44.071 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.conf 2024-03-26T18:46:44.091 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:46:44.091 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:46:44.149 INFO:tasks.cephadm:Adding host smithi190 to orchestrator... 2024-03-26T18:46:44.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph orch host add smithi190 2024-03-26T18:46:44.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:44 smithi102 ceph-mon[29002]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:44.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:44 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:44.563 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:46:45.622 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:45 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:45.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:45 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:45.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:45 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:45.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:45 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:45.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:45 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:46:45.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:45 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-26T18:46:45.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:45 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:46:45.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:45 smithi102 ceph-mon[29002]: Deploying daemon crash.smithi102 on smithi102 2024-03-26T18:46:46.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.101+0000 7fad0e5ad700 1 -- 172.21.15.102:0/3864676368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad080fed30 msgr2=0x7fad080ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:46.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.101+0000 7fad0e5ad700 1 --2- 172.21.15.102:0/3864676368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad080fed30 0x7fad080ff140 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7facfc009910 tx=0x7facfc009c20 comp rx=0 tx=0).stop 2024-03-26T18:46:46.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.101+0000 7fad0e5ad700 1 -- 172.21.15.102:0/3864676368 shutdown_connections 2024-03-26T18:46:46.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.101+0000 7fad0e5ad700 1 --2- 172.21.15.102:0/3864676368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad080fed30 0x7fad080ff140 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:46.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.101+0000 7fad0e5ad700 1 -- 172.21.15.102:0/3864676368 >> 172.21.15.102:0/3864676368 conn(0x7fad080fa2e0 msgr2=0x7fad080fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:46.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.101+0000 7fad0e5ad700 1 -- 172.21.15.102:0/3864676368 shutdown_connections 2024-03-26T18:46:46.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.101+0000 7fad0e5ad700 1 -- 172.21.15.102:0/3864676368 wait complete. 2024-03-26T18:46:46.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad0e5ad700 1 Processor -- start 2024-03-26T18:46:46.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad0e5ad700 1 -- start start 2024-03-26T18:46:46.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad0e5ad700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad080fed30 0x7fad08193ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:46.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad0e5ad700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad08194410 con 0x7fad080fed30 2024-03-26T18:46:46.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad07fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad080fed30 0x7fad08193ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:46.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad07fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad080fed30 0x7fad08193ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39524/0 (socket says 172.21.15.102:39524) 2024-03-26T18:46:46.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad07fff700 1 -- 172.21.15.102:0/4102793834 learned_addr learned my addr 172.21.15.102:0/4102793834 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad07fff700 1 -- 172.21.15.102:0/4102793834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc0095f0 con 0x7fad080fed30 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad07fff700 1 --2- 172.21.15.102:0/4102793834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad080fed30 0x7fad08193ed0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7facfc005b40 tx=0x7facfc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad057fa700 1 -- 172.21.15.102:0/4102793834 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facfc01d070 con 0x7fad080fed30 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad057fa700 1 -- 172.21.15.102:0/4102793834 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7facfc00be10 con 0x7fad080fed30 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.102+0000 7fad057fa700 1 -- 172.21.15.102:0/4102793834 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facfc00f460 con 0x7fad080fed30 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.103+0000 7fad0e5ad700 1 -- 172.21.15.102:0/4102793834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad08194610 con 0x7fad080fed30 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.103+0000 7fad0e5ad700 1 -- 172.21.15.102:0/4102793834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad08194ab0 con 0x7fad080fed30 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.104+0000 7fad057fa700 1 -- 172.21.15.102:0/4102793834 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7facfc00b960 con 0x7fad080fed30 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.104+0000 7fad0e5ad700 1 -- 172.21.15.102:0/4102793834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0818da10 con 0x7fad080fed30 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.104+0000 7fad057fa700 1 --2- 172.21.15.102:0/4102793834 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7facf0038490 0x7facf003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.104+0000 7fad057fa700 1 -- 172.21.15.102:0/4102793834 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7facfc04c3e0 con 0x7fad080fed30 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.104+0000 7fad077fe700 1 --2- 172.21.15.102:0/4102793834 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7facf0038490 0x7facf003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.104+0000 7fad077fe700 1 --2- 172.21.15.102:0/4102793834 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7facf0038490 0x7facf003a940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7facf8006fd0 tx=0x7facf8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.108+0000 7fad057fa700 1 -- 172.21.15.102:0/4102793834 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7facfc02a360 con 0x7fad080fed30 2024-03-26T18:46:46.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:46.308+0000 7fad0e5ad700 1 -- 172.21.15.102:0/4102793834 --> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}) v1 -- 0x7fad08061190 con 0x7facf0038490 2024-03-26T18:46:46.874 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:46 smithi102 ceph-mon[29002]: from='client.14187 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:46:47.858 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:47 smithi102 ceph-mon[29002]: Deploying cephadm binary to smithi190 2024-03-26T18:46:48.955 INFO:teuthology.orchestra.run.smithi102.stdout:Added host 'smithi190' with addr '172.21.15.190' 2024-03-26T18:46:48.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:48.962+0000 7fad057fa700 1 -- 172.21.15.102:0/4102793834 <== mgr.14160 v2:172.21.15.102:6800/3517043895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fad08061190 con 0x7facf0038490 2024-03-26T18:46:48.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:48.964+0000 7fad0e5ad700 1 -- 172.21.15.102:0/4102793834 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7facf0038490 msgr2=0x7facf003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:48.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:48.964+0000 7fad0e5ad700 1 --2- 172.21.15.102:0/4102793834 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7facf0038490 0x7facf003a940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7facf8006fd0 tx=0x7facf8006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:48.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:48.964+0000 7fad0e5ad700 1 -- 172.21.15.102:0/4102793834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad080fed30 msgr2=0x7fad08193ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:48.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:48.964+0000 7fad0e5ad700 1 --2- 172.21.15.102:0/4102793834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad080fed30 0x7fad08193ed0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7facfc005b40 tx=0x7facfc004740 comp rx=0 tx=0).stop 2024-03-26T18:46:48.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:48.964+0000 7fad0e5ad700 1 -- 172.21.15.102:0/4102793834 shutdown_connections 2024-03-26T18:46:48.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:48.964+0000 7fad0e5ad700 1 --2- 172.21.15.102:0/4102793834 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7facf0038490 0x7facf003a940 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:48.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:48.964+0000 7fad0e5ad700 1 --2- 172.21.15.102:0/4102793834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad080fed30 0x7fad08193ed0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:48.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:48.964+0000 7fad0e5ad700 1 -- 172.21.15.102:0/4102793834 >> 172.21.15.102:0/4102793834 conn(0x7fad080fa2e0 msgr2=0x7fad080faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:48.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:48.965+0000 7fad0e5ad700 1 -- 172.21.15.102:0/4102793834 shutdown_connections 2024-03-26T18:46:48.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:48.965+0000 7fad0e5ad700 1 -- 172.21.15.102:0/4102793834 wait complete. 2024-03-26T18:46:49.248 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:49 smithi102 ceph-mon[29002]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:46:49.248 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:49 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:49.248 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:49 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:49.248 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:49 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:49.248 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:49 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:49.249 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:49 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:49.249 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:49 smithi102 ceph-mon[29002]: Deploying daemon node-exporter.smithi102 on smithi102 2024-03-26T18:46:49.249 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:49 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:49.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T18:46:49.784 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:46:50.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:50 smithi102 ceph-mon[29002]: Added host smithi190 2024-03-26T18:46:51.370 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:51 smithi102 ceph-mon[29002]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:46:52.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.288+0000 7f5440d2e700 1 -- 172.21.15.102:0/2632553096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f543c100340 msgr2=0x7f543c100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.288+0000 7f5440d2e700 1 --2- 172.21.15.102:0/2632553096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f543c100340 0x7f543c100750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f542c009910 tx=0x7f542c009c20 comp rx=0 tx=0).stop 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.288+0000 7f5440d2e700 1 -- 172.21.15.102:0/2632553096 shutdown_connections 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.288+0000 7f5440d2e700 1 --2- 172.21.15.102:0/2632553096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f543c100340 0x7f543c100750 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.288+0000 7f5440d2e700 1 -- 172.21.15.102:0/2632553096 >> 172.21.15.102:0/2632553096 conn(0x7f543c0f9a60 msgr2=0x7f543c0fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.288+0000 7f5440d2e700 1 -- 172.21.15.102:0/2632553096 shutdown_connections 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.288+0000 7f5440d2e700 1 -- 172.21.15.102:0/2632553096 wait complete. 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.289+0000 7f5440d2e700 1 Processor -- start 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.289+0000 7f5440d2e700 1 -- start start 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.289+0000 7f5440d2e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f543c100340 0x7f543c0fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.289+0000 7f5440d2e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c0fbf30 con 0x7f543c100340 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.289+0000 7f543a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f543c100340 0x7f543c0fb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.289+0000 7f543a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f543c100340 0x7f543c0fb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47504/0 (socket says 172.21.15.102:47504) 2024-03-26T18:46:52.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.289+0000 7f543a59c700 1 -- 172.21.15.102:0/2119466925 learned_addr learned my addr 172.21.15.102:0/2119466925 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.289+0000 7f543a59c700 1 -- 172.21.15.102:0/2119466925 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c0095f0 con 0x7f543c100340 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.289+0000 7f543a59c700 1 --2- 172.21.15.102:0/2119466925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f543c100340 0x7f543c0fb9f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f542c005950 tx=0x7f542c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.290+0000 7f542b7fe700 1 -- 172.21.15.102:0/2119466925 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f542c01d070 con 0x7f543c100340 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.290+0000 7f542b7fe700 1 -- 172.21.15.102:0/2119466925 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f542c00be10 con 0x7f543c100340 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.290+0000 7f5440d2e700 1 -- 172.21.15.102:0/2119466925 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543c0fa040 con 0x7f543c100340 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.290+0000 7f542b7fe700 1 -- 172.21.15.102:0/2119466925 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f542c00f460 con 0x7f543c100340 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.290+0000 7f5440d2e700 1 -- 172.21.15.102:0/2119466925 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543c0fa4e0 con 0x7f543c100340 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.291+0000 7f542b7fe700 1 -- 172.21.15.102:0/2119466925 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f542c00f690 con 0x7f543c100340 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.291+0000 7f5440d2e700 1 -- 172.21.15.102:0/2119466925 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543c062380 con 0x7f543c100340 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.291+0000 7f542b7fe700 1 --2- 172.21.15.102:0/2119466925 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5424038490 0x7f542403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.291+0000 7f542b7fe700 1 -- 172.21.15.102:0/2119466925 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f542c04c430 con 0x7f543c100340 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.291+0000 7f5439d9b700 1 --2- 172.21.15.102:0/2119466925 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5424038490 0x7f542403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.291+0000 7f5439d9b700 1 --2- 172.21.15.102:0/2119466925 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5424038490 0x7f542403a940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5430006fd0 tx=0x7f5430006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.296+0000 7f542b7fe700 1 -- 172.21.15.102:0/2119466925 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f542c02a360 con 0x7f543c100340 2024-03-26T18:46:52.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.497+0000 7f5440d2e700 1 -- 172.21.15.102:0/2119466925 --> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f543c0fa720 con 0x7f5424038490 2024-03-26T18:46:52.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.499+0000 7f542b7fe700 1 -- 172.21.15.102:0/2119466925 <== mgr.14160 v2:172.21.15.102:6800/3517043895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f543c0fa720 con 0x7f5424038490 2024-03-26T18:46:52.490 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:46:52.490 INFO:teuthology.orchestra.run.smithi102.stdout:[{"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}, {"addr": "172.21.15.190", "hostname": "smithi190", "labels": [], "status": ""}] 2024-03-26T18:46:52.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.501+0000 7f5440d2e700 1 -- 172.21.15.102:0/2119466925 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5424038490 msgr2=0x7f542403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:52.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.501+0000 7f5440d2e700 1 --2- 172.21.15.102:0/2119466925 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5424038490 0x7f542403a940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5430006fd0 tx=0x7f5430006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:52.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.501+0000 7f5440d2e700 1 -- 172.21.15.102:0/2119466925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f543c100340 msgr2=0x7f543c0fb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:52.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.501+0000 7f5440d2e700 1 --2- 172.21.15.102:0/2119466925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f543c100340 0x7f543c0fb9f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f542c005950 tx=0x7f542c009e60 comp rx=0 tx=0).stop 2024-03-26T18:46:52.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.501+0000 7f5440d2e700 1 -- 172.21.15.102:0/2119466925 shutdown_connections 2024-03-26T18:46:52.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.501+0000 7f5440d2e700 1 --2- 172.21.15.102:0/2119466925 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5424038490 0x7f542403a940 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:52.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.501+0000 7f5440d2e700 1 --2- 172.21.15.102:0/2119466925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f543c100340 0x7f543c0fb9f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:52.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.501+0000 7f5440d2e700 1 -- 172.21.15.102:0/2119466925 >> 172.21.15.102:0/2119466925 conn(0x7f543c0f9a60 msgr2=0x7f543c0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:52.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.501+0000 7f5440d2e700 1 -- 172.21.15.102:0/2119466925 shutdown_connections 2024-03-26T18:46:52.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:52.501+0000 7f5440d2e700 1 -- 172.21.15.102:0/2119466925 wait complete. 2024-03-26T18:46:53.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:53 smithi102 ceph-mon[29002]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:46:53.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:53 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:53.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:53 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:53.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:53 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:53.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:53 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:53.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:53 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:46:53.496 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T18:46:53.496 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T18:46:53.716 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:46:54.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:54 smithi102 ceph-mon[29002]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T18:46:54.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:54 smithi102 ceph-mon[29002]: Deploying daemon alertmanager.smithi102 on smithi102 2024-03-26T18:46:55.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:55 smithi102 ceph-mon[29002]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:46:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.584+0000 7f5cb9906700 1 -- 172.21.15.102:0/2749126149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb40fed30 msgr2=0x7f5cb40ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.584+0000 7f5cb9906700 1 --2- 172.21.15.102:0/2749126149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb40fed30 0x7f5cb40ff140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009910 tx=0x7f5ca4009c20 comp rx=0 tx=0).stop 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.585+0000 7f5cb9906700 1 -- 172.21.15.102:0/2749126149 shutdown_connections 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.585+0000 7f5cb9906700 1 --2- 172.21.15.102:0/2749126149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb40fed30 0x7f5cb40ff140 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.585+0000 7f5cb9906700 1 -- 172.21.15.102:0/2749126149 >> 172.21.15.102:0/2749126149 conn(0x7f5cb40fa2e0 msgr2=0x7f5cb40fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.585+0000 7f5cb9906700 1 -- 172.21.15.102:0/2749126149 shutdown_connections 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.585+0000 7f5cb9906700 1 -- 172.21.15.102:0/2749126149 wait complete. 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.585+0000 7f5cb9906700 1 Processor -- start 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.585+0000 7f5cb9906700 1 -- start start 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.585+0000 7f5cb9906700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb40fed30 0x7f5cb4194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.585+0000 7f5cb9906700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb4195500 con 0x7f5cb40fed30 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.586+0000 7f5cb2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb40fed30 0x7f5cb4194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.586+0000 7f5cb2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb40fed30 0x7f5cb4194fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47514/0 (socket says 172.21.15.102:47514) 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.586+0000 7f5cb2ffd700 1 -- 172.21.15.102:0/602795811 learned_addr learned my addr 172.21.15.102:0/602795811 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:46:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.586+0000 7f5cb2ffd700 1 -- 172.21.15.102:0/602795811 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca40095f0 con 0x7f5cb40fed30 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.586+0000 7f5cb2ffd700 1 --2- 172.21.15.102:0/602795811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb40fed30 0x7f5cb4194fc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4005b40 tx=0x7f5ca4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.586+0000 7f5cb8904700 1 -- 172.21.15.102:0/602795811 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca401d070 con 0x7f5cb40fed30 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.586+0000 7f5cb9906700 1 -- 172.21.15.102:0/602795811 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb4195700 con 0x7f5cb40fed30 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.586+0000 7f5cb8904700 1 -- 172.21.15.102:0/602795811 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5ca400be10 con 0x7f5cb40fed30 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.586+0000 7f5cb8904700 1 -- 172.21.15.102:0/602795811 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca400f460 con 0x7f5cb40fed30 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.586+0000 7f5cb9906700 1 -- 172.21.15.102:0/602795811 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb4195ba0 con 0x7f5cb40fed30 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.587+0000 7f5cb8904700 1 -- 172.21.15.102:0/602795811 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f5ca400f6d0 con 0x7f5cb40fed30 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.587+0000 7f5cb9906700 1 -- 172.21.15.102:0/602795811 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb418ea20 con 0x7f5cb40fed30 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.587+0000 7f5cb8904700 1 --2- 172.21.15.102:0/602795811 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5ca0038490 0x7f5ca003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.587+0000 7f5cb8904700 1 -- 172.21.15.102:0/602795811 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5ca404c690 con 0x7f5cb40fed30 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.587+0000 7f5caa7fc700 1 --2- 172.21.15.102:0/602795811 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5ca0038490 0x7f5ca003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.587+0000 7f5caa7fc700 1 --2- 172.21.15.102:0/602795811 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5ca0038490 0x7f5ca003a940 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c006fd0 tx=0x7f5c9c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.592+0000 7f5cb8904700 1 -- 172.21.15.102:0/602795811 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5ca402a360 con 0x7f5cb40fed30 2024-03-26T18:46:55.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:55.796+0000 7f5cb9906700 1 -- 172.21.15.102:0/602795811 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5cb402cc30 con 0x7f5cb40fed30 2024-03-26T18:46:56.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:56.127+0000 7f5cb8904700 1 -- 172.21.15.102:0/602795811 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5ca4027070 con 0x7f5cb40fed30 2024-03-26T18:46:56.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:56.129+0000 7f5cb9906700 1 -- 172.21.15.102:0/602795811 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5ca0038490 msgr2=0x7f5ca003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:56.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:56.129+0000 7f5cb9906700 1 --2- 172.21.15.102:0/602795811 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5ca0038490 0x7f5ca003a940 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c006fd0 tx=0x7f5c9c006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:56.129+0000 7f5cb9906700 1 -- 172.21.15.102:0/602795811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb40fed30 msgr2=0x7f5cb4194fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:56.129+0000 7f5cb9906700 1 --2- 172.21.15.102:0/602795811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb40fed30 0x7f5cb4194fc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4005b40 tx=0x7f5ca4004740 comp rx=0 tx=0).stop 2024-03-26T18:46:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:56.129+0000 7f5cb9906700 1 -- 172.21.15.102:0/602795811 shutdown_connections 2024-03-26T18:46:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:56.129+0000 7f5cb9906700 1 --2- 172.21.15.102:0/602795811 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f5ca0038490 0x7f5ca003a940 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:56.129+0000 7f5cb9906700 1 --2- 172.21.15.102:0/602795811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb40fed30 0x7f5cb4194fc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:56.129+0000 7f5cb9906700 1 -- 172.21.15.102:0/602795811 >> 172.21.15.102:0/602795811 conn(0x7f5cb40fa2e0 msgr2=0x7f5cb40690c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:56.129+0000 7f5cb9906700 1 -- 172.21.15.102:0/602795811 shutdown_connections 2024-03-26T18:46:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:46:56.129+0000 7f5cb9906700 1 -- 172.21.15.102:0/602795811 wait complete. 2024-03-26T18:46:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2024-03-26T18:46:56.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:56 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/602795811' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T18:46:57.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:57 smithi102 ceph-mon[29002]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:46:57.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:57 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/602795811' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T18:46:57.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:57 smithi102 ceph-mon[29002]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T18:46:57.571 INFO:tasks.cephadm:Adding mon.smithi102 on smithi102 2024-03-26T18:46:57.571 INFO:tasks.cephadm:Adding mon.smithi190 on smithi190 2024-03-26T18:46:57.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph orch apply mon '2;smithi102:172.21.15.102=smithi102;smithi190:172.21.15.190=smithi190' 2024-03-26T18:46:57.771 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:46:57.817 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:46:59.279 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:46:59 smithi102 ceph-mon[29002]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.289+0000 7fd36d131700 1 -- 172.21.15.190:0/1457047336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3680fed30 msgr2=0x7fd3680ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.289+0000 7fd36d131700 1 --2- 172.21.15.190:0/1457047336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3680fed30 0x7fd3680ff140 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd358009910 tx=0x7fd358009c20 comp rx=0 tx=0).stop 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.289+0000 7fd36d131700 1 -- 172.21.15.190:0/1457047336 shutdown_connections 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.289+0000 7fd36d131700 1 --2- 172.21.15.190:0/1457047336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3680fed30 0x7fd3680ff140 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.289+0000 7fd36d131700 1 -- 172.21.15.190:0/1457047336 >> 172.21.15.190:0/1457047336 conn(0x7fd3680fa2e0 msgr2=0x7fd3680fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.289+0000 7fd36d131700 1 -- 172.21.15.190:0/1457047336 shutdown_connections 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.289+0000 7fd36d131700 1 -- 172.21.15.190:0/1457047336 wait complete. 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.290+0000 7fd36d131700 1 Processor -- start 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.290+0000 7fd36d131700 1 -- start start 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.290+0000 7fd36d131700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3680fed30 0x7fd368194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.290+0000 7fd36d131700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368195500 con 0x7fd3680fed30 2024-03-26T18:46:59.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.290+0000 7fd366d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3680fed30 0x7fd368194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:59.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.290+0000 7fd366d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3680fed30 0x7fd368194fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:38288/0 (socket says 172.21.15.190:38288) 2024-03-26T18:46:59.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.290+0000 7fd366d9d700 1 -- 172.21.15.190:0/4233220575 learned_addr learned my addr 172.21.15.190:0/4233220575 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:46:59.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.291+0000 7fd366d9d700 1 -- 172.21.15.190:0/4233220575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3580095f0 con 0x7fd3680fed30 2024-03-26T18:46:59.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.291+0000 7fd366d9d700 1 --2- 172.21.15.190:0/4233220575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3680fed30 0x7fd368194fc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd358005b40 tx=0x7fd358004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:59.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.291+0000 7fd34ffff700 1 -- 172.21.15.190:0/4233220575 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd35801d070 con 0x7fd3680fed30 2024-03-26T18:46:59.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.291+0000 7fd36d131700 1 -- 172.21.15.190:0/4233220575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd368195700 con 0x7fd3680fed30 2024-03-26T18:46:59.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.291+0000 7fd34ffff700 1 -- 172.21.15.190:0/4233220575 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd35800be10 con 0x7fd3680fed30 2024-03-26T18:46:59.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.291+0000 7fd34ffff700 1 -- 172.21.15.190:0/4233220575 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd35800f460 con 0x7fd3680fed30 2024-03-26T18:46:59.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.291+0000 7fd36d131700 1 -- 172.21.15.190:0/4233220575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd368195ba0 con 0x7fd3680fed30 2024-03-26T18:46:59.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.292+0000 7fd34ffff700 1 -- 172.21.15.190:0/4233220575 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7fd35800f6d0 con 0x7fd3680fed30 2024-03-26T18:46:59.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.292+0000 7fd36d131700 1 -- 172.21.15.190:0/4233220575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd36818ea20 con 0x7fd3680fed30 2024-03-26T18:46:59.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.292+0000 7fd34ffff700 1 --2- 172.21.15.190:0/4233220575 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fd350038490 0x7fd35003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:46:59.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.292+0000 7fd34ffff700 1 -- 172.21.15.190:0/4233220575 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd35804c710 con 0x7fd3680fed30 2024-03-26T18:46:59.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.292+0000 7fd36659c700 1 --2- 172.21.15.190:0/4233220575 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fd350038490 0x7fd35003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:46:59.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.293+0000 7fd36659c700 1 --2- 172.21.15.190:0/4233220575 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fd350038490 0x7fd35003a940 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd35c006fd0 tx=0x7fd35c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:46:59.288 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.298+0000 7fd34ffff700 1 -- 172.21.15.190:0/4233220575 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd35802a360 con 0x7fd3680fed30 2024-03-26T18:46:59.486 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.495+0000 7fd36d131700 1 -- 172.21.15.190:0/4233220575 --> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi102:172.21.15.102=smithi102;smithi190:172.21.15.190=smithi190", "target": ["mon-mgr", ""]}) v1 -- 0x7fd368061300 con 0x7fd350038490 2024-03-26T18:46:59.493 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.502+0000 7fd34ffff700 1 -- 172.21.15.190:0/4233220575 <== mgr.14160 v2:172.21.15.102:6800/3517043895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd368061300 con 0x7fd350038490 2024-03-26T18:46:59.493 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mon update... 2024-03-26T18:46:59.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.503+0000 7fd36d131700 1 -- 172.21.15.190:0/4233220575 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fd350038490 msgr2=0x7fd35003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:59.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.503+0000 7fd36d131700 1 --2- 172.21.15.190:0/4233220575 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fd350038490 0x7fd35003a940 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd35c006fd0 tx=0x7fd35c006e40 comp rx=0 tx=0).stop 2024-03-26T18:46:59.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.503+0000 7fd36d131700 1 -- 172.21.15.190:0/4233220575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3680fed30 msgr2=0x7fd368194fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:46:59.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.503+0000 7fd36d131700 1 --2- 172.21.15.190:0/4233220575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3680fed30 0x7fd368194fc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd358005b40 tx=0x7fd358004740 comp rx=0 tx=0).stop 2024-03-26T18:46:59.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.503+0000 7fd36d131700 1 -- 172.21.15.190:0/4233220575 shutdown_connections 2024-03-26T18:46:59.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.503+0000 7fd36d131700 1 --2- 172.21.15.190:0/4233220575 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7fd350038490 0x7fd35003a940 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:59.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.503+0000 7fd36d131700 1 --2- 172.21.15.190:0/4233220575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3680fed30 0x7fd368194fc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:46:59.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.504+0000 7fd36d131700 1 -- 172.21.15.190:0/4233220575 >> 172.21.15.190:0/4233220575 conn(0x7fd3680fa2e0 msgr2=0x7fd3680687f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:46:59.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.504+0000 7fd36d131700 1 -- 172.21.15.190:0/4233220575 shutdown_connections 2024-03-26T18:46:59.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:46:59.504+0000 7fd36d131700 1 -- 172.21.15.190:0/4233220575 wait complete. 2024-03-26T18:47:00.071 DEBUG:teuthology.orchestra.run.smithi190:mon.smithi190> sudo journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi190.service 2024-03-26T18:47:00.073 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:00.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:00.298 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:00.342 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:00.590 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:00 smithi102 ceph-mon[29002]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi102:172.21.15.102=smithi102;smithi190:172.21.15.190=smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:47:00.590 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:00 smithi102 ceph-mon[29002]: Saving service mon spec with placement smithi102:172.21.15.102=smithi102;smithi190:172.21.15.190=smithi190;count:2 2024-03-26T18:47:00.590 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:00 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:00.590 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:00 smithi102 ceph-mon[29002]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:01.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.799+0000 7f4cc3728700 1 -- 172.21.15.190:0/1580572824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cbc0ffe40 msgr2=0x7f4cbc100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:01.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.799+0000 7f4cc3728700 1 --2- 172.21.15.190:0/1580572824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cbc0ffe40 0x7f4cbc100250 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009910 tx=0x7f4cb4009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:01.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.799+0000 7f4cc3728700 1 -- 172.21.15.190:0/1580572824 shutdown_connections 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.799+0000 7f4cc3728700 1 --2- 172.21.15.190:0/1580572824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cbc0ffe40 0x7f4cbc100250 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.799+0000 7f4cc3728700 1 -- 172.21.15.190:0/1580572824 >> 172.21.15.190:0/1580572824 conn(0x7f4cbc0fb3d0 msgr2=0x7f4cbc0fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.800+0000 7f4cc3728700 1 -- 172.21.15.190:0/1580572824 shutdown_connections 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.800+0000 7f4cc3728700 1 -- 172.21.15.190:0/1580572824 wait complete. 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.800+0000 7f4cc3728700 1 Processor -- start 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.800+0000 7f4cc3728700 1 -- start start 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.800+0000 7f4cc3728700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cbc0ffe40 0x7f4cbc194f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.800+0000 7f4cc3728700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc1954d0 con 0x7f4cbc0ffe40 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.800+0000 7f4cc14c4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cbc0ffe40 0x7f4cbc194f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.800+0000 7f4cc14c4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cbc0ffe40 0x7f4cbc194f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:38302/0 (socket says 172.21.15.190:38302) 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.800+0000 7f4cc14c4700 1 -- 172.21.15.190:0/3001326618 learned_addr learned my addr 172.21.15.190:0/3001326618 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.801+0000 7f4cc14c4700 1 -- 172.21.15.190:0/3001326618 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb40095f0 con 0x7f4cbc0ffe40 2024-03-26T18:47:01.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.801+0000 7f4cc14c4700 1 --2- 172.21.15.190:0/3001326618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cbc0ffe40 0x7f4cbc194f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4005b40 tx=0x7f4cb4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:01.795 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.801+0000 7f4cba7fc700 1 -- 172.21.15.190:0/3001326618 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cb401d070 con 0x7f4cbc0ffe40 2024-03-26T18:47:01.795 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.801+0000 7f4cc3728700 1 -- 172.21.15.190:0/3001326618 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cbc1956d0 con 0x7f4cbc0ffe40 2024-03-26T18:47:01.795 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.801+0000 7f4cba7fc700 1 -- 172.21.15.190:0/3001326618 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4cb400be10 con 0x7f4cbc0ffe40 2024-03-26T18:47:01.795 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.801+0000 7f4cba7fc700 1 -- 172.21.15.190:0/3001326618 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cb400f460 con 0x7f4cbc0ffe40 2024-03-26T18:47:01.795 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.801+0000 7f4cc3728700 1 -- 172.21.15.190:0/3001326618 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc195b70 con 0x7f4cbc0ffe40 2024-03-26T18:47:01.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.802+0000 7f4cba7fc700 1 -- 172.21.15.190:0/3001326618 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f4cb400b960 con 0x7f4cbc0ffe40 2024-03-26T18:47:01.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.802+0000 7f4cc3728700 1 -- 172.21.15.190:0/3001326618 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cbc18eaa0 con 0x7f4cbc0ffe40 2024-03-26T18:47:01.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.802+0000 7f4cba7fc700 1 --2- 172.21.15.190:0/3001326618 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ca4038490 0x7f4ca403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:01.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.802+0000 7f4cba7fc700 1 -- 172.21.15.190:0/3001326618 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4cb404c340 con 0x7f4cbc0ffe40 2024-03-26T18:47:01.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.802+0000 7f4cc0cc3700 1 --2- 172.21.15.190:0/3001326618 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ca4038490 0x7f4ca403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:01.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.803+0000 7f4cc0cc3700 1 --2- 172.21.15.190:0/3001326618 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ca4038490 0x7f4ca403a940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0006fd0 tx=0x7f4cb0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:01.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:01.811+0000 7f4cba7fc700 1 -- 172.21.15.190:0/3001326618 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4cb402bba0 con 0x7f4cbc0ffe40 2024-03-26T18:47:02.067 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.075+0000 7f4cc3728700 1 -- 172.21.15.190:0/3001326618 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4cbc02cc30 con 0x7f4cbc0ffe40 2024-03-26T18:47:02.067 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.075+0000 7f4cba7fc700 1 -- 172.21.15.190:0/3001326618 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f4cb4027070 con 0x7f4cbc0ffe40 2024-03-26T18:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:02.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.077+0000 7f4cc3728700 1 -- 172.21.15.190:0/3001326618 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ca4038490 msgr2=0x7f4ca403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:02.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.077+0000 7f4cc3728700 1 --2- 172.21.15.190:0/3001326618 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ca4038490 0x7f4ca403a940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0006fd0 tx=0x7f4cb0006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:02.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.077+0000 7f4cc3728700 1 -- 172.21.15.190:0/3001326618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cbc0ffe40 msgr2=0x7f4cbc194f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:02.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.077+0000 7f4cc3728700 1 --2- 172.21.15.190:0/3001326618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cbc0ffe40 0x7f4cbc194f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4005b40 tx=0x7f4cb4004740 comp rx=0 tx=0).stop 2024-03-26T18:47:02.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.077+0000 7f4cc3728700 1 -- 172.21.15.190:0/3001326618 shutdown_connections 2024-03-26T18:47:02.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.077+0000 7f4cc3728700 1 --2- 172.21.15.190:0/3001326618 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f4ca4038490 0x7f4ca403a940 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:02.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.077+0000 7f4cc3728700 1 --2- 172.21.15.190:0/3001326618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cbc0ffe40 0x7f4cbc194f90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:02.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.077+0000 7f4cc3728700 1 -- 172.21.15.190:0/3001326618 >> 172.21.15.190:0/3001326618 conn(0x7f4cbc0fb3d0 msgr2=0x7f4cbc0fc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:02.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.078+0000 7f4cc3728700 1 -- 172.21.15.190:0/3001326618 shutdown_connections 2024-03-26T18:47:02.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:02.078+0000 7f4cc3728700 1 -- 172.21.15.190:0/3001326618 wait complete. 2024-03-26T18:47:02.071 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:03.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:03.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:03.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:03.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:03.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T18:47:03.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:03.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:03.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T18:47:03.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T18:47:03.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:03.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: Deploying daemon grafana.smithi102 on smithi102 2024-03-26T18:47:03.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:03.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:02 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/3001326618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:03.654 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:03.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:03.855 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:03.901 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:04.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:04 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:05.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.337+0000 7f9188ba8700 1 -- 172.21.15.190:0/907711241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9184100920 msgr2=0x7f9184102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:05.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.337+0000 7f9188ba8700 1 --2- 172.21.15.190:0/907711241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9184100920 0x7f9184102d00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9174009910 tx=0x7f9174009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:05.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.338+0000 7f9188ba8700 1 -- 172.21.15.190:0/907711241 shutdown_connections 2024-03-26T18:47:05.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.338+0000 7f9188ba8700 1 --2- 172.21.15.190:0/907711241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9184100920 0x7f9184102d00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:05.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.338+0000 7f9188ba8700 1 -- 172.21.15.190:0/907711241 >> 172.21.15.190:0/907711241 conn(0x7f91840fa2e0 msgr2=0x7f91840fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:05.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.338+0000 7f9188ba8700 1 -- 172.21.15.190:0/907711241 shutdown_connections 2024-03-26T18:47:05.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.338+0000 7f9188ba8700 1 -- 172.21.15.190:0/907711241 wait complete. 2024-03-26T18:47:05.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.338+0000 7f9188ba8700 1 Processor -- start 2024-03-26T18:47:05.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.338+0000 7f9188ba8700 1 -- start start 2024-03-26T18:47:05.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.339+0000 7f9188ba8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9184100920 0x7f9184194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:05.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.339+0000 7f9188ba8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91841954b0 con 0x7f9184100920 2024-03-26T18:47:05.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.339+0000 7f918259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9184100920 0x7f9184194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:05.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.339+0000 7f918259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9184100920 0x7f9184194f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:49764/0 (socket says 172.21.15.190:49764) 2024-03-26T18:47:05.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.339+0000 7f918259c700 1 -- 172.21.15.190:0/964311405 learned_addr learned my addr 172.21.15.190:0/964311405 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:05.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.339+0000 7f918259c700 1 -- 172.21.15.190:0/964311405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91740095f0 con 0x7f9184100920 2024-03-26T18:47:05.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.339+0000 7f918259c700 1 --2- 172.21.15.190:0/964311405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9184100920 0x7f9184194f70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9174005b40 tx=0x7f9174004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:05.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.339+0000 7f91737fe700 1 -- 172.21.15.190:0/964311405 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f917401d070 con 0x7f9184100920 2024-03-26T18:47:05.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.339+0000 7f91737fe700 1 -- 172.21.15.190:0/964311405 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f917400be10 con 0x7f9184100920 2024-03-26T18:47:05.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.339+0000 7f9188ba8700 1 -- 172.21.15.190:0/964311405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91841956b0 con 0x7f9184100920 2024-03-26T18:47:05.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.339+0000 7f91737fe700 1 -- 172.21.15.190:0/964311405 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f917400f460 con 0x7f9184100920 2024-03-26T18:47:05.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.340+0000 7f9188ba8700 1 -- 172.21.15.190:0/964311405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9184195b50 con 0x7f9184100920 2024-03-26T18:47:05.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.340+0000 7f91737fe700 1 -- 172.21.15.190:0/964311405 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f91740227e0 con 0x7f9184100920 2024-03-26T18:47:05.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.340+0000 7f9188ba8700 1 -- 172.21.15.190:0/964311405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918418eb10 con 0x7f9184100920 2024-03-26T18:47:05.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.341+0000 7f91737fe700 1 --2- 172.21.15.190:0/964311405 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f916c038490 0x7f916c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:05.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.341+0000 7f91737fe700 1 -- 172.21.15.190:0/964311405 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f917404c4f0 con 0x7f9184100920 2024-03-26T18:47:05.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.341+0000 7f9181d9b700 1 --2- 172.21.15.190:0/964311405 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f916c038490 0x7f916c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:05.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.341+0000 7f9181d9b700 1 --2- 172.21.15.190:0/964311405 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f916c038490 0x7f916c03a940 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9178006fd0 tx=0x7f9178006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:05.336 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.345+0000 7f91737fe700 1 -- 172.21.15.190:0/964311405 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f917402bba0 con 0x7f9184100920 2024-03-26T18:47:05.373 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:05 smithi102 ceph-mon[29002]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:05.602 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.611+0000 7f9188ba8700 1 -- 172.21.15.190:0/964311405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f918402cc30 con 0x7f9184100920 2024-03-26T18:47:05.602 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.611+0000 7f91737fe700 1 -- 172.21.15.190:0/964311405 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f9174027070 con 0x7f9184100920 2024-03-26T18:47:05.602 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:05.602 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:05.604 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.613+0000 7f9188ba8700 1 -- 172.21.15.190:0/964311405 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f916c038490 msgr2=0x7f916c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:05.604 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.613+0000 7f9188ba8700 1 --2- 172.21.15.190:0/964311405 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f916c038490 0x7f916c03a940 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9178006fd0 tx=0x7f9178006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:05.604 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.613+0000 7f9188ba8700 1 -- 172.21.15.190:0/964311405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9184100920 msgr2=0x7f9184194f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:05.604 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.613+0000 7f9188ba8700 1 --2- 172.21.15.190:0/964311405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9184100920 0x7f9184194f70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9174005b40 tx=0x7f9174004740 comp rx=0 tx=0).stop 2024-03-26T18:47:05.604 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.613+0000 7f9188ba8700 1 -- 172.21.15.190:0/964311405 shutdown_connections 2024-03-26T18:47:05.604 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.613+0000 7f9188ba8700 1 --2- 172.21.15.190:0/964311405 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f916c038490 0x7f916c03a940 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:05.605 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.613+0000 7f9188ba8700 1 --2- 172.21.15.190:0/964311405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9184100920 0x7f9184194f70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:05.605 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.613+0000 7f9188ba8700 1 -- 172.21.15.190:0/964311405 >> 172.21.15.190:0/964311405 conn(0x7f91840fa2e0 msgr2=0x7f91840faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:05.605 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.613+0000 7f9188ba8700 1 -- 172.21.15.190:0/964311405 shutdown_connections 2024-03-26T18:47:05.605 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:05.613+0000 7f9188ba8700 1 -- 172.21.15.190:0/964311405 wait complete. 2024-03-26T18:47:05.605 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:06.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:06 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/964311405' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:07.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:07.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:07.363 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:07.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:07 smithi102 ceph-mon[29002]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:07.409 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:08.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.885+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3083836298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ec0ffe40 msgr2=0x7f13ec100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:08.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.885+0000 7f13f1d8a700 1 --2- 172.21.15.190:0/3083836298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ec0ffe40 0x7f13ec100250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009910 tx=0x7f13dc009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:08.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.885+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3083836298 shutdown_connections 2024-03-26T18:47:08.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.885+0000 7f13f1d8a700 1 --2- 172.21.15.190:0/3083836298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ec0ffe40 0x7f13ec100250 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:08.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.885+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3083836298 >> 172.21.15.190:0/3083836298 conn(0x7f13ec0fb3f0 msgr2=0x7f13ec0fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:08.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.885+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3083836298 shutdown_connections 2024-03-26T18:47:08.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.885+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3083836298 wait complete. 2024-03-26T18:47:08.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.886+0000 7f13f1d8a700 1 Processor -- start 2024-03-26T18:47:08.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.886+0000 7f13f1d8a700 1 -- start start 2024-03-26T18:47:08.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.886+0000 7f13f1d8a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ec0ffe40 0x7f13ec194fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:08.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.886+0000 7f13f1d8a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec1954e0 con 0x7f13ec0ffe40 2024-03-26T18:47:08.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.886+0000 7f13eb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ec0ffe40 0x7f13ec194fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:08.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.886+0000 7f13eb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ec0ffe40 0x7f13ec194fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:49790/0 (socket says 172.21.15.190:49790) 2024-03-26T18:47:08.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.886+0000 7f13eb7fe700 1 -- 172.21.15.190:0/3261016892 learned_addr learned my addr 172.21.15.190:0/3261016892 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:08.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.887+0000 7f13eb7fe700 1 -- 172.21.15.190:0/3261016892 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc0095f0 con 0x7f13ec0ffe40 2024-03-26T18:47:08.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.887+0000 7f13eb7fe700 1 --2- 172.21.15.190:0/3261016892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ec0ffe40 0x7f13ec194fa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0092e0 tx=0x7f13dc009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:08.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.887+0000 7f13e97fa700 1 -- 172.21.15.190:0/3261016892 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13dc01d070 con 0x7f13ec0ffe40 2024-03-26T18:47:08.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.887+0000 7f13e97fa700 1 -- 172.21.15.190:0/3261016892 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f13dc00be10 con 0x7f13ec0ffe40 2024-03-26T18:47:08.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.887+0000 7f13e97fa700 1 -- 172.21.15.190:0/3261016892 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13dc00f460 con 0x7f13ec0ffe40 2024-03-26T18:47:08.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.887+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3261016892 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec1956e0 con 0x7f13ec0ffe40 2024-03-26T18:47:08.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.887+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3261016892 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec195b80 con 0x7f13ec0ffe40 2024-03-26T18:47:08.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.888+0000 7f13e97fa700 1 -- 172.21.15.190:0/3261016892 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f13dc00b960 con 0x7f13ec0ffe40 2024-03-26T18:47:08.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.888+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3261016892 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ec18eaa0 con 0x7f13ec0ffe40 2024-03-26T18:47:08.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.888+0000 7f13e97fa700 1 --2- 172.21.15.190:0/3261016892 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f13d8038490 0x7f13d803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:08.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.888+0000 7f13e97fa700 1 -- 172.21.15.190:0/3261016892 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f13dc04c3b0 con 0x7f13ec0ffe40 2024-03-26T18:47:08.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.888+0000 7f13e2ffd700 1 --2- 172.21.15.190:0/3261016892 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f13d8038490 0x7f13d803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:08.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.889+0000 7f13e2ffd700 1 --2- 172.21.15.190:0/3261016892 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f13d8038490 0x7f13d803a940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f13d4006fd0 tx=0x7f13d4006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:08.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:08.893+0000 7f13e97fa700 1 -- 172.21.15.190:0/3261016892 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f13dc022790 con 0x7f13ec0ffe40 2024-03-26T18:47:09.148 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.156+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3261016892 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13ec02cc30 con 0x7f13ec0ffe40 2024-03-26T18:47:09.148 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.157+0000 7f13e97fa700 1 -- 172.21.15.190:0/3261016892 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f13dc027070 con 0x7f13ec0ffe40 2024-03-26T18:47:09.148 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:09.148 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:09.150 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.159+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3261016892 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f13d8038490 msgr2=0x7f13d803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:09.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.159+0000 7f13f1d8a700 1 --2- 172.21.15.190:0/3261016892 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f13d8038490 0x7f13d803a940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f13d4006fd0 tx=0x7f13d4006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:09.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.159+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3261016892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ec0ffe40 msgr2=0x7f13ec194fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:09.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.159+0000 7f13f1d8a700 1 --2- 172.21.15.190:0/3261016892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ec0ffe40 0x7f13ec194fa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0092e0 tx=0x7f13dc009e60 comp rx=0 tx=0).stop 2024-03-26T18:47:09.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.159+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3261016892 shutdown_connections 2024-03-26T18:47:09.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.159+0000 7f13f1d8a700 1 --2- 172.21.15.190:0/3261016892 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f13d8038490 0x7f13d803a940 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:09.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.159+0000 7f13f1d8a700 1 --2- 172.21.15.190:0/3261016892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ec0ffe40 0x7f13ec194fa0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:09.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.159+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3261016892 >> 172.21.15.190:0/3261016892 conn(0x7f13ec0fb3f0 msgr2=0x7f13ec0fc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:09.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.159+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3261016892 shutdown_connections 2024-03-26T18:47:09.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:09.159+0000 7f13f1d8a700 1 -- 172.21.15.190:0/3261016892 wait complete. 2024-03-26T18:47:09.152 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:09.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:09 smithi102 ceph-mon[29002]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:10.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:10 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/3261016892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:10.884 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:10.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:11.087 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:11.133 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:11.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:11 smithi102 ceph-mon[29002]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:12.613 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.622+0000 7f64d1052700 1 -- 172.21.15.190:0/91514843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64cc0695e0 msgr2=0x7f64cc104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:12.613 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.622+0000 7f64d1052700 1 --2- 172.21.15.190:0/91514843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64cc0695e0 0x7f64cc104dc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f64c400b0a0 tx=0x7f64c400b3b0 comp rx=0 tx=0).stop 2024-03-26T18:47:12.613 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.622+0000 7f64d1052700 1 -- 172.21.15.190:0/91514843 shutdown_connections 2024-03-26T18:47:12.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.622+0000 7f64d1052700 1 --2- 172.21.15.190:0/91514843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64cc0695e0 0x7f64cc104dc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:12.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.622+0000 7f64d1052700 1 -- 172.21.15.190:0/91514843 >> 172.21.15.190:0/91514843 conn(0x7f64cc0fa390 msgr2=0x7f64cc0fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:12.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.622+0000 7f64d1052700 1 -- 172.21.15.190:0/91514843 shutdown_connections 2024-03-26T18:47:12.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.622+0000 7f64d1052700 1 -- 172.21.15.190:0/91514843 wait complete. 2024-03-26T18:47:12.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.623+0000 7f64d1052700 1 Processor -- start 2024-03-26T18:47:12.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.623+0000 7f64d1052700 1 -- start start 2024-03-26T18:47:12.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.623+0000 7f64d1052700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64cc197170 0x7f64cc197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:12.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.623+0000 7f64d1052700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc197ac0 con 0x7f64cc197170 2024-03-26T18:47:12.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.623+0000 7f64cad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64cc197170 0x7f64cc197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:12.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.623+0000 7f64cad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64cc197170 0x7f64cc197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:49802/0 (socket says 172.21.15.190:49802) 2024-03-26T18:47:12.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.623+0000 7f64cad9d700 1 -- 172.21.15.190:0/3778811057 learned_addr learned my addr 172.21.15.190:0/3778811057 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:12.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.624+0000 7f64cad9d700 1 -- 172.21.15.190:0/3778811057 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c4009d00 con 0x7f64cc197170 2024-03-26T18:47:12.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.624+0000 7f64cad9d700 1 --2- 172.21.15.190:0/3778811057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64cc197170 0x7f64cc197580 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f64c400b830 tx=0x7f64c400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:12.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.624+0000 7f64b3fff700 1 -- 172.21.15.190:0/3778811057 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c400e040 con 0x7f64cc197170 2024-03-26T18:47:12.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.624+0000 7f64b3fff700 1 -- 172.21.15.190:0/3778811057 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f64c4012af0 con 0x7f64cc197170 2024-03-26T18:47:12.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.624+0000 7f64d1052700 1 -- 172.21.15.190:0/3778811057 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc197cc0 con 0x7f64cc197170 2024-03-26T18:47:12.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.624+0000 7f64b3fff700 1 -- 172.21.15.190:0/3778811057 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c401d9f0 con 0x7f64cc197170 2024-03-26T18:47:12.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.624+0000 7f64d1052700 1 -- 172.21.15.190:0/3778811057 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc19a920 con 0x7f64cc197170 2024-03-26T18:47:12.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.625+0000 7f64b3fff700 1 -- 172.21.15.190:0/3778811057 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f64c401dc60 con 0x7f64cc197170 2024-03-26T18:47:12.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.625+0000 7f64d1052700 1 -- 172.21.15.190:0/3778811057 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc190c80 con 0x7f64cc197170 2024-03-26T18:47:12.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.625+0000 7f64b3fff700 1 --2- 172.21.15.190:0/3778811057 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f64b40384e0 0x7f64b403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:12.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.625+0000 7f64b3fff700 1 -- 172.21.15.190:0/3778811057 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f64c404cac0 con 0x7f64cc197170 2024-03-26T18:47:12.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.625+0000 7f64ca59c700 1 --2- 172.21.15.190:0/3778811057 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f64b40384e0 0x7f64b403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:12.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.626+0000 7f64ca59c700 1 --2- 172.21.15.190:0/3778811057 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f64b40384e0 0x7f64b403a990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f64c0006fd0 tx=0x7f64c0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:12.621 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.630+0000 7f64b3fff700 1 -- 172.21.15.190:0/3778811057 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f64c401a820 con 0x7f64cc197170 2024-03-26T18:47:12.889 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.898+0000 7f64d1052700 1 -- 172.21.15.190:0/3778811057 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64cc0695e0 con 0x7f64cc197170 2024-03-26T18:47:12.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.899+0000 7f64b3fff700 1 -- 172.21.15.190:0/3778811057 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f64c401a820 con 0x7f64cc197170 2024-03-26T18:47:12.890 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:12.890 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:12.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.901+0000 7f64d1052700 1 -- 172.21.15.190:0/3778811057 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f64b40384e0 msgr2=0x7f64b403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:12.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.901+0000 7f64d1052700 1 --2- 172.21.15.190:0/3778811057 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f64b40384e0 0x7f64b403a990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f64c0006fd0 tx=0x7f64c0006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:12.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.901+0000 7f64d1052700 1 -- 172.21.15.190:0/3778811057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64cc197170 msgr2=0x7f64cc197580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:12.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.901+0000 7f64d1052700 1 --2- 172.21.15.190:0/3778811057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64cc197170 0x7f64cc197580 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f64c400b830 tx=0x7f64c400bd40 comp rx=0 tx=0).stop 2024-03-26T18:47:12.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.901+0000 7f64d1052700 1 -- 172.21.15.190:0/3778811057 shutdown_connections 2024-03-26T18:47:12.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.901+0000 7f64d1052700 1 --2- 172.21.15.190:0/3778811057 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f64b40384e0 0x7f64b403a990 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:12.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.901+0000 7f64d1052700 1 --2- 172.21.15.190:0/3778811057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64cc197170 0x7f64cc197580 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:12.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.901+0000 7f64d1052700 1 -- 172.21.15.190:0/3778811057 >> 172.21.15.190:0/3778811057 conn(0x7f64cc0fa390 msgr2=0x7f64cc0fb010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:12.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.901+0000 7f64d1052700 1 -- 172.21.15.190:0/3778811057 shutdown_connections 2024-03-26T18:47:12.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:12.901+0000 7f64d1052700 1 -- 172.21.15.190:0/3778811057 wait complete. 2024-03-26T18:47:12.893 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:13.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:13 smithi102 ceph-mon[29002]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:13.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:13 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/3778811057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:14.487 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:14.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:14.688 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:14.733 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:15.365 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:15 smithi102 ceph-mon[29002]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:16.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.240+0000 7f3af83f3700 1 -- 172.21.15.190:0/3727034415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af0074d80 msgr2=0x7f3af00731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:16.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.240+0000 7f3af83f3700 1 --2- 172.21.15.190:0/3727034415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af0074d80 0x7f3af00731e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009910 tx=0x7f3aec009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.240+0000 7f3af83f3700 1 -- 172.21.15.190:0/3727034415 shutdown_connections 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.240+0000 7f3af83f3700 1 --2- 172.21.15.190:0/3727034415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af0074d80 0x7f3af00731e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.240+0000 7f3af83f3700 1 -- 172.21.15.190:0/3727034415 >> 172.21.15.190:0/3727034415 conn(0x7f3af00fb380 msgr2=0x7f3af00fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.240+0000 7f3af83f3700 1 -- 172.21.15.190:0/3727034415 shutdown_connections 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.240+0000 7f3af83f3700 1 -- 172.21.15.190:0/3727034415 wait complete. 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3af83f3700 1 Processor -- start 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3af83f3700 1 -- start start 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3af83f3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af0074d80 0x7f3af019b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3af83f3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af019ba40 con 0x7f3af0074d80 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3af618f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af0074d80 0x7f3af019b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3af618f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af0074d80 0x7f3af019b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:47232/0 (socket says 172.21.15.190:47232) 2024-03-26T18:47:16.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3af618f700 1 -- 172.21.15.190:0/3150018225 learned_addr learned my addr 172.21.15.190:0/3150018225 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:16.234 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3af618f700 1 -- 172.21.15.190:0/3150018225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec0095f0 con 0x7f3af0074d80 2024-03-26T18:47:16.234 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3af618f700 1 --2- 172.21.15.190:0/3150018225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af0074d80 0x7f3af019b500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3aec005b40 tx=0x7f3aec004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:16.234 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3ae77fe700 1 -- 172.21.15.190:0/3150018225 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3aec01d070 con 0x7f3af0074d80 2024-03-26T18:47:16.234 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3ae77fe700 1 -- 172.21.15.190:0/3150018225 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3aec00be10 con 0x7f3af0074d80 2024-03-26T18:47:16.234 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3af83f3700 1 -- 172.21.15.190:0/3150018225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af019bc40 con 0x7f3af0074d80 2024-03-26T18:47:16.234 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.241+0000 7f3ae77fe700 1 -- 172.21.15.190:0/3150018225 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3aec00f460 con 0x7f3af0074d80 2024-03-26T18:47:16.234 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.242+0000 7f3af83f3700 1 -- 172.21.15.190:0/3150018225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af019c0e0 con 0x7f3af0074d80 2024-03-26T18:47:16.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.242+0000 7f3af83f3700 1 -- 172.21.15.190:0/3150018225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af0195090 con 0x7f3af0074d80 2024-03-26T18:47:16.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.242+0000 7f3ae77fe700 1 -- 172.21.15.190:0/3150018225 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f3aec00b960 con 0x7f3af0074d80 2024-03-26T18:47:16.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.242+0000 7f3ae77fe700 1 --2- 172.21.15.190:0/3150018225 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3adc0384a0 0x7f3adc03a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:16.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.243+0000 7f3ae77fe700 1 -- 172.21.15.190:0/3150018225 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3aec04c510 con 0x7f3af0074d80 2024-03-26T18:47:16.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.243+0000 7f3af598e700 1 --2- 172.21.15.190:0/3150018225 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3adc0384a0 0x7f3adc03a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:16.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.243+0000 7f3af598e700 1 --2- 172.21.15.190:0/3150018225 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3adc0384a0 0x7f3adc03a950 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0006fd0 tx=0x7f3ae0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:16.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.247+0000 7f3ae77fe700 1 -- 172.21.15.190:0/3150018225 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3aec02a360 con 0x7f3af0074d80 2024-03-26T18:47:16.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.511+0000 7f3af83f3700 1 -- 172.21.15.190:0/3150018225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3af002d050 con 0x7f3af0074d80 2024-03-26T18:47:16.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.511+0000 7f3ae77fe700 1 -- 172.21.15.190:0/3150018225 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3aec027020 con 0x7f3af0074d80 2024-03-26T18:47:16.503 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:16.503 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:16.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.514+0000 7f3af83f3700 1 -- 172.21.15.190:0/3150018225 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3adc0384a0 msgr2=0x7f3adc03a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:16.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.514+0000 7f3af83f3700 1 --2- 172.21.15.190:0/3150018225 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3adc0384a0 0x7f3adc03a950 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0006fd0 tx=0x7f3ae0006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:16.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.514+0000 7f3af83f3700 1 -- 172.21.15.190:0/3150018225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af0074d80 msgr2=0x7f3af019b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:16.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.514+0000 7f3af83f3700 1 --2- 172.21.15.190:0/3150018225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af0074d80 0x7f3af019b500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3aec005b40 tx=0x7f3aec004740 comp rx=0 tx=0).stop 2024-03-26T18:47:16.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.514+0000 7f3af83f3700 1 -- 172.21.15.190:0/3150018225 shutdown_connections 2024-03-26T18:47:16.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.514+0000 7f3af83f3700 1 --2- 172.21.15.190:0/3150018225 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3adc0384a0 0x7f3adc03a950 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:16.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.514+0000 7f3af83f3700 1 --2- 172.21.15.190:0/3150018225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af0074d80 0x7f3af019b500 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:16.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.514+0000 7f3af83f3700 1 -- 172.21.15.190:0/3150018225 >> 172.21.15.190:0/3150018225 conn(0x7f3af00fb380 msgr2=0x7f3af00fc000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:16.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.514+0000 7f3af83f3700 1 -- 172.21.15.190:0/3150018225 shutdown_connections 2024-03-26T18:47:16.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:16.514+0000 7f3af83f3700 1 -- 172.21.15.190:0/3150018225 wait complete. 2024-03-26T18:47:16.506 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:16.530 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:16 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:16.530 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:16 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:16.530 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:16 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:16.530 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:16 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:16.530 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:16 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:16.530 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:16 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:17.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:17 smithi102 ceph-mon[29002]: Deploying daemon prometheus.smithi102 on smithi102 2024-03-26T18:47:17.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:17 smithi102 ceph-mon[29002]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:17.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:17 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/3150018225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:17.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:17.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:18.203 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:18.257 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:19.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:19 smithi102 ceph-mon[29002]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:19.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:19 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:19.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.750+0000 7f0170d8b700 1 -- 172.21.15.190:0/108616394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f016c0fed30 msgr2=0x7f016c0ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:19.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.750+0000 7f0170d8b700 1 --2- 172.21.15.190:0/108616394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f016c0fed30 0x7f016c0ff140 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f015c009910 tx=0x7f015c009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:19.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.750+0000 7f0170d8b700 1 -- 172.21.15.190:0/108616394 shutdown_connections 2024-03-26T18:47:19.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.750+0000 7f0170d8b700 1 --2- 172.21.15.190:0/108616394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f016c0fed30 0x7f016c0ff140 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:19.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.750+0000 7f0170d8b700 1 -- 172.21.15.190:0/108616394 >> 172.21.15.190:0/108616394 conn(0x7f016c0fa2c0 msgr2=0x7f016c0fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:19.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.750+0000 7f0170d8b700 1 -- 172.21.15.190:0/108616394 shutdown_connections 2024-03-26T18:47:19.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.750+0000 7f0170d8b700 1 -- 172.21.15.190:0/108616394 wait complete. 2024-03-26T18:47:19.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.751+0000 7f0170d8b700 1 Processor -- start 2024-03-26T18:47:19.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.751+0000 7f0170d8b700 1 -- start start 2024-03-26T18:47:19.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.751+0000 7f0170d8b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f016c0fed30 0x7f016c0fc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:19.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.751+0000 7f0170d8b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c0fc7b0 con 0x7f016c0fed30 2024-03-26T18:47:19.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.751+0000 7f016a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f016c0fed30 0x7f016c0fc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:19.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.751+0000 7f016a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f016c0fed30 0x7f016c0fc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:47248/0 (socket says 172.21.15.190:47248) 2024-03-26T18:47:19.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.751+0000 7f016a59c700 1 -- 172.21.15.190:0/1729922997 learned_addr learned my addr 172.21.15.190:0/1729922997 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:19.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.752+0000 7f016a59c700 1 -- 172.21.15.190:0/1729922997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c0095f0 con 0x7f016c0fed30 2024-03-26T18:47:19.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.752+0000 7f016a59c700 1 --2- 172.21.15.190:0/1729922997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f016c0fed30 0x7f016c0fc270 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f015c005b40 tx=0x7f015c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:19.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.752+0000 7f01637fe700 1 -- 172.21.15.190:0/1729922997 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f015c01d070 con 0x7f016c0fed30 2024-03-26T18:47:19.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.752+0000 7f0170d8b700 1 -- 172.21.15.190:0/1729922997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016c0fa8c0 con 0x7f016c0fed30 2024-03-26T18:47:19.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.752+0000 7f01637fe700 1 -- 172.21.15.190:0/1729922997 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f015c00be10 con 0x7f016c0fed30 2024-03-26T18:47:19.748 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.752+0000 7f01637fe700 1 -- 172.21.15.190:0/1729922997 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f015c00f460 con 0x7f016c0fed30 2024-03-26T18:47:19.748 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.752+0000 7f0170d8b700 1 -- 172.21.15.190:0/1729922997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016c0fad60 con 0x7f016c0fed30 2024-03-26T18:47:19.748 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.753+0000 7f01637fe700 1 -- 172.21.15.190:0/1729922997 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f015c00b960 con 0x7f016c0fed30 2024-03-26T18:47:19.748 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.753+0000 7f0170d8b700 1 -- 172.21.15.190:0/1729922997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016c062380 con 0x7f016c0fed30 2024-03-26T18:47:19.748 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.753+0000 7f01637fe700 1 --2- 172.21.15.190:0/1729922997 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f0158038490 0x7f015803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:19.748 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.753+0000 7f01637fe700 1 -- 172.21.15.190:0/1729922997 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f015c04c4b0 con 0x7f016c0fed30 2024-03-26T18:47:19.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.753+0000 7f0163fff700 1 --2- 172.21.15.190:0/1729922997 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f0158038490 0x7f015803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:19.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.753+0000 7f0163fff700 1 --2- 172.21.15.190:0/1729922997 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f0158038490 0x7f015803a940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0154006fd0 tx=0x7f0154006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:19.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:19.758+0000 7f01637fe700 1 -- 172.21.15.190:0/1729922997 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f015c02a360 con 0x7f016c0fed30 2024-03-26T18:47:20.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.024+0000 7f0170d8b700 1 -- 172.21.15.190:0/1729922997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f016c18e400 con 0x7f016c0fed30 2024-03-26T18:47:20.017 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.025+0000 7f01637fe700 1 -- 172.21.15.190:0/1729922997 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f015c027030 con 0x7f016c0fed30 2024-03-26T18:47:20.017 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:20.017 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:20.019 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.027+0000 7f0170d8b700 1 -- 172.21.15.190:0/1729922997 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f0158038490 msgr2=0x7f015803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:20.019 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.027+0000 7f0170d8b700 1 --2- 172.21.15.190:0/1729922997 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f0158038490 0x7f015803a940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0154006fd0 tx=0x7f0154006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:20.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.027+0000 7f0170d8b700 1 -- 172.21.15.190:0/1729922997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f016c0fed30 msgr2=0x7f016c0fc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:20.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.027+0000 7f0170d8b700 1 --2- 172.21.15.190:0/1729922997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f016c0fed30 0x7f016c0fc270 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f015c005b40 tx=0x7f015c004740 comp rx=0 tx=0).stop 2024-03-26T18:47:20.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.027+0000 7f0170d8b700 1 -- 172.21.15.190:0/1729922997 shutdown_connections 2024-03-26T18:47:20.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.027+0000 7f0170d8b700 1 --2- 172.21.15.190:0/1729922997 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f0158038490 0x7f015803a940 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:20.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.027+0000 7f0170d8b700 1 --2- 172.21.15.190:0/1729922997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f016c0fed30 0x7f016c0fc270 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:20.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.027+0000 7f0170d8b700 1 -- 172.21.15.190:0/1729922997 >> 172.21.15.190:0/1729922997 conn(0x7f016c0fa2c0 msgr2=0x7f016c069060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:20.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.027+0000 7f0170d8b700 1 -- 172.21.15.190:0/1729922997 shutdown_connections 2024-03-26T18:47:20.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:20.027+0000 7f0170d8b700 1 -- 172.21.15.190:0/1729922997 wait complete. 2024-03-26T18:47:20.021 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:20.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:20 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/1729922997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:21.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:21 smithi102 ceph-mon[29002]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:21.568 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:21.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:21.772 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:21.817 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:23.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.285+0000 7f853899c700 1 -- 172.21.15.190:0/3677162221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85340fe110 msgr2=0x7f85340fe520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:23.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.285+0000 7f853899c700 1 --2- 172.21.15.190:0/3677162221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85340fe110 0x7f85340fe520 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8524009910 tx=0x7f8524009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:23.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.285+0000 7f853899c700 1 -- 172.21.15.190:0/3677162221 shutdown_connections 2024-03-26T18:47:23.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.285+0000 7f853899c700 1 --2- 172.21.15.190:0/3677162221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85340fe110 0x7f85340fe520 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:23.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.285+0000 7f853899c700 1 -- 172.21.15.190:0/3677162221 >> 172.21.15.190:0/3677162221 conn(0x7f85340f9a60 msgr2=0x7f85340fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:23.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.285+0000 7f853899c700 1 -- 172.21.15.190:0/3677162221 shutdown_connections 2024-03-26T18:47:23.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.285+0000 7f853899c700 1 -- 172.21.15.190:0/3677162221 wait complete. 2024-03-26T18:47:23.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.286+0000 7f853899c700 1 Processor -- start 2024-03-26T18:47:23.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.286+0000 7f853899c700 1 -- start start 2024-03-26T18:47:23.279 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.286+0000 7f853899c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85340fe110 0x7f8534194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:23.279 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.286+0000 7f853899c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8534195500 con 0x7f85340fe110 2024-03-26T18:47:23.279 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.286+0000 7f853259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85340fe110 0x7f8534194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:23.279 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.286+0000 7f853259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85340fe110 0x7f8534194fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:47262/0 (socket says 172.21.15.190:47262) 2024-03-26T18:47:23.279 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.286+0000 7f853259c700 1 -- 172.21.15.190:0/3591502820 learned_addr learned my addr 172.21.15.190:0/3591502820 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:23.279 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.287+0000 7f853259c700 1 -- 172.21.15.190:0/3591502820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85240095f0 con 0x7f85340fe110 2024-03-26T18:47:23.279 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.287+0000 7f853259c700 1 --2- 172.21.15.190:0/3591502820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85340fe110 0x7f8534194fc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8524005b40 tx=0x7f8524004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:23.280 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.287+0000 7f85237fe700 1 -- 172.21.15.190:0/3591502820 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f852401d070 con 0x7f85340fe110 2024-03-26T18:47:23.280 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.287+0000 7f85237fe700 1 -- 172.21.15.190:0/3591502820 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f852400be10 con 0x7f85340fe110 2024-03-26T18:47:23.280 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.287+0000 7f853899c700 1 -- 172.21.15.190:0/3591502820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8534195700 con 0x7f85340fe110 2024-03-26T18:47:23.280 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.287+0000 7f85237fe700 1 -- 172.21.15.190:0/3591502820 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f852400f460 con 0x7f85340fe110 2024-03-26T18:47:23.280 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.287+0000 7f853899c700 1 -- 172.21.15.190:0/3591502820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8534195ba0 con 0x7f85340fe110 2024-03-26T18:47:23.280 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.288+0000 7f853899c700 1 -- 172.21.15.190:0/3591502820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853418eb20 con 0x7f85340fe110 2024-03-26T18:47:23.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.288+0000 7f85237fe700 1 -- 172.21.15.190:0/3591502820 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f85240227e0 con 0x7f85340fe110 2024-03-26T18:47:23.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.288+0000 7f85237fe700 1 --2- 172.21.15.190:0/3591502820 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f851c038490 0x7f851c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:23.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.288+0000 7f85237fe700 1 -- 172.21.15.190:0/3591502820 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f852404c3e0 con 0x7f85340fe110 2024-03-26T18:47:23.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.288+0000 7f8531d9b700 1 --2- 172.21.15.190:0/3591502820 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f851c038490 0x7f851c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:23.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.289+0000 7f8531d9b700 1 --2- 172.21.15.190:0/3591502820 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f851c038490 0x7f851c03a940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8528006fd0 tx=0x7f8528006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:23.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.293+0000 7f85237fe700 1 -- 172.21.15.190:0/3591502820 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f852402bba0 con 0x7f85340fe110 2024-03-26T18:47:23.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:23 smithi102 ceph-mon[29002]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:23.547 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.556+0000 7f853899c700 1 -- 172.21.15.190:0/3591502820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f853402cc30 con 0x7f85340fe110 2024-03-26T18:47:23.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.557+0000 7f85237fe700 1 -- 172.21.15.190:0/3591502820 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8524027030 con 0x7f85340fe110 2024-03-26T18:47:23.548 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:23.548 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:23.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.559+0000 7f853899c700 1 -- 172.21.15.190:0/3591502820 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f851c038490 msgr2=0x7f851c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:23.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.559+0000 7f853899c700 1 --2- 172.21.15.190:0/3591502820 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f851c038490 0x7f851c03a940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8528006fd0 tx=0x7f8528006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:23.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.559+0000 7f853899c700 1 -- 172.21.15.190:0/3591502820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85340fe110 msgr2=0x7f8534194fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:23.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.559+0000 7f853899c700 1 --2- 172.21.15.190:0/3591502820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85340fe110 0x7f8534194fc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8524005b40 tx=0x7f8524004740 comp rx=0 tx=0).stop 2024-03-26T18:47:23.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.559+0000 7f853899c700 1 -- 172.21.15.190:0/3591502820 shutdown_connections 2024-03-26T18:47:23.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.559+0000 7f853899c700 1 --2- 172.21.15.190:0/3591502820 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f851c038490 0x7f851c03a940 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:23.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.559+0000 7f853899c700 1 --2- 172.21.15.190:0/3591502820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85340fe110 0x7f8534194fc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:23.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.559+0000 7f853899c700 1 -- 172.21.15.190:0/3591502820 >> 172.21.15.190:0/3591502820 conn(0x7f85340f9a60 msgr2=0x7f85340fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:23.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.559+0000 7f853899c700 1 -- 172.21.15.190:0/3591502820 shutdown_connections 2024-03-26T18:47:23.552 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:23.559+0000 7f853899c700 1 -- 172.21.15.190:0/3591502820 wait complete. 2024-03-26T18:47:23.552 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:24.192 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:24 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/3591502820' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:25.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:25.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:25.290 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:25.325 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:25 smithi102 ceph-mon[29002]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:25.336 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:26.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.775+0000 7f3c0b228700 1 -- 172.21.15.190:0/2829178438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c040fed30 msgr2=0x7f3c040ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:26.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.775+0000 7f3c0b228700 1 --2- 172.21.15.190:0/2829178438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c040fed30 0x7f3c040ff140 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8009910 tx=0x7f3bf8009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.775+0000 7f3c0b228700 1 -- 172.21.15.190:0/2829178438 shutdown_connections 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.775+0000 7f3c0b228700 1 --2- 172.21.15.190:0/2829178438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c040fed30 0x7f3c040ff140 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.775+0000 7f3c0b228700 1 -- 172.21.15.190:0/2829178438 >> 172.21.15.190:0/2829178438 conn(0x7f3c040fa2e0 msgr2=0x7f3c040fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.775+0000 7f3c0b228700 1 -- 172.21.15.190:0/2829178438 shutdown_connections 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.775+0000 7f3c0b228700 1 -- 172.21.15.190:0/2829178438 wait complete. 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.776+0000 7f3c0b228700 1 Processor -- start 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.776+0000 7f3c0b228700 1 -- start start 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.776+0000 7f3c0b228700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c040fed30 0x7f3c04194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.776+0000 7f3c0b228700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c04195500 con 0x7f3c040fed30 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.776+0000 7f3c08fc4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c040fed30 0x7f3c04194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.776+0000 7f3c08fc4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c040fed30 0x7f3c04194fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:48680/0 (socket says 172.21.15.190:48680) 2024-03-26T18:47:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.776+0000 7f3c08fc4700 1 -- 172.21.15.190:0/220215188 learned_addr learned my addr 172.21.15.190:0/220215188 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:26.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.777+0000 7f3c08fc4700 1 -- 172.21.15.190:0/220215188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf80095f0 con 0x7f3c040fed30 2024-03-26T18:47:26.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.777+0000 7f3c08fc4700 1 --2- 172.21.15.190:0/220215188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c040fed30 0x7f3c04194fc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8005b40 tx=0x7f3bf8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:26.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.777+0000 7f3c01ffb700 1 -- 172.21.15.190:0/220215188 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bf801d070 con 0x7f3c040fed30 2024-03-26T18:47:26.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.777+0000 7f3c0b228700 1 -- 172.21.15.190:0/220215188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c04195700 con 0x7f3c040fed30 2024-03-26T18:47:26.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.777+0000 7f3c01ffb700 1 -- 172.21.15.190:0/220215188 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3bf800be10 con 0x7f3c040fed30 2024-03-26T18:47:26.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.777+0000 7f3c01ffb700 1 -- 172.21.15.190:0/220215188 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bf800f460 con 0x7f3c040fed30 2024-03-26T18:47:26.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.777+0000 7f3c0b228700 1 -- 172.21.15.190:0/220215188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c04195ba0 con 0x7f3c040fed30 2024-03-26T18:47:26.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.778+0000 7f3c0b228700 1 -- 172.21.15.190:0/220215188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0418ea20 con 0x7f3c040fed30 2024-03-26T18:47:26.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.778+0000 7f3c01ffb700 1 -- 172.21.15.190:0/220215188 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f3bf80227e0 con 0x7f3c040fed30 2024-03-26T18:47:26.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.778+0000 7f3c01ffb700 1 --2- 172.21.15.190:0/220215188 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3bf4038490 0x7f3bf403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:26.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.778+0000 7f3c01ffb700 1 -- 172.21.15.190:0/220215188 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3bf804c4b0 con 0x7f3c040fed30 2024-03-26T18:47:26.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.779+0000 7f3c03fff700 1 --2- 172.21.15.190:0/220215188 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3bf4038490 0x7f3bf403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:26.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.779+0000 7f3c03fff700 1 --2- 172.21.15.190:0/220215188 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3bf4038490 0x7f3bf403a940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0006fd0 tx=0x7f3bf0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:26.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:26.784+0000 7f3c01ffb700 1 -- 172.21.15.190:0/220215188 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3bf802bba0 con 0x7f3c040fed30 2024-03-26T18:47:27.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.046+0000 7f3c0b228700 1 -- 172.21.15.190:0/220215188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c0402cc30 con 0x7f3c040fed30 2024-03-26T18:47:27.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.046+0000 7f3c01ffb700 1 -- 172.21.15.190:0/220215188 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3bf8027030 con 0x7f3c040fed30 2024-03-26T18:47:27.038 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:27.038 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:27.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.048+0000 7f3c0b228700 1 -- 172.21.15.190:0/220215188 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3bf4038490 msgr2=0x7f3bf403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:27.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.048+0000 7f3c0b228700 1 --2- 172.21.15.190:0/220215188 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3bf4038490 0x7f3bf403a940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0006fd0 tx=0x7f3bf0006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:27.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.048+0000 7f3c0b228700 1 -- 172.21.15.190:0/220215188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c040fed30 msgr2=0x7f3c04194fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:27.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.048+0000 7f3c0b228700 1 --2- 172.21.15.190:0/220215188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c040fed30 0x7f3c04194fc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8005b40 tx=0x7f3bf8004740 comp rx=0 tx=0).stop 2024-03-26T18:47:27.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.048+0000 7f3c0b228700 1 -- 172.21.15.190:0/220215188 shutdown_connections 2024-03-26T18:47:27.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.048+0000 7f3c0b228700 1 --2- 172.21.15.190:0/220215188 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3bf4038490 0x7f3bf403a940 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:27.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.048+0000 7f3c0b228700 1 --2- 172.21.15.190:0/220215188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c040fed30 0x7f3c04194fc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:27.041 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.048+0000 7f3c0b228700 1 -- 172.21.15.190:0/220215188 >> 172.21.15.190:0/220215188 conn(0x7f3c040fa2e0 msgr2=0x7f3c040687f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:27.041 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.048+0000 7f3c0b228700 1 -- 172.21.15.190:0/220215188 shutdown_connections 2024-03-26T18:47:27.041 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:27.049+0000 7f3c0b228700 1 -- 172.21.15.190:0/220215188 wait complete. 2024-03-26T18:47:27.041 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:27.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:27 smithi102 ceph-mon[29002]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:27.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:27 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:27.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:27 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:27.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:27 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:27.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:27 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T18:47:27.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:27 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/220215188' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:28.426 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:28 smithi102 ceph-mon[29002]: from='mgr.14160 172.21.15.102:0/1214542305' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T18:47:28.426 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:28 smithi102 ceph-mon[29002]: mgrmap e14: smithi102.rsfpnv(active, since 59s) 2024-03-26T18:47:28.640 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:28.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:28.843 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:28.889 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:30.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.191+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/1543748244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed40ffe40 msgr2=0x7f9ed4100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.191+0000 7f9ed9b9a700 1 --2- 172.21.15.190:0/1543748244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed40ffe40 0x7f9ed4100250 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009910 tx=0x7f9ec4009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.191+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/1543748244 shutdown_connections 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.192+0000 7f9ed9b9a700 1 --2- 172.21.15.190:0/1543748244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed40ffe40 0x7f9ed4100250 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.192+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/1543748244 >> 172.21.15.190:0/1543748244 conn(0x7f9ed40fb3d0 msgr2=0x7f9ed40fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.192+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/1543748244 shutdown_connections 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.192+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/1543748244 wait complete. 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.192+0000 7f9ed9b9a700 1 Processor -- start 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.192+0000 7f9ed9b9a700 1 -- start start 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.192+0000 7f9ed9b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed40ffe40 0x7f9ed4194f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.192+0000 7f9ed9b9a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed41954d0 con 0x7f9ed40ffe40 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.192+0000 7f9ed37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed40ffe40 0x7f9ed4194f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.192+0000 7f9ed37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed40ffe40 0x7f9ed4194f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:48696/0 (socket says 172.21.15.190:48696) 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.192+0000 7f9ed37fe700 1 -- 172.21.15.190:0/4212411449 learned_addr learned my addr 172.21.15.190:0/4212411449 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.193+0000 7f9ed37fe700 1 -- 172.21.15.190:0/4212411449 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec40095f0 con 0x7f9ed40ffe40 2024-03-26T18:47:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.193+0000 7f9ed37fe700 1 --2- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed40ffe40 0x7f9ed4194f90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4005b40 tx=0x7f9ec4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.193+0000 7f9ed17fa700 1 -- 172.21.15.190:0/4212411449 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec401d070 con 0x7f9ed40ffe40 2024-03-26T18:47:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.193+0000 7f9ed17fa700 1 -- 172.21.15.190:0/4212411449 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9ec400be10 con 0x7f9ed40ffe40 2024-03-26T18:47:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.193+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/4212411449 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed41956d0 con 0x7f9ed40ffe40 2024-03-26T18:47:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.193+0000 7f9ed17fa700 1 -- 172.21.15.190:0/4212411449 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec400f460 con 0x7f9ed40ffe40 2024-03-26T18:47:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.193+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/4212411449 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed4195b70 con 0x7f9ed40ffe40 2024-03-26T18:47:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.194+0000 7f9ed17fa700 1 -- 172.21.15.190:0/4212411449 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 14) v1 ==== 45155+0+0 (secure 0 0 0) 0x7f9ec400b960 con 0x7f9ed40ffe40 2024-03-26T18:47:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.194+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/4212411449 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed418eaa0 con 0x7f9ed40ffe40 2024-03-26T18:47:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.194+0000 7f9ed17fa700 1 --2- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f9ec00384e0 0x7f9ec003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.194+0000 7f9ed17fa700 1 -- 172.21.15.190:0/4212411449 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9ec404c3b0 con 0x7f9ed40ffe40 2024-03-26T18:47:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.194+0000 7f9ecbfff700 1 -- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f9ec00384e0 msgr2=0x7f9ec003a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3517043895 2024-03-26T18:47:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.194+0000 7f9ecbfff700 1 --2- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f9ec00384e0 0x7f9ec003a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T18:47:30.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.199+0000 7f9ed17fa700 1 -- 172.21.15.190:0/4212411449 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9ec402bba0 con 0x7f9ed40ffe40 2024-03-26T18:47:30.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.395+0000 7f9ecbfff700 1 -- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f9ec00384e0 msgr2=0x7f9ec003a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3517043895 2024-03-26T18:47:30.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.395+0000 7f9ecbfff700 1 --2- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f9ec00384e0 0x7f9ec003a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T18:47:30.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.467+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/4212411449 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ed402cc30 con 0x7f9ed40ffe40 2024-03-26T18:47:30.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.468+0000 7f9ed17fa700 1 -- 172.21.15.190:0/4212411449 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f9ec4027070 con 0x7f9ed40ffe40 2024-03-26T18:47:30.459 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:30.459 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:30.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.470+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f9ec00384e0 msgr2=0x7f9ec003a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T18:47:30.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.470+0000 7f9ed9b9a700 1 --2- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f9ec00384e0 0x7f9ec003a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:30.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.470+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed40ffe40 msgr2=0x7f9ed4194f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:30.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.470+0000 7f9ed9b9a700 1 --2- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed40ffe40 0x7f9ed4194f90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4005b40 tx=0x7f9ec4004740 comp rx=0 tx=0).stop 2024-03-26T18:47:30.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.470+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/4212411449 shutdown_connections 2024-03-26T18:47:30.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.470+0000 7f9ed9b9a700 1 --2- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f9ec00384e0 0x7f9ec003a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:30.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.470+0000 7f9ed9b9a700 1 --2- 172.21.15.190:0/4212411449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed40ffe40 0x7f9ed4194f90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:30.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.470+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/4212411449 >> 172.21.15.190:0/4212411449 conn(0x7f9ed40fb3d0 msgr2=0x7f9ed40fc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:30.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.470+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/4212411449 shutdown_connections 2024-03-26T18:47:30.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:30.470+0000 7f9ed9b9a700 1 -- 172.21.15.190:0/4212411449 wait complete. 2024-03-26T18:47:30.462 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:31.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:31 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/4212411449' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:32.035 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:32.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:32.236 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:32.281 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:33.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.764+0000 7f3ff3335700 1 -- 172.21.15.190:0/2249237268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fec0fed30 msgr2=0x7f3fec0ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:33.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.764+0000 7f3ff3335700 1 --2- 172.21.15.190:0/2249237268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fec0fed30 0x7f3fec0ff140 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009910 tx=0x7f3fdc009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:33.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.764+0000 7f3ff3335700 1 -- 172.21.15.190:0/2249237268 shutdown_connections 2024-03-26T18:47:33.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.764+0000 7f3ff3335700 1 --2- 172.21.15.190:0/2249237268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fec0fed30 0x7f3fec0ff140 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:33.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.764+0000 7f3ff3335700 1 -- 172.21.15.190:0/2249237268 >> 172.21.15.190:0/2249237268 conn(0x7f3fec0fa2e0 msgr2=0x7f3fec0fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:33.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.764+0000 7f3ff3335700 1 -- 172.21.15.190:0/2249237268 shutdown_connections 2024-03-26T18:47:33.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.764+0000 7f3ff3335700 1 -- 172.21.15.190:0/2249237268 wait complete. 2024-03-26T18:47:33.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.765+0000 7f3ff3335700 1 Processor -- start 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.765+0000 7f3ff3335700 1 -- start start 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.765+0000 7f3ff3335700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fec0fed30 0x7f3fec068b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.765+0000 7f3ff3335700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec197cc0 con 0x7f3fec0fed30 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.765+0000 7f3ff10d1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fec0fed30 0x7f3fec068b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.765+0000 7f3ff10d1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fec0fed30 0x7f3fec068b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:48704/0 (socket says 172.21.15.190:48704) 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.765+0000 7f3ff10d1700 1 -- 172.21.15.190:0/2676057691 learned_addr learned my addr 172.21.15.190:0/2676057691 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.765+0000 7f3ff10d1700 1 -- 172.21.15.190:0/2676057691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc0095f0 con 0x7f3fec0fed30 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.765+0000 7f3ff10d1700 1 --2- 172.21.15.190:0/2676057691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fec0fed30 0x7f3fec068b20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc005b40 tx=0x7f3fdc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.766+0000 7f3fe27fc700 1 -- 172.21.15.190:0/2676057691 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fdc01d070 con 0x7f3fec0fed30 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.766+0000 7f3fe27fc700 1 -- 172.21.15.190:0/2676057691 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3fdc00be10 con 0x7f3fec0fed30 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.766+0000 7f3ff3335700 1 -- 172.21.15.190:0/2676057691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fec197e00 con 0x7f3fec0fed30 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.766+0000 7f3fe27fc700 1 -- 172.21.15.190:0/2676057691 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fdc00f460 con 0x7f3fec0fed30 2024-03-26T18:47:33.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.766+0000 7f3ff3335700 1 -- 172.21.15.190:0/2676057691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fec1982a0 con 0x7f3fec0fed30 2024-03-26T18:47:33.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.766+0000 7f3fe27fc700 1 -- 172.21.15.190:0/2676057691 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 14) v1 ==== 45155+0+0 (secure 0 0 0) 0x7f3fdc0227e0 con 0x7f3fec0fed30 2024-03-26T18:47:33.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.766+0000 7f3ff3335700 1 -- 172.21.15.190:0/2676057691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fec0623f0 con 0x7f3fec0fed30 2024-03-26T18:47:33.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.767+0000 7f3fe27fc700 1 --2- 172.21.15.190:0/2676057691 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3fd80384e0 0x7f3fd803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:33.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.767+0000 7f3fe27fc700 1 -- 172.21.15.190:0/2676057691 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3fdc04c4e0 con 0x7f3fec0fed30 2024-03-26T18:47:33.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.767+0000 7f3ff08d0700 1 -- 172.21.15.190:0/2676057691 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3fd80384e0 msgr2=0x7f3fd803a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3517043895 2024-03-26T18:47:33.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.767+0000 7f3ff08d0700 1 --2- 172.21.15.190:0/2676057691 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3fd80384e0 0x7f3fd803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T18:47:33.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.771+0000 7f3fe27fc700 1 -- 172.21.15.190:0/2676057691 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3fec0623f0 con 0x7f3fec0fed30 2024-03-26T18:47:33.834 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.842+0000 7f3fe27fc700 1 -- 172.21.15.190:0/2676057691 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 15) v1 ==== 44878+0+0 (secure 0 0 0) 0x7f3fdc00f6d0 con 0x7f3fec0fed30 2024-03-26T18:47:33.834 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.842+0000 7f3fe27fc700 1 -- 172.21.15.190:0/2676057691 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3fd80384e0 msgr2=0x7f3fd803a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T18:47:33.834 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:33.842+0000 7f3fe27fc700 1 --2- 172.21.15.190:0/2676057691 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3fd80384e0 0x7f3fd803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:34.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:34.042+0000 7f3ff3335700 1 -- 172.21.15.190:0/2676057691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fec18da10 con 0x7f3fec0fed30 2024-03-26T18:47:34.034 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:34.043+0000 7f3fe27fc700 1 -- 172.21.15.190:0/2676057691 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3fdc027020 con 0x7f3fec0fed30 2024-03-26T18:47:34.034 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:34.034 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:34.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:34.045+0000 7f3ff3335700 1 -- 172.21.15.190:0/2676057691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fec0fed30 msgr2=0x7f3fec068b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:34.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:34.045+0000 7f3ff3335700 1 --2- 172.21.15.190:0/2676057691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fec0fed30 0x7f3fec068b20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc005b40 tx=0x7f3fdc004740 comp rx=0 tx=0).stop 2024-03-26T18:47:34.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:34.045+0000 7f3ff3335700 1 -- 172.21.15.190:0/2676057691 shutdown_connections 2024-03-26T18:47:34.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:34.045+0000 7f3ff3335700 1 --2- 172.21.15.190:0/2676057691 >> [v2:172.21.15.102:6800/3517043895,v1:172.21.15.102:6801/3517043895] conn(0x7f3fd80384e0 0x7f3fd803a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:34.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:34.045+0000 7f3ff3335700 1 --2- 172.21.15.190:0/2676057691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fec0fed30 0x7f3fec068b20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:34.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:34.045+0000 7f3ff3335700 1 -- 172.21.15.190:0/2676057691 >> 172.21.15.190:0/2676057691 conn(0x7f3fec0fa2e0 msgr2=0x7f3fec0faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:34.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:34.045+0000 7f3ff3335700 1 -- 172.21.15.190:0/2676057691 shutdown_connections 2024-03-26T18:47:34.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:34.045+0000 7f3ff3335700 1 -- 172.21.15.190:0/2676057691 wait complete. 2024-03-26T18:47:34.037 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:34.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: Active manager daemon smithi102.rsfpnv restarted 2024-03-26T18:47:34.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: Activating manager daemon smithi102.rsfpnv 2024-03-26T18:47:34.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: osdmap e5: 0 total, 0 up, 0 in 2024-03-26T18:47:34.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: mgrmap e15: smithi102.rsfpnv(active, starting, since 0.00529764s) 2024-03-26T18:47:34.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:47:34.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi102.rsfpnv", "id": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:47:34.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T18:47:34.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T18:47:34.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T18:47:34.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: Manager daemon smithi102.rsfpnv is now available 2024-03-26T18:47:34.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:47:34.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:33 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:47:35.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:35.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:47:35.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:47:35.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/trash_purge_schedule"}]: dispatch 2024-03-26T18:47:35.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:34 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/2676057691' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:35.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:34 smithi102 ceph-mon[29002]: [26/Mar/2024:18:47:34] ENGINE Bus STARTING 2024-03-26T18:47:35.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:35.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:34 smithi102 ceph-mon[29002]: mgrmap e16: smithi102.rsfpnv(active, since 1.00567s) 2024-03-26T18:47:35.571 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:35.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:35.763 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:35.811 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:36.353 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:36 smithi102 ceph-mon[29002]: [26/Mar/2024:18:47:34] ENGINE Serving on http://172.21.15.102:8765 2024-03-26T18:47:36.354 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:36 smithi102 ceph-mon[29002]: [26/Mar/2024:18:47:34] ENGINE Serving on https://172.21.15.102:7150 2024-03-26T18:47:36.354 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:36 smithi102 ceph-mon[29002]: [26/Mar/2024:18:47:34] ENGINE Bus STARTED 2024-03-26T18:47:36.354 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:36 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:36.354 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:36 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:37.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:37 smithi102 ceph-mon[29002]: mgrmap e17: smithi102.rsfpnv(active, since 2s) 2024-03-26T18:47:37.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.308+0000 7f6f6b849700 1 -- 172.21.15.190:0/3740987411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f640ffe40 msgr2=0x7f6f64100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:37.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.308+0000 7f6f6b849700 1 --2- 172.21.15.190:0/3740987411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f640ffe40 0x7f6f64100250 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00b0a0 tx=0x7f6f5c00b3b0 comp rx=0 tx=0).stop 2024-03-26T18:47:37.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.308+0000 7f6f6b849700 1 -- 172.21.15.190:0/3740987411 shutdown_connections 2024-03-26T18:47:37.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.308+0000 7f6f6b849700 1 --2- 172.21.15.190:0/3740987411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f640ffe40 0x7f6f64100250 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:37.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.308+0000 7f6f6b849700 1 -- 172.21.15.190:0/3740987411 >> 172.21.15.190:0/3740987411 conn(0x7f6f640fb3d0 msgr2=0x7f6f640fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:37.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.308+0000 7f6f6b849700 1 -- 172.21.15.190:0/3740987411 shutdown_connections 2024-03-26T18:47:37.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.308+0000 7f6f6b849700 1 -- 172.21.15.190:0/3740987411 wait complete. 2024-03-26T18:47:37.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.308+0000 7f6f6b849700 1 Processor -- start 2024-03-26T18:47:37.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.308+0000 7f6f6b849700 1 -- start start 2024-03-26T18:47:37.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.308+0000 7f6f6b849700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f640ffe40 0x7f6f64197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:37.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.308+0000 7f6f6b849700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f641976b0 con 0x7f6f640ffe40 2024-03-26T18:47:37.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.309+0000 7f6f695e5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f640ffe40 0x7f6f64197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:37.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.309+0000 7f6f695e5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f640ffe40 0x7f6f64197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:58686/0 (socket says 172.21.15.190:58686) 2024-03-26T18:47:37.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.309+0000 7f6f695e5700 1 -- 172.21.15.190:0/4185347541 learned_addr learned my addr 172.21.15.190:0/4185347541 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:37.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.309+0000 7f6f695e5700 1 -- 172.21.15.190:0/4185347541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c009d00 con 0x7f6f640ffe40 2024-03-26T18:47:37.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.309+0000 7f6f695e5700 1 --2- 172.21.15.190:0/4185347541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f640ffe40 0x7f6f64197170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c015040 tx=0x7f6f5c00bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:37.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.309+0000 7f6f627fc700 1 -- 172.21.15.190:0/4185347541 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f5c00e040 con 0x7f6f640ffe40 2024-03-26T18:47:37.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.309+0000 7f6f6b849700 1 -- 172.21.15.190:0/4185347541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f641978b0 con 0x7f6f640ffe40 2024-03-26T18:47:37.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.309+0000 7f6f6b849700 1 -- 172.21.15.190:0/4185347541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f64197cd0 con 0x7f6f640ffe40 2024-03-26T18:47:37.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.309+0000 7f6f627fc700 1 -- 172.21.15.190:0/4185347541 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6f5c01d930 con 0x7f6f640ffe40 2024-03-26T18:47:37.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.309+0000 7f6f627fc700 1 -- 172.21.15.190:0/4185347541 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f5c027980 con 0x7f6f640ffe40 2024-03-26T18:47:37.303 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.310+0000 7f6f627fc700 1 -- 172.21.15.190:0/4185347541 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 45199+0+0 (secure 0 0 0) 0x7f6f5c01d480 con 0x7f6f640ffe40 2024-03-26T18:47:37.303 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.310+0000 7f6f627fc700 1 --2- 172.21.15.190:0/4185347541 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f4c038580 0x7f6f4c03aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:37.303 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.311+0000 7f6f627fc700 1 -- 172.21.15.190:0/4185347541 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f6f5c04c9b0 con 0x7f6f640ffe40 2024-03-26T18:47:37.303 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.311+0000 7f6f6b849700 1 -- 172.21.15.190:0/4185347541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f641008d0 con 0x7f6f640ffe40 2024-03-26T18:47:37.303 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.311+0000 7f6f68de4700 1 --2- 172.21.15.190:0/4185347541 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f4c038580 0x7f6f4c03aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:37.303 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.311+0000 7f6f68de4700 1 --2- 172.21.15.190:0/4185347541 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f4c038580 0x7f6f4c03aa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6f58006fd0 tx=0x7f6f58006e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:37.307 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.316+0000 7f6f627fc700 1 -- 172.21.15.190:0/4185347541 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6f641008d0 con 0x7f6f640ffe40 2024-03-26T18:47:37.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.589+0000 7f6f6b849700 1 -- 172.21.15.190:0/4185347541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f64190ea0 con 0x7f6f640ffe40 2024-03-26T18:47:37.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.590+0000 7f6f627fc700 1 -- 172.21.15.190:0/4185347541 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f6f5c0263c0 con 0x7f6f640ffe40 2024-03-26T18:47:37.582 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:37.582 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:37.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.592+0000 7f6f6b849700 1 -- 172.21.15.190:0/4185347541 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f4c038580 msgr2=0x7f6f4c03aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:37.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.592+0000 7f6f6b849700 1 --2- 172.21.15.190:0/4185347541 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f4c038580 0x7f6f4c03aa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6f58006fd0 tx=0x7f6f58006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:37.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.592+0000 7f6f6b849700 1 -- 172.21.15.190:0/4185347541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f640ffe40 msgr2=0x7f6f64197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:37.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.592+0000 7f6f6b849700 1 --2- 172.21.15.190:0/4185347541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f640ffe40 0x7f6f64197170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c015040 tx=0x7f6f5c00bb10 comp rx=0 tx=0).stop 2024-03-26T18:47:37.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.592+0000 7f6f6b849700 1 -- 172.21.15.190:0/4185347541 shutdown_connections 2024-03-26T18:47:37.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.592+0000 7f6f6b849700 1 --2- 172.21.15.190:0/4185347541 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f4c038580 0x7f6f4c03aa30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:37.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.592+0000 7f6f6b849700 1 --2- 172.21.15.190:0/4185347541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f640ffe40 0x7f6f64197170 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:37.585 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.592+0000 7f6f6b849700 1 -- 172.21.15.190:0/4185347541 >> 172.21.15.190:0/4185347541 conn(0x7f6f640fb3d0 msgr2=0x7f6f640fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:37.585 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.592+0000 7f6f6b849700 1 -- 172.21.15.190:0/4185347541 shutdown_connections 2024-03-26T18:47:37.585 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:37.592+0000 7f6f6b849700 1 -- 172.21.15.190:0/4185347541 wait complete. 2024-03-26T18:47:37.585 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:38.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:38 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/4185347541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:39.529 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:39 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:39.529 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:39 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:39.529 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:39 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:39.529 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:39 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:39.529 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:39 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:47:39.816 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:39.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:40.297 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:47:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:47:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:47:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:47:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:47:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:47:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-26T18:47:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:47:42.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.062+0000 7f1f77577700 1 -- 172.21.15.190:0/305074265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f700fec90 msgr2=0x7f1f700ff060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:42.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.062+0000 7f1f77577700 1 --2- 172.21.15.190:0/305074265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f700fec90 0x7f1f700ff060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009910 tx=0x7f1f6c009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:42.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.062+0000 7f1f77577700 1 -- 172.21.15.190:0/305074265 shutdown_connections 2024-03-26T18:47:42.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.062+0000 7f1f77577700 1 --2- 172.21.15.190:0/305074265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f700fec90 0x7f1f700ff060 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:42.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.062+0000 7f1f77577700 1 -- 172.21.15.190:0/305074265 >> 172.21.15.190:0/305074265 conn(0x7f1f70075780 msgr2=0x7f1f70075b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:42.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.062+0000 7f1f77577700 1 -- 172.21.15.190:0/305074265 shutdown_connections 2024-03-26T18:47:42.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.062+0000 7f1f77577700 1 -- 172.21.15.190:0/305074265 wait complete. 2024-03-26T18:47:42.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.062+0000 7f1f77577700 1 Processor -- start 2024-03-26T18:47:42.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.063+0000 7f1f77577700 1 -- start start 2024-03-26T18:47:42.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.063+0000 7f1f77577700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f700fec90 0x7f1f701952d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:42.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.063+0000 7f1f77577700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70195810 con 0x7f1f700fec90 2024-03-26T18:47:42.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.063+0000 7f1f75313700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f700fec90 0x7f1f701952d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:42.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.063+0000 7f1f75313700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f700fec90 0x7f1f701952d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:58712/0 (socket says 172.21.15.190:58712) 2024-03-26T18:47:42.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.063+0000 7f1f75313700 1 -- 172.21.15.190:0/1166368435 learned_addr learned my addr 172.21.15.190:0/1166368435 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.063+0000 7f1f75313700 1 -- 172.21.15.190:0/1166368435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c0095f0 con 0x7f1f700fec90 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.063+0000 7f1f75313700 1 --2- 172.21.15.190:0/1166368435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f700fec90 0x7f1f701952d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c005b40 tx=0x7f1f6c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.064+0000 7f1f667fc700 1 -- 172.21.15.190:0/1166368435 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f6c01d070 con 0x7f1f700fec90 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.064+0000 7f1f667fc700 1 -- 172.21.15.190:0/1166368435 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1f6c0044d0 con 0x7f1f700fec90 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.064+0000 7f1f77577700 1 -- 172.21.15.190:0/1166368435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f70195a10 con 0x7f1f700fec90 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.064+0000 7f1f667fc700 1 -- 172.21.15.190:0/1166368435 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f6c003930 con 0x7f1f700fec90 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.064+0000 7f1f77577700 1 -- 172.21.15.190:0/1166368435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f70195e50 con 0x7f1f700fec90 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.065+0000 7f1f667fc700 1 -- 172.21.15.190:0/1166368435 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 45199+0+0 (secure 0 0 0) 0x7f1f6c003bf0 con 0x7f1f700fec90 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.065+0000 7f1f77577700 1 -- 172.21.15.190:0/1166368435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f70062380 con 0x7f1f700fec90 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.065+0000 7f1f667fc700 1 --2- 172.21.15.190:0/1166368435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1f5c038580 0x7f1f5c03aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.065+0000 7f1f667fc700 1 -- 172.21.15.190:0/1166368435 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f1f6c04c520 con 0x7f1f700fec90 2024-03-26T18:47:42.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.065+0000 7f1f74b12700 1 --2- 172.21.15.190:0/1166368435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1f5c038580 0x7f1f5c03aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:42.058 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.066+0000 7f1f74b12700 1 --2- 172.21.15.190:0/1166368435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1f5c038580 0x7f1f5c03aa30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1f60006fd0 tx=0x7f1f60006e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:42.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.070+0000 7f1f667fc700 1 -- 172.21.15.190:0/1166368435 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1f6c02a540 con 0x7f1f700fec90 2024-03-26T18:47:42.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.351+0000 7f1f77577700 1 -- 172.21.15.190:0/1166368435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f70196630 con 0x7f1f700fec90 2024-03-26T18:47:42.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.352+0000 7f1f667fc700 1 -- 172.21.15.190:0/1166368435 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f1f6c004170 con 0x7f1f700fec90 2024-03-26T18:47:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:42.343 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:42.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.354+0000 7f1f77577700 1 -- 172.21.15.190:0/1166368435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1f5c038580 msgr2=0x7f1f5c03aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:42.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.354+0000 7f1f77577700 1 --2- 172.21.15.190:0/1166368435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1f5c038580 0x7f1f5c03aa30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1f60006fd0 tx=0x7f1f60006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:42.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.354+0000 7f1f77577700 1 -- 172.21.15.190:0/1166368435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f700fec90 msgr2=0x7f1f701952d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:42.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.354+0000 7f1f77577700 1 --2- 172.21.15.190:0/1166368435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f700fec90 0x7f1f701952d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c005b40 tx=0x7f1f6c004c80 comp rx=0 tx=0).stop 2024-03-26T18:47:42.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.354+0000 7f1f77577700 1 -- 172.21.15.190:0/1166368435 shutdown_connections 2024-03-26T18:47:42.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.354+0000 7f1f77577700 1 --2- 172.21.15.190:0/1166368435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1f5c038580 0x7f1f5c03aa30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:42.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.354+0000 7f1f77577700 1 --2- 172.21.15.190:0/1166368435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f700fec90 0x7f1f701952d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:42.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.354+0000 7f1f77577700 1 -- 172.21.15.190:0/1166368435 >> 172.21.15.190:0/1166368435 conn(0x7f1f70075780 msgr2=0x7f1f701045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:42.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.354+0000 7f1f77577700 1 -- 172.21.15.190:0/1166368435 shutdown_connections 2024-03-26T18:47:42.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:42.354+0000 7f1f77577700 1 -- 172.21.15.190:0/1166368435 wait complete. 2024-03-26T18:47:42.346 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:42.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:42 smithi102 ceph-mon[29002]: Deploying daemon ceph-exporter.smithi190 on smithi190 2024-03-26T18:47:43.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:43 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/1166368435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:44.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:44.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:44.475 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:47:45.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.871+0000 7fda264f5700 1 -- 172.21.15.190:0/1648837860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda200ff020 msgr2=0x7fda200ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:45.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.871+0000 7fda264f5700 1 --2- 172.21.15.190:0/1648837860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda200ff020 0x7fda200ff3f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fda10009910 tx=0x7fda10009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:45.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.871+0000 7fda264f5700 1 -- 172.21.15.190:0/1648837860 shutdown_connections 2024-03-26T18:47:45.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.871+0000 7fda264f5700 1 --2- 172.21.15.190:0/1648837860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda200ff020 0x7fda200ff3f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:45.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.871+0000 7fda264f5700 1 -- 172.21.15.190:0/1648837860 >> 172.21.15.190:0/1648837860 conn(0x7fda20074af0 msgr2=0x7fda20074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:45.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.871+0000 7fda264f5700 1 -- 172.21.15.190:0/1648837860 shutdown_connections 2024-03-26T18:47:45.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.871+0000 7fda264f5700 1 -- 172.21.15.190:0/1648837860 wait complete. 2024-03-26T18:47:45.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.872+0000 7fda264f5700 1 Processor -- start 2024-03-26T18:47:45.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.872+0000 7fda264f5700 1 -- start start 2024-03-26T18:47:45.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.872+0000 7fda264f5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda200ff020 0x7fda20101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:45.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.872+0000 7fda264f5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20103240 con 0x7fda200ff020 2024-03-26T18:47:45.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.873+0000 7fda1ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda200ff020 0x7fda20101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:45.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.873+0000 7fda1ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda200ff020 0x7fda20101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:42204/0 (socket says 172.21.15.190:42204) 2024-03-26T18:47:45.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.873+0000 7fda1ffff700 1 -- 172.21.15.190:0/399116551 learned_addr learned my addr 172.21.15.190:0/399116551 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:45.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.873+0000 7fda1ffff700 1 -- 172.21.15.190:0/399116551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda100095f0 con 0x7fda200ff020 2024-03-26T18:47:45.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.873+0000 7fda1ffff700 1 --2- 172.21.15.190:0/399116551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda200ff020 0x7fda20101270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda10005b40 tx=0x7fda10004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:45.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.873+0000 7fda1d7fa700 1 -- 172.21.15.190:0/399116551 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda1001d070 con 0x7fda200ff020 2024-03-26T18:47:45.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.873+0000 7fda1d7fa700 1 -- 172.21.15.190:0/399116551 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fda100044d0 con 0x7fda200ff020 2024-03-26T18:47:45.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.873+0000 7fda264f5700 1 -- 172.21.15.190:0/399116551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda20101810 con 0x7fda200ff020 2024-03-26T18:47:45.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.873+0000 7fda1d7fa700 1 -- 172.21.15.190:0/399116551 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda10003930 con 0x7fda200ff020 2024-03-26T18:47:45.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.873+0000 7fda264f5700 1 -- 172.21.15.190:0/399116551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda20101c30 con 0x7fda200ff020 2024-03-26T18:47:45.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.874+0000 7fda1d7fa700 1 -- 172.21.15.190:0/399116551 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 45199+0+0 (secure 0 0 0) 0x7fda10003bf0 con 0x7fda200ff020 2024-03-26T18:47:45.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.875+0000 7fda1d7fa700 1 --2- 172.21.15.190:0/399116551 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fda08038580 0x7fda0803aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:45.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.875+0000 7fda1d7fa700 1 -- 172.21.15.190:0/399116551 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fda1004d480 con 0x7fda200ff020 2024-03-26T18:47:45.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.875+0000 7fda264f5700 1 -- 172.21.15.190:0/399116551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda20062380 con 0x7fda200ff020 2024-03-26T18:47:45.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.875+0000 7fda1f7fe700 1 --2- 172.21.15.190:0/399116551 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fda08038580 0x7fda0803aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:45.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.875+0000 7fda1f7fe700 1 --2- 172.21.15.190:0/399116551 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fda08038580 0x7fda0803aa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda14006fd0 tx=0x7fda14006e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:45.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:45.880+0000 7fda1d7fa700 1 -- 172.21.15.190:0/399116551 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fda1002a950 con 0x7fda200ff020 2024-03-26T18:47:46.150 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.159+0000 7fda264f5700 1 -- 172.21.15.190:0/399116551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda20101f20 con 0x7fda200ff020 2024-03-26T18:47:46.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.160+0000 7fda1d7fa700 1 -- 172.21.15.190:0/399116551 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fda1002a360 con 0x7fda200ff020 2024-03-26T18:47:46.153 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:46.154 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:46.154 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.162+0000 7fda264f5700 1 -- 172.21.15.190:0/399116551 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fda08038580 msgr2=0x7fda0803aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:46.154 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.162+0000 7fda264f5700 1 --2- 172.21.15.190:0/399116551 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fda08038580 0x7fda0803aa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda14006fd0 tx=0x7fda14006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:46.154 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.162+0000 7fda264f5700 1 -- 172.21.15.190:0/399116551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda200ff020 msgr2=0x7fda20101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:46.154 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.162+0000 7fda264f5700 1 --2- 172.21.15.190:0/399116551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda200ff020 0x7fda20101270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda10005b40 tx=0x7fda10004c80 comp rx=0 tx=0).stop 2024-03-26T18:47:46.154 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.162+0000 7fda264f5700 1 -- 172.21.15.190:0/399116551 shutdown_connections 2024-03-26T18:47:46.154 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.162+0000 7fda264f5700 1 --2- 172.21.15.190:0/399116551 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fda08038580 0x7fda0803aa30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:46.154 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.162+0000 7fda264f5700 1 --2- 172.21.15.190:0/399116551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda200ff020 0x7fda20101270 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:46.155 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.162+0000 7fda264f5700 1 -- 172.21.15.190:0/399116551 >> 172.21.15.190:0/399116551 conn(0x7fda20074af0 msgr2=0x7fda20106ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:46.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.162+0000 7fda264f5700 1 -- 172.21.15.190:0/399116551 shutdown_connections 2024-03-26T18:47:46.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:46.162+0000 7fda264f5700 1 -- 172.21.15.190:0/399116551 wait complete. 2024-03-26T18:47:46.158 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:46.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:46 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/399116551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:47.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:47.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:48.005 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:47:48.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:48.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:48.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:48.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:48.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:47:48.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-26T18:47:48.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:47:48.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:47 smithi102 ceph-mon[29002]: Deploying daemon crash.smithi190 on smithi190 2024-03-26T18:47:50.003 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.010+0000 7f459f112700 1 -- 172.21.15.190:0/2805922938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980fec90 msgr2=0x7f45980ff060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:50.003 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.010+0000 7f459f112700 1 --2- 172.21.15.190:0/2805922938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980fec90 0x7f45980ff060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f458c009910 tx=0x7f458c009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.010+0000 7f459f112700 1 -- 172.21.15.190:0/2805922938 shutdown_connections 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.010+0000 7f459f112700 1 --2- 172.21.15.190:0/2805922938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980fec90 0x7f45980ff060 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.010+0000 7f459f112700 1 -- 172.21.15.190:0/2805922938 >> 172.21.15.190:0/2805922938 conn(0x7f4598075780 msgr2=0x7f4598075b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.010+0000 7f459f112700 1 -- 172.21.15.190:0/2805922938 shutdown_connections 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.010+0000 7f459f112700 1 -- 172.21.15.190:0/2805922938 wait complete. 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.011+0000 7f459f112700 1 Processor -- start 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.011+0000 7f459f112700 1 -- start start 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.011+0000 7f459f112700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980fec90 0x7f45981952b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.011+0000 7f459f112700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45981957f0 con 0x7f45980fec90 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.011+0000 7f459ceae700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980fec90 0x7f45981952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.011+0000 7f459ceae700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980fec90 0x7f45981952b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:42238/0 (socket says 172.21.15.190:42238) 2024-03-26T18:47:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.011+0000 7f459ceae700 1 -- 172.21.15.190:0/1191124096 learned_addr learned my addr 172.21.15.190:0/1191124096 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:50.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.012+0000 7f459ceae700 1 -- 172.21.15.190:0/1191124096 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458c0095f0 con 0x7f45980fec90 2024-03-26T18:47:50.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.012+0000 7f459ceae700 1 --2- 172.21.15.190:0/1191124096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980fec90 0x7f45981952b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f458c0092e0 tx=0x7f458c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:50.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.012+0000 7f4595ffb700 1 -- 172.21.15.190:0/1191124096 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f458c01d070 con 0x7f45980fec90 2024-03-26T18:47:50.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.012+0000 7f4595ffb700 1 -- 172.21.15.190:0/1191124096 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f458c00be10 con 0x7f45980fec90 2024-03-26T18:47:50.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.012+0000 7f459f112700 1 -- 172.21.15.190:0/1191124096 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45981959f0 con 0x7f45980fec90 2024-03-26T18:47:50.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.012+0000 7f4595ffb700 1 -- 172.21.15.190:0/1191124096 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f458c022e50 con 0x7f45980fec90 2024-03-26T18:47:50.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.012+0000 7f459f112700 1 -- 172.21.15.190:0/1191124096 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4598195e90 con 0x7f45980fec90 2024-03-26T18:47:50.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.013+0000 7f4595ffb700 1 -- 172.21.15.190:0/1191124096 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 45199+0+0 (secure 0 0 0) 0x7f458c00f460 con 0x7f45980fec90 2024-03-26T18:47:50.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.013+0000 7f4595ffb700 1 --2- 172.21.15.190:0/1191124096 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4588038530 0x7f458803a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:50.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.013+0000 7f4595ffb700 1 -- 172.21.15.190:0/1191124096 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f458c04d590 con 0x7f45980fec90 2024-03-26T18:47:50.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.013+0000 7f459f112700 1 -- 172.21.15.190:0/1191124096 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4598062380 con 0x7f45980fec90 2024-03-26T18:47:50.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.013+0000 7f4597fff700 1 --2- 172.21.15.190:0/1191124096 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4588038530 0x7f458803a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:50.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.014+0000 7f4597fff700 1 --2- 172.21.15.190:0/1191124096 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4588038530 0x7f458803a9e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4584006fd0 tx=0x7f4584006e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:50.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.018+0000 7f4595ffb700 1 -- 172.21.15.190:0/1191124096 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f458c027070 con 0x7f45980fec90 2024-03-26T18:47:50.149 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:49 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:50.149 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:49 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:47:50.294 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.303+0000 7f459f112700 1 -- 172.21.15.190:0/1191124096 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4598106f70 con 0x7f45980fec90 2024-03-26T18:47:50.294 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.303+0000 7f4595ffb700 1 -- 172.21.15.190:0/1191124096 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f458c034070 con 0x7f45980fec90 2024-03-26T18:47:50.295 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:50.295 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:50.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.305+0000 7f459f112700 1 -- 172.21.15.190:0/1191124096 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4588038530 msgr2=0x7f458803a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:50.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.305+0000 7f459f112700 1 --2- 172.21.15.190:0/1191124096 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4588038530 0x7f458803a9e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4584006fd0 tx=0x7f4584006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:50.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.305+0000 7f459f112700 1 -- 172.21.15.190:0/1191124096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980fec90 msgr2=0x7f45981952b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:50.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.305+0000 7f459f112700 1 --2- 172.21.15.190:0/1191124096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980fec90 0x7f45981952b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f458c0092e0 tx=0x7f458c004c80 comp rx=0 tx=0).stop 2024-03-26T18:47:50.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.305+0000 7f459f112700 1 -- 172.21.15.190:0/1191124096 shutdown_connections 2024-03-26T18:47:50.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.305+0000 7f459f112700 1 --2- 172.21.15.190:0/1191124096 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4588038530 0x7f458803a9e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:50.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.305+0000 7f459f112700 1 --2- 172.21.15.190:0/1191124096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980fec90 0x7f45981952b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:50.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.305+0000 7f459f112700 1 -- 172.21.15.190:0/1191124096 >> 172.21.15.190:0/1191124096 conn(0x7f4598075780 msgr2=0x7f45981045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:50.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.305+0000 7f459f112700 1 -- 172.21.15.190:0/1191124096 shutdown_connections 2024-03-26T18:47:50.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:50.305+0000 7f459f112700 1 -- 172.21.15.190:0/1191124096 wait complete. 2024-03-26T18:47:50.298 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:51.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:51 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/1191124096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:51.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:51.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:51.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:51.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:52.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:52.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:52.273 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:47:52.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:52 smithi102 ceph-mon[29002]: Deploying daemon node-exporter.smithi190 on smithi190 2024-03-26T18:47:55.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.242+0000 7f8344155700 1 -- 172.21.15.190:0/3195703984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0fefa0 msgr2=0x7f833c0ff370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:55.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.242+0000 7f8344155700 1 --2- 172.21.15.190:0/3195703984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0fefa0 0x7f833c0ff370 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8338009910 tx=0x7f8338009c20 comp rx=0 tx=0).stop 2024-03-26T18:47:55.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.242+0000 7f8344155700 1 -- 172.21.15.190:0/3195703984 shutdown_connections 2024-03-26T18:47:55.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.242+0000 7f8344155700 1 --2- 172.21.15.190:0/3195703984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0fefa0 0x7f833c0ff370 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:55.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.242+0000 7f8344155700 1 -- 172.21.15.190:0/3195703984 >> 172.21.15.190:0/3195703984 conn(0x7f833c0fa860 msgr2=0x7f833c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:55.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.243+0000 7f8344155700 1 -- 172.21.15.190:0/3195703984 shutdown_connections 2024-03-26T18:47:55.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.243+0000 7f8344155700 1 -- 172.21.15.190:0/3195703984 wait complete. 2024-03-26T18:47:55.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.243+0000 7f8344155700 1 Processor -- start 2024-03-26T18:47:55.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.243+0000 7f8344155700 1 -- start start 2024-03-26T18:47:55.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.243+0000 7f8344155700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0fefa0 0x7f833c1009b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:55.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.243+0000 7f8344155700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c102980 con 0x7f833c0fefa0 2024-03-26T18:47:55.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.244+0000 7f8341ef1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0fefa0 0x7f833c1009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:55.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.244+0000 7f8341ef1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0fefa0 0x7f833c1009b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:40736/0 (socket says 172.21.15.190:40736) 2024-03-26T18:47:55.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.244+0000 7f8341ef1700 1 -- 172.21.15.190:0/2964786195 learned_addr learned my addr 172.21.15.190:0/2964786195 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:47:55.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.244+0000 7f8341ef1700 1 -- 172.21.15.190:0/2964786195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83380095f0 con 0x7f833c0fefa0 2024-03-26T18:47:55.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.244+0000 7f8341ef1700 1 --2- 172.21.15.190:0/2964786195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0fefa0 0x7f833c1009b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8338005b40 tx=0x7f8338004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:55.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.244+0000 7f8332ffd700 1 -- 172.21.15.190:0/2964786195 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f833801d070 con 0x7f833c0fefa0 2024-03-26T18:47:55.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.244+0000 7f8344155700 1 -- 172.21.15.190:0/2964786195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c100f50 con 0x7f833c0fefa0 2024-03-26T18:47:55.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.244+0000 7f8332ffd700 1 -- 172.21.15.190:0/2964786195 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f83380044d0 con 0x7f833c0fefa0 2024-03-26T18:47:55.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.244+0000 7f8332ffd700 1 -- 172.21.15.190:0/2964786195 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8338003930 con 0x7f833c0fefa0 2024-03-26T18:47:55.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.244+0000 7f8344155700 1 -- 172.21.15.190:0/2964786195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c101370 con 0x7f833c0fefa0 2024-03-26T18:47:55.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.245+0000 7f8344155700 1 -- 172.21.15.190:0/2964786195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c062380 con 0x7f833c0fefa0 2024-03-26T18:47:55.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.245+0000 7f8332ffd700 1 -- 172.21.15.190:0/2964786195 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 45199+0+0 (secure 0 0 0) 0x7f833800b960 con 0x7f833c0fefa0 2024-03-26T18:47:55.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.245+0000 7f8332ffd700 1 --2- 172.21.15.190:0/2964786195 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8328038580 0x7f832803aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:47:55.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.245+0000 7f8332ffd700 1 -- 172.21.15.190:0/2964786195 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f833804c4b0 con 0x7f833c0fefa0 2024-03-26T18:47:55.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.246+0000 7f83416f0700 1 --2- 172.21.15.190:0/2964786195 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8328038580 0x7f832803aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:47:55.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.246+0000 7f83416f0700 1 --2- 172.21.15.190:0/2964786195 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8328038580 0x7f832803aa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f832c006fd0 tx=0x7f832c006e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:47:55.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.251+0000 7f8332ffd700 1 -- 172.21.15.190:0/2964786195 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f833802a540 con 0x7f833c0fefa0 2024-03-26T18:47:55.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:54 smithi102 ceph-mon[29002]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:55.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:55.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:55.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:55.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:55.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:55.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.zrixos", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:47:55.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.zrixos", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T18:47:55.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:47:55.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:47:55.525 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.534+0000 7f8344155700 1 -- 172.21.15.190:0/2964786195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f833c1017b0 con 0x7f833c0fefa0 2024-03-26T18:47:55.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.535+0000 7f8332ffd700 1 -- 172.21.15.190:0/2964786195 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f833802a360 con 0x7f833c0fefa0 2024-03-26T18:47:55.526 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:47:55.527 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:47:55.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.537+0000 7f8344155700 1 -- 172.21.15.190:0/2964786195 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8328038580 msgr2=0x7f832803aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:55.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.537+0000 7f8344155700 1 --2- 172.21.15.190:0/2964786195 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8328038580 0x7f832803aa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f832c006fd0 tx=0x7f832c006e40 comp rx=0 tx=0).stop 2024-03-26T18:47:55.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.537+0000 7f8344155700 1 -- 172.21.15.190:0/2964786195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0fefa0 msgr2=0x7f833c1009b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:47:55.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.537+0000 7f8344155700 1 --2- 172.21.15.190:0/2964786195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0fefa0 0x7f833c1009b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8338005b40 tx=0x7f8338004c80 comp rx=0 tx=0).stop 2024-03-26T18:47:55.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.537+0000 7f8344155700 1 -- 172.21.15.190:0/2964786195 shutdown_connections 2024-03-26T18:47:55.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.537+0000 7f8344155700 1 --2- 172.21.15.190:0/2964786195 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8328038580 0x7f832803aa30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:55.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.537+0000 7f8344155700 1 --2- 172.21.15.190:0/2964786195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0fefa0 0x7f833c1009b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:47:55.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.537+0000 7f8344155700 1 -- 172.21.15.190:0/2964786195 >> 172.21.15.190:0/2964786195 conn(0x7f833c0fa860 msgr2=0x7f833c0687f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:47:55.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.537+0000 7f8344155700 1 -- 172.21.15.190:0/2964786195 shutdown_connections 2024-03-26T18:47:55.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:47:55.537+0000 7f8344155700 1 -- 172.21.15.190:0/2964786195 wait complete. 2024-03-26T18:47:55.530 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:47:56.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:56 smithi102 ceph-mon[29002]: Deploying daemon mgr.smithi190.zrixos on smithi190 2024-03-26T18:47:56.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:56 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/2964786195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:47:57.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:57 smithi102 ceph-mon[29002]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:47:58.088 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:47:58.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:47:59.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:58 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:59.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:58 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:59.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:58 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:59.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:58 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:47:59.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:58 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:47:59.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:58 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:47:59.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:58 smithi102 ceph-mon[29002]: Deploying daemon mon.smithi190 on smithi190 2024-03-26T18:47:59.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:58 smithi102 ceph-mon[29002]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:00.140 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi190/config 2024-03-26T18:48:00.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:47:59 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:01.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:01 smithi102 ceph-mon[29002]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:01.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.804+0000 7fcdee25d700 1 -- 172.21.15.190:0/4233738790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde81001d0 msgr2=0x7fcde81005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:01.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.804+0000 7fcdee25d700 1 --2- 172.21.15.190:0/4233738790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde81001d0 0x7fcde81005a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009910 tx=0x7fcddc009c20 comp rx=0 tx=0).stop 2024-03-26T18:48:01.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.804+0000 7fcdee25d700 1 -- 172.21.15.190:0/4233738790 shutdown_connections 2024-03-26T18:48:01.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.804+0000 7fcdee25d700 1 --2- 172.21.15.190:0/4233738790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde81001d0 0x7fcde81005a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:01.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.804+0000 7fcdee25d700 1 -- 172.21.15.190:0/4233738790 >> 172.21.15.190:0/4233738790 conn(0x7fcde80fba50 msgr2=0x7fcde80fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:48:01.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.804+0000 7fcdee25d700 1 -- 172.21.15.190:0/4233738790 shutdown_connections 2024-03-26T18:48:01.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.804+0000 7fcdee25d700 1 -- 172.21.15.190:0/4233738790 wait complete. 2024-03-26T18:48:01.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.805+0000 7fcdee25d700 1 Processor -- start 2024-03-26T18:48:01.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.805+0000 7fcdee25d700 1 -- start start 2024-03-26T18:48:01.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.805+0000 7fcdee25d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde81001d0 0x7fcde8195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:01.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.805+0000 7fcdee25d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde8195990 con 0x7fcde81001d0 2024-03-26T18:48:01.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.806+0000 7fcde77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde81001d0 0x7fcde8195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:01.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.806+0000 7fcde77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde81001d0 0x7fcde8195450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.190:40764/0 (socket says 172.21.15.190:40764) 2024-03-26T18:48:01.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.806+0000 7fcde77fe700 1 -- 172.21.15.190:0/2520029732 learned_addr learned my addr 172.21.15.190:0/2520029732 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:48:01.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.806+0000 7fcde77fe700 1 -- 172.21.15.190:0/2520029732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc0095f0 con 0x7fcde81001d0 2024-03-26T18:48:01.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.806+0000 7fcde77fe700 1 --2- 172.21.15.190:0/2520029732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde81001d0 0x7fcde8195450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcddc005b40 tx=0x7fcddc004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:48:01.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.806+0000 7fcde4ff9700 1 -- 172.21.15.190:0/2520029732 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcddc01d070 con 0x7fcde81001d0 2024-03-26T18:48:01.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.806+0000 7fcdee25d700 1 -- 172.21.15.190:0/2520029732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde8195b90 con 0x7fcde81001d0 2024-03-26T18:48:01.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.806+0000 7fcde4ff9700 1 -- 172.21.15.190:0/2520029732 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcddc0044d0 con 0x7fcde81001d0 2024-03-26T18:48:01.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.806+0000 7fcde4ff9700 1 -- 172.21.15.190:0/2520029732 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcddc003930 con 0x7fcde81001d0 2024-03-26T18:48:01.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.806+0000 7fcdee25d700 1 -- 172.21.15.190:0/2520029732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde8195f50 con 0x7fcde81001d0 2024-03-26T18:48:01.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.807+0000 7fcdee25d700 1 -- 172.21.15.190:0/2520029732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde8062380 con 0x7fcde81001d0 2024-03-26T18:48:01.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.807+0000 7fcde4ff9700 1 -- 172.21.15.190:0/2520029732 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 45199+0+0 (secure 0 0 0) 0x7fcddc003bf0 con 0x7fcde81001d0 2024-03-26T18:48:01.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.807+0000 7fcde4ff9700 1 --2- 172.21.15.190:0/2520029732 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcdd0038580 0x7fcdd003aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:01.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.807+0000 7fcde4ff9700 1 -- 172.21.15.190:0/2520029732 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fcddc04c4c0 con 0x7fcde81001d0 2024-03-26T18:48:01.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.808+0000 7fcde6ffd700 1 --2- 172.21.15.190:0/2520029732 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcdd0038580 0x7fcdd003aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:01.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.808+0000 7fcde6ffd700 1 --2- 172.21.15.190:0/2520029732 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcdd0038580 0x7fcdd003aa30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8006fd0 tx=0x7fcdd8006e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:48:01.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:01.812+0000 7fcde4ff9700 1 -- 172.21.15.190:0/2520029732 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcddc02a540 con 0x7fcde81001d0 2024-03-26T18:48:02.085 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.094+0000 7fcdee25d700 1 -- 172.21.15.190:0/2520029732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcde8107f70 con 0x7fcde81001d0 2024-03-26T18:48:02.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.094+0000 7fcde4ff9700 1 -- 172.21.15.190:0/2520029732 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fcddc02a360 con 0x7fcde81001d0 2024-03-26T18:48:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:48:02.086 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:45:22.493426Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T18:48:02.088 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.096+0000 7fcdee25d700 1 -- 172.21.15.190:0/2520029732 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcdd0038580 msgr2=0x7fcdd003aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:02.088 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.096+0000 7fcdee25d700 1 --2- 172.21.15.190:0/2520029732 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcdd0038580 0x7fcdd003aa30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8006fd0 tx=0x7fcdd8006e40 comp rx=0 tx=0).stop 2024-03-26T18:48:02.088 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.096+0000 7fcdee25d700 1 -- 172.21.15.190:0/2520029732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde81001d0 msgr2=0x7fcde8195450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:02.088 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.096+0000 7fcdee25d700 1 --2- 172.21.15.190:0/2520029732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde81001d0 0x7fcde8195450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcddc005b40 tx=0x7fcddc004c80 comp rx=0 tx=0).stop 2024-03-26T18:48:02.088 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.096+0000 7fcdee25d700 1 -- 172.21.15.190:0/2520029732 shutdown_connections 2024-03-26T18:48:02.088 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.096+0000 7fcdee25d700 1 --2- 172.21.15.190:0/2520029732 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcdd0038580 0x7fcdd003aa30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:02.088 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.096+0000 7fcdee25d700 1 --2- 172.21.15.190:0/2520029732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde81001d0 0x7fcde8195450 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:02.088 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.096+0000 7fcdee25d700 1 -- 172.21.15.190:0/2520029732 >> 172.21.15.190:0/2520029732 conn(0x7fcde80fba50 msgr2=0x7fcde81043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:48:02.089 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.097+0000 7fcdee25d700 1 -- 172.21.15.190:0/2520029732 shutdown_connections 2024-03-26T18:48:02.089 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:02.097+0000 7fcdee25d700 1 -- 172.21.15.190:0/2520029732 wait complete. 2024-03-26T18:48:02.089 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-26T18:48:02.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:02 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/2520029732' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:48:04.193 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T18:48:04.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T18:48:04.398 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi190/config 2024-03-26T18:48:07.778 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:07 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: mon.smithi102 calling monitor election 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: mon.smithi190 calling monitor election 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/2742524819' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/crt"}]: dispatch 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:48:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: mon.smithi102 is new leader, mons smithi102,smithi190 in quorum (ranks 0,1) 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: monmap e2: 2 mons at {smithi102=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],smithi190=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: fsmap 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: osdmap e5: 0 total, 0 up, 0 in 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: mgrmap e17: smithi102.rsfpnv(active, since 33s) 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: Standby manager daemon smithi190.zrixos started 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/2742524819' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: overall HEALTH_OK 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/2742524819' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/key"}]: dispatch 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/2742524819' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:07.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:08.778 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:08 smithi190 ceph-mon[35043]: mgrmap e18: smithi102.rsfpnv(active, since 33s), standbys: smithi190.zrixos 2024-03-26T18:48:08.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:08 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi190.zrixos", "id": "smithi190.zrixos"}]: dispatch 2024-03-26T18:48:08.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:08 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:48:08.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:08 smithi190 ceph-mon[35043]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:08.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:08 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.853+0000 7f8b16eca700 1 -- 172.21.15.190:0/3953186896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b000035c0 msgr2=0x7f8b00005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.853+0000 7f8b16eca700 1 --2- 172.21.15.190:0/3953186896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b000035c0 0x7f8b00005a40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8b10068dc0 tx=0x7f8b0400feb0 comp rx=0 tx=0).stop 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.853+0000 7f8b16eca700 1 -- 172.21.15.190:0/3953186896 shutdown_connections 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.853+0000 7f8b16eca700 1 --2- 172.21.15.190:0/3953186896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b000035c0 0x7f8b00005a40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.853+0000 7f8b16eca700 1 -- 172.21.15.190:0/3953186896 >> 172.21.15.190:0/3953186896 conn(0x7f8b100fa9a0 msgr2=0x7f8b100fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.853+0000 7f8b16eca700 1 -- 172.21.15.190:0/3953186896 shutdown_connections 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.853+0000 7f8b16eca700 1 -- 172.21.15.190:0/3953186896 wait complete. 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b16eca700 1 Processor -- start 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b16eca700 1 -- start start 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b16eca700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b000035c0 0x7f8b10195560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b16eca700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b10195aa0 0x7f8b10199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:08.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b16eca700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b101960b0 con 0x7f8b000035c0 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b16eca700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b10196220 con 0x7f8b10195aa0 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b0ffff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b10195aa0 0x7f8b10199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b0ffff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b10195aa0 0x7f8b10199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:48878/0 (socket says 172.21.15.190:48878) 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b0ffff700 1 -- 172.21.15.190:0/2056444532 learned_addr learned my addr 172.21.15.190:0/2056444532 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b0ffff700 1 -- 172.21.15.190:0/2056444532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b10195aa0 msgr2=0x7f8b10199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b0ffff700 1 -- 172.21.15.190:0/2056444532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b10195aa0 msgr2=0x7f8b10199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b0ffff700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b10195aa0 0x7f8b10199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b14c66700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b000035c0 0x7f8b10195560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.854+0000 7f8b0ffff700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b10195aa0 0x7f8b10199f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.855+0000 7f8b14c66700 1 -- 172.21.15.190:0/2056444532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b10195aa0 msgr2=0x7f8b10199f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.855+0000 7f8b14c66700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b10195aa0 0x7f8b10199f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.855+0000 7f8b14c66700 1 -- 172.21.15.190:0/2056444532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0400f900 con 0x7f8b000035c0 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.855+0000 7f8b14c66700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b000035c0 0x7f8b10195560 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8b10068dc0 tx=0x7f8b04004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:48:08.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.855+0000 7f8b0dffb700 1 -- 172.21.15.190:0/2056444532 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b04013b30 con 0x7f8b000035c0 2024-03-26T18:48:08.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.855+0000 7f8b16eca700 1 -- 172.21.15.190:0/2056444532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1019a510 con 0x7f8b000035c0 2024-03-26T18:48:08.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.855+0000 7f8b0dffb700 1 -- 172.21.15.190:0/2056444532 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8b040044e0 con 0x7f8b000035c0 2024-03-26T18:48:08.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.855+0000 7f8b0dffb700 1 -- 172.21.15.190:0/2056444532 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b04003e20 con 0x7f8b000035c0 2024-03-26T18:48:08.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.855+0000 7f8b16eca700 1 -- 172.21.15.190:0/2056444532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1019aa60 con 0x7f8b000035c0 2024-03-26T18:48:08.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.857+0000 7f8b16eca700 1 -- 172.21.15.190:0/2056444532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b10062380 con 0x7f8b000035c0 2024-03-26T18:48:08.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.857+0000 7f8b0dffb700 1 -- 172.21.15.190:0/2056444532 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7f8b04004030 con 0x7f8b000035c0 2024-03-26T18:48:08.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.857+0000 7f8b0dffb700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8af406c750 0x7f8af406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:08.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.857+0000 7f8b0dffb700 1 -- 172.21.15.190:0/2056444532 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f8b0408d1b0 con 0x7f8b000035c0 2024-03-26T18:48:08.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.857+0000 7f8b0ffff700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8af406c750 0x7f8af406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:08.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.858+0000 7f8b0ffff700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8af406c750 0x7f8af406ec00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009180 tx=0x7f8afc008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:48:08.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:08.862+0000 7f8b0dffb700 1 -- 172.21.15.190:0/2056444532 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8b04058970 con 0x7f8b000035c0 2024-03-26T18:48:08.874 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:08 smithi102 ceph-mon[29002]: mgrmap e18: smithi102.rsfpnv(active, since 33s), standbys: smithi190.zrixos 2024-03-26T18:48:08.874 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:08 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi190.zrixos", "id": "smithi190.zrixos"}]: dispatch 2024-03-26T18:48:08.874 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:08 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:48:08.874 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:08 smithi102 ceph-mon[29002]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:08.874 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:08 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:48:09.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.150+0000 7f8b16eca700 1 -- 172.21.15.190:0/2056444532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b101a01b0 con 0x7f8b000035c0 2024-03-26T18:48:09.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.151+0000 7f8b0dffb700 1 -- 172.21.15.190:0/2056444532 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1026 (secure 0 0 0) 0x7f8b04031090 con 0x7f8b000035c0 2024-03-26T18:48:09.142 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-26T18:48:09.142 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":2,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","modified":"2024-03-26T18:48:02.282584Z","created":"2024-03-26T18:45:22.493426Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi190","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T18:48:09.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.153+0000 7f8b16eca700 1 -- 172.21.15.190:0/2056444532 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8af406c750 msgr2=0x7f8af406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:09.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.153+0000 7f8b16eca700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8af406c750 0x7f8af406ec00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009180 tx=0x7f8afc008040 comp rx=0 tx=0).stop 2024-03-26T18:48:09.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.153+0000 7f8b16eca700 1 -- 172.21.15.190:0/2056444532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b000035c0 msgr2=0x7f8b10195560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:09.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.153+0000 7f8b16eca700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b000035c0 0x7f8b10195560 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8b10068dc0 tx=0x7f8b04004f90 comp rx=0 tx=0).stop 2024-03-26T18:48:09.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.153+0000 7f8b16eca700 1 -- 172.21.15.190:0/2056444532 shutdown_connections 2024-03-26T18:48:09.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.153+0000 7f8b16eca700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8af406c750 0x7f8af406ec00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:09.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.153+0000 7f8b16eca700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b000035c0 0x7f8b10195560 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:09.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.153+0000 7f8b16eca700 1 --2- 172.21.15.190:0/2056444532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b10195aa0 0x7f8b10199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:09.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.153+0000 7f8b16eca700 1 -- 172.21.15.190:0/2056444532 >> 172.21.15.190:0/2056444532 conn(0x7f8b100fa9a0 msgr2=0x7f8b10101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:48:09.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.153+0000 7f8b16eca700 1 -- 172.21.15.190:0/2056444532 shutdown_connections 2024-03-26T18:48:09.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:48:09.153+0000 7f8b16eca700 1 -- 172.21.15.190:0/2056444532 wait complete. 2024-03-26T18:48:09.146 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 2 2024-03-26T18:48:09.858 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T18:48:09.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T18:48:10.076 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:48:10.198 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:09 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:10.199 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:09 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/2056444532' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:48:10.278 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:09 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:10.278 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:09 smithi190 ceph-mon[35043]: from='client.? 172.21.15.190:0/2056444532' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T18:48:11.278 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:48:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:48:11.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:11.341 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:48:11.341 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:48:11.341 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:12.182 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:12 smithi102 ceph-mon[29002]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:48:12.182 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:12 smithi102 ceph-mon[29002]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:48:12.182 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:12 smithi102 ceph-mon[29002]: Reconfiguring mon.smithi102 (unknown last config time)... 2024-03-26T18:48:12.182 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:12 smithi102 ceph-mon[29002]: Reconfiguring daemon mon.smithi102 on smithi102 2024-03-26T18:48:12.278 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:12 smithi190 ceph-mon[35043]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:48:12.278 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:12 smithi190 ceph-mon[35043]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:48:12.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:12 smithi190 ceph-mon[35043]: Reconfiguring mon.smithi102 (unknown last config time)... 2024-03-26T18:48:12.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:12 smithi190 ceph-mon[35043]: Reconfiguring daemon mon.smithi102 on smithi102 2024-03-26T18:48:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.291+0000 7f8500951700 1 -- 172.21.15.102:0/2654166739 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84fc1000b0 msgr2=0x7f84fc100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.291+0000 7f8500951700 1 --2- 172.21.15.102:0/2654166739 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84fc1000b0 0x7f84fc100480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f84f0009910 tx=0x7f84f0009c20 comp rx=0 tx=0).stop 2024-03-26T18:48:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.291+0000 7f8500951700 1 -- 172.21.15.102:0/2654166739 shutdown_connections 2024-03-26T18:48:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.291+0000 7f8500951700 1 --2- 172.21.15.102:0/2654166739 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84fc1000b0 0x7f84fc100480 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.291+0000 7f8500951700 1 -- 172.21.15.102:0/2654166739 >> 172.21.15.102:0/2654166739 conn(0x7f84fc0fb990 msgr2=0x7f84fc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:48:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.291+0000 7f8500951700 1 -- 172.21.15.102:0/2654166739 shutdown_connections 2024-03-26T18:48:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.291+0000 7f8500951700 1 -- 172.21.15.102:0/2654166739 wait complete. 2024-03-26T18:48:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f8500951700 1 Processor -- start 2024-03-26T18:48:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f8500951700 1 -- start start 2024-03-26T18:48:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f8500951700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84fc1000b0 0x7f84fc073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f8500951700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84fc073590 0x7f84fc073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f8500951700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1a16e0 con 0x7f84fc073590 2024-03-26T18:48:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f8500951700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1a1820 con 0x7f84fc1000b0 2024-03-26T18:48:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f84f9d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84fc073590 0x7f84fc073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f84f9d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84fc073590 0x7f84fc073a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59962/0 (socket says 172.21.15.102:59962) 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f84f9d9b700 1 -- 172.21.15.102:0/2354299486 learned_addr learned my addr 172.21.15.102:0/2354299486 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f84fa59c700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84fc1000b0 0x7f84fc073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f84f9d9b700 1 -- 172.21.15.102:0/2354299486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84fc1000b0 msgr2=0x7f84fc073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.292+0000 7f84f9d9b700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84fc1000b0 0x7f84fc073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.293+0000 7f84f9d9b700 1 -- 172.21.15.102:0/2354299486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f00095f0 con 0x7f84fc073590 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.293+0000 7f84fa59c700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84fc1000b0 0x7f84fc073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.293+0000 7f84f9d9b700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84fc073590 0x7f84fc073a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00b730 tx=0x7f84ec00baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.293+0000 7f84eb7fe700 1 -- 172.21.15.102:0/2354299486 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84ec004200 con 0x7f84fc073590 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.293+0000 7f84eb7fe700 1 -- 172.21.15.102:0/2354299486 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f84ec004360 con 0x7f84fc073590 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.293+0000 7f8500951700 1 -- 172.21.15.102:0/2354299486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84fc1a1960 con 0x7f84fc073590 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.293+0000 7f84eb7fe700 1 -- 172.21.15.102:0/2354299486 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84ec011be0 con 0x7f84fc073590 2024-03-26T18:48:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.293+0000 7f8500951700 1 -- 172.21.15.102:0/2354299486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84fc1a1eb0 con 0x7f84fc073590 2024-03-26T18:48:12.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.294+0000 7f8500951700 1 -- 172.21.15.102:0/2354299486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84fc062380 con 0x7f84fc073590 2024-03-26T18:48:12.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.294+0000 7f84eb7fe700 1 -- 172.21.15.102:0/2354299486 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7f84ec0036a0 con 0x7f84fc073590 2024-03-26T18:48:12.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.295+0000 7f84eb7fe700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f84e406c760 0x7f84e406ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:12.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.295+0000 7f84eb7fe700 1 -- 172.21.15.102:0/2354299486 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f84ec08a310 con 0x7f84fc073590 2024-03-26T18:48:12.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.295+0000 7f84fa59c700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f84e406c760 0x7f84e406ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:12.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.295+0000 7f84fa59c700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f84e406c760 0x7f84e406ec10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f84f0006010 tx=0x7f84f000b540 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:48:12.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.299+0000 7f84eb7fe700 1 -- 172.21.15.102:0/2354299486 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f84ec055ad0 con 0x7f84fc073590 2024-03-26T18:48:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.496+0000 7f8500951700 1 -- 172.21.15.102:0/2354299486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f84fc074580 con 0x7f84fc073590 2024-03-26T18:48:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.497+0000 7f84eb7fe700 1 -- 172.21.15.102:0/2354299486 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f84ec089ad0 con 0x7f84fc073590 2024-03-26T18:48:12.488 INFO:teuthology.orchestra.run.smithi102.stdout:# minimal ceph.conf for ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:48:12.488 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-03-26T18:48:12.489 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:48:12.489 INFO:teuthology.orchestra.run.smithi102.stdout: mon_host = [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-03-26T18:48:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.499+0000 7f8500951700 1 -- 172.21.15.102:0/2354299486 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f84e406c760 msgr2=0x7f84e406ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.499+0000 7f8500951700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f84e406c760 0x7f84e406ec10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f84f0006010 tx=0x7f84f000b540 comp rx=0 tx=0).stop 2024-03-26T18:48:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.499+0000 7f8500951700 1 -- 172.21.15.102:0/2354299486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84fc073590 msgr2=0x7f84fc073a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.499+0000 7f8500951700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84fc073590 0x7f84fc073a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00b730 tx=0x7f84ec00baf0 comp rx=0 tx=0).stop 2024-03-26T18:48:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.499+0000 7f8500951700 1 -- 172.21.15.102:0/2354299486 shutdown_connections 2024-03-26T18:48:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.499+0000 7f8500951700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f84e406c760 0x7f84e406ec10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.499+0000 7f8500951700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84fc1000b0 0x7f84fc073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.499+0000 7f8500951700 1 --2- 172.21.15.102:0/2354299486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84fc073590 0x7f84fc073a00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.499+0000 7f8500951700 1 -- 172.21.15.102:0/2354299486 >> 172.21.15.102:0/2354299486 conn(0x7f84fc0fb990 msgr2=0x7f84fc105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:48:12.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.499+0000 7f8500951700 1 -- 172.21.15.102:0/2354299486 shutdown_connections 2024-03-26T18:48:12.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:12.499+0000 7f8500951700 1 -- 172.21.15.102:0/2354299486 wait complete. 2024-03-26T18:48:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:13 smithi190 ceph-mon[35043]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:13 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2354299486' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:13 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:13 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:13 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.rsfpnv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:48:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:13 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:48:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:13 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:13.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:13 smithi102 ceph-mon[29002]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:13.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:13 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2354299486' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:13.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:13 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:13.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:13 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:13.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:13 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.rsfpnv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:48:13.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:13 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:48:13.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:13 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:13.413 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T18:48:13.413 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:48:13.413 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T18:48:13.444 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:48:13.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:48:13.511 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:48:13.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T18:48:13.545 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:48:13.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:48:13.616 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T18:48:13.616 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:48:13.616 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T18:48:13.631 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T18:48:13.631 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-03-26T18:48:13.686 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T18:48:13.686 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T18:48:13.686 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T18:48:13.686 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T18:48:13.686 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T18:48:13.686 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-26 18:47:25.490127049 +0000 2024-03-26T18:48:13.686 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-26 18:41:45.110050150 +0000 2024-03-26T18:48:13.686 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-26 18:41:45.110050150 +0000 2024-03-26T18:48:13.687 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-26 18:41:45.110050150 +0000 2024-03-26T18:48:13.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T18:48:13.753 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-26T18:48:13.753 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-26T18:48:13.754 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000272664 s, 1.9 MB/s 2024-03-26T18:48:13.755 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T18:48:13.812 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-03-26T18:48:13.867 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T18:48:13.867 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T18:48:13.867 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T18:48:13.867 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T18:48:13.867 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T18:48:13.868 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-26 18:47:25.490127049 +0000 2024-03-26T18:48:13.868 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-26 18:41:45.458042861 +0000 2024-03-26T18:48:13.868 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-26 18:41:45.458042861 +0000 2024-03-26T18:48:13.868 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-26 18:41:45.458042861 +0000 2024-03-26T18:48:13.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T18:48:13.934 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-26T18:48:13.934 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-26T18:48:13.934 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000182809 s, 2.8 MB/s 2024-03-26T18:48:13.935 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T18:48:13.991 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-03-26T18:48:14.046 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T18:48:14.046 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T18:48:14.046 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-26T18:48:14.046 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T18:48:14.046 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T18:48:14.046 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-26 18:47:25.491127029 +0000 2024-03-26T18:48:14.047 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-26 18:41:45.803035636 +0000 2024-03-26T18:48:14.047 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-26 18:41:45.803035636 +0000 2024-03-26T18:48:14.047 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-26 18:41:45.803035636 +0000 2024-03-26T18:48:14.047 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T18:48:14.112 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-26T18:48:14.112 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-26T18:48:14.112 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000173602 s, 2.9 MB/s 2024-03-26T18:48:14.113 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T18:48:14.169 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-03-26T18:48:14.224 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T18:48:14.224 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T18:48:14.224 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-26T18:48:14.224 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T18:48:14.224 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T18:48:14.224 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-26 18:47:25.491127029 +0000 2024-03-26T18:48:14.225 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-26 18:41:46.148028409 +0000 2024-03-26T18:48:14.225 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-26 18:41:46.148028409 +0000 2024-03-26T18:48:14.225 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-26 18:41:46.148028409 +0000 2024-03-26T18:48:14.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T18:48:14.287 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:14 smithi102 ceph-mon[29002]: Reconfiguring mgr.smithi102.rsfpnv (unknown last config time)... 2024-03-26T18:48:14.287 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:14 smithi102 ceph-mon[29002]: Reconfiguring daemon mgr.smithi102.rsfpnv on smithi102 2024-03-26T18:48:14.290 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-26T18:48:14.290 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-26T18:48:14.290 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000190288 s, 2.7 MB/s 2024-03-26T18:48:14.291 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T18:48:14.346 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:48:14.347 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T18:48:14.365 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T18:48:14.365 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-03-26T18:48:14.423 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T18:48:14.423 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T18:48:14.423 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T18:48:14.423 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T18:48:14.423 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T18:48:14.423 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-26 18:48:00.448353914 +0000 2024-03-26T18:48:14.423 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-26 18:41:44.994831864 +0000 2024-03-26T18:48:14.423 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-26 18:41:44.994831864 +0000 2024-03-26T18:48:14.423 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-26 18:41:44.994831864 +0000 2024-03-26T18:48:14.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T18:48:14.490 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:14 smithi190 ceph-mon[35043]: Reconfiguring mgr.smithi102.rsfpnv (unknown last config time)... 2024-03-26T18:48:14.490 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:14 smithi190 ceph-mon[35043]: Reconfiguring daemon mgr.smithi102.rsfpnv on smithi102 2024-03-26T18:48:14.492 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-26T18:48:14.493 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-26T18:48:14.493 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000297472 s, 1.7 MB/s 2024-03-26T18:48:14.494 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T18:48:14.551 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-03-26T18:48:14.608 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T18:48:14.608 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T18:48:14.608 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T18:48:14.608 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T18:48:14.609 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T18:48:14.609 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-26 18:48:00.448353914 +0000 2024-03-26T18:48:14.609 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-26 18:41:45.343822031 +0000 2024-03-26T18:48:14.609 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-26 18:41:45.343822031 +0000 2024-03-26T18:48:14.609 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-26 18:41:45.343822031 +0000 2024-03-26T18:48:14.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T18:48:14.675 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-26T18:48:14.675 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-26T18:48:14.675 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000207224 s, 2.5 MB/s 2024-03-26T18:48:14.677 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T18:48:14.734 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-03-26T18:48:14.790 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T18:48:14.790 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T18:48:14.791 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-26T18:48:14.791 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T18:48:14.791 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T18:48:14.791 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-26 18:48:00.449353886 +0000 2024-03-26T18:48:14.791 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-26 18:41:45.682812480 +0000 2024-03-26T18:48:14.791 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-26 18:41:45.682812480 +0000 2024-03-26T18:48:14.791 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-26 18:41:45.682812480 +0000 2024-03-26T18:48:14.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T18:48:14.860 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-26T18:48:14.860 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-26T18:48:14.860 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000227086 s, 2.3 MB/s 2024-03-26T18:48:14.861 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T18:48:14.918 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-03-26T18:48:14.974 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T18:48:14.974 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T18:48:14.974 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T18:48:14.974 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T18:48:14.974 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T18:48:14.975 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-26 18:48:00.449353886 +0000 2024-03-26T18:48:14.975 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-26 18:41:46.024802845 +0000 2024-03-26T18:48:14.975 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-26 18:41:46.024802845 +0000 2024-03-26T18:48:14.975 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-26 18:41:46.024802845 +0000 2024-03-26T18:48:14.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T18:48:15.041 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-26T18:48:15.041 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-26T18:48:15.041 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000208877 s, 2.5 MB/s 2024-03-26T18:48:15.043 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T18:48:15.099 INFO:tasks.cephadm:Deploying osd.0 on smithi102 with /dev/vg_nvme/lv_4... 2024-03-26T18:48:15.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T18:48:15.278 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:15 smithi190 ceph-mon[35043]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:15.278 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:15.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:15.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:48:15.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:15.325 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:15 smithi102 ceph-mon[29002]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:15.325 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:15.325 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:15.325 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:48:15.325 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:48:16.348 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:16 smithi102 ceph-mon[29002]: Reconfiguring ceph-exporter.smithi102 (monmap changed)... 2024-03-26T18:48:16.348 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:16 smithi102 ceph-mon[29002]: Reconfiguring daemon ceph-exporter.smithi102 on smithi102 2024-03-26T18:48:16.348 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:16 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:16.348 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:16 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:16.348 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:16 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:48:16.348 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:16 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:16.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:16 smithi190 ceph-mon[35043]: Reconfiguring ceph-exporter.smithi102 (monmap changed)... 2024-03-26T18:48:16.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:16 smithi190 ceph-mon[35043]: Reconfiguring daemon ceph-exporter.smithi102 on smithi102 2024-03-26T18:48:16.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:16 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:16.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:16 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:16.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:16 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:48:16.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:16 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:17.254 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:17 smithi102 ceph-mon[29002]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:17.254 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:17 smithi102 ceph-mon[29002]: Reconfiguring crash.smithi102 (monmap changed)... 2024-03-26T18:48:17.254 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:17 smithi102 ceph-mon[29002]: Reconfiguring daemon crash.smithi102 on smithi102 2024-03-26T18:48:17.278 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:17 smithi190 ceph-mon[35043]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:17.278 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:17 smithi190 ceph-mon[35043]: Reconfiguring crash.smithi102 (monmap changed)... 2024-03-26T18:48:17.278 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:17 smithi190 ceph-mon[35043]: Reconfiguring daemon crash.smithi102 on smithi102 2024-03-26T18:48:18.350 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T18:48:18.350 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T18:48:18.350 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-26T18:48:18.350 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-26T18:48:18.350 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148288 s, 707 MB/s 2024-03-26T18:48:18.350 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-26T18:48:19.097 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi102:vg_nvme/lv_4 2024-03-26T18:48:19.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:19 smithi102 ceph-mon[29002]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:19.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:19.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:19.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:48:19.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:19 smithi190 ceph-mon[35043]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:19.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:19.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:19.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:48:20.232 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:20 smithi102 ceph-mon[29002]: Reconfiguring alertmanager.smithi102 (dependencies changed)... 2024-03-26T18:48:20.232 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:20 smithi102 ceph-mon[29002]: Reconfiguring daemon alertmanager.smithi102 on smithi102 2024-03-26T18:48:20.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:20 smithi190 ceph-mon[35043]: Reconfiguring alertmanager.smithi102 (dependencies changed)... 2024-03-26T18:48:20.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:20 smithi190 ceph-mon[35043]: Reconfiguring daemon alertmanager.smithi102 on smithi102 2024-03-26T18:48:21.323 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:21 smithi102 ceph-mon[29002]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:21.424 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:48:21.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:21 smithi190 ceph-mon[35043]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.910+0000 7f7bc62f7700 1 -- 172.21.15.102:0/1619706333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc0103560 msgr2=0x7f7bc0105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.910+0000 7f7bc62f7700 1 --2- 172.21.15.102:0/1619706333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc0103560 0x7f7bc0105940 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009910 tx=0x7f7bb4009c20 comp rx=0 tx=0).stop 2024-03-26T18:48:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.910+0000 7f7bc62f7700 1 -- 172.21.15.102:0/1619706333 shutdown_connections 2024-03-26T18:48:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.910+0000 7f7bc62f7700 1 --2- 172.21.15.102:0/1619706333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc0103560 0x7f7bc0105940 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.910+0000 7f7bc62f7700 1 --2- 172.21.15.102:0/1619706333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bc0100c40 0x7f7bc0103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.910+0000 7f7bc62f7700 1 -- 172.21.15.102:0/1619706333 >> 172.21.15.102:0/1619706333 conn(0x7f7bc00fa5e0 msgr2=0x7f7bc00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:48:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.910+0000 7f7bc62f7700 1 -- 172.21.15.102:0/1619706333 shutdown_connections 2024-03-26T18:48:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.910+0000 7f7bc62f7700 1 -- 172.21.15.102:0/1619706333 wait complete. 2024-03-26T18:48:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.911+0000 7f7bc62f7700 1 Processor -- start 2024-03-26T18:48:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.911+0000 7f7bc62f7700 1 -- start start 2024-03-26T18:48:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.911+0000 7f7bc62f7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc0100c40 0x7f7bc0195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.911+0000 7f7bc62f7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bc0103560 0x7f7bc0195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.911+0000 7f7bc62f7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc01964b0 con 0x7f7bc0100c40 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.911+0000 7f7bc62f7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc01965f0 con 0x7f7bc0103560 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.911+0000 7f7bbffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc0100c40 0x7f7bc0195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.911+0000 7f7bbffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc0100c40 0x7f7bc0195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39780/0 (socket says 172.21.15.102:39780) 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.911+0000 7f7bbffff700 1 -- 172.21.15.102:0/2354909389 learned_addr learned my addr 172.21.15.102:0/2354909389 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.912+0000 7f7bbf7fe700 1 --2- 172.21.15.102:0/2354909389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bc0103560 0x7f7bc0195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.912+0000 7f7bbffff700 1 -- 172.21.15.102:0/2354909389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bc0103560 msgr2=0x7f7bc0195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.912+0000 7f7bbffff700 1 --2- 172.21.15.102:0/2354909389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bc0103560 0x7f7bc0195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.912+0000 7f7bbffff700 1 -- 172.21.15.102:0/2354909389 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb40095f0 con 0x7f7bc0100c40 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.912+0000 7f7bbffff700 1 --2- 172.21.15.102:0/2354909389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc0100c40 0x7f7bc0195950 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7bb000d8d0 tx=0x7f7bb000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:48:22.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.912+0000 7f7bbd7fa700 1 -- 172.21.15.102:0/2354909389 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bb0009940 con 0x7f7bc0100c40 2024-03-26T18:48:22.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.912+0000 7f7bbd7fa700 1 -- 172.21.15.102:0/2354909389 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7bb0010460 con 0x7f7bc0100c40 2024-03-26T18:48:22.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.912+0000 7f7bc62f7700 1 -- 172.21.15.102:0/2354909389 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc019b0a0 con 0x7f7bc0100c40 2024-03-26T18:48:22.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.912+0000 7f7bbd7fa700 1 -- 172.21.15.102:0/2354909389 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bb000f5f0 con 0x7f7bc0100c40 2024-03-26T18:48:22.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.912+0000 7f7bc62f7700 1 -- 172.21.15.102:0/2354909389 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc019b5f0 con 0x7f7bc0100c40 2024-03-26T18:48:22.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.914+0000 7f7bbd7fa700 1 -- 172.21.15.102:0/2354909389 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7f7bb000f7e0 con 0x7f7bc0100c40 2024-03-26T18:48:22.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.914+0000 7f7bc62f7700 1 -- 172.21.15.102:0/2354909389 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc018fb70 con 0x7f7bc0100c40 2024-03-26T18:48:22.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.914+0000 7f7bbd7fa700 1 --2- 172.21.15.102:0/2354909389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ba806c600 0x7f7ba806eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:22.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.914+0000 7f7bbd7fa700 1 -- 172.21.15.102:0/2354909389 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f7bb008a6f0 con 0x7f7bc0100c40 2024-03-26T18:48:22.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.914+0000 7f7bbf7fe700 1 --2- 172.21.15.102:0/2354909389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ba806c600 0x7f7ba806eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:22.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.915+0000 7f7bbf7fe700 1 --2- 172.21.15.102:0/2354909389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ba806c600 0x7f7ba806eab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005b40 tx=0x7f7bb4005a90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:48:22.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:22.919+0000 7f7bbd7fa700 1 -- 172.21.15.102:0/2354909389 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7bb0059b80 con 0x7f7bc0100c40 2024-03-26T18:48:23.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:23.130+0000 7f7bc62f7700 1 -- 172.21.15.102:0/2354909389 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bc0061190 con 0x7f7ba806c600 2024-03-26T18:48:23.362 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:23 smithi102 ceph-mon[29002]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:23.362 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:23 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:48:23.362 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:23 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:48:23.362 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:23 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:23.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:23 smithi190 ceph-mon[35043]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:23.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:23 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:48:23.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:23 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:48:23.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:23 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:24.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:24 smithi102 ceph-mon[29002]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:48:24.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:24 smithi190 ceph-mon[35043]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:48:25.425 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:25 smithi102 ceph-mon[29002]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:25.425 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:25.425 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:25.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:25 smithi190 ceph-mon[35043]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:25.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:25.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:26.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:26 smithi102 ceph-mon[29002]: Reconfiguring grafana.smithi102 (dependencies changed)... 2024-03-26T18:48:26.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:26 smithi102 ceph-mon[29002]: Reconfiguring daemon grafana.smithi102 on smithi102 2024-03-26T18:48:26.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:26 smithi190 ceph-mon[35043]: Reconfiguring grafana.smithi102 (dependencies changed)... 2024-03-26T18:48:26.538 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:26 smithi190 ceph-mon[35043]: Reconfiguring daemon grafana.smithi102 on smithi102 2024-03-26T18:48:27.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:27 smithi190 ceph-mon[35043]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:27.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:27 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3455373970' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5458d389-eb88-4798-92bc-b442168b37f4"}]: dispatch 2024-03-26T18:48:27.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:27 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3455373970' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5458d389-eb88-4798-92bc-b442168b37f4"}]': finished 2024-03-26T18:48:27.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:27 smithi190 ceph-mon[35043]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T18:48:27.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:27 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:27.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:27 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2854902588' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:48:27.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:27 smithi102 ceph-mon[29002]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:27.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:27 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3455373970' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5458d389-eb88-4798-92bc-b442168b37f4"}]: dispatch 2024-03-26T18:48:27.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:27 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3455373970' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5458d389-eb88-4798-92bc-b442168b37f4"}]': finished 2024-03-26T18:48:27.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:27 smithi102 ceph-mon[29002]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T18:48:27.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:27 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:27.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:27 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2854902588' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:48:29.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:29 smithi190 ceph-mon[35043]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:29.594 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:29 smithi102 ceph-mon[29002]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:31.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:31 smithi190 ceph-mon[35043]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:31.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:31 smithi102 ceph-mon[29002]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:33.313 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:33 smithi102 ceph-mon[29002]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:33.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:33 smithi190 ceph-mon[35043]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:34.268 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:48:34.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:48:35.455 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:35 smithi102 ceph-mon[29002]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:35.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:35 smithi190 ceph-mon[35043]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:36.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:36 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:36.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:36 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:36.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:36 smithi102 ceph-mon[29002]: Reconfiguring prometheus.smithi102 (dependencies changed)... 2024-03-26T18:48:36.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:36 smithi102 ceph-mon[29002]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:36.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:36 smithi102 ceph-mon[29002]: Reconfiguring daemon prometheus.smithi102 on smithi102 2024-03-26T18:48:36.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:36 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:36.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:36 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:36.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:36 smithi190 ceph-mon[35043]: Reconfiguring prometheus.smithi102 (dependencies changed)... 2024-03-26T18:48:36.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:36 smithi190 ceph-mon[35043]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:36.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:36 smithi190 ceph-mon[35043]: Reconfiguring daemon prometheus.smithi102 on smithi102 2024-03-26T18:48:38.028 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:37 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T18:48:38.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:37 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:38.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:37 smithi190 ceph-mon[35043]: Deploying daemon osd.0 on smithi102 2024-03-26T18:48:38.102 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:37 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T18:48:38.102 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:37 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:38.102 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:37 smithi102 ceph-mon[29002]: Deploying daemon osd.0 on smithi102 2024-03-26T18:48:38.844 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:38 smithi102 ceph-mon[29002]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:39.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:38 smithi190 ceph-mon[35043]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:41.359 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:41 smithi102 ceph-mon[29002]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:41.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:41 smithi190 ceph-mon[35043]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:42 smithi190 ceph-mon[35043]: Reconfiguring ceph-exporter.smithi190 (monmap changed)... 2024-03-26T18:48:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:48:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:42 smithi190 ceph-mon[35043]: Reconfiguring daemon ceph-exporter.smithi190 on smithi190 2024-03-26T18:48:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:42 smithi190 ceph-mon[35043]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:42 smithi102 ceph-mon[29002]: Reconfiguring ceph-exporter.smithi190 (monmap changed)... 2024-03-26T18:48:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:48:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:42 smithi102 ceph-mon[29002]: Reconfiguring daemon ceph-exporter.smithi190 on smithi190 2024-03-26T18:48:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:42 smithi102 ceph-mon[29002]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:43.972 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:43 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:43.972 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:43 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:43.972 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:43 smithi102 ceph-mon[29002]: Reconfiguring crash.smithi190 (monmap changed)... 2024-03-26T18:48:43.972 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:43 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:48:43.972 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:43 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:43.972 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:43 smithi102 ceph-mon[29002]: Reconfiguring daemon crash.smithi190 on smithi190 2024-03-26T18:48:44.248 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:43 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:44.248 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:43 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:44.248 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:43 smithi190 ceph-mon[35043]: Reconfiguring crash.smithi190 (monmap changed)... 2024-03-26T18:48:44.248 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:43 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:48:44.248 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:43 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:44.248 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:43 smithi190 ceph-mon[35043]: Reconfiguring daemon crash.smithi190 on smithi190 2024-03-26T18:48:45.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:44 smithi190 ceph-mon[35043]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:45.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:44 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:45.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:44 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:45.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:44 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.zrixos", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:48:45.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:44 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:48:45.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:44 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:45.274 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:44 smithi102 ceph-mon[29002]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:45.274 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:44 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:45.274 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:44 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:45.274 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:44 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.zrixos", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:48:45.275 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:44 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:48:45.275 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:44 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:46.316 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:46 smithi102 ceph-mon[29002]: Reconfiguring mgr.smithi190.zrixos (monmap changed)... 2024-03-26T18:48:46.317 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:46 smithi102 ceph-mon[29002]: Reconfiguring daemon mgr.smithi190.zrixos on smithi190 2024-03-26T18:48:46.317 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:46 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:46.317 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:46 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:46.317 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:46 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:48:46.317 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:46 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:48:46.317 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:46 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:46.465 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:46 smithi190 ceph-mon[35043]: Reconfiguring mgr.smithi190.zrixos (monmap changed)... 2024-03-26T18:48:46.465 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:46 smithi190 ceph-mon[35043]: Reconfiguring daemon mgr.smithi190.zrixos on smithi190 2024-03-26T18:48:46.466 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:46 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:46.466 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:46 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:46.466 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:46 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:48:46.466 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:46 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:48:46.466 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:46 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:47.227 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:47 smithi190 ceph-mon[35043]: Reconfiguring mon.smithi190 (monmap changed)... 2024-03-26T18:48:47.227 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:47 smithi190 ceph-mon[35043]: Reconfiguring daemon mon.smithi190 on smithi190 2024-03-26T18:48:47.227 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:47 smithi190 ceph-mon[35043]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:47.227 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:47 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:47.227 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:47 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:47.343 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:47 smithi102 ceph-mon[29002]: Reconfiguring mon.smithi190 (monmap changed)... 2024-03-26T18:48:47.343 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:47 smithi102 ceph-mon[29002]: Reconfiguring daemon mon.smithi190 on smithi190 2024-03-26T18:48:47.343 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:47 smithi102 ceph-mon[29002]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:47.343 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:47.343 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T18:48:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T18:48:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T18:48:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T18:48:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T18:48:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T18:48:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi102.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T18:48:48.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi102.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T18:48:48.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:48.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:48:48.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:48:48.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi102.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T18:48:48.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi102.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T18:48:48.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:48.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:48:48.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:48 smithi190 ceph-mon[35043]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:48.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:48.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:48.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T18:48:48.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T18:48:48.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T18:48:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T18:48:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T18:48:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T18:48:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi102.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T18:48:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi102.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T18:48:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:48:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:48:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi102.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T18:48:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi102.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T18:48:48.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:48.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:48:48.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:48 smithi102 ceph-mon[29002]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:49.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:49 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:48:49.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:49 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:49.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:49 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:49.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.764+0000 7f7bbd7fa700 1 -- 172.21.15.102:0/2354909389 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7bc0061190 con 0x7f7ba806c600 2024-03-26T18:48:49.756 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 0 on host 'smithi102' 2024-03-26T18:48:49.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.766+0000 7f7bc62f7700 1 -- 172.21.15.102:0/2354909389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ba806c600 msgr2=0x7f7ba806eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:49.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.766+0000 7f7bc62f7700 1 --2- 172.21.15.102:0/2354909389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ba806c600 0x7f7ba806eab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005b40 tx=0x7f7bb4005a90 comp rx=0 tx=0).stop 2024-03-26T18:48:49.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.766+0000 7f7bc62f7700 1 -- 172.21.15.102:0/2354909389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc0100c40 msgr2=0x7f7bc0195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.766+0000 7f7bc62f7700 1 --2- 172.21.15.102:0/2354909389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc0100c40 0x7f7bc0195950 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7bb000d8d0 tx=0x7f7bb000dc90 comp rx=0 tx=0).stop 2024-03-26T18:48:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.766+0000 7f7bc62f7700 1 -- 172.21.15.102:0/2354909389 shutdown_connections 2024-03-26T18:48:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.766+0000 7f7bc62f7700 1 --2- 172.21.15.102:0/2354909389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ba806c600 0x7f7ba806eab0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.766+0000 7f7bc62f7700 1 --2- 172.21.15.102:0/2354909389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc0100c40 0x7f7bc0195950 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.766+0000 7f7bc62f7700 1 --2- 172.21.15.102:0/2354909389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bc0103560 0x7f7bc0195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.766+0000 7f7bc62f7700 1 -- 172.21.15.102:0/2354909389 >> 172.21.15.102:0/2354909389 conn(0x7f7bc00fa5e0 msgr2=0x7f7bc00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:48:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.766+0000 7f7bc62f7700 1 -- 172.21.15.102:0/2354909389 shutdown_connections 2024-03-26T18:48:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:49.766+0000 7f7bc62f7700 1 -- 172.21.15.102:0/2354909389 wait complete. 2024-03-26T18:48:49.778 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:49 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:48:49.778 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:49 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:49.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:49 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:50.434 DEBUG:teuthology.orchestra.run.smithi102:osd.0> sudo journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T18:48:50.436 INFO:tasks.cephadm:Deploying osd.1 on smithi102 with /dev/vg_nvme/lv_3... 2024-03-26T18:48:50.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T18:48:51.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:50 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:51.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:50 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:51.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:50 smithi190 ceph-mon[35043]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:51.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:50 smithi190 ceph-mon[35043]: from='osd.0 [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T18:48:51.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:50 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:51.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:50 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:51.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:50 smithi102 ceph-mon[29002]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:51.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:50 smithi102 ceph-mon[29002]: from='osd.0 [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T18:48:51.771 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:48:52.199 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:52 smithi102 ceph-mon[29002]: from='osd.0 [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T18:48:52.199 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:52 smithi102 ceph-mon[29002]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T18:48:52.199 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:52 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:52.199 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:52 smithi102 ceph-mon[29002]: from='osd.0 [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:48:52.199 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:48:52 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[43870]: 2024-03-26T18:48:52.142+0000 7f520d37c700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T18:48:52.199 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:48:52 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[43870]: 2024-03-26T18:48:52.148+0000 7f520616b700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T18:48:52.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:52 smithi190 ceph-mon[35043]: from='osd.0 [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T18:48:52.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:52 smithi190 ceph-mon[35043]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T18:48:52.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:52 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:52.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:52 smithi190 ceph-mon[35043]: from='osd.0 [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:48:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:53 smithi102 ceph-mon[29002]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:53 smithi102 ceph-mon[29002]: from='osd.0 [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-26T18:48:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:53 smithi102 ceph-mon[29002]: osdmap e8: 1 total, 0 up, 1 in 2024-03-26T18:48:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:53 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:53 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:53 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:53 smithi190 ceph-mon[35043]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:53 smithi190 ceph-mon[35043]: from='osd.0 [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-26T18:48:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:53 smithi190 ceph-mon[35043]: osdmap e8: 1 total, 0 up, 1 in 2024-03-26T18:48:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:53 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:53 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:53 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:54.001 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T18:48:54.001 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T18:48:54.001 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-26T18:48:54.001 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-26T18:48:54.002 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139806 s, 750 MB/s 2024-03-26T18:48:54.002 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-26T18:48:54.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:54 smithi102 ceph-mon[29002]: purged_snaps scrub starts 2024-03-26T18:48:54.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:54 smithi102 ceph-mon[29002]: purged_snaps scrub ok 2024-03-26T18:48:54.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:54 smithi102 ceph-mon[29002]: osd.0 [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310] boot 2024-03-26T18:48:54.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:54 smithi102 ceph-mon[29002]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T18:48:54.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:54.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:54 smithi190 ceph-mon[35043]: purged_snaps scrub starts 2024-03-26T18:48:54.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:54 smithi190 ceph-mon[35043]: purged_snaps scrub ok 2024-03-26T18:48:54.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:54 smithi190 ceph-mon[35043]: osd.0 [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310] boot 2024-03-26T18:48:54.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:54 smithi190 ceph-mon[35043]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T18:48:54.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:54 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:48:54.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi102:vg_nvme/lv_3 2024-03-26T18:48:55.072 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:48:55.195 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:55 smithi102 ceph-mon[29002]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:55.196 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:55 smithi102 ceph-mon[29002]: osdmap e10: 1 total, 1 up, 1 in 2024-03-26T18:48:55.196 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:55 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:55.196 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:55 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:55.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:55 smithi190 ceph-mon[35043]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T18:48:55.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:55 smithi190 ceph-mon[35043]: osdmap e10: 1 total, 1 up, 1 in 2024-03-26T18:48:55.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:55 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:55.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:55 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:48:56.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.639+0000 7f1e27493700 1 -- 172.21.15.102:0/4182162325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e200fe820 msgr2=0x7f1e200fec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:56.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.639+0000 7f1e27493700 1 --2- 172.21.15.102:0/4182162325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e200fe820 0x7f1e200fec30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800b0a0 tx=0x7f1e1800b3b0 comp rx=0 tx=0).stop 2024-03-26T18:48:56.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.639+0000 7f1e27493700 1 -- 172.21.15.102:0/4182162325 shutdown_connections 2024-03-26T18:48:56.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.639+0000 7f1e27493700 1 --2- 172.21.15.102:0/4182162325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e200ffa20 0x7f1e200ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:56.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.639+0000 7f1e27493700 1 --2- 172.21.15.102:0/4182162325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e200fe820 0x7f1e200fec30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.639+0000 7f1e27493700 1 -- 172.21.15.102:0/4182162325 >> 172.21.15.102:0/4182162325 conn(0x7f1e200f9db0 msgr2=0x7f1e200fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.639+0000 7f1e27493700 1 -- 172.21.15.102:0/4182162325 shutdown_connections 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.640+0000 7f1e27493700 1 -- 172.21.15.102:0/4182162325 wait complete. 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.640+0000 7f1e27493700 1 Processor -- start 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.640+0000 7f1e27493700 1 -- start start 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.640+0000 7f1e27493700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e200fe820 0x7f1e20195a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.640+0000 7f1e27493700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e200ffa20 0x7f1e20195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.640+0000 7f1e27493700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e20196570 con 0x7f1e200ffa20 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.640+0000 7f1e27493700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e201966b0 con 0x7f1e200fe820 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.640+0000 7f1e24a2e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e200ffa20 0x7f1e20195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.640+0000 7f1e24a2e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e200ffa20 0x7f1e20195f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39292/0 (socket says 172.21.15.102:39292) 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.640+0000 7f1e24a2e700 1 -- 172.21.15.102:0/3740692582 learned_addr learned my addr 172.21.15.102:0/3740692582 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:48:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.640+0000 7f1e2522f700 1 --2- 172.21.15.102:0/3740692582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e200fe820 0x7f1e20195a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.641+0000 7f1e24a2e700 1 -- 172.21.15.102:0/3740692582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e200fe820 msgr2=0x7f1e20195a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:48:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.641+0000 7f1e24a2e700 1 --2- 172.21.15.102:0/3740692582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e200fe820 0x7f1e20195a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:48:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.641+0000 7f1e24a2e700 1 -- 172.21.15.102:0/3740692582 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e18009d00 con 0x7f1e200ffa20 2024-03-26T18:48:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.641+0000 7f1e24a2e700 1 --2- 172.21.15.102:0/3740692582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e200ffa20 0x7f1e20195f50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1e1000c8a0 tx=0x7f1e1000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:48:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.641+0000 7f1e167fc700 1 -- 172.21.15.102:0/3740692582 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e1000cea0 con 0x7f1e200ffa20 2024-03-26T18:48:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.641+0000 7f1e167fc700 1 -- 172.21.15.102:0/3740692582 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1e10004830 con 0x7f1e200ffa20 2024-03-26T18:48:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.641+0000 7f1e27493700 1 -- 172.21.15.102:0/3740692582 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2019b110 con 0x7f1e200ffa20 2024-03-26T18:48:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.641+0000 7f1e167fc700 1 -- 172.21.15.102:0/3740692582 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e100055b0 con 0x7f1e200ffa20 2024-03-26T18:48:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.641+0000 7f1e27493700 1 -- 172.21.15.102:0/3740692582 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2019b6e0 con 0x7f1e200ffa20 2024-03-26T18:48:56.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.642+0000 7f1e27493700 1 -- 172.21.15.102:0/3740692582 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e20107370 con 0x7f1e200ffa20 2024-03-26T18:48:56.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.642+0000 7f1e167fc700 1 -- 172.21.15.102:0/3740692582 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7f1e10007ad0 con 0x7f1e200ffa20 2024-03-26T18:48:56.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.643+0000 7f1e167fc700 1 --2- 172.21.15.102:0/3740692582 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1e0c06c600 0x7f1e0c06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:48:56.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.643+0000 7f1e167fc700 1 -- 172.21.15.102:0/3740692582 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f1e1008a730 con 0x7f1e200ffa20 2024-03-26T18:48:56.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.643+0000 7f1e2522f700 1 --2- 172.21.15.102:0/3740692582 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1e0c06c600 0x7f1e0c06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:48:56.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.643+0000 7f1e2522f700 1 --2- 172.21.15.102:0/3740692582 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1e0c06c600 0x7f1e0c06eab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1e18000f80 tx=0x7f1e1800b7b0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:48:56.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.647+0000 7f1e167fc700 1 -- 172.21.15.102:0/3740692582 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1e10059720 con 0x7f1e200ffa20 2024-03-26T18:48:56.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:48:56.856+0000 7f1e27493700 1 -- 172.21.15.102:0/3740692582 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e200fbfd0 con 0x7f1e0c06c600 2024-03-26T18:48:57.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:57 smithi190 ceph-mon[35043]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:48:57.313 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:57 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:48:57.313 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:57 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:48:57.314 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:57 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:57.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:57 smithi102 ceph-mon[29002]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:48:57.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:57 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:48:57.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:57 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:48:57.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:57 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:48:58.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:58 smithi102 ceph-mon[29002]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:48:58.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:58 smithi190 ceph-mon[35043]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:48:59.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:48:59 smithi102 ceph-mon[29002]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:48:59.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:48:59 smithi190 ceph-mon[35043]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:00.601 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:00 smithi102 ceph-mon[29002]: Detected new or changed devices on smithi102 2024-03-26T18:49:00.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:00.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:00.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:00.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:00 smithi102 ceph-mon[29002]: Adjusting osd_memory_target on smithi102 to 11840M 2024-03-26T18:49:00.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:00.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:00.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:00.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:00.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:49:00.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:00 smithi102 ceph-mon[29002]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:00.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:00 smithi190 ceph-mon[35043]: Detected new or changed devices on smithi102 2024-03-26T18:49:00.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:00.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:00.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:00.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:00 smithi190 ceph-mon[35043]: Adjusting osd_memory_target on smithi102 to 11840M 2024-03-26T18:49:00.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:00.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:00.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:00.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:00.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:49:00.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:00 smithi190 ceph-mon[35043]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:01.622 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:01 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/4051128261' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87508f24-8b3c-4f6d-add2-00ad59300da8"}]: dispatch 2024-03-26T18:49:01.622 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:01 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/4051128261' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87508f24-8b3c-4f6d-add2-00ad59300da8"}]': finished 2024-03-26T18:49:01.622 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:01 smithi102 ceph-mon[29002]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T18:49:01.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:01 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:01.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:01 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:01.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:01 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:01.778 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:01 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/4051128261' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87508f24-8b3c-4f6d-add2-00ad59300da8"}]: dispatch 2024-03-26T18:49:01.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:01 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/4051128261' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87508f24-8b3c-4f6d-add2-00ad59300da8"}]': finished 2024-03-26T18:49:01.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:01 smithi190 ceph-mon[35043]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T18:49:01.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:01 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:01.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:01 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:01.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:01 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:02.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:02 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2514504426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:49:02.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:02 smithi190 ceph-mon[35043]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:02.850 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:02 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2514504426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:49:02.851 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:02 smithi102 ceph-mon[29002]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:04.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:04 smithi102 ceph-mon[29002]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:04.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:04 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:04.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:04 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:04.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:04 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:04.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:04 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:04.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:04 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:04.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:04 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:49:05.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:04 smithi190 ceph-mon[35043]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:05.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:04 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:05.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:04 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:05.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:04 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:05.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:04 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:05.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:04 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:05.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:04 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:49:07.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:07 smithi190 ceph-mon[35043]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:07.312 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:07 smithi102 ceph-mon[29002]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:08.449 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:08 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T18:49:08.449 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:08 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:08.576 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:08 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T18:49:08.576 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:08 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:09.457 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:09 smithi102 ceph-mon[29002]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:09.457 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:09 smithi102 ceph-mon[29002]: Deploying daemon osd.1 on smithi102 2024-03-26T18:49:09.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:09 smithi190 ceph-mon[35043]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:09.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:09 smithi190 ceph-mon[35043]: Deploying daemon osd.1 on smithi102 2024-03-26T18:49:11.400 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:11 smithi102 ceph-mon[29002]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:11.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:11 smithi190 ceph-mon[35043]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:13.454 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:13 smithi102 ceph-mon[29002]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:13.454 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:13 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:13.454 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:13 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:13.454 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:13 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:49:13.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:13 smithi190 ceph-mon[35043]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:13.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:13 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:13.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:13 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:13.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:13 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:49:15.424 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:15 smithi102 ceph-mon[29002]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:15.424 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:15.424 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:15.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:15 smithi190 ceph-mon[35043]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:15.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:15.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:16.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.666+0000 7f1e167fc700 1 -- 172.21.15.102:0/3740692582 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1e200fbfd0 con 0x7f1e0c06c600 2024-03-26T18:49:16.659 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 1 on host 'smithi102' 2024-03-26T18:49:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.669+0000 7f1e27493700 1 -- 172.21.15.102:0/3740692582 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1e0c06c600 msgr2=0x7f1e0c06eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:49:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.669+0000 7f1e27493700 1 --2- 172.21.15.102:0/3740692582 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1e0c06c600 0x7f1e0c06eab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1e18000f80 tx=0x7f1e1800b7b0 comp rx=0 tx=0).stop 2024-03-26T18:49:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.669+0000 7f1e27493700 1 -- 172.21.15.102:0/3740692582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e200ffa20 msgr2=0x7f1e20195f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:49:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.669+0000 7f1e27493700 1 --2- 172.21.15.102:0/3740692582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e200ffa20 0x7f1e20195f50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1e1000c8a0 tx=0x7f1e1000cc60 comp rx=0 tx=0).stop 2024-03-26T18:49:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.669+0000 7f1e27493700 1 -- 172.21.15.102:0/3740692582 shutdown_connections 2024-03-26T18:49:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.669+0000 7f1e27493700 1 --2- 172.21.15.102:0/3740692582 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1e0c06c600 0x7f1e0c06eab0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.669+0000 7f1e27493700 1 --2- 172.21.15.102:0/3740692582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e200fe820 0x7f1e20195a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.669+0000 7f1e27493700 1 --2- 172.21.15.102:0/3740692582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e200ffa20 0x7f1e20195f50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.669+0000 7f1e27493700 1 -- 172.21.15.102:0/3740692582 >> 172.21.15.102:0/3740692582 conn(0x7f1e200f9db0 msgr2=0x7f1e200fb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:49:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.669+0000 7f1e27493700 1 -- 172.21.15.102:0/3740692582 shutdown_connections 2024-03-26T18:49:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:16.669+0000 7f1e27493700 1 -- 172.21.15.102:0/3740692582 wait complete. 2024-03-26T18:49:17.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:17 smithi190 ceph-mon[35043]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:17.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:17 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:17.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:17 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:17.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:17 smithi190 ceph-mon[35043]: from='osd.1 [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T18:49:17.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:17 smithi102 ceph-mon[29002]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:17.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:17 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:17.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:17 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:17.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:17 smithi102 ceph-mon[29002]: from='osd.1 [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T18:49:17.597 DEBUG:teuthology.orchestra.run.smithi102:osd.1> sudo journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T18:49:17.599 INFO:tasks.cephadm:Deploying osd.2 on smithi102 with /dev/vg_nvme/lv_2... 2024-03-26T18:49:17.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T18:49:18.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:18 smithi102 ceph-mon[29002]: from='osd.1 [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T18:49:18.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:18 smithi102 ceph-mon[29002]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T18:49:18.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:18 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:18.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:18 smithi102 ceph-mon[29002]: from='osd.1 [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:49:18.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:18 smithi102 ceph-mon[29002]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:18.875 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:49:18 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[49428]: 2024-03-26T18:49:18.635+0000 7fcce81b6700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T18:49:18.875 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:49:18 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[49428]: 2024-03-26T18:49:18.638+0000 7fcce47ac700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T18:49:19.011 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:49:19.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:18 smithi190 ceph-mon[35043]: from='osd.1 [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T18:49:19.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:18 smithi190 ceph-mon[35043]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T18:49:19.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:18 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:19.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:18 smithi190 ceph-mon[35043]: from='osd.1 [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:49:19.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:18 smithi190 ceph-mon[35043]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:19.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:19 smithi102 ceph-mon[29002]: from='osd.1 [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-26T18:49:19.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:19 smithi102 ceph-mon[29002]: osdmap e13: 2 total, 1 up, 2 in 2024-03-26T18:49:19.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:19.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:19.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:49:19.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:19.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:19 smithi190 ceph-mon[35043]: from='osd.1 [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-26T18:49:19.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:19 smithi190 ceph-mon[35043]: osdmap e13: 2 total, 1 up, 2 in 2024-03-26T18:49:19.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:19.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:19.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:49:19.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:20.980 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:20 smithi102 ceph-mon[29002]: purged_snaps scrub starts 2024-03-26T18:49:20.980 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:20 smithi102 ceph-mon[29002]: purged_snaps scrub ok 2024-03-26T18:49:20.980 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:20 smithi102 ceph-mon[29002]: osd.1 [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241] boot 2024-03-26T18:49:20.980 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:20 smithi102 ceph-mon[29002]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T18:49:20.980 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:20.980 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:20 smithi102 ceph-mon[29002]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:20.980 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:21.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:20 smithi190 ceph-mon[35043]: purged_snaps scrub starts 2024-03-26T18:49:21.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:20 smithi190 ceph-mon[35043]: purged_snaps scrub ok 2024-03-26T18:49:21.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:20 smithi190 ceph-mon[35043]: osd.1 [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241] boot 2024-03-26T18:49:21.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:20 smithi190 ceph-mon[35043]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T18:49:21.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:49:21.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:20 smithi190 ceph-mon[35043]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T18:49:21.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:21.386 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T18:49:21.386 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T18:49:21.386 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-26T18:49:21.386 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-26T18:49:21.386 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124074 s, 845 MB/s 2024-03-26T18:49:21.386 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-26T18:49:21.995 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:21 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:21.995 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:21 smithi190 ceph-mon[35043]: osdmap e15: 2 total, 2 up, 2 in 2024-03-26T18:49:22.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:21 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:22.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:21 smithi102 ceph-mon[29002]: osdmap e15: 2 total, 2 up, 2 in 2024-03-26T18:49:22.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi102:vg_nvme/lv_2 2024-03-26T18:49:22.824 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:22 smithi102 ceph-mon[29002]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:23.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:22 smithi190 ceph-mon[35043]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:49:25.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.429+0000 7f1ba6b27700 1 -- 172.21.15.102:0/2863097526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba0100270 msgr2=0x7f1ba01006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:49:25.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.429+0000 7f1ba6b27700 1 --2- 172.21.15.102:0/2863097526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba0100270 0x7f1ba01006c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009910 tx=0x7f1b90009c20 comp rx=0 tx=0).stop 2024-03-26T18:49:25.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.429+0000 7f1ba6b27700 1 -- 172.21.15.102:0/2863097526 shutdown_connections 2024-03-26T18:49:25.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.429+0000 7f1ba6b27700 1 --2- 172.21.15.102:0/2863097526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba0100270 0x7f1ba01006c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:25.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.429+0000 7f1ba6b27700 1 --2- 172.21.15.102:0/2863097526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ba00ff070 0x7f1ba00ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:25.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.429+0000 7f1ba6b27700 1 -- 172.21.15.102:0/2863097526 >> 172.21.15.102:0/2863097526 conn(0x7f1ba00fa600 msgr2=0x7f1ba00fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:49:25.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.429+0000 7f1ba6b27700 1 -- 172.21.15.102:0/2863097526 shutdown_connections 2024-03-26T18:49:25.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.429+0000 7f1ba6b27700 1 -- 172.21.15.102:0/2863097526 wait complete. 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1ba6b27700 1 Processor -- start 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1ba6b27700 1 -- start start 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1ba6b27700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ba00ff070 0x7f1ba0195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1ba6b27700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba0100270 0x7f1ba0195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1ba6b27700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba01964e0 con 0x7f1ba0100270 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1ba6b27700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba0196620 con 0x7f1ba00ff070 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1b9ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba0100270 0x7f1ba0195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1b9ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba0100270 0x7f1ba0195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55882/0 (socket says 172.21.15.102:55882) 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1b9ffff700 1 -- 172.21.15.102:0/3407221681 learned_addr learned my addr 172.21.15.102:0/3407221681 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1ba48c3700 1 --2- 172.21.15.102:0/3407221681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ba00ff070 0x7f1ba0195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1b9ffff700 1 -- 172.21.15.102:0/3407221681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ba00ff070 msgr2=0x7f1ba0195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1b9ffff700 1 --2- 172.21.15.102:0/3407221681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ba00ff070 0x7f1ba0195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1b9ffff700 1 -- 172.21.15.102:0/3407221681 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b900095f0 con 0x7f1ba0100270 2024-03-26T18:49:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1b9ffff700 1 --2- 172.21.15.102:0/3407221681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba0100270 0x7f1ba0195ec0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1b90006010 tx=0x7f1b9000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:49:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.430+0000 7f1b9dffb700 1 -- 172.21.15.102:0/3407221681 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b9001d070 con 0x7f1ba0100270 2024-03-26T18:49:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.431+0000 7f1ba6b27700 1 -- 172.21.15.102:0/3407221681 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba019b070 con 0x7f1ba0100270 2024-03-26T18:49:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.431+0000 7f1b9dffb700 1 -- 172.21.15.102:0/3407221681 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1b9000f460 con 0x7f1ba0100270 2024-03-26T18:49:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.431+0000 7f1b9dffb700 1 -- 172.21.15.102:0/3407221681 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b90021640 con 0x7f1ba0100270 2024-03-26T18:49:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.431+0000 7f1ba6b27700 1 -- 172.21.15.102:0/3407221681 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba019b560 con 0x7f1ba0100270 2024-03-26T18:49:25.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.432+0000 7f1ba6b27700 1 -- 172.21.15.102:0/3407221681 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba0107bc0 con 0x7f1ba0100270 2024-03-26T18:49:25.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.432+0000 7f1b9dffb700 1 -- 172.21.15.102:0/3407221681 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7f1b9000fab0 con 0x7f1ba0100270 2024-03-26T18:49:25.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.432+0000 7f1b9dffb700 1 --2- 172.21.15.102:0/3407221681 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1b8c06c600 0x7f1b8c06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:49:25.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.433+0000 7f1b9dffb700 1 -- 172.21.15.102:0/3407221681 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f1b9008be50 con 0x7f1ba0100270 2024-03-26T18:49:25.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.433+0000 7f1ba48c3700 1 --2- 172.21.15.102:0/3407221681 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1b8c06c600 0x7f1b8c06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:49:25.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.433+0000 7f1ba48c3700 1 --2- 172.21.15.102:0/3407221681 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1b8c06c600 0x7f1b8c06eab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1b98009550 tx=0x7f1b980091b0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:49:25.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.437+0000 7f1b9dffb700 1 -- 172.21.15.102:0/3407221681 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1b9005ac90 con 0x7f1ba0100270 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: Detected new or changed devices on smithi102 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: Adjusting osd_memory_target on smithi102 to 5920M 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:49:25.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:25.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: Detected new or changed devices on smithi102 2024-03-26T18:49:25.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:25.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:25.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:25.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:25.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: Adjusting osd_memory_target on smithi102 to 5920M 2024-03-26T18:49:25.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:25.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:25.586 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:25.586 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:25.586 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:49:25.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:25.640+0000 7f1ba6b27700 1 -- 172.21.15.102:0/3407221681 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ba019bc60 con 0x7f1b8c06c600 2024-03-26T18:49:26.523 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:26 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:49:26.523 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:26 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:49:26.523 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:26 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:26.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:26 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:49:26.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:26 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:49:26.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:26 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:27.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:27 smithi190 ceph-mon[35043]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:49:27.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:27 smithi190 ceph-mon[35043]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:27.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:27 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:27.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:27 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:27.548 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:27 smithi102 ceph-mon[29002]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:49:27.549 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:27 smithi102 ceph-mon[29002]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:27.549 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:27 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:27.549 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:27 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:29.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:29 smithi102 ceph-mon[29002]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:29.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:29 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1358201435' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0179546-10c5-41f2-af0c-be93c2bbf5c7"}]: dispatch 2024-03-26T18:49:29.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:29 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1358201435' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0179546-10c5-41f2-af0c-be93c2bbf5c7"}]': finished 2024-03-26T18:49:29.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:29 smithi102 ceph-mon[29002]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T18:49:29.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:29 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:29.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:29 smithi190 ceph-mon[35043]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:29.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:29 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/1358201435' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0179546-10c5-41f2-af0c-be93c2bbf5c7"}]: dispatch 2024-03-26T18:49:29.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:29 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/1358201435' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0179546-10c5-41f2-af0c-be93c2bbf5c7"}]': finished 2024-03-26T18:49:29.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:29 smithi190 ceph-mon[35043]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T18:49:29.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:29 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:30.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:30 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3792439903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:49:30.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:30 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3792439903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:49:31.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:31 smithi190 ceph-mon[35043]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:31.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:31.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:31.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:31.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:31.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:31.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:31 smithi102 ceph-mon[29002]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:31.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:31.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:31.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:31.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:31.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:32.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:32 smithi102 ceph-mon[29002]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:32.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:32 smithi190 ceph-mon[35043]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:34.101 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:33 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:49:34.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:33 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:49:34.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:34 smithi102 ceph-mon[29002]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:35.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:34 smithi190 ceph-mon[35043]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:36.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:36 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T18:49:36.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:36 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:36.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:36 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T18:49:36.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:36 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:37.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:37 smithi190 ceph-mon[35043]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:37.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:37 smithi190 ceph-mon[35043]: Deploying daemon osd.2 on smithi102 2024-03-26T18:49:37.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:37 smithi102 ceph-mon[29002]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:37.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:37 smithi102 ceph-mon[29002]: Deploying daemon osd.2 on smithi102 2024-03-26T18:49:39.400 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:39 smithi102 ceph-mon[29002]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:39.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:39 smithi190 ceph-mon[35043]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:41.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:41 smithi102 ceph-mon[29002]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:41.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:41 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:41.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:41 smithi190 ceph-mon[35043]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:41.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:41 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:42.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:42.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:49:42.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:42.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:49:43.461 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:43 smithi102 ceph-mon[29002]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:43.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:43 smithi190 ceph-mon[35043]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:44.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.994+0000 7f1b9dffb700 1 -- 172.21.15.102:0/3407221681 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1ba019bc60 con 0x7f1b8c06c600 2024-03-26T18:49:44.986 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 2 on host 'smithi102' 2024-03-26T18:49:44.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.996+0000 7f1ba6b27700 1 -- 172.21.15.102:0/3407221681 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1b8c06c600 msgr2=0x7f1b8c06eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:49:44.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.996+0000 7f1ba6b27700 1 --2- 172.21.15.102:0/3407221681 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1b8c06c600 0x7f1b8c06eab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1b98009550 tx=0x7f1b980091b0 comp rx=0 tx=0).stop 2024-03-26T18:49:44.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.996+0000 7f1ba6b27700 1 -- 172.21.15.102:0/3407221681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba0100270 msgr2=0x7f1ba0195ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:49:44.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.996+0000 7f1ba6b27700 1 --2- 172.21.15.102:0/3407221681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba0100270 0x7f1ba0195ec0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1b90006010 tx=0x7f1b9000bd70 comp rx=0 tx=0).stop 2024-03-26T18:49:44.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.996+0000 7f1ba6b27700 1 -- 172.21.15.102:0/3407221681 shutdown_connections 2024-03-26T18:49:44.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.996+0000 7f1ba6b27700 1 --2- 172.21.15.102:0/3407221681 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f1b8c06c600 0x7f1b8c06eab0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:44.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.996+0000 7f1ba6b27700 1 --2- 172.21.15.102:0/3407221681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ba00ff070 0x7f1ba0195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:44.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.996+0000 7f1ba6b27700 1 --2- 172.21.15.102:0/3407221681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba0100270 0x7f1ba0195ec0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:44.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.996+0000 7f1ba6b27700 1 -- 172.21.15.102:0/3407221681 >> 172.21.15.102:0/3407221681 conn(0x7f1ba00fa600 msgr2=0x7f1ba0068d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:49:44.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.996+0000 7f1ba6b27700 1 -- 172.21.15.102:0/3407221681 shutdown_connections 2024-03-26T18:49:44.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:49:44.996+0000 7f1ba6b27700 1 -- 172.21.15.102:0/3407221681 wait complete. 2024-03-26T18:49:45.339 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:45 smithi102 ceph-mon[29002]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:45.339 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:45 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:45.339 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:45 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:45.339 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:45 smithi102 ceph-mon[29002]: from='osd.2 [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T18:49:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:45 smithi190 ceph-mon[35043]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:45 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:45 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:45 smithi190 ceph-mon[35043]: from='osd.2 [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T18:49:45.622 DEBUG:teuthology.orchestra.run.smithi102:osd.2> sudo journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T18:49:45.624 INFO:tasks.cephadm:Deploying osd.3 on smithi190 with /dev/vg_nvme/lv_4... 2024-03-26T18:49:45.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T18:49:45.837 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi190/config 2024-03-26T18:49:47.243 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:46 smithi190 ceph-mon[35043]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:47.243 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:46 smithi190 ceph-mon[35043]: from='osd.2 [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T18:49:47.243 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:46 smithi190 ceph-mon[35043]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T18:49:47.243 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:46 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:47.243 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:46 smithi190 ceph-mon[35043]: from='osd.2 [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:49:47.283 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:46 smithi102 ceph-mon[29002]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:47.283 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:46 smithi102 ceph-mon[29002]: from='osd.2 [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T18:49:47.283 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:46 smithi102 ceph-mon[29002]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T18:49:47.283 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:46 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:47.283 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:46 smithi102 ceph-mon[29002]: from='osd.2 [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:49:47.284 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:49:46 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[54930]: 2024-03-26T18:49:46.968+0000 7f6e4c4cd700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T18:49:47.284 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:49:46 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[54930]: 2024-03-26T18:49:46.976+0000 7f6e44abb700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T18:49:47.725 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T18:49:47.725 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T18:49:47.725 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-03-26T18:49:47.725 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-03-26T18:49:47.725 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113314 s, 925 MB/s 2024-03-26T18:49:47.725 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-03-26T18:49:48.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:47 smithi190 ceph-mon[35043]: from='osd.2 [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-26T18:49:48.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:47 smithi190 ceph-mon[35043]: osdmap e18: 3 total, 2 up, 3 in 2024-03-26T18:49:48.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:47 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:48.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:47 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:48.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:47 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:48.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:47 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:48.282 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:47 smithi102 ceph-mon[29002]: from='osd.2 [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-26T18:49:48.282 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:47 smithi102 ceph-mon[29002]: osdmap e18: 3 total, 2 up, 3 in 2024-03-26T18:49:48.282 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:48.282 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:48.282 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:48.283 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:47 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:48.359 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi190:vg_nvme/lv_4 2024-03-26T18:49:48.562 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi190/config 2024-03-26T18:49:49.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:48 smithi102 ceph-mon[29002]: purged_snaps scrub starts 2024-03-26T18:49:49.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:48 smithi102 ceph-mon[29002]: purged_snaps scrub ok 2024-03-26T18:49:49.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:48 smithi102 ceph-mon[29002]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:49.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:49.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:48 smithi102 ceph-mon[29002]: osd.2 [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840] boot 2024-03-26T18:49:49.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:48 smithi102 ceph-mon[29002]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T18:49:49.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:49.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:49:49.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:48 smithi190 ceph-mon[35043]: purged_snaps scrub starts 2024-03-26T18:49:49.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:48 smithi190 ceph-mon[35043]: purged_snaps scrub ok 2024-03-26T18:49:49.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:48 smithi190 ceph-mon[35043]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T18:49:49.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:49.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:48 smithi190 ceph-mon[35043]: osd.2 [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840] boot 2024-03-26T18:49:49.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:48 smithi190 ceph-mon[35043]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T18:49:49.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:49:49.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:49:50.133 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.140+0000 7f7d4846e700 1 -- 172.21.15.190:0/3617690112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d40100270 msgr2=0x7f7d401006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:49:50.133 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.140+0000 7f7d4846e700 1 --2- 172.21.15.190:0/3617690112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d40100270 0x7f7d401006c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009910 tx=0x7f7d30009c20 comp rx=0 tx=0).stop 2024-03-26T18:49:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.140+0000 7f7d4846e700 1 -- 172.21.15.190:0/3617690112 shutdown_connections 2024-03-26T18:49:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.140+0000 7f7d4846e700 1 --2- 172.21.15.190:0/3617690112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d40100270 0x7f7d401006c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.140+0000 7f7d4846e700 1 --2- 172.21.15.190:0/3617690112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ff070 0x7f7d400ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.140+0000 7f7d4846e700 1 -- 172.21.15.190:0/3617690112 >> 172.21.15.190:0/3617690112 conn(0x7f7d400fa600 msgr2=0x7f7d400fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:49:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.140+0000 7f7d4846e700 1 -- 172.21.15.190:0/3617690112 shutdown_connections 2024-03-26T18:49:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.140+0000 7f7d4846e700 1 -- 172.21.15.190:0/3617690112 wait complete. 2024-03-26T18:49:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.140+0000 7f7d4846e700 1 Processor -- start 2024-03-26T18:49:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.140+0000 7f7d4846e700 1 -- start start 2024-03-26T18:49:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d4846e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ff070 0x7f7d401959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:49:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d4846e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d40100270 0x7f7d40195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:49:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d4846e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d40196510 con 0x7f7d400ff070 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d4846e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d40196650 con 0x7f7d40100270 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d45a09700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d40100270 0x7f7d40195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d45a09700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d40100270 0x7f7d40195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:42696/0 (socket says 172.21.15.190:42696) 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d45a09700 1 -- 172.21.15.190:0/1314865538 learned_addr learned my addr 172.21.15.190:0/1314865538 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d4620a700 1 --2- 172.21.15.190:0/1314865538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ff070 0x7f7d401959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d45a09700 1 -- 172.21.15.190:0/1314865538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ff070 msgr2=0x7f7d401959b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d45a09700 1 --2- 172.21.15.190:0/1314865538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ff070 0x7f7d401959b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d45a09700 1 -- 172.21.15.190:0/1314865538 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d300095f0 con 0x7f7d40100270 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d45a09700 1 --2- 172.21.15.190:0/1314865538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d40100270 0x7f7d40195ef0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7d30006010 tx=0x7f7d3000bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d377fe700 1 -- 172.21.15.190:0/1314865538 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d3001d070 con 0x7f7d40100270 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d377fe700 1 -- 172.21.15.190:0/1314865538 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7d3000f460 con 0x7f7d40100270 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d4846e700 1 -- 172.21.15.190:0/1314865538 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4019b0a0 con 0x7f7d40100270 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d377fe700 1 -- 172.21.15.190:0/1314865538 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d30021640 con 0x7f7d40100270 2024-03-26T18:49:50.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.141+0000 7f7d4846e700 1 -- 172.21.15.190:0/1314865538 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4019b590 con 0x7f7d40100270 2024-03-26T18:49:50.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.142+0000 7f7d4846e700 1 -- 172.21.15.190:0/1314865538 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d40107bc0 con 0x7f7d40100270 2024-03-26T18:49:50.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.143+0000 7f7d377fe700 1 -- 172.21.15.190:0/1314865538 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7f7d3000fab0 con 0x7f7d40100270 2024-03-26T18:49:50.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.143+0000 7f7d377fe700 1 --2- 172.21.15.190:0/1314865538 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d2c06c600 0x7f7d2c06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:49:50.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.143+0000 7f7d377fe700 1 -- 172.21.15.190:0/1314865538 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2846+0+0 (secure 0 0 0) 0x7f7d3008c000 con 0x7f7d40100270 2024-03-26T18:49:50.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.143+0000 7f7d4620a700 1 --2- 172.21.15.190:0/1314865538 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d2c06c600 0x7f7d2c06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:49:50.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.144+0000 7f7d4620a700 1 --2- 172.21.15.190:0/1314865538 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d2c06c600 0x7f7d2c06eab0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7d401000d0 tx=0x7f7d3c00b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:49:50.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.148+0000 7f7d377fe700 1 -- 172.21.15.190:0/1314865538 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7d3005ac90 con 0x7f7d40100270 2024-03-26T18:49:50.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:50.350+0000 7f7d4846e700 1 -- 172.21.15.190:0/1314865538 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d40069460 con 0x7f7d2c06c600 2024-03-26T18:49:50.480 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:50 smithi102 ceph-mon[29002]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T18:49:50.528 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:50 smithi190 ceph-mon[35043]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T18:49:51.492 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: pgmap v76: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:49:51.492 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T18:49:51.492 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T18:49:51.492 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T18:49:51.492 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T18:49:51.492 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:49:51.492 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:49:51.492 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:51.492 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:51.492 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:51.492 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:51.493 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:51.493 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:51.493 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:51.493 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:51.493 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:51.493 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:51.493 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:51 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:49:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: pgmap v76: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:49:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T18:49:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T18:49:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T18:49:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T18:49:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:49:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:49:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:49:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:51 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:49:52.318 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:52 smithi102 ceph-mon[29002]: from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:49:52.318 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:52 smithi102 ceph-mon[29002]: Detected new or changed devices on smithi102 2024-03-26T18:49:52.319 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:52 smithi102 ceph-mon[29002]: Adjusting osd_memory_target on smithi102 to 3946M 2024-03-26T18:49:52.319 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:52 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T18:49:52.319 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:52 smithi102 ceph-mon[29002]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T18:49:52.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:52 smithi190 ceph-mon[35043]: from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:49:52.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:52 smithi190 ceph-mon[35043]: Detected new or changed devices on smithi102 2024-03-26T18:49:52.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:52 smithi190 ceph-mon[35043]: Adjusting osd_memory_target on smithi102 to 3946M 2024-03-26T18:49:52.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:52 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T18:49:52.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:52 smithi190 ceph-mon[35043]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T18:49:52.583 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57523]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T18:49:52.583 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57523]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T18:49:52.583 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57523]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T18:49:52.583 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57523]: pam_unix(sudo:session): session closed for user root 2024-03-26T18:49:52.863 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57572]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T18:49:52.863 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57572]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T18:49:52.864 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57572]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T18:49:52.864 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57572]: pam_unix(sudo:session): session closed for user root 2024-03-26T18:49:53.122 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57602]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T18:49:53.122 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57602]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T18:49:53.122 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57602]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T18:49:53.122 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:49:52 smithi102 sudo[57602]: pam_unix(sudo:session): session closed for user root 2024-03-26T18:49:53.374 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57605]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T18:49:53.375 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57605]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T18:49:53.375 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57605]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T18:49:53.375 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57605]: pam_unix(sudo:session): session closed for user root 2024-03-26T18:49:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 ceph-mon[29002]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:49:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 ceph-mon[29002]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T18:49:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/3668305787' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d524f7b-8b31-469b-b0c2-b94079febaeb"}]: dispatch 2024-03-26T18:49:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 ceph-mon[29002]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d524f7b-8b31-469b-b0c2-b94079febaeb"}]: dispatch 2024-03-26T18:49:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 ceph-mon[29002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d524f7b-8b31-469b-b0c2-b94079febaeb"}]': finished 2024-03-26T18:49:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 ceph-mon[29002]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T18:49:53.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:49:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:53 smithi190 ceph-mon[35043]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:49:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:53 smithi190 ceph-mon[35043]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T18:49:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:53 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:53 smithi190 ceph-mon[35043]: from='client.? 172.21.15.190:0/3668305787' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d524f7b-8b31-469b-b0c2-b94079febaeb"}]: dispatch 2024-03-26T18:49:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:53 smithi190 ceph-mon[35043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d524f7b-8b31-469b-b0c2-b94079febaeb"}]: dispatch 2024-03-26T18:49:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:53 smithi190 ceph-mon[35043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d524f7b-8b31-469b-b0c2-b94079febaeb"}]': finished 2024-03-26T18:49:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:53 smithi190 ceph-mon[35043]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T18:49:53.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:53 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:49:53.644 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57653]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T18:49:53.645 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57653]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T18:49:53.645 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57653]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T18:49:53.645 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57653]: pam_unix(sudo:session): session closed for user root 2024-03-26T18:49:53.661 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:49:53.670+0000 7f7d377fe700 1 -- 172.21.15.190:0/1314865538 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f7d300575d0 con 0x7f7d40100270 2024-03-26T18:49:53.897 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57684]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T18:49:53.897 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57684]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T18:49:53.897 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57684]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T18:49:53.897 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57684]: pam_unix(sudo:session): session closed for user root 2024-03-26T18:49:54.164 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57722]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T18:49:54.164 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57722]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T18:49:54.164 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57722]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T18:49:54.164 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:53 smithi102 sudo[57722]: pam_unix(sudo:session): session closed for user root 2024-03-26T18:49:54.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 sudo[57765]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T18:49:54.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 sudo[57765]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T18:49:54.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 sudo[57765]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T18:49:54.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 sudo[57765]: pam_unix(sudo:session): session closed for user root 2024-03-26T18:49:54.672 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 sudo[39289]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T18:49:54.673 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 sudo[39289]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T18:49:54.673 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 sudo[39289]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T18:49:54.673 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 sudo[39289]: pam_unix(sudo:session): session closed for user root 2024-03-26T18:49:54.968 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/611522435' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:49:54.968 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 ceph-mon[29002]: mgrmap e19: smithi102.rsfpnv(active, since 2m), standbys: smithi190.zrixos 2024-03-26T18:49:54.968 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 ceph-mon[29002]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T18:49:54.968 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 ceph-mon[29002]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T18:49:54.968 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:49:54.968 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:49:54.968 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:49:54.968 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:54 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:49:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 sudo[39477]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T18:49:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 sudo[39477]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T18:49:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 sudo[39477]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T18:49:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 sudo[39477]: pam_unix(sudo:session): session closed for user root 2024-03-26T18:49:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 ceph-mon[35043]: from='client.? 172.21.15.190:0/611522435' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:49:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 ceph-mon[35043]: mgrmap e19: smithi102.rsfpnv(active, since 2m), standbys: smithi190.zrixos 2024-03-26T18:49:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 ceph-mon[35043]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T18:49:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 ceph-mon[35043]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T18:49:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:49:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:49:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:49:55.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:54 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:49:56.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:55 smithi190 ceph-mon[35043]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:49:56.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:55 smithi190 ceph-mon[35043]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T18:49:56.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:55 smithi190 ceph-mon[35043]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T18:49:56.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:55 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:56.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:55 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:56.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:55 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:56.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:55 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:56.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:55 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:56.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:55 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:56.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:55 smithi102 ceph-mon[29002]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:49:56.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:55 smithi102 ceph-mon[29002]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T18:49:56.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:55 smithi102 ceph-mon[29002]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T18:49:56.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:55 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:56.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:55 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:56.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:55 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:56.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:55 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:49:56.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:55 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:49:56.064 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:55 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:49:57.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:56 smithi190 ceph-mon[35043]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:49:57.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:56 smithi102 ceph-mon[29002]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:49:59.015 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:49:58 smithi190 ceph-mon[35043]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:49:59.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:49:58 smithi102 ceph-mon[29002]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:01.177 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:00 smithi190 ceph-mon[35043]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:01.177 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:00 smithi190 ceph-mon[35043]: overall HEALTH_OK 2024-03-26T18:50:01.177 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T18:50:01.177 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:01.177 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:00 smithi190 ceph-mon[35043]: Deploying daemon osd.3 on smithi190 2024-03-26T18:50:01.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:00 smithi102 ceph-mon[29002]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:01.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:00 smithi102 ceph-mon[29002]: overall HEALTH_OK 2024-03-26T18:50:01.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T18:50:01.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:01.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:00 smithi102 ceph-mon[29002]: Deploying daemon osd.3 on smithi190 2024-03-26T18:50:03.078 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:02 smithi190 ceph-mon[35043]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:03.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:02 smithi102 ceph-mon[29002]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:04.149 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:03 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:50:04.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:03 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:50:05.154 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:04 smithi190 ceph-mon[35043]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:05.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:04 smithi102 ceph-mon[29002]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:06.688 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:06.688 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:06.688 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:50:06.688 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:06 smithi190 ceph-mon[35043]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:06.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:06.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:06.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:50:06.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:06 smithi102 ceph-mon[29002]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:08.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.765+0000 7f7d377fe700 1 -- 172.21.15.190:0/1314865538 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7d40069460 con 0x7f7d2c06c600 2024-03-26T18:50:08.757 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 3 on host 'smithi190' 2024-03-26T18:50:08.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.768+0000 7f7d4846e700 1 -- 172.21.15.190:0/1314865538 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d2c06c600 msgr2=0x7f7d2c06eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:50:08.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.768+0000 7f7d4846e700 1 --2- 172.21.15.190:0/1314865538 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d2c06c600 0x7f7d2c06eab0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7d401000d0 tx=0x7f7d3c00b410 comp rx=0 tx=0).stop 2024-03-26T18:50:08.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.768+0000 7f7d4846e700 1 -- 172.21.15.190:0/1314865538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d40100270 msgr2=0x7f7d40195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:50:08.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.768+0000 7f7d4846e700 1 --2- 172.21.15.190:0/1314865538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d40100270 0x7f7d40195ef0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7d30006010 tx=0x7f7d3000bd70 comp rx=0 tx=0).stop 2024-03-26T18:50:08.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.768+0000 7f7d4846e700 1 -- 172.21.15.190:0/1314865538 shutdown_connections 2024-03-26T18:50:08.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.768+0000 7f7d4846e700 1 --2- 172.21.15.190:0/1314865538 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d2c06c600 0x7f7d2c06eab0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:08.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.768+0000 7f7d4846e700 1 --2- 172.21.15.190:0/1314865538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ff070 0x7f7d401959b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:08.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.768+0000 7f7d4846e700 1 --2- 172.21.15.190:0/1314865538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d40100270 0x7f7d40195ef0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:08.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.768+0000 7f7d4846e700 1 -- 172.21.15.190:0/1314865538 >> 172.21.15.190:0/1314865538 conn(0x7f7d400fa600 msgr2=0x7f7d40068d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:50:08.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.768+0000 7f7d4846e700 1 -- 172.21.15.190:0/1314865538 shutdown_connections 2024-03-26T18:50:08.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:08.768+0000 7f7d4846e700 1 -- 172.21.15.190:0/1314865538 wait complete. 2024-03-26T18:50:09.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:08 smithi190 ceph-mon[35043]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:09.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:08 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:09.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:08 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:09.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:08 smithi102 ceph-mon[29002]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:09.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:08 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:09.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:08 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:09.703 DEBUG:teuthology.orchestra.run.smithi190:osd.3> sudo journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T18:50:09.706 INFO:tasks.cephadm:Deploying osd.4 on smithi190 with /dev/vg_nvme/lv_3... 2024-03-26T18:50:09.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T18:50:09.969 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi190/config 2024-03-26T18:50:10.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:09 smithi190 ceph-mon[35043]: from='osd.3 [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T18:50:10.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:09 smithi190 ceph-mon[35043]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T18:50:10.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:09 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:10.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:09 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:10.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:09 smithi102 ceph-mon[29002]: from='osd.3 [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T18:50:10.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:09 smithi102 ceph-mon[29002]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T18:50:10.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:09 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:10.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:09 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:11.073 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:10 smithi190 ceph-mon[35043]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:11.073 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:10 smithi190 ceph-mon[35043]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T18:50:11.073 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:10 smithi190 ceph-mon[35043]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T18:50:11.073 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:10 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:50:11.073 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:10 smithi190 ceph-mon[35043]: from='osd.3 [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:50:11.073 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:10 smithi190 ceph-mon[35043]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:50:11.073 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:50:10 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[41121]: 2024-03-26T18:50:10.935+0000 7f0d2fc13700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T18:50:11.073 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:50:10 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[41121]: 2024-03-26T18:50:10.942+0000 7f0d2a205700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T18:50:11.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:10 smithi102 ceph-mon[29002]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:11.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:10 smithi102 ceph-mon[29002]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T18:50:11.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:10 smithi102 ceph-mon[29002]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T18:50:11.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:10 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:50:11.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:10 smithi102 ceph-mon[29002]: from='osd.3 [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:50:11.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:10 smithi102 ceph-mon[29002]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:50:11.845 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T18:50:11.845 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T18:50:11.845 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-03-26T18:50:11.845 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-03-26T18:50:11.845 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121683 s, 862 MB/s 2024-03-26T18:50:11.845 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-03-26T18:50:12.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:11 smithi190 ceph-mon[35043]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-26T18:50:12.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:11 smithi190 ceph-mon[35043]: osdmap e26: 4 total, 3 up, 4 in 2024-03-26T18:50:12.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:50:12.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:11 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:50:12.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:11 smithi102 ceph-mon[29002]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-26T18:50:12.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:11 smithi102 ceph-mon[29002]: osdmap e26: 4 total, 3 up, 4 in 2024-03-26T18:50:12.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:50:12.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:11 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:50:12.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi190:vg_nvme/lv_3 2024-03-26T18:50:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:12 smithi190 ceph-mon[35043]: purged_snaps scrub starts 2024-03-26T18:50:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:12 smithi190 ceph-mon[35043]: purged_snaps scrub ok 2024-03-26T18:50:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:12 smithi190 ceph-mon[35043]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:12 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:50:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:12 smithi190 ceph-mon[35043]: osd.3 [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793] boot 2024-03-26T18:50:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:12 smithi190 ceph-mon[35043]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T18:50:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:12 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:50:13.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:12 smithi102 ceph-mon[29002]: purged_snaps scrub starts 2024-03-26T18:50:13.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:12 smithi102 ceph-mon[29002]: purged_snaps scrub ok 2024-03-26T18:50:13.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:12 smithi102 ceph-mon[29002]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T18:50:13.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:12 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:50:13.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:12 smithi102 ceph-mon[29002]: osd.3 [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793] boot 2024-03-26T18:50:13.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:12 smithi102 ceph-mon[29002]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T18:50:13.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:12 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:50:13.910 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi190/config 2024-03-26T18:50:14.198 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:13 smithi190 ceph-mon[35043]: osdmap e28: 4 total, 4 up, 4 in 2024-03-26T18:50:14.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:13 smithi102 ceph-mon[29002]: osdmap e28: 4 total, 4 up, 4 in 2024-03-26T18:50:15.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:14 smithi190 ceph-mon[35043]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:15.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:14 smithi190 ceph-mon[35043]: osdmap e29: 4 total, 4 up, 4 in 2024-03-26T18:50:15.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:14 smithi102 ceph-mon[29002]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:15.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:14 smithi102 ceph-mon[29002]: osdmap e29: 4 total, 4 up, 4 in 2024-03-26T18:50:16.106 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: osdmap e30: 4 total, 4 up, 4 in 2024-03-26T18:50:16.106 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:16.106 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:16.106 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:50:16.106 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:16.107 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:16.107 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:50:16.107 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:16.107 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:50:16.107 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:16.107 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:50:16.107 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:15 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:16.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.322+0000 7f3f134a8700 1 -- 172.21.15.190:0/2556849871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0c101350 msgr2=0x7f3f0c1017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:50:16.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.322+0000 7f3f134a8700 1 --2- 172.21.15.190:0/2556849871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0c101350 0x7f3f0c1017a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009910 tx=0x7f3efc009c20 comp rx=0 tx=0).stop 2024-03-26T18:50:16.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.322+0000 7f3f134a8700 1 -- 172.21.15.190:0/2556849871 shutdown_connections 2024-03-26T18:50:16.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.322+0000 7f3f134a8700 1 --2- 172.21.15.190:0/2556849871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0c101350 0x7f3f0c1017a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:16.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.322+0000 7f3f134a8700 1 --2- 172.21.15.190:0/2556849871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c100150 0x7f3f0c100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:16.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.322+0000 7f3f134a8700 1 -- 172.21.15.190:0/2556849871 >> 172.21.15.190:0/2556849871 conn(0x7f3f0c0fb6e0 msgr2=0x7f3f0c0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:50:16.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.322+0000 7f3f134a8700 1 -- 172.21.15.190:0/2556849871 shutdown_connections 2024-03-26T18:50:16.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.322+0000 7f3f134a8700 1 -- 172.21.15.190:0/2556849871 wait complete. 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.322+0000 7f3f134a8700 1 Processor -- start 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f134a8700 1 -- start start 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f134a8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0c100150 0x7f3f0c074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f134a8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c101350 0x7f3f0c073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f134a8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c073680 con 0x7f3f0c101350 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f134a8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0737c0 con 0x7f3f0c100150 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f11244700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0c100150 0x7f3f0c074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f11244700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0c100150 0x7f3f0c074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:56356/0 (socket says 172.21.15.190:56356) 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f11244700 1 -- 172.21.15.190:0/2726500223 learned_addr learned my addr 172.21.15.190:0/2726500223 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f10a43700 1 --2- 172.21.15.190:0/2726500223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c101350 0x7f3f0c073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f11244700 1 -- 172.21.15.190:0/2726500223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c101350 msgr2=0x7f3f0c073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f11244700 1 --2- 172.21.15.190:0/2726500223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c101350 0x7f3f0c073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f11244700 1 -- 172.21.15.190:0/2726500223 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc0095f0 con 0x7f3f0c100150 2024-03-26T18:50:16.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f11244700 1 --2- 172.21.15.190:0/2726500223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0c100150 0x7f3f0c074af0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800ada0 tx=0x7f3f0800c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:50:16.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f027fc700 1 -- 172.21.15.190:0/2726500223 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f0800f070 con 0x7f3f0c100150 2024-03-26T18:50:16.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f134a8700 1 -- 172.21.15.190:0/2726500223 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c073aa0 con 0x7f3f0c100150 2024-03-26T18:50:16.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f027fc700 1 -- 172.21.15.190:0/2726500223 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3f0800ae90 con 0x7f3f0c100150 2024-03-26T18:50:16.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.323+0000 7f3f027fc700 1 -- 172.21.15.190:0/2726500223 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f080083f0 con 0x7f3f0c100150 2024-03-26T18:50:16.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.324+0000 7f3f134a8700 1 -- 172.21.15.190:0/2726500223 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c19d9b0 con 0x7f3f0c100150 2024-03-26T18:50:16.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.324+0000 7f3f134a8700 1 -- 172.21.15.190:0/2726500223 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c108ca0 con 0x7f3f0c100150 2024-03-26T18:50:16.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.325+0000 7f3f027fc700 1 -- 172.21.15.190:0/2726500223 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f3f0800cad0 con 0x7f3f0c100150 2024-03-26T18:50:16.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.325+0000 7f3f027fc700 1 --2- 172.21.15.190:0/2726500223 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3ef806c680 0x7f3ef806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:50:16.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.325+0000 7f3f027fc700 1 -- 172.21.15.190:0/2726500223 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f3f0808ab60 con 0x7f3f0c100150 2024-03-26T18:50:16.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.325+0000 7f3f10a43700 1 --2- 172.21.15.190:0/2726500223 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3ef806c680 0x7f3ef806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:50:16.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.326+0000 7f3f10a43700 1 --2- 172.21.15.190:0/2726500223 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3ef806c680 0x7f3ef806eb30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3efc006010 tx=0x7f3efc00b540 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:50:16.322 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.330+0000 7f3f027fc700 1 -- 172.21.15.190:0/2726500223 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3f08059370 con 0x7f3f0c100150 2024-03-26T18:50:16.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: osdmap e30: 4 total, 4 up, 4 in 2024-03-26T18:50:16.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:16.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:16.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:50:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:50:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:50:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:50:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:15 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:16.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:16.527+0000 7f3f134a8700 1 -- 172.21.15.190:0/2726500223 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f0c105ca0 con 0x7f3ef806c680 2024-03-26T18:50:17.256 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:16 smithi190 ceph-mon[35043]: Detected new or changed devices on smithi190 2024-03-26T18:50:17.256 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:16 smithi190 ceph-mon[35043]: Adjusting osd_memory_target on smithi190 to 14912M 2024-03-26T18:50:17.256 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:16 smithi190 ceph-mon[35043]: pgmap v99: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T18:50:17.256 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:16 smithi190 ceph-mon[35043]: osdmap e31: 4 total, 4 up, 4 in 2024-03-26T18:50:17.256 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:16 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:50:17.256 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:16 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:50:17.256 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:16 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:17.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:16 smithi102 ceph-mon[29002]: Detected new or changed devices on smithi190 2024-03-26T18:50:17.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:16 smithi102 ceph-mon[29002]: Adjusting osd_memory_target on smithi190 to 14912M 2024-03-26T18:50:17.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:16 smithi102 ceph-mon[29002]: pgmap v99: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T18:50:17.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:16 smithi102 ceph-mon[29002]: osdmap e31: 4 total, 4 up, 4 in 2024-03-26T18:50:17.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:16 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:50:17.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:16 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:50:17.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:16 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:18.191 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:17 smithi190 ceph-mon[35043]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:50:18.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:17 smithi102 ceph-mon[29002]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:50:19.276 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:18 smithi102 ceph-mon[29002]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-26T18:50:19.276 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:18 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:50:19.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:18 smithi190 ceph-mon[35043]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-26T18:50:19.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:18 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:50:20.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:20 smithi190 ceph-mon[35043]: from='client.? 172.21.15.190:0/486916667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "865baef8-dd72-4449-b3a4-893801bc2958"}]: dispatch 2024-03-26T18:50:20.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:20 smithi190 ceph-mon[35043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "865baef8-dd72-4449-b3a4-893801bc2958"}]: dispatch 2024-03-26T18:50:20.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:20 smithi190 ceph-mon[35043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "865baef8-dd72-4449-b3a4-893801bc2958"}]': finished 2024-03-26T18:50:20.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:20 smithi190 ceph-mon[35043]: osdmap e32: 5 total, 4 up, 5 in 2024-03-26T18:50:20.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:50:20.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:20 smithi190 ceph-mon[35043]: from='client.? 172.21.15.190:0/3452104447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:50:20.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:19 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/486916667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "865baef8-dd72-4449-b3a4-893801bc2958"}]: dispatch 2024-03-26T18:50:20.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:19 smithi102 ceph-mon[29002]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "865baef8-dd72-4449-b3a4-893801bc2958"}]: dispatch 2024-03-26T18:50:20.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:19 smithi102 ceph-mon[29002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "865baef8-dd72-4449-b3a4-893801bc2958"}]': finished 2024-03-26T18:50:20.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:19 smithi102 ceph-mon[29002]: osdmap e32: 5 total, 4 up, 5 in 2024-03-26T18:50:20.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:50:20.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:19 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/3452104447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:50:21.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:21 smithi190 ceph-mon[35043]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-26T18:50:21.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:21 smithi102 ceph-mon[29002]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-26T18:50:23.459 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:23 smithi190 ceph-mon[35043]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-26T18:50:23.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:23 smithi102 ceph-mon[29002]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-26T18:50:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:25 smithi190 ceph-mon[35043]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:25.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:25 smithi102 ceph-mon[29002]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:27.419 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:27 smithi190 ceph-mon[35043]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:27.420 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:27 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T18:50:27.420 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:27 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:27.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:27 smithi102 ceph-mon[29002]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:27.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:27 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T18:50:27.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:27 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:28.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:28 smithi190 ceph-mon[35043]: Deploying daemon osd.4 on smithi190 2024-03-26T18:50:28.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:28 smithi102 ceph-mon[29002]: Deploying daemon osd.4 on smithi190 2024-03-26T18:50:29.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:29 smithi190 ceph-mon[35043]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:29.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:29 smithi102 ceph-mon[29002]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:31.479 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:31 smithi190 ceph-mon[35043]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:31.624 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:31 smithi102 ceph-mon[29002]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:32.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:32 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:32.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:32 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:32.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:32 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:50:32.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:32 smithi102 ceph-mon[29002]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:32.969 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:32 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:32.969 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:32 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:32.969 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:32 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:50:32.970 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:32 smithi190 ceph-mon[35043]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:34.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:33 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:50:34.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:33 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:50:35.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:34 smithi190 ceph-mon[35043]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:35.270 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:34 smithi102 ceph-mon[29002]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:35.486 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.495+0000 7f3f027fc700 1 -- 172.21.15.190:0/2726500223 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3f0c105ca0 con 0x7f3ef806c680 2024-03-26T18:50:35.487 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 4 on host 'smithi190' 2024-03-26T18:50:35.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.496+0000 7f3f134a8700 1 -- 172.21.15.190:0/2726500223 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3ef806c680 msgr2=0x7f3ef806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:50:35.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.496+0000 7f3f134a8700 1 --2- 172.21.15.190:0/2726500223 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3ef806c680 0x7f3ef806eb30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3efc006010 tx=0x7f3efc00b540 comp rx=0 tx=0).stop 2024-03-26T18:50:35.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.497+0000 7f3f134a8700 1 -- 172.21.15.190:0/2726500223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0c100150 msgr2=0x7f3f0c074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:50:35.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.497+0000 7f3f134a8700 1 --2- 172.21.15.190:0/2726500223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0c100150 0x7f3f0c074af0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800ada0 tx=0x7f3f0800c330 comp rx=0 tx=0).stop 2024-03-26T18:50:35.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.497+0000 7f3f134a8700 1 -- 172.21.15.190:0/2726500223 shutdown_connections 2024-03-26T18:50:35.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.497+0000 7f3f134a8700 1 --2- 172.21.15.190:0/2726500223 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3ef806c680 0x7f3ef806eb30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:35.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.497+0000 7f3f134a8700 1 --2- 172.21.15.190:0/2726500223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0c100150 0x7f3f0c074af0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:35.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.497+0000 7f3f134a8700 1 --2- 172.21.15.190:0/2726500223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c101350 0x7f3f0c073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:35.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.497+0000 7f3f134a8700 1 -- 172.21.15.190:0/2726500223 >> 172.21.15.190:0/2726500223 conn(0x7f3f0c0fb6e0 msgr2=0x7f3f0c104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:50:35.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.497+0000 7f3f134a8700 1 -- 172.21.15.190:0/2726500223 shutdown_connections 2024-03-26T18:50:35.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:35.497+0000 7f3f134a8700 1 -- 172.21.15.190:0/2726500223 wait complete. 2024-03-26T18:50:36.450 DEBUG:teuthology.orchestra.run.smithi190:osd.4> sudo journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T18:50:36.453 INFO:tasks.cephadm:Deploying osd.5 on smithi190 with /dev/vg_nvme/lv_2... 2024-03-26T18:50:36.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T18:50:36.705 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi190/config 2024-03-26T18:50:36.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:36 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:36.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:36 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:36.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:36 smithi190 ceph-mon[35043]: from='osd.4 [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T18:50:36.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:36 smithi190 ceph-mon[35043]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T18:50:36.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:36 smithi190 ceph-mon[35043]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:36.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:36 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:36.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:36 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:36.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:36 smithi102 ceph-mon[29002]: from='osd.4 [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T18:50:36.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:36 smithi102 ceph-mon[29002]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T18:50:36.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:36 smithi102 ceph-mon[29002]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:37.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:37 smithi190 ceph-mon[35043]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T18:50:37.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:37 smithi190 ceph-mon[35043]: osdmap e33: 5 total, 4 up, 5 in 2024-03-26T18:50:37.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:37 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:50:37.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:37 smithi190 ceph-mon[35043]: from='osd.4 [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:50:37.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:37 smithi190 ceph-mon[35043]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:50:37.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:37 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:37.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:37 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:37.780 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 18:50:37 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[45750]: 2024-03-26T18:50:37.471+0000 7f865679e700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T18:50:37.780 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 18:50:37 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[45750]: 2024-03-26T18:50:37.480+0000 7f8652d94700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T18:50:37.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:37 smithi102 ceph-mon[29002]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T18:50:37.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:37 smithi102 ceph-mon[29002]: osdmap e33: 5 total, 4 up, 5 in 2024-03-26T18:50:37.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:37 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:50:37.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:37 smithi102 ceph-mon[29002]: from='osd.4 [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:50:37.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:37 smithi102 ceph-mon[29002]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:50:37.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:37 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:37.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:37 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:38.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:38 smithi190 ceph-mon[35043]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-26T18:50:38.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:38 smithi190 ceph-mon[35043]: osdmap e34: 5 total, 4 up, 5 in 2024-03-26T18:50:38.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:38 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:50:38.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:38 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:50:38.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:38 smithi190 ceph-mon[35043]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:38.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:38 smithi102 ceph-mon[29002]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-26T18:50:38.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:38 smithi102 ceph-mon[29002]: osdmap e34: 5 total, 4 up, 5 in 2024-03-26T18:50:38.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:38 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:50:38.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:38 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:50:38.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:38 smithi102 ceph-mon[29002]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T18:50:39.655 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:39 smithi190 ceph-mon[35043]: purged_snaps scrub starts 2024-03-26T18:50:39.655 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:39 smithi190 ceph-mon[35043]: purged_snaps scrub ok 2024-03-26T18:50:39.655 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:39 smithi190 ceph-mon[35043]: osd.4 [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306] boot 2024-03-26T18:50:39.655 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:39 smithi190 ceph-mon[35043]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T18:50:39.655 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:39 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:50:39.655 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T18:50:39.655 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T18:50:39.656 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-03-26T18:50:39.656 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-03-26T18:50:39.656 INFO:teuthology.orchestra.run.smithi190.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123621 s, 848 MB/s 2024-03-26T18:50:39.656 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-03-26T18:50:39.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:39 smithi102 ceph-mon[29002]: purged_snaps scrub starts 2024-03-26T18:50:39.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:39 smithi102 ceph-mon[29002]: purged_snaps scrub ok 2024-03-26T18:50:39.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:39 smithi102 ceph-mon[29002]: osd.4 [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306] boot 2024-03-26T18:50:39.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:39 smithi102 ceph-mon[29002]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T18:50:39.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:39 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:50:40.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi190:vg_nvme/lv_2 2024-03-26T18:50:40.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:40 smithi102 ceph-mon[29002]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T18:50:40.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:40 smithi102 ceph-mon[29002]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:40.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:40 smithi190 ceph-mon[35043]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T18:50:40.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:40 smithi190 ceph-mon[35043]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:41.212 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi190/config 2024-03-26T18:50:42.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: Detected new or changed devices on smithi190 2024-03-26T18:50:42.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:42.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:42.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:50:42.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:50:42.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: Adjusting osd_memory_target on smithi190 to 7456M 2024-03-26T18:50:42.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:42.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:42.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:50:42.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:42.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:50:42.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:42.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:50:42.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: Detected new or changed devices on smithi190 2024-03-26T18:50:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:50:42.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:50:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: Adjusting osd_memory_target on smithi190 to 7456M 2024-03-26T18:50:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:50:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:50:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:50:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:42.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.792+0000 7fb220b4c700 1 -- 172.21.15.190:0/2530858523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb21c100170 msgr2=0x7fb21c100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.792+0000 7fb220b4c700 1 --2- 172.21.15.190:0/2530858523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb21c100170 0x7fb21c100580 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb210009910 tx=0x7fb210009c20 comp rx=0 tx=0).stop 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.792+0000 7fb220b4c700 1 -- 172.21.15.190:0/2530858523 shutdown_connections 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.792+0000 7fb220b4c700 1 --2- 172.21.15.190:0/2530858523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb21c101370 0x7fb21c1017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.792+0000 7fb220b4c700 1 --2- 172.21.15.190:0/2530858523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb21c100170 0x7fb21c100580 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.792+0000 7fb220b4c700 1 -- 172.21.15.190:0/2530858523 >> 172.21.15.190:0/2530858523 conn(0x7fb21c0fb6e0 msgr2=0x7fb21c0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.792+0000 7fb220b4c700 1 -- 172.21.15.190:0/2530858523 shutdown_connections 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.792+0000 7fb220b4c700 1 -- 172.21.15.190:0/2530858523 wait complete. 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.793+0000 7fb220b4c700 1 Processor -- start 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.793+0000 7fb220b4c700 1 -- start start 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.793+0000 7fb220b4c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb21c100170 0x7fb21c1959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.793+0000 7fb220b4c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb21c101370 0x7fb21c195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.793+0000 7fb220b4c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c196510 con 0x7fb21c101370 2024-03-26T18:50:42.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.793+0000 7fb220b4c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c196650 con 0x7fb21c100170 2024-03-26T18:50:42.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.793+0000 7fb21a59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb21c100170 0x7fb21c1959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:50:42.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.793+0000 7fb21a59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb21c100170 0x7fb21c1959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:43124/0 (socket says 172.21.15.190:43124) 2024-03-26T18:50:42.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.793+0000 7fb21a59c700 1 -- 172.21.15.190:0/1410077149 learned_addr learned my addr 172.21.15.190:0/1410077149 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:50:42.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.794+0000 7fb219d9b700 1 --2- 172.21.15.190:0/1410077149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb21c101370 0x7fb21c195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:50:42.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.794+0000 7fb21a59c700 1 -- 172.21.15.190:0/1410077149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb21c101370 msgr2=0x7fb21c195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:50:42.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.794+0000 7fb21a59c700 1 --2- 172.21.15.190:0/1410077149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb21c101370 0x7fb21c195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:50:42.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.794+0000 7fb21a59c700 1 -- 172.21.15.190:0/1410077149 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2100095f0 con 0x7fb21c100170 2024-03-26T18:50:42.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.794+0000 7fb21a59c700 1 --2- 172.21.15.190:0/1410077149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb21c100170 0x7fb21c1959b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb210000c00 tx=0x7fb21000f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:50:42.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.794+0000 7fb20b7fe700 1 -- 172.21.15.190:0/1410077149 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb21001d070 con 0x7fb21c100170 2024-03-26T18:50:42.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.794+0000 7fb220b4c700 1 -- 172.21.15.190:0/1410077149 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21c19b0a0 con 0x7fb21c100170 2024-03-26T18:50:42.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.794+0000 7fb20b7fe700 1 -- 172.21.15.190:0/1410077149 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb210005dc0 con 0x7fb21c100170 2024-03-26T18:50:42.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.794+0000 7fb220b4c700 1 -- 172.21.15.190:0/1410077149 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21c19b590 con 0x7fb21c100170 2024-03-26T18:50:42.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.794+0000 7fb20b7fe700 1 -- 172.21.15.190:0/1410077149 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2100226a0 con 0x7fb21c100170 2024-03-26T18:50:42.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.795+0000 7fb220b4c700 1 -- 172.21.15.190:0/1410077149 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c108cc0 con 0x7fb21c100170 2024-03-26T18:50:42.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.796+0000 7fb20b7fe700 1 -- 172.21.15.190:0/1410077149 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fb210003910 con 0x7fb21c100170 2024-03-26T18:50:42.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.796+0000 7fb20b7fe700 1 --2- 172.21.15.190:0/1410077149 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb20406c680 0x7fb20406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:50:42.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.796+0000 7fb20b7fe700 1 -- 172.21.15.190:0/1410077149 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4206+0+0 (secure 0 0 0) 0x7fb21008c4c0 con 0x7fb21c100170 2024-03-26T18:50:42.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.796+0000 7fb219d9b700 1 --2- 172.21.15.190:0/1410077149 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb20406c680 0x7fb20406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:50:42.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.797+0000 7fb219d9b700 1 --2- 172.21.15.190:0/1410077149 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb20406c680 0x7fb20406eb30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009bf0 tx=0x7fb20c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:50:42.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.801+0000 7fb20b7fe700 1 -- 172.21.15.190:0/1410077149 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb21005ab20 con 0x7fb21c100170 2024-03-26T18:50:42.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:50:42.996+0000 7fb220b4c700 1 -- 172.21.15.190:0/1410077149 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb21c19bce0 con 0x7fb20406c680 2024-03-26T18:50:43.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:43 smithi190 ceph-mon[35043]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:43.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:43 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:50:43.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:43 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:50:43.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:43 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:43.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:43 smithi102 ceph-mon[29002]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:43.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:43 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T18:50:43.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:43 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T18:50:43.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:43 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:44.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:44 smithi190 ceph-mon[35043]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:50:44.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:44 smithi102 ceph-mon[29002]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:50:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:45 smithi190 ceph-mon[35043]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:45.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:45 smithi102 ceph-mon[29002]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:46.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:46 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/3051802089' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5fd67007-d7ae-43b6-8002-258fbfc5a797"}]: dispatch 2024-03-26T18:50:46.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:46 smithi102 ceph-mon[29002]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5fd67007-d7ae-43b6-8002-258fbfc5a797"}]: dispatch 2024-03-26T18:50:46.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:46 smithi102 ceph-mon[29002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5fd67007-d7ae-43b6-8002-258fbfc5a797"}]': finished 2024-03-26T18:50:46.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:46 smithi102 ceph-mon[29002]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T18:50:46.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:46 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:50:46.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:46 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/3528228121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:50:46.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:46 smithi190 ceph-mon[35043]: from='client.? 172.21.15.190:0/3051802089' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5fd67007-d7ae-43b6-8002-258fbfc5a797"}]: dispatch 2024-03-26T18:50:46.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:46 smithi190 ceph-mon[35043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5fd67007-d7ae-43b6-8002-258fbfc5a797"}]: dispatch 2024-03-26T18:50:46.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:46 smithi190 ceph-mon[35043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5fd67007-d7ae-43b6-8002-258fbfc5a797"}]': finished 2024-03-26T18:50:46.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:46 smithi190 ceph-mon[35043]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T18:50:46.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:46 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:50:46.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:46 smithi190 ceph-mon[35043]: from='client.? 172.21.15.190:0/3528228121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T18:50:47.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:47 smithi190 ceph-mon[35043]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:47.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:47 smithi102 ceph-mon[29002]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:49.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:49 smithi190 ceph-mon[35043]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:49.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:49 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:50:49.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:49 smithi102 ceph-mon[29002]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:49.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:49 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:50:51.385 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:51 smithi190 ceph-mon[35043]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:51 smithi102 ceph-mon[29002]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:52.555 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:52 smithi190 ceph-mon[35043]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:52.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:52 smithi102 ceph-mon[29002]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:53.432 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:53 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T18:50:53.432 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:53 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:53.432 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:53 smithi190 ceph-mon[35043]: Deploying daemon osd.5 on smithi190 2024-03-26T18:50:53.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:53 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T18:50:53.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:53 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:50:53.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:53 smithi102 ceph-mon[29002]: Deploying daemon osd.5 on smithi190 2024-03-26T18:50:54.479 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:54 smithi190 ceph-mon[35043]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:54.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:54 smithi102 ceph-mon[29002]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:57.270 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:57 smithi190 ceph-mon[35043]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:57.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:57 smithi102 ceph-mon[29002]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:59.130 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:58 smithi190 ceph-mon[35043]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:59.130 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:58 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:59.131 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:58 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:59.131 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:50:58 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:50:59.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:58 smithi102 ceph-mon[29002]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:50:59.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:58 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:59.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:58 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:50:59.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:50:58 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:51:01.517 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:01 smithi190 ceph-mon[35043]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:51:01.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.609+0000 7fb20b7fe700 1 -- 172.21.15.190:0/1410077149 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb21c19bce0 con 0x7fb20406c680 2024-03-26T18:51:01.601 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 5 on host 'smithi190' 2024-03-26T18:51:01.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.611+0000 7fb220b4c700 1 -- 172.21.15.190:0/1410077149 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb20406c680 msgr2=0x7fb20406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:01.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.611+0000 7fb220b4c700 1 --2- 172.21.15.190:0/1410077149 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb20406c680 0x7fb20406eb30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009bf0 tx=0x7fb20c009260 comp rx=0 tx=0).stop 2024-03-26T18:51:01.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.611+0000 7fb220b4c700 1 -- 172.21.15.190:0/1410077149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb21c100170 msgr2=0x7fb21c1959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:01.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.611+0000 7fb220b4c700 1 --2- 172.21.15.190:0/1410077149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb21c100170 0x7fb21c1959b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb210000c00 tx=0x7fb21000f740 comp rx=0 tx=0).stop 2024-03-26T18:51:01.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.611+0000 7fb220b4c700 1 -- 172.21.15.190:0/1410077149 shutdown_connections 2024-03-26T18:51:01.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.611+0000 7fb220b4c700 1 --2- 172.21.15.190:0/1410077149 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb20406c680 0x7fb20406eb30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:01.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.611+0000 7fb220b4c700 1 --2- 172.21.15.190:0/1410077149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb21c100170 0x7fb21c1959b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:01.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.611+0000 7fb220b4c700 1 --2- 172.21.15.190:0/1410077149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb21c101370 0x7fb21c195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:01.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.611+0000 7fb220b4c700 1 -- 172.21.15.190:0/1410077149 >> 172.21.15.190:0/1410077149 conn(0x7fb21c0fb6e0 msgr2=0x7fb21c1045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:01.604 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.611+0000 7fb220b4c700 1 -- 172.21.15.190:0/1410077149 shutdown_connections 2024-03-26T18:51:01.604 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:01.611+0000 7fb220b4c700 1 -- 172.21.15.190:0/1410077149 wait complete. 2024-03-26T18:51:01.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:01 smithi102 ceph-mon[29002]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:51:02.434 DEBUG:teuthology.orchestra.run.smithi190:osd.5> sudo journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T18:51:02.436 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-26T18:51:02.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T18:51:02.663 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:02.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:02 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:02.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:02 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:02.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:02 smithi190 ceph-mon[35043]: from='osd.5 [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T18:51:02.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:02 smithi190 ceph-mon[35043]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T18:51:02.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:02 smithi190 ceph-mon[35043]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:51:02.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:02 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:02.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:02 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:02.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:02 smithi102 ceph-mon[29002]: from='osd.5 [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T18:51:02.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:02 smithi102 ceph-mon[29002]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T18:51:02.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:02 smithi102 ceph-mon[29002]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:51:03.746 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:03 smithi190 ceph-mon[35043]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T18:51:03.746 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:03 smithi190 ceph-mon[35043]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T18:51:03.746 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:03 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:51:03.746 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:03 smithi190 ceph-mon[35043]: from='osd.5 [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:51:03.746 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:03 smithi190 ceph-mon[35043]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:51:03.746 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:03 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:03.746 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:03 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:03.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 18:51:03 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[50392]: 2024-03-26T18:51:03.570+0000 7fd824e1a700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T18:51:03.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 18:51:03 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[50392]: 2024-03-26T18:51:03.578+0000 7fd81ec0b700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T18:51:03.852 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:03 smithi102 ceph-mon[29002]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T18:51:03.852 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:03 smithi102 ceph-mon[29002]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T18:51:03.852 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:03 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:51:03.852 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:03 smithi102 ceph-mon[29002]: from='osd.5 [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:51:03.853 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:03 smithi102 ceph-mon[29002]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:51:03.853 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:03 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:03.853 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:03 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.391+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3288617986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c0101390 msgr2=0x7ff0c01017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.391+0000 7ff0c4fcf700 1 --2- 172.21.15.102:0/3288617986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c0101390 0x7ff0c01017e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009910 tx=0x7ff0b0009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.391+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3288617986 shutdown_connections 2024-03-26T18:51:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.391+0000 7ff0c4fcf700 1 --2- 172.21.15.102:0/3288617986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c0101390 0x7ff0c01017e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.391+0000 7ff0c4fcf700 1 --2- 172.21.15.102:0/3288617986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0c0100190 0x7ff0c01005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.391+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3288617986 >> 172.21.15.102:0/3288617986 conn(0x7ff0c00fb720 msgr2=0x7ff0c00fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.391+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3288617986 shutdown_connections 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.391+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3288617986 wait complete. 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0c4fcf700 1 Processor -- start 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0c4fcf700 1 -- start start 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0c4fcf700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c0100190 0x7ff0c0074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0c4fcf700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0c0101390 0x7ff0c0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0c4fcf700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c0073680 con 0x7ff0c0100190 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0c4fcf700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c00737c0 con 0x7ff0c0101390 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0be59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c0100190 0x7ff0c0074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0be59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c0100190 0x7ff0c0074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57530/0 (socket says 172.21.15.102:57530) 2024-03-26T18:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0be59c700 1 -- 172.21.15.102:0/3336827610 learned_addr learned my addr 172.21.15.102:0/3336827610 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0bdd9b700 1 --2- 172.21.15.102:0/3336827610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0c0101390 0x7ff0c0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0be59c700 1 -- 172.21.15.102:0/3336827610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0c0101390 msgr2=0x7ff0c0073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0be59c700 1 --2- 172.21.15.102:0/3336827610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0c0101390 0x7ff0c0073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.392+0000 7ff0be59c700 1 -- 172.21.15.102:0/3336827610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b00095f0 con 0x7ff0c0100190 2024-03-26T18:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.393+0000 7ff0be59c700 1 --2- 172.21.15.102:0/3336827610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c0100190 0x7ff0c0074af0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff0b400d900 tx=0x7ff0b400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.393+0000 7ff0af7fe700 1 -- 172.21.15.102:0/3336827610 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0b40041f0 con 0x7ff0c0100190 2024-03-26T18:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.393+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3336827610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c0073aa0 con 0x7ff0c0100190 2024-03-26T18:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.393+0000 7ff0af7fe700 1 -- 172.21.15.102:0/3336827610 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff0b4004350 con 0x7ff0c0100190 2024-03-26T18:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.393+0000 7ff0af7fe700 1 -- 172.21.15.102:0/3336827610 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0b4003da0 con 0x7ff0c0100190 2024-03-26T18:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.393+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3336827610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c019da00 con 0x7ff0c0100190 2024-03-26T18:51:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.394+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3336827610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c0108ce0 con 0x7ff0c0100190 2024-03-26T18:51:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.394+0000 7ff0af7fe700 1 -- 172.21.15.102:0/3336827610 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7ff0b4009730 con 0x7ff0c0100190 2024-03-26T18:51:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.394+0000 7ff0af7fe700 1 --2- 172.21.15.102:0/3336827610 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff0a806c680 0x7ff0a806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.395+0000 7ff0af7fe700 1 -- 172.21.15.102:0/3336827610 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4358+0+0 (secure 0 0 0) 0x7ff0b408b5a0 con 0x7ff0c0100190 2024-03-26T18:51:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.395+0000 7ff0bdd9b700 1 --2- 172.21.15.102:0/3336827610 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff0a806c680 0x7ff0a806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.395+0000 7ff0bdd9b700 1 --2- 172.21.15.102:0/3336827610 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff0a806c680 0x7ff0a806eb30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff0b000b5c0 tx=0x7ff0b0005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:04.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.399+0000 7ff0af7fe700 1 -- 172.21.15.102:0/3336827610 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff0b4059b60 con 0x7ff0c0100190 2024-03-26T18:51:04.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.592+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3336827610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7ff0c0108ea0 con 0x7ff0c0100190 2024-03-26T18:51:04.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.592+0000 7ff0af7fe700 1 -- 172.21.15.102:0/3336827610 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7ff0b40596f0 con 0x7ff0c0100190 2024-03-26T18:51:04.585 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:51:04.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.594+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3336827610 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff0a806c680 msgr2=0x7ff0a806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:04.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.594+0000 7ff0c4fcf700 1 --2- 172.21.15.102:0/3336827610 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff0a806c680 0x7ff0a806eb30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff0b000b5c0 tx=0x7ff0b0005960 comp rx=0 tx=0).stop 2024-03-26T18:51:04.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.594+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3336827610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c0100190 msgr2=0x7ff0c0074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:04.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.594+0000 7ff0c4fcf700 1 --2- 172.21.15.102:0/3336827610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c0100190 0x7ff0c0074af0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff0b400d900 tx=0x7ff0b400dcc0 comp rx=0 tx=0).stop 2024-03-26T18:51:04.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.594+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3336827610 shutdown_connections 2024-03-26T18:51:04.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.594+0000 7ff0c4fcf700 1 --2- 172.21.15.102:0/3336827610 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff0a806c680 0x7ff0a806eb30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:04.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.594+0000 7ff0c4fcf700 1 --2- 172.21.15.102:0/3336827610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c0100190 0x7ff0c0074af0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:04.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.594+0000 7ff0c4fcf700 1 --2- 172.21.15.102:0/3336827610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0c0101390 0x7ff0c0073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:04.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.594+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3336827610 >> 172.21.15.102:0/3336827610 conn(0x7ff0c00fb720 msgr2=0x7ff0c01045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:04.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.595+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3336827610 shutdown_connections 2024-03-26T18:51:04.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:04.595+0000 7ff0c4fcf700 1 -- 172.21.15.102:0/3336827610 wait complete. 2024-03-26T18:51:04.829 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:04 smithi190 ceph-mon[35043]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-26T18:51:04.829 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:04 smithi190 ceph-mon[35043]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T18:51:04.829 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:04 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:51:04.829 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:04 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:51:04.829 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:04 smithi190 ceph-mon[35043]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:51:04.830 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:04 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:51:04.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:04 smithi102 ceph-mon[29002]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-26T18:51:04.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:04 smithi102 ceph-mon[29002]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T18:51:04.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:04 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:51:04.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:04 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:51:04.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:04 smithi102 ceph-mon[29002]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T18:51:04.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:04 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:51:05.182 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1711479064,"num_in_osds":6,"osd_in_since":1711479045,"num_remapped_pgs":0} 2024-03-26T18:51:05.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T18:51:05.399 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:05.729 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:05 smithi102 ceph-mon[29002]: purged_snaps scrub starts 2024-03-26T18:51:05.730 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:05 smithi102 ceph-mon[29002]: purged_snaps scrub ok 2024-03-26T18:51:05.730 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:05 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:51:05.730 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:05 smithi102 ceph-mon[29002]: osd.5 [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415] boot 2024-03-26T18:51:05.730 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:05 smithi102 ceph-mon[29002]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T18:51:05.730 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:05 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:51:05.730 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:05 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3336827610' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T18:51:05.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:05 smithi190 ceph-mon[35043]: purged_snaps scrub starts 2024-03-26T18:51:05.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:05 smithi190 ceph-mon[35043]: purged_snaps scrub ok 2024-03-26T18:51:05.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:05 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:51:05.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:05 smithi190 ceph-mon[35043]: osd.5 [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415] boot 2024-03-26T18:51:05.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:05 smithi190 ceph-mon[35043]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T18:51:05.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:05 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:51:05.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:05 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3336827610' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T18:51:06.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T18:51:06.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:06.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: Detected new or changed devices on smithi190 2024-03-26T18:51:06.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:06.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:06.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:51:06.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:51:06.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:51:06.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: Adjusting osd_memory_target on smithi190 to 4970M 2024-03-26T18:51:06.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:06.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:51:06.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:51:06.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:06.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:06 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:51:07.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T18:51:07.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:07.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: Detected new or changed devices on smithi190 2024-03-26T18:51:07.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:07.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:07.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:51:07.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:51:07.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:51:07.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: Adjusting osd_memory_target on smithi190 to 4970M 2024-03-26T18:51:07.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:07.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:51:07.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:51:07.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:07.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:06 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.123+0000 7ff2c957e700 1 -- 172.21.15.102:0/1804617597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c4100d70 msgr2=0x7ff2c4104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.123+0000 7ff2c957e700 1 --2- 172.21.15.102:0/1804617597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c4100d70 0x7ff2c4104d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8009910 tx=0x7ff2b8009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.123+0000 7ff2c957e700 1 -- 172.21.15.102:0/1804617597 shutdown_connections 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.123+0000 7ff2c957e700 1 --2- 172.21.15.102:0/1804617597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c4100d70 0x7ff2c4104d40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.123+0000 7ff2c957e700 1 --2- 172.21.15.102:0/1804617597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c41003d0 0x7ff2c41007a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.123+0000 7ff2c957e700 1 -- 172.21.15.102:0/1804617597 >> 172.21.15.102:0/1804617597 conn(0x7ff2c40fbc90 msgr2=0x7ff2c40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.123+0000 7ff2c957e700 1 -- 172.21.15.102:0/1804617597 shutdown_connections 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.123+0000 7ff2c957e700 1 -- 172.21.15.102:0/1804617597 wait complete. 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.124+0000 7ff2c957e700 1 Processor -- start 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.124+0000 7ff2c957e700 1 -- start start 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.124+0000 7ff2c957e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c41003d0 0x7ff2c4195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.124+0000 7ff2c957e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c4100d70 0x7ff2c4196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:07.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.124+0000 7ff2c957e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c4196970 con 0x7ff2c41003d0 2024-03-26T18:51:07.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.124+0000 7ff2c957e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c419a700 con 0x7ff2c4100d70 2024-03-26T18:51:07.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.124+0000 7ff2c2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c41003d0 0x7ff2c4195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:07.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.124+0000 7ff2c2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c41003d0 0x7ff2c4195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57550/0 (socket says 172.21.15.102:57550) 2024-03-26T18:51:07.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.124+0000 7ff2c2ffd700 1 -- 172.21.15.102:0/366541008 learned_addr learned my addr 172.21.15.102:0/366541008 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:07.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.124+0000 7ff2c27fc700 1 --2- 172.21.15.102:0/366541008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c4100d70 0x7ff2c4196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:07.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.125+0000 7ff2c2ffd700 1 -- 172.21.15.102:0/366541008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c4100d70 msgr2=0x7ff2c4196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:07.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.125+0000 7ff2c2ffd700 1 --2- 172.21.15.102:0/366541008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c4100d70 0x7ff2c4196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:07.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.125+0000 7ff2c2ffd700 1 -- 172.21.15.102:0/366541008 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b80095f0 con 0x7ff2c41003d0 2024-03-26T18:51:07.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.125+0000 7ff2c2ffd700 1 --2- 172.21.15.102:0/366541008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c41003d0 0x7ff2c4195d50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0060b0 tx=0x7ff2bc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:07.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.125+0000 7ff2abfff700 1 -- 172.21.15.102:0/366541008 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2bc00dda0 con 0x7ff2c41003d0 2024-03-26T18:51:07.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.125+0000 7ff2c957e700 1 -- 172.21.15.102:0/366541008 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c419a9e0 con 0x7ff2c41003d0 2024-03-26T18:51:07.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.125+0000 7ff2abfff700 1 -- 172.21.15.102:0/366541008 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff2bc00f040 con 0x7ff2c41003d0 2024-03-26T18:51:07.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.125+0000 7ff2c957e700 1 -- 172.21.15.102:0/366541008 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c419af30 con 0x7ff2c41003d0 2024-03-26T18:51:07.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.125+0000 7ff2abfff700 1 -- 172.21.15.102:0/366541008 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2bc0146f0 con 0x7ff2c41003d0 2024-03-26T18:51:07.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.126+0000 7ff2c957e700 1 -- 172.21.15.102:0/366541008 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c4108740 con 0x7ff2c41003d0 2024-03-26T18:51:07.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.126+0000 7ff2abfff700 1 -- 172.21.15.102:0/366541008 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7ff2bc015400 con 0x7ff2c41003d0 2024-03-26T18:51:07.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.127+0000 7ff2abfff700 1 --2- 172.21.15.102:0/366541008 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2ac06c680 0x7ff2ac06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:07.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.127+0000 7ff2abfff700 1 -- 172.21.15.102:0/366541008 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff2bc08af80 con 0x7ff2c41003d0 2024-03-26T18:51:07.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.127+0000 7ff2c27fc700 1 --2- 172.21.15.102:0/366541008 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2ac06c680 0x7ff2ac06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:07.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.127+0000 7ff2c27fc700 1 --2- 172.21.15.102:0/366541008 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2ac06c680 0x7ff2ac06eb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4197370 tx=0x7ff2b801a040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:07.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.131+0000 7ff2abfff700 1 -- 172.21.15.102:0/366541008 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff2bc04f720 con 0x7ff2c41003d0 2024-03-26T18:51:07.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.322+0000 7ff2c957e700 1 -- 172.21.15.102:0/366541008 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2c404ea50 con 0x7ff2c41003d0 2024-03-26T18:51:07.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.323+0000 7ff2abfff700 1 -- 172.21.15.102:0/366541008 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11364 (secure 0 0 0) 0x7ff2bc055a50 con 0x7ff2c41003d0 2024-03-26T18:51:07.315 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:51:07.315 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","created":"2024-03-26T18:45:25.446329+0000","modified":"2024-03-26T18:51:05.573960+0000","last_up_change":"2024-03-26T18:51:04.568813+0000","last_in_change":"2024-03-26T18:50:45.452094+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T18:49:50.172501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5458d389-eb88-4798-92bc-b442168b37f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6803","nonce":689794310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6805","nonce":689794310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6809","nonce":689794310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6807","nonce":689794310}]},"public_addr":"172.21.15.102:6803/689794310","cluster_addr":"172.21.15.102:6805/689794310","heartbeat_back_addr":"172.21.15.102:6809/689794310","heartbeat_front_addr":"172.21.15.102:6807/689794310","state":["exists","up"]},{"osd":1,"uuid":"87508f24-8b3c-4f6d-add2-00ad59300da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6811","nonce":391196241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6813","nonce":391196241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6817","nonce":391196241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6815","nonce":391196241}]},"public_addr":"172.21.15.102:6811/391196241","cluster_addr":"172.21.15.102:6813/391196241","heartbeat_back_addr":"172.21.15.102:6817/391196241","heartbeat_front_addr":"172.21.15.102:6815/391196241","state":["exists","up"]},{"osd":2,"uuid":"c0179546-10c5-41f2-af0c-be93c2bbf5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6819","nonce":3930922840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6821","nonce":3930922840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6825","nonce":3930922840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6823","nonce":3930922840}]},"public_addr":"172.21.15.102:6819/3930922840","cluster_addr":"172.21.15.102:6821/3930922840","heartbeat_back_addr":"172.21.15.102:6825/3930922840","heartbeat_front_addr":"172.21.15.102:6823/3930922840","state":["exists","up"]},{"osd":3,"uuid":"4d524f7b-8b31-469b-b0c2-b94079febaeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6801","nonce":1428928793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6803","nonce":1428928793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6807","nonce":1428928793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6805","nonce":1428928793}]},"public_addr":"172.21.15.190:6801/1428928793","cluster_addr":"172.21.15.190:6803/1428928793","heartbeat_back_addr":"172.21.15.190:6807/1428928793","heartbeat_front_addr":"172.21.15.190:6805/1428928793","state":["exists","up"]},{"osd":4,"uuid":"865baef8-dd72-4449-b3a4-893801bc2958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6809","nonce":320320306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6811","nonce":320320306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6815","nonce":320320306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6813","nonce":320320306}]},"public_addr":"172.21.15.190:6809/320320306","cluster_addr":"172.21.15.190:6811/320320306","heartbeat_back_addr":"172.21.15.190:6815/320320306","heartbeat_front_addr":"172.21.15.190:6813/320320306","state":["exists","up"]},{"osd":5,"uuid":"5fd67007-d7ae-43b6-8002-258fbfc5a797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6817","nonce":3245135415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6819","nonce":3245135415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6823","nonce":3245135415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6821","nonce":3245135415}]},"public_addr":"172.21.15.190:6817/3245135415","cluster_addr":"172.21.15.190:6819/3245135415","heartbeat_back_addr":"172.21.15.190:6823/3245135415","heartbeat_front_addr":"172.21.15.190:6821/3245135415","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:48:51.274164+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:18.020320+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:46.066766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:10.365857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:36.582495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:51:02.797707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6800/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/3353740154":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1842664791":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1212290737":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6801/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6800/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/1994330698":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/1295801893":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/4276276941":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1380712530":"2024-03-27T18:45:55.925005+0000","172.21.15.102:6801/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:6800/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/2686253586":"2024-03-27T18:45:55.925005+0000","172.21.15.102:6801/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/2752725029":"2024-03-27T18:46:28.035077+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T18:51:07.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.325+0000 7ff2c957e700 1 -- 172.21.15.102:0/366541008 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2ac06c680 msgr2=0x7ff2ac06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:07.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.325+0000 7ff2c957e700 1 --2- 172.21.15.102:0/366541008 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2ac06c680 0x7ff2ac06eb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4197370 tx=0x7ff2b801a040 comp rx=0 tx=0).stop 2024-03-26T18:51:07.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.325+0000 7ff2c957e700 1 -- 172.21.15.102:0/366541008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c41003d0 msgr2=0x7ff2c4195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:07.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.325+0000 7ff2c957e700 1 --2- 172.21.15.102:0/366541008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c41003d0 0x7ff2c4195d50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0060b0 tx=0x7ff2bc00d3b0 comp rx=0 tx=0).stop 2024-03-26T18:51:07.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.325+0000 7ff2c957e700 1 -- 172.21.15.102:0/366541008 shutdown_connections 2024-03-26T18:51:07.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.325+0000 7ff2c957e700 1 --2- 172.21.15.102:0/366541008 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2ac06c680 0x7ff2ac06eb30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:07.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.325+0000 7ff2c957e700 1 --2- 172.21.15.102:0/366541008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c41003d0 0x7ff2c4195d50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:07.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.325+0000 7ff2c957e700 1 --2- 172.21.15.102:0/366541008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c4100d70 0x7ff2c4196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:07.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.325+0000 7ff2c957e700 1 -- 172.21.15.102:0/366541008 >> 172.21.15.102:0/366541008 conn(0x7ff2c40fbc90 msgr2=0x7ff2c40fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:07.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.325+0000 7ff2c957e700 1 -- 172.21.15.102:0/366541008 shutdown_connections 2024-03-26T18:51:07.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:07.325+0000 7ff2c957e700 1 -- 172.21.15.102:0/366541008 wait complete. 2024-03-26T18:51:07.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:07 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:51:07.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:07 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:51:07.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:07 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:07.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:07 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/366541008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T18:51:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:51:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:51:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:07 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:51:07.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:07 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/366541008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T18:51:07.984 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T18:49:50.172501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T18:51:07.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T18:51:08.195 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:08.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:08 smithi102 ceph-mon[29002]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:09.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:08 smithi190 ceph-mon[35043]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.881+0000 7f446edd8700 1 -- 172.21.15.102:0/856651039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44680ff390 msgr2=0x7f44680ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.881+0000 7f446edd8700 1 --2- 172.21.15.102:0/856651039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44680ff390 0x7f44680ff760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f445c009910 tx=0x7f445c009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.881+0000 7f446edd8700 1 -- 172.21.15.102:0/856651039 shutdown_connections 2024-03-26T18:51:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.881+0000 7f446edd8700 1 --2- 172.21.15.102:0/856651039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44680ffd30 0x7f4468103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.881+0000 7f446edd8700 1 --2- 172.21.15.102:0/856651039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44680ff390 0x7f44680ff760 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.881+0000 7f446edd8700 1 -- 172.21.15.102:0/856651039 >> 172.21.15.102:0/856651039 conn(0x7f4468074f50 msgr2=0x7f4468075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.881+0000 7f446edd8700 1 -- 172.21.15.102:0/856651039 shutdown_connections 2024-03-26T18:51:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.881+0000 7f446edd8700 1 -- 172.21.15.102:0/856651039 wait complete. 2024-03-26T18:51:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446edd8700 1 Processor -- start 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446edd8700 1 -- start start 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446edd8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44680ff390 0x7f4468194d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446edd8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44680ffd30 0x7f4468195290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446edd8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4468195970 con 0x7f44680ff390 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446edd8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4468199700 con 0x7f44680ffd30 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446cb74700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44680ff390 0x7f4468194d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446cb74700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44680ff390 0x7f4468194d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39664/0 (socket says 172.21.15.102:39664) 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446cb74700 1 -- 172.21.15.102:0/1094005034 learned_addr learned my addr 172.21.15.102:0/1094005034 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446cb74700 1 -- 172.21.15.102:0/1094005034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44680ffd30 msgr2=0x7f4468195290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446cb74700 1 --2- 172.21.15.102:0/1094005034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44680ffd30 0x7f4468195290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446cb74700 1 -- 172.21.15.102:0/1094005034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c0095f0 con 0x7f44680ff390 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.882+0000 7f446cb74700 1 --2- 172.21.15.102:0/1094005034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44680ff390 0x7f4468194d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f445c006010 tx=0x7f445c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.883+0000 7f4465ffb700 1 -- 172.21.15.102:0/1094005034 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f445c01d070 con 0x7f44680ff390 2024-03-26T18:51:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.883+0000 7f4465ffb700 1 -- 172.21.15.102:0/1094005034 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f445c00f460 con 0x7f44680ff390 2024-03-26T18:51:09.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.883+0000 7f4465ffb700 1 -- 172.21.15.102:0/1094005034 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f445c021640 con 0x7f44680ff390 2024-03-26T18:51:09.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.883+0000 7f446edd8700 1 -- 172.21.15.102:0/1094005034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4468199980 con 0x7f44680ff390 2024-03-26T18:51:09.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.883+0000 7f446edd8700 1 -- 172.21.15.102:0/1094005034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4468199e70 con 0x7f44680ff390 2024-03-26T18:51:09.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.884+0000 7f446edd8700 1 -- 172.21.15.102:0/1094005034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4468107650 con 0x7f44680ff390 2024-03-26T18:51:09.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.884+0000 7f4465ffb700 1 -- 172.21.15.102:0/1094005034 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f445c00faf0 con 0x7f44680ff390 2024-03-26T18:51:09.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.885+0000 7f4465ffb700 1 --2- 172.21.15.102:0/1094005034 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f445806c680 0x7f445806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:09.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.885+0000 7f4465ffb700 1 -- 172.21.15.102:0/1094005034 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f445c08c7c0 con 0x7f44680ff390 2024-03-26T18:51:09.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.885+0000 7f4467fff700 1 --2- 172.21.15.102:0/1094005034 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f445806c680 0x7f445806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:09.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.885+0000 7f4467fff700 1 --2- 172.21.15.102:0/1094005034 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f445806c680 0x7f445806eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4468196370 tx=0x7f4454006d20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:09.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:09.889+0000 7f4465ffb700 1 -- 172.21.15.102:0/1094005034 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f445c057260 con 0x7f44680ff390 2024-03-26T18:51:10.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.077+0000 7f446edd8700 1 -- 172.21.15.102:0/1094005034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f446802ccf0 con 0x7f44680ff390 2024-03-26T18:51:10.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.078+0000 7f4465ffb700 1 -- 172.21.15.102:0/1094005034 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f445c0260a0 con 0x7f44680ff390 2024-03-26T18:51:10.070 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-26T18:51:10.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.080+0000 7f446edd8700 1 -- 172.21.15.102:0/1094005034 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f445806c680 msgr2=0x7f445806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:10.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.080+0000 7f446edd8700 1 --2- 172.21.15.102:0/1094005034 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f445806c680 0x7f445806eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4468196370 tx=0x7f4454006d20 comp rx=0 tx=0).stop 2024-03-26T18:51:10.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.080+0000 7f446edd8700 1 -- 172.21.15.102:0/1094005034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44680ff390 msgr2=0x7f4468194d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:10.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.080+0000 7f446edd8700 1 --2- 172.21.15.102:0/1094005034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44680ff390 0x7f4468194d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f445c006010 tx=0x7f445c00be20 comp rx=0 tx=0).stop 2024-03-26T18:51:10.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.080+0000 7f446edd8700 1 -- 172.21.15.102:0/1094005034 shutdown_connections 2024-03-26T18:51:10.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.080+0000 7f446edd8700 1 --2- 172.21.15.102:0/1094005034 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f445806c680 0x7f445806eb30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:10.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.080+0000 7f446edd8700 1 --2- 172.21.15.102:0/1094005034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44680ff390 0x7f4468194d50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:10.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.080+0000 7f446edd8700 1 --2- 172.21.15.102:0/1094005034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44680ffd30 0x7f4468195290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:10.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.080+0000 7f446edd8700 1 -- 172.21.15.102:0/1094005034 >> 172.21.15.102:0/1094005034 conn(0x7f4468074f50 msgr2=0x7f44680fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:10.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.080+0000 7f446edd8700 1 -- 172.21.15.102:0/1094005034 shutdown_connections 2024-03-26T18:51:10.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:10.080+0000 7f446edd8700 1 -- 172.21.15.102:0/1094005034 wait complete. 2024-03-26T18:51:10.746 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T18:51:10.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T18:51:10.950 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:11.252 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:10 smithi102 ceph-mon[29002]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:11.252 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:10 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1094005034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T18:51:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:10 smithi190 ceph-mon[35043]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:11.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:10 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/1094005034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T18:51:12.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.584+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2965272800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa568101370 msgr2=0x7fa5681017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:12.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.584+0000 7fa56e6f8700 1 --2- 172.21.15.102:0/2965272800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa568101370 0x7fa5681017c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009910 tx=0x7fa55c009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:12.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.584+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2965272800 shutdown_connections 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.584+0000 7fa56e6f8700 1 --2- 172.21.15.102:0/2965272800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa568101370 0x7fa5681017c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.584+0000 7fa56e6f8700 1 --2- 172.21.15.102:0/2965272800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa568100170 0x7fa568100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.584+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2965272800 >> 172.21.15.102:0/2965272800 conn(0x7fa5680fb6e0 msgr2=0x7fa5680fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.584+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2965272800 shutdown_connections 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.584+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2965272800 wait complete. 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.584+0000 7fa56e6f8700 1 Processor -- start 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.584+0000 7fa56e6f8700 1 -- start start 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa56e6f8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa568100170 0x7fa568195a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa56e6f8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa568101370 0x7fa568195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa56e6f8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa568196590 con 0x7fa568101370 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa56e6f8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5681966d0 con 0x7fa568100170 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa5677fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa568101370 0x7fa568195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa5677fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa568101370 0x7fa568195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39688/0 (socket says 172.21.15.102:39688) 2024-03-26T18:51:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa5677fe700 1 -- 172.21.15.102:0/2471605651 learned_addr learned my addr 172.21.15.102:0/2471605651 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa567fff700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa568100170 0x7fa568195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa5677fe700 1 -- 172.21.15.102:0/2471605651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa568100170 msgr2=0x7fa568195a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa5677fe700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa568100170 0x7fa568195a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa5677fe700 1 -- 172.21.15.102:0/2471605651 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c0095f0 con 0x7fa568101370 2024-03-26T18:51:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa567fff700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa568100170 0x7fa568195a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T18:51:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa5677fe700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa568101370 0x7fa568195f70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa55c006010 tx=0x7fa55c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa5657fa700 1 -- 172.21.15.102:0/2471605651 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa55c01d070 con 0x7fa568101370 2024-03-26T18:51:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa5657fa700 1 -- 172.21.15.102:0/2471605651 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa55c00f460 con 0x7fa568101370 2024-03-26T18:51:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.585+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2471605651 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56819b120 con 0x7fa568101370 2024-03-26T18:51:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.586+0000 7fa5657fa700 1 -- 172.21.15.102:0/2471605651 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa55c021640 con 0x7fa568101370 2024-03-26T18:51:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.586+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2471605651 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56819b690 con 0x7fa568101370 2024-03-26T18:51:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.586+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2471605651 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa568108d60 con 0x7fa568101370 2024-03-26T18:51:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.587+0000 7fa5657fa700 1 -- 172.21.15.102:0/2471605651 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fa55c00fab0 con 0x7fa568101370 2024-03-26T18:51:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.587+0000 7fa5657fa700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa55006c680 0x7fa55006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.588+0000 7fa567fff700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa55006c680 0x7fa55006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.588+0000 7fa5657fa700 1 -- 172.21.15.102:0/2471605651 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa55c08c9c0 con 0x7fa568101370 2024-03-26T18:51:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.588+0000 7fa567fff700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa55006c680 0x7fa55006eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa5681011d0 tx=0x7fa558008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:12.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.592+0000 7fa5657fa700 1 -- 172.21.15.102:0/2471605651 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa55c05ae70 con 0x7fa568101370 2024-03-26T18:51:12.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.863+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2471605651 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fa56819c040 con 0x7fa568101370 2024-03-26T18:51:12.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.866+0000 7fa5657fa700 1 -- 172.21.15.102:0/2471605651 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7fa55c0260f0 con 0x7fa568101370 2024-03-26T18:51:12.859 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-03-26T18:51:12.859 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQAgGQNmGMCHMxAA/J88ow+NQgMh0xDEdnjVsw== 2024-03-26T18:51:12.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.868+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2471605651 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa55006c680 msgr2=0x7fa55006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:12.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.868+0000 7fa56e6f8700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa55006c680 0x7fa55006eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa5681011d0 tx=0x7fa558008040 comp rx=0 tx=0).stop 2024-03-26T18:51:12.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.868+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2471605651 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa568101370 msgr2=0x7fa568195f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:12.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.868+0000 7fa56e6f8700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa568101370 0x7fa568195f70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa55c006010 tx=0x7fa55c00be20 comp rx=0 tx=0).stop 2024-03-26T18:51:12.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.869+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2471605651 shutdown_connections 2024-03-26T18:51:12.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.869+0000 7fa56e6f8700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa55006c680 0x7fa55006eb30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:12.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.869+0000 7fa56e6f8700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa568100170 0x7fa568195a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:12.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.869+0000 7fa56e6f8700 1 --2- 172.21.15.102:0/2471605651 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa568101370 0x7fa568195f70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:12.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.869+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2471605651 >> 172.21.15.102:0/2471605651 conn(0x7fa5680fb6e0 msgr2=0x7fa5681045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:12.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.869+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2471605651 shutdown_connections 2024-03-26T18:51:12.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:12.869+0000 7fa56e6f8700 1 -- 172.21.15.102:0/2471605651 wait complete. 2024-03-26T18:51:13.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:12 smithi102 ceph-mon[29002]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:13.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:12 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2471605651' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T18:51:13.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:12 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2471605651' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T18:51:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:12 smithi190 ceph-mon[35043]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:12 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2471605651' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T18:51:13.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:12 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2471605651' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T18:51:13.585 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:51:13.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T18:51:13.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T18:51:13.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T18:51:13.843 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi190/config 2024-03-26T18:51:15.236 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:14 smithi190 ceph-mon[35043]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:15.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:14 smithi102 ceph-mon[29002]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:15.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.445+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/3574671151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf0100250 msgr2=0x7f6bf01006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:15.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.445+0000 7f6bf4fa7700 1 --2- 172.21.15.190:0/3574671151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf0100250 0x7f6bf01006a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009910 tx=0x7f6be4009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:15.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.445+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/3574671151 shutdown_connections 2024-03-26T18:51:15.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.445+0000 7f6bf4fa7700 1 --2- 172.21.15.190:0/3574671151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf0100250 0x7f6bf01006a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:15.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.445+0000 7f6bf4fa7700 1 --2- 172.21.15.190:0/3574671151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bf00ff050 0x7f6bf00ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:15.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.445+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/3574671151 >> 172.21.15.190:0/3574671151 conn(0x7f6bf00fa5e0 msgr2=0x7f6bf00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:15.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.445+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/3574671151 shutdown_connections 2024-03-26T18:51:15.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.445+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/3574671151 wait complete. 2024-03-26T18:51:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bf4fa7700 1 Processor -- start 2024-03-26T18:51:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bf4fa7700 1 -- start start 2024-03-26T18:51:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bf4fa7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf00ff050 0x7f6bf0194860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bf4fa7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bf0100250 0x7f6bf0194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bf4fa7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf01953c0 con 0x7f6bf0100250 2024-03-26T18:51:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bf4fa7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf0195500 con 0x7f6bf00ff050 2024-03-26T18:51:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bee59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf00ff050 0x7f6bf0194860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bee59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf00ff050 0x7f6bf0194860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:40828/0 (socket says 172.21.15.190:40828) 2024-03-26T18:51:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bee59c700 1 -- 172.21.15.190:0/1605894064 learned_addr learned my addr 172.21.15.190:0/1605894064 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-26T18:51:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bedd9b700 1 --2- 172.21.15.190:0/1605894064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bf0100250 0x7f6bf0194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:15.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bee59c700 1 -- 172.21.15.190:0/1605894064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bf0100250 msgr2=0x7f6bf0194da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:15.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bee59c700 1 --2- 172.21.15.190:0/1605894064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bf0100250 0x7f6bf0194da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:15.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bee59c700 1 -- 172.21.15.190:0/1605894064 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be40095f0 con 0x7f6bf00ff050 2024-03-26T18:51:15.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.446+0000 7f6bee59c700 1 --2- 172.21.15.190:0/1605894064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf00ff050 0x7f6bf0194860 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6be0007e90 tx=0x7f6be000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:15.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.447+0000 7f6bdf7fe700 1 -- 172.21.15.190:0/1605894064 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6be0017070 con 0x7f6bf00ff050 2024-03-26T18:51:15.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.447+0000 7f6bdf7fe700 1 -- 172.21.15.190:0/1605894064 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6be000f040 con 0x7f6bf00ff050 2024-03-26T18:51:15.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.447+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/1605894064 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf0199fb0 con 0x7f6bf00ff050 2024-03-26T18:51:15.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.447+0000 7f6bdf7fe700 1 -- 172.21.15.190:0/1605894064 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6be0004880 con 0x7f6bf00ff050 2024-03-26T18:51:15.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.447+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/1605894064 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf019a470 con 0x7f6bf00ff050 2024-03-26T18:51:15.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.448+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/1605894064 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf0107ba0 con 0x7f6bf00ff050 2024-03-26T18:51:15.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.448+0000 7f6bdf7fe700 1 -- 172.21.15.190:0/1605894064 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f6be00040b0 con 0x7f6bf00ff050 2024-03-26T18:51:15.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.449+0000 7f6bdf7fe700 1 --2- 172.21.15.190:0/1605894064 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6bd806c680 0x7f6bd806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:15.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.449+0000 7f6bdf7fe700 1 -- 172.21.15.190:0/1605894064 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6be008ae70 con 0x7f6bf00ff050 2024-03-26T18:51:15.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.449+0000 7f6bedd9b700 1 --2- 172.21.15.190:0/1605894064 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6bd806c680 0x7f6bd806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:15.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.449+0000 7f6bedd9b700 1 --2- 172.21.15.190:0/1605894064 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6bd806c680 0x7f6bd806eb30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b5c0 tx=0x7f6be4005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:15.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.453+0000 7f6bdf7fe700 1 -- 172.21.15.190:0/1605894064 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6be00593a0 con 0x7f6bf00ff050 2024-03-26T18:51:15.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.717+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/1605894064 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f6bf019adb0 con 0x7f6bf00ff050 2024-03-26T18:51:15.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.722+0000 7f6bdf7fe700 1 -- 172.21.15.190:0/1605894064 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f6be0058f30 con 0x7f6bf00ff050 2024-03-26T18:51:15.714 INFO:teuthology.orchestra.run.smithi190.stdout:[client.1] 2024-03-26T18:51:15.714 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAjGQNmMEbgKhAA3cB9HXNNmWUW6VEcsnfu7Q== 2024-03-26T18:51:15.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.724+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/1605894064 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6bd806c680 msgr2=0x7f6bd806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:15.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.724+0000 7f6bf4fa7700 1 --2- 172.21.15.190:0/1605894064 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6bd806c680 0x7f6bd806eb30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b5c0 tx=0x7f6be4005960 comp rx=0 tx=0).stop 2024-03-26T18:51:15.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.724+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/1605894064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf00ff050 msgr2=0x7f6bf0194860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.724+0000 7f6bf4fa7700 1 --2- 172.21.15.190:0/1605894064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf00ff050 0x7f6bf0194860 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6be0007e90 tx=0x7f6be000d3b0 comp rx=0 tx=0).stop 2024-03-26T18:51:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.724+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/1605894064 shutdown_connections 2024-03-26T18:51:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.724+0000 7f6bf4fa7700 1 --2- 172.21.15.190:0/1605894064 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6bd806c680 0x7f6bd806eb30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.724+0000 7f6bf4fa7700 1 --2- 172.21.15.190:0/1605894064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf00ff050 0x7f6bf0194860 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.724+0000 7f6bf4fa7700 1 --2- 172.21.15.190:0/1605894064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bf0100250 0x7f6bf0194da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.724+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/1605894064 >> 172.21.15.190:0/1605894064 conn(0x7f6bf00fa5e0 msgr2=0x7f6bf0103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.725+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/1605894064 shutdown_connections 2024-03-26T18:51:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:51:15.725+0000 7f6bf4fa7700 1 -- 172.21.15.190:0/1605894064 wait complete. 2024-03-26T18:51:16.364 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:16 smithi190 ceph-mon[35043]: from='client.? 172.21.15.190:0/1605894064' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T18:51:16.364 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:16 smithi190 ceph-mon[35043]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T18:51:16.364 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:16 smithi190 ceph-mon[35043]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T18:51:16.386 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:51:16.386 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T18:51:16.386 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T18:51:16.429 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T18:51:16.429 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T18:51:16.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T18:51:16.457 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:16 smithi102 ceph-mon[29002]: from='client.? 172.21.15.190:0/1605894064' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T18:51:16.457 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:16 smithi102 ceph-mon[29002]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T18:51:16.457 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:16 smithi102 ceph-mon[29002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T18:51:16.649 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:17.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:17 smithi102 ceph-mon[29002]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:17.439 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:17 smithi190 ceph-mon[35043]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:18.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.266+0000 7f1495551700 1 -- 172.21.15.102:0/400286511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14900ff390 msgr2=0x7f14900ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:18.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.266+0000 7f1495551700 1 --2- 172.21.15.102:0/400286511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14900ff390 0x7f14900ff760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f148800b0a0 tx=0x7f148800b3b0 comp rx=0 tx=0).stop 2024-03-26T18:51:18.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.266+0000 7f1495551700 1 -- 172.21.15.102:0/400286511 shutdown_connections 2024-03-26T18:51:18.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.266+0000 7f1495551700 1 --2- 172.21.15.102:0/400286511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14900ffd30 0x7f1490103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.266+0000 7f1495551700 1 --2- 172.21.15.102:0/400286511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14900ff390 0x7f14900ff760 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.266+0000 7f1495551700 1 -- 172.21.15.102:0/400286511 >> 172.21.15.102:0/400286511 conn(0x7f1490074f50 msgr2=0x7f1490075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.266+0000 7f1495551700 1 -- 172.21.15.102:0/400286511 shutdown_connections 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.266+0000 7f1495551700 1 -- 172.21.15.102:0/400286511 wait complete. 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f1495551700 1 Processor -- start 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f1495551700 1 -- start start 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f1495551700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14900ffd30 0x7f1490196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f1495551700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1490196550 0x7f149019a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f1495551700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1490196b60 con 0x7f1490196550 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f1495551700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1490196cd0 con 0x7f14900ffd30 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f148e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1490196550 0x7f149019a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f148e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1490196550 0x7f149019a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39716/0 (socket says 172.21.15.102:39716) 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f148e7fc700 1 -- 172.21.15.102:0/3442996255 learned_addr learned my addr 172.21.15.102:0/3442996255 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f148effd700 1 --2- 172.21.15.102:0/3442996255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14900ffd30 0x7f1490196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:18.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f148e7fc700 1 -- 172.21.15.102:0/3442996255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14900ffd30 msgr2=0x7f1490196010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:18.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f148e7fc700 1 --2- 172.21.15.102:0/3442996255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14900ffd30 0x7f1490196010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:18.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.267+0000 7f148e7fc700 1 -- 172.21.15.102:0/3442996255 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1488009d00 con 0x7f1490196550 2024-03-26T18:51:18.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.268+0000 7f148e7fc700 1 --2- 172.21.15.102:0/3442996255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1490196550 0x7f149019a9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f147800d930 tx=0x7f147800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:18.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.268+0000 7f1487fff700 1 -- 172.21.15.102:0/3442996255 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1478009900 con 0x7f1490196550 2024-03-26T18:51:18.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.268+0000 7f1487fff700 1 -- 172.21.15.102:0/3442996255 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1478010460 con 0x7f1490196550 2024-03-26T18:51:18.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.268+0000 7f1495551700 1 -- 172.21.15.102:0/3442996255 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149019afc0 con 0x7f1490196550 2024-03-26T18:51:18.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.268+0000 7f1487fff700 1 -- 172.21.15.102:0/3442996255 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f147800f5f0 con 0x7f1490196550 2024-03-26T18:51:18.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.268+0000 7f1495551700 1 -- 172.21.15.102:0/3442996255 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149019b510 con 0x7f1490196550 2024-03-26T18:51:18.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.269+0000 7f1495551700 1 -- 172.21.15.102:0/3442996255 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f149004ea50 con 0x7f1490196550 2024-03-26T18:51:18.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.269+0000 7f1487fff700 1 -- 172.21.15.102:0/3442996255 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f14780105d0 con 0x7f1490196550 2024-03-26T18:51:18.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.270+0000 7f1487fff700 1 --2- 172.21.15.102:0/3442996255 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f147c06c680 0x7f147c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:18.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.270+0000 7f148effd700 1 --2- 172.21.15.102:0/3442996255 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f147c06c680 0x7f147c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:18.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.270+0000 7f1487fff700 1 -- 172.21.15.102:0/3442996255 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f147808b140 con 0x7f1490196550 2024-03-26T18:51:18.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.270+0000 7f148effd700 1 --2- 172.21.15.102:0/3442996255 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f147c06c680 0x7f147c06eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1488006070 tx=0x7f1488006000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:18.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.274+0000 7f1487fff700 1 -- 172.21.15.102:0/3442996255 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1478055b60 con 0x7f1490196550 2024-03-26T18:51:18.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.516+0000 7f1495551700 1 -- 172.21.15.102:0/3442996255 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f149019b790 con 0x7f1490196550 2024-03-26T18:51:18.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.519+0000 7f1487fff700 1 -- 172.21.15.102:0/3442996255 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+172860 (secure 0 0 0) 0x7f1478059180 con 0x7f1490196550 2024-03-26T18:51:18.514 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:51:18.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.522+0000 7f1495551700 1 -- 172.21.15.102:0/3442996255 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f147c06c680 msgr2=0x7f147c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.522+0000 7f1495551700 1 --2- 172.21.15.102:0/3442996255 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f147c06c680 0x7f147c06eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1488006070 tx=0x7f1488006000 comp rx=0 tx=0).stop 2024-03-26T18:51:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.522+0000 7f1495551700 1 -- 172.21.15.102:0/3442996255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1490196550 msgr2=0x7f149019a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.522+0000 7f1495551700 1 --2- 172.21.15.102:0/3442996255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1490196550 0x7f149019a9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f147800d930 tx=0x7f147800dcf0 comp rx=0 tx=0).stop 2024-03-26T18:51:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.522+0000 7f1495551700 1 -- 172.21.15.102:0/3442996255 shutdown_connections 2024-03-26T18:51:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.522+0000 7f1495551700 1 --2- 172.21.15.102:0/3442996255 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f147c06c680 0x7f147c06eb30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.522+0000 7f1495551700 1 --2- 172.21.15.102:0/3442996255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14900ffd30 0x7f1490196010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.522+0000 7f1495551700 1 --2- 172.21.15.102:0/3442996255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1490196550 0x7f149019a9c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.522+0000 7f1495551700 1 -- 172.21.15.102:0/3442996255 >> 172.21.15.102:0/3442996255 conn(0x7f1490074f50 msgr2=0x7f14901035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.522+0000 7f1495551700 1 -- 172.21.15.102:0/3442996255 shutdown_connections 2024-03-26T18:51:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:18.522+0000 7f1495551700 1 -- 172.21.15.102:0/3442996255 wait complete. 2024-03-26T18:51:19.172 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":19,"active_gid":14211,"active_name":"smithi102.rsfpnv","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":748923615},{"type":"v1","addr":"172.21.15.102:6801","nonce":748923615}]},"active_addr":"172.21.15.102:6801/748923615","active_change":"2024-03-26T18:47:33.839295+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14234,"name":"smithi190.zrixos","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2094779473}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":216723823}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1615769300}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1480032448}]}]} 2024-03-26T18:51:19.176 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T18:51:19.176 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T18:51:19.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T18:51:19.417 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:19.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:19 smithi190 ceph-mon[35043]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:19.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:19 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3442996255' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T18:51:19.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:51:19.537 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:19 smithi102 ceph-mon[29002]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:19.538 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:19 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3442996255' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T18:51:19.538 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:51:21.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.048+0000 7f511cc4b700 1 -- 172.21.15.102:0/1543652237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51180ff2c0 msgr2=0x7f51180ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:21.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.048+0000 7f511cc4b700 1 --2- 172.21.15.102:0/1543652237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51180ff2c0 0x7f51180ff690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5108009910 tx=0x7f5108009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:21.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.048+0000 7f511cc4b700 1 -- 172.21.15.102:0/1543652237 shutdown_connections 2024-03-26T18:51:21.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.048+0000 7f511cc4b700 1 --2- 172.21.15.102:0/1543652237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51180ffc60 0x7f5118103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:21.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.048+0000 7f511cc4b700 1 --2- 172.21.15.102:0/1543652237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51180ff2c0 0x7f51180ff690 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:21.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.048+0000 7f511cc4b700 1 -- 172.21.15.102:0/1543652237 >> 172.21.15.102:0/1543652237 conn(0x7f51180fab80 msgr2=0x7f51180fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:21.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.048+0000 7f511cc4b700 1 -- 172.21.15.102:0/1543652237 shutdown_connections 2024-03-26T18:51:21.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.048+0000 7f511cc4b700 1 -- 172.21.15.102:0/1543652237 wait complete. 2024-03-26T18:51:21.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.049+0000 7f511cc4b700 1 Processor -- start 2024-03-26T18:51:21.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.049+0000 7f511cc4b700 1 -- start start 2024-03-26T18:51:21.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.049+0000 7f511cc4b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51180ff2c0 0x7f5118196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:21.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.049+0000 7f511cc4b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51180ffc60 0x7f51181970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:21.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.049+0000 7f511cc4b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51181976f0 con 0x7f51180ff2c0 2024-03-26T18:51:21.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.049+0000 7f511cc4b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5118197830 con 0x7f51180ffc60 2024-03-26T18:51:21.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.049+0000 7f511659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51180ff2c0 0x7f5118196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:21.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.050+0000 7f511659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51180ff2c0 0x7f5118196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50380/0 (socket says 172.21.15.102:50380) 2024-03-26T18:51:21.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.050+0000 7f511659c700 1 -- 172.21.15.102:0/777038750 learned_addr learned my addr 172.21.15.102:0/777038750 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:21.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.050+0000 7f5115d9b700 1 --2- 172.21.15.102:0/777038750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51180ffc60 0x7f51181970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:21.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.050+0000 7f511659c700 1 -- 172.21.15.102:0/777038750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51180ffc60 msgr2=0x7f51181970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:21.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.050+0000 7f511659c700 1 --2- 172.21.15.102:0/777038750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51180ffc60 0x7f51181970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:21.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.050+0000 7f511659c700 1 -- 172.21.15.102:0/777038750 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51080095f0 con 0x7f51180ff2c0 2024-03-26T18:51:21.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.050+0000 7f511659c700 1 --2- 172.21.15.102:0/777038750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51180ff2c0 0x7f5118196b90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5108006010 tx=0x7f510800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:21.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.050+0000 7f51077fe700 1 -- 172.21.15.102:0/777038750 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f510801d070 con 0x7f51180ff2c0 2024-03-26T18:51:21.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.050+0000 7f51077fe700 1 -- 172.21.15.102:0/777038750 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f510800f460 con 0x7f51180ff2c0 2024-03-26T18:51:21.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.050+0000 7f511cc4b700 1 -- 172.21.15.102:0/777038750 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511819fc00 con 0x7f51180ff2c0 2024-03-26T18:51:21.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.050+0000 7f51077fe700 1 -- 172.21.15.102:0/777038750 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5108017630 con 0x7f51180ff2c0 2024-03-26T18:51:21.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.051+0000 7f511cc4b700 1 -- 172.21.15.102:0/777038750 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51181a0010 con 0x7f51180ff2c0 2024-03-26T18:51:21.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.052+0000 7f51077fe700 1 -- 172.21.15.102:0/777038750 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f51080178f0 con 0x7f51180ff2c0 2024-03-26T18:51:21.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.052+0000 7f511cc4b700 1 -- 172.21.15.102:0/777038750 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51181075d0 con 0x7f51180ff2c0 2024-03-26T18:51:21.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.052+0000 7f51077fe700 1 --2- 172.21.15.102:0/777038750 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f510006c680 0x7f510006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:21.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.052+0000 7f51077fe700 1 -- 172.21.15.102:0/777038750 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f510808c260 con 0x7f51180ff2c0 2024-03-26T18:51:21.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.052+0000 7f5115d9b700 1 --2- 172.21.15.102:0/777038750 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f510006c680 0x7f510006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:21.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.053+0000 7f5115d9b700 1 --2- 172.21.15.102:0/777038750 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f510006c680 0x7f510006eb30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5118069510 tx=0x7f510c00b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:21.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.057+0000 7f51077fe700 1 -- 172.21.15.102:0/777038750 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5108056c30 con 0x7f51180ff2c0 2024-03-26T18:51:21.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.259+0000 7f511cc4b700 1 -- 172.21.15.102:0/777038750 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f511804ea50 con 0x7f51180ff2c0 2024-03-26T18:51:21.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.260+0000 7f51077fe700 1 -- 172.21.15.102:0/777038750 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11364 (secure 0 0 0) 0x7f510805a250 con 0x7f51180ff2c0 2024-03-26T18:51:21.253 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:51:21.253 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","created":"2024-03-26T18:45:25.446329+0000","modified":"2024-03-26T18:51:05.573960+0000","last_up_change":"2024-03-26T18:51:04.568813+0000","last_in_change":"2024-03-26T18:50:45.452094+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T18:49:50.172501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5458d389-eb88-4798-92bc-b442168b37f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6803","nonce":689794310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6805","nonce":689794310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6809","nonce":689794310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6807","nonce":689794310}]},"public_addr":"172.21.15.102:6803/689794310","cluster_addr":"172.21.15.102:6805/689794310","heartbeat_back_addr":"172.21.15.102:6809/689794310","heartbeat_front_addr":"172.21.15.102:6807/689794310","state":["exists","up"]},{"osd":1,"uuid":"87508f24-8b3c-4f6d-add2-00ad59300da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6811","nonce":391196241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6813","nonce":391196241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6817","nonce":391196241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6815","nonce":391196241}]},"public_addr":"172.21.15.102:6811/391196241","cluster_addr":"172.21.15.102:6813/391196241","heartbeat_back_addr":"172.21.15.102:6817/391196241","heartbeat_front_addr":"172.21.15.102:6815/391196241","state":["exists","up"]},{"osd":2,"uuid":"c0179546-10c5-41f2-af0c-be93c2bbf5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6819","nonce":3930922840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6821","nonce":3930922840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6825","nonce":3930922840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6823","nonce":3930922840}]},"public_addr":"172.21.15.102:6819/3930922840","cluster_addr":"172.21.15.102:6821/3930922840","heartbeat_back_addr":"172.21.15.102:6825/3930922840","heartbeat_front_addr":"172.21.15.102:6823/3930922840","state":["exists","up"]},{"osd":3,"uuid":"4d524f7b-8b31-469b-b0c2-b94079febaeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6801","nonce":1428928793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6803","nonce":1428928793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6807","nonce":1428928793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6805","nonce":1428928793}]},"public_addr":"172.21.15.190:6801/1428928793","cluster_addr":"172.21.15.190:6803/1428928793","heartbeat_back_addr":"172.21.15.190:6807/1428928793","heartbeat_front_addr":"172.21.15.190:6805/1428928793","state":["exists","up"]},{"osd":4,"uuid":"865baef8-dd72-4449-b3a4-893801bc2958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6809","nonce":320320306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6811","nonce":320320306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6815","nonce":320320306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6813","nonce":320320306}]},"public_addr":"172.21.15.190:6809/320320306","cluster_addr":"172.21.15.190:6811/320320306","heartbeat_back_addr":"172.21.15.190:6815/320320306","heartbeat_front_addr":"172.21.15.190:6813/320320306","state":["exists","up"]},{"osd":5,"uuid":"5fd67007-d7ae-43b6-8002-258fbfc5a797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6817","nonce":3245135415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6819","nonce":3245135415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6823","nonce":3245135415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6821","nonce":3245135415}]},"public_addr":"172.21.15.190:6817/3245135415","cluster_addr":"172.21.15.190:6819/3245135415","heartbeat_back_addr":"172.21.15.190:6823/3245135415","heartbeat_front_addr":"172.21.15.190:6821/3245135415","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:48:51.274164+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:18.020320+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:46.066766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:10.365857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:36.582495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:51:02.797707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6800/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/3353740154":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1842664791":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1212290737":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6801/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6800/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/1994330698":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/1295801893":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/4276276941":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1380712530":"2024-03-27T18:45:55.925005+0000","172.21.15.102:6801/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:6800/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/2686253586":"2024-03-27T18:45:55.925005+0000","172.21.15.102:6801/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/2752725029":"2024-03-27T18:46:28.035077+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T18:51:21.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.262+0000 7f511cc4b700 1 -- 172.21.15.102:0/777038750 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f510006c680 msgr2=0x7f510006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:21.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.262+0000 7f511cc4b700 1 --2- 172.21.15.102:0/777038750 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f510006c680 0x7f510006eb30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5118069510 tx=0x7f510c00b410 comp rx=0 tx=0).stop 2024-03-26T18:51:21.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.262+0000 7f511cc4b700 1 -- 172.21.15.102:0/777038750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51180ff2c0 msgr2=0x7f5118196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:21.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.262+0000 7f511cc4b700 1 --2- 172.21.15.102:0/777038750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51180ff2c0 0x7f5118196b90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5108006010 tx=0x7f510800bd70 comp rx=0 tx=0).stop 2024-03-26T18:51:21.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.263+0000 7f511cc4b700 1 -- 172.21.15.102:0/777038750 shutdown_connections 2024-03-26T18:51:21.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.263+0000 7f511cc4b700 1 --2- 172.21.15.102:0/777038750 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f510006c680 0x7f510006eb30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:21.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.263+0000 7f511cc4b700 1 --2- 172.21.15.102:0/777038750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51180ff2c0 0x7f5118196b90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:21.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.263+0000 7f511cc4b700 1 --2- 172.21.15.102:0/777038750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51180ffc60 0x7f51181970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:21.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.263+0000 7f511cc4b700 1 -- 172.21.15.102:0/777038750 >> 172.21.15.102:0/777038750 conn(0x7f51180fab80 msgr2=0x7f51180fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:21.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.263+0000 7f511cc4b700 1 -- 172.21.15.102:0/777038750 shutdown_connections 2024-03-26T18:51:21.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:21.263+0000 7f511cc4b700 1 -- 172.21.15.102:0/777038750 wait complete. 2024-03-26T18:51:21.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:21 smithi102 ceph-mon[29002]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:21.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:21 smithi190 ceph-mon[35043]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:21.913 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T18:51:21.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T18:51:22.123 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:22.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:22 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/777038750' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T18:51:22.450 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:22 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/777038750' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T18:51:23.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:23 smithi190 ceph-mon[35043]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:23.537 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:23 smithi102 ceph-mon[29002]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:23.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.750+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/3727673637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b80ffd30 msgr2=0x7fb8b8103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:23.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.750+0000 7fb8bdeb6700 1 --2- 172.21.15.102:0/3727673637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b80ffd30 0x7fb8b8103ce0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009910 tx=0x7fb8ac009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:23.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.750+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/3727673637 shutdown_connections 2024-03-26T18:51:23.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.750+0000 7fb8bdeb6700 1 --2- 172.21.15.102:0/3727673637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b80ffd30 0x7fb8b8103ce0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:23.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.750+0000 7fb8bdeb6700 1 --2- 172.21.15.102:0/3727673637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8b80ff390 0x7fb8b80ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.750+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/3727673637 >> 172.21.15.102:0/3727673637 conn(0x7fb8b8074f50 msgr2=0x7fb8b8075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.750+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/3727673637 shutdown_connections 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.750+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/3727673637 wait complete. 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8bdeb6700 1 Processor -- start 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8bdeb6700 1 -- start start 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8bdeb6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8b80ff390 0x7fb8b8196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8bdeb6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b80ffd30 0x7fb8b8197060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8bdeb6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b81976b0 con 0x7fb8b80ffd30 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8bdeb6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b81977f0 con 0x7fb8b80ff390 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b80ffd30 0x7fb8b8197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b80ffd30 0x7fb8b8197060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50402/0 (socket says 172.21.15.102:50402) 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8b6ffd700 1 -- 172.21.15.102:0/2643532901 learned_addr learned my addr 172.21.15.102:0/2643532901 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:23.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8b77fe700 1 --2- 172.21.15.102:0/2643532901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8b80ff390 0x7fb8b8196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:23.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8b6ffd700 1 -- 172.21.15.102:0/2643532901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8b80ff390 msgr2=0x7fb8b8196b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:23.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8b6ffd700 1 --2- 172.21.15.102:0/2643532901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8b80ff390 0x7fb8b8196b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:23.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.751+0000 7fb8b6ffd700 1 -- 172.21.15.102:0/2643532901 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0095f0 con 0x7fb8b80ffd30 2024-03-26T18:51:23.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.752+0000 7fb8b6ffd700 1 --2- 172.21.15.102:0/2643532901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b80ffd30 0x7fb8b8197060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0092e0 tx=0x7fb8ac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:23.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.752+0000 7fb8b4ff9700 1 -- 172.21.15.102:0/2643532901 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8ac01d070 con 0x7fb8b80ffd30 2024-03-26T18:51:23.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.752+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/2643532901 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b819fb90 con 0x7fb8b80ffd30 2024-03-26T18:51:23.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.752+0000 7fb8b4ff9700 1 -- 172.21.15.102:0/2643532901 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb8ac022470 con 0x7fb8b80ffd30 2024-03-26T18:51:23.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.752+0000 7fb8b4ff9700 1 -- 172.21.15.102:0/2643532901 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8ac00f690 con 0x7fb8b80ffd30 2024-03-26T18:51:23.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.752+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/2643532901 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b819ffa0 con 0x7fb8b80ffd30 2024-03-26T18:51:23.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.753+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/2643532901 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b8107600 con 0x7fb8b80ffd30 2024-03-26T18:51:23.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.753+0000 7fb8b4ff9700 1 -- 172.21.15.102:0/2643532901 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fb8ac0225e0 con 0x7fb8b80ffd30 2024-03-26T18:51:23.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.754+0000 7fb8b4ff9700 1 --2- 172.21.15.102:0/2643532901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb8a006c680 0x7fb8a006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:23.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.754+0000 7fb8b77fe700 1 --2- 172.21.15.102:0/2643532901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb8a006c680 0x7fb8a006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:23.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.754+0000 7fb8b4ff9700 1 -- 172.21.15.102:0/2643532901 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb8ac08cad0 con 0x7fb8b80ffd30 2024-03-26T18:51:23.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.754+0000 7fb8b77fe700 1 --2- 172.21.15.102:0/2643532901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb8a006c680 0x7fb8a006eb30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8005ea0 tx=0x7fb8a8005e30 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:23.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.758+0000 7fb8b4ff9700 1 -- 172.21.15.102:0/2643532901 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb8ac0575a0 con 0x7fb8b80ffd30 2024-03-26T18:51:23.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.950+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/2643532901 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8b804ea50 con 0x7fb8b80ffd30 2024-03-26T18:51:23.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.951+0000 7fb8b4ff9700 1 -- 172.21.15.102:0/2643532901 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11364 (secure 0 0 0) 0x7fb8ac027090 con 0x7fb8b80ffd30 2024-03-26T18:51:23.943 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:51:23.944 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","created":"2024-03-26T18:45:25.446329+0000","modified":"2024-03-26T18:51:05.573960+0000","last_up_change":"2024-03-26T18:51:04.568813+0000","last_in_change":"2024-03-26T18:50:45.452094+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T18:49:50.172501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5458d389-eb88-4798-92bc-b442168b37f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6803","nonce":689794310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6805","nonce":689794310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6809","nonce":689794310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6807","nonce":689794310}]},"public_addr":"172.21.15.102:6803/689794310","cluster_addr":"172.21.15.102:6805/689794310","heartbeat_back_addr":"172.21.15.102:6809/689794310","heartbeat_front_addr":"172.21.15.102:6807/689794310","state":["exists","up"]},{"osd":1,"uuid":"87508f24-8b3c-4f6d-add2-00ad59300da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6811","nonce":391196241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6813","nonce":391196241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6817","nonce":391196241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6815","nonce":391196241}]},"public_addr":"172.21.15.102:6811/391196241","cluster_addr":"172.21.15.102:6813/391196241","heartbeat_back_addr":"172.21.15.102:6817/391196241","heartbeat_front_addr":"172.21.15.102:6815/391196241","state":["exists","up"]},{"osd":2,"uuid":"c0179546-10c5-41f2-af0c-be93c2bbf5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6819","nonce":3930922840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6821","nonce":3930922840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6825","nonce":3930922840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6823","nonce":3930922840}]},"public_addr":"172.21.15.102:6819/3930922840","cluster_addr":"172.21.15.102:6821/3930922840","heartbeat_back_addr":"172.21.15.102:6825/3930922840","heartbeat_front_addr":"172.21.15.102:6823/3930922840","state":["exists","up"]},{"osd":3,"uuid":"4d524f7b-8b31-469b-b0c2-b94079febaeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6801","nonce":1428928793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6803","nonce":1428928793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6807","nonce":1428928793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6805","nonce":1428928793}]},"public_addr":"172.21.15.190:6801/1428928793","cluster_addr":"172.21.15.190:6803/1428928793","heartbeat_back_addr":"172.21.15.190:6807/1428928793","heartbeat_front_addr":"172.21.15.190:6805/1428928793","state":["exists","up"]},{"osd":4,"uuid":"865baef8-dd72-4449-b3a4-893801bc2958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6809","nonce":320320306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6811","nonce":320320306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6815","nonce":320320306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6813","nonce":320320306}]},"public_addr":"172.21.15.190:6809/320320306","cluster_addr":"172.21.15.190:6811/320320306","heartbeat_back_addr":"172.21.15.190:6815/320320306","heartbeat_front_addr":"172.21.15.190:6813/320320306","state":["exists","up"]},{"osd":5,"uuid":"5fd67007-d7ae-43b6-8002-258fbfc5a797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6817","nonce":3245135415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6819","nonce":3245135415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6823","nonce":3245135415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6821","nonce":3245135415}]},"public_addr":"172.21.15.190:6817/3245135415","cluster_addr":"172.21.15.190:6819/3245135415","heartbeat_back_addr":"172.21.15.190:6823/3245135415","heartbeat_front_addr":"172.21.15.190:6821/3245135415","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:48:51.274164+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:18.020320+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:46.066766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:10.365857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:36.582495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:51:02.797707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6800/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/3353740154":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1842664791":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1212290737":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6801/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6800/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/1994330698":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/1295801893":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/4276276941":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1380712530":"2024-03-27T18:45:55.925005+0000","172.21.15.102:6801/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:6800/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/2686253586":"2024-03-27T18:45:55.925005+0000","172.21.15.102:6801/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/2752725029":"2024-03-27T18:46:28.035077+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T18:51:23.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.953+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/2643532901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb8a006c680 msgr2=0x7fb8a006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:23.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.953+0000 7fb8bdeb6700 1 --2- 172.21.15.102:0/2643532901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb8a006c680 0x7fb8a006eb30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8005ea0 tx=0x7fb8a8005e30 comp rx=0 tx=0).stop 2024-03-26T18:51:23.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.953+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/2643532901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b80ffd30 msgr2=0x7fb8b8197060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:23.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.953+0000 7fb8bdeb6700 1 --2- 172.21.15.102:0/2643532901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b80ffd30 0x7fb8b8197060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0092e0 tx=0x7fb8ac00be20 comp rx=0 tx=0).stop 2024-03-26T18:51:23.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.953+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/2643532901 shutdown_connections 2024-03-26T18:51:23.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.953+0000 7fb8bdeb6700 1 --2- 172.21.15.102:0/2643532901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb8a006c680 0x7fb8a006eb30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:23.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.953+0000 7fb8bdeb6700 1 --2- 172.21.15.102:0/2643532901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8b80ff390 0x7fb8b8196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:23.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.953+0000 7fb8bdeb6700 1 --2- 172.21.15.102:0/2643532901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b80ffd30 0x7fb8b8197060 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:23.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.953+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/2643532901 >> 172.21.15.102:0/2643532901 conn(0x7fb8b8074f50 msgr2=0x7fb8b80fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:23.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.953+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/2643532901 shutdown_connections 2024-03-26T18:51:23.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:23.953+0000 7fb8bdeb6700 1 -- 172.21.15.102:0/2643532901 wait complete. 2024-03-26T18:51:24.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:24 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2643532901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T18:51:24.571 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:24 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2643532901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T18:51:24.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T18:51:24.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T18:51:24.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T18:51:24.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T18:51:24.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T18:51:24.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T18:51:24.897 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:24.898 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:24.950 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:25.300 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:25 smithi102 ceph-mon[29002]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:25 smithi190 ceph-mon[35043]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:26.806 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:26.806 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:26.807 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:27.515 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:27 smithi190 ceph-mon[35043]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:27.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:27 smithi102 ceph-mon[29002]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:28.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.529+0000 7f862b6f9700 1 -- 172.21.15.102:0/698995185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8624073a00 msgr2=0x7f8624110c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:28.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.529+0000 7f862b6f9700 1 --2- 172.21.15.102:0/698995185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8624073a00 0x7f8624110c40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8614009910 tx=0x7f8614009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 -- 172.21.15.102:0/698995185 shutdown_connections 2024-03-26T18:51:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 --2- 172.21.15.102:0/698995185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8624073a00 0x7f8624110c40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 --2- 172.21.15.102:0/698995185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86240730f0 0x7f86240734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 -- 172.21.15.102:0/698995185 >> 172.21.15.102:0/698995185 conn(0x7f86240fbc90 msgr2=0x7f86240fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 -- 172.21.15.102:0/698995185 shutdown_connections 2024-03-26T18:51:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 -- 172.21.15.102:0/698995185 wait complete. 2024-03-26T18:51:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 Processor -- start 2024-03-26T18:51:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 -- start start 2024-03-26T18:51:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86240730f0 0x7f862419c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8624073a00 0x7f862419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862419cef0 con 0x7f8624073a00 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.530+0000 7f862b6f9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86241a0c80 con 0x7f86240730f0 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f8628c94700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8624073a00 0x7f862419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f8628c94700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8624073a00 0x7f862419c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50422/0 (socket says 172.21.15.102:50422) 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f8628c94700 1 -- 172.21.15.102:0/374207533 learned_addr learned my addr 172.21.15.102:0/374207533 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f8629495700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86240730f0 0x7f862419c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f8628c94700 1 -- 172.21.15.102:0/374207533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86240730f0 msgr2=0x7f862419c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f8628c94700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86240730f0 0x7f862419c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f8628c94700 1 -- 172.21.15.102:0/374207533 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86140095f0 con 0x7f8624073a00 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f8628c94700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8624073a00 0x7f862419c810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8614006010 tx=0x7f861400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f861a7fc700 1 -- 172.21.15.102:0/374207533 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f861401d070 con 0x7f8624073a00 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f862b6f9700 1 -- 172.21.15.102:0/374207533 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86241a0f60 con 0x7f8624073a00 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f861a7fc700 1 -- 172.21.15.102:0/374207533 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f861400f460 con 0x7f8624073a00 2024-03-26T18:51:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f861a7fc700 1 -- 172.21.15.102:0/374207533 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8614017630 con 0x7f8624073a00 2024-03-26T18:51:28.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.531+0000 7f862b6f9700 1 -- 172.21.15.102:0/374207533 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86241a14b0 con 0x7f8624073a00 2024-03-26T18:51:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.532+0000 7f862b6f9700 1 -- 172.21.15.102:0/374207533 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8624113b60 con 0x7f8624073a00 2024-03-26T18:51:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.533+0000 7f861a7fc700 1 -- 172.21.15.102:0/374207533 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f861400f5d0 con 0x7f8624073a00 2024-03-26T18:51:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.533+0000 7f861a7fc700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f861006c680 0x7f861006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.533+0000 7f861a7fc700 1 -- 172.21.15.102:0/374207533 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f861408c920 con 0x7f8624073a00 2024-03-26T18:51:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.533+0000 7f8629495700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f861006c680 0x7f861006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.533+0000 7f861a7fc700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415] conn(0x7f8610072350 0x7f8610074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.533+0000 7f861a7fc700 1 -- 172.21.15.102:0/374207533 --> [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8610074e10 con 0x7f8610072350 2024-03-26T18:51:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.533+0000 7f861a7fc700 1 -- 172.21.15.102:0/374207533 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86140912a0 con 0x7f8624073a00 2024-03-26T18:51:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.533+0000 7f8629495700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f861006c680 0x7f861006eb30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8620005950 tx=0x7f862000b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.533+0000 7f8629c96700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415] conn(0x7f8610072350 0x7f8610074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.534+0000 7f8629c96700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415] conn(0x7f8610072350 0x7f8610074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:28.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.535+0000 7f861a7fc700 1 -- 172.21.15.102:0/374207533 <== osd.5 v2:172.21.15.190:6816/3245135415 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8610074e10 con 0x7f8610072350 2024-03-26T18:51:28.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.558+0000 7f862b6f9700 1 -- 172.21.15.102:0/374207533 --> [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f862404ea50 con 0x7f8610072350 2024-03-26T18:51:28.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f861a7fc700 1 -- 172.21.15.102:0/374207533 <== osd.5 v2:172.21.15.190:6816/3245135415 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f862404ea50 con 0x7f8610072350 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 -- 172.21.15.102:0/374207533 >> [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415] conn(0x7f8610072350 msgr2=0x7f8610074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415] conn(0x7f8610072350 0x7f8610074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 -- 172.21.15.102:0/374207533 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f861006c680 msgr2=0x7f861006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f861006c680 0x7f861006eb30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8620005950 tx=0x7f862000b410 comp rx=0 tx=0).stop 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 -- 172.21.15.102:0/374207533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8624073a00 msgr2=0x7f862419c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8624073a00 0x7f862419c810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8614006010 tx=0x7f861400bd70 comp rx=0 tx=0).stop 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 -- 172.21.15.102:0/374207533 shutdown_connections 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f861006c680 0x7f861006eb30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.190:6816/3245135415,v1:172.21.15.190:6817/3245135415] conn(0x7f8610072350 0x7f8610074760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86240730f0 0x7f862419c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 --2- 172.21.15.102:0/374207533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8624073a00 0x7f862419c810 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 -- 172.21.15.102:0/374207533 >> 172.21.15.102:0/374207533 conn(0x7f86240fbc90 msgr2=0x7f8624102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:28.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 -- 172.21.15.102:0/374207533 shutdown_connections 2024-03-26T18:51:28.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.559+0000 7f862b6f9700 1 -- 172.21.15.102:0/374207533 wait complete. 2024-03-26T18:51:28.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.958+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1368264280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8f01003d0 msgr2=0x7fd8f01007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:28.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.958+0000 7fd8f77f4700 1 --2- 172.21.15.102:0/1368264280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8f01003d0 0x7fd8f01007a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009910 tx=0x7fd8ec009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:28.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.958+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1368264280 shutdown_connections 2024-03-26T18:51:28.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.958+0000 7fd8f77f4700 1 --2- 172.21.15.102:0/1368264280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8f0100d70 0x7fd8f0104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.958+0000 7fd8f77f4700 1 --2- 172.21.15.102:0/1368264280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8f01003d0 0x7fd8f01007a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.958+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1368264280 >> 172.21.15.102:0/1368264280 conn(0x7fd8f00fbc90 msgr2=0x7fd8f00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.958+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1368264280 shutdown_connections 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.958+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1368264280 wait complete. 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.959+0000 7fd8f77f4700 1 Processor -- start 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.959+0000 7fd8f77f4700 1 -- start start 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.959+0000 7fd8f77f4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8f01003d0 0x7fd8f0195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.959+0000 7fd8f77f4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8f0100d70 0x7fd8f01962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.959+0000 7fd8f77f4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f01968f0 con 0x7fd8f01003d0 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.959+0000 7fd8f77f4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f0196a30 con 0x7fd8f0100d70 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.959+0000 7fd8f5590700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8f01003d0 0x7fd8f0195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.959+0000 7fd8f5590700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8f01003d0 0x7fd8f0195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43012/0 (socket says 172.21.15.102:43012) 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.959+0000 7fd8f5590700 1 -- 172.21.15.102:0/1248336405 learned_addr learned my addr 172.21.15.102:0/1248336405 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.960+0000 7fd8f4d8f700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8f0100d70 0x7fd8f01962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.960+0000 7fd8f5590700 1 -- 172.21.15.102:0/1248336405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8f0100d70 msgr2=0x7fd8f01962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:28.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.960+0000 7fd8f5590700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8f0100d70 0x7fd8f01962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.960+0000 7fd8f5590700 1 -- 172.21.15.102:0/1248336405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec0095f0 con 0x7fd8f01003d0 2024-03-26T18:51:28.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.960+0000 7fd8f5590700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8f01003d0 0x7fd8f0195d60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec006010 tx=0x7fd8ec00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:28.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.960+0000 7fd8e67fc700 1 -- 172.21.15.102:0/1248336405 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8ec01d070 con 0x7fd8f01003d0 2024-03-26T18:51:28.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.960+0000 7fd8e67fc700 1 -- 172.21.15.102:0/1248336405 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd8ec004fa0 con 0x7fd8f01003d0 2024-03-26T18:51:28.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.960+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1248336405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f019a820 con 0x7fd8f01003d0 2024-03-26T18:51:28.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.960+0000 7fd8e67fc700 1 -- 172.21.15.102:0/1248336405 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8ec00f690 con 0x7fd8f01003d0 2024-03-26T18:51:28.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.960+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1248336405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f019ad10 con 0x7fd8f01003d0 2024-03-26T18:51:28.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.961+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1248336405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd8f010d5b0 con 0x7fd8f01003d0 2024-03-26T18:51:28.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.961+0000 7fd8e67fc700 1 -- 172.21.15.102:0/1248336405 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fd8ec022470 con 0x7fd8f01003d0 2024-03-26T18:51:28.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.962+0000 7fd8e67fc700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd8dc06c680 0x7fd8dc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.962+0000 7fd8f4d8f700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd8dc06c680 0x7fd8dc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.962+0000 7fd8e67fc700 1 -- 172.21.15.102:0/1248336405 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd8ec08cc30 con 0x7fd8f01003d0 2024-03-26T18:51:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.962+0000 7fd8e67fc700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840] conn(0x7fd8dc072350 0x7fd8dc074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.962+0000 7fd8e67fc700 1 -- 172.21.15.102:0/1248336405 --> [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8dc074e10 con 0x7fd8dc072350 2024-03-26T18:51:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.962+0000 7fd8e67fc700 1 -- 172.21.15.102:0/1248336405 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8ec092310 con 0x7fd8f01003d0 2024-03-26T18:51:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.962+0000 7fd8f5d91700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840] conn(0x7fd8dc072350 0x7fd8dc074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.962+0000 7fd8f4d8f700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd8dc06c680 0x7fd8dc06eb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0197380 tx=0x7fd8e000a380 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.963+0000 7fd8f5d91700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840] conn(0x7fd8dc072350 0x7fd8dc074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.963+0000 7fd8e67fc700 1 -- 172.21.15.102:0/1248336405 <== osd.2 v2:172.21.15.102:6818/3930922840 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd8dc074e10 con 0x7fd8dc072350 2024-03-26T18:51:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1248336405 --> [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd8f004ea50 con 0x7fd8dc072350 2024-03-26T18:51:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8e67fc700 1 -- 172.21.15.102:0/1248336405 <== osd.2 v2:172.21.15.102:6818/3930922840 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd8f004ea50 con 0x7fd8dc072350 2024-03-26T18:51:28.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840] conn(0x7fd8dc072350 msgr2=0x7fd8dc074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:28.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840] conn(0x7fd8dc072350 0x7fd8dc074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd8dc06c680 msgr2=0x7fd8dc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:28.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd8dc06c680 0x7fd8dc06eb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0197380 tx=0x7fd8e000a380 comp rx=0 tx=0).stop 2024-03-26T18:51:28.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8f01003d0 msgr2=0x7fd8f0195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:28.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8f01003d0 0x7fd8f0195d60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec006010 tx=0x7fd8ec00bd70 comp rx=0 tx=0).stop 2024-03-26T18:51:28.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1248336405 shutdown_connections 2024-03-26T18:51:28.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6818/3930922840,v1:172.21.15.102:6819/3930922840] conn(0x7fd8dc072350 0x7fd8dc074760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd8dc06c680 0x7fd8dc06eb30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8f01003d0 0x7fd8f0195d60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 --2- 172.21.15.102:0/1248336405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8f0100d70 0x7fd8f01962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:28.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1248336405 >> 172.21.15.102:0/1248336405 conn(0x7fd8f00fbc90 msgr2=0x7fd8f00fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:28.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.979+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1248336405 shutdown_connections 2024-03-26T18:51:28.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:28.980+0000 7fd8f77f4700 1 -- 172.21.15.102:0/1248336405 wait complete. 2024-03-26T18:51:29.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.176+0000 7faf2836e700 1 -- 172.21.15.102:0/2002138188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf200ff2c0 msgr2=0x7faf200ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:29.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.176+0000 7faf2836e700 1 --2- 172.21.15.102:0/2002138188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf200ff2c0 0x7faf200ff690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faf1c009910 tx=0x7faf1c009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:29.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.176+0000 7faf2836e700 1 -- 172.21.15.102:0/2002138188 shutdown_connections 2024-03-26T18:51:29.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.176+0000 7faf2836e700 1 --2- 172.21.15.102:0/2002138188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf200ffc60 0x7faf20103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:29.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.176+0000 7faf2836e700 1 --2- 172.21.15.102:0/2002138188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf200ff2c0 0x7faf200ff690 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.176+0000 7faf2836e700 1 -- 172.21.15.102:0/2002138188 >> 172.21.15.102:0/2002138188 conn(0x7faf200fab80 msgr2=0x7faf200fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.176+0000 7faf2836e700 1 -- 172.21.15.102:0/2002138188 shutdown_connections 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.176+0000 7faf2836e700 1 -- 172.21.15.102:0/2002138188 wait complete. 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.177+0000 7faf2836e700 1 Processor -- start 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.177+0000 7faf2836e700 1 -- start start 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.177+0000 7faf2836e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf200ff2c0 0x7faf20195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.177+0000 7faf2836e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf200ffc60 0x7faf20196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.177+0000 7faf2836e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf20196a00 con 0x7faf200ffc60 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.177+0000 7faf2836e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2019a790 con 0x7faf200ff2c0 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.177+0000 7faf25909700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf200ffc60 0x7faf20196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.177+0000 7faf25909700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf200ffc60 0x7faf20196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43026/0 (socket says 172.21.15.102:43026) 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.177+0000 7faf25909700 1 -- 172.21.15.102:0/2242500136 learned_addr learned my addr 172.21.15.102:0/2242500136 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.177+0000 7faf2610a700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf200ff2c0 0x7faf20195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.178+0000 7faf25909700 1 -- 172.21.15.102:0/2242500136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf200ff2c0 msgr2=0x7faf20195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:29.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.178+0000 7faf25909700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf200ff2c0 0x7faf20195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:29.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.178+0000 7faf25909700 1 -- 172.21.15.102:0/2242500136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1c0095f0 con 0x7faf200ffc60 2024-03-26T18:51:29.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.178+0000 7faf25909700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf200ffc60 0x7faf20196320 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faf1000d900 tx=0x7faf1000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:29.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.178+0000 7faf177fe700 1 -- 172.21.15.102:0/2242500136 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf10009900 con 0x7faf200ffc60 2024-03-26T18:51:29.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.178+0000 7faf177fe700 1 -- 172.21.15.102:0/2242500136 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faf10010460 con 0x7faf200ffc60 2024-03-26T18:51:29.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.178+0000 7faf2836e700 1 -- 172.21.15.102:0/2242500136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf2019aa70 con 0x7faf200ffc60 2024-03-26T18:51:29.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.178+0000 7faf177fe700 1 -- 172.21.15.102:0/2242500136 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf1000f5f0 con 0x7faf200ffc60 2024-03-26T18:51:29.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.178+0000 7faf2836e700 1 -- 172.21.15.102:0/2242500136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2019afc0 con 0x7faf200ffc60 2024-03-26T18:51:29.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.179+0000 7faf2836e700 1 -- 172.21.15.102:0/2242500136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faf2010d560 con 0x7faf200ffc60 2024-03-26T18:51:29.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.179+0000 7faf177fe700 1 -- 172.21.15.102:0/2242500136 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7faf100105d0 con 0x7faf200ffc60 2024-03-26T18:51:29.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.180+0000 7faf177fe700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7faf0c06c680 0x7faf0c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:29.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.180+0000 7faf177fe700 1 -- 172.21.15.102:0/2242500136 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7faf1008b0d0 con 0x7faf200ffc60 2024-03-26T18:51:29.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.180+0000 7faf2610a700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7faf0c06c680 0x7faf0c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:29.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.180+0000 7faf177fe700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310] conn(0x7faf0c072350 0x7faf0c074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:29.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.180+0000 7faf177fe700 1 -- 172.21.15.102:0/2242500136 --> [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf0c074e10 con 0x7faf0c072350 2024-03-26T18:51:29.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.180+0000 7faf177fe700 1 -- 172.21.15.102:0/2242500136 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf1008d7a0 con 0x7faf200ffc60 2024-03-26T18:51:29.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.180+0000 7faf2690b700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310] conn(0x7faf0c072350 0x7faf0c074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:29.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.180+0000 7faf2610a700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7faf0c06c680 0x7faf0c06eb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf1c006010 tx=0x7faf1c01a040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:29.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.180+0000 7faf2690b700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310] conn(0x7faf0c072350 0x7faf0c074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:29.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.181+0000 7faf177fe700 1 -- 172.21.15.102:0/2242500136 <== osd.0 v2:172.21.15.102:6802/689794310 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7faf0c074e10 con 0x7faf0c072350 2024-03-26T18:51:29.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.197+0000 7faf2836e700 1 -- 172.21.15.102:0/2242500136 --> [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faf2004ea50 con 0x7faf0c072350 2024-03-26T18:51:29.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.197+0000 7faf177fe700 1 -- 172.21.15.102:0/2242500136 <== osd.0 v2:172.21.15.102:6802/689794310 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7faf2004ea50 con 0x7faf0c072350 2024-03-26T18:51:29.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.197+0000 7faf2836e700 1 -- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310] conn(0x7faf0c072350 msgr2=0x7faf0c074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:29.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.197+0000 7faf2836e700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310] conn(0x7faf0c072350 0x7faf0c074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:29.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.197+0000 7faf2836e700 1 -- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7faf0c06c680 msgr2=0x7faf0c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:29.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.197+0000 7faf2836e700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7faf0c06c680 0x7faf0c06eb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf1c006010 tx=0x7faf1c01a040 comp rx=0 tx=0).stop 2024-03-26T18:51:29.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.197+0000 7faf2836e700 1 -- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf200ffc60 msgr2=0x7faf20196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:29.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.197+0000 7faf2836e700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf200ffc60 0x7faf20196320 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faf1000d900 tx=0x7faf1000dc10 comp rx=0 tx=0).stop 2024-03-26T18:51:29.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.198+0000 7faf2836e700 1 -- 172.21.15.102:0/2242500136 shutdown_connections 2024-03-26T18:51:29.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.198+0000 7faf2836e700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6802/689794310,v1:172.21.15.102:6803/689794310] conn(0x7faf0c072350 0x7faf0c074760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:29.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.198+0000 7faf2836e700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7faf0c06c680 0x7faf0c06eb30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:29.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.198+0000 7faf2836e700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf200ff2c0 0x7faf20195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:29.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.198+0000 7faf2836e700 1 --2- 172.21.15.102:0/2242500136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf200ffc60 0x7faf20196320 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:29.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.198+0000 7faf2836e700 1 -- 172.21.15.102:0/2242500136 >> 172.21.15.102:0/2242500136 conn(0x7faf200fab80 msgr2=0x7faf200fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:29.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.198+0000 7faf2836e700 1 -- 172.21.15.102:0/2242500136 shutdown_connections 2024-03-26T18:51:29.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:29.198+0000 7faf2836e700 1 -- 172.21.15.102:0/2242500136 wait complete. 2024-03-26T18:51:29.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:29 smithi190 ceph-mon[35043]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:29.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:29 smithi102 ceph-mon[29002]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:30.004 INFO:teuthology.orchestra.run.smithi102.stdout:38654705699 2024-03-26T18:51:30.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T18:51:30.689 INFO:teuthology.orchestra.run.smithi102.stdout:171798691847 2024-03-26T18:51:30.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T18:51:31.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:31 smithi102 ceph-mon[29002]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:31.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:31 smithi190 ceph-mon[35043]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:32.569 INFO:teuthology.orchestra.run.smithi102.stdout:81604378648 2024-03-26T18:51:32.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T18:51:33.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:33 smithi102 ceph-mon[29002]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:33.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:33 smithi190 ceph-mon[35043]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:34.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:51:34.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:51:35.339 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:35 smithi102 ceph-mon[29002]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.463+0000 7fc46283b700 1 -- 172.21.15.102:0/2923416057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc45c0ff2c0 msgr2=0x7fc45c0ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.463+0000 7fc46283b700 1 --2- 172.21.15.102:0/2923416057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc45c0ff2c0 0x7fc45c0ff690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009910 tx=0x7fc44c009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.463+0000 7fc46283b700 1 -- 172.21.15.102:0/2923416057 shutdown_connections 2024-03-26T18:51:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.463+0000 7fc46283b700 1 --2- 172.21.15.102:0/2923416057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc45c0ffc60 0x7fc45c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.463+0000 7fc46283b700 1 --2- 172.21.15.102:0/2923416057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc45c0ff2c0 0x7fc45c0ff690 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.463+0000 7fc46283b700 1 -- 172.21.15.102:0/2923416057 >> 172.21.15.102:0/2923416057 conn(0x7fc45c0fab80 msgr2=0x7fc45c0fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.463+0000 7fc46283b700 1 -- 172.21.15.102:0/2923416057 shutdown_connections 2024-03-26T18:51:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.463+0000 7fc46283b700 1 -- 172.21.15.102:0/2923416057 wait complete. 2024-03-26T18:51:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.464+0000 7fc46283b700 1 Processor -- start 2024-03-26T18:51:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.464+0000 7fc46283b700 1 -- start start 2024-03-26T18:51:35.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.464+0000 7fc46283b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc45c0ff2c0 0x7fc45c196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:35.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.464+0000 7fc46283b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc45c0ffc60 0x7fc45c197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:35.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.464+0000 7fc46283b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c197750 con 0x7fc45c0ff2c0 2024-03-26T18:51:35.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.464+0000 7fc46283b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c197890 con 0x7fc45c0ffc60 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.464+0000 7fc45bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc45c0ff2c0 0x7fc45c196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.464+0000 7fc45bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc45c0ff2c0 0x7fc45c196b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43046/0 (socket says 172.21.15.102:43046) 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.464+0000 7fc45bfff700 1 -- 172.21.15.102:0/3508616813 learned_addr learned my addr 172.21.15.102:0/3508616813 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.464+0000 7fc45b7fe700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc45c0ffc60 0x7fc45c197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.465+0000 7fc45bfff700 1 -- 172.21.15.102:0/3508616813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc45c0ffc60 msgr2=0x7fc45c197070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.465+0000 7fc45bfff700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc45c0ffc60 0x7fc45c197070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.465+0000 7fc45bfff700 1 -- 172.21.15.102:0/3508616813 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c0095f0 con 0x7fc45c0ff2c0 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.465+0000 7fc45bfff700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc45c0ff2c0 0x7fc45c196b30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00fa00 tx=0x7fc44c00fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.465+0000 7fc4597fa700 1 -- 172.21.15.102:0/3508616813 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc44c01d070 con 0x7fc45c0ff2c0 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.465+0000 7fc46283b700 1 -- 172.21.15.102:0/3508616813 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c194920 con 0x7fc45c0ff2c0 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.465+0000 7fc4597fa700 1 -- 172.21.15.102:0/3508616813 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc44c005640 con 0x7fc45c0ff2c0 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.465+0000 7fc4597fa700 1 -- 172.21.15.102:0/3508616813 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc44c0226a0 con 0x7fc45c0ff2c0 2024-03-26T18:51:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.465+0000 7fc46283b700 1 -- 172.21.15.102:0/3508616813 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c194de0 con 0x7fc45c0ff2c0 2024-03-26T18:51:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.466+0000 7fc46283b700 1 -- 172.21.15.102:0/3508616813 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc45c10c4a0 con 0x7fc45c0ff2c0 2024-03-26T18:51:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.467+0000 7fc4597fa700 1 -- 172.21.15.102:0/3508616813 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fc44c003680 con 0x7fc45c0ff2c0 2024-03-26T18:51:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.467+0000 7fc4597fa700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc44406c680 0x7fc44406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.467+0000 7fc45b7fe700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc44406c680 0x7fc44406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.467+0000 7fc4597fa700 1 -- 172.21.15.102:0/3508616813 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc44c08cc00 con 0x7fc45c0ff2c0 2024-03-26T18:51:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.467+0000 7fc4597fa700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241] conn(0x7fc444072350 0x7fc444074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.467+0000 7fc4597fa700 1 -- 172.21.15.102:0/3508616813 --> [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc444074e10 con 0x7fc444072350 2024-03-26T18:51:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.467+0000 7fc4597fa700 1 -- 172.21.15.102:0/3508616813 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc44c08d060 con 0x7fc45c0ff2c0 2024-03-26T18:51:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.468+0000 7fc460dd8700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241] conn(0x7fc444072350 0x7fc444074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.468+0000 7fc45b7fe700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc44406c680 0x7fc44406eb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc45c069d40 tx=0x7fc450009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.468+0000 7fc460dd8700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241] conn(0x7fc444072350 0x7fc444074760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:35.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.469+0000 7fc4597fa700 1 -- 172.21.15.102:0/3508616813 <== osd.1 v2:172.21.15.102:6810/391196241 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc444074e10 con 0x7fc444072350 2024-03-26T18:51:35.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.484+0000 7fc46283b700 1 -- 172.21.15.102:0/3508616813 --> [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc45c04ea50 con 0x7fc444072350 2024-03-26T18:51:35.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc4597fa700 1 -- 172.21.15.102:0/3508616813 <== osd.1 v2:172.21.15.102:6810/391196241 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc45c04ea50 con 0x7fc444072350 2024-03-26T18:51:35.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 -- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241] conn(0x7fc444072350 msgr2=0x7fc444074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241] conn(0x7fc444072350 0x7fc444074760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 -- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc44406c680 msgr2=0x7fc44406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc44406c680 0x7fc44406eb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc45c069d40 tx=0x7fc450009260 comp rx=0 tx=0).stop 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 -- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc45c0ff2c0 msgr2=0x7fc45c196b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc45c0ff2c0 0x7fc45c196b30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00fa00 tx=0x7fc44c00fae0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 -- 172.21.15.102:0/3508616813 shutdown_connections 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6810/391196241,v1:172.21.15.102:6811/391196241] conn(0x7fc444072350 0x7fc444074760 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc44406c680 0x7fc44406eb30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc45c0ff2c0 0x7fc45c196b30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 --2- 172.21.15.102:0/3508616813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc45c0ffc60 0x7fc45c197070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 -- 172.21.15.102:0/3508616813 >> 172.21.15.102:0/3508616813 conn(0x7fc45c0fab80 msgr2=0x7fc45c0fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 -- 172.21.15.102:0/3508616813 shutdown_connections 2024-03-26T18:51:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.485+0000 7fc46283b700 1 -- 172.21.15.102:0/3508616813 wait complete. 2024-03-26T18:51:35.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:35 smithi190 ceph-mon[35043]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.654+0000 7fb455271700 1 -- 172.21.15.102:0/2724203148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4500ffc80 msgr2=0x7fb450103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.654+0000 7fb455271700 1 --2- 172.21.15.102:0/2724203148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4500ffc80 0x7fb450103c50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb444009910 tx=0x7fb444009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.654+0000 7fb455271700 1 -- 172.21.15.102:0/2724203148 shutdown_connections 2024-03-26T18:51:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.654+0000 7fb455271700 1 --2- 172.21.15.102:0/2724203148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4500ffc80 0x7fb450103c50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.654+0000 7fb455271700 1 --2- 172.21.15.102:0/2724203148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4500ff2e0 0x7fb4500ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.654+0000 7fb455271700 1 -- 172.21.15.102:0/2724203148 >> 172.21.15.102:0/2724203148 conn(0x7fb4500fab80 msgr2=0x7fb4500fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.654+0000 7fb455271700 1 -- 172.21.15.102:0/2724203148 shutdown_connections 2024-03-26T18:51:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.654+0000 7fb455271700 1 -- 172.21.15.102:0/2724203148 wait complete. 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.655+0000 7fb455271700 1 Processor -- start 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.655+0000 7fb455271700 1 -- start start 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.655+0000 7fb455271700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4500ff2e0 0x7fb450100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.655+0000 7fb455271700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4500ffc80 0x7fb450101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.655+0000 7fb455271700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb450101770 con 0x7fb4500ff2e0 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.655+0000 7fb455271700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4501018b0 con 0x7fb4500ffc80 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.655+0000 7fb44effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4500ff2e0 0x7fb450100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.655+0000 7fb44effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4500ff2e0 0x7fb450100cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43064/0 (socket says 172.21.15.102:43064) 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.655+0000 7fb44effd700 1 -- 172.21.15.102:0/171855674 learned_addr learned my addr 172.21.15.102:0/171855674 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.655+0000 7fb44e7fc700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4500ffc80 0x7fb450101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.656+0000 7fb44effd700 1 -- 172.21.15.102:0/171855674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4500ffc80 msgr2=0x7fb450101230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.656+0000 7fb44effd700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4500ffc80 0x7fb450101230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.656+0000 7fb44effd700 1 -- 172.21.15.102:0/171855674 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4440095f0 con 0x7fb4500ff2e0 2024-03-26T18:51:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.656+0000 7fb44effd700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4500ff2e0 0x7fb450100cf0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb44000d8d0 tx=0x7fb44000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.656+0000 7fb437fff700 1 -- 172.21.15.102:0/171855674 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4400098a0 con 0x7fb4500ff2e0 2024-03-26T18:51:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.656+0000 7fb437fff700 1 -- 172.21.15.102:0/171855674 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb440010460 con 0x7fb4500ff2e0 2024-03-26T18:51:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.656+0000 7fb455271700 1 -- 172.21.15.102:0/171855674 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4501a1ee0 con 0x7fb4500ff2e0 2024-03-26T18:51:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.656+0000 7fb437fff700 1 -- 172.21.15.102:0/171855674 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb44000f5f0 con 0x7fb4500ff2e0 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.656+0000 7fb455271700 1 -- 172.21.15.102:0/171855674 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4501a22f0 con 0x7fb4500ff2e0 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.657+0000 7fb455271700 1 -- 172.21.15.102:0/171855674 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb45010c490 con 0x7fb4500ff2e0 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.658+0000 7fb437fff700 1 -- 172.21.15.102:0/171855674 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fb440009a00 con 0x7fb4500ff2e0 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.658+0000 7fb437fff700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb43806c680 0x7fb43806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.658+0000 7fb44e7fc700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb43806c680 0x7fb43806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.658+0000 7fb437fff700 1 -- 172.21.15.102:0/171855674 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb44008b970 con 0x7fb4500ff2e0 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.658+0000 7fb437fff700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306] conn(0x7fb438072350 0x7fb438074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.658+0000 7fb437fff700 1 -- 172.21.15.102:0/171855674 --> [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb438074e10 con 0x7fb438072350 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.658+0000 7fb437fff700 1 -- 172.21.15.102:0/171855674 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb44008bdd0 con 0x7fb4500ff2e0 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.659+0000 7fb44f7fe700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306] conn(0x7fb438072350 0x7fb438074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.659+0000 7fb44e7fc700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb43806c680 0x7fb43806eb30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb4501024e0 tx=0x7fb44401a040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.659+0000 7fb44f7fe700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306] conn(0x7fb438072350 0x7fb438074760 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.660+0000 7fb437fff700 1 -- 172.21.15.102:0/171855674 <== osd.4 v2:172.21.15.190:6808/320320306 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb438074e10 con 0x7fb438072350 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.676+0000 7fb455271700 1 -- 172.21.15.102:0/171855674 --> [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb45004ea50 con 0x7fb438072350 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.676+0000 7fb437fff700 1 -- 172.21.15.102:0/171855674 <== osd.4 v2:172.21.15.190:6808/320320306 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb45004ea50 con 0x7fb438072350 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 -- 172.21.15.102:0/171855674 >> [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306] conn(0x7fb438072350 msgr2=0x7fb438074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306] conn(0x7fb438072350 0x7fb438074760 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 -- 172.21.15.102:0/171855674 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb43806c680 msgr2=0x7fb43806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb43806c680 0x7fb43806eb30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb4501024e0 tx=0x7fb44401a040 comp rx=0 tx=0).stop 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 -- 172.21.15.102:0/171855674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4500ff2e0 msgr2=0x7fb450100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4500ff2e0 0x7fb450100cf0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb44000d8d0 tx=0x7fb44000dbe0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 -- 172.21.15.102:0/171855674 shutdown_connections 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb43806c680 0x7fb43806eb30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.190:6808/320320306,v1:172.21.15.190:6809/320320306] conn(0x7fb438072350 0x7fb438074760 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4500ff2e0 0x7fb450100cf0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 --2- 172.21.15.102:0/171855674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4500ffc80 0x7fb450101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 -- 172.21.15.102:0/171855674 >> 172.21.15.102:0/171855674 conn(0x7fb4500fab80 msgr2=0x7fb450068e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 -- 172.21.15.102:0/171855674 shutdown_connections 2024-03-26T18:51:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:35.677+0000 7fb455271700 1 -- 172.21.15.102:0/171855674 wait complete. 2024-03-26T18:51:36.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.339+0000 7fbe973b1700 1 -- 172.21.15.102:0/243392475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe900ff7c0 msgr2=0x7fbe9010c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:36.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.339+0000 7fbe973b1700 1 --2- 172.21.15.102:0/243392475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe900ff7c0 0x7fbe9010c520 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009910 tx=0x7fbe80009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:36.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.339+0000 7fbe973b1700 1 -- 172.21.15.102:0/243392475 shutdown_connections 2024-03-26T18:51:36.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.339+0000 7fbe973b1700 1 --2- 172.21.15.102:0/243392475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe900ff7c0 0x7fbe9010c520 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:36.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.339+0000 7fbe973b1700 1 --2- 172.21.15.102:0/243392475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe900feeb0 0x7fbe900ff280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:36.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.339+0000 7fbe973b1700 1 -- 172.21.15.102:0/243392475 >> 172.21.15.102:0/243392475 conn(0x7fbe900fab80 msgr2=0x7fbe900fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:36.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.339+0000 7fbe973b1700 1 -- 172.21.15.102:0/243392475 shutdown_connections 2024-03-26T18:51:36.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.339+0000 7fbe973b1700 1 -- 172.21.15.102:0/243392475 wait complete. 2024-03-26T18:51:36.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.339+0000 7fbe973b1700 1 Processor -- start 2024-03-26T18:51:36.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe973b1700 1 -- start start 2024-03-26T18:51:36.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe973b1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe900feeb0 0x7fbe90101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe973b1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe900ff7c0 0x7fbe901016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe973b1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe90105320 con 0x7fbe900feeb0 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe973b1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe90101c10 con 0x7fbe900ff7c0 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe9514d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe900feeb0 0x7fbe90101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe9514d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe900feeb0 0x7fbe90101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43094/0 (socket says 172.21.15.102:43094) 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe9514d700 1 -- 172.21.15.102:0/960195755 learned_addr learned my addr 172.21.15.102:0/960195755 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe9494c700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe900ff7c0 0x7fbe901016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe9514d700 1 -- 172.21.15.102:0/960195755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe900ff7c0 msgr2=0x7fbe901016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe9514d700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe900ff7c0 0x7fbe901016d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.340+0000 7fbe9514d700 1 -- 172.21.15.102:0/960195755 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe800095f0 con 0x7fbe900feeb0 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.341+0000 7fbe9514d700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe900feeb0 0x7fbe90101190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00e920 tx=0x7fbe8c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.341+0000 7fbe867fc700 1 -- 172.21.15.102:0/960195755 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe8c009790 con 0x7fbe900feeb0 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.341+0000 7fbe867fc700 1 -- 172.21.15.102:0/960195755 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbe8c00cba0 con 0x7fbe900feeb0 2024-03-26T18:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.341+0000 7fbe867fc700 1 -- 172.21.15.102:0/960195755 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe8c018630 con 0x7fbe900feeb0 2024-03-26T18:51:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.341+0000 7fbe973b1700 1 -- 172.21.15.102:0/960195755 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe90101ef0 con 0x7fbe900feeb0 2024-03-26T18:51:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.341+0000 7fbe973b1700 1 -- 172.21.15.102:0/960195755 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe901a66e0 con 0x7fbe900feeb0 2024-03-26T18:51:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.342+0000 7fbe973b1700 1 -- 172.21.15.102:0/960195755 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbe9010f720 con 0x7fbe900feeb0 2024-03-26T18:51:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.342+0000 7fbe867fc700 1 -- 172.21.15.102:0/960195755 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fbe8c00cd10 con 0x7fbe900feeb0 2024-03-26T18:51:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.343+0000 7fbe867fc700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbe7c06c680 0x7fbe7c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.343+0000 7fbe867fc700 1 -- 172.21.15.102:0/960195755 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbe8c014070 con 0x7fbe900feeb0 2024-03-26T18:51:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.343+0000 7fbe9494c700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbe7c06c680 0x7fbe7c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.343+0000 7fbe867fc700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793] conn(0x7fbe7c072350 0x7fbe7c074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.343+0000 7fbe867fc700 1 -- 172.21.15.102:0/960195755 --> [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe7c074e10 con 0x7fbe7c072350 2024-03-26T18:51:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.343+0000 7fbe867fc700 1 -- 172.21.15.102:0/960195755 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe8c08de40 con 0x7fbe900feeb0 2024-03-26T18:51:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.343+0000 7fbe9494c700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbe7c06c680 0x7fbe7c06eb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009fd0 tx=0x7fbe80005940 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:36.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.343+0000 7fbe9594e700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793] conn(0x7fbe7c072350 0x7fbe7c074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:36.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.344+0000 7fbe9594e700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793] conn(0x7fbe7c072350 0x7fbe7c074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:36.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.345+0000 7fbe867fc700 1 -- 172.21.15.102:0/960195755 <== osd.3 v2:172.21.15.190:6800/1428928793 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fbe7c074e10 con 0x7fbe7c072350 2024-03-26T18:51:36.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.368+0000 7fbe973b1700 1 -- 172.21.15.102:0/960195755 --> [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbe9004ea50 con 0x7fbe7c072350 2024-03-26T18:51:36.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe867fc700 1 -- 172.21.15.102:0/960195755 <== osd.3 v2:172.21.15.190:6800/1428928793 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fbe9004ea50 con 0x7fbe7c072350 2024-03-26T18:51:36.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 -- 172.21.15.102:0/960195755 >> [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793] conn(0x7fbe7c072350 msgr2=0x7fbe7c074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:36.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793] conn(0x7fbe7c072350 0x7fbe7c074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:36.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 -- 172.21.15.102:0/960195755 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbe7c06c680 msgr2=0x7fbe7c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:36.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbe7c06c680 0x7fbe7c06eb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009fd0 tx=0x7fbe80005940 comp rx=0 tx=0).stop 2024-03-26T18:51:36.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 -- 172.21.15.102:0/960195755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe900feeb0 msgr2=0x7fbe90101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:36.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe900feeb0 0x7fbe90101190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00e920 tx=0x7fbe8c00ece0 comp rx=0 tx=0).stop 2024-03-26T18:51:36.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 -- 172.21.15.102:0/960195755 shutdown_connections 2024-03-26T18:51:36.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbe7c06c680 0x7fbe7c06eb30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:36.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.190:6800/1428928793,v1:172.21.15.190:6801/1428928793] conn(0x7fbe7c072350 0x7fbe7c074760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:36.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe900feeb0 0x7fbe90101190 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:36.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 --2- 172.21.15.102:0/960195755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe900ff7c0 0x7fbe901016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:36.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 -- 172.21.15.102:0/960195755 >> 172.21.15.102:0/960195755 conn(0x7fbe900fab80 msgr2=0x7fbe900fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:36.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 -- 172.21.15.102:0/960195755 shutdown_connections 2024-03-26T18:51:36.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:36.369+0000 7fbe973b1700 1 -- 172.21.15.102:0/960195755 wait complete. 2024-03-26T18:51:36.864 INFO:teuthology.orchestra.run.smithi102.stdout:60129542173 2024-03-26T18:51:36.865 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T18:51:37.349 INFO:teuthology.orchestra.run.smithi102.stdout:150323855374 2024-03-26T18:51:37.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T18:51:37.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:37 smithi190 ceph-mon[35043]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:37.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:37 smithi102 ceph-mon[29002]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:37.931 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:38.554 INFO:teuthology.orchestra.run.smithi102.stdout:115964117012 2024-03-26T18:51:38.555 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T18:51:39.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:39 smithi102 ceph-mon[29002]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:39.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:39 smithi190 ceph-mon[35043]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:41.156 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:41.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:41 smithi190 ceph-mon[35043]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:41 smithi102 ceph-mon[29002]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:42.868 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:43.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:43 smithi102 ceph-mon[29002]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:43.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:43 smithi190 ceph-mon[35043]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:43.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.181+0000 7f8adbeb8700 1 -- 172.21.15.102:0/2778529804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad40ff390 msgr2=0x7f8ad40ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:43.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.181+0000 7f8adbeb8700 1 --2- 172.21.15.102:0/2778529804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad40ff390 0x7f8ad40ff760 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0009910 tx=0x7f8ad0009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:43.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.181+0000 7f8adbeb8700 1 -- 172.21.15.102:0/2778529804 shutdown_connections 2024-03-26T18:51:43.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.181+0000 7f8adbeb8700 1 --2- 172.21.15.102:0/2778529804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ad40ffd30 0x7f8ad4103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:43.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.181+0000 7f8adbeb8700 1 --2- 172.21.15.102:0/2778529804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad40ff390 0x7f8ad40ff760 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.181+0000 7f8adbeb8700 1 -- 172.21.15.102:0/2778529804 >> 172.21.15.102:0/2778529804 conn(0x7f8ad4074f50 msgr2=0x7f8ad4075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.181+0000 7f8adbeb8700 1 -- 172.21.15.102:0/2778529804 shutdown_connections 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.181+0000 7f8adbeb8700 1 -- 172.21.15.102:0/2778529804 wait complete. 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8adbeb8700 1 Processor -- start 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8adbeb8700 1 -- start start 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8adbeb8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ad40ff390 0x7f8ad4195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8adbeb8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad40ffd30 0x7f8ad4196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8adbeb8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad41969e0 con 0x7f8ad40ffd30 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8adbeb8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad419a770 con 0x7f8ad40ff390 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8ad9453700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad40ffd30 0x7f8ad4196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8ad9453700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad40ffd30 0x7f8ad4196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34402/0 (socket says 172.21.15.102:34402) 2024-03-26T18:51:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8ad9453700 1 -- 172.21.15.102:0/3677906460 learned_addr learned my addr 172.21.15.102:0/3677906460 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:43.878 INFO:teuthology.orchestra.run.smithi102.stdout:81604378650 2024-03-26T18:51:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8ad9c54700 1 --2- 172.21.15.102:0/3677906460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ad40ff390 0x7f8ad4195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8ad9453700 1 -- 172.21.15.102:0/3677906460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ad40ff390 msgr2=0x7f8ad4195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8ad9453700 1 --2- 172.21.15.102:0/3677906460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ad40ff390 0x7f8ad4195dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8ad9453700 1 -- 172.21.15.102:0/3677906460 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad00095f0 con 0x7f8ad40ffd30 2024-03-26T18:51:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.182+0000 7f8ad9453700 1 --2- 172.21.15.102:0/3677906460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad40ffd30 0x7f8ad4196300 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400d900 tx=0x7f8ac400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.183+0000 7f8acaffd700 1 -- 172.21.15.102:0/3677906460 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ac4009900 con 0x7f8ad40ffd30 2024-03-26T18:51:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.183+0000 7f8adbeb8700 1 -- 172.21.15.102:0/3677906460 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad419aa50 con 0x7f8ad40ffd30 2024-03-26T18:51:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.183+0000 7f8acaffd700 1 -- 172.21.15.102:0/3677906460 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8ac4010460 con 0x7f8ad40ffd30 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.183+0000 7f8acaffd700 1 -- 172.21.15.102:0/3677906460 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ac400f5f0 con 0x7f8ad40ffd30 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.183+0000 7f8adbeb8700 1 -- 172.21.15.102:0/3677906460 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad419afa0 con 0x7f8ad40ffd30 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.183+0000 7f8adbeb8700 1 -- 172.21.15.102:0/3677906460 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad404ea50 con 0x7f8ad40ffd30 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.184+0000 7f8acaffd700 1 -- 172.21.15.102:0/3677906460 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f8ac40105d0 con 0x7f8ad40ffd30 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.184+0000 7f8acaffd700 1 --2- 172.21.15.102:0/3677906460 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ac006c680 0x7f8ac006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.184+0000 7f8acaffd700 1 -- 172.21.15.102:0/3677906460 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8ac408b0d0 con 0x7f8ad40ffd30 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.184+0000 7f8ad9c54700 1 --2- 172.21.15.102:0/3677906460 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ac006c680 0x7f8ac006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.185+0000 7f8ad9c54700 1 --2- 172.21.15.102:0/3677906460 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ac006c680 0x7f8ac006eb30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0006010 tx=0x7f8ad001a040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.189+0000 7f8acaffd700 1 -- 172.21.15.102:0/3677906460 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8ac4055af0 con 0x7f8ad40ffd30 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.388+0000 7f8adbeb8700 1 -- 172.21.15.102:0/3677906460 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8ad4197120 con 0x7f8ad40ffd30 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.388+0000 7f8acaffd700 1 -- 172.21.15.102:0/3677906460 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8ac4059110 con 0x7f8ad40ffd30 2024-03-26T18:51:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.390+0000 7f8adbeb8700 1 -- 172.21.15.102:0/3677906460 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ac006c680 msgr2=0x7f8ac006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.390+0000 7f8adbeb8700 1 --2- 172.21.15.102:0/3677906460 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ac006c680 0x7f8ac006eb30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0006010 tx=0x7f8ad001a040 comp rx=0 tx=0).stop 2024-03-26T18:51:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.390+0000 7f8adbeb8700 1 -- 172.21.15.102:0/3677906460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad40ffd30 msgr2=0x7f8ad4196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.390+0000 7f8adbeb8700 1 --2- 172.21.15.102:0/3677906460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad40ffd30 0x7f8ad4196300 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400d900 tx=0x7f8ac400dc10 comp rx=0 tx=0).stop 2024-03-26T18:51:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.390+0000 7f8adbeb8700 1 -- 172.21.15.102:0/3677906460 shutdown_connections 2024-03-26T18:51:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.390+0000 7f8adbeb8700 1 --2- 172.21.15.102:0/3677906460 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ac006c680 0x7f8ac006eb30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.390+0000 7f8adbeb8700 1 --2- 172.21.15.102:0/3677906460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ad40ff390 0x7f8ad4195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.390+0000 7f8adbeb8700 1 --2- 172.21.15.102:0/3677906460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad40ffd30 0x7f8ad4196300 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.390+0000 7f8adbeb8700 1 -- 172.21.15.102:0/3677906460 >> 172.21.15.102:0/3677906460 conn(0x7f8ad4074f50 msgr2=0x7f8ad40fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.390+0000 7f8adbeb8700 1 -- 172.21.15.102:0/3677906460 shutdown_connections 2024-03-26T18:51:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:43.390+0000 7f8adbeb8700 1 -- 172.21.15.102:0/3677906460 wait complete. 2024-03-26T18:51:44.513 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:44 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3677906460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T18:51:44.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:44 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3677906460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T18:51:44.538 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378650 for osd.2 2024-03-26T18:51:44.538 DEBUG:teuthology.parallel:result is None 2024-03-26T18:51:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.709+0000 7f7f0f81d700 1 -- 172.21.15.102:0/1706466325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f081003b0 msgr2=0x7f7f08100780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.709+0000 7f7f0f81d700 1 --2- 172.21.15.102:0/1706466325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f081003b0 0x7f7f08100780 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009910 tx=0x7f7f04009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.709+0000 7f7f0f81d700 1 -- 172.21.15.102:0/1706466325 shutdown_connections 2024-03-26T18:51:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.709+0000 7f7f0f81d700 1 --2- 172.21.15.102:0/1706466325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f08100d50 0x7f7f08104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.709+0000 7f7f0f81d700 1 --2- 172.21.15.102:0/1706466325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f081003b0 0x7f7f08100780 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.709+0000 7f7f0f81d700 1 -- 172.21.15.102:0/1706466325 >> 172.21.15.102:0/1706466325 conn(0x7f7f080fbc70 msgr2=0x7f7f080fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:44.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.709+0000 7f7f0f81d700 1 -- 172.21.15.102:0/1706466325 shutdown_connections 2024-03-26T18:51:44.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.709+0000 7f7f0f81d700 1 -- 172.21.15.102:0/1706466325 wait complete. 2024-03-26T18:51:44.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.710+0000 7f7f0f81d700 1 Processor -- start 2024-03-26T18:51:44.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.710+0000 7f7f0f81d700 1 -- start start 2024-03-26T18:51:44.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.710+0000 7f7f0f81d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f081003b0 0x7f7f08195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.710+0000 7f7f0f81d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f08100d50 0x7f7f081962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.710+0000 7f7f0f81d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f08196980 con 0x7f7f081003b0 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.710+0000 7f7f0f81d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0819a710 con 0x7f7f08100d50 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7f0d5b9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f081003b0 0x7f7f08195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7f0d5b9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f081003b0 0x7f7f08195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34420/0 (socket says 172.21.15.102:34420) 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7f0d5b9700 1 -- 172.21.15.102:0/3828893241 learned_addr learned my addr 172.21.15.102:0/3828893241 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7f0cdb8700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f08100d50 0x7f7f081962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7f0d5b9700 1 -- 172.21.15.102:0/3828893241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f08100d50 msgr2=0x7f7f081962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7f0d5b9700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f08100d50 0x7f7f081962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7f0d5b9700 1 -- 172.21.15.102:0/3828893241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f040095f0 con 0x7f7f081003b0 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7f0cdb8700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f08100d50 0x7f7f081962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7f0d5b9700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f081003b0 0x7f7f08195d60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7f04006010 tx=0x7f7f0400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7efe7fc700 1 -- 172.21.15.102:0/3828893241 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f0401d070 con 0x7f7f081003b0 2024-03-26T18:51:44.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7efe7fc700 1 -- 172.21.15.102:0/3828893241 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7f0400f460 con 0x7f7f081003b0 2024-03-26T18:51:44.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7f0f81d700 1 -- 172.21.15.102:0/3828893241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0819a990 con 0x7f7f081003b0 2024-03-26T18:51:44.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7efe7fc700 1 -- 172.21.15.102:0/3828893241 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f04021640 con 0x7f7f081003b0 2024-03-26T18:51:44.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.711+0000 7f7f0f81d700 1 -- 172.21.15.102:0/3828893241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0819ae80 con 0x7f7f081003b0 2024-03-26T18:51:44.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.712+0000 7f7f0f81d700 1 -- 172.21.15.102:0/3828893241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f081086c0 con 0x7f7f081003b0 2024-03-26T18:51:44.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.713+0000 7f7efe7fc700 1 -- 172.21.15.102:0/3828893241 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f7f0400faf0 con 0x7f7f081003b0 2024-03-26T18:51:44.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.713+0000 7f7efe7fc700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ef406c680 0x7f7ef406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:44.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.713+0000 7f7efe7fc700 1 -- 172.21.15.102:0/3828893241 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7f0408c8f0 con 0x7f7f081003b0 2024-03-26T18:51:44.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.713+0000 7f7f0cdb8700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ef406c680 0x7f7ef406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:44.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.714+0000 7f7f0cdb8700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ef406c680 0x7f7ef406eb30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7f08197380 tx=0x7f7ef8008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:44.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.718+0000 7f7efe7fc700 1 -- 172.21.15.102:0/3828893241 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7f04057310 con 0x7f7f081003b0 2024-03-26T18:51:44.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.912+0000 7f7f0f81d700 1 -- 172.21.15.102:0/3828893241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7f0819b160 con 0x7f7f081003b0 2024-03-26T18:51:44.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.912+0000 7f7efe7fc700 1 -- 172.21.15.102:0/3828893241 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7f04026090 con 0x7f7f081003b0 2024-03-26T18:51:44.905 INFO:teuthology.orchestra.run.smithi102.stdout:38654705701 2024-03-26T18:51:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.915+0000 7f7f0f81d700 1 -- 172.21.15.102:0/3828893241 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ef406c680 msgr2=0x7f7ef406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.915+0000 7f7f0f81d700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ef406c680 0x7f7ef406eb30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7f08197380 tx=0x7f7ef8008040 comp rx=0 tx=0).stop 2024-03-26T18:51:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.915+0000 7f7f0f81d700 1 -- 172.21.15.102:0/3828893241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f081003b0 msgr2=0x7f7f08195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.915+0000 7f7f0f81d700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f081003b0 0x7f7f08195d60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7f04006010 tx=0x7f7f0400be20 comp rx=0 tx=0).stop 2024-03-26T18:51:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.915+0000 7f7f0f81d700 1 -- 172.21.15.102:0/3828893241 shutdown_connections 2024-03-26T18:51:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.915+0000 7f7f0f81d700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7ef406c680 0x7f7ef406eb30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.915+0000 7f7f0f81d700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f081003b0 0x7f7f08195d60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:44.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.915+0000 7f7f0f81d700 1 --2- 172.21.15.102:0/3828893241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f08100d50 0x7f7f081962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:44.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.915+0000 7f7f0f81d700 1 -- 172.21.15.102:0/3828893241 >> 172.21.15.102:0/3828893241 conn(0x7f7f080fbc70 msgr2=0x7f7f080fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:44.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.915+0000 7f7f0f81d700 1 -- 172.21.15.102:0/3828893241 shutdown_connections 2024-03-26T18:51:44.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:44.915+0000 7f7f0f81d700 1 -- 172.21.15.102:0/3828893241 wait complete. 2024-03-26T18:51:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.327+0000 7fd613c95700 1 -- 172.21.15.102:0/3723681833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd60c0ffd30 msgr2=0x7fd60c103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.327+0000 7fd613c95700 1 --2- 172.21.15.102:0/3723681833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd60c0ffd30 0x7fd60c103ce0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd608009910 tx=0x7fd608009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.327+0000 7fd613c95700 1 -- 172.21.15.102:0/3723681833 shutdown_connections 2024-03-26T18:51:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.327+0000 7fd613c95700 1 --2- 172.21.15.102:0/3723681833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd60c0ffd30 0x7fd60c103ce0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.327+0000 7fd613c95700 1 --2- 172.21.15.102:0/3723681833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd60c0ff390 0x7fd60c0ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.327+0000 7fd613c95700 1 -- 172.21.15.102:0/3723681833 >> 172.21.15.102:0/3723681833 conn(0x7fd60c074f50 msgr2=0x7fd60c075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.327+0000 7fd613c95700 1 -- 172.21.15.102:0/3723681833 shutdown_connections 2024-03-26T18:51:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.327+0000 7fd613c95700 1 -- 172.21.15.102:0/3723681833 wait complete. 2024-03-26T18:51:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd613c95700 1 Processor -- start 2024-03-26T18:51:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd613c95700 1 -- start start 2024-03-26T18:51:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd613c95700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd60c0ff390 0x7fd60c196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd613c95700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd60c0ffd30 0x7fd60c1970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd613c95700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c1976e0 con 0x7fd60c0ffd30 2024-03-26T18:51:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd613c95700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c197820 con 0x7fd60c0ff390 2024-03-26T18:51:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd611230700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd60c0ffd30 0x7fd60c1970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd611230700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd60c0ffd30 0x7fd60c1970c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34442/0 (socket says 172.21.15.102:34442) 2024-03-26T18:51:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd611230700 1 -- 172.21.15.102:0/2428620052 learned_addr learned my addr 172.21.15.102:0/2428620052 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd611a31700 1 --2- 172.21.15.102:0/2428620052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd60c0ff390 0x7fd60c196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd611230700 1 -- 172.21.15.102:0/2428620052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd60c0ff390 msgr2=0x7fd60c196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd611230700 1 --2- 172.21.15.102:0/2428620052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd60c0ff390 0x7fd60c196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd611230700 1 -- 172.21.15.102:0/2428620052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6080095f0 con 0x7fd60c0ffd30 2024-03-26T18:51:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd611230700 1 --2- 172.21.15.102:0/2428620052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd60c0ffd30 0x7fd60c1970c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd6080092e0 tx=0x7fd60800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.328+0000 7fd602ffd700 1 -- 172.21.15.102:0/2428620052 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd60801d070 con 0x7fd60c0ffd30 2024-03-26T18:51:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.329+0000 7fd613c95700 1 -- 172.21.15.102:0/2428620052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c19fbf0 con 0x7fd60c0ffd30 2024-03-26T18:51:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.329+0000 7fd602ffd700 1 -- 172.21.15.102:0/2428620052 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd60800f460 con 0x7fd60c0ffd30 2024-03-26T18:51:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.329+0000 7fd602ffd700 1 -- 172.21.15.102:0/2428620052 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd608017630 con 0x7fd60c0ffd30 2024-03-26T18:51:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.329+0000 7fd613c95700 1 -- 172.21.15.102:0/2428620052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c1a0000 con 0x7fd60c0ffd30 2024-03-26T18:51:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.330+0000 7fd613c95700 1 -- 172.21.15.102:0/2428620052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60c107600 con 0x7fd60c0ffd30 2024-03-26T18:51:45.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.330+0000 7fd602ffd700 1 -- 172.21.15.102:0/2428620052 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fd60800f5d0 con 0x7fd60c0ffd30 2024-03-26T18:51:45.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.330+0000 7fd602ffd700 1 --2- 172.21.15.102:0/2428620052 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd5f806c680 0x7fd5f806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:45.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.331+0000 7fd602ffd700 1 -- 172.21.15.102:0/2428620052 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd60808c9a0 con 0x7fd60c0ffd30 2024-03-26T18:51:45.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.331+0000 7fd611a31700 1 --2- 172.21.15.102:0/2428620052 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd5f806c680 0x7fd5f806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:45.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.331+0000 7fd611a31700 1 --2- 172.21.15.102:0/2428620052 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd5f806c680 0x7fd5f806eb30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc005950 tx=0x7fd5fc0058e0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:45.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.335+0000 7fd602ffd700 1 -- 172.21.15.102:0/2428620052 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd6080573c0 con 0x7fd60c0ffd30 2024-03-26T18:51:45.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.530+0000 7fd613c95700 1 -- 172.21.15.102:0/2428620052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd60c04ea50 con 0x7fd60c0ffd30 2024-03-26T18:51:45.523 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:45 smithi102 ceph-mon[29002]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:45.523 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:45 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3828893241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T18:51:45.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.530+0000 7fd602ffd700 1 -- 172.21.15.102:0/2428620052 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd60805a9e0 con 0x7fd60c0ffd30 2024-03-26T18:51:45.523 INFO:teuthology.orchestra.run.smithi102.stdout:171798691850 2024-03-26T18:51:45.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.532+0000 7fd613c95700 1 -- 172.21.15.102:0/2428620052 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd5f806c680 msgr2=0x7fd5f806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:45.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.532+0000 7fd613c95700 1 --2- 172.21.15.102:0/2428620052 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd5f806c680 0x7fd5f806eb30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc005950 tx=0x7fd5fc0058e0 comp rx=0 tx=0).stop 2024-03-26T18:51:45.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.532+0000 7fd613c95700 1 -- 172.21.15.102:0/2428620052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd60c0ffd30 msgr2=0x7fd60c1970c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:45.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.532+0000 7fd613c95700 1 --2- 172.21.15.102:0/2428620052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd60c0ffd30 0x7fd60c1970c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd6080092e0 tx=0x7fd60800be20 comp rx=0 tx=0).stop 2024-03-26T18:51:45.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.532+0000 7fd613c95700 1 -- 172.21.15.102:0/2428620052 shutdown_connections 2024-03-26T18:51:45.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.532+0000 7fd613c95700 1 --2- 172.21.15.102:0/2428620052 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd5f806c680 0x7fd5f806eb30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:45.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.532+0000 7fd613c95700 1 --2- 172.21.15.102:0/2428620052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd60c0ff390 0x7fd60c196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:45.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.532+0000 7fd613c95700 1 --2- 172.21.15.102:0/2428620052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd60c0ffd30 0x7fd60c1970c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:45.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.532+0000 7fd613c95700 1 -- 172.21.15.102:0/2428620052 >> 172.21.15.102:0/2428620052 conn(0x7fd60c074f50 msgr2=0x7fd60c0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:45.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.532+0000 7fd613c95700 1 -- 172.21.15.102:0/2428620052 shutdown_connections 2024-03-26T18:51:45.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:45.532+0000 7fd613c95700 1 -- 172.21.15.102:0/2428620052 wait complete. 2024-03-26T18:51:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:45 smithi190 ceph-mon[35043]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:45 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3828893241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T18:51:46.369 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:46 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2428620052' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T18:51:46.512 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705701 for osd.0 2024-03-26T18:51:46.512 DEBUG:teuthology.parallel:result is None 2024-03-26T18:51:46.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:46 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2428620052' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T18:51:46.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.894+0000 7fbebcb73700 1 -- 172.21.15.102:0/2805636193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb80ff2c0 msgr2=0x7fbeb80ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:46.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.894+0000 7fbebcb73700 1 --2- 172.21.15.102:0/2805636193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb80ff2c0 0x7fbeb80ff690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009910 tx=0x7fbeb0009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:46.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.894+0000 7fbebcb73700 1 -- 172.21.15.102:0/2805636193 shutdown_connections 2024-03-26T18:51:46.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.894+0000 7fbebcb73700 1 --2- 172.21.15.102:0/2805636193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbeb80ffc60 0x7fbeb8103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:46.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.894+0000 7fbebcb73700 1 --2- 172.21.15.102:0/2805636193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb80ff2c0 0x7fbeb80ff690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:46.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.894+0000 7fbebcb73700 1 -- 172.21.15.102:0/2805636193 >> 172.21.15.102:0/2805636193 conn(0x7fbeb80fab80 msgr2=0x7fbeb80fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:46.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.894+0000 7fbebcb73700 1 -- 172.21.15.102:0/2805636193 shutdown_connections 2024-03-26T18:51:46.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.894+0000 7fbebcb73700 1 -- 172.21.15.102:0/2805636193 wait complete. 2024-03-26T18:51:46.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbebcb73700 1 Processor -- start 2024-03-26T18:51:46.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbebcb73700 1 -- start start 2024-03-26T18:51:46.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbebcb73700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb80ff2c0 0x7fbeb8194cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:46.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbebcb73700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbeb80ffc60 0x7fbeb81951f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:46.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbebcb73700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb8195880 con 0x7fbeb80ff2c0 2024-03-26T18:51:46.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbebcb73700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb81985b0 con 0x7fbeb80ffc60 2024-03-26T18:51:46.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbeb659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb80ff2c0 0x7fbeb8194cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:46.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbeb659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb80ff2c0 0x7fbeb8194cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34454/0 (socket says 172.21.15.102:34454) 2024-03-26T18:51:46.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbeb659c700 1 -- 172.21.15.102:0/514819679 learned_addr learned my addr 172.21.15.102:0/514819679 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:46.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbeb5d9b700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbeb80ffc60 0x7fbeb81951f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:46.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbeb659c700 1 -- 172.21.15.102:0/514819679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbeb80ffc60 msgr2=0x7fbeb81951f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:46.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbeb659c700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbeb80ffc60 0x7fbeb81951f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:46.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbeb659c700 1 -- 172.21.15.102:0/514819679 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb00095f0 con 0x7fbeb80ff2c0 2024-03-26T18:51:46.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbeb5d9b700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbeb80ffc60 0x7fbeb81951f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T18:51:46.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.895+0000 7fbeb659c700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb80ff2c0 0x7fbeb8194cb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0006010 tx=0x7fbeb000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:46.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.896+0000 7fbea77fe700 1 -- 172.21.15.102:0/514819679 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb001d070 con 0x7fbeb80ff2c0 2024-03-26T18:51:46.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.896+0000 7fbebcb73700 1 -- 172.21.15.102:0/514819679 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb8198830 con 0x7fbeb80ff2c0 2024-03-26T18:51:46.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.896+0000 7fbea77fe700 1 -- 172.21.15.102:0/514819679 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbeb000f460 con 0x7fbeb80ff2c0 2024-03-26T18:51:46.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.896+0000 7fbea77fe700 1 -- 172.21.15.102:0/514819679 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb0021640 con 0x7fbeb80ff2c0 2024-03-26T18:51:46.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.896+0000 7fbebcb73700 1 -- 172.21.15.102:0/514819679 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb8198d20 con 0x7fbeb80ff2c0 2024-03-26T18:51:46.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.897+0000 7fbebcb73700 1 -- 172.21.15.102:0/514819679 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb81075d0 con 0x7fbeb80ff2c0 2024-03-26T18:51:46.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.897+0000 7fbea77fe700 1 -- 172.21.15.102:0/514819679 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fbeb0004b70 con 0x7fbeb80ff2c0 2024-03-26T18:51:46.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.897+0000 7fbea77fe700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbea006c680 0x7fbea006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:46.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.898+0000 7fbea77fe700 1 -- 172.21.15.102:0/514819679 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbeb008c8c0 con 0x7fbeb80ff2c0 2024-03-26T18:51:46.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.898+0000 7fbeb5d9b700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbea006c680 0x7fbea006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:46.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.898+0000 7fbeb5d9b700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbea006c680 0x7fbea006eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8196290 tx=0x7fbeac0073c0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:46.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:46.902+0000 7fbea77fe700 1 -- 172.21.15.102:0/514819679 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbeb00572e0 con 0x7fbeb80ff2c0 2024-03-26T18:51:47.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.096+0000 7fbebcb73700 1 -- 172.21.15.102:0/514819679 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbeb8106fb0 con 0x7fbeb80ff2c0 2024-03-26T18:51:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.097+0000 7fbea77fe700 1 -- 172.21.15.102:0/514819679 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbeb0026090 con 0x7fbeb80ff2c0 2024-03-26T18:51:47.089 INFO:teuthology.orchestra.run.smithi102.stdout:60129542176 2024-03-26T18:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.099+0000 7fbebcb73700 1 -- 172.21.15.102:0/514819679 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbea006c680 msgr2=0x7fbea006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.099+0000 7fbebcb73700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbea006c680 0x7fbea006eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8196290 tx=0x7fbeac0073c0 comp rx=0 tx=0).stop 2024-03-26T18:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.099+0000 7fbebcb73700 1 -- 172.21.15.102:0/514819679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb80ff2c0 msgr2=0x7fbeb8194cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.099+0000 7fbebcb73700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb80ff2c0 0x7fbeb8194cb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0006010 tx=0x7fbeb000be20 comp rx=0 tx=0).stop 2024-03-26T18:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.099+0000 7fbebcb73700 1 -- 172.21.15.102:0/514819679 shutdown_connections 2024-03-26T18:51:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.099+0000 7fbebcb73700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fbea006c680 0x7fbea006eb30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.099+0000 7fbebcb73700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb80ff2c0 0x7fbeb8194cb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.099+0000 7fbebcb73700 1 --2- 172.21.15.102:0/514819679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbeb80ffc60 0x7fbeb81951f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.099+0000 7fbebcb73700 1 -- 172.21.15.102:0/514819679 >> 172.21.15.102:0/514819679 conn(0x7fbeb80fab80 msgr2=0x7fbeb80fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.099+0000 7fbebcb73700 1 -- 172.21.15.102:0/514819679 shutdown_connections 2024-03-26T18:51:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:47.099+0000 7fbebcb73700 1 -- 172.21.15.102:0/514819679 wait complete. 2024-03-26T18:51:47.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:47 smithi190 ceph-mon[35043]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:47.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:47 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/514819679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T18:51:47.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:47 smithi102 ceph-mon[29002]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:47.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:47 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/514819679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T18:51:48.311 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542176 for osd.1 2024-03-26T18:51:48.311 DEBUG:teuthology.parallel:result is None 2024-03-26T18:51:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:48 smithi190 ceph-mon[35043]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:48.535 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:48 smithi102 ceph-mon[29002]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:48.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.986+0000 7f7d975f6700 1 -- 172.21.15.102:0/1054245615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d901013d0 msgr2=0x7f7d901017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:48.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.986+0000 7f7d975f6700 1 --2- 172.21.15.102:0/1054245615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d901013d0 0x7f7d901017a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009910 tx=0x7f7d88009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:48.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.986+0000 7f7d975f6700 1 -- 172.21.15.102:0/1054245615 shutdown_connections 2024-03-26T18:51:48.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.986+0000 7f7d975f6700 1 --2- 172.21.15.102:0/1054245615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d90101ce0 0x7f7d9010a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:48.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.986+0000 7f7d975f6700 1 --2- 172.21.15.102:0/1054245615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d901013d0 0x7f7d901017a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:48.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.986+0000 7f7d975f6700 1 -- 172.21.15.102:0/1054245615 >> 172.21.15.102:0/1054245615 conn(0x7f7d900fabb0 msgr2=0x7f7d900fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:48.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.986+0000 7f7d975f6700 1 -- 172.21.15.102:0/1054245615 shutdown_connections 2024-03-26T18:51:48.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.986+0000 7f7d975f6700 1 -- 172.21.15.102:0/1054245615 wait complete. 2024-03-26T18:51:48.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d975f6700 1 Processor -- start 2024-03-26T18:51:48.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d975f6700 1 -- start start 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d975f6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d901013d0 0x7f7d9019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d975f6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d90101ce0 0x7f7d9019bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d975f6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9019c680 con 0x7f7d90101ce0 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d975f6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d90195b30 con 0x7f7d901013d0 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d94b91700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d90101ce0 0x7f7d9019bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d94b91700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d90101ce0 0x7f7d9019bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34412/0 (socket says 172.21.15.102:34412) 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d94b91700 1 -- 172.21.15.102:0/275034014 learned_addr learned my addr 172.21.15.102:0/275034014 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d95392700 1 --2- 172.21.15.102:0/275034014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d901013d0 0x7f7d9019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d94b91700 1 -- 172.21.15.102:0/275034014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d901013d0 msgr2=0x7f7d9019bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d94b91700 1 --2- 172.21.15.102:0/275034014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d901013d0 0x7f7d9019bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d94b91700 1 -- 172.21.15.102:0/275034014 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d880095f0 con 0x7f7d90101ce0 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d94b91700 1 --2- 172.21.15.102:0/275034014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d90101ce0 0x7f7d9019bff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000d900 tx=0x7f7d8000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.987+0000 7f7d867fc700 1 -- 172.21.15.102:0/275034014 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d80004a00 con 0x7f7d90101ce0 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.988+0000 7f7d867fc700 1 -- 172.21.15.102:0/275034014 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7d800054d0 con 0x7f7d90101ce0 2024-03-26T18:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.988+0000 7f7d975f6700 1 -- 172.21.15.102:0/275034014 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d90195e10 con 0x7f7d90101ce0 2024-03-26T18:51:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.988+0000 7f7d867fc700 1 -- 172.21.15.102:0/275034014 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d80009d00 con 0x7f7d90101ce0 2024-03-26T18:51:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.988+0000 7f7d975f6700 1 -- 172.21.15.102:0/275034014 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d90196360 con 0x7f7d90101ce0 2024-03-26T18:51:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.988+0000 7f7d975f6700 1 -- 172.21.15.102:0/275034014 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d90107880 con 0x7f7d90101ce0 2024-03-26T18:51:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.989+0000 7f7d867fc700 1 -- 172.21.15.102:0/275034014 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f7d80005020 con 0x7f7d90101ce0 2024-03-26T18:51:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.989+0000 7f7d867fc700 1 --2- 172.21.15.102:0/275034014 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d7c06c680 0x7f7d7c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.989+0000 7f7d867fc700 1 -- 172.21.15.102:0/275034014 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7d8008b3e0 con 0x7f7d90101ce0 2024-03-26T18:51:48.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.989+0000 7f7d95392700 1 --2- 172.21.15.102:0/275034014 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d7c06c680 0x7f7d7c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:48.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.990+0000 7f7d95392700 1 --2- 172.21.15.102:0/275034014 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d7c06c680 0x7f7d7c06eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7d88006010 tx=0x7f7d8801a040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:48.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:48.994+0000 7f7d867fc700 1 -- 172.21.15.102:0/275034014 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7d80059910 con 0x7f7d90101ce0 2024-03-26T18:51:49.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.195+0000 7f7d975f6700 1 -- 172.21.15.102:0/275034014 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f7d9004ea50 con 0x7f7d90101ce0 2024-03-26T18:51:49.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.195+0000 7f7d867fc700 1 -- 172.21.15.102:0/275034014 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7d80009e60 con 0x7f7d90101ce0 2024-03-26T18:51:49.188 INFO:teuthology.orchestra.run.smithi102.stdout:150323855376 2024-03-26T18:51:49.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7f7d975f6700 1 -- 172.21.15.102:0/275034014 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d7c06c680 msgr2=0x7f7d7c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:49.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7f7d975f6700 1 --2- 172.21.15.102:0/275034014 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d7c06c680 0x7f7d7c06eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7d88006010 tx=0x7f7d8801a040 comp rx=0 tx=0).stop 2024-03-26T18:51:49.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7f7d975f6700 1 -- 172.21.15.102:0/275034014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d90101ce0 msgr2=0x7f7d9019bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:49.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7f7d975f6700 1 --2- 172.21.15.102:0/275034014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d90101ce0 0x7f7d9019bff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000d900 tx=0x7f7d8000dc10 comp rx=0 tx=0).stop 2024-03-26T18:51:49.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7f7d975f6700 1 -- 172.21.15.102:0/275034014 shutdown_connections 2024-03-26T18:51:49.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7f7d975f6700 1 --2- 172.21.15.102:0/275034014 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7d7c06c680 0x7f7d7c06eb30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:49.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7f7d975f6700 1 --2- 172.21.15.102:0/275034014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d901013d0 0x7f7d9019bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:49.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7f7d975f6700 1 --2- 172.21.15.102:0/275034014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d90101ce0 0x7f7d9019bff0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:49.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7f7d975f6700 1 -- 172.21.15.102:0/275034014 >> 172.21.15.102:0/275034014 conn(0x7f7d900fabb0 msgr2=0x7f7d901049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:49.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7f7d975f6700 1 -- 172.21.15.102:0/275034014 shutdown_connections 2024-03-26T18:51:49.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7f7d975f6700 1 -- 172.21.15.102:0/275034014 wait complete. 2024-03-26T18:51:49.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.194+0000 7ff2e547b700 1 -- 172.21.15.102:0/272690160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e0100da0 msgr2=0x7ff2e0104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:49.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.194+0000 7ff2e547b700 1 --2- 172.21.15.102:0/272690160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e0100da0 0x7ff2e0104d70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0009910 tx=0x7ff2d0009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:49.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.194+0000 7ff2e547b700 1 -- 172.21.15.102:0/272690160 shutdown_connections 2024-03-26T18:51:49.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.194+0000 7ff2e547b700 1 --2- 172.21.15.102:0/272690160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e0100da0 0x7ff2e0104d70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:49.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.194+0000 7ff2e547b700 1 --2- 172.21.15.102:0/272690160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2e0100400 0x7ff2e01007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.194+0000 7ff2e547b700 1 -- 172.21.15.102:0/272690160 >> 172.21.15.102:0/272690160 conn(0x7ff2e00fbca0 msgr2=0x7ff2e00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.195+0000 7ff2e547b700 1 -- 172.21.15.102:0/272690160 shutdown_connections 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.195+0000 7ff2e547b700 1 -- 172.21.15.102:0/272690160 wait complete. 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.195+0000 7ff2e547b700 1 Processor -- start 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.195+0000 7ff2e547b700 1 -- start start 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2e547b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2e0100400 0x7ff2e0195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2e547b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e0100da0 0x7ff2e0196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2e547b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e0196970 con 0x7ff2e0100da0 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2e547b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e019a700 con 0x7ff2e0100400 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2de7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e0100da0 0x7ff2e0196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2de7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e0100da0 0x7ff2e0196290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34430/0 (socket says 172.21.15.102:34430) 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2de7fc700 1 -- 172.21.15.102:0/3495546437 learned_addr learned my addr 172.21.15.102:0/3495546437 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2deffd700 1 --2- 172.21.15.102:0/3495546437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2e0100400 0x7ff2e0195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:49.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2de7fc700 1 -- 172.21.15.102:0/3495546437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2e0100400 msgr2=0x7ff2e0195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2de7fc700 1 --2- 172.21.15.102:0/3495546437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2e0100400 0x7ff2e0195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2de7fc700 1 -- 172.21.15.102:0/3495546437 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d00095f0 con 0x7ff2e0100da0 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2de7fc700 1 --2- 172.21.15.102:0/3495546437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e0100da0 0x7ff2e0196290 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0000c00 tx=0x7ff2d000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2c7fff700 1 -- 172.21.15.102:0/3495546437 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2d001d070 con 0x7ff2e0100da0 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2c7fff700 1 -- 172.21.15.102:0/3495546437 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff2d0005dc0 con 0x7ff2e0100da0 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2e547b700 1 -- 172.21.15.102:0/3495546437 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e019a980 con 0x7ff2e0100da0 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2c7fff700 1 -- 172.21.15.102:0/3495546437 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2d0017400 con 0x7ff2e0100da0 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.196+0000 7ff2e547b700 1 -- 172.21.15.102:0/3495546437 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e019ae70 con 0x7ff2e0100da0 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.197+0000 7ff2e547b700 1 -- 172.21.15.102:0/3495546437 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e0108770 con 0x7ff2e0100da0 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7ff2c7fff700 1 -- 172.21.15.102:0/3495546437 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7ff2d000f460 con 0x7ff2e0100da0 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.198+0000 7ff2c7fff700 1 --2- 172.21.15.102:0/3495546437 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2c806c680 0x7ff2c806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.199+0000 7ff2c7fff700 1 -- 172.21.15.102:0/3495546437 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff2d008ca10 con 0x7ff2e0100da0 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.199+0000 7ff2deffd700 1 --2- 172.21.15.102:0/3495546437 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2c806c680 0x7ff2c806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.199+0000 7ff2deffd700 1 --2- 172.21.15.102:0/3495546437 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2c806c680 0x7ff2c806eb30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4005950 tx=0x7ff2d400b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:49.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.203+0000 7ff2c7fff700 1 -- 172.21.15.102:0/3495546437 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff2d0057430 con 0x7ff2e0100da0 2024-03-26T18:51:49.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:49 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:51:49.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:49 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/275034014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T18:51:49.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.404+0000 7ff2e547b700 1 -- 172.21.15.102:0/3495546437 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff2e004ea50 con 0x7ff2e0100da0 2024-03-26T18:51:49.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.404+0000 7ff2c7fff700 1 -- 172.21.15.102:0/3495546437 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff2d005aa50 con 0x7ff2e0100da0 2024-03-26T18:51:49.397 INFO:teuthology.orchestra.run.smithi102.stdout:115964117014 2024-03-26T18:51:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.406+0000 7ff2e547b700 1 -- 172.21.15.102:0/3495546437 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2c806c680 msgr2=0x7ff2c806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.406+0000 7ff2e547b700 1 --2- 172.21.15.102:0/3495546437 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2c806c680 0x7ff2c806eb30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4005950 tx=0x7ff2d400b410 comp rx=0 tx=0).stop 2024-03-26T18:51:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.406+0000 7ff2e547b700 1 -- 172.21.15.102:0/3495546437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e0100da0 msgr2=0x7ff2e0196290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.406+0000 7ff2e547b700 1 --2- 172.21.15.102:0/3495546437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e0100da0 0x7ff2e0196290 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0000c00 tx=0x7ff2d000bd70 comp rx=0 tx=0).stop 2024-03-26T18:51:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.406+0000 7ff2e547b700 1 -- 172.21.15.102:0/3495546437 shutdown_connections 2024-03-26T18:51:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.406+0000 7ff2e547b700 1 --2- 172.21.15.102:0/3495546437 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff2c806c680 0x7ff2c806eb30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.406+0000 7ff2e547b700 1 --2- 172.21.15.102:0/3495546437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2e0100400 0x7ff2e0195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.406+0000 7ff2e547b700 1 --2- 172.21.15.102:0/3495546437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e0100da0 0x7ff2e0196290 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.406+0000 7ff2e547b700 1 -- 172.21.15.102:0/3495546437 >> 172.21.15.102:0/3495546437 conn(0x7ff2e00fbca0 msgr2=0x7ff2e00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.406+0000 7ff2e547b700 1 -- 172.21.15.102:0/3495546437 shutdown_connections 2024-03-26T18:51:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:49.406+0000 7ff2e547b700 1 -- 172.21.15.102:0/3495546437 wait complete. 2024-03-26T18:51:49.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:49 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:51:49.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:49 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/275034014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T18:51:50.135 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117012 got 115964117014 for osd.3 2024-03-26T18:51:50.135 DEBUG:teuthology.parallel:result is None 2024-03-26T18:51:50.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:50 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3495546437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T18:51:50.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:50 smithi102 ceph-mon[29002]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:50.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:50 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3495546437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T18:51:50.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:50 smithi190 ceph-mon[35043]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:50.683 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855374 got 150323855376 for osd.4 2024-03-26T18:51:50.683 DEBUG:teuthology.parallel:result is None 2024-03-26T18:51:51.041 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691847 got 171798691850 for osd.5 2024-03-26T18:51:51.042 DEBUG:teuthology.parallel:result is None 2024-03-26T18:51:51.042 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T18:51:51.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T18:51:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:52.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.898+0000 7f56d5a59700 1 -- 172.21.15.102:0/4057717914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56d00ff2c0 msgr2=0x7f56d00ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:52.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.898+0000 7f56d5a59700 1 --2- 172.21.15.102:0/4057717914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56d00ff2c0 0x7f56d00ff690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f56c4009910 tx=0x7f56c4009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:52.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.898+0000 7f56d5a59700 1 -- 172.21.15.102:0/4057717914 shutdown_connections 2024-03-26T18:51:52.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.898+0000 7f56d5a59700 1 --2- 172.21.15.102:0/4057717914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56d00ffc60 0x7f56d0103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:52.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.898+0000 7f56d5a59700 1 --2- 172.21.15.102:0/4057717914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56d00ff2c0 0x7f56d00ff690 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:52.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.898+0000 7f56d5a59700 1 -- 172.21.15.102:0/4057717914 >> 172.21.15.102:0/4057717914 conn(0x7f56d00fab60 msgr2=0x7f56d00fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:52.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.898+0000 7f56d5a59700 1 -- 172.21.15.102:0/4057717914 shutdown_connections 2024-03-26T18:51:52.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.898+0000 7f56d5a59700 1 -- 172.21.15.102:0/4057717914 wait complete. 2024-03-26T18:51:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.899+0000 7f56d5a59700 1 Processor -- start 2024-03-26T18:51:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.899+0000 7f56d5a59700 1 -- start start 2024-03-26T18:51:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.899+0000 7f56d5a59700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56d00ff2c0 0x7f56d019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.899+0000 7f56d5a59700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56d00ffc60 0x7f56d019afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.899+0000 7f56d5a59700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d019b630 con 0x7f56d00ff2c0 2024-03-26T18:51:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.899+0000 7f56d5a59700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d0194ae0 con 0x7f56d00ffc60 2024-03-26T18:51:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56ceffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56d00ff2c0 0x7f56d019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56ceffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56d00ff2c0 0x7f56d019aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34456/0 (socket says 172.21.15.102:34456) 2024-03-26T18:51:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56ceffd700 1 -- 172.21.15.102:0/3784095628 learned_addr learned my addr 172.21.15.102:0/3784095628 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56ce7fc700 1 --2- 172.21.15.102:0/3784095628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56d00ffc60 0x7f56d019afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56ceffd700 1 -- 172.21.15.102:0/3784095628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56d00ffc60 msgr2=0x7f56d019afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56ceffd700 1 --2- 172.21.15.102:0/3784095628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56d00ffc60 0x7f56d019afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56ceffd700 1 -- 172.21.15.102:0/3784095628 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c40095f0 con 0x7f56d00ff2c0 2024-03-26T18:51:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56ceffd700 1 --2- 172.21.15.102:0/3784095628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56d00ff2c0 0x7f56d019aa60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f56c4006010 tx=0x7f56c400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56d4a57700 1 -- 172.21.15.102:0/3784095628 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56c401d070 con 0x7f56d00ff2c0 2024-03-26T18:51:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56d4a57700 1 -- 172.21.15.102:0/3784095628 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f56c400f460 con 0x7f56d00ff2c0 2024-03-26T18:51:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56d5a59700 1 -- 172.21.15.102:0/3784095628 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d0194d60 con 0x7f56d00ff2c0 2024-03-26T18:51:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56d4a57700 1 -- 172.21.15.102:0/3784095628 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56c4017630 con 0x7f56d00ff2c0 2024-03-26T18:51:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.900+0000 7f56d5a59700 1 -- 172.21.15.102:0/3784095628 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d0195250 con 0x7f56d00ff2c0 2024-03-26T18:51:52.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.901+0000 7f56d5a59700 1 -- 172.21.15.102:0/3784095628 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d01075d0 con 0x7f56d00ff2c0 2024-03-26T18:51:52.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.902+0000 7f56d4a57700 1 -- 172.21.15.102:0/3784095628 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f56c400f5d0 con 0x7f56d00ff2c0 2024-03-26T18:51:52.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.902+0000 7f56d4a57700 1 --2- 172.21.15.102:0/3784095628 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f56b806c680 0x7f56b806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:52.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.902+0000 7f56d4a57700 1 -- 172.21.15.102:0/3784095628 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f56c408cbd0 con 0x7f56d00ff2c0 2024-03-26T18:51:52.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.902+0000 7f56ce7fc700 1 --2- 172.21.15.102:0/3784095628 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f56b806c680 0x7f56b806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:52.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.902+0000 7f56ce7fc700 1 --2- 172.21.15.102:0/3784095628 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f56b806c680 0x7f56b806eb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f56c0006fd0 tx=0x7f56c0008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:52.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:52.906+0000 7f56d4a57700 1 -- 172.21.15.102:0/3784095628 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f56c405b080 con 0x7f56d00ff2c0 2024-03-26T18:51:53.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:52 smithi190 ceph-mon[35043]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:53.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.094+0000 7f56d5a59700 1 -- 172.21.15.102:0/3784095628 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56d0195470 con 0x7f56b806c680 2024-03-26T18:51:53.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.096+0000 7f56d4a57700 1 -- 172.21.15.102:0/3784095628 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27986 (secure 0 0 0) 0x7f56d0195470 con 0x7f56b806c680 2024-03-26T18:51:53.089 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:51:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.098+0000 7f56d5a59700 1 -- 172.21.15.102:0/3784095628 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f56b806c680 msgr2=0x7f56b806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.098+0000 7f56d5a59700 1 --2- 172.21.15.102:0/3784095628 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f56b806c680 0x7f56b806eb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f56c0006fd0 tx=0x7f56c0008040 comp rx=0 tx=0).stop 2024-03-26T18:51:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.098+0000 7f56d5a59700 1 -- 172.21.15.102:0/3784095628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56d00ff2c0 msgr2=0x7f56d019aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.098+0000 7f56d5a59700 1 --2- 172.21.15.102:0/3784095628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56d00ff2c0 0x7f56d019aa60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f56c4006010 tx=0x7f56c400be20 comp rx=0 tx=0).stop 2024-03-26T18:51:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.098+0000 7f56d5a59700 1 -- 172.21.15.102:0/3784095628 shutdown_connections 2024-03-26T18:51:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.098+0000 7f56d5a59700 1 --2- 172.21.15.102:0/3784095628 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f56b806c680 0x7f56b806eb30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.098+0000 7f56d5a59700 1 --2- 172.21.15.102:0/3784095628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56d00ff2c0 0x7f56d019aa60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.098+0000 7f56d5a59700 1 --2- 172.21.15.102:0/3784095628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56d00ffc60 0x7f56d019afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.098+0000 7f56d5a59700 1 -- 172.21.15.102:0/3784095628 >> 172.21.15.102:0/3784095628 conn(0x7f56d00fab60 msgr2=0x7f56d00fc120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.098+0000 7f56d5a59700 1 -- 172.21.15.102:0/3784095628 shutdown_connections 2024-03-26T18:51:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:53.098+0000 7f56d5a59700 1 -- 172.21.15.102:0/3784095628 wait complete. 2024-03-26T18:51:53.092 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-26T18:51:53.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:52 smithi102 ceph-mon[29002]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:53.767 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-03-26T18:51:51.889235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001966"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":187,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-26T18:51:04.575578+0000","last_change":"2024-03-26T18:50:38.471499+0000","last_active":"2024-03-26T18:51:04.575578+0000","last_peered":"2024-03-26T18:51:04.575578+0000","last_clean":"2024-03-26T18:51:04.575578+0000","last_became_active":"2024-03-26T18:50:38.471189+0000","last_became_peered":"2024-03-26T18:50:38.471189+0000","last_unstale":"2024-03-26T18:51:04.575578+0000","last_undegraded":"2024-03-26T18:51:04.575578+0000","last_fullsized":"2024-03-26T18:51:04.575578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T18:49:50.222676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T18:49:50.222676+0000","last_clean_scrub_stamp":"2024-03-26T18:49:50.222676+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:21:21.829759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]}]},{"osd":4,"up_from":35,"seq":150323855377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.495}]},{"osd":1,"last update":"Tue Mar 26 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.47999999999999998}]},{"osd":2,"last update":"Tue Mar 26 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.438},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.52800000000000002}]},{"osd":3,"last update":"Tue Mar 26 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.37}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 18:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.51400000000000001}]},{"osd":1,"last update":"Tue Mar 26 18:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.496}]},{"osd":2,"last update":"Tue Mar 26 18:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.46800000000000003}]},{"osd":4,"last update":"Tue Mar 26 18:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 18:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.27600000000000002}]},{"osd":1,"last update":"Tue Mar 26 18:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.32700000000000001}]},{"osd":3,"last update":"Tue Mar 26 18:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.499},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.44800000000000001}]},{"osd":4,"last update":"Tue Mar 26 18:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 26 18:51:25 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.308,"15min":0.30299999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.432,"5min":0.432,"15min":0.432},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.318,"15min":0.312},"min":{"1min":0.28299999999999997,"5min":0.193,"15min":0.193},"max":{"1min":0.49299999999999999,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.26900000000000002}]},{"osd":2,"last update":"Tue Mar 26 18:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.309}]},{"osd":3,"last update":"Tue Mar 26 18:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.41499999999999998}]},{"osd":4,"last update":"Tue Mar 26 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 18:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.188,"5min":0.161,"15min":0.161},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.379},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.315,"15min":0.31},"min":{"1min":0.22800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.317}]},{"osd":2,"last update":"Tue Mar 26 18:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.33300000000000002}]},{"osd":3,"last update":"Tue Mar 26 18:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.55800000000000005}]},{"osd":4,"last update":"Tue Mar 26 18:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.48399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T18:51:53.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T18:51:53.973 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:54.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:53 smithi102 ceph-mon[29002]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T18:51:54.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:53 smithi190 ceph-mon[35043]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T18:51:55.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:54 smithi190 ceph-mon[35043]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:55.299 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:54 smithi102 ceph-mon[29002]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.686+0000 7f42abbeb700 1 -- 172.21.15.102:0/1125720958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40ffc70 msgr2=0x7f42a4103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.686+0000 7f42abbeb700 1 --2- 172.21.15.102:0/1125720958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40ffc70 0x7f42a4103c40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009910 tx=0x7f42a0009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.686+0000 7f42abbeb700 1 -- 172.21.15.102:0/1125720958 shutdown_connections 2024-03-26T18:51:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.686+0000 7f42abbeb700 1 --2- 172.21.15.102:0/1125720958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40ffc70 0x7f42a4103c40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.686+0000 7f42abbeb700 1 --2- 172.21.15.102:0/1125720958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42a40ff2d0 0x7f42a40ff6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.686+0000 7f42abbeb700 1 -- 172.21.15.102:0/1125720958 >> 172.21.15.102:0/1125720958 conn(0x7f42a40fabb0 msgr2=0x7f42a40fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.686+0000 7f42abbeb700 1 -- 172.21.15.102:0/1125720958 shutdown_connections 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.686+0000 7f42abbeb700 1 -- 172.21.15.102:0/1125720958 wait complete. 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.687+0000 7f42abbeb700 1 Processor -- start 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.687+0000 7f42abbeb700 1 -- start start 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.687+0000 7f42abbeb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42a40ff2d0 0x7f42a4194d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.687+0000 7f42abbeb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40ffc70 0x7f42a41952c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.687+0000 7f42abbeb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a41959a0 con 0x7f42a40ffc70 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.687+0000 7f42abbeb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4199730 con 0x7f42a40ff2d0 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f42a9186700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40ffc70 0x7f42a41952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f42a9186700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40ffc70 0x7f42a41952c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34472/0 (socket says 172.21.15.102:34472) 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f42a9186700 1 -- 172.21.15.102:0/1321214081 learned_addr learned my addr 172.21.15.102:0/1321214081 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f42a9987700 1 --2- 172.21.15.102:0/1321214081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42a40ff2d0 0x7f42a4194d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f42a9186700 1 -- 172.21.15.102:0/1321214081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42a40ff2d0 msgr2=0x7f42a4194d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f42a9186700 1 --2- 172.21.15.102:0/1321214081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42a40ff2d0 0x7f42a4194d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f42a9186700 1 -- 172.21.15.102:0/1321214081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a00095f0 con 0x7f42a40ffc70 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f42a9186700 1 --2- 172.21.15.102:0/1321214081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40ffc70 0x7f42a41952c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f42a00092e0 tx=0x7f42a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f429affd700 1 -- 172.21.15.102:0/1321214081 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42a001d070 con 0x7f42a40ffc70 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f429affd700 1 -- 172.21.15.102:0/1321214081 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f42a0022470 con 0x7f42a40ffc70 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f42abbeb700 1 -- 172.21.15.102:0/1321214081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a41999b0 con 0x7f42a40ffc70 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f429affd700 1 -- 172.21.15.102:0/1321214081 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42a000f690 con 0x7f42a40ffc70 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.688+0000 7f42abbeb700 1 -- 172.21.15.102:0/1321214081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a4199ea0 con 0x7f42a40ffc70 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.689+0000 7f42abbeb700 1 -- 172.21.15.102:0/1321214081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a4107590 con 0x7f42a40ffc70 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.690+0000 7f429affd700 1 -- 172.21.15.102:0/1321214081 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f42a00225e0 con 0x7f42a40ffc70 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.690+0000 7f429affd700 1 --2- 172.21.15.102:0/1321214081 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f429006c680 0x7f429006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.690+0000 7f429affd700 1 -- 172.21.15.102:0/1321214081 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f42a008caf0 con 0x7f42a40ffc70 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.690+0000 7f42a9987700 1 --2- 172.21.15.102:0/1321214081 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f429006c680 0x7f429006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.690+0000 7f42a9987700 1 --2- 172.21.15.102:0/1321214081 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f429006c680 0x7f429006eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4294006fd0 tx=0x7f4294008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.695+0000 7f429affd700 1 -- 172.21.15.102:0/1321214081 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f42a00575c0 con 0x7f42a40ffc70 2024-03-26T18:51:55.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.879+0000 7f42abbeb700 1 -- 172.21.15.102:0/1321214081 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f42a419a1f0 con 0x7f429006c680 2024-03-26T18:51:55.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.880+0000 7f429affd700 1 -- 172.21.15.102:0/1321214081 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28001 (secure 0 0 0) 0x7f42a419a1f0 con 0x7f429006c680 2024-03-26T18:51:55.873 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.883+0000 7f42abbeb700 1 -- 172.21.15.102:0/1321214081 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f429006c680 msgr2=0x7f429006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.883+0000 7f42abbeb700 1 --2- 172.21.15.102:0/1321214081 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f429006c680 0x7f429006eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4294006fd0 tx=0x7f4294008040 comp rx=0 tx=0).stop 2024-03-26T18:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.883+0000 7f42abbeb700 1 -- 172.21.15.102:0/1321214081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40ffc70 msgr2=0x7f42a41952c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.883+0000 7f42abbeb700 1 --2- 172.21.15.102:0/1321214081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40ffc70 0x7f42a41952c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f42a00092e0 tx=0x7f42a000be20 comp rx=0 tx=0).stop 2024-03-26T18:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.883+0000 7f42abbeb700 1 -- 172.21.15.102:0/1321214081 shutdown_connections 2024-03-26T18:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.883+0000 7f42abbeb700 1 --2- 172.21.15.102:0/1321214081 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f429006c680 0x7f429006eb30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.883+0000 7f42abbeb700 1 --2- 172.21.15.102:0/1321214081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42a40ff2d0 0x7f42a4194d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.883+0000 7f42abbeb700 1 --2- 172.21.15.102:0/1321214081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40ffc70 0x7f42a41952c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.883+0000 7f42abbeb700 1 -- 172.21.15.102:0/1321214081 >> 172.21.15.102:0/1321214081 conn(0x7f42a40fabb0 msgr2=0x7f42a40fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.883+0000 7f42abbeb700 1 -- 172.21.15.102:0/1321214081 shutdown_connections 2024-03-26T18:51:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:55.883+0000 7f42abbeb700 1 -- 172.21.15.102:0/1321214081 wait complete. 2024-03-26T18:51:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-26T18:51:56.506 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-03-26T18:51:53.889520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001918"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":187,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-26T18:51:04.575578+0000","last_change":"2024-03-26T18:50:38.471499+0000","last_active":"2024-03-26T18:51:04.575578+0000","last_peered":"2024-03-26T18:51:04.575578+0000","last_clean":"2024-03-26T18:51:04.575578+0000","last_became_active":"2024-03-26T18:50:38.471189+0000","last_became_peered":"2024-03-26T18:50:38.471189+0000","last_unstale":"2024-03-26T18:51:04.575578+0000","last_undegraded":"2024-03-26T18:51:04.575578+0000","last_fullsized":"2024-03-26T18:51:04.575578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T18:49:50.222676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T18:49:50.222676+0000","last_clean_scrub_stamp":"2024-03-26T18:49:50.222676+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:21:21.829759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35999999999999999}]},{"osd":1,"last update":"Tue Mar 26 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.372},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.38100000000000001}]},{"osd":2,"last update":"Tue Mar 26 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.44600000000000001}]},{"osd":3,"last update":"Tue Mar 26 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.33400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 18:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.502}]},{"osd":1,"last update":"Tue Mar 26 18:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Tue Mar 26 18:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.438},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.46400000000000002}]},{"osd":4,"last update":"Tue Mar 26 18:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 18:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.27600000000000002}]},{"osd":1,"last update":"Tue Mar 26 18:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.32700000000000001}]},{"osd":3,"last update":"Tue Mar 26 18:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.499},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.44800000000000001}]},{"osd":4,"last update":"Tue Mar 26 18:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 26 18:51:25 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.308,"15min":0.30299999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.432,"5min":0.432,"15min":0.432},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.318,"15min":0.312},"min":{"1min":0.28299999999999997,"5min":0.193,"15min":0.193},"max":{"1min":0.49299999999999999,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.26900000000000002}]},{"osd":2,"last update":"Tue Mar 26 18:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.309}]},{"osd":3,"last update":"Tue Mar 26 18:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.41499999999999998}]},{"osd":4,"last update":"Tue Mar 26 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 18:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.188,"5min":0.161,"15min":0.161},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.379},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.315,"15min":0.31},"min":{"1min":0.22800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.317}]},{"osd":2,"last update":"Tue Mar 26 18:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.33300000000000002}]},{"osd":3,"last update":"Tue Mar 26 18:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.55800000000000005}]},{"osd":4,"last update":"Tue Mar 26 18:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.48399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T18:51:56.507 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T18:51:56.507 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T18:51:56.507 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T18:51:56.507 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T18:51:56.712 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:57.391 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:57 smithi102 ceph-mon[29002]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T18:51:57.391 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:57 smithi102 ceph-mon[29002]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:57.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:57 smithi190 ceph-mon[35043]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T18:51:57.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:57 smithi190 ceph-mon[35043]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.191+0000 7f28c1ad0700 1 -- 172.21.15.102:0/1689595793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28bc103860 msgr2=0x7f28bc103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.191+0000 7f28c1ad0700 1 --2- 172.21.15.102:0/1689595793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28bc103860 0x7f28bc103c30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009910 tx=0x7f28ac009c20 comp rx=0 tx=0).stop 2024-03-26T18:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.191+0000 7f28c1ad0700 1 -- 172.21.15.102:0/1689595793 shutdown_connections 2024-03-26T18:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.191+0000 7f28c1ad0700 1 --2- 172.21.15.102:0/1689595793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28bc0ff320 0x7f28bc0ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.191+0000 7f28c1ad0700 1 --2- 172.21.15.102:0/1689595793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28bc103860 0x7f28bc103c30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.191+0000 7f28c1ad0700 1 -- 172.21.15.102:0/1689595793 >> 172.21.15.102:0/1689595793 conn(0x7f28bc0748b0 msgr2=0x7f28bc074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.191+0000 7f28c1ad0700 1 -- 172.21.15.102:0/1689595793 shutdown_connections 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.191+0000 7f28c1ad0700 1 -- 172.21.15.102:0/1689595793 wait complete. 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28c1ad0700 1 Processor -- start 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28c1ad0700 1 -- start start 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28c1ad0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28bc0ff320 0x7f28bc100510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28c1ad0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28bc103860 0x7f28bc100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28c1ad0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc100f90 con 0x7f28bc0ff320 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28c1ad0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc1010d0 con 0x7f28bc103860 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28bb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28bc0ff320 0x7f28bc100510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28bb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28bc0ff320 0x7f28bc100510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34498/0 (socket says 172.21.15.102:34498) 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28bb7fe700 1 -- 172.21.15.102:0/4269894913 learned_addr learned my addr 172.21.15.102:0/4269894913 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28baffd700 1 --2- 172.21.15.102:0/4269894913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28bc103860 0x7f28bc100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28bb7fe700 1 -- 172.21.15.102:0/4269894913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28bc103860 msgr2=0x7f28bc100a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28bb7fe700 1 --2- 172.21.15.102:0/4269894913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28bc103860 0x7f28bc100a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.192+0000 7f28bb7fe700 1 -- 172.21.15.102:0/4269894913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ac0095f0 con 0x7f28bc0ff320 2024-03-26T18:51:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.193+0000 7f28bb7fe700 1 --2- 172.21.15.102:0/4269894913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28bc0ff320 0x7f28bc100510 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f28ac006010 tx=0x7f28ac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.193+0000 7f28b8ff9700 1 -- 172.21.15.102:0/4269894913 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28ac01d070 con 0x7f28bc0ff320 2024-03-26T18:51:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.193+0000 7f28b8ff9700 1 -- 172.21.15.102:0/4269894913 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f28ac00f460 con 0x7f28bc0ff320 2024-03-26T18:51:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.193+0000 7f28c1ad0700 1 -- 172.21.15.102:0/4269894913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc101350 con 0x7f28bc0ff320 2024-03-26T18:51:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.193+0000 7f28b8ff9700 1 -- 172.21.15.102:0/4269894913 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28ac021640 con 0x7f28bc0ff320 2024-03-26T18:51:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.193+0000 7f28c1ad0700 1 -- 172.21.15.102:0/4269894913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28bc1a2280 con 0x7f28bc0ff320 2024-03-26T18:51:58.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.194+0000 7f28c1ad0700 1 -- 172.21.15.102:0/4269894913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28bc108540 con 0x7f28bc0ff320 2024-03-26T18:51:58.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.194+0000 7f28b8ff9700 1 -- 172.21.15.102:0/4269894913 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f28ac00fab0 con 0x7f28bc0ff320 2024-03-26T18:51:58.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.195+0000 7f28b8ff9700 1 --2- 172.21.15.102:0/4269894913 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f28a406c680 0x7f28a406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:51:58.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.195+0000 7f28b8ff9700 1 -- 172.21.15.102:0/4269894913 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f28ac08c900 con 0x7f28bc0ff320 2024-03-26T18:51:58.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.195+0000 7f28baffd700 1 --2- 172.21.15.102:0/4269894913 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f28a406c680 0x7f28a406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:51:58.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.195+0000 7f28baffd700 1 --2- 172.21.15.102:0/4269894913 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f28a406c680 0x7f28a406eb30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f28b0006fd0 tx=0x7f28b0008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:51:58.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.199+0000 7f28b8ff9700 1 -- 172.21.15.102:0/4269894913 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f28ac057320 con 0x7f28bc0ff320 2024-03-26T18:51:58.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.434+0000 7f28c1ad0700 1 -- 172.21.15.102:0/4269894913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f28bc04ea50 con 0x7f28bc0ff320 2024-03-26T18:51:58.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.435+0000 7f28b8ff9700 1 -- 172.21.15.102:0/4269894913 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f28ac026040 con 0x7f28bc0ff320 2024-03-26T18:51:58.427 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:51:58.427 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T18:51:58.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.436+0000 7f28c1ad0700 1 -- 172.21.15.102:0/4269894913 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f28a406c680 msgr2=0x7f28a406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:58.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.436+0000 7f28c1ad0700 1 --2- 172.21.15.102:0/4269894913 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f28a406c680 0x7f28a406eb30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f28b0006fd0 tx=0x7f28b0008040 comp rx=0 tx=0).stop 2024-03-26T18:51:58.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.436+0000 7f28c1ad0700 1 -- 172.21.15.102:0/4269894913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28bc0ff320 msgr2=0x7f28bc100510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:51:58.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.436+0000 7f28c1ad0700 1 --2- 172.21.15.102:0/4269894913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28bc0ff320 0x7f28bc100510 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f28ac006010 tx=0x7f28ac00be20 comp rx=0 tx=0).stop 2024-03-26T18:51:58.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.437+0000 7f28c1ad0700 1 -- 172.21.15.102:0/4269894913 shutdown_connections 2024-03-26T18:51:58.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.437+0000 7f28c1ad0700 1 --2- 172.21.15.102:0/4269894913 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f28a406c680 0x7f28a406eb30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:58.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.437+0000 7f28c1ad0700 1 --2- 172.21.15.102:0/4269894913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28bc0ff320 0x7f28bc100510 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:58.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.437+0000 7f28c1ad0700 1 --2- 172.21.15.102:0/4269894913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28bc103860 0x7f28bc100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:51:58.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.437+0000 7f28c1ad0700 1 -- 172.21.15.102:0/4269894913 >> 172.21.15.102:0/4269894913 conn(0x7f28bc0748b0 msgr2=0x7f28bc105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:51:58.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.437+0000 7f28c1ad0700 1 -- 172.21.15.102:0/4269894913 shutdown_connections 2024-03-26T18:51:58.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:51:58.437+0000 7f28c1ad0700 1 -- 172.21.15.102:0/4269894913 wait complete. 2024-03-26T18:51:59.098 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T18:51:59.099 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T18:51:59.099 INFO:teuthology.run_tasks:Running task print... 2024-03-26T18:51:59.108 INFO:teuthology.task.print:**** done end installing v18.2.0 cephadm ... 2024-03-26T18:51:59.108 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T18:51:59.133 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T18:51:59.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-26T18:51:59.347 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:51:59.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:59 smithi102 ceph-mon[29002]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:59.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:51:59 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/4269894913' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T18:51:59.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:59 smithi190 ceph-mon[35043]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:51:59.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:51:59 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/4269894913' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T18:52:01.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.146+0000 7f8f454e2700 1 -- 172.21.15.102:0/596579979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f401035a0 msgr2=0x7f8f40105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:01.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.146+0000 7f8f454e2700 1 --2- 172.21.15.102:0/596579979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f401035a0 0x7f8f40105980 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009910 tx=0x7f8f28009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:01.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.146+0000 7f8f454e2700 1 -- 172.21.15.102:0/596579979 shutdown_connections 2024-03-26T18:52:01.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.146+0000 7f8f454e2700 1 --2- 172.21.15.102:0/596579979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f401035a0 0x7f8f40105980 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:01.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.146+0000 7f8f454e2700 1 --2- 172.21.15.102:0/596579979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f40100c80 0x7f8f40103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:01.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.146+0000 7f8f454e2700 1 -- 172.21.15.102:0/596579979 >> 172.21.15.102:0/596579979 conn(0x7f8f400fa600 msgr2=0x7f8f400fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:01.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.146+0000 7f8f454e2700 1 -- 172.21.15.102:0/596579979 shutdown_connections 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.146+0000 7f8f454e2700 1 -- 172.21.15.102:0/596579979 wait complete. 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.146+0000 7f8f454e2700 1 Processor -- start 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f454e2700 1 -- start start 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f454e2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f40100c80 0x7f8f401959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f454e2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f401035a0 0x7f8f40195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f454e2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f40196530 con 0x7f8f401035a0 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f454e2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f40196670 con 0x7f8f40100c80 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f367fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f401035a0 0x7f8f40195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f367fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f401035a0 0x7f8f40195f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59996/0 (socket says 172.21.15.102:59996) 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f367fc700 1 -- 172.21.15.102:0/2966245822 learned_addr learned my addr 172.21.15.102:0/2966245822 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f3effd700 1 --2- 172.21.15.102:0/2966245822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f40100c80 0x7f8f401959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f367fc700 1 -- 172.21.15.102:0/2966245822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f40100c80 msgr2=0x7f8f401959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:01.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f367fc700 1 --2- 172.21.15.102:0/2966245822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f40100c80 0x7f8f401959d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:01.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f367fc700 1 -- 172.21.15.102:0/2966245822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f280095f0 con 0x7f8f401035a0 2024-03-26T18:52:01.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f367fc700 1 --2- 172.21.15.102:0/2966245822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f401035a0 0x7f8f40195f10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8f28006010 tx=0x7f8f2800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:01.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f3cff9700 1 -- 172.21.15.102:0/2966245822 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f2801d070 con 0x7f8f401035a0 2024-03-26T18:52:01.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f454e2700 1 -- 172.21.15.102:0/2966245822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4019b0c0 con 0x7f8f401035a0 2024-03-26T18:52:01.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f3cff9700 1 -- 172.21.15.102:0/2966245822 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8f2800f460 con 0x7f8f401035a0 2024-03-26T18:52:01.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f3cff9700 1 -- 172.21.15.102:0/2966245822 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f28021640 con 0x7f8f401035a0 2024-03-26T18:52:01.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.147+0000 7f8f454e2700 1 -- 172.21.15.102:0/2966245822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4019b630 con 0x7f8f401035a0 2024-03-26T18:52:01.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.148+0000 7f8f454e2700 1 -- 172.21.15.102:0/2966245822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4018fc10 con 0x7f8f401035a0 2024-03-26T18:52:01.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.149+0000 7f8f3cff9700 1 -- 172.21.15.102:0/2966245822 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f8f2800fab0 con 0x7f8f401035a0 2024-03-26T18:52:01.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.149+0000 7f8f3cff9700 1 --2- 172.21.15.102:0/2966245822 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8f2c06c680 0x7f8f2c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:01.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.149+0000 7f8f3cff9700 1 -- 172.21.15.102:0/2966245822 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8f2808ca20 con 0x7f8f401035a0 2024-03-26T18:52:01.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.149+0000 7f8f3effd700 1 --2- 172.21.15.102:0/2966245822 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8f2c06c680 0x7f8f2c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:01.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.150+0000 7f8f3effd700 1 --2- 172.21.15.102:0/2966245822 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8f2c06c680 0x7f8f2c06eb30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8f30006fd0 tx=0x7f8f30008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:01.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.154+0000 7f8f3cff9700 1 -- 172.21.15.102:0/2966245822 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8f2805aed0 con 0x7f8f401035a0 2024-03-26T18:52:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.343+0000 7f8f454e2700 1 -- 172.21.15.102:0/2966245822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f8f40061960 con 0x7f8f401035a0 2024-03-26T18:52:01.336 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:01 smithi102 ceph-mon[29002]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:01.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.347+0000 7f8f3cff9700 1 -- 172.21.15.102:0/2966245822 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f8f280260d0 con 0x7f8f401035a0 2024-03-26T18:52:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.349+0000 7f8f454e2700 1 -- 172.21.15.102:0/2966245822 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8f2c06c680 msgr2=0x7f8f2c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.349+0000 7f8f454e2700 1 --2- 172.21.15.102:0/2966245822 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8f2c06c680 0x7f8f2c06eb30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8f30006fd0 tx=0x7f8f30008040 comp rx=0 tx=0).stop 2024-03-26T18:52:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.349+0000 7f8f454e2700 1 -- 172.21.15.102:0/2966245822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f401035a0 msgr2=0x7f8f40195f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.349+0000 7f8f454e2700 1 --2- 172.21.15.102:0/2966245822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f401035a0 0x7f8f40195f10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8f28006010 tx=0x7f8f2800be20 comp rx=0 tx=0).stop 2024-03-26T18:52:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.349+0000 7f8f454e2700 1 -- 172.21.15.102:0/2966245822 shutdown_connections 2024-03-26T18:52:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.349+0000 7f8f454e2700 1 --2- 172.21.15.102:0/2966245822 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8f2c06c680 0x7f8f2c06eb30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.349+0000 7f8f454e2700 1 --2- 172.21.15.102:0/2966245822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f40100c80 0x7f8f401959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.349+0000 7f8f454e2700 1 --2- 172.21.15.102:0/2966245822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f401035a0 0x7f8f40195f10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.349+0000 7f8f454e2700 1 -- 172.21.15.102:0/2966245822 >> 172.21.15.102:0/2966245822 conn(0x7f8f400fa600 msgr2=0x7f8f400fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:01.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.350+0000 7f8f454e2700 1 -- 172.21.15.102:0/2966245822 shutdown_connections 2024-03-26T18:52:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:01.350+0000 7f8f454e2700 1 -- 172.21.15.102:0/2966245822 wait complete. 2024-03-26T18:52:01.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:01 smithi190 ceph-mon[35043]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:01.933 INFO:teuthology.run_tasks:Running task print... 2024-03-26T18:52:01.942 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-26T18:52:01.942 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T18:52:01.950 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T18:52:01.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-26T18:52:02.155 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:02.615 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:02 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2966245822' entity='client.admin' 2024-03-26T18:52:02.615 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:02 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:52:02.615 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:02 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:02.615 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:02 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:52:02.615 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:02 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:02.616 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:02 smithi102 ceph-mon[29002]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:02.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:02 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2966245822' entity='client.admin' 2024-03-26T18:52:02.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:02 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:52:02.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:02 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:02.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:02 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:52:02.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:02 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:02.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:02 smithi190 ceph-mon[35043]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:03.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.756+0000 7f4e5b72f700 1 -- 172.21.15.102:0/538203530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e540ff050 msgr2=0x7f4e540ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:03.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.756+0000 7f4e5b72f700 1 --2- 172.21.15.102:0/538203530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e540ff050 0x7f4e540ff460 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00b0a0 tx=0x7f4e4c00b3b0 comp rx=0 tx=0).stop 2024-03-26T18:52:03.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.756+0000 7f4e5b72f700 1 -- 172.21.15.102:0/538203530 shutdown_connections 2024-03-26T18:52:03.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.756+0000 7f4e5b72f700 1 --2- 172.21.15.102:0/538203530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e54100250 0x7f4e541006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:03.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.756+0000 7f4e5b72f700 1 --2- 172.21.15.102:0/538203530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e540ff050 0x7f4e540ff460 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:03.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.756+0000 7f4e5b72f700 1 -- 172.21.15.102:0/538203530 >> 172.21.15.102:0/538203530 conn(0x7f4e540fa5e0 msgr2=0x7f4e540fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:03.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.756+0000 7f4e5b72f700 1 -- 172.21.15.102:0/538203530 shutdown_connections 2024-03-26T18:52:03.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.756+0000 7f4e5b72f700 1 -- 172.21.15.102:0/538203530 wait complete. 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.756+0000 7f4e5b72f700 1 Processor -- start 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.756+0000 7f4e5b72f700 1 -- start start 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e5b72f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e540ff050 0x7f4e54102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e5b72f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e54100250 0x7f4e54101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e5b72f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e54101b70 con 0x7f4e540ff050 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e5b72f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e54101cb0 con 0x7f4e54100250 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e594cb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e540ff050 0x7f4e54102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e594cb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e540ff050 0x7f4e54102fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60026/0 (socket says 172.21.15.102:60026) 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e594cb700 1 -- 172.21.15.102:0/2391609003 learned_addr learned my addr 172.21.15.102:0/2391609003 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e58cca700 1 --2- 172.21.15.102:0/2391609003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e54100250 0x7f4e54101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e594cb700 1 -- 172.21.15.102:0/2391609003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e54100250 msgr2=0x7f4e54101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e594cb700 1 --2- 172.21.15.102:0/2391609003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e54100250 0x7f4e54101630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e594cb700 1 -- 172.21.15.102:0/2391609003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c009d00 con 0x7f4e540ff050 2024-03-26T18:52:03.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e594cb700 1 --2- 172.21.15.102:0/2391609003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e540ff050 0x7f4e54102fe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009fd0 tx=0x7f4e4c009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:03.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e527fc700 1 -- 172.21.15.102:0/2391609003 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e4c00e040 con 0x7f4e540ff050 2024-03-26T18:52:03.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e527fc700 1 -- 172.21.15.102:0/2391609003 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4e4c003de0 con 0x7f4e540ff050 2024-03-26T18:52:03.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e5b72f700 1 -- 172.21.15.102:0/2391609003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e54101f30 con 0x7f4e540ff050 2024-03-26T18:52:03.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e527fc700 1 -- 172.21.15.102:0/2391609003 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e4c01d980 con 0x7f4e540ff050 2024-03-26T18:52:03.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.757+0000 7f4e5b72f700 1 -- 172.21.15.102:0/2391609003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e54102420 con 0x7f4e540ff050 2024-03-26T18:52:03.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.758+0000 7f4e5b72f700 1 -- 172.21.15.102:0/2391609003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e54107ba0 con 0x7f4e540ff050 2024-03-26T18:52:03.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.759+0000 7f4e527fc700 1 -- 172.21.15.102:0/2391609003 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f4e4c004040 con 0x7f4e540ff050 2024-03-26T18:52:03.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.759+0000 7f4e527fc700 1 --2- 172.21.15.102:0/2391609003 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4e3c06c680 0x7f4e3c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:03.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.759+0000 7f4e527fc700 1 -- 172.21.15.102:0/2391609003 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4e4c08cf20 con 0x7f4e540ff050 2024-03-26T18:52:03.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.759+0000 7f4e58cca700 1 --2- 172.21.15.102:0/2391609003 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4e3c06c680 0x7f4e3c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:03.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.760+0000 7f4e58cca700 1 --2- 172.21.15.102:0/2391609003 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4e3c06c680 0x7f4e3c06eb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4e48005950 tx=0x7f4e480058e0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:03.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.764+0000 7f4e527fc700 1 -- 172.21.15.102:0/2391609003 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4e4c0bf720 con 0x7f4e540ff050 2024-03-26T18:52:03.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.976+0000 7f4e5b72f700 1 -- 172.21.15.102:0/2391609003 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e54068ae0 con 0x7f4e3c06c680 2024-03-26T18:52:03.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.977+0000 7f4e527fc700 1 -- 172.21.15.102:0/2391609003 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f4e54068ae0 con 0x7f4e3c06c680 2024-03-26T18:52:03.970 INFO:teuthology.orchestra.run.smithi102.stdout:Backend: cephadm 2024-03-26T18:52:03.970 INFO:teuthology.orchestra.run.smithi102.stdout:Available: Yes 2024-03-26T18:52:03.970 INFO:teuthology.orchestra.run.smithi102.stdout:Paused: No 2024-03-26T18:52:03.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.979+0000 7f4e5b72f700 1 -- 172.21.15.102:0/2391609003 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4e3c06c680 msgr2=0x7f4e3c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:03.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.979+0000 7f4e5b72f700 1 --2- 172.21.15.102:0/2391609003 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4e3c06c680 0x7f4e3c06eb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4e48005950 tx=0x7f4e480058e0 comp rx=0 tx=0).stop 2024-03-26T18:52:03.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.979+0000 7f4e5b72f700 1 -- 172.21.15.102:0/2391609003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e540ff050 msgr2=0x7f4e54102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:03.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.979+0000 7f4e5b72f700 1 --2- 172.21.15.102:0/2391609003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e540ff050 0x7f4e54102fe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009fd0 tx=0x7f4e4c009790 comp rx=0 tx=0).stop 2024-03-26T18:52:03.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.979+0000 7f4e5b72f700 1 -- 172.21.15.102:0/2391609003 shutdown_connections 2024-03-26T18:52:03.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.980+0000 7f4e5b72f700 1 --2- 172.21.15.102:0/2391609003 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4e3c06c680 0x7f4e3c06eb30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:03.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.980+0000 7f4e5b72f700 1 --2- 172.21.15.102:0/2391609003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e540ff050 0x7f4e54102fe0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:03.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.980+0000 7f4e5b72f700 1 --2- 172.21.15.102:0/2391609003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e54100250 0x7f4e54101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:03.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.980+0000 7f4e5b72f700 1 -- 172.21.15.102:0/2391609003 >> 172.21.15.102:0/2391609003 conn(0x7f4e540fa5e0 msgr2=0x7f4e540683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:03.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.980+0000 7f4e5b72f700 1 -- 172.21.15.102:0/2391609003 shutdown_connections 2024-03-26T18:52:03.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:03.980+0000 7f4e5b72f700 1 -- 172.21.15.102:0/2391609003 wait complete. 2024-03-26T18:52:04.232 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:03 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:52:04.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:03 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:52:04.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-26T18:52:04.786 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:05.202 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:04 smithi102 ceph-mon[29002]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:05.203 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:04 smithi102 ceph-mon[29002]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:05.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:04 smithi190 ceph-mon[35043]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:05.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:04 smithi190 ceph-mon[35043]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:06.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.300+0000 7fb005c50700 1 -- 172.21.15.102:0/2319443409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb000103500 msgr2=0x7fb0001058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:06.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.300+0000 7fb005c50700 1 --2- 172.21.15.102:0/2319443409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb000103500 0x7fb0001058e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faff4009910 tx=0x7faff4009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:06.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.300+0000 7fb005c50700 1 -- 172.21.15.102:0/2319443409 shutdown_connections 2024-03-26T18:52:06.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.300+0000 7fb005c50700 1 --2- 172.21.15.102:0/2319443409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb000103500 0x7fb0001058e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:06.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.300+0000 7fb005c50700 1 --2- 172.21.15.102:0/2319443409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000100be0 0x7fb000102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:06.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.300+0000 7fb005c50700 1 -- 172.21.15.102:0/2319443409 >> 172.21.15.102:0/2319443409 conn(0x7fb0000fa5e0 msgr2=0x7fb0000fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:06.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.300+0000 7fb005c50700 1 -- 172.21.15.102:0/2319443409 shutdown_connections 2024-03-26T18:52:06.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.300+0000 7fb005c50700 1 -- 172.21.15.102:0/2319443409 wait complete. 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.300+0000 7fb005c50700 1 Processor -- start 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fb005c50700 1 -- start start 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fb005c50700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb000100be0 0x7fb0001959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fb005c50700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000103500 0x7fb000195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fb005c50700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb000196500 con 0x7fb000100be0 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fb005c50700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb000196640 con 0x7fb000103500 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fafff7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb000100be0 0x7fb0001959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fafff7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb000100be0 0x7fb0001959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60046/0 (socket says 172.21.15.102:60046) 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fafff7fe700 1 -- 172.21.15.102:0/126556988 learned_addr learned my addr 172.21.15.102:0/126556988 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7faffeffd700 1 --2- 172.21.15.102:0/126556988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000103500 0x7fb000195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fafff7fe700 1 -- 172.21.15.102:0/126556988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000103500 msgr2=0x7fb000195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fafff7fe700 1 --2- 172.21.15.102:0/126556988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000103500 0x7fb000195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fafff7fe700 1 -- 172.21.15.102:0/126556988 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff40095f0 con 0x7fb000100be0 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7fafff7fe700 1 --2- 172.21.15.102:0/126556988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb000100be0 0x7fb0001959a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faff000b4b0 tx=0x7faff000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7faffcff9700 1 -- 172.21.15.102:0/126556988 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faff0011800 con 0x7fb000100be0 2024-03-26T18:52:06.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.301+0000 7faffcff9700 1 -- 172.21.15.102:0/126556988 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faff0011e10 con 0x7fb000100be0 2024-03-26T18:52:06.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.302+0000 7fb005c50700 1 -- 172.21.15.102:0/126556988 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00019b0f0 con 0x7fb000100be0 2024-03-26T18:52:06.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.302+0000 7faffcff9700 1 -- 172.21.15.102:0/126556988 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faff000f2e0 con 0x7fb000100be0 2024-03-26T18:52:06.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.302+0000 7fb005c50700 1 -- 172.21.15.102:0/126556988 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00019b6c0 con 0x7fb000100be0 2024-03-26T18:52:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.303+0000 7fb005c50700 1 -- 172.21.15.102:0/126556988 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00018fbf0 con 0x7fb000100be0 2024-03-26T18:52:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.303+0000 7faffcff9700 1 -- 172.21.15.102:0/126556988 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7faff0010380 con 0x7fb000100be0 2024-03-26T18:52:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.303+0000 7faffcff9700 1 --2- 172.21.15.102:0/126556988 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fafe806c680 0x7fafe806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.303+0000 7faffcff9700 1 -- 172.21.15.102:0/126556988 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7faff008b090 con 0x7fb000100be0 2024-03-26T18:52:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.303+0000 7faffeffd700 1 --2- 172.21.15.102:0/126556988 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fafe806c680 0x7fafe806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.304+0000 7faffeffd700 1 --2- 172.21.15.102:0/126556988 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fafe806c680 0x7fafe806eb30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faff4006010 tx=0x7faff400b580 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:06.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.308+0000 7faffcff9700 1 -- 172.21.15.102:0/126556988 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7faff0059670 con 0x7fb000100be0 2024-03-26T18:52:06.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.510+0000 7fb005c50700 1 -- 172.21.15.102:0/126556988 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb000061190 con 0x7fafe806c680 2024-03-26T18:52:06.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.519+0000 7faffcff9700 1 -- 172.21.15.102:0/126556988 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2820 (secure 0 0 0) 0x7fb000061190 con 0x7fafe806c680 2024-03-26T18:52:06.511 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (3m) 2m ago 5m 26.5M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (5m) 2m ago 5m 7017k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (4m) 63s ago 4m 6975k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (5m) 2m ago 5m 6665k - 18.2.0 dc2bc1663786 bfac00217c68 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (4m) 63s ago 4m 6656k - 18.2.0 dc2bc1663786 7379f1f6b746 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (3m) 2m ago 4m 80.8M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:9283,8765,8443 running (6m) 2m ago 6m 476M - 18.2.0 dc2bc1663786 61d2bd7101b6 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (4m) 63s ago 4m 439M - 18.2.0 dc2bc1663786 57e44bfc8ede 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (6m) 2m ago 6m 43.3M 2048M 18.2.0 dc2bc1663786 b5abd46c3973 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (4m) 63s ago 4m 39.4M 2048M 18.2.0 dc2bc1663786 6bd913d483f0 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (5m) 2m ago 5m 16.3M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (4m) 63s ago 4m 15.3M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (3m) 2m ago 3m 37.9M 3946M 18.2.0 dc2bc1663786 3e2065fb5fbc 2024-03-26T18:52:06.512 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (2m) 2m ago 2m 38.8M 3946M 18.2.0 dc2bc1663786 0bf773515cc1 2024-03-26T18:52:06.513 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (2m) 2m ago 2m 37.2M 3946M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:52:06.513 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (2m) 63s ago 2m 43.7M 4970M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:52:06.513 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (95s) 63s ago 94s 40.0M 4970M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:52:06.513 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (69s) 63s ago 68s 11.5M 4970M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:52:06.513 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (3m) 2m ago 4m 40.4M - 2.43.0 a07b618ecd1d d0dc2ca7f41f 2024-03-26T18:52:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.521+0000 7fb005c50700 1 -- 172.21.15.102:0/126556988 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fafe806c680 msgr2=0x7fafe806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.521+0000 7fb005c50700 1 --2- 172.21.15.102:0/126556988 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fafe806c680 0x7fafe806eb30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faff4006010 tx=0x7faff400b580 comp rx=0 tx=0).stop 2024-03-26T18:52:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.521+0000 7fb005c50700 1 -- 172.21.15.102:0/126556988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb000100be0 msgr2=0x7fb0001959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.521+0000 7fb005c50700 1 --2- 172.21.15.102:0/126556988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb000100be0 0x7fb0001959a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faff000b4b0 tx=0x7faff000b870 comp rx=0 tx=0).stop 2024-03-26T18:52:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.521+0000 7fb005c50700 1 -- 172.21.15.102:0/126556988 shutdown_connections 2024-03-26T18:52:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.521+0000 7fb005c50700 1 --2- 172.21.15.102:0/126556988 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fafe806c680 0x7fafe806eb30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.521+0000 7fb005c50700 1 --2- 172.21.15.102:0/126556988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb000100be0 0x7fb0001959a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.521+0000 7fb005c50700 1 --2- 172.21.15.102:0/126556988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000103500 0x7fb000195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.521+0000 7fb005c50700 1 -- 172.21.15.102:0/126556988 >> 172.21.15.102:0/126556988 conn(0x7fb0000fa5e0 msgr2=0x7fb0000fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.521+0000 7fb005c50700 1 -- 172.21.15.102:0/126556988 shutdown_connections 2024-03-26T18:52:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:06.521+0000 7fb005c50700 1 -- 172.21.15.102:0/126556988 wait complete. 2024-03-26T18:52:07.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-26T18:52:07.411 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:07.476 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:07 smithi102 ceph-mon[29002]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:07.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:07 smithi190 ceph-mon[35043]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:08.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:08 smithi190 ceph-mon[35043]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:08.536 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:08 smithi102 ceph-mon[29002]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.029+0000 7fa0a536c700 1 -- 172.21.15.102:0/3113633725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a0100280 msgr2=0x7fa0a01006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.029+0000 7fa0a536c700 1 --2- 172.21.15.102:0/3113633725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a0100280 0x7fa0a01006d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa094009910 tx=0x7fa094009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.029+0000 7fa0a536c700 1 -- 172.21.15.102:0/3113633725 shutdown_connections 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.029+0000 7fa0a536c700 1 --2- 172.21.15.102:0/3113633725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a0100280 0x7fa0a01006d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.029+0000 7fa0a536c700 1 --2- 172.21.15.102:0/3113633725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a00ff080 0x7fa0a00ff490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.029+0000 7fa0a536c700 1 -- 172.21.15.102:0/3113633725 >> 172.21.15.102:0/3113633725 conn(0x7fa0a00fa630 msgr2=0x7fa0a00fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.029+0000 7fa0a536c700 1 -- 172.21.15.102:0/3113633725 shutdown_connections 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.029+0000 7fa0a536c700 1 -- 172.21.15.102:0/3113633725 wait complete. 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.030+0000 7fa0a536c700 1 Processor -- start 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.030+0000 7fa0a536c700 1 -- start start 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.030+0000 7fa0a536c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a00ff080 0x7fa0a0195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.030+0000 7fa0a536c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a0100280 0x7fa0a0195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.030+0000 7fa0a536c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a01964f0 con 0x7fa0a0100280 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.030+0000 7fa0a536c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a0196630 con 0x7fa0a00ff080 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.030+0000 7fa09e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a0100280 0x7fa0a0195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.030+0000 7fa09e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a0100280 0x7fa0a0195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39232/0 (socket says 172.21.15.102:39232) 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.030+0000 7fa09e7fc700 1 -- 172.21.15.102:0/1522846200 learned_addr learned my addr 172.21.15.102:0/1522846200 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.030+0000 7fa09effd700 1 --2- 172.21.15.102:0/1522846200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a00ff080 0x7fa0a0195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.031+0000 7fa09e7fc700 1 -- 172.21.15.102:0/1522846200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a00ff080 msgr2=0x7fa0a0195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.031+0000 7fa09e7fc700 1 --2- 172.21.15.102:0/1522846200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a00ff080 0x7fa0a0195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.031+0000 7fa09e7fc700 1 -- 172.21.15.102:0/1522846200 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0940095f0 con 0x7fa0a0100280 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.031+0000 7fa09e7fc700 1 --2- 172.21.15.102:0/1522846200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a0100280 0x7fa0a0195ed0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa09400b5c0 tx=0x7fa09400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.031+0000 7fa087fff700 1 -- 172.21.15.102:0/1522846200 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa09401d070 con 0x7fa0a0100280 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.031+0000 7fa0a536c700 1 -- 172.21.15.102:0/1522846200 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a019b080 con 0x7fa0a0100280 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.031+0000 7fa087fff700 1 -- 172.21.15.102:0/1522846200 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa094022470 con 0x7fa0a0100280 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.031+0000 7fa087fff700 1 -- 172.21.15.102:0/1522846200 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa09400f690 con 0x7fa0a0100280 2024-03-26T18:52:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.031+0000 7fa0a536c700 1 -- 172.21.15.102:0/1522846200 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a019b570 con 0x7fa0a0100280 2024-03-26T18:52:09.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.032+0000 7fa0a536c700 1 -- 172.21.15.102:0/1522846200 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a0107bd0 con 0x7fa0a0100280 2024-03-26T18:52:09.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.033+0000 7fa087fff700 1 -- 172.21.15.102:0/1522846200 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fa0940225e0 con 0x7fa0a0100280 2024-03-26T18:52:09.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.033+0000 7fa087fff700 1 --2- 172.21.15.102:0/1522846200 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa08806c680 0x7fa08806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.033+0000 7fa09effd700 1 --2- 172.21.15.102:0/1522846200 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa08806c680 0x7fa08806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.033+0000 7fa087fff700 1 -- 172.21.15.102:0/1522846200 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa09408cd10 con 0x7fa0a0100280 2024-03-26T18:52:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.033+0000 7fa09effd700 1 --2- 172.21.15.102:0/1522846200 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa08806c680 0x7fa08806eb30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa0a01000e0 tx=0x7fa09000a380 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:09.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.037+0000 7fa087fff700 1 -- 172.21.15.102:0/1522846200 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa09405b1c0 con 0x7fa0a0100280 2024-03-26T18:52:09.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.237+0000 7fa0a536c700 1 -- 172.21.15.102:0/1522846200 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0a0069400 con 0x7fa08806c680 2024-03-26T18:52:09.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.241+0000 7fa087fff700 1 -- 172.21.15.102:0/1522846200 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1310 (secure 0 0 0) 0x7fa0a0069400 con 0x7fa08806c680 2024-03-26T18:52:09.233 INFO:teuthology.orchestra.run.smithi102.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-26T18:52:09.234 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-03-26T18:52:09.234 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter 2/2 2m ago 5m * 2024-03-26T18:52:09.234 INFO:teuthology.orchestra.run.smithi102.stdout:crash 2/2 2m ago 5m * 2024-03-26T18:52:09.234 INFO:teuthology.orchestra.run.smithi102.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-03-26T18:52:09.234 INFO:teuthology.orchestra.run.smithi102.stdout:mgr 2/2 2m ago 6m count:2 2024-03-26T18:52:09.234 INFO:teuthology.orchestra.run.smithi102.stdout:mon 2/2 2m ago 5m smithi102:172.21.15.102=smithi102;smithi190:172.21.15.190=smithi190;count:2 2024-03-26T18:52:09.234 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-03-26T18:52:09.234 INFO:teuthology.orchestra.run.smithi102.stdout:osd 6 2m ago - 2024-03-26T18:52:09.234 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-03-26T18:52:09.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.243+0000 7fa0a536c700 1 -- 172.21.15.102:0/1522846200 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa08806c680 msgr2=0x7fa08806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:09.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.243+0000 7fa0a536c700 1 --2- 172.21.15.102:0/1522846200 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa08806c680 0x7fa08806eb30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa0a01000e0 tx=0x7fa09000a380 comp rx=0 tx=0).stop 2024-03-26T18:52:09.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.243+0000 7fa0a536c700 1 -- 172.21.15.102:0/1522846200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a0100280 msgr2=0x7fa0a0195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:09.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.243+0000 7fa0a536c700 1 --2- 172.21.15.102:0/1522846200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a0100280 0x7fa0a0195ed0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa09400b5c0 tx=0x7fa09400bd70 comp rx=0 tx=0).stop 2024-03-26T18:52:09.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.243+0000 7fa0a536c700 1 -- 172.21.15.102:0/1522846200 shutdown_connections 2024-03-26T18:52:09.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.243+0000 7fa0a536c700 1 --2- 172.21.15.102:0/1522846200 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa08806c680 0x7fa08806eb30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:09.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.243+0000 7fa0a536c700 1 --2- 172.21.15.102:0/1522846200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a00ff080 0x7fa0a0195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:09.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.243+0000 7fa0a536c700 1 --2- 172.21.15.102:0/1522846200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a0100280 0x7fa0a0195ed0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:09.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.243+0000 7fa0a536c700 1 -- 172.21.15.102:0/1522846200 >> 172.21.15.102:0/1522846200 conn(0x7fa0a00fa630 msgr2=0x7fa0a0068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:09.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.243+0000 7fa0a536c700 1 -- 172.21.15.102:0/1522846200 shutdown_connections 2024-03-26T18:52:09.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:09.243+0000 7fa0a536c700 1 -- 172.21.15.102:0/1522846200 wait complete. 2024-03-26T18:52:09.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:09 smithi190 ceph-mon[35043]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:09.576 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:09 smithi102 ceph-mon[29002]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:09.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-26T18:52:10.117 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:10.464 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:10 smithi102 ceph-mon[29002]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:10.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:10 smithi190 ceph-mon[35043]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:11.280 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:11 smithi102 ceph-mon[29002]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:11.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:11 smithi190 ceph-mon[35043]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.662+0000 7f4eea131700 1 -- 172.21.15.102:0/3256678702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee4103580 msgr2=0x7f4ee4105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.662+0000 7f4eea131700 1 --2- 172.21.15.102:0/3256678702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee4103580 0x7f4ee4105960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4ed800b0a0 tx=0x7f4ed800b3b0 comp rx=0 tx=0).stop 2024-03-26T18:52:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.662+0000 7f4eea131700 1 -- 172.21.15.102:0/3256678702 shutdown_connections 2024-03-26T18:52:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.662+0000 7f4eea131700 1 --2- 172.21.15.102:0/3256678702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee4103580 0x7f4ee4105960 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.662+0000 7f4eea131700 1 --2- 172.21.15.102:0/3256678702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee4100c60 0x7f4ee4103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.662+0000 7f4eea131700 1 -- 172.21.15.102:0/3256678702 >> 172.21.15.102:0/3256678702 conn(0x7f4ee40fa600 msgr2=0x7f4ee40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.662+0000 7f4eea131700 1 -- 172.21.15.102:0/3256678702 shutdown_connections 2024-03-26T18:52:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.662+0000 7f4eea131700 1 -- 172.21.15.102:0/3256678702 wait complete. 2024-03-26T18:52:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.662+0000 7f4eea131700 1 Processor -- start 2024-03-26T18:52:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4eea131700 1 -- start start 2024-03-26T18:52:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4eea131700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee4100c60 0x7f4ee4195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4eea131700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee4103580 0x7f4ee4195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4eea131700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee41964b0 con 0x7f4ee4100c60 2024-03-26T18:52:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4eea131700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee41965f0 con 0x7f4ee4103580 2024-03-26T18:52:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4ee37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee4100c60 0x7f4ee4195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4ee37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee4100c60 0x7f4ee4195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39238/0 (socket says 172.21.15.102:39238) 2024-03-26T18:52:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4ee37fe700 1 -- 172.21.15.102:0/1333963904 learned_addr learned my addr 172.21.15.102:0/1333963904 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4ee2ffd700 1 --2- 172.21.15.102:0/1333963904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee4103580 0x7f4ee4195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4ee37fe700 1 -- 172.21.15.102:0/1333963904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee4103580 msgr2=0x7f4ee4195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4ee37fe700 1 --2- 172.21.15.102:0/1333963904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee4103580 0x7f4ee4195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4ee37fe700 1 -- 172.21.15.102:0/1333963904 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed8009d00 con 0x7f4ee4100c60 2024-03-26T18:52:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4ee37fe700 1 --2- 172.21.15.102:0/1333963904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee4100c60 0x7f4ee4195950 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00d8d0 tx=0x7f4edc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4ee0ff9700 1 -- 172.21.15.102:0/1333963904 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4edc0098a0 con 0x7f4ee4100c60 2024-03-26T18:52:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4ee0ff9700 1 -- 172.21.15.102:0/1333963904 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4edc010460 con 0x7f4ee4100c60 2024-03-26T18:52:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4ee0ff9700 1 -- 172.21.15.102:0/1333963904 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4edc00f5f0 con 0x7f4ee4100c60 2024-03-26T18:52:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.663+0000 7f4eea131700 1 -- 172.21.15.102:0/1333963904 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee419b0a0 con 0x7f4ee4100c60 2024-03-26T18:52:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.664+0000 7f4eea131700 1 -- 172.21.15.102:0/1333963904 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee419b670 con 0x7f4ee4100c60 2024-03-26T18:52:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.664+0000 7f4eea131700 1 -- 172.21.15.102:0/1333963904 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee418fb70 con 0x7f4ee4100c60 2024-03-26T18:52:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.665+0000 7f4ee0ff9700 1 -- 172.21.15.102:0/1333963904 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f4edc009a00 con 0x7f4ee4100c60 2024-03-26T18:52:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.665+0000 7f4ee0ff9700 1 --2- 172.21.15.102:0/1333963904 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4ecc06c680 0x7f4ecc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.665+0000 7f4ee2ffd700 1 --2- 172.21.15.102:0/1333963904 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4ecc06c680 0x7f4ecc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:11.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.666+0000 7f4ee0ff9700 1 -- 172.21.15.102:0/1333963904 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4edc08b370 con 0x7f4ee4100c60 2024-03-26T18:52:11.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.666+0000 7f4ee2ffd700 1 --2- 172.21.15.102:0/1333963904 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4ecc06c680 0x7f4ecc06eb30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009fd0 tx=0x7f4ed80060c0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:11.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.670+0000 7f4ee0ff9700 1 -- 172.21.15.102:0/1333963904 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4edc0598a0 con 0x7f4ee4100c60 2024-03-26T18:52:11.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.863+0000 7f4eea131700 1 -- 172.21.15.102:0/1333963904 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ee4061190 con 0x7f4ecc06c680 2024-03-26T18:52:11.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.865+0000 7f4ee0ff9700 1 -- 172.21.15.102:0/1333963904 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f4ee4061190 con 0x7f4ecc06c680 2024-03-26T18:52:11.857 INFO:teuthology.orchestra.run.smithi102.stdout:HOST ADDR LABELS STATUS 2024-03-26T18:52:11.857 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 172.21.15.102 2024-03-26T18:52:11.857 INFO:teuthology.orchestra.run.smithi102.stdout:smithi190 172.21.15.190 2024-03-26T18:52:11.857 INFO:teuthology.orchestra.run.smithi102.stdout:2 hosts in cluster 2024-03-26T18:52:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.867+0000 7f4eea131700 1 -- 172.21.15.102:0/1333963904 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4ecc06c680 msgr2=0x7f4ecc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.867+0000 7f4eea131700 1 --2- 172.21.15.102:0/1333963904 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4ecc06c680 0x7f4ecc06eb30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009fd0 tx=0x7f4ed80060c0 comp rx=0 tx=0).stop 2024-03-26T18:52:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.867+0000 7f4eea131700 1 -- 172.21.15.102:0/1333963904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee4100c60 msgr2=0x7f4ee4195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.867+0000 7f4eea131700 1 --2- 172.21.15.102:0/1333963904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee4100c60 0x7f4ee4195950 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00d8d0 tx=0x7f4edc00dbe0 comp rx=0 tx=0).stop 2024-03-26T18:52:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.867+0000 7f4eea131700 1 -- 172.21.15.102:0/1333963904 shutdown_connections 2024-03-26T18:52:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.867+0000 7f4eea131700 1 --2- 172.21.15.102:0/1333963904 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4ecc06c680 0x7f4ecc06eb30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.867+0000 7f4eea131700 1 --2- 172.21.15.102:0/1333963904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee4100c60 0x7f4ee4195950 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.867+0000 7f4eea131700 1 --2- 172.21.15.102:0/1333963904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee4103580 0x7f4ee4195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.867+0000 7f4eea131700 1 -- 172.21.15.102:0/1333963904 >> 172.21.15.102:0/1333963904 conn(0x7f4ee40fa600 msgr2=0x7f4ee40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.868+0000 7f4eea131700 1 -- 172.21.15.102:0/1333963904 shutdown_connections 2024-03-26T18:52:11.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:11.868+0000 7f4eea131700 1 -- 172.21.15.102:0/1333963904 wait complete. 2024-03-26T18:52:12.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-26T18:52:12.744 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:13.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:13 smithi190 ceph-mon[35043]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:13.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:13 smithi190 ceph-mon[35043]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:13.546 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:13 smithi102 ceph-mon[29002]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:13.546 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:13 smithi102 ceph-mon[29002]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:14.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.434+0000 7f3f93ba4700 1 -- 172.21.15.102:0/814347272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f8c100170 msgr2=0x7f3f8c100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:14.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.434+0000 7f3f93ba4700 1 --2- 172.21.15.102:0/814347272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f8c100170 0x7f3f8c100580 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009910 tx=0x7f3f88009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.434+0000 7f3f93ba4700 1 -- 172.21.15.102:0/814347272 shutdown_connections 2024-03-26T18:52:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.434+0000 7f3f93ba4700 1 --2- 172.21.15.102:0/814347272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f8c101370 0x7f3f8c1017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.434+0000 7f3f93ba4700 1 --2- 172.21.15.102:0/814347272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f8c100170 0x7f3f8c100580 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.434+0000 7f3f93ba4700 1 -- 172.21.15.102:0/814347272 >> 172.21.15.102:0/814347272 conn(0x7f3f8c0fb6e0 msgr2=0x7f3f8c0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.434+0000 7f3f93ba4700 1 -- 172.21.15.102:0/814347272 shutdown_connections 2024-03-26T18:52:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.434+0000 7f3f93ba4700 1 -- 172.21.15.102:0/814347272 wait complete. 2024-03-26T18:52:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.434+0000 7f3f93ba4700 1 Processor -- start 2024-03-26T18:52:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f93ba4700 1 -- start start 2024-03-26T18:52:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f93ba4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f8c100170 0x7f3f8c074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f93ba4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f8c101370 0x7f3f8c073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f93ba4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c073680 con 0x7f3f8c101370 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f93ba4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0737c0 con 0x7f3f8c100170 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f9113f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f8c101370 0x7f3f8c073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f9113f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f8c101370 0x7f3f8c073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39248/0 (socket says 172.21.15.102:39248) 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f9113f700 1 -- 172.21.15.102:0/654068604 learned_addr learned my addr 172.21.15.102:0/654068604 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f91940700 1 --2- 172.21.15.102:0/654068604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f8c100170 0x7f3f8c074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f9113f700 1 -- 172.21.15.102:0/654068604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f8c100170 msgr2=0x7f3f8c074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f9113f700 1 --2- 172.21.15.102:0/654068604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f8c100170 0x7f3f8c074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f9113f700 1 -- 172.21.15.102:0/654068604 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f880095f0 con 0x7f3f8c101370 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f9113f700 1 --2- 172.21.15.102:0/654068604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f8c101370 0x7f3f8c073140 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00b4b0 tx=0x7f3f7c00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f82ffd700 1 -- 172.21.15.102:0/654068604 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f7c011800 con 0x7f3f8c101370 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f82ffd700 1 -- 172.21.15.102:0/654068604 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3f7c011e10 con 0x7f3f8c101370 2024-03-26T18:52:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.435+0000 7f3f82ffd700 1 -- 172.21.15.102:0/654068604 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f7c00f2e0 con 0x7f3f8c101370 2024-03-26T18:52:14.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.436+0000 7f3f93ba4700 1 -- 172.21.15.102:0/654068604 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8c073aa0 con 0x7f3f8c101370 2024-03-26T18:52:14.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.436+0000 7f3f93ba4700 1 -- 172.21.15.102:0/654068604 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8c073fa0 con 0x7f3f8c101370 2024-03-26T18:52:14.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.436+0000 7f3f93ba4700 1 -- 172.21.15.102:0/654068604 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8c108d40 con 0x7f3f8c101370 2024-03-26T18:52:14.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.437+0000 7f3f82ffd700 1 -- 172.21.15.102:0/654068604 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f3f7c010380 con 0x7f3f8c101370 2024-03-26T18:52:14.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.437+0000 7f3f82ffd700 1 --2- 172.21.15.102:0/654068604 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f7806c680 0x7f3f7806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:14.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.437+0000 7f3f82ffd700 1 -- 172.21.15.102:0/654068604 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3f7c08b110 con 0x7f3f8c101370 2024-03-26T18:52:14.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.437+0000 7f3f91940700 1 --2- 172.21.15.102:0/654068604 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f7806c680 0x7f3f7806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:14.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.438+0000 7f3f91940700 1 --2- 172.21.15.102:0/654068604 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f7806c680 0x7f3f7806eb30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3f88006010 tx=0x7f3f8800b540 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:14.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.442+0000 7f3f82ffd700 1 -- 172.21.15.102:0/654068604 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3f7c0595c0 con 0x7f3f8c101370 2024-03-26T18:52:14.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.637+0000 7f3f93ba4700 1 -- 172.21.15.102:0/654068604 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f8c105cc0 con 0x7f3f7806c680 2024-03-26T18:52:14.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.639+0000 7f3f82ffd700 1 -- 172.21.15.102:0/654068604 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f3f8c105cc0 con 0x7f3f7806c680 2024-03-26T18:52:14.631 INFO:teuthology.orchestra.run.smithi102.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-26T18:52:14.631 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VK400BGN 372G No 2m ago LVM detected, locked 2024-03-26T18:52:14.631 INFO:teuthology.orchestra.run.smithi102.stdout:smithi190 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT63210012400BGN 372G No 68s ago LVM detected, locked 2024-03-26T18:52:14.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.641+0000 7f3f93ba4700 1 -- 172.21.15.102:0/654068604 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f7806c680 msgr2=0x7f3f7806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:14.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.641+0000 7f3f93ba4700 1 --2- 172.21.15.102:0/654068604 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f7806c680 0x7f3f7806eb30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3f88006010 tx=0x7f3f8800b540 comp rx=0 tx=0).stop 2024-03-26T18:52:14.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.641+0000 7f3f93ba4700 1 -- 172.21.15.102:0/654068604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f8c101370 msgr2=0x7f3f8c073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:14.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.641+0000 7f3f93ba4700 1 --2- 172.21.15.102:0/654068604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f8c101370 0x7f3f8c073140 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00b4b0 tx=0x7f3f7c00b7c0 comp rx=0 tx=0).stop 2024-03-26T18:52:14.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.641+0000 7f3f93ba4700 1 -- 172.21.15.102:0/654068604 shutdown_connections 2024-03-26T18:52:14.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.641+0000 7f3f93ba4700 1 --2- 172.21.15.102:0/654068604 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f7806c680 0x7f3f7806eb30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:14.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.641+0000 7f3f93ba4700 1 --2- 172.21.15.102:0/654068604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f8c100170 0x7f3f8c074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:14.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.641+0000 7f3f93ba4700 1 --2- 172.21.15.102:0/654068604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f8c101370 0x7f3f8c073140 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:14.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.641+0000 7f3f93ba4700 1 -- 172.21.15.102:0/654068604 >> 172.21.15.102:0/654068604 conn(0x7f3f8c0fb6e0 msgr2=0x7f3f8c1045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:14.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.641+0000 7f3f93ba4700 1 -- 172.21.15.102:0/654068604 shutdown_connections 2024-03-26T18:52:14.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:14.641+0000 7f3f93ba4700 1 -- 172.21.15.102:0/654068604 wait complete. 2024-03-26T18:52:15.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:15 smithi102 ceph-mon[29002]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:15.459 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T18:52:15.470 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T18:52:15.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-26T18:52:15.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:15 smithi190 ceph-mon[35043]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:15.676 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:16.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:16 smithi102 ceph-mon[29002]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:16.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:16 smithi190 ceph-mon[35043]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.266+0000 7fae786cc700 1 -- 172.21.15.102:0/1465080685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70100270 msgr2=0x7fae701006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.266+0000 7fae786cc700 1 --2- 172.21.15.102:0/1465080685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70100270 0x7fae701006c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009910 tx=0x7fae6c009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.266+0000 7fae786cc700 1 -- 172.21.15.102:0/1465080685 shutdown_connections 2024-03-26T18:52:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.266+0000 7fae786cc700 1 --2- 172.21.15.102:0/1465080685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70100270 0x7fae701006c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.266+0000 7fae786cc700 1 --2- 172.21.15.102:0/1465080685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae700ff070 0x7fae700ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.266+0000 7fae786cc700 1 -- 172.21.15.102:0/1465080685 >> 172.21.15.102:0/1465080685 conn(0x7fae700fa600 msgr2=0x7fae700fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.266+0000 7fae786cc700 1 -- 172.21.15.102:0/1465080685 shutdown_connections 2024-03-26T18:52:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.266+0000 7fae786cc700 1 -- 172.21.15.102:0/1465080685 wait complete. 2024-03-26T18:52:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae786cc700 1 Processor -- start 2024-03-26T18:52:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae786cc700 1 -- start start 2024-03-26T18:52:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae786cc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae700ff070 0x7fae70195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae786cc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae70100270 0x7fae70195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae786cc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae701964e0 con 0x7fae700ff070 2024-03-26T18:52:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae786cc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae70196620 con 0x7fae70100270 2024-03-26T18:52:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae76468700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae700ff070 0x7fae70195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae76468700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae700ff070 0x7fae70195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39264/0 (socket says 172.21.15.102:39264) 2024-03-26T18:52:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae76468700 1 -- 172.21.15.102:0/3264034155 learned_addr learned my addr 172.21.15.102:0/3264034155 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae75c67700 1 --2- 172.21.15.102:0/3264034155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae70100270 0x7fae70195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae76468700 1 -- 172.21.15.102:0/3264034155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae70100270 msgr2=0x7fae70195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae76468700 1 --2- 172.21.15.102:0/3264034155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae70100270 0x7fae70195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.267+0000 7fae76468700 1 -- 172.21.15.102:0/3264034155 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6c0095f0 con 0x7fae700ff070 2024-03-26T18:52:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.268+0000 7fae76468700 1 --2- 172.21.15.102:0/3264034155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae700ff070 0x7fae70195980 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fae6000d900 tx=0x7fae6000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.268+0000 7fae677fe700 1 -- 172.21.15.102:0/3264034155 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae600041f0 con 0x7fae700ff070 2024-03-26T18:52:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.268+0000 7fae677fe700 1 -- 172.21.15.102:0/3264034155 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fae60004350 con 0x7fae700ff070 2024-03-26T18:52:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.268+0000 7fae786cc700 1 -- 172.21.15.102:0/3264034155 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7019b0d0 con 0x7fae700ff070 2024-03-26T18:52:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.268+0000 7fae677fe700 1 -- 172.21.15.102:0/3264034155 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae60003d20 con 0x7fae700ff070 2024-03-26T18:52:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.268+0000 7fae786cc700 1 -- 172.21.15.102:0/3264034155 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7019b620 con 0x7fae700ff070 2024-03-26T18:52:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.269+0000 7fae786cc700 1 -- 172.21.15.102:0/3264034155 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae70107bc0 con 0x7fae700ff070 2024-03-26T18:52:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.269+0000 7fae677fe700 1 -- 172.21.15.102:0/3264034155 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fae60009730 con 0x7fae700ff070 2024-03-26T18:52:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.270+0000 7fae677fe700 1 --2- 172.21.15.102:0/3264034155 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fae5c06c680 0x7fae5c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.270+0000 7fae677fe700 1 -- 172.21.15.102:0/3264034155 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fae6008b350 con 0x7fae700ff070 2024-03-26T18:52:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.270+0000 7fae75c67700 1 --2- 172.21.15.102:0/3264034155 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fae5c06c680 0x7fae5c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.270+0000 7fae75c67700 1 --2- 172.21.15.102:0/3264034155 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fae5c06c680 0x7fae5c06eb30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fae6c006010 tx=0x7fae6c00b540 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:17.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.274+0000 7fae677fe700 1 -- 172.21.15.102:0/3264034155 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fae60059880 con 0x7fae700ff070 2024-03-26T18:52:17.267 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:17 smithi102 ceph-mon[29002]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:17.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:17.489+0000 7fae786cc700 1 -- 172.21.15.102:0/3264034155 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fae70069460 con 0x7fae5c06c680 2024-03-26T18:52:17.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:17 smithi190 ceph-mon[35043]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:18.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:18 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-26T18:52:18.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:18 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-26T18:52:19.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.271+0000 7fae677fe700 1 -- 172.21.15.102:0/3264034155 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fae70069460 con 0x7fae5c06c680 2024-03-26T18:52:19.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.274+0000 7fae786cc700 1 -- 172.21.15.102:0/3264034155 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fae5c06c680 msgr2=0x7fae5c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:19.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.274+0000 7fae786cc700 1 --2- 172.21.15.102:0/3264034155 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fae5c06c680 0x7fae5c06eb30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fae6c006010 tx=0x7fae6c00b540 comp rx=0 tx=0).stop 2024-03-26T18:52:19.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.274+0000 7fae786cc700 1 -- 172.21.15.102:0/3264034155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae700ff070 msgr2=0x7fae70195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:19.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.274+0000 7fae786cc700 1 --2- 172.21.15.102:0/3264034155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae700ff070 0x7fae70195980 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fae6000d900 tx=0x7fae6000dc10 comp rx=0 tx=0).stop 2024-03-26T18:52:19.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.274+0000 7fae786cc700 1 -- 172.21.15.102:0/3264034155 shutdown_connections 2024-03-26T18:52:19.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.274+0000 7fae786cc700 1 --2- 172.21.15.102:0/3264034155 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fae5c06c680 0x7fae5c06eb30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:19.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.274+0000 7fae786cc700 1 --2- 172.21.15.102:0/3264034155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae700ff070 0x7fae70195980 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:19.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.274+0000 7fae786cc700 1 --2- 172.21.15.102:0/3264034155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae70100270 0x7fae70195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:19.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.274+0000 7fae786cc700 1 -- 172.21.15.102:0/3264034155 >> 172.21.15.102:0/3264034155 conn(0x7fae700fa600 msgr2=0x7fae70068d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:19.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.274+0000 7fae786cc700 1 -- 172.21.15.102:0/3264034155 shutdown_connections 2024-03-26T18:52:19.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:19.274+0000 7fae786cc700 1 -- 172.21.15.102:0/3264034155 wait complete. 2024-03-26T18:52:19.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:19 smithi190 ceph-mon[35043]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:19.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:19 smithi190 ceph-mon[35043]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:19.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-26T18:52:19.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:19 smithi190 ceph-mon[35043]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T18:52:19.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-26T18:52:19.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:52:19.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:19 smithi102 ceph-mon[29002]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:52:19.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:19 smithi102 ceph-mon[29002]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:19.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-26T18:52:19.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:19 smithi102 ceph-mon[29002]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T18:52:19.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-26T18:52:19.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:52:19.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:19 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102[28998]: 2024-03-26T18:52:19.250+0000 7f0bc4d5d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T18:52:19.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-26T18:52:20.118 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:20.521 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-26T18:52:20.521 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T18:52:20.521 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-26T18:52:20.521 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T18:52:20.521 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-26T18:52:20.521 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T18:52:20.521 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-26T18:52:20.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: fsmap cephfs:0 2024-03-26T18:52:20.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: Saving service mds.cephfs spec with placement count:4 2024-03-26T18:52:20.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:20.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:52:20.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:20.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:52:20.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:20.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.tvsqaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:52:20.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.tvsqaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T18:52:20.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:20 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:20.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-26T18:52:20.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T18:52:20.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-26T18:52:20.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T18:52:20.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-26T18:52:20.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T18:52:20.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-26T18:52:20.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: fsmap cephfs:0 2024-03-26T18:52:20.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: Saving service mds.cephfs spec with placement count:4 2024-03-26T18:52:20.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:20.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:52:20.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:20.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:52:20.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:20.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.tvsqaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:52:20.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.tvsqaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T18:52:20.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:20 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:21.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:21 smithi190 ceph-mon[35043]: pgmap v175: 65 pgs: 11 creating+peering, 1 active+clean, 53 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:21.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:21 smithi190 ceph-mon[35043]: Deploying daemon mds.cephfs.smithi102.tvsqaa on smithi102 2024-03-26T18:52:21.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:21 smithi190 ceph-mon[35043]: osdmap e45: 6 total, 6 up, 6 in 2024-03-26T18:52:21.565 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:21 smithi102 ceph-mon[29002]: pgmap v175: 65 pgs: 11 creating+peering, 1 active+clean, 53 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:21.565 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:21 smithi102 ceph-mon[29002]: Deploying daemon mds.cephfs.smithi102.tvsqaa on smithi102 2024-03-26T18:52:21.565 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:21 smithi102 ceph-mon[29002]: osdmap e45: 6 total, 6 up, 6 in 2024-03-26T18:52:22.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.159+0000 7fb75ea17700 1 -- 172.21.15.102:0/263543890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb758103500 msgr2=0x7fb7581058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:22.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.159+0000 7fb75ea17700 1 --2- 172.21.15.102:0/263543890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb758103500 0x7fb7581058e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb74c009910 tx=0x7fb74c009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:22.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.159+0000 7fb75ea17700 1 -- 172.21.15.102:0/263543890 shutdown_connections 2024-03-26T18:52:22.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.159+0000 7fb75ea17700 1 --2- 172.21.15.102:0/263543890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb758103500 0x7fb7581058e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:22.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.159+0000 7fb75ea17700 1 --2- 172.21.15.102:0/263543890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb758100be0 0x7fb758102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:22.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.159+0000 7fb75ea17700 1 -- 172.21.15.102:0/263543890 >> 172.21.15.102:0/263543890 conn(0x7fb7580fa5e0 msgr2=0x7fb7580fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:22.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.160+0000 7fb75ea17700 1 -- 172.21.15.102:0/263543890 shutdown_connections 2024-03-26T18:52:22.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.160+0000 7fb75ea17700 1 -- 172.21.15.102:0/263543890 wait complete. 2024-03-26T18:52:22.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.160+0000 7fb75ea17700 1 Processor -- start 2024-03-26T18:52:22.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.160+0000 7fb75ea17700 1 -- start start 2024-03-26T18:52:22.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb75ea17700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb758100be0 0x7fb758195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:22.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb75ea17700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb758103500 0x7fb758195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb75ea17700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb758196490 con 0x7fb758100be0 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb75ea17700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7581965d0 con 0x7fb758103500 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb757fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb758100be0 0x7fb758195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb757fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb758100be0 0x7fb758195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55926/0 (socket says 172.21.15.102:55926) 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb757fff700 1 -- 172.21.15.102:0/248108446 learned_addr learned my addr 172.21.15.102:0/248108446 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb7577fe700 1 --2- 172.21.15.102:0/248108446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb758103500 0x7fb758195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb757fff700 1 -- 172.21.15.102:0/248108446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb758103500 msgr2=0x7fb758195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb757fff700 1 --2- 172.21.15.102:0/248108446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb758103500 0x7fb758195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb757fff700 1 -- 172.21.15.102:0/248108446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0095f0 con 0x7fb758100be0 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb757fff700 1 --2- 172.21.15.102:0/248108446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb758100be0 0x7fb758195930 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb74800d8d0 tx=0x7fb74800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.161+0000 7fb7557fa700 1 -- 172.21.15.102:0/248108446 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7480098a0 con 0x7fb758100be0 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.162+0000 7fb75ea17700 1 -- 172.21.15.102:0/248108446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75819b080 con 0x7fb758100be0 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.162+0000 7fb7557fa700 1 -- 172.21.15.102:0/248108446 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb748010460 con 0x7fb758100be0 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.162+0000 7fb7557fa700 1 -- 172.21.15.102:0/248108446 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb74800f5f0 con 0x7fb758100be0 2024-03-26T18:52:22.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.162+0000 7fb75ea17700 1 -- 172.21.15.102:0/248108446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75819b5d0 con 0x7fb758100be0 2024-03-26T18:52:22.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.162+0000 7fb75ea17700 1 -- 172.21.15.102:0/248108446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75818fb70 con 0x7fb758100be0 2024-03-26T18:52:22.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.163+0000 7fb7557fa700 1 -- 172.21.15.102:0/248108446 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fb748009a00 con 0x7fb758100be0 2024-03-26T18:52:22.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.163+0000 7fb7557fa700 1 --2- 172.21.15.102:0/248108446 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb74006c680 0x7fb74006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:22.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.164+0000 7fb7557fa700 1 -- 172.21.15.102:0/248108446 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fb74808b760 con 0x7fb758100be0 2024-03-26T18:52:22.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.164+0000 7fb7577fe700 1 --2- 172.21.15.102:0/248108446 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb74006c680 0x7fb74006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:22.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.164+0000 7fb7577fe700 1 --2- 172.21.15.102:0/248108446 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb74006c680 0x7fb74006eb30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb74c006010 tx=0x7fb74c00b560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:22.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.172+0000 7fb7557fa700 1 -- 172.21.15.102:0/248108446 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb7480598d0 con 0x7fb758100be0 2024-03-26T18:52:22.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.418+0000 7fb75ea17700 1 -- 172.21.15.102:0/248108446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb758061960 con 0x7fb758100be0 2024-03-26T18:52:22.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.418+0000 7fb7557fa700 1 -- 172.21.15.102:0/248108446 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7fb748059460 con 0x7fb758100be0 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:e2 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 2 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:52:19.251466+0000 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:52:22.411 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 0 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 1 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:in 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:up {} 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:52:22.413 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:52:22.413 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 0 2024-03-26T18:52:22.413 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:52:22.413 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:52:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.421+0000 7fb75ea17700 1 -- 172.21.15.102:0/248108446 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb74006c680 msgr2=0x7fb74006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.421+0000 7fb75ea17700 1 --2- 172.21.15.102:0/248108446 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb74006c680 0x7fb74006eb30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb74c006010 tx=0x7fb74c00b560 comp rx=0 tx=0).stop 2024-03-26T18:52:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.421+0000 7fb75ea17700 1 -- 172.21.15.102:0/248108446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb758100be0 msgr2=0x7fb758195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.421+0000 7fb75ea17700 1 --2- 172.21.15.102:0/248108446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb758100be0 0x7fb758195930 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb74800d8d0 tx=0x7fb74800dbe0 comp rx=0 tx=0).stop 2024-03-26T18:52:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.421+0000 7fb75ea17700 1 -- 172.21.15.102:0/248108446 shutdown_connections 2024-03-26T18:52:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.421+0000 7fb75ea17700 1 --2- 172.21.15.102:0/248108446 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb74006c680 0x7fb74006eb30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.421+0000 7fb75ea17700 1 --2- 172.21.15.102:0/248108446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb758100be0 0x7fb758195930 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.421+0000 7fb75ea17700 1 --2- 172.21.15.102:0/248108446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb758103500 0x7fb758195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.421+0000 7fb75ea17700 1 -- 172.21.15.102:0/248108446 >> 172.21.15.102:0/248108446 conn(0x7fb7580fa5e0 msgr2=0x7fb7580fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.421+0000 7fb75ea17700 1 -- 172.21.15.102:0/248108446 shutdown_connections 2024-03-26T18:52:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:22.421+0000 7fb75ea17700 1 -- 172.21.15.102:0/248108446 wait complete. 2024-03-26T18:52:22.415 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 2 2024-03-26T18:52:22.424 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:22 smithi102 ceph-mon[29002]: osdmap e46: 6 total, 6 up, 6 in 2024-03-26T18:52:22.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:22 smithi190 ceph-mon[35043]: osdmap e46: 6 total, 6 up, 6 in 2024-03-26T18:52:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:23 smithi102 ceph-mon[29002]: pgmap v178: 65 pgs: 8 creating+activating, 11 creating+peering, 18 active+clean, 28 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:23 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/248108446' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:52:23.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:23 smithi190 ceph-mon[35043]: pgmap v178: 65 pgs: 8 creating+activating, 11 creating+peering, 18 active+clean, 28 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:23.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:23 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/248108446' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:52:23.601 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T18:52:23.611 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T18:52:23.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-03-26T18:52:24.252 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:25.357 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: pgmap v179: 65 pgs: 8 creating+activating, 8 creating+peering, 39 active+clean, 10 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.102:6826/2550866692,v1:172.21.15.102:6827/2550866692] up:boot 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: daemon mds.cephfs.smithi102.tvsqaa assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: Cluster is now healthy 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: fsmap cephfs:0 1 up:standby 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:creating} 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.vykizb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:52:25.358 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.vykizb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T18:52:25.359 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:25.359 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:25 smithi102 ceph-mon[29002]: daemon mds.cephfs.smithi102.tvsqaa is now active in filesystem cephfs as rank 0 2024-03-26T18:52:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: pgmap v179: 65 pgs: 8 creating+activating, 8 creating+peering, 39 active+clean, 10 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:52:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.102:6826/2550866692,v1:172.21.15.102:6827/2550866692] up:boot 2024-03-26T18:52:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: daemon mds.cephfs.smithi102.tvsqaa assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-26T18:52:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T18:52:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-26T18:52:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: Cluster is now healthy 2024-03-26T18:52:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: fsmap cephfs:0 1 up:standby 2024-03-26T18:52:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T18:52:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:creating} 2024-03-26T18:52:25.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:25.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.vykizb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:52:25.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.vykizb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T18:52:25.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:25.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:25 smithi190 ceph-mon[35043]: daemon mds.cephfs.smithi102.tvsqaa is now active in filesystem cephfs as rank 0 2024-03-26T18:52:25.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.864+0000 7f4fcd970700 1 -- 172.21.15.102:0/3545038655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc8100260 msgr2=0x7f4fc81006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:25.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.864+0000 7f4fcd970700 1 --2- 172.21.15.102:0/3545038655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc8100260 0x7f4fc81006b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc009910 tx=0x7f4fbc009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:25.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.864+0000 7f4fcd970700 1 -- 172.21.15.102:0/3545038655 shutdown_connections 2024-03-26T18:52:25.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.864+0000 7f4fcd970700 1 --2- 172.21.15.102:0/3545038655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc8100260 0x7f4fc81006b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:25.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.864+0000 7f4fcd970700 1 --2- 172.21.15.102:0/3545038655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fc80ff060 0x7f4fc80ff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:25.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.864+0000 7f4fcd970700 1 -- 172.21.15.102:0/3545038655 >> 172.21.15.102:0/3545038655 conn(0x7f4fc80fa610 msgr2=0x7f4fc80fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:25.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.864+0000 7f4fcd970700 1 -- 172.21.15.102:0/3545038655 shutdown_connections 2024-03-26T18:52:25.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.864+0000 7f4fcd970700 1 -- 172.21.15.102:0/3545038655 wait complete. 2024-03-26T18:52:25.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fcd970700 1 Processor -- start 2024-03-26T18:52:25.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fcd970700 1 -- start start 2024-03-26T18:52:25.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fcd970700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc80ff060 0x7f4fc8195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:25.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fcd970700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fc8100260 0x7f4fc8195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:25.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fcd970700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc81964d0 con 0x7f4fc80ff060 2024-03-26T18:52:25.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fcd970700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc8196610 con 0x7f4fc8100260 2024-03-26T18:52:25.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fc6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc80ff060 0x7f4fc8195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:25.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fc6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc80ff060 0x7f4fc8195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55980/0 (socket says 172.21.15.102:55980) 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fc6ffd700 1 -- 172.21.15.102:0/2180634544 learned_addr learned my addr 172.21.15.102:0/2180634544 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fc67fc700 1 --2- 172.21.15.102:0/2180634544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fc8100260 0x7f4fc8195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fc6ffd700 1 -- 172.21.15.102:0/2180634544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fc8100260 msgr2=0x7f4fc8195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fc6ffd700 1 --2- 172.21.15.102:0/2180634544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fc8100260 0x7f4fc8195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.865+0000 7f4fc6ffd700 1 -- 172.21.15.102:0/2180634544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fbc0095f0 con 0x7f4fc80ff060 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.866+0000 7f4fc6ffd700 1 --2- 172.21.15.102:0/2180634544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc80ff060 0x7f4fc8195970 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800d900 tx=0x7f4fb800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.866+0000 7f4fcc96e700 1 -- 172.21.15.102:0/2180634544 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fb80041f0 con 0x7f4fc80ff060 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.866+0000 7f4fcd970700 1 -- 172.21.15.102:0/2180634544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc819b0c0 con 0x7f4fc80ff060 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.866+0000 7f4fcc96e700 1 -- 172.21.15.102:0/2180634544 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4fb8004350 con 0x7f4fc80ff060 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.866+0000 7f4fcc96e700 1 -- 172.21.15.102:0/2180634544 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fb8003d20 con 0x7f4fc80ff060 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.866+0000 7f4fcd970700 1 -- 172.21.15.102:0/2180634544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc819b610 con 0x7f4fc80ff060 2024-03-26T18:52:25.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.867+0000 7f4fcd970700 1 -- 172.21.15.102:0/2180634544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc8107bb0 con 0x7f4fc80ff060 2024-03-26T18:52:25.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.867+0000 7f4fcc96e700 1 -- 172.21.15.102:0/2180634544 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f4fb8009730 con 0x7f4fc80ff060 2024-03-26T18:52:25.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.868+0000 7f4fcc96e700 1 --2- 172.21.15.102:0/2180634544 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4fb006c680 0x7f4fb006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:25.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.868+0000 7f4fc67fc700 1 --2- 172.21.15.102:0/2180634544 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4fb006c680 0x7f4fb006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:25.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.868+0000 7f4fcc96e700 1 -- 172.21.15.102:0/2180634544 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f4fb808b780 con 0x7f4fc80ff060 2024-03-26T18:52:25.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.868+0000 7f4fc67fc700 1 --2- 172.21.15.102:0/2180634544 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4fb006c680 0x7f4fb006eb30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc006010 tx=0x7f4fbc00b540 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:25.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:25.872+0000 7f4fcc96e700 1 -- 172.21.15.102:0/2180634544 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4fb80598f0 con 0x7f4fc80ff060 2024-03-26T18:52:26.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.114+0000 7f4fcd970700 1 -- 172.21.15.102:0/2180634544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f4fc819bcb0 con 0x7f4fc80ff060 2024-03-26T18:52:26.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.302+0000 7f4fcc96e700 1 -- 172.21.15.102:0/2180634544 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f4fb8003e80 con 0x7f4fc80ff060 2024-03-26T18:52:26.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.304+0000 7f4fcd970700 1 -- 172.21.15.102:0/2180634544 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4fb006c680 msgr2=0x7f4fb006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:26.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.304+0000 7f4fcd970700 1 --2- 172.21.15.102:0/2180634544 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4fb006c680 0x7f4fb006eb30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc006010 tx=0x7f4fbc00b540 comp rx=0 tx=0).stop 2024-03-26T18:52:26.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.304+0000 7f4fcd970700 1 -- 172.21.15.102:0/2180634544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc80ff060 msgr2=0x7f4fc8195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:26.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.304+0000 7f4fcd970700 1 --2- 172.21.15.102:0/2180634544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc80ff060 0x7f4fc8195970 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800d900 tx=0x7f4fb800dc10 comp rx=0 tx=0).stop 2024-03-26T18:52:26.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.304+0000 7f4fcd970700 1 -- 172.21.15.102:0/2180634544 shutdown_connections 2024-03-26T18:52:26.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.304+0000 7f4fcd970700 1 --2- 172.21.15.102:0/2180634544 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f4fb006c680 0x7f4fb006eb30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:26.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.304+0000 7f4fcd970700 1 --2- 172.21.15.102:0/2180634544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc80ff060 0x7f4fc8195970 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:26.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.304+0000 7f4fcd970700 1 --2- 172.21.15.102:0/2180634544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fc8100260 0x7f4fc8195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:26.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.304+0000 7f4fcd970700 1 -- 172.21.15.102:0/2180634544 >> 172.21.15.102:0/2180634544 conn(0x7f4fc80fa610 msgr2=0x7f4fc8068d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:26.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.305+0000 7f4fcd970700 1 -- 172.21.15.102:0/2180634544 shutdown_connections 2024-03-26T18:52:26.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:26.305+0000 7f4fcd970700 1 -- 172.21.15.102:0/2180634544 wait complete. 2024-03-26T18:52:26.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:26 smithi190 ceph-mon[35043]: Deploying daemon mds.cephfs.smithi190.vykizb on smithi190 2024-03-26T18:52:26.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:26 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.102:6826/2550866692,v1:172.21.15.102:6827/2550866692] up:active 2024-03-26T18:52:26.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:26 smithi190 ceph-mon[35043]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:active} 2024-03-26T18:52:26.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:26 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2180634544' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-26T18:52:26.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:26 smithi102 ceph-mon[29002]: Deploying daemon mds.cephfs.smithi190.vykizb on smithi190 2024-03-26T18:52:26.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:26 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.102:6826/2550866692,v1:172.21.15.102:6827/2550866692] up:active 2024-03-26T18:52:26.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:26 smithi102 ceph-mon[29002]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:active} 2024-03-26T18:52:26.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:26 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2180634544' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-26T18:52:26.921 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T18:52:26.931 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T18:52:26.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-03-26T18:52:27.143 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:27.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:27 smithi190 ceph-mon[35043]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 308 B/s wr, 0 op/s 2024-03-26T18:52:27.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:27 smithi190 ceph-mon[35043]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-26T18:52:27.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:27 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2180634544' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-26T18:52:27.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:27 smithi190 ceph-mon[35043]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:active} 2024-03-26T18:52:27.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:27 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:27.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:27 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:27.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:27 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:27.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:27 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hrbrsp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:52:27.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:27 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hrbrsp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T18:52:27.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:27 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:27.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:27 smithi190 ceph-mon[35043]: Deploying daemon mds.cephfs.smithi102.hrbrsp on smithi102 2024-03-26T18:52:27.550 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:27 smithi102 ceph-mon[29002]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 308 B/s wr, 0 op/s 2024-03-26T18:52:27.550 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:27 smithi102 ceph-mon[29002]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-26T18:52:27.550 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:27 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2180634544' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-26T18:52:27.550 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:27 smithi102 ceph-mon[29002]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:active} 2024-03-26T18:52:27.550 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:27 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:27.551 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:27 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:27.551 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:27 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:27.551 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:27 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hrbrsp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:52:27.551 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:27 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hrbrsp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T18:52:27.551 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:27 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:27.551 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:27 smithi102 ceph-mon[29002]: Deploying daemon mds.cephfs.smithi102.hrbrsp on smithi102 2024-03-26T18:52:28.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:28 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] up:boot 2024-03-26T18:52:28.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:28 smithi102 ceph-mon[29002]: daemon mds.cephfs.smithi190.vykizb assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-26T18:52:28.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:28 smithi102 ceph-mon[29002]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-26T18:52:28.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:28 smithi102 ceph-mon[29002]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-26T18:52:28.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:28 smithi102 ceph-mon[29002]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:active} 1 up:standby 2024-03-26T18:52:28.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:28 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T18:52:28.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:28 smithi102 ceph-mon[29002]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi190.vykizb=up:creating} 2024-03-26T18:52:28.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:28 smithi102 ceph-mon[29002]: daemon mds.cephfs.smithi190.vykizb is now active in filesystem cephfs as rank 1 2024-03-26T18:52:28.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:28 smithi102 ceph-mon[29002]: pgmap v181: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T18:52:28.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:28 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] up:boot 2024-03-26T18:52:28.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:28 smithi190 ceph-mon[35043]: daemon mds.cephfs.smithi190.vykizb assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-26T18:52:28.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:28 smithi190 ceph-mon[35043]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-26T18:52:28.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:28 smithi190 ceph-mon[35043]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-26T18:52:28.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:28 smithi190 ceph-mon[35043]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:active} 1 up:standby 2024-03-26T18:52:28.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:28 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T18:52:28.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:28 smithi190 ceph-mon[35043]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi190.vykizb=up:creating} 2024-03-26T18:52:28.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:28 smithi190 ceph-mon[35043]: daemon mds.cephfs.smithi190.vykizb is now active in filesystem cephfs as rank 1 2024-03-26T18:52:28.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:28 smithi190 ceph-mon[35043]: pgmap v181: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T18:52:29.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:29 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] up:active 2024-03-26T18:52:29.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:29 smithi190 ceph-mon[35043]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi190.vykizb=up:active} 2024-03-26T18:52:29.583 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:29 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] up:active 2024-03-26T18:52:29.583 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:29 smithi102 ceph-mon[29002]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi190.vykizb=up:active} 2024-03-26T18:52:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.770+0000 7f7096307700 1 -- 172.21.15.102:0/1635446717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7090100290 msgr2=0x7f70901006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.770+0000 7f7096307700 1 --2- 172.21.15.102:0/1635446717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7090100290 0x7f70901006e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7078009910 tx=0x7f7078009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.770+0000 7f7096307700 1 -- 172.21.15.102:0/1635446717 shutdown_connections 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.770+0000 7f7096307700 1 --2- 172.21.15.102:0/1635446717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7090100290 0x7f70901006e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.770+0000 7f7096307700 1 --2- 172.21.15.102:0/1635446717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70900ff090 0x7f70900ff4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.770+0000 7f7096307700 1 -- 172.21.15.102:0/1635446717 >> 172.21.15.102:0/1635446717 conn(0x7f70900fa620 msgr2=0x7f70900fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.770+0000 7f7096307700 1 -- 172.21.15.102:0/1635446717 shutdown_connections 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.770+0000 7f7096307700 1 -- 172.21.15.102:0/1635446717 wait complete. 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f7096307700 1 Processor -- start 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f7096307700 1 -- start start 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f7096307700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70900ff090 0x7f7090195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f7096307700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7090100290 0x7f7090195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f7096307700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7090196460 con 0x7f70900ff090 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f7096307700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70901965a0 con 0x7f7090100290 2024-03-26T18:52:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f708ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70900ff090 0x7f7090195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f708ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70900ff090 0x7f7090195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52306/0 (socket says 172.21.15.102:52306) 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f708ffff700 1 -- 172.21.15.102:0/4242442225 learned_addr learned my addr 172.21.15.102:0/4242442225 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f7087fff700 1 --2- 172.21.15.102:0/4242442225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7090100290 0x7f7090195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f708ffff700 1 -- 172.21.15.102:0/4242442225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7090100290 msgr2=0x7f7090195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f708ffff700 1 --2- 172.21.15.102:0/4242442225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7090100290 0x7f7090195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f708ffff700 1 -- 172.21.15.102:0/4242442225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70780095f0 con 0x7f70900ff090 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.771+0000 7f708ffff700 1 --2- 172.21.15.102:0/4242442225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70900ff090 0x7f7090195990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f708000c910 tx=0x7f708000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.772+0000 7f708dffb700 1 -- 172.21.15.102:0/4242442225 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70800041f0 con 0x7f70900ff090 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.772+0000 7f708dffb700 1 -- 172.21.15.102:0/4242442225 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7080007db0 con 0x7f70900ff090 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.772+0000 7f7096307700 1 -- 172.21.15.102:0/4242442225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f709019b060 con 0x7f70900ff090 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.772+0000 7f708dffb700 1 -- 172.21.15.102:0/4242442225 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7080003d60 con 0x7f70900ff090 2024-03-26T18:52:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.772+0000 7f7096307700 1 -- 172.21.15.102:0/4242442225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f709019b580 con 0x7f70900ff090 2024-03-26T18:52:29.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.773+0000 7f7096307700 1 -- 172.21.15.102:0/4242442225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f709018fad0 con 0x7f70900ff090 2024-03-26T18:52:29.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.773+0000 7f708dffb700 1 -- 172.21.15.102:0/4242442225 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f7080007900 con 0x7f70900ff090 2024-03-26T18:52:29.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.774+0000 7f708dffb700 1 --2- 172.21.15.102:0/4242442225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 0x7f707c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:29.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.774+0000 7f708dffb700 1 -- 172.21.15.102:0/4242442225 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f708008b580 con 0x7f70900ff090 2024-03-26T18:52:29.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.774+0000 7f7087fff700 1 --2- 172.21.15.102:0/4242442225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 0x7f707c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.774+0000 7f7087fff700 1 --2- 172.21.15.102:0/4242442225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 0x7f707c06eb30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7078006010 tx=0x7f707800b580 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:29.778+0000 7f708dffb700 1 -- 172.21.15.102:0/4242442225 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f70800596f0 con 0x7f70900ff090 2024-03-26T18:52:30.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.032+0000 7f7096307700 1 -- 172.21.15.102:0/4242442225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f709002cc30 con 0x7f70900ff090 2024-03-26T18:52:30.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.327+0000 7f708dffb700 1 -- 172.21.15.102:0/4242442225 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7080059280 con 0x7f70900ff090 2024-03-26T18:52:30.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.329+0000 7f7096307700 1 -- 172.21.15.102:0/4242442225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 msgr2=0x7f707c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:30.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.329+0000 7f7096307700 1 --2- 172.21.15.102:0/4242442225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 0x7f707c06eb30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7078006010 tx=0x7f707800b580 comp rx=0 tx=0).stop 2024-03-26T18:52:30.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.329+0000 7f7096307700 1 -- 172.21.15.102:0/4242442225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70900ff090 msgr2=0x7f7090195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:30.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.329+0000 7f7096307700 1 --2- 172.21.15.102:0/4242442225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70900ff090 0x7f7090195990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f708000c910 tx=0x7f708000ccd0 comp rx=0 tx=0).stop 2024-03-26T18:52:30.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.329+0000 7f7096307700 1 -- 172.21.15.102:0/4242442225 shutdown_connections 2024-03-26T18:52:30.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.329+0000 7f7096307700 1 --2- 172.21.15.102:0/4242442225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 0x7f707c06eb30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:30.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.329+0000 7f7096307700 1 --2- 172.21.15.102:0/4242442225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70900ff090 0x7f7090195990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:30.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.329+0000 7f7096307700 1 --2- 172.21.15.102:0/4242442225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7090100290 0x7f7090195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:30.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.329+0000 7f7096307700 1 -- 172.21.15.102:0/4242442225 >> 172.21.15.102:0/4242442225 conn(0x7f70900fa620 msgr2=0x7f70900fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:30.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.329+0000 7f7096307700 1 -- 172.21.15.102:0/4242442225 shutdown_connections 2024-03-26T18:52:30.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:30.330+0000 7f7096307700 1 -- 172.21.15.102:0/4242442225 wait complete. 2024-03-26T18:52:30.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:30 smithi102 ceph-mon[29002]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 6 op/s 2024-03-26T18:52:30.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:30 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/4242442225' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-03-26T18:52:30.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:30 smithi190 ceph-mon[35043]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 6 op/s 2024-03-26T18:52:30.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:30 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/4242442225' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-03-26T18:52:30.951 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T18:52:30.962 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T18:52:30.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data false' 2024-03-26T18:52:31.171 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:31.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-26T18:52:31.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: Cluster is now healthy 2024-03-26T18:52:31.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/4242442225' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-03-26T18:52:31.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:boot 2024-03-26T18:52:31.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi190.vykizb=up:active} 1 up:standby 2024-03-26T18:52:31.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T18:52:31.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:31.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:31.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:31.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.hwsvdg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:52:31.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.hwsvdg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T18:52:31.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:31.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:31 smithi190 ceph-mon[35043]: Deploying daemon mds.cephfs.smithi190.hwsvdg on smithi190 2024-03-26T18:52:31.558 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-26T18:52:31.558 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: Cluster is now healthy 2024-03-26T18:52:31.558 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/4242442225' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-03-26T18:52:31.558 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:boot 2024-03-26T18:52:31.558 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi190.vykizb=up:active} 1 up:standby 2024-03-26T18:52:31.559 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T18:52:31.559 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:31.559 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:31.559 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:31.559 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.hwsvdg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:52:31.559 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.hwsvdg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T18:52:31.559 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:31.559 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:31 smithi102 ceph-mon[29002]: Deploying daemon mds.cephfs.smithi190.hwsvdg on smithi190 2024-03-26T18:52:32.475 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:32 smithi102 ceph-mon[29002]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s wr, 7 op/s 2024-03-26T18:52:32.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:32 smithi190 ceph-mon[35043]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s wr, 7 op/s 2024-03-26T18:52:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.862+0000 7f2645d00700 1 -- 172.21.15.102:0/1894107725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2640100bf0 msgr2=0x7f2640102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.862+0000 7f2645d00700 1 --2- 172.21.15.102:0/1894107725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2640100bf0 0x7f2640102fd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f263000b0a0 tx=0x7f263000b3b0 comp rx=0 tx=0).stop 2024-03-26T18:52:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.862+0000 7f2645d00700 1 -- 172.21.15.102:0/1894107725 shutdown_connections 2024-03-26T18:52:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.862+0000 7f2645d00700 1 --2- 172.21.15.102:0/1894107725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2640103510 0x7f26401058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.862+0000 7f2645d00700 1 --2- 172.21.15.102:0/1894107725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2640100bf0 0x7f2640102fd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.862+0000 7f2645d00700 1 -- 172.21.15.102:0/1894107725 >> 172.21.15.102:0/1894107725 conn(0x7f26400fa610 msgr2=0x7f26400fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.862+0000 7f2645d00700 1 -- 172.21.15.102:0/1894107725 shutdown_connections 2024-03-26T18:52:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.862+0000 7f2645d00700 1 -- 172.21.15.102:0/1894107725 wait complete. 2024-03-26T18:52:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f2645d00700 1 Processor -- start 2024-03-26T18:52:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f2645d00700 1 -- start start 2024-03-26T18:52:32.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f2645d00700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2640100bf0 0x7f2640195a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:32.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f2645d00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2640103510 0x7f2640195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:32.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f2645d00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2640196590 con 0x7f2640103510 2024-03-26T18:52:32.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f2645d00700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26401966d0 con 0x7f2640100bf0 2024-03-26T18:52:32.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f263effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2640103510 0x7f2640195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:32.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f263effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2640103510 0x7f2640195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52334/0 (socket says 172.21.15.102:52334) 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f263effd700 1 -- 172.21.15.102:0/1189513586 learned_addr learned my addr 172.21.15.102:0/1189513586 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f263f7fe700 1 --2- 172.21.15.102:0/1189513586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2640100bf0 0x7f2640195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f263effd700 1 -- 172.21.15.102:0/1189513586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2640100bf0 msgr2=0x7f2640195a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f263effd700 1 --2- 172.21.15.102:0/1189513586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2640100bf0 0x7f2640195a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f263effd700 1 -- 172.21.15.102:0/1189513586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2630009d00 con 0x7f2640103510 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.863+0000 7f263effd700 1 --2- 172.21.15.102:0/1189513586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2640103510 0x7f2640195f70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f263400b4b0 tx=0x7f263400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.864+0000 7f263cff9700 1 -- 172.21.15.102:0/1189513586 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2634011800 con 0x7f2640103510 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.864+0000 7f263cff9700 1 -- 172.21.15.102:0/1189513586 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2634011e10 con 0x7f2640103510 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.864+0000 7f263cff9700 1 -- 172.21.15.102:0/1189513586 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f263400f2e0 con 0x7f2640103510 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.864+0000 7f2645d00700 1 -- 172.21.15.102:0/1189513586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f264019b180 con 0x7f2640103510 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.864+0000 7f2645d00700 1 -- 172.21.15.102:0/1189513586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264019b750 con 0x7f2640103510 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.865+0000 7f2645d00700 1 -- 172.21.15.102:0/1189513586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264018fc00 con 0x7f2640103510 2024-03-26T18:52:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.865+0000 7f263cff9700 1 -- 172.21.15.102:0/1189513586 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f2634010380 con 0x7f2640103510 2024-03-26T18:52:32.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.865+0000 7f263cff9700 1 --2- 172.21.15.102:0/1189513586 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f262806c680 0x7f262806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:32.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.865+0000 7f263cff9700 1 -- 172.21.15.102:0/1189513586 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f263408b3d0 con 0x7f2640103510 2024-03-26T18:52:32.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.865+0000 7f263f7fe700 1 --2- 172.21.15.102:0/1189513586 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f262806c680 0x7f262806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:32.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.866+0000 7f263f7fe700 1 --2- 172.21.15.102:0/1189513586 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f262806c680 0x7f262806eb30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2630000f80 tx=0x7f263000b7b0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:32.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:32.870+0000 7f263cff9700 1 -- 172.21.15.102:0/1189513586 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2634059670 con 0x7f2640103510 2024-03-26T18:52:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.111+0000 7f2645d00700 1 -- 172.21.15.102:0/1189513586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7f264002cc30 con 0x7f2640103510 2024-03-26T18:52:33.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.328+0000 7f263cff9700 1 -- 172.21.15.102:0/1189513586 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v11) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2634059200 con 0x7f2640103510 2024-03-26T18:52:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.330+0000 7f2645d00700 1 -- 172.21.15.102:0/1189513586 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f262806c680 msgr2=0x7f262806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.330+0000 7f2645d00700 1 --2- 172.21.15.102:0/1189513586 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f262806c680 0x7f262806eb30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2630000f80 tx=0x7f263000b7b0 comp rx=0 tx=0).stop 2024-03-26T18:52:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.330+0000 7f2645d00700 1 -- 172.21.15.102:0/1189513586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2640103510 msgr2=0x7f2640195f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.330+0000 7f2645d00700 1 --2- 172.21.15.102:0/1189513586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2640103510 0x7f2640195f70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f263400b4b0 tx=0x7f263400b870 comp rx=0 tx=0).stop 2024-03-26T18:52:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.331+0000 7f2645d00700 1 -- 172.21.15.102:0/1189513586 shutdown_connections 2024-03-26T18:52:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.331+0000 7f2645d00700 1 --2- 172.21.15.102:0/1189513586 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f262806c680 0x7f262806eb30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.331+0000 7f2645d00700 1 --2- 172.21.15.102:0/1189513586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2640100bf0 0x7f2640195a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.331+0000 7f2645d00700 1 --2- 172.21.15.102:0/1189513586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2640103510 0x7f2640195f70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:33.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.331+0000 7f2645d00700 1 -- 172.21.15.102:0/1189513586 >> 172.21.15.102:0/1189513586 conn(0x7f26400fa610 msgr2=0x7f26400fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:33.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.331+0000 7f2645d00700 1 -- 172.21.15.102:0/1189513586 shutdown_connections 2024-03-26T18:52:33.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:33.331+0000 7f2645d00700 1 -- 172.21.15.102:0/1189513586 wait complete. 2024-03-26T18:52:33.324 INFO:teuthology.orchestra.run.smithi102.stderr:inline data disabled 2024-03-26T18:52:33.928 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T18:52:33.939 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T18:52:33.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1189513586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] up:boot 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1189513586' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T18:52:34.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:52:34.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/1189513586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] up:boot 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/1189513586' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:52:34.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:34.697 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:35.075 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:35 smithi102 ceph-mon[29002]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-26T18:52:35.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:35 smithi190 ceph-mon[35043]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-26T18:52:36.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:36 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:standby 2024-03-26T18:52:36.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:36 smithi102 ceph-mon[29002]: Dropping low affinity active daemon mds.cephfs.smithi102.tvsqaa in favor of higher affinity standby. 2024-03-26T18:52:36.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:36 smithi102 ceph-mon[29002]: Replacing daemon mds.cephfs.smithi102.tvsqaa as rank 0 with standby daemon mds.cephfs.smithi102.hrbrsp 2024-03-26T18:52:36.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:36 smithi102 ceph-mon[29002]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T18:52:36.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:36 smithi102 ceph-mon[29002]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:36.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:36 smithi102 ceph-mon[29002]: osdmap e47: 6 total, 6 up, 6 in 2024-03-26T18:52:36.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:36 smithi102 ceph-mon[29002]: fsmap cephfs:2/2 {0=cephfs.smithi102.hrbrsp=up:replay,1=cephfs.smithi190.vykizb=up:active} 1 up:standby 2024-03-26T18:52:36.063 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:36 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:36.064 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:36 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:36.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:36 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:standby 2024-03-26T18:52:36.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:36 smithi190 ceph-mon[35043]: Dropping low affinity active daemon mds.cephfs.smithi102.tvsqaa in favor of higher affinity standby. 2024-03-26T18:52:36.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:36 smithi190 ceph-mon[35043]: Replacing daemon mds.cephfs.smithi102.tvsqaa as rank 0 with standby daemon mds.cephfs.smithi102.hrbrsp 2024-03-26T18:52:36.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:36 smithi190 ceph-mon[35043]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T18:52:36.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:36 smithi190 ceph-mon[35043]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:36.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:36 smithi190 ceph-mon[35043]: osdmap e47: 6 total, 6 up, 6 in 2024-03-26T18:52:36.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:36 smithi190 ceph-mon[35043]: fsmap cephfs:2/2 {0=cephfs.smithi102.hrbrsp=up:replay,1=cephfs.smithi190.vykizb=up:active} 1 up:standby 2024-03-26T18:52:36.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:36 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:36.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:36 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:37.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.040+0000 7f0108689700 1 -- 172.21.15.102:0/3202976332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01000fe9d0 msgr2=0x7f01000fede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.040+0000 7f0108689700 1 --2- 172.21.15.102:0/3202976332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01000fe9d0 0x7f01000fede0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f00fc009910 tx=0x7f00fc009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.040+0000 7f0108689700 1 -- 172.21.15.102:0/3202976332 shutdown_connections 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.040+0000 7f0108689700 1 --2- 172.21.15.102:0/3202976332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01000ff320 0x7f01000ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.040+0000 7f0108689700 1 --2- 172.21.15.102:0/3202976332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01000fe9d0 0x7f01000fede0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.040+0000 7f0108689700 1 -- 172.21.15.102:0/3202976332 >> 172.21.15.102:0/3202976332 conn(0x7f01000fa600 msgr2=0x7f01000fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.040+0000 7f0108689700 1 -- 172.21.15.102:0/3202976332 shutdown_connections 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.040+0000 7f0108689700 1 -- 172.21.15.102:0/3202976332 wait complete. 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.041+0000 7f0108689700 1 Processor -- start 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.041+0000 7f0108689700 1 -- start start 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.041+0000 7f0108689700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01000fe9d0 0x7f0100197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.041+0000 7f0108689700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01000ff320 0x7f0100198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.041+0000 7f0108689700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01001986a0 con 0x7f01000ff320 2024-03-26T18:52:37.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.041+0000 7f0108689700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01001987e0 con 0x7f01000fe9d0 2024-03-26T18:52:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.041+0000 7f0106425700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01000fe9d0 0x7f0100197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.041+0000 7f0106425700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01000fe9d0 0x7f0100197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:45598/0 (socket says 172.21.15.102:45598) 2024-03-26T18:52:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.041+0000 7f0106425700 1 -- 172.21.15.102:0/2270125901 learned_addr learned my addr 172.21.15.102:0/2270125901 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.042+0000 7f0105c24700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01000ff320 0x7f0100198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.042+0000 7f0106425700 1 -- 172.21.15.102:0/2270125901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01000ff320 msgr2=0x7f0100198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.042+0000 7f0106425700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01000ff320 0x7f0100198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.042+0000 7f0106425700 1 -- 172.21.15.102:0/2270125901 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc0095f0 con 0x7f01000fe9d0 2024-03-26T18:52:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.042+0000 7f0105c24700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01000ff320 0x7f0100198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:52:37.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.042+0000 7f0106425700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01000fe9d0 0x7f0100197b40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00fc006010 tx=0x7f00fc00bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:37.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.042+0000 7f00f77fe700 1 -- 172.21.15.102:0/2270125901 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00fc01d070 con 0x7f01000fe9d0 2024-03-26T18:52:37.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.042+0000 7f0108689700 1 -- 172.21.15.102:0/2270125901 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010019d230 con 0x7f01000fe9d0 2024-03-26T18:52:37.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.042+0000 7f00f77fe700 1 -- 172.21.15.102:0/2270125901 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f00fc00f460 con 0x7f01000fe9d0 2024-03-26T18:52:37.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.042+0000 7f00f77fe700 1 -- 172.21.15.102:0/2270125901 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00fc021640 con 0x7f01000fe9d0 2024-03-26T18:52:37.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.042+0000 7f0108689700 1 -- 172.21.15.102:0/2270125901 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010019d720 con 0x7f01000fe9d0 2024-03-26T18:52:37.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.043+0000 7f0108689700 1 -- 172.21.15.102:0/2270125901 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f010010ae80 con 0x7f01000fe9d0 2024-03-26T18:52:37.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.044+0000 7f00f77fe700 1 -- 172.21.15.102:0/2270125901 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f00fc00fab0 con 0x7f01000fe9d0 2024-03-26T18:52:37.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.045+0000 7f00f77fe700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f00ec06c680 0x7f00ec06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:37.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.045+0000 7f00f77fe700 1 -- 172.21.15.102:0/2270125901 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f00fc08cc10 con 0x7f01000fe9d0 2024-03-26T18:52:37.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.045+0000 7f0105c24700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f00ec06c680 0x7f00ec06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:37.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.045+0000 7f0105c24700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f00ec06c680 0x7f00ec06eb30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f00f0005950 tx=0x7f00f000b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:37.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.049+0000 7f00f77fe700 1 -- 172.21.15.102:0/2270125901 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f00fc05acc0 con 0x7f01000fe9d0 2024-03-26T18:52:37.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:37 smithi102 ceph-mon[29002]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 2.4 KiB/s wr, 8 op/s 2024-03-26T18:52:37.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:37 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:resolve 2024-03-26T18:52:37.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:37 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] up:boot 2024-03-26T18:52:37.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:37 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] up:active 2024-03-26T18:52:37.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:37 smithi102 ceph-mon[29002]: fsmap cephfs:2/2 {0=cephfs.smithi102.hrbrsp=up:resolve,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:37.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:37 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T18:52:37.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:37 smithi190 ceph-mon[35043]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 2.4 KiB/s wr, 8 op/s 2024-03-26T18:52:37.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:37 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:resolve 2024-03-26T18:52:37.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:37 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] up:boot 2024-03-26T18:52:37.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:37 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] up:active 2024-03-26T18:52:37.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:37 smithi190 ceph-mon[35043]: fsmap cephfs:2/2 {0=cephfs.smithi102.hrbrsp=up:resolve,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:37.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:37 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T18:52:37.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.291+0000 7f0108689700 1 -- 172.21.15.102:0/2270125901 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f010019ddb0 con 0x7f01000fe9d0 2024-03-26T18:52:37.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.292+0000 7f00f77fe700 1 -- 172.21.15.102:0/2270125901 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1840 (secure 0 0 0) 0x7f00fc026740 con 0x7f01000fe9d0 2024-03-26T18:52:37.284 INFO:teuthology.orchestra.run.smithi102.stdout:e15 2024-03-26T18:52:37.284 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:52:37.284 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:52:37.284 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:52:37.284 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:52:37.284 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 15 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:52:37.035606+0000 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:52:37.285 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:reconnect seq 4 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:52:37.286 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:52:37.287 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:52:37.287 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:52:37.287 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:52:37.287 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:52:37.287 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:52:37.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.294+0000 7f0108689700 1 -- 172.21.15.102:0/2270125901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f00ec06c680 msgr2=0x7f00ec06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.294+0000 7f0108689700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f00ec06c680 0x7f00ec06eb30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f00f0005950 tx=0x7f00f000b410 comp rx=0 tx=0).stop 2024-03-26T18:52:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.294+0000 7f0108689700 1 -- 172.21.15.102:0/2270125901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01000fe9d0 msgr2=0x7f0100197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.294+0000 7f0108689700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01000fe9d0 0x7f0100197b40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00fc006010 tx=0x7f00fc00bd70 comp rx=0 tx=0).stop 2024-03-26T18:52:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.294+0000 7f0108689700 1 -- 172.21.15.102:0/2270125901 shutdown_connections 2024-03-26T18:52:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.294+0000 7f0108689700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f00ec06c680 0x7f00ec06eb30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.294+0000 7f0108689700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01000fe9d0 0x7f0100197b40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.294+0000 7f0108689700 1 --2- 172.21.15.102:0/2270125901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01000ff320 0x7f0100198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.294+0000 7f0108689700 1 -- 172.21.15.102:0/2270125901 >> 172.21.15.102:0/2270125901 conn(0x7f01000fa600 msgr2=0x7f0100107000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.295+0000 7f0108689700 1 -- 172.21.15.102:0/2270125901 shutdown_connections 2024-03-26T18:52:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:37.295+0000 7f0108689700 1 -- 172.21.15.102:0/2270125901 wait complete. 2024-03-26T18:52:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 15 2024-03-26T18:52:38.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-26T18:52:38.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:38 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:reconnect 2024-03-26T18:52:38.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:38 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] up:standby 2024-03-26T18:52:38.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:38 smithi102 ceph-mon[29002]: fsmap cephfs:2/2 {0=cephfs.smithi102.hrbrsp=up:reconnect,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:38.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:38 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2270125901' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:52:38.466 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:38.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:38 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:reconnect 2024-03-26T18:52:38.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:38 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] up:standby 2024-03-26T18:52:38.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:38 smithi190 ceph-mon[35043]: fsmap cephfs:2/2 {0=cephfs.smithi102.hrbrsp=up:reconnect,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:38.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:38 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2270125901' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:52:39.337 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:39 smithi102 ceph-mon[29002]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 2.4 KiB/s wr, 8 op/s 2024-03-26T18:52:39.338 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:39 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:rejoin 2024-03-26T18:52:39.338 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:39 smithi102 ceph-mon[29002]: fsmap cephfs:2/2 {0=cephfs.smithi102.hrbrsp=up:rejoin,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:39.338 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:39 smithi102 ceph-mon[29002]: daemon mds.cephfs.smithi102.hrbrsp is now active in filesystem cephfs as rank 0 2024-03-26T18:52:39.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:39 smithi190 ceph-mon[35043]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 2.4 KiB/s wr, 8 op/s 2024-03-26T18:52:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:39 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:rejoin 2024-03-26T18:52:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:39 smithi190 ceph-mon[35043]: fsmap cephfs:2/2 {0=cephfs.smithi102.hrbrsp=up:rejoin,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:39 smithi190 ceph-mon[35043]: daemon mds.cephfs.smithi102.hrbrsp is now active in filesystem cephfs as rank 0 2024-03-26T18:52:40.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:40 smithi102 ceph-mon[29002]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T18:52:40.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:40 smithi102 ceph-mon[29002]: Cluster is now healthy 2024-03-26T18:52:40.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:40 smithi102 ceph-mon[29002]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:active 2024-03-26T18:52:40.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:40 smithi102 ceph-mon[29002]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:40.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:40 smithi190 ceph-mon[35043]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T18:52:40.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:40 smithi190 ceph-mon[35043]: Cluster is now healthy 2024-03-26T18:52:40.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:40 smithi190 ceph-mon[35043]: mds.? [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] up:active 2024-03-26T18:52:40.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:40 smithi190 ceph-mon[35043]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:52:41.518 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:41 smithi102 ceph-mon[29002]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-26T18:52:41.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:41 smithi190 ceph-mon[35043]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-26T18:52:41.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.545+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/882474097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fb4100690 msgr2=0x7f3fb4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:41.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.545+0000 7f3fb8a1b700 1 --2- 172.21.15.102:0/882474097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fb4100690 0x7f3fb4100aa0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8009910 tx=0x7f3fa8009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:41.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.545+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/882474097 shutdown_connections 2024-03-26T18:52:41.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.545+0000 7f3fb8a1b700 1 --2- 172.21.15.102:0/882474097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fb4100fe0 0x7f3fb40ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:41.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.545+0000 7f3fb8a1b700 1 --2- 172.21.15.102:0/882474097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fb4100690 0x7f3fb4100aa0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:41.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.545+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/882474097 >> 172.21.15.102:0/882474097 conn(0x7f3fb40f9e50 msgr2=0x7f3fb40fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:41.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.545+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/882474097 shutdown_connections 2024-03-26T18:52:41.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.545+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/882474097 wait complete. 2024-03-26T18:52:41.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.546+0000 7f3fb8a1b700 1 Processor -- start 2024-03-26T18:52:41.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.546+0000 7f3fb8a1b700 1 -- start start 2024-03-26T18:52:41.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.546+0000 7f3fb8a1b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fb4100690 0x7f3fb4197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:41.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.546+0000 7f3fb8a1b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fb4100fe0 0x7f3fb41980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:41.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.546+0000 7f3fb8a1b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb41986e0 con 0x7f3fb4100690 2024-03-26T18:52:41.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.546+0000 7f3fb8a1b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb4198820 con 0x7f3fb4100fe0 2024-03-26T18:52:41.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fb259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fb4100690 0x7f3fb4197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:41.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fb259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fb4100690 0x7f3fb4197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45510/0 (socket says 172.21.15.102:45510) 2024-03-26T18:52:41.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fb259c700 1 -- 172.21.15.102:0/1621307403 learned_addr learned my addr 172.21.15.102:0/1621307403 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:41.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fb1d9b700 1 --2- 172.21.15.102:0/1621307403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fb4100fe0 0x7f3fb41980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:41.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fb259c700 1 -- 172.21.15.102:0/1621307403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fb4100fe0 msgr2=0x7f3fb41980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:41.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fb259c700 1 --2- 172.21.15.102:0/1621307403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fb4100fe0 0x7f3fb41980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:41.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fb259c700 1 -- 172.21.15.102:0/1621307403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa80095f0 con 0x7f3fb4100690 2024-03-26T18:52:41.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fb259c700 1 --2- 172.21.15.102:0/1621307403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fb4100690 0x7f3fb4197b80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8005b40 tx=0x7f3fa800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:41.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fa37fe700 1 -- 172.21.15.102:0/1621307403 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fa801d070 con 0x7f3fb4100690 2024-03-26T18:52:41.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/1621307403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb419d270 con 0x7f3fb4100690 2024-03-26T18:52:41.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fa37fe700 1 -- 172.21.15.102:0/1621307403 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3fa8005dc0 con 0x7f3fb4100690 2024-03-26T18:52:41.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fa37fe700 1 -- 172.21.15.102:0/1621307403 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fa8022470 con 0x7f3fb4100690 2024-03-26T18:52:41.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.547+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/1621307403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb419d760 con 0x7f3fb4100690 2024-03-26T18:52:41.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.548+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/1621307403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb4107b40 con 0x7f3fb4100690 2024-03-26T18:52:41.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.549+0000 7f3fa37fe700 1 -- 172.21.15.102:0/1621307403 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f3fa80226a0 con 0x7f3fb4100690 2024-03-26T18:52:41.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.549+0000 7f3fa37fe700 1 --2- 172.21.15.102:0/1621307403 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f9c06c680 0x7f3f9c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:41.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.549+0000 7f3fa37fe700 1 -- 172.21.15.102:0/1621307403 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f3fa808ced0 con 0x7f3fb4100690 2024-03-26T18:52:41.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.549+0000 7f3fb1d9b700 1 --2- 172.21.15.102:0/1621307403 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f9c06c680 0x7f3f9c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:41.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.549+0000 7f3fb1d9b700 1 --2- 172.21.15.102:0/1621307403 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f9c06c680 0x7f3f9c06eb30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009c60 tx=0x7f3fa4009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:41.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.554+0000 7f3fa37fe700 1 -- 172.21.15.102:0/1621307403 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3fa805b0c0 con 0x7f3fb4100690 2024-03-26T18:52:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.816+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/1621307403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fb4107d00 con 0x7f3fb4100690 2024-03-26T18:52:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.817+0000 7f3fa37fe700 1 -- 172.21.15.102:0/1621307403 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4760 (secure 0 0 0) 0x7f3fa805ac50 con 0x7f3fb4100690 2024-03-26T18:52:41.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.819+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/1621307403 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f9c06c680 msgr2=0x7f3f9c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:41.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.819+0000 7f3fb8a1b700 1 --2- 172.21.15.102:0/1621307403 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f9c06c680 0x7f3f9c06eb30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009c60 tx=0x7f3fa4009260 comp rx=0 tx=0).stop 2024-03-26T18:52:41.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.819+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/1621307403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fb4100690 msgr2=0x7f3fb4197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:41.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.819+0000 7f3fb8a1b700 1 --2- 172.21.15.102:0/1621307403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fb4100690 0x7f3fb4197b80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8005b40 tx=0x7f3fa800fa00 comp rx=0 tx=0).stop 2024-03-26T18:52:41.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.819+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/1621307403 shutdown_connections 2024-03-26T18:52:41.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.819+0000 7f3fb8a1b700 1 --2- 172.21.15.102:0/1621307403 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f3f9c06c680 0x7f3f9c06eb30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:41.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.819+0000 7f3fb8a1b700 1 --2- 172.21.15.102:0/1621307403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fb4100690 0x7f3fb4197b80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:41.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.819+0000 7f3fb8a1b700 1 --2- 172.21.15.102:0/1621307403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fb4100fe0 0x7f3fb41980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:41.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.819+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/1621307403 >> 172.21.15.102:0/1621307403 conn(0x7f3fb40f9e50 msgr2=0x7f3fb40fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:41.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.819+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/1621307403 shutdown_connections 2024-03-26T18:52:41.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:41.819+0000 7f3fb8a1b700 1 -- 172.21.15.102:0/1621307403 wait complete. 2024-03-26T18:52:41.813 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 17 2024-03-26T18:52:41.822 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:52:42.430 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:42 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1621307403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T18:52:42.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:42.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:42.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:42.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:52:42.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:42.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:52:42.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-26T18:52:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:42 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/1621307403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T18:52:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:52:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:42.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:52:43.427 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:43.505 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:43 smithi190 ceph-mon[35043]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T18:52:43.549 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:43 smithi102 ceph-mon[29002]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T18:52:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.009+0000 7fb541a21700 1 -- 172.21.15.102:0/382213209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb53c0ff060 msgr2=0x7fb53c0ff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.009+0000 7fb541a21700 1 --2- 172.21.15.102:0/382213209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb53c0ff060 0x7fb53c0ff470 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009910 tx=0x7fb52c009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.009+0000 7fb541a21700 1 -- 172.21.15.102:0/382213209 shutdown_connections 2024-03-26T18:52:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.009+0000 7fb541a21700 1 --2- 172.21.15.102:0/382213209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c100260 0x7fb53c1006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.009+0000 7fb541a21700 1 --2- 172.21.15.102:0/382213209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb53c0ff060 0x7fb53c0ff470 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.009+0000 7fb541a21700 1 -- 172.21.15.102:0/382213209 >> 172.21.15.102:0/382213209 conn(0x7fb53c0fa610 msgr2=0x7fb53c0fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.010+0000 7fb541a21700 1 -- 172.21.15.102:0/382213209 shutdown_connections 2024-03-26T18:52:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.010+0000 7fb541a21700 1 -- 172.21.15.102:0/382213209 wait complete. 2024-03-26T18:52:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.010+0000 7fb541a21700 1 Processor -- start 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.010+0000 7fb541a21700 1 -- start start 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.010+0000 7fb541a21700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c0ff060 0x7fb53c069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.010+0000 7fb541a21700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb53c100260 0x7fb53c19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.010+0000 7fb541a21700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c19aca0 con 0x7fb53c100260 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.010+0000 7fb541a21700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c19ade0 con 0x7fb53c0ff060 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb53a7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb53c100260 0x7fb53c19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb53a7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb53c100260 0x7fb53c19a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45520/0 (socket says 172.21.15.102:45520) 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb53a7fc700 1 -- 172.21.15.102:0/3483184190 learned_addr learned my addr 172.21.15.102:0/3483184190 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb53affd700 1 --2- 172.21.15.102:0/3483184190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c0ff060 0x7fb53c069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb53a7fc700 1 -- 172.21.15.102:0/3483184190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c0ff060 msgr2=0x7fb53c069930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb53a7fc700 1 --2- 172.21.15.102:0/3483184190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c0ff060 0x7fb53c069930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb53a7fc700 1 -- 172.21.15.102:0/3483184190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c0095f0 con 0x7fb53c100260 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb53a7fc700 1 --2- 172.21.15.102:0/3483184190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb53c100260 0x7fb53c19a680 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb53000d8d0 tx=0x7fb53000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb540a1f700 1 -- 172.21.15.102:0/3483184190 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5300098a0 con 0x7fb53c100260 2024-03-26T18:52:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb541a21700 1 -- 172.21.15.102:0/3483184190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53c19b0c0 con 0x7fb53c100260 2024-03-26T18:52:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb540a1f700 1 -- 172.21.15.102:0/3483184190 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb530010460 con 0x7fb53c100260 2024-03-26T18:52:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb540a1f700 1 -- 172.21.15.102:0/3483184190 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb53000f5f0 con 0x7fb53c100260 2024-03-26T18:52:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.011+0000 7fb541a21700 1 -- 172.21.15.102:0/3483184190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53c19b610 con 0x7fb53c100260 2024-03-26T18:52:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.012+0000 7fb541a21700 1 -- 172.21.15.102:0/3483184190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53c107bb0 con 0x7fb53c100260 2024-03-26T18:52:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.012+0000 7fb540a1f700 1 -- 172.21.15.102:0/3483184190 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fb530009a00 con 0x7fb53c100260 2024-03-26T18:52:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.013+0000 7fb540a1f700 1 --2- 172.21.15.102:0/3483184190 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb52406c680 0x7fb52406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.013+0000 7fb53affd700 1 --2- 172.21.15.102:0/3483184190 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb52406c680 0x7fb52406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.013+0000 7fb540a1f700 1 -- 172.21.15.102:0/3483184190 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb53008b7c0 con 0x7fb53c100260 2024-03-26T18:52:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.013+0000 7fb53affd700 1 --2- 172.21.15.102:0/3483184190 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb52406c680 0x7fb52406eb30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb52c006010 tx=0x7fb52c00b540 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.017+0000 7fb540a1f700 1 -- 172.21.15.102:0/3483184190 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb5300598d0 con 0x7fb53c100260 2024-03-26T18:52:45.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.293+0000 7fb541a21700 1 -- 172.21.15.102:0/3483184190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fb53c107da0 con 0x7fb53c100260 2024-03-26T18:52:45.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.293+0000 7fb540a1f700 1 -- 172.21.15.102:0/3483184190 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v17) v1 ==== 78+0+83 (secure 0 0 0) 0x7fb530059460 con 0x7fb53c100260 2024-03-26T18:52:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.295+0000 7fb541a21700 1 -- 172.21.15.102:0/3483184190 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb52406c680 msgr2=0x7fb52406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.295+0000 7fb541a21700 1 --2- 172.21.15.102:0/3483184190 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb52406c680 0x7fb52406eb30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb52c006010 tx=0x7fb52c00b540 comp rx=0 tx=0).stop 2024-03-26T18:52:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.295+0000 7fb541a21700 1 -- 172.21.15.102:0/3483184190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb53c100260 msgr2=0x7fb53c19a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.295+0000 7fb541a21700 1 --2- 172.21.15.102:0/3483184190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb53c100260 0x7fb53c19a680 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb53000d8d0 tx=0x7fb53000dbe0 comp rx=0 tx=0).stop 2024-03-26T18:52:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.295+0000 7fb541a21700 1 -- 172.21.15.102:0/3483184190 shutdown_connections 2024-03-26T18:52:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.295+0000 7fb541a21700 1 --2- 172.21.15.102:0/3483184190 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fb52406c680 0x7fb52406eb30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.295+0000 7fb541a21700 1 --2- 172.21.15.102:0/3483184190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c0ff060 0x7fb53c069930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.295+0000 7fb541a21700 1 --2- 172.21.15.102:0/3483184190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb53c100260 0x7fb53c19a680 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.295+0000 7fb541a21700 1 -- 172.21.15.102:0/3483184190 >> 172.21.15.102:0/3483184190 conn(0x7fb53c0fa610 msgr2=0x7fb53c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.295+0000 7fb541a21700 1 -- 172.21.15.102:0/3483184190 shutdown_connections 2024-03-26T18:52:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:45.295+0000 7fb541a21700 1 -- 172.21.15.102:0/3483184190 wait complete. 2024-03-26T18:52:45.298 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:52:45.478 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:45 smithi102 ceph-mon[29002]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-26T18:52:45.478 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:45 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:45.478 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:45 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:45 smithi190 ceph-mon[35043]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-26T18:52:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:45 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:45 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:46.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:46 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3483184190' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-26T18:52:46.440 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-26T18:52:46.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T18:52:46.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:46 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3483184190' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-26T18:52:46.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.573+0000 7fa1c6b39640 1 -- 172.21.15.102:0/591190053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1c0100860 msgr2=0x7fa1c0100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:46.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.573+0000 7fa1c6b39640 1 --2- 172.21.15.102:0/591190053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1c0100860 0x7fa1c0100cd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009880 tx=0x7fa1ac02bce0 comp rx=0 tx=0).stop 2024-03-26T18:52:46.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.573+0000 7fa1c6b39640 1 -- 172.21.15.102:0/591190053 shutdown_connections 2024-03-26T18:52:46.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.573+0000 7fa1c6b39640 1 --2- 172.21.15.102:0/591190053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1c0100860 0x7fa1c0100cd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:46.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.573+0000 7fa1c6b39640 1 --2- 172.21.15.102:0/591190053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c00ff670 0x7fa1c00ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:46.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.573+0000 7fa1c6b39640 1 -- 172.21.15.102:0/591190053 >> 172.21.15.102:0/591190053 conn(0x7fa1c00fabd0 msgr2=0x7fa1c00fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:46.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.573+0000 7fa1c6b39640 1 -- 172.21.15.102:0/591190053 shutdown_connections 2024-03-26T18:52:46.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.573+0000 7fa1c6b39640 1 -- 172.21.15.102:0/591190053 wait complete. 2024-03-26T18:52:46.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1c6b39640 1 Processor -- start 2024-03-26T18:52:46.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1c6b39640 1 -- start start 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1c6b39640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c00ff670 0x7fa1c0194e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1c6b39640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1c0100860 0x7fa1c01953d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1c6b39640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c01959a0 con 0x7fa1c0100860 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1c6b39640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c0195b10 con 0x7fa1c00ff670 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1bffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1c0100860 0x7fa1c01953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1bffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1c0100860 0x7fa1c01953d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45544/0 (socket says 172.21.15.102:45544) 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1bffff640 1 -- 172.21.15.102:0/2022783231 learned_addr learned my addr 172.21.15.102:0/2022783231 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1c48d5640 1 --2- 172.21.15.102:0/2022783231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c00ff670 0x7fa1c0194e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1bffff640 1 -- 172.21.15.102:0/2022783231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c00ff670 msgr2=0x7fa1c0194e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1bffff640 1 --2- 172.21.15.102:0/2022783231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c00ff670 0x7fa1c0194e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1bffff640 1 -- 172.21.15.102:0/2022783231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac009560 con 0x7fa1c0100860 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1bffff640 1 --2- 172.21.15.102:0/2022783231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1c0100860 0x7fa1c01953d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009230 tx=0x7fa1ac004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.574+0000 7fa1bdffb640 1 -- 172.21.15.102:0/2022783231 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1ac004520 con 0x7fa1c0100860 2024-03-26T18:52:46.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.575+0000 7fa1c6b39640 1 -- 172.21.15.102:0/2022783231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c019a540 con 0x7fa1c0100860 2024-03-26T18:52:46.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.575+0000 7fa1bdffb640 1 -- 172.21.15.102:0/2022783231 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa1ac0054b0 con 0x7fa1c0100860 2024-03-26T18:52:46.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.575+0000 7fa1bdffb640 1 -- 172.21.15.102:0/2022783231 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1ac034670 con 0x7fa1c0100860 2024-03-26T18:52:46.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.575+0000 7fa1c6b39640 1 -- 172.21.15.102:0/2022783231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c019aa40 con 0x7fa1c0100860 2024-03-26T18:52:46.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.576+0000 7fa1c6b39640 1 -- 172.21.15.102:0/2022783231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c01083b0 con 0x7fa1c0100860 2024-03-26T18:52:46.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.576+0000 7fa1bdffb640 1 -- 172.21.15.102:0/2022783231 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fa1ac0038b0 con 0x7fa1c0100860 2024-03-26T18:52:46.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.576+0000 7fa1bdffb640 1 --2- 172.21.15.102:0/2022783231 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa1b006c6b0 0x7fa1b006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:46.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.577+0000 7fa1bdffb640 1 -- 172.21.15.102:0/2022783231 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa1ac0adeb0 con 0x7fa1c0100860 2024-03-26T18:52:46.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.577+0000 7fa1c48d5640 1 --2- 172.21.15.102:0/2022783231 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa1b006c6b0 0x7fa1b006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:46.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.577+0000 7fa1c48d5640 1 --2- 172.21.15.102:0/2022783231 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa1b006c6b0 0x7fa1b006eb60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4005fd0 tx=0x7fa1b4005850 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:46.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.581+0000 7fa1bdffb640 1 -- 172.21.15.102:0/2022783231 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa1ac078530 con 0x7fa1c0100860 2024-03-26T18:52:46.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.760+0000 7fa1c6b39640 1 -- 172.21.15.102:0/2022783231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1c019ac60 con 0x7fa1c0100860 2024-03-26T18:52:46.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.761+0000 7fa1bdffb640 1 -- 172.21.15.102:0/2022783231 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+15159 (secure 0 0 0) 0x7fa1ac08fe60 con 0x7fa1c0100860 2024-03-26T18:52:46.754 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:52:46.754 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":47,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","created":"2024-03-26T18:45:25.446329+0000","modified":"2024-03-26T18:52:35.022281+0000","last_up_change":"2024-03-26T18:51:04.568813+0000","last_in_change":"2024-03-26T18:50:45.452094+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T18:49:50.172501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-26T18:52:17.727534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-26T18:52:18.482241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5458d389-eb88-4798-92bc-b442168b37f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6803","nonce":689794310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6805","nonce":689794310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6809","nonce":689794310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":689794310},{"type":"v1","addr":"172.21.15.102:6807","nonce":689794310}]},"public_addr":"172.21.15.102:6803/689794310","cluster_addr":"172.21.15.102:6805/689794310","heartbeat_back_addr":"172.21.15.102:6809/689794310","heartbeat_front_addr":"172.21.15.102:6807/689794310","state":["exists","up"]},{"osd":1,"uuid":"87508f24-8b3c-4f6d-add2-00ad59300da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6811","nonce":391196241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6813","nonce":391196241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6817","nonce":391196241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":391196241},{"type":"v1","addr":"172.21.15.102:6815","nonce":391196241}]},"public_addr":"172.21.15.102:6811/391196241","cluster_addr":"172.21.15.102:6813/391196241","heartbeat_back_addr":"172.21.15.102:6817/391196241","heartbeat_front_addr":"172.21.15.102:6815/391196241","state":["exists","up"]},{"osd":2,"uuid":"c0179546-10c5-41f2-af0c-be93c2bbf5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6819","nonce":3930922840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6821","nonce":3930922840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6825","nonce":3930922840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":3930922840},{"type":"v1","addr":"172.21.15.102:6823","nonce":3930922840}]},"public_addr":"172.21.15.102:6819/3930922840","cluster_addr":"172.21.15.102:6821/3930922840","heartbeat_back_addr":"172.21.15.102:6825/3930922840","heartbeat_front_addr":"172.21.15.102:6823/3930922840","state":["exists","up"]},{"osd":3,"uuid":"4d524f7b-8b31-469b-b0c2-b94079febaeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6801","nonce":1428928793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6803","nonce":1428928793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6807","nonce":1428928793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1428928793},{"type":"v1","addr":"172.21.15.190:6805","nonce":1428928793}]},"public_addr":"172.21.15.190:6801/1428928793","cluster_addr":"172.21.15.190:6803/1428928793","heartbeat_back_addr":"172.21.15.190:6807/1428928793","heartbeat_front_addr":"172.21.15.190:6805/1428928793","state":["exists","up"]},{"osd":4,"uuid":"865baef8-dd72-4449-b3a4-893801bc2958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6809","nonce":320320306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6811","nonce":320320306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6815","nonce":320320306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":320320306},{"type":"v1","addr":"172.21.15.190:6813","nonce":320320306}]},"public_addr":"172.21.15.190:6809/320320306","cluster_addr":"172.21.15.190:6811/320320306","heartbeat_back_addr":"172.21.15.190:6815/320320306","heartbeat_front_addr":"172.21.15.190:6813/320320306","state":["exists","up"]},{"osd":5,"uuid":"5fd67007-d7ae-43b6-8002-258fbfc5a797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6817","nonce":3245135415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6819","nonce":3245135415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6823","nonce":3245135415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":3245135415},{"type":"v1","addr":"172.21.15.190:6821","nonce":3245135415}]},"public_addr":"172.21.15.190:6817/3245135415","cluster_addr":"172.21.15.190:6819/3245135415","heartbeat_back_addr":"172.21.15.190:6823/3245135415","heartbeat_front_addr":"172.21.15.190:6821/3245135415","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:48:51.274164+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:18.020320+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:46.066766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:10.365857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:36.582495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:51:02.797707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6827/2550866692":"2024-03-27T18:52:35.022250+0000","172.21.15.102:0/2752725029":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6800/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.102:6801/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:6801/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/1380712530":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/4276276941":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1295801893":"2024-03-27T18:45:55.925005+0000","172.21.15.102:6800/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6826/2550866692":"2024-03-27T18:52:35.022250+0000","172.21.15.102:6801/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/2686253586":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/1212290737":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/1842664791":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1994330698":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/3353740154":"2024-03-27T18:47:33.839189+0000","172.21.15.102:6800/3517043895":"2024-03-27T18:47:33.839189+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T18:52:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.763+0000 7fa1c6b39640 1 -- 172.21.15.102:0/2022783231 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa1b006c6b0 msgr2=0x7fa1b006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.763+0000 7fa1c6b39640 1 --2- 172.21.15.102:0/2022783231 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa1b006c6b0 0x7fa1b006eb60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4005fd0 tx=0x7fa1b4005850 comp rx=0 tx=0).stop 2024-03-26T18:52:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.763+0000 7fa1c6b39640 1 -- 172.21.15.102:0/2022783231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1c0100860 msgr2=0x7fa1c01953d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.763+0000 7fa1c6b39640 1 --2- 172.21.15.102:0/2022783231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1c0100860 0x7fa1c01953d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009230 tx=0x7fa1ac004e30 comp rx=0 tx=0).stop 2024-03-26T18:52:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.763+0000 7fa1c6b39640 1 -- 172.21.15.102:0/2022783231 shutdown_connections 2024-03-26T18:52:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.763+0000 7fa1c6b39640 1 --2- 172.21.15.102:0/2022783231 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa1b006c6b0 0x7fa1b006eb60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.763+0000 7fa1c6b39640 1 --2- 172.21.15.102:0/2022783231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1c0100860 0x7fa1c01953d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.763+0000 7fa1c6b39640 1 --2- 172.21.15.102:0/2022783231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c00ff670 0x7fa1c0194e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.763+0000 7fa1c6b39640 1 -- 172.21.15.102:0/2022783231 >> 172.21.15.102:0/2022783231 conn(0x7fa1c00fabd0 msgr2=0x7fa1c0103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.763+0000 7fa1c6b39640 1 -- 172.21.15.102:0/2022783231 shutdown_connections 2024-03-26T18:52:46.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.763+0000 7fa1c6b39640 1 -- 172.21.15.102:0/2022783231 wait complete. 2024-03-26T18:52:46.767 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T18:49:50.172501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-26T18:52:17.727534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-26T18:52:18.482241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T18:52:46.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T18:52:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.880+0000 7fa50aa2d640 1 -- 172.21.15.102:0/557032942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040ff730 msgr2=0x7fa5040ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.880+0000 7fa50aa2d640 1 --2- 172.21.15.102:0/557032942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040ff730 0x7fa5040ffba0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009880 tx=0x7fa4f802bce0 comp rx=0 tx=0).stop 2024-03-26T18:52:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.880+0000 7fa50aa2d640 1 -- 172.21.15.102:0/557032942 shutdown_connections 2024-03-26T18:52:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.880+0000 7fa50aa2d640 1 --2- 172.21.15.102:0/557032942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040ff730 0x7fa5040ffba0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.880+0000 7fa50aa2d640 1 --2- 172.21.15.102:0/557032942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5040fe540 0x7fa5040fe950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.880+0000 7fa50aa2d640 1 -- 172.21.15.102:0/557032942 >> 172.21.15.102:0/557032942 conn(0x7fa5040f9a80 msgr2=0x7fa5040fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.880+0000 7fa50aa2d640 1 -- 172.21.15.102:0/557032942 shutdown_connections 2024-03-26T18:52:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.880+0000 7fa50aa2d640 1 -- 172.21.15.102:0/557032942 wait complete. 2024-03-26T18:52:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa50aa2d640 1 Processor -- start 2024-03-26T18:52:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa50aa2d640 1 -- start start 2024-03-26T18:52:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa50aa2d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5040fe540 0x7fa504067e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa50aa2d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040ff730 0x7fa504199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa50aa2d640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50419a240 con 0x7fa5040ff730 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa50aa2d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50419a380 con 0x7fa5040fe540 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa5037fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040ff730 0x7fa504199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa5037fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040ff730 0x7fa504199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45566/0 (socket says 172.21.15.102:45566) 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa5037fe640 1 -- 172.21.15.102:0/1697130707 learned_addr learned my addr 172.21.15.102:0/1697130707 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa503fff640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5040fe540 0x7fa504067e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa5037fe640 1 -- 172.21.15.102:0/1697130707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5040fe540 msgr2=0x7fa504067e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa5037fe640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5040fe540 0x7fa504067e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa5037fe640 1 -- 172.21.15.102:0/1697130707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f8009560 con 0x7fa5040ff730 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.881+0000 7fa503fff640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5040fe540 0x7fa504067e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.882+0000 7fa5037fe640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040ff730 0x7fa504199c70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009230 tx=0x7fa4f802cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.882+0000 7fa5017fa640 1 -- 172.21.15.102:0/1697130707 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4f8004420 con 0x7fa5040ff730 2024-03-26T18:52:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.882+0000 7fa5017fa640 1 -- 172.21.15.102:0/1697130707 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa4f80054b0 con 0x7fa5040ff730 2024-03-26T18:52:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.882+0000 7fa50aa2d640 1 -- 172.21.15.102:0/1697130707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50419a5d0 con 0x7fa5040ff730 2024-03-26T18:52:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.882+0000 7fa5017fa640 1 -- 172.21.15.102:0/1697130707 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4f8034440 con 0x7fa5040ff730 2024-03-26T18:52:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.882+0000 7fa50aa2d640 1 -- 172.21.15.102:0/1697130707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50419aad0 con 0x7fa5040ff730 2024-03-26T18:52:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.883+0000 7fa50aa2d640 1 -- 172.21.15.102:0/1697130707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa504107280 con 0x7fa5040ff730 2024-03-26T18:52:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.883+0000 7fa5017fa640 1 -- 172.21.15.102:0/1697130707 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fa4f80037d0 con 0x7fa5040ff730 2024-03-26T18:52:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.883+0000 7fa5017fa640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4ec06c5f0 0x7fa4ec06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.884+0000 7fa5017fa640 1 -- 172.21.15.102:0/1697130707 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa4f80ade10 con 0x7fa5040ff730 2024-03-26T18:52:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.884+0000 7fa503fff640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4ec06c5f0 0x7fa4ec06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.884+0000 7fa503fff640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4ec06c5f0 0x7fa4ec06eaa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa5040ff590 tx=0x7fa4f4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:46.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:46.888+0000 7fa5017fa640 1 -- 172.21.15.102:0/1697130707 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa4f8078490 con 0x7fa5040ff730 2024-03-26T18:52:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.059+0000 7fa50aa2d640 1 -- 172.21.15.102:0/1697130707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa50419acf0 con 0x7fa5040ff730 2024-03-26T18:52:47.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.059+0000 7fa5017fa640 1 -- 172.21.15.102:0/1697130707 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa4f804b020 con 0x7fa5040ff730 2024-03-26T18:52:47.052 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-26T18:52:47.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.061+0000 7fa50aa2d640 1 -- 172.21.15.102:0/1697130707 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4ec06c5f0 msgr2=0x7fa4ec06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.061+0000 7fa50aa2d640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4ec06c5f0 0x7fa4ec06eaa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa5040ff590 tx=0x7fa4f4005f90 comp rx=0 tx=0).stop 2024-03-26T18:52:47.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.061+0000 7fa50aa2d640 1 -- 172.21.15.102:0/1697130707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040ff730 msgr2=0x7fa504199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.061+0000 7fa50aa2d640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040ff730 0x7fa504199c70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009230 tx=0x7fa4f802cec0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.061+0000 7fa50aa2d640 1 -- 172.21.15.102:0/1697130707 shutdown_connections 2024-03-26T18:52:47.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.061+0000 7fa50aa2d640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4ec06c5f0 0x7fa4ec06eaa0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.061+0000 7fa50aa2d640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040ff730 0x7fa504199c70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.061+0000 7fa50aa2d640 1 --2- 172.21.15.102:0/1697130707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5040fe540 0x7fa504067e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.061+0000 7fa50aa2d640 1 -- 172.21.15.102:0/1697130707 >> 172.21.15.102:0/1697130707 conn(0x7fa5040f9a80 msgr2=0x7fa504102960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:47.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.061+0000 7fa50aa2d640 1 -- 172.21.15.102:0/1697130707 shutdown_connections 2024-03-26T18:52:47.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.061+0000 7fa50aa2d640 1 -- 172.21.15.102:0/1697130707 wait complete. 2024-03-26T18:52:47.065 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-26T18:52:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.184+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3440710520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53f80ff760 msgr2=0x7f53f80ffbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.184+0000 7f53ff5b1640 1 --2- 172.21.15.102:0/3440710520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53f80ff760 0x7f53f80ffbd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009880 tx=0x7f53e802bce0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.184+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3440710520 shutdown_connections 2024-03-26T18:52:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.184+0000 7f53ff5b1640 1 --2- 172.21.15.102:0/3440710520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53f80ff760 0x7f53f80ffbd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.184+0000 7f53ff5b1640 1 --2- 172.21.15.102:0/3440710520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80fe570 0x7f53f80fe980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.184+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3440710520 >> 172.21.15.102:0/3440710520 conn(0x7f53f80f9ad0 msgr2=0x7f53f80fbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.184+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3440710520 shutdown_connections 2024-03-26T18:52:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.184+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3440710520 wait complete. 2024-03-26T18:52:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53ff5b1640 1 Processor -- start 2024-03-26T18:52:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53ff5b1640 1 -- start start 2024-03-26T18:52:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53ff5b1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80fe570 0x7f53f8194f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53ff5b1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53f80ff760 0x7f53f8195480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53ff5b1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8195a50 con 0x7f53f80ff760 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53ff5b1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8195bc0 con 0x7f53f80fe570 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53fcb4c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53f80ff760 0x7f53f8195480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53fcb4c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53f80ff760 0x7f53f8195480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45588/0 (socket says 172.21.15.102:45588) 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53fcb4c640 1 -- 172.21.15.102:0/3148137601 learned_addr learned my addr 172.21.15.102:0/3148137601 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53fd34d640 1 --2- 172.21.15.102:0/3148137601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80fe570 0x7f53f8194f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53fcb4c640 1 -- 172.21.15.102:0/3148137601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80fe570 msgr2=0x7f53f8194f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53fcb4c640 1 --2- 172.21.15.102:0/3148137601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80fe570 0x7f53f8194f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.185+0000 7f53fcb4c640 1 -- 172.21.15.102:0/3148137601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e8009560 con 0x7f53f80ff760 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.186+0000 7f53fcb4c640 1 --2- 172.21.15.102:0/3148137601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53f80ff760 0x7f53f8195480 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009230 tx=0x7f53e8004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.186+0000 7f53ee7fc640 1 -- 172.21.15.102:0/3148137601 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53e8004520 con 0x7f53f80ff760 2024-03-26T18:52:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.186+0000 7f53ee7fc640 1 -- 172.21.15.102:0/3148137601 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f53e80054b0 con 0x7f53f80ff760 2024-03-26T18:52:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.186+0000 7f53ee7fc640 1 -- 172.21.15.102:0/3148137601 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53e8034670 con 0x7f53f80ff760 2024-03-26T18:52:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.186+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3148137601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f819a5f0 con 0x7f53f80ff760 2024-03-26T18:52:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.186+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3148137601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f819aaf0 con 0x7f53f80ff760 2024-03-26T18:52:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.187+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3148137601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f8108380 con 0x7f53f80ff760 2024-03-26T18:52:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.187+0000 7f53ee7fc640 1 -- 172.21.15.102:0/3148137601 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f53e80038b0 con 0x7f53f80ff760 2024-03-26T18:52:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.188+0000 7f53ee7fc640 1 --2- 172.21.15.102:0/3148137601 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f53e406c5f0 0x7f53e406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.188+0000 7f53fd34d640 1 --2- 172.21.15.102:0/3148137601 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f53e406c5f0 0x7f53e406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.188+0000 7f53ee7fc640 1 -- 172.21.15.102:0/3148137601 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f53e80adea0 con 0x7f53f80ff760 2024-03-26T18:52:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.188+0000 7f53fd34d640 1 --2- 172.21.15.102:0/3148137601 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f53e406c5f0 0x7f53e406eaa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f53f4005d00 tx=0x7f53f4005c50 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:47.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.192+0000 7f53ee7fc640 1 -- 172.21.15.102:0/3148137601 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f53e8078480 con 0x7f53f80ff760 2024-03-26T18:52:47.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.373+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3148137601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f53f819b230 con 0x7f53f80ff760 2024-03-26T18:52:47.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.373+0000 7f53ee7fc640 1 -- 172.21.15.102:0/3148137601 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f53e804b020 con 0x7f53f80ff760 2024-03-26T18:52:47.366 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-26T18:52:47.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.375+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3148137601 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f53e406c5f0 msgr2=0x7f53e406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.375+0000 7f53ff5b1640 1 --2- 172.21.15.102:0/3148137601 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f53e406c5f0 0x7f53e406eaa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f53f4005d00 tx=0x7f53f4005c50 comp rx=0 tx=0).stop 2024-03-26T18:52:47.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.375+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3148137601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53f80ff760 msgr2=0x7f53f8195480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.375+0000 7f53ff5b1640 1 --2- 172.21.15.102:0/3148137601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53f80ff760 0x7f53f8195480 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009230 tx=0x7f53e8004e30 comp rx=0 tx=0).stop 2024-03-26T18:52:47.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.375+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3148137601 shutdown_connections 2024-03-26T18:52:47.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.375+0000 7f53ff5b1640 1 --2- 172.21.15.102:0/3148137601 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f53e406c5f0 0x7f53e406eaa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.375+0000 7f53ff5b1640 1 --2- 172.21.15.102:0/3148137601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53f80ff760 0x7f53f8195480 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.375+0000 7f53ff5b1640 1 --2- 172.21.15.102:0/3148137601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80fe570 0x7f53f8194f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.375+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3148137601 >> 172.21.15.102:0/3148137601 conn(0x7f53f80f9ad0 msgr2=0x7f53f8102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:47.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.375+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3148137601 shutdown_connections 2024-03-26T18:52:47.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.375+0000 7f53ff5b1640 1 -- 172.21.15.102:0/3148137601 wait complete. 2024-03-26T18:52:47.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:47 smithi102 ceph-mon[29002]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T18:52:47.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:47 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2022783231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T18:52:47.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:47 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1697130707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T18:52:47.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-26T18:52:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.495+0000 7f6f753bb640 1 -- 172.21.15.102:0/3471757833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f701029e0 msgr2=0x7f6f70104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.495+0000 7f6f753bb640 1 --2- 172.21.15.102:0/3471757833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f701029e0 0x7f6f70104dc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6f600098b0 tx=0x7f6f6002bcd0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.495+0000 7f6f753bb640 1 -- 172.21.15.102:0/3471757833 shutdown_connections 2024-03-26T18:52:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.495+0000 7f6f753bb640 1 --2- 172.21.15.102:0/3471757833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f701029e0 0x7f6f70104dc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.495+0000 7f6f753bb640 1 --2- 172.21.15.102:0/3471757833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f701000c0 0x7f6f701024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.495+0000 7f6f753bb640 1 -- 172.21.15.102:0/3471757833 >> 172.21.15.102:0/3471757833 conn(0x7f6f700f9a80 msgr2=0x7f6f700fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.495+0000 7f6f753bb640 1 -- 172.21.15.102:0/3471757833 shutdown_connections 2024-03-26T18:52:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.495+0000 7f6f753bb640 1 -- 172.21.15.102:0/3471757833 wait complete. 2024-03-26T18:52:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.496+0000 7f6f753bb640 1 Processor -- start 2024-03-26T18:52:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.496+0000 7f6f753bb640 1 -- start start 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.496+0000 7f6f753bb640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f701000c0 0x7f6f70194ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.496+0000 7f6f753bb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f701029e0 0x7f6f70195530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.496+0000 7f6f753bb640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70195b00 con 0x7f6f701000c0 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.496+0000 7f6f753bb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70195c70 con 0x7f6f701029e0 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.496+0000 7f6f6effd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f701000c0 0x7f6f70194ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.496+0000 7f6f6effd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f701000c0 0x7f6f70194ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45594/0 (socket says 172.21.15.102:45594) 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.496+0000 7f6f6effd640 1 -- 172.21.15.102:0/3694271874 learned_addr learned my addr 172.21.15.102:0/3694271874 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.496+0000 7f6f6e7fc640 1 --2- 172.21.15.102:0/3694271874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f701029e0 0x7f6f70195530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.497+0000 7f6f6effd640 1 -- 172.21.15.102:0/3694271874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f701029e0 msgr2=0x7f6f70195530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.497+0000 7f6f6effd640 1 --2- 172.21.15.102:0/3694271874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f701029e0 0x7f6f70195530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.497+0000 7f6f6effd640 1 -- 172.21.15.102:0/3694271874 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f60009560 con 0x7f6f701000c0 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.497+0000 7f6f6effd640 1 --2- 172.21.15.102:0/3694271874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f701000c0 0x7f6f70194ff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400e970 tx=0x7f6f6400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.497+0000 7f6f57fff640 1 -- 172.21.15.102:0/3694271874 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f6400cc90 con 0x7f6f701000c0 2024-03-26T18:52:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.497+0000 7f6f753bb640 1 -- 172.21.15.102:0/3694271874 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7019a700 con 0x7f6f701000c0 2024-03-26T18:52:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.497+0000 7f6f57fff640 1 -- 172.21.15.102:0/3694271874 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6f6400cdf0 con 0x7f6f701000c0 2024-03-26T18:52:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.497+0000 7f6f57fff640 1 -- 172.21.15.102:0/3694271874 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f64010430 con 0x7f6f701000c0 2024-03-26T18:52:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.497+0000 7f6f753bb640 1 -- 172.21.15.102:0/3694271874 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7019ac30 con 0x7f6f701000c0 2024-03-26T18:52:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.498+0000 7f6f753bb640 1 -- 172.21.15.102:0/3694271874 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f70108220 con 0x7f6f701000c0 2024-03-26T18:52:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.499+0000 7f6f57fff640 1 -- 172.21.15.102:0/3694271874 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f6f640046c0 con 0x7f6f701000c0 2024-03-26T18:52:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.499+0000 7f6f57fff640 1 --2- 172.21.15.102:0/3694271874 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f5806c640 0x7f6f5806eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.500+0000 7f6f57fff640 1 -- 172.21.15.102:0/3694271874 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6f64014070 con 0x7f6f701000c0 2024-03-26T18:52:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.500+0000 7f6f6e7fc640 1 --2- 172.21.15.102:0/3694271874 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f5806c640 0x7f6f5806eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.500+0000 7f6f6e7fc640 1 --2- 172.21.15.102:0/3694271874 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f5806c640 0x7f6f5806eaf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009880 tx=0x7f6f60005fd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:47.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.508+0000 7f6f57fff640 1 -- 172.21.15.102:0/3694271874 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6f6408f030 con 0x7f6f701000c0 2024-03-26T18:52:47.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:47 smithi190 ceph-mon[35043]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T18:52:47.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:47 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2022783231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T18:52:47.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:47 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/1697130707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T18:52:47.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.681+0000 7f6f753bb640 1 -- 172.21.15.102:0/3694271874 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f6f7019b0b0 con 0x7f6f701000c0 2024-03-26T18:52:47.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.681+0000 7f6f57fff640 1 -- 172.21.15.102:0/3694271874 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6f64056570 con 0x7f6f701000c0 2024-03-26T18:52:47.674 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-26T18:52:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.683+0000 7f6f753bb640 1 -- 172.21.15.102:0/3694271874 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f5806c640 msgr2=0x7f6f5806eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.683+0000 7f6f753bb640 1 --2- 172.21.15.102:0/3694271874 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f5806c640 0x7f6f5806eaf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009880 tx=0x7f6f60005fd0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.683+0000 7f6f753bb640 1 -- 172.21.15.102:0/3694271874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f701000c0 msgr2=0x7f6f70194ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.683+0000 7f6f753bb640 1 --2- 172.21.15.102:0/3694271874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f701000c0 0x7f6f70194ff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400e970 tx=0x7f6f6400ed70 comp rx=0 tx=0).stop 2024-03-26T18:52:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.683+0000 7f6f753bb640 1 -- 172.21.15.102:0/3694271874 shutdown_connections 2024-03-26T18:52:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.683+0000 7f6f753bb640 1 --2- 172.21.15.102:0/3694271874 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6f5806c640 0x7f6f5806eaf0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.683+0000 7f6f753bb640 1 --2- 172.21.15.102:0/3694271874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f701029e0 0x7f6f70195530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.683+0000 7f6f753bb640 1 --2- 172.21.15.102:0/3694271874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f701000c0 0x7f6f70194ff0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.683+0000 7f6f753bb640 1 -- 172.21.15.102:0/3694271874 >> 172.21.15.102:0/3694271874 conn(0x7f6f700f9a80 msgr2=0x7f6f700fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.683+0000 7f6f753bb640 1 -- 172.21.15.102:0/3694271874 shutdown_connections 2024-03-26T18:52:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.683+0000 7f6f753bb640 1 -- 172.21.15.102:0/3694271874 wait complete. 2024-03-26T18:52:47.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T18:52:47.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.795+0000 7f6add4d2640 1 -- 172.21.15.102:0/989867113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ad80ff740 msgr2=0x7f6ad80ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.795+0000 7f6add4d2640 1 --2- 172.21.15.102:0/989867113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ad80ff740 0x7f6ad80ffbb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009880 tx=0x7f6ac002bce0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.795+0000 7f6add4d2640 1 -- 172.21.15.102:0/989867113 shutdown_connections 2024-03-26T18:52:47.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.795+0000 7f6add4d2640 1 --2- 172.21.15.102:0/989867113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ad80ff740 0x7f6ad80ffbb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.795+0000 7f6add4d2640 1 --2- 172.21.15.102:0/989867113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ad80fe550 0x7f6ad80fe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.795+0000 7f6add4d2640 1 -- 172.21.15.102:0/989867113 >> 172.21.15.102:0/989867113 conn(0x7f6ad80f9af0 msgr2=0x7f6ad80fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:47.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.795+0000 7f6add4d2640 1 -- 172.21.15.102:0/989867113 shutdown_connections 2024-03-26T18:52:47.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.795+0000 7f6add4d2640 1 -- 172.21.15.102:0/989867113 wait complete. 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6add4d2640 1 Processor -- start 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6add4d2640 1 -- start start 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6add4d2640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ad80fe550 0x7f6ad8193e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6add4d2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ad80ff740 0x7f6ad8194380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6add4d2640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad8194950 con 0x7f6ad80fe550 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6add4d2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad8194ac0 con 0x7f6ad80ff740 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6ad6ffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ad80fe550 0x7f6ad8193e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6ad6ffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ad80fe550 0x7f6ad8193e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45618/0 (socket says 172.21.15.102:45618) 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6ad6ffd640 1 -- 172.21.15.102:0/1153142503 learned_addr learned my addr 172.21.15.102:0/1153142503 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6acffff640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ad80ff740 0x7f6ad8194380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6ad6ffd640 1 -- 172.21.15.102:0/1153142503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ad80ff740 msgr2=0x7f6ad8194380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6ad6ffd640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ad80ff740 0x7f6ad8194380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6ad6ffd640 1 -- 172.21.15.102:0/1153142503 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac0009560 con 0x7f6ad80fe550 2024-03-26T18:52:47.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6acffff640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ad80ff740 0x7f6ad8194380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T18:52:47.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6ad6ffd640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ad80fe550 0x7f6ad8193e40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800d8d0 tx=0x7f6ac800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:47.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.796+0000 7f6ad4ff9640 1 -- 172.21.15.102:0/1153142503 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ac8009a30 con 0x7f6ad80fe550 2024-03-26T18:52:47.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.797+0000 7f6ad4ff9640 1 -- 172.21.15.102:0/1153142503 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6ac8009b90 con 0x7f6ad80fe550 2024-03-26T18:52:47.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.797+0000 7f6add4d2640 1 -- 172.21.15.102:0/1153142503 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad8199550 con 0x7f6ad80fe550 2024-03-26T18:52:47.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.797+0000 7f6ad4ff9640 1 -- 172.21.15.102:0/1153142503 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ac800f680 con 0x7f6ad80fe550 2024-03-26T18:52:47.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.797+0000 7f6add4d2640 1 -- 172.21.15.102:0/1153142503 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad8199ab0 con 0x7f6ad80fe550 2024-03-26T18:52:47.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.797+0000 7f6add4d2640 1 -- 172.21.15.102:0/1153142503 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad8107290 con 0x7f6ad80fe550 2024-03-26T18:52:47.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.798+0000 7f6ad4ff9640 1 -- 172.21.15.102:0/1153142503 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f6ac8009d00 con 0x7f6ad80fe550 2024-03-26T18:52:47.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.798+0000 7f6ad4ff9640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6ac406c700 0x7f6ac406ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:47.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.798+0000 7f6ad4ff9640 1 -- 172.21.15.102:0/1153142503 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6ac808b7e0 con 0x7f6ad80fe550 2024-03-26T18:52:47.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.798+0000 7f6acffff640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6ac406c700 0x7f6ac406ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:47.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.799+0000 7f6acffff640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6ac406c700 0x7f6ac406ebb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8195350 tx=0x7f6ac002c7c0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:47.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:47.803+0000 7f6ad4ff9640 1 -- 172.21.15.102:0/1153142503 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6ac8055dc0 con 0x7f6ad80fe550 2024-03-26T18:52:48.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.019+0000 7f6add4d2640 1 -- 172.21.15.102:0/1153142503 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ad8199cd0 con 0x7f6ad80fe550 2024-03-26T18:52:48.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.019+0000 7f6ad4ff9640 1 -- 172.21.15.102:0/1153142503 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4760 (secure 0 0 0) 0x7f6ac80593e0 con 0x7f6ad80fe550 2024-03-26T18:52:48.012 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:52:48.012 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14502,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2674848848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2674848848},{"type":"v1","addr":"172.21.15.102:6827","nonce":2674848848}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T18:52:39.046175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14490,"mds_1":24283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14490":{"gid":14490,"name":"cephfs.smithi102.hrbrsp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.102:6829/989894576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":989894576},{"type":"v1","addr":"172.21.15.102:6829","nonce":989894576}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24283":{"gid":24283,"name":"cephfs.smithi190.vykizb","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.190:6825/2769333078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2769333078},{"type":"v1","addr":"172.21.15.190:6825","nonce":2769333078}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-26T18:52:48.012 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 17 2024-03-26T18:52:48.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.021+0000 7f6add4d2640 1 -- 172.21.15.102:0/1153142503 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6ac406c700 msgr2=0x7f6ac406ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:48.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.021+0000 7f6add4d2640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6ac406c700 0x7f6ac406ebb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8195350 tx=0x7f6ac002c7c0 comp rx=0 tx=0).stop 2024-03-26T18:52:48.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.021+0000 7f6add4d2640 1 -- 172.21.15.102:0/1153142503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ad80fe550 msgr2=0x7f6ad8193e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:48.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.021+0000 7f6add4d2640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ad80fe550 0x7f6ad8193e40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800d8d0 tx=0x7f6ac800dd90 comp rx=0 tx=0).stop 2024-03-26T18:52:48.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.021+0000 7f6add4d2640 1 -- 172.21.15.102:0/1153142503 shutdown_connections 2024-03-26T18:52:48.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.021+0000 7f6add4d2640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f6ac406c700 0x7f6ac406ebb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:48.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.021+0000 7f6add4d2640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ad80ff740 0x7f6ad8194380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:48.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.021+0000 7f6add4d2640 1 --2- 172.21.15.102:0/1153142503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ad80fe550 0x7f6ad8193e40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:48.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.021+0000 7f6add4d2640 1 -- 172.21.15.102:0/1153142503 >> 172.21.15.102:0/1153142503 conn(0x7f6ad80f9af0 msgr2=0x7f6ad8102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:48.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.021+0000 7f6add4d2640 1 -- 172.21.15.102:0/1153142503 shutdown_connections 2024-03-26T18:52:48.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:48.021+0000 7f6add4d2640 1 -- 172.21.15.102:0/1153142503 wait complete. 2024-03-26T18:52:48.025 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 17, 'max_mds': 2, 'flags': 18} 2024-03-26T18:52:48.025 INFO:teuthology.run_tasks:Running task kclient... 2024-03-26T18:52:48.052 INFO:tasks.kclient:Mounting kernel clients... 2024-03-26T18:52:48.053 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-03-26T18:52:48.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:48.053 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-26T18:52:48.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:48.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-03-26T18:52:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:Cannot find device "ceph-brx" 2024-03-26T18:52:48.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T18:52:48.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:48.142 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2024-03-26T18:52:48.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:48.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link delete ceph-brx 2024-03-26T18:52:48.234 INFO:teuthology.orchestra.run.smithi190.stderr:Cannot find device "ceph-brx" 2024-03-26T18:52:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T18:52:48.236 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-03-26T18:52:48.236 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-26T18:52:48.236 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-26T18:52:48.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-26T18:52:48.261 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:48 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3148137601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-26T18:52:48.261 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:48 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3694271874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-26T18:52:48.261 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:48 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1153142503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T18:52:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:48 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3148137601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-26T18:52:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:48 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3694271874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-26T18:52:48.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:48 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/1153142503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T18:52:48.635 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:49.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:49 smithi102 ceph-mon[29002]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-26T18:52:49.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:49 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:52:49.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:49 smithi190 ceph-mon[35043]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-26T18:52:49.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:49 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:52:50.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.358+0000 7fcc72c09700 1 -- 172.21.15.102:0/287033144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc6c100460 msgr2=0x7fcc6c100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:50.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.358+0000 7fcc72c09700 1 --2- 172.21.15.102:0/287033144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc6c100460 0x7fcc6c100830 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcc60009910 tx=0x7fcc60009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:50.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.358+0000 7fcc72c09700 1 -- 172.21.15.102:0/287033144 shutdown_connections 2024-03-26T18:52:50.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.358+0000 7fcc72c09700 1 --2- 172.21.15.102:0/287033144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc6c100e00 0x7fcc6c104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.358+0000 7fcc72c09700 1 --2- 172.21.15.102:0/287033144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc6c100460 0x7fcc6c100830 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.358+0000 7fcc72c09700 1 -- 172.21.15.102:0/287033144 >> 172.21.15.102:0/287033144 conn(0x7fcc6c0fbd00 msgr2=0x7fcc6c0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.358+0000 7fcc72c09700 1 -- 172.21.15.102:0/287033144 shutdown_connections 2024-03-26T18:52:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.358+0000 7fcc72c09700 1 -- 172.21.15.102:0/287033144 wait complete. 2024-03-26T18:52:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.358+0000 7fcc72c09700 1 Processor -- start 2024-03-26T18:52:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc72c09700 1 -- start start 2024-03-26T18:52:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc72c09700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc6c100460 0x7fcc6c195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc72c09700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc6c100e00 0x7fcc6c196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc72c09700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c196a00 con 0x7fcc6c100e00 2024-03-26T18:52:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc72c09700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c19a790 con 0x7fcc6c100460 2024-03-26T18:52:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc6bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc6c100e00 0x7fcc6c196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc6bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc6c100e00 0x7fcc6c196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52112/0 (socket says 172.21.15.102:52112) 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc6bfff700 1 -- 172.21.15.102:0/3118845205 learned_addr learned my addr 172.21.15.102:0/3118845205 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc709a5700 1 --2- 172.21.15.102:0/3118845205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc6c100460 0x7fcc6c195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc6bfff700 1 -- 172.21.15.102:0/3118845205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc6c100460 msgr2=0x7fcc6c195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc6bfff700 1 --2- 172.21.15.102:0/3118845205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc6c100460 0x7fcc6c195de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc6bfff700 1 -- 172.21.15.102:0/3118845205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc600095f0 con 0x7fcc6c100e00 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc6bfff700 1 --2- 172.21.15.102:0/3118845205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc6c100e00 0x7fcc6c196320 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800c8f0 tx=0x7fcc5800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc69ffb700 1 -- 172.21.15.102:0/3118845205 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc58007ad0 con 0x7fcc6c100e00 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc72c09700 1 -- 172.21.15.102:0/3118845205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6c19aa70 con 0x7fcc6c100e00 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc69ffb700 1 -- 172.21.15.102:0/3118845205 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcc5800ce90 con 0x7fcc6c100e00 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.359+0000 7fcc69ffb700 1 -- 172.21.15.102:0/3118845205 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc58018630 con 0x7fcc6c100e00 2024-03-26T18:52:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.360+0000 7fcc72c09700 1 -- 172.21.15.102:0/3118845205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6c19afc0 con 0x7fcc6c100e00 2024-03-26T18:52:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.361+0000 7fcc72c09700 1 -- 172.21.15.102:0/3118845205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6c108770 con 0x7fcc6c100e00 2024-03-26T18:52:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.361+0000 7fcc69ffb700 1 -- 172.21.15.102:0/3118845205 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fcc58007c30 con 0x7fcc6c100e00 2024-03-26T18:52:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.361+0000 7fcc69ffb700 1 --2- 172.21.15.102:0/3118845205 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcc5c06c680 0x7fcc5c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.361+0000 7fcc69ffb700 1 -- 172.21.15.102:0/3118845205 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcc5808b920 con 0x7fcc6c100e00 2024-03-26T18:52:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.361+0000 7fcc709a5700 1 --2- 172.21.15.102:0/3118845205 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcc5c06c680 0x7fcc5c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:50.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.362+0000 7fcc709a5700 1 --2- 172.21.15.102:0/3118845205 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcc5c06c680 0x7fcc5c06eb30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcc6000b5c0 tx=0x7fcc60005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:50.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.366+0000 7fcc69ffb700 1 -- 172.21.15.102:0/3118845205 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcc58055fa0 con 0x7fcc6c100e00 2024-03-26T18:52:50.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.600+0000 7fcc72c09700 1 -- 172.21.15.102:0/3118845205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcc6c19b2a0 con 0x7fcc6c100e00 2024-03-26T18:52:50.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.600+0000 7fcc69ffb700 1 -- 172.21.15.102:0/3118845205 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+83 (secure 0 0 0) 0x7fcc580595c0 con 0x7fcc6c100e00 2024-03-26T18:52:50.593 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-26T18:52:50.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.602+0000 7fcc72c09700 1 -- 172.21.15.102:0/3118845205 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcc5c06c680 msgr2=0x7fcc5c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:50.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.602+0000 7fcc72c09700 1 --2- 172.21.15.102:0/3118845205 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcc5c06c680 0x7fcc5c06eb30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcc6000b5c0 tx=0x7fcc60005960 comp rx=0 tx=0).stop 2024-03-26T18:52:50.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.603+0000 7fcc72c09700 1 -- 172.21.15.102:0/3118845205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc6c100e00 msgr2=0x7fcc6c196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:50.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.603+0000 7fcc72c09700 1 --2- 172.21.15.102:0/3118845205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc6c100e00 0x7fcc6c196320 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800c8f0 tx=0x7fcc5800ccb0 comp rx=0 tx=0).stop 2024-03-26T18:52:50.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.603+0000 7fcc72c09700 1 -- 172.21.15.102:0/3118845205 shutdown_connections 2024-03-26T18:52:50.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.603+0000 7fcc72c09700 1 --2- 172.21.15.102:0/3118845205 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcc5c06c680 0x7fcc5c06eb30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:50.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.603+0000 7fcc72c09700 1 --2- 172.21.15.102:0/3118845205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc6c100460 0x7fcc6c195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:50.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.603+0000 7fcc72c09700 1 --2- 172.21.15.102:0/3118845205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc6c100e00 0x7fcc6c196320 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:50.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.603+0000 7fcc72c09700 1 -- 172.21.15.102:0/3118845205 >> 172.21.15.102:0/3118845205 conn(0x7fcc6c0fbd00 msgr2=0x7fcc6c0fd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:50.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.603+0000 7fcc72c09700 1 -- 172.21.15.102:0/3118845205 shutdown_connections 2024-03-26T18:52:50.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:50.603+0000 7fcc72c09700 1 -- 172.21.15.102:0/3118845205 wait complete. 2024-03-26T18:52:50.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:50 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:50.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:50 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:50.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:50 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:50.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:50 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:52:50.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:50 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:50.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:50 smithi190 ceph-mon[35043]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-26T18:52:50.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:50 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:50.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:50 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:50.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:50 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:52:50.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:50 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:52:50.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:50 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:52:50.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:50 smithi102 ceph-mon[29002]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-26T18:52:51.141 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-26T18:52:51.141 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-26T18:52:51.141 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-26T18:52:51.141 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-26T18:52:51.141 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-26T18:52:51.141 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-26T18:52:51.141 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-26T18:52:51.141 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-26T18:52:51.141 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-26T18:52:51.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:51.142 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-26T18:52:51.197 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-26T18:52:51.197 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-26T18:52:51.197 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-26T18:52:51.197 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-26T18:52:51.197 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-26T18:52:51.197 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-26T18:52:51.197 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42289sec preferred_lft 42289sec 2024-03-26T18:52:51.198 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-26T18:52:51.199 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-26T18:52:51.199 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-26T18:52:51.199 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-26T18:52:51.199 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-26T18:52:51.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add name ceph-brx type bridge 2024-03-26T18:52:51.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr flush dev ceph-brx 2024-03-26T18:52:51.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set ceph-brx up 2024-03-26T18:52:51.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-26T18:52:51.199 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-26T18:52:51.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:51.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:51.417 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-26T18:52:51.444 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-26T18:52:51.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:51.445 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-26T18:52:51.523 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-26T18:52:51.523 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-26T18:52:51.523 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-26T18:52:51.523 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-26T18:52:51.523 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-26T18:52:51.523 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-26T18:52:51.524 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-26T18:52:51.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-26T18:52:51.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-26T18:52:51.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-26T18:52:51.524 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-26T18:52:51.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:51.723 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:51 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3118845205' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-26T18:52:51.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:51 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3118845205' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-26T18:52:51.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:51.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:51.839 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-26T18:52:51.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:51.854 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2024-03-26T18:52:51.908 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-26T18:52:51.909 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-26T18:52:51.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-26T18:52:51.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-26T18:52:51.909 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-26T18:52:51.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:52.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:52.032 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-26T18:52:52.033 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-26T18:52:52.033 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-26T18:52:52.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-26T18:52:52.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-26T18:52:52.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-26T18:52:52.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-26T18:52:52.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-26T18:52:52.033 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-26T18:52:52.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:52.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:52.172 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-26T18:52:52.172 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-26T18:52:52.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-03-26T18:52:52.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.0 master ceph-brx 2024-03-26T18:52:52.172 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-26T18:52:52.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:52.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:52.254 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-26T18:52:52.254 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-26T18:52:52.269 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-26T18:52:52.269 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-26T18:52:52.269 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-26T18:52:52.323 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-26T18:52:52.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:52.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-26T18:52:52.428 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-26T18:52:52.428 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-26T18:52:52.428 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-26T18:52:52.533 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-26T18:52:52.533 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-26T18:52:52.533 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-26T18:52:52.533 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.190:3300" 2024-03-26T18:52:52.533 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-26T18:52:52.550 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-26T18:52:52.551 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-26T18:52:52.551 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-26T18:52:52.551 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.190:3300" 2024-03-26T18:52:52.551 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-26T18:52:52.551 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.190:6789" 2024-03-26T18:52:52.551 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.190:6789:/ 2024-03-26T18:52:52.551 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ed4e027c-eba0-11ee-95d0-87774f69a715" will pass to kernel 2024-03-26T18:52:52.552 INFO:teuthology.orchestra.run.smithi102.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-26T18:52:52.552 INFO:teuthology.orchestra.run.smithi102.stdout: You just mounted a file system that supports labels which does not 2024-03-26T18:52:52.552 INFO:teuthology.orchestra.run.smithi102.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-26T18:52:52.552 INFO:teuthology.orchestra.run.smithi102.stdout: applications will generate AVC messages and not be allowed access to 2024-03-26T18:52:52.552 INFO:teuthology.orchestra.run.smithi102.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-26T18:52:52.553 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-26T18:52:52.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:52.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-26T18:52:52.584 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-26T18:52:52.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-26T18:52:52.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:52.685 INFO:teuthology.orchestra.run.smithi102.stdout:client14540 2024-03-26T18:52:52.685 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:52:52.685 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-26T18:52:52.702 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-26T18:52:52.741 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:52 smithi190 ceph-mon[35043]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-26T18:52:52.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-26T18:52:52.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:52.801 INFO:teuthology.orchestra.run.smithi102.stdout:client14540 2024-03-26T18:52:52.802 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-26T18:52:52.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-26T18:52:52.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:52.862 INFO:teuthology.orchestra.run.smithi102.stdout:client14540 2024-03-26T18:52:52.862 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715.client14540/status') 2024-03-26T18:52:52.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:52 smithi102 ceph-mon[29002]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-26T18:52:52.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:52.926 INFO:teuthology.orchestra.run.smithi102.stdout:instance: client.14540 (3)192.168.0.1:0/2355540585 2024-03-26T18:52:52.926 INFO:teuthology.orchestra.run.smithi102.stdout:blocklisted: false 2024-03-26T18:52:52.926 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records in 2024-03-26T18:52:52.926 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records out 2024-03-26T18:52:52.927 INFO:teuthology.orchestra.run.smithi102.stderr:70 bytes copied, 2.0339e-05 s, 3.4 MB/s 2024-03-26T18:52:52.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:52.933 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-03-26T18:52:52.933 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-26T18:52:52.933 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-26T18:52:52.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-26T18:52:53.180 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.780+0000 7fad93903700 1 -- 172.21.15.102:0/942697765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad8c0ffd30 msgr2=0x7fad8c103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.780+0000 7fad93903700 1 --2- 172.21.15.102:0/942697765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad8c0ffd30 0x7fad8c103ce0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fad88009910 tx=0x7fad88009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.780+0000 7fad93903700 1 -- 172.21.15.102:0/942697765 shutdown_connections 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.780+0000 7fad93903700 1 --2- 172.21.15.102:0/942697765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad8c0ffd30 0x7fad8c103ce0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.780+0000 7fad93903700 1 --2- 172.21.15.102:0/942697765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad8c0ff390 0x7fad8c0ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.780+0000 7fad93903700 1 -- 172.21.15.102:0/942697765 >> 172.21.15.102:0/942697765 conn(0x7fad8c074f50 msgr2=0x7fad8c075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.780+0000 7fad93903700 1 -- 172.21.15.102:0/942697765 shutdown_connections 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.780+0000 7fad93903700 1 -- 172.21.15.102:0/942697765 wait complete. 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad93903700 1 Processor -- start 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad93903700 1 -- start start 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad93903700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad8c0ff390 0x7fad8c100d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad93903700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad8c0ffd30 0x7fad8c1012c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad93903700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c101890 con 0x7fad8c0ffd30 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad93903700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c101a00 con 0x7fad8c0ff390 2024-03-26T18:52:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad90e9e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad8c0ffd30 0x7fad8c1012c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad90e9e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad8c0ffd30 0x7fad8c1012c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52126/0 (socket says 172.21.15.102:52126) 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad90e9e700 1 -- 172.21.15.102:0/2504235375 learned_addr learned my addr 172.21.15.102:0/2504235375 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad9169f700 1 --2- 172.21.15.102:0/2504235375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad8c0ff390 0x7fad8c100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad90e9e700 1 -- 172.21.15.102:0/2504235375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad8c0ff390 msgr2=0x7fad8c100d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad90e9e700 1 --2- 172.21.15.102:0/2504235375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad8c0ff390 0x7fad8c100d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad90e9e700 1 -- 172.21.15.102:0/2504235375 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad880095f0 con 0x7fad8c0ffd30 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.781+0000 7fad90e9e700 1 --2- 172.21.15.102:0/2504235375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad8c0ffd30 0x7fad8c1012c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fad880092e0 tx=0x7fad8800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.782+0000 7fad827fc700 1 -- 172.21.15.102:0/2504235375 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad8801d070 con 0x7fad8c0ffd30 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.782+0000 7fad827fc700 1 -- 172.21.15.102:0/2504235375 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fad88022470 con 0x7fad8c0ffd30 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.782+0000 7fad93903700 1 -- 172.21.15.102:0/2504235375 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8c1a2020 con 0x7fad8c0ffd30 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.782+0000 7fad827fc700 1 -- 172.21.15.102:0/2504235375 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad8800f690 con 0x7fad8c0ffd30 2024-03-26T18:52:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.782+0000 7fad93903700 1 -- 172.21.15.102:0/2504235375 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8c1a23d0 con 0x7fad8c0ffd30 2024-03-26T18:52:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.783+0000 7fad93903700 1 -- 172.21.15.102:0/2504235375 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad8c108680 con 0x7fad8c0ffd30 2024-03-26T18:52:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.783+0000 7fad827fc700 1 -- 172.21.15.102:0/2504235375 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fad880225e0 con 0x7fad8c0ffd30 2024-03-26T18:52:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.783+0000 7fad827fc700 1 --2- 172.21.15.102:0/2504235375 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fad7806c680 0x7fad7806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.783+0000 7fad827fc700 1 -- 172.21.15.102:0/2504235375 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fad8808cf00 con 0x7fad8c0ffd30 2024-03-26T18:52:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.783+0000 7fad9169f700 1 --2- 172.21.15.102:0/2504235375 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fad7806c680 0x7fad7806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.784+0000 7fad9169f700 1 --2- 172.21.15.102:0/2504235375 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fad7806c680 0x7fad7806eb30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fad7c006fd0 tx=0x7fad7c008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:54.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:54.788+0000 7fad827fc700 1 -- 172.21.15.102:0/2504235375 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fad88057630 con 0x7fad8c0ffd30 2024-03-26T18:52:55.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.027+0000 7fad93903700 1 -- 172.21.15.102:0/2504235375 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fad8c04ea50 con 0x7fad8c0ffd30 2024-03-26T18:52:55.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.027+0000 7fad827fc700 1 -- 172.21.15.102:0/2504235375 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+83 (secure 0 0 0) 0x7fad88027070 con 0x7fad8c0ffd30 2024-03-26T18:52:55.020 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-26T18:52:55.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.029+0000 7fad93903700 1 -- 172.21.15.102:0/2504235375 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fad7806c680 msgr2=0x7fad7806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:55.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.029+0000 7fad93903700 1 --2- 172.21.15.102:0/2504235375 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fad7806c680 0x7fad7806eb30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fad7c006fd0 tx=0x7fad7c008040 comp rx=0 tx=0).stop 2024-03-26T18:52:55.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.029+0000 7fad93903700 1 -- 172.21.15.102:0/2504235375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad8c0ffd30 msgr2=0x7fad8c1012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:55.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.029+0000 7fad93903700 1 --2- 172.21.15.102:0/2504235375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad8c0ffd30 0x7fad8c1012c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fad880092e0 tx=0x7fad8800be20 comp rx=0 tx=0).stop 2024-03-26T18:52:55.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.029+0000 7fad93903700 1 -- 172.21.15.102:0/2504235375 shutdown_connections 2024-03-26T18:52:55.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.029+0000 7fad93903700 1 --2- 172.21.15.102:0/2504235375 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fad7806c680 0x7fad7806eb30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:55.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.029+0000 7fad93903700 1 --2- 172.21.15.102:0/2504235375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad8c0ff390 0x7fad8c100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:55.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.029+0000 7fad93903700 1 --2- 172.21.15.102:0/2504235375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad8c0ffd30 0x7fad8c1012c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:55.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.029+0000 7fad93903700 1 -- 172.21.15.102:0/2504235375 >> 172.21.15.102:0/2504235375 conn(0x7fad8c074f50 msgr2=0x7fad8c0686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:55.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.029+0000 7fad93903700 1 -- 172.21.15.102:0/2504235375 shutdown_connections 2024-03-26T18:52:55.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:55.029+0000 7fad93903700 1 -- 172.21.15.102:0/2504235375 wait complete. 2024-03-26T18:52:55.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:54 smithi190 ceph-mon[35043]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-26T18:52:55.353 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:54 smithi102 ceph-mon[29002]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-26T18:52:55.690 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-26T18:52:55.691 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-26T18:52:55.691 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi190.front.sepia.ceph.com 2024-03-26T18:52:55.691 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-26T18:52:55.691 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-26T18:52:55.691 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-26T18:52:55.691 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-26T18:52:55.691 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-26T18:52:55.691 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-26T18:52:55.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:55.691 DEBUG:teuthology.orchestra.run.smithi190:> ip addr 2024-03-26T18:52:55.710 INFO:teuthology.orchestra.run.smithi190.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-26T18:52:55.710 INFO:teuthology.orchestra.run.smithi190.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-26T18:52:55.710 INFO:teuthology.orchestra.run.smithi190.stdout: inet 127.0.0.1/8 scope host lo 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 ::1/128 scope host 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:d9:36:d0 brd ff:ff:ff:ff:ff:ff 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout: altname enp7s0f0 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:d9:36:d1 brd ff:ff:ff:ff:ff:ff 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout: altname enp7s0f1 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:88:77:62 brd ff:ff:ff:ff:ff:ff 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:88:77:63 brd ff:ff:ff:ff:ff:ff 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout: inet 172.21.15.190/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-26T18:52:55.711 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft 42287sec preferred_lft 42287sec 2024-03-26T18:52:55.712 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 fe80::ec4:7aff:fe88:7763/64 scope link 2024-03-26T18:52:55.712 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2024-03-26T18:52:55.712 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-26T18:52:55.712 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-26T18:52:55.712 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2024-03-26T18:52:55.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link add name ceph-brx type bridge 2024-03-26T18:52:55.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip addr flush dev ceph-brx 2024-03-26T18:52:55.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link set ceph-brx up 2024-03-26T18:52:55.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-26T18:52:55.712 DEBUG:teuthology.orchestra.run.smithi190:> ') 2024-03-26T18:52:55.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:55.934 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:55.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:55.941 DEBUG:teuthology.orchestra.run.smithi190:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-26T18:52:55.968 INFO:teuthology.orchestra.run.smithi190.stdout:1 2024-03-26T18:52:55.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:55.969 DEBUG:teuthology.orchestra.run.smithi190:> route 2024-03-26T18:52:56.043 INFO:teuthology.orchestra.run.smithi190.stdout:Kernel IP routing table 2024-03-26T18:52:56.043 INFO:teuthology.orchestra.run.smithi190.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-26T18:52:56.043 INFO:teuthology.orchestra.run.smithi190.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-26T18:52:56.043 INFO:teuthology.orchestra.run.smithi190.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-26T18:52:56.043 INFO:teuthology.orchestra.run.smithi190.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-26T18:52:56.043 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-26T18:52:56.044 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2024-03-26T18:52:56.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-26T18:52:56.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-26T18:52:56.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-26T18:52:56.044 DEBUG:teuthology.orchestra.run.smithi190:> ') 2024-03-26T18:52:56.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:56.141 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:55 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2504235375' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-26T18:52:56.369 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:56.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:56.375 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2024-03-26T18:52:56.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:55 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2504235375' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-26T18:52:56.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:56.391 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list-id 2024-03-26T18:52:56.446 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-26T18:52:56.446 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2024-03-26T18:52:56.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-26T18:52:56.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-26T18:52:56.446 DEBUG:teuthology.orchestra.run.smithi190:> ') 2024-03-26T18:52:56.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:56.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:56.568 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-26T18:52:56.568 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-26T18:52:56.568 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2024-03-26T18:52:56.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-26T18:52:56.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-26T18:52:56.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-26T18:52:56.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-26T18:52:56.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-26T18:52:56.569 DEBUG:teuthology.orchestra.run.smithi190:> ') 2024-03-26T18:52:56.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:56.702 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:56.708 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-26T18:52:56.708 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2024-03-26T18:52:56.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link set brx.0 up 2024-03-26T18:52:56.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link set dev brx.0 master ceph-brx 2024-03-26T18:52:56.709 DEBUG:teuthology.orchestra.run.smithi190:> ') 2024-03-26T18:52:56.755 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:56.783 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:56.789 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-26T18:52:56.789 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-26T18:52:56.817 INFO:teuthology.orchestra.run.smithi190.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-26T18:52:56.817 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-26T18:52:56.817 DEBUG:teuthology.orchestra.run.smithi190:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-26T18:52:56.871 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-26T18:52:56.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:56.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-26T18:52:57.026 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-26T18:52:57.026 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-26T18:52:57.026 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2024-03-26T18:52:57.131 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-26T18:52:57.131 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-26T18:52:57.131 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2024-03-26T18:52:57.131 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.190:3300" 2024-03-26T18:52:57.131 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-26T18:52:57.136 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-26T18:52:57.136 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-26T18:52:57.136 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2024-03-26T18:52:57.136 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.190:3300" 2024-03-26T18:52:57.136 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-26T18:52:57.136 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.190:6789" 2024-03-26T18:52:57.136 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.190:6789:/ 2024-03-26T18:52:57.136 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=ed4e027c-eba0-11ee-95d0-87774f69a715" will pass to kernel 2024-03-26T18:52:57.137 INFO:teuthology.orchestra.run.smithi190.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-26T18:52:57.137 INFO:teuthology.orchestra.run.smithi190.stdout: You just mounted a file system that supports labels which does not 2024-03-26T18:52:57.137 INFO:teuthology.orchestra.run.smithi190.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-26T18:52:57.137 INFO:teuthology.orchestra.run.smithi190.stdout: applications will generate AVC messages and not be allowed access to 2024-03-26T18:52:57.138 INFO:teuthology.orchestra.run.smithi190.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-26T18:52:57.139 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-26T18:52:57.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T18:52:57.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-26T18:52:57.174 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-26T18:52:57.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-26T18:52:57.265 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:57.272 INFO:teuthology.orchestra.run.smithi190.stdout:client14546 2024-03-26T18:52:57.272 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:52:57.272 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-26T18:52:57.288 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-26T18:52:57.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-26T18:52:57.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:57.384 INFO:teuthology.orchestra.run.smithi190.stdout:client14546 2024-03-26T18:52:57.384 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-26T18:52:57.433 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-26T18:52:57.434 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:57.440 INFO:teuthology.orchestra.run.smithi190.stdout:client14546 2024-03-26T18:52:57.441 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715.client14546/status') 2024-03-26T18:52:57.488 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:57 smithi190 ceph-mon[35043]: pgmap v196: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 341 B/s wr, 2 op/s 2024-03-26T18:52:57.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-26T18:52:57.503 INFO:teuthology.orchestra.run.smithi190.stdout:instance: client.14546 (3)172.21.15.190:0/1799840371 2024-03-26T18:52:57.503 INFO:teuthology.orchestra.run.smithi190.stdout:blocklisted: false 2024-03-26T18:52:57.503 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2024-03-26T18:52:57.503 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2024-03-26T18:52:57.503 INFO:teuthology.orchestra.run.smithi190.stderr:72 bytes copied, 2.1109e-05 s, 3.4 MB/s 2024-03-26T18:52:57.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-26T18:52:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-26T18:52:57.510 INFO:teuthology.run_tasks:Running task print... 2024-03-26T18:52:57.521 INFO:teuthology.task.print:**** done client 2024-03-26T18:52:57.521 INFO:teuthology.run_tasks:Running task parallel... 2024-03-26T18:52:57.531 INFO:teuthology.task.parallel:starting parallel... 2024-03-26T18:52:57.531 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-26T18:52:57.532 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-26T18:52:57.532 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T18:52:57.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-03-26T18:52:57.533 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-26T18:52:57.533 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-26T18:52:57.536 INFO:tasks.workunit:Pulling workunits from ref 72e2437271dcafb9826f2f49bb1001426fa6de81 2024-03-26T18:52:57.536 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-26T18:52:57.536 INFO:tasks.workunit:timeout=3h 2024-03-26T18:52:57.536 INFO:tasks.workunit:cleanup=True 2024-03-26T18:52:57.537 DEBUG:teuthology.orchestra.run.smithi102:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-26T18:52:57.556 INFO:teuthology.orchestra.run.smithi102.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-26T18:52:57.556 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-26T18:52:57.556 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 94h/148d Inode: 1 Links: 2 2024-03-26T18:52:57.556 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T18:52:57.556 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-26T18:52:57.556 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-26T18:52:57.556 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-26 18:52:24.280192752 +0000 2024-03-26T18:52:57.556 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-26 18:52:52.586446228 +0000 2024-03-26T18:52:57.556 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-26 18:52:24.280192752 +0000 2024-03-26T18:52:57.556 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-26T18:52:57.556 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-26T18:52:57.565 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:57 smithi102 ceph-mon[29002]: pgmap v196: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 341 B/s wr, 2 op/s 2024-03-26T18:52:57.599 DEBUG:teuthology.orchestra.run.smithi190:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-26T18:52:57.616 INFO:teuthology.orchestra.run.smithi190.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-26T18:52:57.616 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-03-26T18:52:57.616 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 7eh/126d Inode: 1 Links: 3 2024-03-26T18:52:57.616 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T18:52:57.616 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-26T18:52:57.616 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-26T18:52:57.616 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-26 18:52:57.601344352 +0000 2024-03-26T18:52:57.616 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-26 18:52:57.601344352 +0000 2024-03-26T18:52:57.616 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-26 18:52:24.280192752 +0000 2024-03-26T18:52:57.617 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-26T18:52:57.617 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-26T18:52:57.688 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ljflores/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 72e2437271dcafb9826f2f49bb1001426fa6de81 2024-03-26T18:52:57.689 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ljflores/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 72e2437271dcafb9826f2f49bb1001426fa6de81 2024-03-26T18:52:57.704 INFO:tasks.workunit.client.0.smithi102.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-26T18:52:57.744 INFO:tasks.workunit.client.1.smithi190.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-26T18:52:57.760 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:52:59.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.367+0000 7f8b04883700 1 -- 172.21.15.102:0/2370889366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00100170 msgr2=0x7f8b00100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:59.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.367+0000 7f8b04883700 1 --2- 172.21.15.102:0/2370889366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00100170 0x7f8b00100580 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009910 tx=0x7f8af0009c20 comp rx=0 tx=0).stop 2024-03-26T18:52:59.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.367+0000 7f8b04883700 1 -- 172.21.15.102:0/2370889366 shutdown_connections 2024-03-26T18:52:59.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.367+0000 7f8b04883700 1 --2- 172.21.15.102:0/2370889366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b00101370 0x7f8b001017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:59.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.367+0000 7f8b04883700 1 --2- 172.21.15.102:0/2370889366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00100170 0x7f8b00100580 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:59.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.367+0000 7f8b04883700 1 -- 172.21.15.102:0/2370889366 >> 172.21.15.102:0/2370889366 conn(0x7f8b000fb6e0 msgr2=0x7f8b000fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:59.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.367+0000 7f8b04883700 1 -- 172.21.15.102:0/2370889366 shutdown_connections 2024-03-26T18:52:59.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.367+0000 7f8b04883700 1 -- 172.21.15.102:0/2370889366 wait complete. 2024-03-26T18:52:59.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.368+0000 7f8b04883700 1 Processor -- start 2024-03-26T18:52:59.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.368+0000 7f8b04883700 1 -- start start 2024-03-26T18:52:59.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.368+0000 7f8b04883700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b00100170 0x7f8b001959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:59.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.368+0000 7f8b04883700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00101370 0x7f8b00195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:59.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.368+0000 7f8b04883700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b00196520 con 0x7f8b00101370 2024-03-26T18:52:59.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.368+0000 7f8b04883700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b00196660 con 0x7f8b00100170 2024-03-26T18:52:59.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.368+0000 7f8afdd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00101370 0x7f8b00195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.368+0000 7f8afdd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00101370 0x7f8b00195f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51686/0 (socket says 172.21.15.102:51686) 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.368+0000 7f8afdd9b700 1 -- 172.21.15.102:0/3809630435 learned_addr learned my addr 172.21.15.102:0/3809630435 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.368+0000 7f8afe59c700 1 --2- 172.21.15.102:0/3809630435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b00100170 0x7f8b001959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.369+0000 7f8afdd9b700 1 -- 172.21.15.102:0/3809630435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b00100170 msgr2=0x7f8b001959c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.369+0000 7f8afdd9b700 1 --2- 172.21.15.102:0/3809630435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b00100170 0x7f8b001959c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.369+0000 7f8afdd9b700 1 -- 172.21.15.102:0/3809630435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af00095f0 con 0x7f8b00101370 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.369+0000 7f8afdd9b700 1 --2- 172.21.15.102:0/3809630435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00101370 0x7f8b00195f00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8af400c8a0 tx=0x7f8af400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.369+0000 7f8aef7fe700 1 -- 172.21.15.102:0/3809630435 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8af400cea0 con 0x7f8b00101370 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.369+0000 7f8aef7fe700 1 -- 172.21.15.102:0/3809630435 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8af4004830 con 0x7f8b00101370 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.369+0000 7f8aef7fe700 1 -- 172.21.15.102:0/3809630435 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8af40055b0 con 0x7f8b00101370 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.369+0000 7f8b04883700 1 -- 172.21.15.102:0/3809630435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0019b110 con 0x7f8b00101370 2024-03-26T18:52:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.369+0000 7f8b04883700 1 -- 172.21.15.102:0/3809630435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0019b6e0 con 0x7f8b00101370 2024-03-26T18:52:59.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.370+0000 7f8b04883700 1 -- 172.21.15.102:0/3809630435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b00108cc0 con 0x7f8b00101370 2024-03-26T18:52:59.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.370+0000 7f8aef7fe700 1 -- 172.21.15.102:0/3809630435 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f8af4007ad0 con 0x7f8b00101370 2024-03-26T18:52:59.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.371+0000 7f8aef7fe700 1 --2- 172.21.15.102:0/3809630435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ae806c680 0x7f8ae806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:52:59.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.371+0000 7f8aef7fe700 1 -- 172.21.15.102:0/3809630435 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f8af408b590 con 0x7f8b00101370 2024-03-26T18:52:59.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.371+0000 7f8afe59c700 1 --2- 172.21.15.102:0/3809630435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ae806c680 0x7f8ae806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:52:59.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.371+0000 7f8afe59c700 1 --2- 172.21.15.102:0/3809630435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ae806c680 0x7f8ae806eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8af0006010 tx=0x7f8af00058e0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:52:59.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.375+0000 7f8aef7fe700 1 -- 172.21.15.102:0/3809630435 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8af4059720 con 0x7f8b00101370 2024-03-26T18:52:59.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:52:59 smithi190 ceph-mon[35043]: pgmap v197: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-03-26T18:52:59.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.572+0000 7f8b04883700 1 -- 172.21.15.102:0/3809630435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f8b00108e80 con 0x7f8b00101370 2024-03-26T18:52:59.565 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:52:59 smithi102 ceph-mon[29002]: pgmap v197: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-03-26T18:52:59.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.577+0000 7f8aef7fe700 1 -- 172.21.15.102:0/3809630435 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v22)=0 v22) v1 ==== 126+0+0 (secure 0 0 0) 0x7f8af40592b0 con 0x7f8b00101370 2024-03-26T18:52:59.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.580+0000 7f8b04883700 1 -- 172.21.15.102:0/3809630435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ae806c680 msgr2=0x7f8ae806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:59.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.580+0000 7f8b04883700 1 --2- 172.21.15.102:0/3809630435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ae806c680 0x7f8ae806eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8af0006010 tx=0x7f8af00058e0 comp rx=0 tx=0).stop 2024-03-26T18:52:59.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.580+0000 7f8b04883700 1 -- 172.21.15.102:0/3809630435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00101370 msgr2=0x7f8b00195f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:52:59.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.580+0000 7f8b04883700 1 --2- 172.21.15.102:0/3809630435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00101370 0x7f8b00195f00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8af400c8a0 tx=0x7f8af400cc60 comp rx=0 tx=0).stop 2024-03-26T18:52:59.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.580+0000 7f8b04883700 1 -- 172.21.15.102:0/3809630435 shutdown_connections 2024-03-26T18:52:59.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.580+0000 7f8b04883700 1 --2- 172.21.15.102:0/3809630435 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8ae806c680 0x7f8ae806eb30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:59.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.580+0000 7f8b04883700 1 --2- 172.21.15.102:0/3809630435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b00100170 0x7f8b001959c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:59.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.580+0000 7f8b04883700 1 --2- 172.21.15.102:0/3809630435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b00101370 0x7f8b00195f00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:52:59.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.580+0000 7f8b04883700 1 -- 172.21.15.102:0/3809630435 >> 172.21.15.102:0/3809630435 conn(0x7f8b000fb6e0 msgr2=0x7f8b001045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:52:59.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.580+0000 7f8b04883700 1 -- 172.21.15.102:0/3809630435 shutdown_connections 2024-03-26T18:52:59.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:52:59.580+0000 7f8b04883700 1 -- 172.21.15.102:0/3809630435 wait complete. 2024-03-26T18:53:00.274 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-26T18:53:00.275 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T18:53:00.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-26T18:53:00.538 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:53:00.818 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:00 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3809630435' entity='client.admin' 2024-03-26T18:53:00.819 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:53:00.819 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:53:00.819 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:53:00.819 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:00 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:53:00.819 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:00 smithi102 ceph-mon[29002]: pgmap v198: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-26T18:53:01.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:00 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3809630435' entity='client.admin' 2024-03-26T18:53:01.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:53:01.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:53:01.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:53:01.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:00 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:53:01.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:00 smithi190 ceph-mon[35043]: pgmap v198: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-26T18:53:02.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.302+0000 7f5726849700 1 -- 172.21.15.102:0/2311207305 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5720100170 msgr2=0x7f5720100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:02.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.302+0000 7f5726849700 1 --2- 172.21.15.102:0/2311207305 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5720100170 0x7f5720100580 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5710009910 tx=0x7f5710009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:02.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.302+0000 7f5726849700 1 -- 172.21.15.102:0/2311207305 shutdown_connections 2024-03-26T18:53:02.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.302+0000 7f5726849700 1 --2- 172.21.15.102:0/2311207305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720101370 0x7f57201017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:02.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.302+0000 7f5726849700 1 --2- 172.21.15.102:0/2311207305 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5720100170 0x7f5720100580 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:02.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.302+0000 7f5726849700 1 -- 172.21.15.102:0/2311207305 >> 172.21.15.102:0/2311207305 conn(0x7f57200fb6e0 msgr2=0x7f57200fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:02.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.302+0000 7f5726849700 1 -- 172.21.15.102:0/2311207305 shutdown_connections 2024-03-26T18:53:02.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.302+0000 7f5726849700 1 -- 172.21.15.102:0/2311207305 wait complete. 2024-03-26T18:53:02.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.303+0000 7f5726849700 1 Processor -- start 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.303+0000 7f5726849700 1 -- start start 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.303+0000 7f5726849700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5720100170 0x7f5720195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.303+0000 7f5726849700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720101370 0x7f5720195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.303+0000 7f5726849700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57201964f0 con 0x7f5720100170 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.303+0000 7f5726849700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5720196630 con 0x7f5720101370 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.303+0000 7f571ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5720100170 0x7f5720195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.303+0000 7f571ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5720100170 0x7f5720195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51708/0 (socket says 172.21.15.102:51708) 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.303+0000 7f571ffff700 1 -- 172.21.15.102:0/2190346565 learned_addr learned my addr 172.21.15.102:0/2190346565 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.303+0000 7f571f7fe700 1 --2- 172.21.15.102:0/2190346565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720101370 0x7f5720195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.304+0000 7f571ffff700 1 -- 172.21.15.102:0/2190346565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720101370 msgr2=0x7f5720195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.304+0000 7f571ffff700 1 --2- 172.21.15.102:0/2190346565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720101370 0x7f5720195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.304+0000 7f571ffff700 1 -- 172.21.15.102:0/2190346565 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57100095f0 con 0x7f5720100170 2024-03-26T18:53:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.304+0000 7f571ffff700 1 --2- 172.21.15.102:0/2190346565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5720100170 0x7f5720195990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5710000c00 tx=0x7f571000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.304+0000 7f571d7fa700 1 -- 172.21.15.102:0/2190346565 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f571001d070 con 0x7f5720100170 2024-03-26T18:53:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.304+0000 7f5726849700 1 -- 172.21.15.102:0/2190346565 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572019b080 con 0x7f5720100170 2024-03-26T18:53:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.304+0000 7f571d7fa700 1 -- 172.21.15.102:0/2190346565 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5710005dc0 con 0x7f5720100170 2024-03-26T18:53:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.304+0000 7f571d7fa700 1 -- 172.21.15.102:0/2190346565 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5710022470 con 0x7f5720100170 2024-03-26T18:53:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.304+0000 7f5726849700 1 -- 172.21.15.102:0/2190346565 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572019b570 con 0x7f5720100170 2024-03-26T18:53:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.305+0000 7f5726849700 1 -- 172.21.15.102:0/2190346565 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5720108cc0 con 0x7f5720100170 2024-03-26T18:53:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.306+0000 7f571d7fa700 1 -- 172.21.15.102:0/2190346565 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f5710003810 con 0x7f5720100170 2024-03-26T18:53:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.306+0000 7f571d7fa700 1 --2- 172.21.15.102:0/2190346565 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f570806c680 0x7f570806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.306+0000 7f571d7fa700 1 -- 172.21.15.102:0/2190346565 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f571008ce00 con 0x7f5720100170 2024-03-26T18:53:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.306+0000 7f571f7fe700 1 --2- 172.21.15.102:0/2190346565 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f570806c680 0x7f570806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.307+0000 7f571f7fe700 1 --2- 172.21.15.102:0/2190346565 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f570806c680 0x7f570806eb30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5714009c60 tx=0x7f5714009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:02.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.311+0000 7f571d7fa700 1 -- 172.21.15.102:0/2190346565 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f571005af10 con 0x7f5720100170 2024-03-26T18:53:02.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.533+0000 7f5726849700 1 -- 172.21.15.102:0/2190346565 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f5720108eb0 con 0x7f5720100170 2024-03-26T18:53:02.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.534+0000 7f571d7fa700 1 -- 172.21.15.102:0/2190346565 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v22)=0 v22) v1 ==== 155+0+0 (secure 0 0 0) 0x7f571005aaa0 con 0x7f5720100170 2024-03-26T18:53:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.536+0000 7f5726849700 1 -- 172.21.15.102:0/2190346565 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f570806c680 msgr2=0x7f570806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.536+0000 7f5726849700 1 --2- 172.21.15.102:0/2190346565 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f570806c680 0x7f570806eb30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5714009c60 tx=0x7f5714009260 comp rx=0 tx=0).stop 2024-03-26T18:53:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.536+0000 7f5726849700 1 -- 172.21.15.102:0/2190346565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5720100170 msgr2=0x7f5720195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.536+0000 7f5726849700 1 --2- 172.21.15.102:0/2190346565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5720100170 0x7f5720195990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5710000c00 tx=0x7f571000f690 comp rx=0 tx=0).stop 2024-03-26T18:53:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.536+0000 7f5726849700 1 -- 172.21.15.102:0/2190346565 shutdown_connections 2024-03-26T18:53:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.536+0000 7f5726849700 1 --2- 172.21.15.102:0/2190346565 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f570806c680 0x7f570806eb30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.536+0000 7f5726849700 1 --2- 172.21.15.102:0/2190346565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5720100170 0x7f5720195990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.536+0000 7f5726849700 1 --2- 172.21.15.102:0/2190346565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720101370 0x7f5720195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.536+0000 7f5726849700 1 -- 172.21.15.102:0/2190346565 >> 172.21.15.102:0/2190346565 conn(0x7f57200fb6e0 msgr2=0x7f57201045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.536+0000 7f5726849700 1 -- 172.21.15.102:0/2190346565 shutdown_connections 2024-03-26T18:53:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:02.536+0000 7f5726849700 1 -- 172.21.15.102:0/2190346565 wait complete. 2024-03-26T18:53:03.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:02 smithi102 ceph-mon[29002]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-03-26T18:53:03.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-26T18:53:03.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:02 smithi190 ceph-mon[35043]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-03-26T18:53:03.407 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:53:04.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:03 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:53:04.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:03 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:53:05.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.212+0000 7f36cfe56700 1 -- 172.21.15.102:0/3885376754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c8100250 msgr2=0x7f36c81006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:05.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.212+0000 7f36cfe56700 1 --2- 172.21.15.102:0/3885376754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c8100250 0x7f36c81006a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009910 tx=0x7f36b8009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:05.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.212+0000 7f36cfe56700 1 -- 172.21.15.102:0/3885376754 shutdown_connections 2024-03-26T18:53:05.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.212+0000 7f36cfe56700 1 --2- 172.21.15.102:0/3885376754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c8100250 0x7f36c81006a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:05.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.212+0000 7f36cfe56700 1 --2- 172.21.15.102:0/3885376754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c80ff050 0x7f36c80ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:05.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.212+0000 7f36cfe56700 1 -- 172.21.15.102:0/3885376754 >> 172.21.15.102:0/3885376754 conn(0x7f36c80fa5e0 msgr2=0x7f36c80fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:05.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.212+0000 7f36cfe56700 1 -- 172.21.15.102:0/3885376754 shutdown_connections 2024-03-26T18:53:05.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.212+0000 7f36cfe56700 1 -- 172.21.15.102:0/3885376754 wait complete. 2024-03-26T18:53:05.206 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:04 smithi102 ceph-mon[29002]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-03-26T18:53:05.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.213+0000 7f36cfe56700 1 Processor -- start 2024-03-26T18:53:05.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.213+0000 7f36cfe56700 1 -- start start 2024-03-26T18:53:05.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.213+0000 7f36cfe56700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c80ff050 0x7f36c8194870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:05.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.213+0000 7f36cfe56700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c8100250 0x7f36c8194db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:05.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.213+0000 7f36cfe56700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c81953d0 con 0x7f36c8100250 2024-03-26T18:53:05.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.213+0000 7f36cfe56700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c8195510 con 0x7f36c80ff050 2024-03-26T18:53:05.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.213+0000 7f36cdbf2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c80ff050 0x7f36c8194870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:05.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.213+0000 7f36cdbf2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c80ff050 0x7f36c8194870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:39378/0 (socket says 172.21.15.102:39378) 2024-03-26T18:53:05.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.213+0000 7f36cdbf2700 1 -- 172.21.15.102:0/1693175737 learned_addr learned my addr 172.21.15.102:0/1693175737 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:05.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.213+0000 7f36cd3f1700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c8100250 0x7f36c8194db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:05.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.213+0000 7f36cd3f1700 1 -- 172.21.15.102:0/1693175737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c80ff050 msgr2=0x7f36c8194870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:05.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.214+0000 7f36cd3f1700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c80ff050 0x7f36c8194870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:05.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.214+0000 7f36cd3f1700 1 -- 172.21.15.102:0/1693175737 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b80095f0 con 0x7f36c8100250 2024-03-26T18:53:05.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.214+0000 7f36cdbf2700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c80ff050 0x7f36c8194870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:53:05.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.214+0000 7f36cd3f1700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c8100250 0x7f36c8194db0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f36b8000c00 tx=0x7f36b800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:05.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.214+0000 7f36beffd700 1 -- 172.21.15.102:0/1693175737 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b801d070 con 0x7f36c8100250 2024-03-26T18:53:05.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.214+0000 7f36beffd700 1 -- 172.21.15.102:0/1693175737 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f36b8005dc0 con 0x7f36c8100250 2024-03-26T18:53:05.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.214+0000 7f36cfe56700 1 -- 172.21.15.102:0/1693175737 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c80683d0 con 0x7f36c8100250 2024-03-26T18:53:05.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.214+0000 7f36beffd700 1 -- 172.21.15.102:0/1693175737 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b8017400 con 0x7f36c8100250 2024-03-26T18:53:05.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.214+0000 7f36cfe56700 1 -- 172.21.15.102:0/1693175737 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c8068800 con 0x7f36c8100250 2024-03-26T18:53:05.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.216+0000 7f36cfe56700 1 -- 172.21.15.102:0/1693175737 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c8107c10 con 0x7f36c8100250 2024-03-26T18:53:05.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.216+0000 7f36beffd700 1 -- 172.21.15.102:0/1693175737 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f36b8003810 con 0x7f36c8100250 2024-03-26T18:53:05.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.216+0000 7f36beffd700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f36b406c680 0x7f36b406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:05.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.216+0000 7f36beffd700 1 -- 172.21.15.102:0/1693175737 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f36b808cd30 con 0x7f36c8100250 2024-03-26T18:53:05.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.217+0000 7f36cdbf2700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f36b406c680 0x7f36b406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:05.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.217+0000 7f36cdbf2700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f36b406c680 0x7f36b406eb30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f36c4005950 tx=0x7f36c4010040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:05.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.222+0000 7f36beffd700 1 -- 172.21.15.102:0/1693175737 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f36b805ae20 con 0x7f36c8100250 2024-03-26T18:53:05.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:04 smithi190 ceph-mon[35043]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-03-26T18:53:05.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.443+0000 7f36cfe56700 1 -- 172.21.15.102:0/1693175737 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f36c8107dd0 con 0x7f36c8100250 2024-03-26T18:53:05.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.443+0000 7f36beffd700 1 -- 172.21.15.102:0/1693175737 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v22)=0 v22) v1 ==== 163+0+0 (secure 0 0 0) 0x7f36b805a9b0 con 0x7f36c8100250 2024-03-26T18:53:05.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.445+0000 7f36cfe56700 1 -- 172.21.15.102:0/1693175737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f36b406c680 msgr2=0x7f36b406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:05.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.445+0000 7f36cfe56700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f36b406c680 0x7f36b406eb30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f36c4005950 tx=0x7f36c4010040 comp rx=0 tx=0).stop 2024-03-26T18:53:05.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.445+0000 7f36cfe56700 1 -- 172.21.15.102:0/1693175737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c8100250 msgr2=0x7f36c8194db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:05.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.445+0000 7f36cfe56700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c8100250 0x7f36c8194db0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f36b8000c00 tx=0x7f36b800f690 comp rx=0 tx=0).stop 2024-03-26T18:53:05.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.445+0000 7f36cfe56700 1 -- 172.21.15.102:0/1693175737 shutdown_connections 2024-03-26T18:53:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.445+0000 7f36cfe56700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f36b406c680 0x7f36b406eb30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.445+0000 7f36cfe56700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c80ff050 0x7f36c8194870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.445+0000 7f36cfe56700 1 --2- 172.21.15.102:0/1693175737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c8100250 0x7f36c8194db0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.445+0000 7f36cfe56700 1 -- 172.21.15.102:0/1693175737 >> 172.21.15.102:0/1693175737 conn(0x7f36c80fa5e0 msgr2=0x7f36c8103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.446+0000 7f36cfe56700 1 -- 172.21.15.102:0/1693175737 shutdown_connections 2024-03-26T18:53:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:05.446+0000 7f36cfe56700 1 -- 172.21.15.102:0/1693175737 wait complete. 2024-03-26T18:53:06.131 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-26T18:53:06.361 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:53:07.516 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:07 smithi102 ceph-mon[29002]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-26T18:53:07.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:07 smithi190 ceph-mon[35043]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-26T18:53:08.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.117+0000 7fba7935e700 1 -- 172.21.15.102:0/3801574853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba741014b0 msgr2=0x7fba74101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.117+0000 7fba7935e700 1 --2- 172.21.15.102:0/3801574853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba741014b0 0x7fba74101900 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009910 tx=0x7fba5c009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.117+0000 7fba7935e700 1 -- 172.21.15.102:0/3801574853 shutdown_connections 2024-03-26T18:53:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.117+0000 7fba7935e700 1 --2- 172.21.15.102:0/3801574853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba741014b0 0x7fba74101900 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.117+0000 7fba7935e700 1 --2- 172.21.15.102:0/3801574853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba740fe430 0x7fba740fe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.117+0000 7fba7935e700 1 -- 172.21.15.102:0/3801574853 >> 172.21.15.102:0/3801574853 conn(0x7fba740f9d80 msgr2=0x7fba740fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.117+0000 7fba7935e700 1 -- 172.21.15.102:0/3801574853 shutdown_connections 2024-03-26T18:53:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.117+0000 7fba7935e700 1 -- 172.21.15.102:0/3801574853 wait complete. 2024-03-26T18:53:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.118+0000 7fba7935e700 1 Processor -- start 2024-03-26T18:53:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.118+0000 7fba7935e700 1 -- start start 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba7935e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba740fe430 0x7fba74195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba7935e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba741014b0 0x7fba74195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba7935e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba741964c0 con 0x7fba741014b0 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba7935e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba74196600 con 0x7fba740fe430 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba72ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba740fe430 0x7fba74195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba6bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba741014b0 0x7fba74195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba72ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba740fe430 0x7fba74195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:39406/0 (socket says 172.21.15.102:39406) 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba72ffd700 1 -- 172.21.15.102:0/1298708996 learned_addr learned my addr 172.21.15.102:0/1298708996 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba72ffd700 1 -- 172.21.15.102:0/1298708996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba741014b0 msgr2=0x7fba74195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba72ffd700 1 --2- 172.21.15.102:0/1298708996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba741014b0 0x7fba74195ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba72ffd700 1 -- 172.21.15.102:0/1298708996 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c0095f0 con 0x7fba740fe430 2024-03-26T18:53:08.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba6bfff700 1 --2- 172.21.15.102:0/1298708996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba741014b0 0x7fba74195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:53:08.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.119+0000 7fba72ffd700 1 --2- 172.21.15.102:0/1298708996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba740fe430 0x7fba74195960 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fba6400e8f0 tx=0x7fba6400ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:08.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.120+0000 7fba70ff9700 1 -- 172.21.15.102:0/1298708996 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba64009790 con 0x7fba740fe430 2024-03-26T18:53:08.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.120+0000 7fba70ff9700 1 -- 172.21.15.102:0/1298708996 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fba6400c810 con 0x7fba740fe430 2024-03-26T18:53:08.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.120+0000 7fba70ff9700 1 -- 172.21.15.102:0/1298708996 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba64003d20 con 0x7fba740fe430 2024-03-26T18:53:08.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.120+0000 7fba7935e700 1 -- 172.21.15.102:0/1298708996 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7419b0b0 con 0x7fba740fe430 2024-03-26T18:53:08.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.120+0000 7fba7935e700 1 -- 172.21.15.102:0/1298708996 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7419b5a0 con 0x7fba740fe430 2024-03-26T18:53:08.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.121+0000 7fba7935e700 1 -- 172.21.15.102:0/1298708996 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba74108c90 con 0x7fba740fe430 2024-03-26T18:53:08.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.122+0000 7fba70ff9700 1 -- 172.21.15.102:0/1298708996 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fba640041f0 con 0x7fba740fe430 2024-03-26T18:53:08.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.122+0000 7fba70ff9700 1 --2- 172.21.15.102:0/1298708996 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fba6006c680 0x7fba6006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:08.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.122+0000 7fba70ff9700 1 -- 172.21.15.102:0/1298708996 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fba64014070 con 0x7fba740fe430 2024-03-26T18:53:08.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.122+0000 7fba6bfff700 1 --2- 172.21.15.102:0/1298708996 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fba6006c680 0x7fba6006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:08.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.122+0000 7fba6bfff700 1 --2- 172.21.15.102:0/1298708996 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fba6006c680 0x7fba6006eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009fd0 tx=0x7fba5c005940 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:08.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.127+0000 7fba70ff9700 1 -- 172.21.15.102:0/1298708996 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fba64059da0 con 0x7fba740fe430 2024-03-26T18:53:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.336+0000 7fba7935e700 1 -- 172.21.15.102:0/1298708996 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fba74108e50 con 0x7fba740fe430 2024-03-26T18:53:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.337+0000 7fba70ff9700 1 -- 172.21.15.102:0/1298708996 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7fba64059930 con 0x7fba740fe430 2024-03-26T18:53:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.339+0000 7fba7935e700 1 -- 172.21.15.102:0/1298708996 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fba6006c680 msgr2=0x7fba6006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.339+0000 7fba7935e700 1 --2- 172.21.15.102:0/1298708996 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fba6006c680 0x7fba6006eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009fd0 tx=0x7fba5c005940 comp rx=0 tx=0).stop 2024-03-26T18:53:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.339+0000 7fba7935e700 1 -- 172.21.15.102:0/1298708996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba740fe430 msgr2=0x7fba74195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.339+0000 7fba7935e700 1 --2- 172.21.15.102:0/1298708996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba740fe430 0x7fba74195960 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fba6400e8f0 tx=0x7fba6400ec00 comp rx=0 tx=0).stop 2024-03-26T18:53:08.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.339+0000 7fba7935e700 1 -- 172.21.15.102:0/1298708996 shutdown_connections 2024-03-26T18:53:08.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.339+0000 7fba7935e700 1 --2- 172.21.15.102:0/1298708996 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fba6006c680 0x7fba6006eb30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:08.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.339+0000 7fba7935e700 1 --2- 172.21.15.102:0/1298708996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba740fe430 0x7fba74195960 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:08.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.339+0000 7fba7935e700 1 --2- 172.21.15.102:0/1298708996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba741014b0 0x7fba74195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:08.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.339+0000 7fba7935e700 1 -- 172.21.15.102:0/1298708996 >> 172.21.15.102:0/1298708996 conn(0x7fba740f9d80 msgr2=0x7fba741046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:08.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.340+0000 7fba7935e700 1 -- 172.21.15.102:0/1298708996 shutdown_connections 2024-03-26T18:53:08.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:08.340+0000 7fba7935e700 1 -- 172.21.15.102:0/1298708996 wait complete. 2024-03-26T18:53:09.015 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-26T18:53:09.241 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:53:09.363 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:09 smithi102 ceph-mon[29002]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-03-26T18:53:09.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:09 smithi190 ceph-mon[35043]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-03-26T18:53:11.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.074+0000 7f490225b700 1 -- 172.21.15.102:0/2246188736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48fc100150 msgr2=0x7f48fc100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:11.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.074+0000 7f490225b700 1 --2- 172.21.15.102:0/2246188736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48fc100150 0x7f48fc100560 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009910 tx=0x7f48ec009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:11.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.074+0000 7f490225b700 1 -- 172.21.15.102:0/2246188736 shutdown_connections 2024-03-26T18:53:11.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.074+0000 7f490225b700 1 --2- 172.21.15.102:0/2246188736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48fc101350 0x7f48fc1017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:11.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.074+0000 7f490225b700 1 --2- 172.21.15.102:0/2246188736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48fc100150 0x7f48fc100560 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.074+0000 7f490225b700 1 -- 172.21.15.102:0/2246188736 >> 172.21.15.102:0/2246188736 conn(0x7f48fc0fb6e0 msgr2=0x7f48fc0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.074+0000 7f490225b700 1 -- 172.21.15.102:0/2246188736 shutdown_connections 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.074+0000 7f490225b700 1 -- 172.21.15.102:0/2246188736 wait complete. 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.074+0000 7f490225b700 1 Processor -- start 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f490225b700 1 -- start start 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f490225b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48fc100150 0x7f48fc1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f490225b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48fc101350 0x7f48fc195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f490225b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc196500 con 0x7f48fc101350 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f490225b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc196640 con 0x7f48fc100150 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f48faffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48fc101350 0x7f48fc195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f48faffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48fc101350 0x7f48fc195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43872/0 (socket says 172.21.15.102:43872) 2024-03-26T18:53:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f48faffd700 1 -- 172.21.15.102:0/2811740000 learned_addr learned my addr 172.21.15.102:0/2811740000 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f48fb7fe700 1 --2- 172.21.15.102:0/2811740000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48fc100150 0x7f48fc1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f48faffd700 1 -- 172.21.15.102:0/2811740000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48fc100150 msgr2=0x7f48fc1959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f48faffd700 1 --2- 172.21.15.102:0/2811740000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48fc100150 0x7f48fc1959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f48faffd700 1 -- 172.21.15.102:0/2811740000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec0095f0 con 0x7f48fc101350 2024-03-26T18:53:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.075+0000 7f48faffd700 1 --2- 172.21.15.102:0/2811740000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48fc101350 0x7f48fc195ee0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f48f000e920 tx=0x7f48f000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.076+0000 7f48f8ff9700 1 -- 172.21.15.102:0/2811740000 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48f000c9d0 con 0x7f48fc101350 2024-03-26T18:53:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.076+0000 7f48f8ff9700 1 -- 172.21.15.102:0/2811740000 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f48f0010430 con 0x7f48fc101350 2024-03-26T18:53:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.076+0000 7f490225b700 1 -- 172.21.15.102:0/2811740000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48fc19b0f0 con 0x7f48fc101350 2024-03-26T18:53:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.076+0000 7f48f8ff9700 1 -- 172.21.15.102:0/2811740000 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48f0018650 con 0x7f48fc101350 2024-03-26T18:53:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.076+0000 7f490225b700 1 -- 172.21.15.102:0/2811740000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48fc19b640 con 0x7f48fc101350 2024-03-26T18:53:11.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.077+0000 7f490225b700 1 -- 172.21.15.102:0/2811740000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48fc108ca0 con 0x7f48fc101350 2024-03-26T18:53:11.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.078+0000 7f48f8ff9700 1 -- 172.21.15.102:0/2811740000 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f48f0018910 con 0x7f48fc101350 2024-03-26T18:53:11.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.078+0000 7f48f8ff9700 1 --2- 172.21.15.102:0/2811740000 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f48e406c680 0x7f48e406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:11.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.078+0000 7f48f8ff9700 1 -- 172.21.15.102:0/2811740000 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f48f0014070 con 0x7f48fc101350 2024-03-26T18:53:11.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.078+0000 7f48fb7fe700 1 --2- 172.21.15.102:0/2811740000 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f48e406c680 0x7f48e406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:11.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.079+0000 7f48fb7fe700 1 --2- 172.21.15.102:0/2811740000 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f48e406c680 0x7f48e406eb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009fd0 tx=0x7f48ec005a90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:11.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.083+0000 7f48f8ff9700 1 -- 172.21.15.102:0/2811740000 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f48f0051720 con 0x7f48fc101350 2024-03-26T18:53:11.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.317+0000 7f490225b700 1 -- 172.21.15.102:0/2811740000 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:360516069d9393362c4cc6eb9371680fe16d66ab", "target": ["mon-mgr", ""]}) v1 -- 0x7f48fc105d40 con 0x7f48e406c680 2024-03-26T18:53:11.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.322+0000 7f48f8ff9700 1 -- 172.21.15.102:0/2811740000 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f48fc105d40 con 0x7f48e406c680 2024-03-26T18:53:11.315 INFO:teuthology.orchestra.run.smithi102.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:53:11.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.324+0000 7f490225b700 1 -- 172.21.15.102:0/2811740000 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f48e406c680 msgr2=0x7f48e406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:11.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.324+0000 7f490225b700 1 --2- 172.21.15.102:0/2811740000 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f48e406c680 0x7f48e406eb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009fd0 tx=0x7f48ec005a90 comp rx=0 tx=0).stop 2024-03-26T18:53:11.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.324+0000 7f490225b700 1 -- 172.21.15.102:0/2811740000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48fc101350 msgr2=0x7f48fc195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:11.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.324+0000 7f490225b700 1 --2- 172.21.15.102:0/2811740000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48fc101350 0x7f48fc195ee0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f48f000e920 tx=0x7f48f000ec30 comp rx=0 tx=0).stop 2024-03-26T18:53:11.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.324+0000 7f490225b700 1 -- 172.21.15.102:0/2811740000 shutdown_connections 2024-03-26T18:53:11.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.324+0000 7f490225b700 1 --2- 172.21.15.102:0/2811740000 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f48e406c680 0x7f48e406eb30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:11.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.324+0000 7f490225b700 1 --2- 172.21.15.102:0/2811740000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48fc100150 0x7f48fc1959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:11.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.324+0000 7f490225b700 1 --2- 172.21.15.102:0/2811740000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48fc101350 0x7f48fc195ee0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:11.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.324+0000 7f490225b700 1 -- 172.21.15.102:0/2811740000 >> 172.21.15.102:0/2811740000 conn(0x7f48fc0fb6e0 msgr2=0x7f48fc104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:11.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.325+0000 7f490225b700 1 -- 172.21.15.102:0/2811740000 shutdown_connections 2024-03-26T18:53:11.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:11.325+0000 7f490225b700 1 -- 172.21.15.102:0/2811740000 wait complete. 2024-03-26T18:53:11.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:11 smithi190 ceph-mon[35043]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-03-26T18:53:11.543 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:11 smithi102 ceph-mon[29002]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-03-26T18:53:12.242 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-26T18:53:12.243 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T18:53:12.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-26T18:53:12.501 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T18:53:12.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:12 smithi102 ceph-mon[29002]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:360516069d9393362c4cc6eb9371680fe16d66ab", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:12.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:12 smithi102 ceph-mon[29002]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:53:12.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:12 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:53:12.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:12 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:53:12.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:12 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:53:12.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:12 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:53:12.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:12 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:53:12.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:12 smithi102 ceph-mon[29002]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:53:12.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:12 smithi102 ceph-mon[29002]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-26T18:53:12.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:12 smithi190 ceph-mon[35043]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:360516069d9393362c4cc6eb9371680fe16d66ab", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:12 smithi190 ceph-mon[35043]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:53:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:12 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:53:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:12 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:53:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:12 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:53:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:12 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:53:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:12 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:53:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:12 smithi190 ceph-mon[35043]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:360516069d9393362c4cc6eb9371680fe16d66ab 2024-03-26T18:53:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:12 smithi190 ceph-mon[35043]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-26T18:53:14.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 -- 172.21.15.102:0/3815765471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45e8100250 msgr2=0x7f45e81006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 --2- 172.21.15.102:0/3815765471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45e8100250 0x7f45e81006a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009910 tx=0x7f45d8009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 -- 172.21.15.102:0/3815765471 shutdown_connections 2024-03-26T18:53:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 --2- 172.21.15.102:0/3815765471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45e8100250 0x7f45e81006a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 --2- 172.21.15.102:0/3815765471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45e80ff050 0x7f45e80ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 -- 172.21.15.102:0/3815765471 >> 172.21.15.102:0/3815765471 conn(0x7f45e80fa5e0 msgr2=0x7f45e80fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 -- 172.21.15.102:0/3815765471 shutdown_connections 2024-03-26T18:53:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 -- 172.21.15.102:0/3815765471 wait complete. 2024-03-26T18:53:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 Processor -- start 2024-03-26T18:53:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 -- start start 2024-03-26T18:53:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45e80ff050 0x7f45e8193760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45e8100250 0x7f45e8193ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.552+0000 7f45f04a2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e81942c0 con 0x7f45e8100250 2024-03-26T18:53:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.553+0000 7f45f04a2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e8194400 con 0x7f45e80ff050 2024-03-26T18:53:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.553+0000 7f45ee23e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45e80ff050 0x7f45e8193760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.553+0000 7f45ee23e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45e80ff050 0x7f45e8193760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:50944/0 (socket says 172.21.15.102:50944) 2024-03-26T18:53:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.553+0000 7f45ee23e700 1 -- 172.21.15.102:0/2342464737 learned_addr learned my addr 172.21.15.102:0/2342464737 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.553+0000 7f45eda3d700 1 --2- 172.21.15.102:0/2342464737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45e8100250 0x7f45e8193ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.553+0000 7f45ee23e700 1 -- 172.21.15.102:0/2342464737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45e8100250 msgr2=0x7f45e8193ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.553+0000 7f45ee23e700 1 --2- 172.21.15.102:0/2342464737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45e8100250 0x7f45e8193ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.553+0000 7f45ee23e700 1 -- 172.21.15.102:0/2342464737 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d80095f0 con 0x7f45e80ff050 2024-03-26T18:53:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.553+0000 7f45ee23e700 1 --2- 172.21.15.102:0/2342464737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45e80ff050 0x7f45e8193760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f45e400e8f0 tx=0x7f45e400ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.553+0000 7f45df7fe700 1 -- 172.21.15.102:0/2342464737 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45e4009790 con 0x7f45e80ff050 2024-03-26T18:53:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.553+0000 7f45f04a2700 1 -- 172.21.15.102:0/2342464737 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e8198eb0 con 0x7f45e80ff050 2024-03-26T18:53:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.554+0000 7f45df7fe700 1 -- 172.21.15.102:0/2342464737 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f45e400c810 con 0x7f45e80ff050 2024-03-26T18:53:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.554+0000 7f45f04a2700 1 -- 172.21.15.102:0/2342464737 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e8199400 con 0x7f45e80ff050 2024-03-26T18:53:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.554+0000 7f45df7fe700 1 -- 172.21.15.102:0/2342464737 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45e4003d20 con 0x7f45e80ff050 2024-03-26T18:53:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.555+0000 7f45f04a2700 1 -- 172.21.15.102:0/2342464737 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45cc005320 con 0x7f45e80ff050 2024-03-26T18:53:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.555+0000 7f45df7fe700 1 -- 172.21.15.102:0/2342464737 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f45e40041f0 con 0x7f45e80ff050 2024-03-26T18:53:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.556+0000 7f45df7fe700 1 --2- 172.21.15.102:0/2342464737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f45d406c680 0x7f45d406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:14.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.556+0000 7f45df7fe700 1 -- 172.21.15.102:0/2342464737 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f45e4014070 con 0x7f45e80ff050 2024-03-26T18:53:14.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.556+0000 7f45eda3d700 1 --2- 172.21.15.102:0/2342464737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f45d406c680 0x7f45d406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:14.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.556+0000 7f45eda3d700 1 --2- 172.21.15.102:0/2342464737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f45d406c680 0x7f45d406eb30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009fd0 tx=0x7f45d8005940 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:14.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.560+0000 7f45df7fe700 1 -- 172.21.15.102:0/2342464737 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f45e4059e20 con 0x7f45e80ff050 2024-03-26T18:53:14.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.850+0000 7f45f04a2700 1 -- 172.21.15.102:0/2342464737 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f45cc000bf0 con 0x7f45d406c680 2024-03-26T18:53:14.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.852+0000 7f45df7fe700 1 -- 172.21.15.102:0/2342464737 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f45cc000bf0 con 0x7f45d406c680 2024-03-26T18:53:14.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.854+0000 7f45f04a2700 1 -- 172.21.15.102:0/2342464737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f45d406c680 msgr2=0x7f45d406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:14.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.854+0000 7f45f04a2700 1 --2- 172.21.15.102:0/2342464737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f45d406c680 0x7f45d406eb30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009fd0 tx=0x7f45d8005940 comp rx=0 tx=0).stop 2024-03-26T18:53:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.854+0000 7f45f04a2700 1 -- 172.21.15.102:0/2342464737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45e80ff050 msgr2=0x7f45e8193760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.854+0000 7f45f04a2700 1 --2- 172.21.15.102:0/2342464737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45e80ff050 0x7f45e8193760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f45e400e8f0 tx=0x7f45e400ec00 comp rx=0 tx=0).stop 2024-03-26T18:53:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.854+0000 7f45f04a2700 1 -- 172.21.15.102:0/2342464737 shutdown_connections 2024-03-26T18:53:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.854+0000 7f45f04a2700 1 --2- 172.21.15.102:0/2342464737 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f45d406c680 0x7f45d406eb30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.854+0000 7f45f04a2700 1 --2- 172.21.15.102:0/2342464737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45e80ff050 0x7f45e8193760 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.854+0000 7f45f04a2700 1 --2- 172.21.15.102:0/2342464737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45e8100250 0x7f45e8193ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.854+0000 7f45f04a2700 1 -- 172.21.15.102:0/2342464737 >> 172.21.15.102:0/2342464737 conn(0x7f45e80fa5e0 msgr2=0x7f45e80683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.854+0000 7f45f04a2700 1 -- 172.21.15.102:0/2342464737 shutdown_connections 2024-03-26T18:53:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.854+0000 7f45f04a2700 1 -- 172.21.15.102:0/2342464737 wait complete. 2024-03-26T18:53:14.859 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:53:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.982+0000 7ff6bafa5700 1 -- 172.21.15.102:0/1695410214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6b4103500 msgr2=0x7ff6b41058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.982+0000 7ff6bafa5700 1 --2- 172.21.15.102:0/1695410214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6b4103500 0x7ff6b41058e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009910 tx=0x7ff6a0009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.982+0000 7ff6bafa5700 1 -- 172.21.15.102:0/1695410214 shutdown_connections 2024-03-26T18:53:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.982+0000 7ff6bafa5700 1 --2- 172.21.15.102:0/1695410214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6b4103500 0x7ff6b41058e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.982+0000 7ff6bafa5700 1 --2- 172.21.15.102:0/1695410214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b4100be0 0x7ff6b4102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.982+0000 7ff6bafa5700 1 -- 172.21.15.102:0/1695410214 >> 172.21.15.102:0/1695410214 conn(0x7ff6b40fa5e0 msgr2=0x7ff6b40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.982+0000 7ff6bafa5700 1 -- 172.21.15.102:0/1695410214 shutdown_connections 2024-03-26T18:53:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.982+0000 7ff6bafa5700 1 -- 172.21.15.102:0/1695410214 wait complete. 2024-03-26T18:53:14.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.983+0000 7ff6bafa5700 1 Processor -- start 2024-03-26T18:53:14.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.983+0000 7ff6bafa5700 1 -- start start 2024-03-26T18:53:14.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.983+0000 7ff6bafa5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b4100be0 0x7ff6b4195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:14.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.983+0000 7ff6bafa5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6b4103500 0x7ff6b4195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:14.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.983+0000 7ff6bafa5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b4196490 con 0x7ff6b4103500 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.983+0000 7ff6bafa5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b41965d0 con 0x7ff6b4100be0 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.984+0000 7ff6b8d41700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b4100be0 0x7ff6b4195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.984+0000 7ff6b8d41700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b4100be0 0x7ff6b4195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:50964/0 (socket says 172.21.15.102:50964) 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.984+0000 7ff6b8d41700 1 -- 172.21.15.102:0/3201918713 learned_addr learned my addr 172.21.15.102:0/3201918713 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.984+0000 7ff6b3fff700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6b4103500 0x7ff6b4195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.984+0000 7ff6b8d41700 1 -- 172.21.15.102:0/3201918713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6b4103500 msgr2=0x7ff6b4195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.984+0000 7ff6b8d41700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6b4103500 0x7ff6b4195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.984+0000 7ff6b8d41700 1 -- 172.21.15.102:0/3201918713 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a00095f0 con 0x7ff6b4100be0 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.984+0000 7ff6b3fff700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6b4103500 0x7ff6b4195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.984+0000 7ff6b8d41700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b4100be0 0x7ff6b4195930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800d8d0 tx=0x7ff6a800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.984+0000 7ff6b1ffb700 1 -- 172.21.15.102:0/3201918713 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6a8009940 con 0x7ff6b4100be0 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.985+0000 7ff6bafa5700 1 -- 172.21.15.102:0/3201918713 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b419b080 con 0x7ff6b4100be0 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.985+0000 7ff6b1ffb700 1 -- 172.21.15.102:0/3201918713 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff6a8010460 con 0x7ff6b4100be0 2024-03-26T18:53:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.985+0000 7ff6b1ffb700 1 -- 172.21.15.102:0/3201918713 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6a800f5f0 con 0x7ff6b4100be0 2024-03-26T18:53:14.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.985+0000 7ff6bafa5700 1 -- 172.21.15.102:0/3201918713 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b419b5d0 con 0x7ff6b4100be0 2024-03-26T18:53:14.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.986+0000 7ff6b1ffb700 1 -- 172.21.15.102:0/3201918713 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7ff6a800f7e0 con 0x7ff6b4100be0 2024-03-26T18:53:14.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.986+0000 7ff6bafa5700 1 -- 172.21.15.102:0/3201918713 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b418fb70 con 0x7ff6b4100be0 2024-03-26T18:53:14.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.987+0000 7ff6b1ffb700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff6a406c680 0x7ff6a406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:14.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.987+0000 7ff6b3fff700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff6a406c680 0x7ff6a406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:14.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.987+0000 7ff6b1ffb700 1 -- 172.21.15.102:0/3201918713 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff6a808b890 con 0x7ff6b4100be0 2024-03-26T18:53:14.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.987+0000 7ff6b3fff700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff6a406c680 0x7ff6a406eb30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff6a000b5c0 tx=0x7ff6a0005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:14.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:14.991+0000 7ff6b1ffb700 1 -- 172.21.15.102:0/3201918713 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff6a8059b00 con 0x7ff6b4100be0 2024-03-26T18:53:15.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:14 smithi102 ceph-mon[29002]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-26T18:53:15.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.279+0000 7ff6bafa5700 1 -- 172.21.15.102:0/3201918713 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6b4061190 con 0x7ff6a406c680 2024-03-26T18:53:15.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.280+0000 7ff6b1ffb700 1 -- 172.21.15.102:0/3201918713 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7ff6b4061190 con 0x7ff6a406c680 2024-03-26T18:53:15.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.283+0000 7ff6bafa5700 1 -- 172.21.15.102:0/3201918713 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff6a406c680 msgr2=0x7ff6a406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:15.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.283+0000 7ff6bafa5700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff6a406c680 0x7ff6a406eb30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff6a000b5c0 tx=0x7ff6a0005960 comp rx=0 tx=0).stop 2024-03-26T18:53:15.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.283+0000 7ff6bafa5700 1 -- 172.21.15.102:0/3201918713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b4100be0 msgr2=0x7ff6b4195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:15.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.283+0000 7ff6bafa5700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b4100be0 0x7ff6b4195930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800d8d0 tx=0x7ff6a800dc90 comp rx=0 tx=0).stop 2024-03-26T18:53:15.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.283+0000 7ff6bafa5700 1 -- 172.21.15.102:0/3201918713 shutdown_connections 2024-03-26T18:53:15.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.283+0000 7ff6bafa5700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff6a406c680 0x7ff6a406eb30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.283+0000 7ff6bafa5700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b4100be0 0x7ff6b4195930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.283+0000 7ff6bafa5700 1 --2- 172.21.15.102:0/3201918713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6b4103500 0x7ff6b4195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.283+0000 7ff6bafa5700 1 -- 172.21.15.102:0/3201918713 >> 172.21.15.102:0/3201918713 conn(0x7ff6b40fa5e0 msgr2=0x7ff6b40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:15.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.283+0000 7ff6bafa5700 1 -- 172.21.15.102:0/3201918713 shutdown_connections 2024-03-26T18:53:15.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.283+0000 7ff6bafa5700 1 -- 172.21.15.102:0/3201918713 wait complete. 2024-03-26T18:53:15.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:14 smithi190 ceph-mon[35043]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-26T18:53:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.412+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/211339142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d80ff050 msgr2=0x7fa4d80ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.412+0000 7fa4dd1f4700 1 --2- 172.21.15.102:0/211339142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d80ff050 0x7fa4d80ff460 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009910 tx=0x7fa4c8009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.412+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/211339142 shutdown_connections 2024-03-26T18:53:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.412+0000 7fa4dd1f4700 1 --2- 172.21.15.102:0/211339142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8100250 0x7fa4d81006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.412+0000 7fa4dd1f4700 1 --2- 172.21.15.102:0/211339142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d80ff050 0x7fa4d80ff460 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.412+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/211339142 >> 172.21.15.102:0/211339142 conn(0x7fa4d80fa5e0 msgr2=0x7fa4d80fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.412+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/211339142 shutdown_connections 2024-03-26T18:53:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.412+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/211339142 wait complete. 2024-03-26T18:53:15.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.413+0000 7fa4dd1f4700 1 Processor -- start 2024-03-26T18:53:15.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.413+0000 7fa4dd1f4700 1 -- start start 2024-03-26T18:53:15.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.413+0000 7fa4dd1f4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d80ff050 0x7fa4d8102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:15.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.413+0000 7fa4dd1f4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8100250 0x7fa4d8101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:15.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.413+0000 7fa4dd1f4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d8101b70 con 0x7fa4d80ff050 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.413+0000 7fa4dd1f4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d8101cb0 con 0x7fa4d8100250 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.413+0000 7fa4d6d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d80ff050 0x7fa4d8102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.413+0000 7fa4d6d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d80ff050 0x7fa4d8102fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43910/0 (socket says 172.21.15.102:43910) 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.413+0000 7fa4d6d9d700 1 -- 172.21.15.102:0/2039465114 learned_addr learned my addr 172.21.15.102:0/2039465114 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.414+0000 7fa4d6d9d700 1 -- 172.21.15.102:0/2039465114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8100250 msgr2=0x7fa4d8101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.414+0000 7fa4d659c700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8100250 0x7fa4d8101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.414+0000 7fa4d6d9d700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8100250 0x7fa4d8101630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.414+0000 7fa4d6d9d700 1 -- 172.21.15.102:0/2039465114 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c80095f0 con 0x7fa4d80ff050 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.414+0000 7fa4d659c700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8100250 0x7fa4d8101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.414+0000 7fa4d6d9d700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d80ff050 0x7fa4d8102fe0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800bda0 tx=0x7fa4c800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.415+0000 7fa4bffff700 1 -- 172.21.15.102:0/2039465114 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4c801d070 con 0x7fa4d80ff050 2024-03-26T18:53:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.415+0000 7fa4bffff700 1 -- 172.21.15.102:0/2039465114 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa4c8004c50 con 0x7fa4d80ff050 2024-03-26T18:53:15.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.415+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/2039465114 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d8101f30 con 0x7fa4d80ff050 2024-03-26T18:53:15.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.415+0000 7fa4bffff700 1 -- 172.21.15.102:0/2039465114 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4c800f690 con 0x7fa4d80ff050 2024-03-26T18:53:15.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.415+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/2039465114 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d8102360 con 0x7fa4d80ff050 2024-03-26T18:53:15.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.416+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/2039465114 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d8107c20 con 0x7fa4d80ff050 2024-03-26T18:53:15.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.417+0000 7fa4bffff700 1 -- 172.21.15.102:0/2039465114 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fa4c800f7f0 con 0x7fa4d80ff050 2024-03-26T18:53:15.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.417+0000 7fa4bffff700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4c006c680 0x7fa4c006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:15.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.417+0000 7fa4bffff700 1 -- 172.21.15.102:0/2039465114 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa4c808d050 con 0x7fa4d80ff050 2024-03-26T18:53:15.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.417+0000 7fa4d659c700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4c006c680 0x7fa4c006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:15.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.418+0000 7fa4d659c700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4c006c680 0x7fa4c006eb30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc007900 tx=0x7fa4cc008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:15.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.422+0000 7fa4bffff700 1 -- 172.21.15.102:0/2039465114 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa4c805b2c0 con 0x7fa4d80ff050 2024-03-26T18:53:15.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.680+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/2039465114 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4d8068ae0 con 0x7fa4c006c680 2024-03-26T18:53:15.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.699+0000 7fa4bffff700 1 -- 172.21.15.102:0/2039465114 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7fa4d8068ae0 con 0x7fa4c006c680 2024-03-26T18:53:15.692 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:53:15.692 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (4m) 26s ago 6m 25.5M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:53:15.692 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (6m) 26s ago 6m 7457k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:53:15.692 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (5m) 31s ago 5m 7356k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:53:15.692 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (6m) 26s ago 6m 6665k - 18.2.0 dc2bc1663786 bfac00217c68 2024-03-26T18:53:15.692 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (5m) 31s ago 5m 6656k - 18.2.0 dc2bc1663786 7379f1f6b746 2024-03-26T18:53:15.692 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (4m) 26s ago 6m 85.5M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (45s) 26s ago 45s 15.9M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (51s) 26s ago 51s 11.5M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (43s) 31s ago 42s 12.1M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (49s) 31s ago 48s 14.9M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:9283,8765,8443 running (7m) 26s ago 7m 487M - 18.2.0 dc2bc1663786 61d2bd7101b6 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (5m) 31s ago 5m 440M - 18.2.0 dc2bc1663786 57e44bfc8ede 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (7m) 26s ago 7m 45.1M 2048M 18.2.0 dc2bc1663786 b5abd46c3973 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (5m) 31s ago 5m 35.3M 2048M 18.2.0 dc2bc1663786 6bd913d483f0 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (6m) 26s ago 6m 15.9M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (5m) 31s ago 5m 15.3M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (4m) 26s ago 4m 47.9M 3946M 18.2.0 dc2bc1663786 3e2065fb5fbc 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (4m) 26s ago 4m 48.5M 3946M 18.2.0 dc2bc1663786 0bf773515cc1 2024-03-26T18:53:15.693 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (3m) 26s ago 3m 46.3M 3946M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:53:15.694 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (3m) 31s ago 3m 49.3M 4970M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:53:15.694 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (2m) 31s ago 2m 44.9M 4970M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:53:15.694 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (2m) 31s ago 2m 44.1M 4970M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:53:15.694 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (4m) 26s ago 5m 61.7M - 2.43.0 a07b618ecd1d d0dc2ca7f41f 2024-03-26T18:53:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.702+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/2039465114 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4c006c680 msgr2=0x7fa4c006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.702+0000 7fa4dd1f4700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4c006c680 0x7fa4c006eb30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc007900 tx=0x7fa4cc008040 comp rx=0 tx=0).stop 2024-03-26T18:53:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.702+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/2039465114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d80ff050 msgr2=0x7fa4d8102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.702+0000 7fa4dd1f4700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d80ff050 0x7fa4d8102fe0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800bda0 tx=0x7fa4c800be80 comp rx=0 tx=0).stop 2024-03-26T18:53:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.702+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/2039465114 shutdown_connections 2024-03-26T18:53:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.702+0000 7fa4dd1f4700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fa4c006c680 0x7fa4c006eb30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.702+0000 7fa4dd1f4700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d80ff050 0x7fa4d8102fe0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.702+0000 7fa4dd1f4700 1 --2- 172.21.15.102:0/2039465114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8100250 0x7fa4d8101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.702+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/2039465114 >> 172.21.15.102:0/2039465114 conn(0x7fa4d80fa5e0 msgr2=0x7fa4d80683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:15.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.703+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/2039465114 shutdown_connections 2024-03-26T18:53:15.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.703+0000 7fa4dd1f4700 1 -- 172.21.15.102:0/2039465114 wait complete. 2024-03-26T18:53:15.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.826+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2944571462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae40fe410 msgr2=0x7f2ae40fe820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.826+0000 7f2aea8e4700 1 --2- 172.21.15.102:0/2944571462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae40fe410 0x7f2ae40fe820 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4009910 tx=0x7f2ad4009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.826+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2944571462 shutdown_connections 2024-03-26T18:53:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.826+0000 7f2aea8e4700 1 --2- 172.21.15.102:0/2944571462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae41013f0 0x7f2ae4101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.826+0000 7f2aea8e4700 1 --2- 172.21.15.102:0/2944571462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae40fe410 0x7f2ae40fe820 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.826+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2944571462 >> 172.21.15.102:0/2944571462 conn(0x7f2ae40f9d80 msgr2=0x7f2ae40fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.826+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2944571462 shutdown_connections 2024-03-26T18:53:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.826+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2944571462 wait complete. 2024-03-26T18:53:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.827+0000 7f2aea8e4700 1 Processor -- start 2024-03-26T18:53:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.827+0000 7f2aea8e4700 1 -- start start 2024-03-26T18:53:15.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.827+0000 7f2aea8e4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae40fe410 0x7f2ae4195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.827+0000 7f2aea8e4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae41013f0 0x7f2ae4195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.827+0000 7f2aea8e4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae41964e0 con 0x7f2ae40fe410 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.827+0000 7f2aea8e4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae4196620 con 0x7f2ae41013f0 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.827+0000 7f2ae3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae40fe410 0x7f2ae4195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2ae3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae40fe410 0x7f2ae4195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43928/0 (socket says 172.21.15.102:43928) 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2ae3fff700 1 -- 172.21.15.102:0/2536057203 learned_addr learned my addr 172.21.15.102:0/2536057203 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2ae37fe700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae41013f0 0x7f2ae4195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2ae3fff700 1 -- 172.21.15.102:0/2536057203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae41013f0 msgr2=0x7f2ae4195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2ae3fff700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae41013f0 0x7f2ae4195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2ae3fff700 1 -- 172.21.15.102:0/2536057203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad40095f0 con 0x7f2ae40fe410 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2ae37fe700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae41013f0 0x7f2ae4195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2ae3fff700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae40fe410 0x7f2ae4195980 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4000c00 tx=0x7f2ad400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2ae17fa700 1 -- 172.21.15.102:0/2536057203 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ad401d070 con 0x7f2ae40fe410 2024-03-26T18:53:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2ae17fa700 1 -- 172.21.15.102:0/2536057203 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2ad4005dc0 con 0x7f2ae40fe410 2024-03-26T18:53:15.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2ae17fa700 1 -- 172.21.15.102:0/2536057203 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ad4022470 con 0x7f2ae40fe410 2024-03-26T18:53:15.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2536057203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae419b070 con 0x7f2ae40fe410 2024-03-26T18:53:15.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.828+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2536057203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae419b560 con 0x7f2ae40fe410 2024-03-26T18:53:15.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.829+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2536057203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae4108c90 con 0x7f2ae40fe410 2024-03-26T18:53:15.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.830+0000 7f2ae17fa700 1 -- 172.21.15.102:0/2536057203 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f2ad4003810 con 0x7f2ae40fe410 2024-03-26T18:53:15.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.831+0000 7f2ae17fa700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f2acc06c680 0x7f2acc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:15.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.831+0000 7f2ae17fa700 1 -- 172.21.15.102:0/2536057203 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2ad408cda0 con 0x7f2ae40fe410 2024-03-26T18:53:15.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.831+0000 7f2ae37fe700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f2acc06c680 0x7f2acc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:15.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.832+0000 7f2ae37fe700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f2acc06c680 0x7f2acc06eb30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8011490 tx=0x7f2ad8009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:15.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:15.836+0000 7f2ae17fa700 1 -- 172.21.15.102:0/2536057203 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2ad405af30 con 0x7f2ae40fe410 2024-03-26T18:53:16.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.198+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2536057203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2ae419bfd0 con 0x7f2ae40fe410 2024-03-26T18:53:16.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.198+0000 7f2ae17fa700 1 -- 172.21.15.102:0/2536057203 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f2ad405aac0 con 0x7f2ae40fe410 2024-03-26T18:53:16.191 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:53:16.191 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:53:16.191 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-03-26T18:53:16.191 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 14 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:53:16.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.201+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2536057203 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f2acc06c680 msgr2=0x7f2acc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:16.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.201+0000 7f2aea8e4700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f2acc06c680 0x7f2acc06eb30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8011490 tx=0x7f2ad8009260 comp rx=0 tx=0).stop 2024-03-26T18:53:16.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.201+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2536057203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae40fe410 msgr2=0x7f2ae4195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:16.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.201+0000 7f2aea8e4700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae40fe410 0x7f2ae4195980 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4000c00 tx=0x7f2ad400f690 comp rx=0 tx=0).stop 2024-03-26T18:53:16.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.201+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2536057203 shutdown_connections 2024-03-26T18:53:16.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.201+0000 7f2aea8e4700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f2acc06c680 0x7f2acc06eb30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.201+0000 7f2aea8e4700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae40fe410 0x7f2ae4195980 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.201+0000 7f2aea8e4700 1 --2- 172.21.15.102:0/2536057203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae41013f0 0x7f2ae4195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.201+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2536057203 >> 172.21.15.102:0/2536057203 conn(0x7f2ae40f9d80 msgr2=0x7f2ae4104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:16.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.201+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2536057203 shutdown_connections 2024-03-26T18:53:16.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.201+0000 7f2aea8e4700 1 -- 172.21.15.102:0/2536057203 wait complete. 2024-03-26T18:53:16.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.327+0000 7f225a0c7700 1 -- 172.21.15.102:0/4000572098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2254100250 msgr2=0x7f22541006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:16.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.327+0000 7f225a0c7700 1 --2- 172.21.15.102:0/4000572098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2254100250 0x7f22541006a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2248009910 tx=0x7f2248009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:16.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.327+0000 7f225a0c7700 1 -- 172.21.15.102:0/4000572098 shutdown_connections 2024-03-26T18:53:16.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.327+0000 7f225a0c7700 1 --2- 172.21.15.102:0/4000572098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2254100250 0x7f22541006a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.327+0000 7f225a0c7700 1 --2- 172.21.15.102:0/4000572098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22540ff050 0x7f22540ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.327+0000 7f225a0c7700 1 -- 172.21.15.102:0/4000572098 >> 172.21.15.102:0/4000572098 conn(0x7f22540fa600 msgr2=0x7f22540fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:16.320 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:16 smithi102 ceph-mon[29002]: from='client.24351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:16.320 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:16 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2536057203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:53:16.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.327+0000 7f225a0c7700 1 -- 172.21.15.102:0/4000572098 shutdown_connections 2024-03-26T18:53:16.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.327+0000 7f225a0c7700 1 -- 172.21.15.102:0/4000572098 wait complete. 2024-03-26T18:53:16.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.328+0000 7f225a0c7700 1 Processor -- start 2024-03-26T18:53:16.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.328+0000 7f225a0c7700 1 -- start start 2024-03-26T18:53:16.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.328+0000 7f225a0c7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22540ff050 0x7f2254102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:16.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.328+0000 7f225a0c7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2254100250 0x7f2254101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:16.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.328+0000 7f225a0c7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22541995f0 con 0x7f22540ff050 2024-03-26T18:53:16.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.328+0000 7f225a0c7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2254199730 con 0x7f2254100250 2024-03-26T18:53:16.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.328+0000 7f22537fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22540ff050 0x7f2254102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:16.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.328+0000 7f2252ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2254100250 0x7f2254101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:16.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.328+0000 7f2252ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2254100250 0x7f2254101630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:51038/0 (socket says 172.21.15.102:51038) 2024-03-26T18:53:16.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.328+0000 7f22537fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22540ff050 0x7f2254102f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43942/0 (socket says 172.21.15.102:43942) 2024-03-26T18:53:16.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.328+0000 7f2252ffd700 1 -- 172.21.15.102:0/3784062689 learned_addr learned my addr 172.21.15.102:0/3784062689 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:16.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.329+0000 7f2252ffd700 1 -- 172.21.15.102:0/3784062689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22540ff050 msgr2=0x7f2254102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:16.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.329+0000 7f2252ffd700 1 --2- 172.21.15.102:0/3784062689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22540ff050 0x7f2254102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.329+0000 7f2252ffd700 1 -- 172.21.15.102:0/3784062689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22480095f0 con 0x7f2254100250 2024-03-26T18:53:16.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.329+0000 7f2252ffd700 1 --2- 172.21.15.102:0/3784062689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2254100250 0x7f2254101630 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2248000c00 tx=0x7f224800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:16.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.329+0000 7f2250ff9700 1 -- 172.21.15.102:0/3784062689 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f224801d070 con 0x7f2254100250 2024-03-26T18:53:16.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.329+0000 7f2250ff9700 1 -- 172.21.15.102:0/3784062689 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2248005dc0 con 0x7f2254100250 2024-03-26T18:53:16.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.329+0000 7f225a0c7700 1 -- 172.21.15.102:0/3784062689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22541999b0 con 0x7f2254100250 2024-03-26T18:53:16.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.329+0000 7f225a0c7700 1 -- 172.21.15.102:0/3784062689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2254199ea0 con 0x7f2254100250 2024-03-26T18:53:16.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.330+0000 7f2250ff9700 1 -- 172.21.15.102:0/3784062689 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22480226a0 con 0x7f2254100250 2024-03-26T18:53:16.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.330+0000 7f225a0c7700 1 -- 172.21.15.102:0/3784062689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2254107ba0 con 0x7f2254100250 2024-03-26T18:53:16.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.332+0000 7f2250ff9700 1 -- 172.21.15.102:0/3784062689 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f22480038f0 con 0x7f2254100250 2024-03-26T18:53:16.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.332+0000 7f2250ff9700 1 --2- 172.21.15.102:0/3784062689 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f223c06c680 0x7f223c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:16.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.332+0000 7f2250ff9700 1 -- 172.21.15.102:0/3784062689 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f224808d2a0 con 0x7f2254100250 2024-03-26T18:53:16.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.333+0000 7f22537fe700 1 --2- 172.21.15.102:0/3784062689 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f223c06c680 0x7f223c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:16.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.334+0000 7f22537fe700 1 --2- 172.21.15.102:0/3784062689 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f223c06c680 0x7f223c06eb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f22541000b0 tx=0x7f2244009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:16.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.336+0000 7f2250ff9700 1 -- 172.21.15.102:0/3784062689 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f224805b4d0 con 0x7f2254100250 2024-03-26T18:53:16.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:16 smithi190 ceph-mon[35043]: from='client.24351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:16.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:16 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2536057203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:53:16.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.628+0000 7f225a0c7700 1 -- 172.21.15.102:0/3784062689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2254107d90 con 0x7f2254100250 2024-03-26T18:53:16.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.628+0000 7f2250ff9700 1 -- 172.21.15.102:0/3784062689 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 17 v17) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f224805b060 con 0x7f2254100250 2024-03-26T18:53:16.621 INFO:teuthology.orchestra.run.smithi102.stdout:e17 2024-03-26T18:53:16.621 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:53:16.621 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:53:16.621 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:53:16.621 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:53:16.621 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:53:16.621 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:53:16.621 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 17 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:52:39.046175+0000 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:53:16.622 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:53:16.623 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:53:16.624 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:53:16.624 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:53:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.631+0000 7f225a0c7700 1 -- 172.21.15.102:0/3784062689 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f223c06c680 msgr2=0x7f223c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.631+0000 7f225a0c7700 1 --2- 172.21.15.102:0/3784062689 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f223c06c680 0x7f223c06eb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f22541000b0 tx=0x7f2244009260 comp rx=0 tx=0).stop 2024-03-26T18:53:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.631+0000 7f225a0c7700 1 -- 172.21.15.102:0/3784062689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2254100250 msgr2=0x7f2254101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.631+0000 7f225a0c7700 1 --2- 172.21.15.102:0/3784062689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2254100250 0x7f2254101630 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2248000c00 tx=0x7f224800f690 comp rx=0 tx=0).stop 2024-03-26T18:53:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.631+0000 7f225a0c7700 1 -- 172.21.15.102:0/3784062689 shutdown_connections 2024-03-26T18:53:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.631+0000 7f225a0c7700 1 --2- 172.21.15.102:0/3784062689 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f223c06c680 0x7f223c06eb30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.631+0000 7f225a0c7700 1 --2- 172.21.15.102:0/3784062689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22540ff050 0x7f2254102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.631+0000 7f225a0c7700 1 --2- 172.21.15.102:0/3784062689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2254100250 0x7f2254101630 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.631+0000 7f225a0c7700 1 -- 172.21.15.102:0/3784062689 >> 172.21.15.102:0/3784062689 conn(0x7f22540fa600 msgr2=0x7f2254068c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.631+0000 7f225a0c7700 1 -- 172.21.15.102:0/3784062689 shutdown_connections 2024-03-26T18:53:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.631+0000 7f225a0c7700 1 -- 172.21.15.102:0/3784062689 wait complete. 2024-03-26T18:53:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 17 2024-03-26T18:53:16.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.789+0000 7f5976bb5700 1 -- 172.21.15.102:0/2662822175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59700ff050 msgr2=0x7f59700ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:16.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.789+0000 7f5976bb5700 1 --2- 172.21.15.102:0/2662822175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59700ff050 0x7f59700ff460 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5964009910 tx=0x7f5964009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:16.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.789+0000 7f5976bb5700 1 -- 172.21.15.102:0/2662822175 shutdown_connections 2024-03-26T18:53:16.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.789+0000 7f5976bb5700 1 --2- 172.21.15.102:0/2662822175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5970100250 0x7f59701006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.789+0000 7f5976bb5700 1 --2- 172.21.15.102:0/2662822175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59700ff050 0x7f59700ff460 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.789+0000 7f5976bb5700 1 -- 172.21.15.102:0/2662822175 >> 172.21.15.102:0/2662822175 conn(0x7f59700fa600 msgr2=0x7f59700fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:16.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.789+0000 7f5976bb5700 1 -- 172.21.15.102:0/2662822175 shutdown_connections 2024-03-26T18:53:16.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.789+0000 7f5976bb5700 1 -- 172.21.15.102:0/2662822175 wait complete. 2024-03-26T18:53:16.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.790+0000 7f5976bb5700 1 Processor -- start 2024-03-26T18:53:16.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.790+0000 7f5976bb5700 1 -- start start 2024-03-26T18:53:16.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.790+0000 7f5976bb5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59700ff050 0x7f5970196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:16.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.790+0000 7f5976bb5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5970100250 0x7f5970197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:16.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.790+0000 7f5976bb5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59701937f0 con 0x7f5970100250 2024-03-26T18:53:16.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.790+0000 7f5976bb5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5970193960 con 0x7f59700ff050 2024-03-26T18:53:16.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.790+0000 7f5974951700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59700ff050 0x7f5970196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:16.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.790+0000 7f5974951700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59700ff050 0x7f5970196d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:51054/0 (socket says 172.21.15.102:51054) 2024-03-26T18:53:16.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.790+0000 7f5974951700 1 -- 172.21.15.102:0/953722351 learned_addr learned my addr 172.21.15.102:0/953722351 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:16.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.790+0000 7f596ffff700 1 --2- 172.21.15.102:0/953722351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5970100250 0x7f5970197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:16.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.791+0000 7f596ffff700 1 -- 172.21.15.102:0/953722351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59700ff050 msgr2=0x7f5970196d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:16.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.791+0000 7f596ffff700 1 --2- 172.21.15.102:0/953722351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59700ff050 0x7f5970196d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:16.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.791+0000 7f596ffff700 1 -- 172.21.15.102:0/953722351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59640095f0 con 0x7f5970100250 2024-03-26T18:53:16.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.791+0000 7f596ffff700 1 --2- 172.21.15.102:0/953722351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5970100250 0x7f5970197240 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f595c00d8d0 tx=0x7f595c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:16.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.791+0000 7f596dffb700 1 -- 172.21.15.102:0/953722351 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f595c00f860 con 0x7f5970100250 2024-03-26T18:53:16.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.791+0000 7f596dffb700 1 -- 172.21.15.102:0/953722351 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f595c00fe70 con 0x7f5970100250 2024-03-26T18:53:16.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.791+0000 7f5976bb5700 1 -- 172.21.15.102:0/953722351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5970193c40 con 0x7f5970100250 2024-03-26T18:53:16.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.792+0000 7f596dffb700 1 -- 172.21.15.102:0/953722351 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f595c00e5e0 con 0x7f5970100250 2024-03-26T18:53:16.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.792+0000 7f5976bb5700 1 -- 172.21.15.102:0/953722351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5970194210 con 0x7f5970100250 2024-03-26T18:53:16.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.793+0000 7f5976bb5700 1 -- 172.21.15.102:0/953722351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5970106a90 con 0x7f5970100250 2024-03-26T18:53:16.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.795+0000 7f596dffb700 1 -- 172.21.15.102:0/953722351 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f595c010460 con 0x7f5970100250 2024-03-26T18:53:16.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.795+0000 7f596dffb700 1 --2- 172.21.15.102:0/953722351 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f596006c7b0 0x7f596006ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:16.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.796+0000 7f596dffb700 1 -- 172.21.15.102:0/953722351 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f595c08b460 con 0x7f5970100250 2024-03-26T18:53:16.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.796+0000 7f5974951700 1 --2- 172.21.15.102:0/953722351 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f596006c7b0 0x7f596006ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:16.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.796+0000 7f5974951700 1 --2- 172.21.15.102:0/953722351 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f596006c7b0 0x7f596006ec60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5964006010 tx=0x7f596400b540 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:16.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:16.801+0000 7f596dffb700 1 -- 172.21.15.102:0/953722351 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f595c0595d0 con 0x7f5970100250 2024-03-26T18:53:17.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.215+0000 7f5976bb5700 1 -- 172.21.15.102:0/953722351 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5970104ba0 con 0x7f596006c7b0 2024-03-26T18:53:17.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.217+0000 7f596dffb700 1 -- 172.21.15.102:0/953722351 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f5970104ba0 con 0x7f596006c7b0 2024-03-26T18:53:17.210 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:53:17.210 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:360516069d9393362c4cc6eb9371680fe16d66ab", 2024-03-26T18:53:17.210 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:53:17.210 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:53:17.210 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-03-26T18:53:17.210 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "", 2024-03-26T18:53:17.210 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:360516069d9393362c4cc6eb9371680fe16d66ab image", 2024-03-26T18:53:17.210 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:53:17.210 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:53:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.219+0000 7f5976bb5700 1 -- 172.21.15.102:0/953722351 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f596006c7b0 msgr2=0x7f596006ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.219+0000 7f5976bb5700 1 --2- 172.21.15.102:0/953722351 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f596006c7b0 0x7f596006ec60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5964006010 tx=0x7f596400b540 comp rx=0 tx=0).stop 2024-03-26T18:53:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.219+0000 7f5976bb5700 1 -- 172.21.15.102:0/953722351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5970100250 msgr2=0x7f5970197240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.219+0000 7f5976bb5700 1 --2- 172.21.15.102:0/953722351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5970100250 0x7f5970197240 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f595c00d8d0 tx=0x7f595c00dbe0 comp rx=0 tx=0).stop 2024-03-26T18:53:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.220+0000 7f5976bb5700 1 -- 172.21.15.102:0/953722351 shutdown_connections 2024-03-26T18:53:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.220+0000 7f5976bb5700 1 --2- 172.21.15.102:0/953722351 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f596006c7b0 0x7f596006ec60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.220+0000 7f5976bb5700 1 --2- 172.21.15.102:0/953722351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59700ff050 0x7f5970196d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:17.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.220+0000 7f5976bb5700 1 --2- 172.21.15.102:0/953722351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5970100250 0x7f5970197240 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:17.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.220+0000 7f5976bb5700 1 -- 172.21.15.102:0/953722351 >> 172.21.15.102:0/953722351 conn(0x7f59700fa600 msgr2=0x7f5970103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:17.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.220+0000 7f5976bb5700 1 -- 172.21.15.102:0/953722351 shutdown_connections 2024-03-26T18:53:17.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.220+0000 7f5976bb5700 1 -- 172.21.15.102:0/953722351 wait complete. 2024-03-26T18:53:17.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.352+0000 7f2697917700 1 -- 172.21.15.102:0/3912679510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26900ff050 msgr2=0x7f26900ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:17.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.352+0000 7f2697917700 1 --2- 172.21.15.102:0/3912679510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26900ff050 0x7f26900ff460 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f268c009910 tx=0x7f268c009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:17.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.352+0000 7f2697917700 1 -- 172.21.15.102:0/3912679510 shutdown_connections 2024-03-26T18:53:17.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.352+0000 7f2697917700 1 --2- 172.21.15.102:0/3912679510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2690100250 0x7f26901006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:17.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.352+0000 7f2697917700 1 --2- 172.21.15.102:0/3912679510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26900ff050 0x7f26900ff460 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:17.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.352+0000 7f2697917700 1 -- 172.21.15.102:0/3912679510 >> 172.21.15.102:0/3912679510 conn(0x7f26900fa600 msgr2=0x7f26900fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:17.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.352+0000 7f2697917700 1 -- 172.21.15.102:0/3912679510 shutdown_connections 2024-03-26T18:53:17.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.352+0000 7f2697917700 1 -- 172.21.15.102:0/3912679510 wait complete. 2024-03-26T18:53:17.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f2697917700 1 Processor -- start 2024-03-26T18:53:17.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f2697917700 1 -- start start 2024-03-26T18:53:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f2697917700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26900ff050 0x7f2690197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f2697917700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2690100250 0x7f26901980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f2697917700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2690198710 con 0x7f2690100250 2024-03-26T18:53:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f2697917700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2690198850 con 0x7f26900ff050 2024-03-26T18:53:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f26956b3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26900ff050 0x7f2690197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f26956b3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26900ff050 0x7f2690197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:51068/0 (socket says 172.21.15.102:51068) 2024-03-26T18:53:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f26956b3700 1 -- 172.21.15.102:0/3146868635 learned_addr learned my addr 172.21.15.102:0/3146868635 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f2694eb2700 1 --2- 172.21.15.102:0/3146868635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2690100250 0x7f26901980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f26956b3700 1 -- 172.21.15.102:0/3146868635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2690100250 msgr2=0x7f26901980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.353+0000 7f26956b3700 1 --2- 172.21.15.102:0/3146868635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2690100250 0x7f26901980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.354+0000 7f26956b3700 1 -- 172.21.15.102:0/3146868635 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c0095f0 con 0x7f26900ff050 2024-03-26T18:53:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.354+0000 7f26956b3700 1 --2- 172.21.15.102:0/3146868635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26900ff050 0x7f2690197bb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f268c00fa00 tx=0x7f268c00fa30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.354+0000 7f26867fc700 1 -- 172.21.15.102:0/3146868635 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268c01d070 con 0x7f26900ff050 2024-03-26T18:53:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.354+0000 7f26867fc700 1 -- 172.21.15.102:0/3146868635 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f268c00fe90 con 0x7f26900ff050 2024-03-26T18:53:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.354+0000 7f26867fc700 1 -- 172.21.15.102:0/3146868635 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268c017630 con 0x7f26900ff050 2024-03-26T18:53:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.354+0000 7f2697917700 1 -- 172.21.15.102:0/3146868635 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269019d250 con 0x7f26900ff050 2024-03-26T18:53:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.354+0000 7f2697917700 1 -- 172.21.15.102:0/3146868635 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269019d740 con 0x7f26900ff050 2024-03-26T18:53:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.355+0000 7f2697917700 1 -- 172.21.15.102:0/3146868635 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2690107ba0 con 0x7f26900ff050 2024-03-26T18:53:17.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.356+0000 7f26867fc700 1 -- 172.21.15.102:0/3146868635 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f268c003680 con 0x7f26900ff050 2024-03-26T18:53:17.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.356+0000 7f26867fc700 1 --2- 172.21.15.102:0/3146868635 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f267c06c680 0x7f267c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:17.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.357+0000 7f26867fc700 1 -- 172.21.15.102:0/3146868635 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f268c08cc60 con 0x7f26900ff050 2024-03-26T18:53:17.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.357+0000 7f2694eb2700 1 --2- 172.21.15.102:0/3146868635 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f267c06c680 0x7f267c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:17.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.357+0000 7f2694eb2700 1 --2- 172.21.15.102:0/3146868635 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f267c06c680 0x7f267c06eb30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2680009c60 tx=0x7f2680009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:17.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.361+0000 7f26867fc700 1 -- 172.21.15.102:0/3146868635 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f268c05aea0 con 0x7f26900ff050 2024-03-26T18:53:17.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:17 smithi190 ceph-mon[35043]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:17.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:17 smithi190 ceph-mon[35043]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:17.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:17 smithi190 ceph-mon[35043]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-26T18:53:17.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:17 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3784062689' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:53:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:17 smithi102 ceph-mon[29002]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:17 smithi102 ceph-mon[29002]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:17 smithi102 ceph-mon[29002]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-26T18:53:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:17 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3784062689' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:53:17.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.777+0000 7f2697917700 1 -- 172.21.15.102:0/3146868635 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2690107d90 con 0x7f26900ff050 2024-03-26T18:53:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.777+0000 7f26867fc700 1 -- 172.21.15.102:0/3146868635 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f268c003e60 con 0x7f26900ff050 2024-03-26T18:53:17.770 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T18:53:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.780+0000 7f2697917700 1 -- 172.21.15.102:0/3146868635 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f267c06c680 msgr2=0x7f267c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.780+0000 7f2697917700 1 --2- 172.21.15.102:0/3146868635 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f267c06c680 0x7f267c06eb30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2680009c60 tx=0x7f2680009260 comp rx=0 tx=0).stop 2024-03-26T18:53:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.780+0000 7f2697917700 1 -- 172.21.15.102:0/3146868635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26900ff050 msgr2=0x7f2690197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.780+0000 7f2697917700 1 --2- 172.21.15.102:0/3146868635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26900ff050 0x7f2690197bb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f268c00fa00 tx=0x7f268c00fa30 comp rx=0 tx=0).stop 2024-03-26T18:53:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.780+0000 7f2697917700 1 -- 172.21.15.102:0/3146868635 shutdown_connections 2024-03-26T18:53:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.780+0000 7f2697917700 1 --2- 172.21.15.102:0/3146868635 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f267c06c680 0x7f267c06eb30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.780+0000 7f2697917700 1 --2- 172.21.15.102:0/3146868635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26900ff050 0x7f2690197bb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.780+0000 7f2697917700 1 --2- 172.21.15.102:0/3146868635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2690100250 0x7f26901980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.780+0000 7f2697917700 1 -- 172.21.15.102:0/3146868635 >> 172.21.15.102:0/3146868635 conn(0x7f26900fa600 msgr2=0x7f2690068c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.780+0000 7f2697917700 1 -- 172.21.15.102:0/3146868635 shutdown_connections 2024-03-26T18:53:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:17.780+0000 7f2697917700 1 -- 172.21.15.102:0/3146868635 wait complete. 2024-03-26T18:53:18.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:18 smithi190 ceph-mon[35043]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:18.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:18 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3146868635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:53:18.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:18 smithi102 ceph-mon[29002]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:18.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:18 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3146868635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:53:19.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:19 smithi102 ceph-mon[29002]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:19.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:53:19.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:19 smithi190 ceph-mon[35043]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:19.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:53:21.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:21 smithi190 ceph-mon[35043]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:21.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:21 smithi102 ceph-mon[29002]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:23.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:23 smithi190 ceph-mon[35043]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:23.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:23 smithi102 ceph-mon[29002]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:25 smithi190 ceph-mon[35043]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:25.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:25 smithi102 ceph-mon[29002]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:27.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:27 smithi190 ceph-mon[35043]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:27.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:27 smithi102 ceph-mon[29002]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:29.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:29 smithi190 ceph-mon[35043]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:29.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:29 smithi102 ceph-mon[29002]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:31.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:31 smithi190 ceph-mon[35043]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:31.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:31 smithi102 ceph-mon[29002]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T18:53:33.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:33 smithi190 ceph-mon[35043]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-26T18:53:33.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:33 smithi102 ceph-mon[29002]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-26T18:53:34.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:34 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:53:34.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:34 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:53:35.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:35 smithi190 ceph-mon[35043]: pgmap v215: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-26T18:53:35.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:35 smithi102 ceph-mon[29002]: pgmap v215: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-26T18:53:37.248 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:37 smithi102 ceph-mon[29002]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-26T18:53:37.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:37 smithi190 ceph-mon[35043]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-26T18:53:39.250 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:39 smithi190 ceph-mon[35043]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-26T18:53:39.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:39 smithi102 ceph-mon[29002]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-26T18:53:41.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:41 smithi190 ceph-mon[35043]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-26T18:53:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:41 smithi102 ceph-mon[29002]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-26T18:53:43.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:43 smithi190 ceph-mon[35043]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-03-26T18:53:43.540 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:43 smithi102 ceph-mon[29002]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-03-26T18:53:45.226 INFO:tasks.workunit.client.1.smithi190.stderr:Note: switching to '72e2437271dcafb9826f2f49bb1001426fa6de81'. 2024-03-26T18:53:45.226 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-03-26T18:53:45.226 INFO:tasks.workunit.client.1.smithi190.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-26T18:53:45.226 INFO:tasks.workunit.client.1.smithi190.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-26T18:53:45.226 INFO:tasks.workunit.client.1.smithi190.stderr:state without impacting any branches by switching back to a branch. 2024-03-26T18:53:45.226 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-03-26T18:53:45.226 INFO:tasks.workunit.client.1.smithi190.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-26T18:53:45.227 INFO:tasks.workunit.client.1.smithi190.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-26T18:53:45.227 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-03-26T18:53:45.227 INFO:tasks.workunit.client.1.smithi190.stderr: git switch -c 2024-03-26T18:53:45.227 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-03-26T18:53:45.227 INFO:tasks.workunit.client.1.smithi190.stderr:Or undo this operation with: 2024-03-26T18:53:45.227 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-03-26T18:53:45.227 INFO:tasks.workunit.client.1.smithi190.stderr: git switch - 2024-03-26T18:53:45.227 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-03-26T18:53:45.227 INFO:tasks.workunit.client.1.smithi190.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-26T18:53:45.227 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-03-26T18:53:45.227 INFO:tasks.workunit.client.1.smithi190.stderr:HEAD is now at 72e2437271d qa/cephfs/overrides: ignore expected PG_DEGRADED health warning 2024-03-26T18:53:45.230 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-26T18:53:45.290 INFO:tasks.workunit.client.1.smithi190.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-26T18:53:45.293 INFO:tasks.workunit.client.1.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-26T18:53:45.294 INFO:tasks.workunit.client.1.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-26T18:53:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:45 smithi190 ceph-mon[35043]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-26T18:53:45.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:45 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:53:45.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:45 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:53:45.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:45 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:53:45.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:45 smithi102 ceph-mon[29002]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-26T18:53:45.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:45 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:53:45.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:45 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:53:45.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:45 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:53:46.335 INFO:tasks.workunit.client.1.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-26T18:53:46.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:46 smithi102 ceph-mon[29002]: Upgrade: Target is version 19.0.0-2311-g36051606 (unknown) 2024-03-26T18:53:46.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:46 smithi102 ceph-mon[29002]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9'] 2024-03-26T18:53:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:46 smithi102 ceph-mon[29002]: Upgrade: Need to upgrade myself (mgr.smithi102.rsfpnv) 2024-03-26T18:53:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:46 smithi102 ceph-mon[29002]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 on smithi190 2024-03-26T18:53:46.402 INFO:tasks.workunit.client.1.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-26T18:53:46.448 INFO:tasks.workunit.client.1.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-26T18:53:46.450 INFO:tasks.workunit.client.1.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-26T18:53:46.450 INFO:tasks.workunit.client.1.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-26T18:53:46.491 INFO:tasks.workunit.client.1.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-26T18:53:46.494 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T18:53:46.494 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-26T18:53:46.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:46 smithi190 ceph-mon[35043]: Upgrade: Target is version 19.0.0-2311-g36051606 (unknown) 2024-03-26T18:53:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:46 smithi190 ceph-mon[35043]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9'] 2024-03-26T18:53:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:46 smithi190 ceph-mon[35043]: Upgrade: Need to upgrade myself (mgr.smithi102.rsfpnv) 2024-03-26T18:53:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:46 smithi190 ceph-mon[35043]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 on smithi190 2024-03-26T18:53:46.545 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-26T18:53:46.548 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-26T18:53:46.548 DEBUG:teuthology.orchestra.run.smithi190:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=72e2437271dcafb9826f2f49bb1001426fa6de81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-26T18:53:46.607 INFO:tasks.workunit.client.1.smithi190.stderr:+ mkdir -p fsstress 2024-03-26T18:53:46.609 INFO:tasks.workunit.client.1.smithi190.stderr:+ pushd fsstress 2024-03-26T18:53:46.609 INFO:tasks.workunit.client.1.smithi190.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-26T18:53:46.609 INFO:tasks.workunit.client.1.smithi190.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-26T18:53:47.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:47 smithi102 ceph-mon[29002]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-26T18:53:47.449 INFO:tasks.workunit.client.1.smithi190.stderr:+ tar xzf ltp-full.tgz 2024-03-26T18:53:47.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:47 smithi190 ceph-mon[35043]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-26T18:53:47.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.902+0000 7f7a44397700 1 -- 172.21.15.102:0/3752315457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3c100be0 msgr2=0x7f7a3c102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:47.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.902+0000 7f7a44397700 1 --2- 172.21.15.102:0/3752315457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3c100be0 0x7f7a3c102fc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009910 tx=0x7f7a38009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:47.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.902+0000 7f7a44397700 1 -- 172.21.15.102:0/3752315457 shutdown_connections 2024-03-26T18:53:47.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.902+0000 7f7a44397700 1 --2- 172.21.15.102:0/3752315457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c103500 0x7f7a3c1058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:47.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.902+0000 7f7a44397700 1 --2- 172.21.15.102:0/3752315457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3c100be0 0x7f7a3c102fc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:47.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.902+0000 7f7a44397700 1 -- 172.21.15.102:0/3752315457 >> 172.21.15.102:0/3752315457 conn(0x7f7a3c0fa600 msgr2=0x7f7a3c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:47.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.903+0000 7f7a44397700 1 -- 172.21.15.102:0/3752315457 shutdown_connections 2024-03-26T18:53:47.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.903+0000 7f7a44397700 1 -- 172.21.15.102:0/3752315457 wait complete. 2024-03-26T18:53:47.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.903+0000 7f7a44397700 1 Processor -- start 2024-03-26T18:53:47.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.903+0000 7f7a44397700 1 -- start start 2024-03-26T18:53:47.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.903+0000 7f7a44397700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3c100be0 0x7f7a3c1004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:47.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.903+0000 7f7a44397700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c103500 0x7f7a3c0feb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:47.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.903+0000 7f7a44397700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c0ff0d0 con 0x7f7a3c100be0 2024-03-26T18:53:47.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.903+0000 7f7a44397700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c0ff210 con 0x7f7a3c103500 2024-03-26T18:53:47.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.903+0000 7f7a42133700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3c100be0 0x7f7a3c1004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a42133700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3c100be0 0x7f7a3c1004f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41286/0 (socket says 172.21.15.102:41286) 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a42133700 1 -- 172.21.15.102:0/1185904473 learned_addr learned my addr 172.21.15.102:0/1185904473 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a41932700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c103500 0x7f7a3c0feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a42133700 1 -- 172.21.15.102:0/1185904473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c103500 msgr2=0x7f7a3c0feb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a42133700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c103500 0x7f7a3c0feb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a42133700 1 -- 172.21.15.102:0/1185904473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c009520 con 0x7f7a3c100be0 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a41932700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c103500 0x7f7a3c0feb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a42133700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3c100be0 0x7f7a3c1004f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7a380092e0 tx=0x7f7a3800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a337fe700 1 -- 172.21.15.102:0/1185904473 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a3801d070 con 0x7f7a3c100be0 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a44397700 1 -- 172.21.15.102:0/1185904473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a380095f0 con 0x7f7a3c100be0 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a44397700 1 -- 172.21.15.102:0/1185904473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c0ff7f0 con 0x7f7a3c100be0 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a337fe700 1 -- 172.21.15.102:0/1185904473 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7a38005dc0 con 0x7f7a3c100be0 2024-03-26T18:53:47.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.904+0000 7f7a337fe700 1 -- 172.21.15.102:0/1185904473 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a38017610 con 0x7f7a3c100be0 2024-03-26T18:53:47.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.905+0000 7f7a44397700 1 -- 172.21.15.102:0/1185904473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3c18fb70 con 0x7f7a3c100be0 2024-03-26T18:53:47.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.906+0000 7f7a337fe700 1 -- 172.21.15.102:0/1185904473 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f7a3800fba0 con 0x7f7a3c100be0 2024-03-26T18:53:47.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.907+0000 7f7a337fe700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7a2806c680 0x7f7a2806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:47.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.907+0000 7f7a337fe700 1 -- 172.21.15.102:0/1185904473 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7a3808cd80 con 0x7f7a3c100be0 2024-03-26T18:53:47.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.907+0000 7f7a41932700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7a2806c680 0x7f7a2806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:47.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.907+0000 7f7a41932700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7a2806c680 0x7f7a2806eb30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c009e40 tx=0x7f7a2c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:47.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:47.912+0000 7f7a337fe700 1 -- 172.21.15.102:0/1185904473 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7a3805afc0 con 0x7f7a3c100be0 2024-03-26T18:53:48.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.179+0000 7f7a44397700 1 -- 172.21.15.102:0/1185904473 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a3c061190 con 0x7f7a2806c680 2024-03-26T18:53:48.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.181+0000 7f7a337fe700 1 -- 172.21.15.102:0/1185904473 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f7a3c061190 con 0x7f7a2806c680 2024-03-26T18:53:48.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.183+0000 7f7a44397700 1 -- 172.21.15.102:0/1185904473 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7a2806c680 msgr2=0x7f7a2806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.183+0000 7f7a44397700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7a2806c680 0x7f7a2806eb30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c009e40 tx=0x7f7a2c009260 comp rx=0 tx=0).stop 2024-03-26T18:53:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.183+0000 7f7a44397700 1 -- 172.21.15.102:0/1185904473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3c100be0 msgr2=0x7f7a3c1004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.183+0000 7f7a44397700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3c100be0 0x7f7a3c1004f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7a380092e0 tx=0x7f7a3800f740 comp rx=0 tx=0).stop 2024-03-26T18:53:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.183+0000 7f7a44397700 1 -- 172.21.15.102:0/1185904473 shutdown_connections 2024-03-26T18:53:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.183+0000 7f7a44397700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7a2806c680 0x7f7a2806eb30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.183+0000 7f7a44397700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3c100be0 0x7f7a3c1004f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.183+0000 7f7a44397700 1 --2- 172.21.15.102:0/1185904473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c103500 0x7f7a3c0feb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.183+0000 7f7a44397700 1 -- 172.21.15.102:0/1185904473 >> 172.21.15.102:0/1185904473 conn(0x7f7a3c0fa600 msgr2=0x7f7a3c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.183+0000 7f7a44397700 1 -- 172.21.15.102:0/1185904473 shutdown_connections 2024-03-26T18:53:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.183+0000 7f7a44397700 1 -- 172.21.15.102:0/1185904473 wait complete. 2024-03-26T18:53:48.188 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:53:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.312+0000 7efc1c467700 1 -- 172.21.15.102:0/3393606394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc14100be0 msgr2=0x7efc14102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.312+0000 7efc1c467700 1 --2- 172.21.15.102:0/3393606394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc14100be0 0x7efc14102fc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efc10009910 tx=0x7efc10009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.312+0000 7efc1c467700 1 -- 172.21.15.102:0/3393606394 shutdown_connections 2024-03-26T18:53:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.312+0000 7efc1c467700 1 --2- 172.21.15.102:0/3393606394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc14103500 0x7efc141058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.312+0000 7efc1c467700 1 --2- 172.21.15.102:0/3393606394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc14100be0 0x7efc14102fc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.312+0000 7efc1c467700 1 -- 172.21.15.102:0/3393606394 >> 172.21.15.102:0/3393606394 conn(0x7efc140fa5e0 msgr2=0x7efc140fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.312+0000 7efc1c467700 1 -- 172.21.15.102:0/3393606394 shutdown_connections 2024-03-26T18:53:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.312+0000 7efc1c467700 1 -- 172.21.15.102:0/3393606394 wait complete. 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.313+0000 7efc1c467700 1 Processor -- start 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.313+0000 7efc1c467700 1 -- start start 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.313+0000 7efc1c467700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc14100be0 0x7efc14195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.313+0000 7efc1c467700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc14103500 0x7efc14195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.313+0000 7efc1c467700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc14196480 con 0x7efc14103500 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.313+0000 7efc1c467700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc141965c0 con 0x7efc14100be0 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.313+0000 7efc19a02700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc14103500 0x7efc14195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.313+0000 7efc1a203700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc14100be0 0x7efc14195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.313+0000 7efc1a203700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc14100be0 0x7efc14195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:33858/0 (socket says 172.21.15.102:33858) 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.313+0000 7efc19a02700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc14103500 0x7efc14195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41308/0 (socket says 172.21.15.102:41308) 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.313+0000 7efc1a203700 1 -- 172.21.15.102:0/971215856 learned_addr learned my addr 172.21.15.102:0/971215856 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.314+0000 7efc19a02700 1 -- 172.21.15.102:0/971215856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc14100be0 msgr2=0x7efc14195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.314+0000 7efc19a02700 1 --2- 172.21.15.102:0/971215856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc14100be0 0x7efc14195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.314+0000 7efc19a02700 1 -- 172.21.15.102:0/971215856 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc100095f0 con 0x7efc14103500 2024-03-26T18:53:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.314+0000 7efc19a02700 1 --2- 172.21.15.102:0/971215856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc14103500 0x7efc14195e60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efc0400e970 tx=0x7efc0400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.314+0000 7efc0b7fe700 1 -- 172.21.15.102:0/971215856 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc0400c9d0 con 0x7efc14103500 2024-03-26T18:53:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.314+0000 7efc1c467700 1 -- 172.21.15.102:0/971215856 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1419b070 con 0x7efc14103500 2024-03-26T18:53:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.314+0000 7efc0b7fe700 1 -- 172.21.15.102:0/971215856 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efc04010430 con 0x7efc14103500 2024-03-26T18:53:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.314+0000 7efc0b7fe700 1 -- 172.21.15.102:0/971215856 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc04018650 con 0x7efc14103500 2024-03-26T18:53:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.314+0000 7efc1c467700 1 -- 172.21.15.102:0/971215856 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1419b5c0 con 0x7efc14103500 2024-03-26T18:53:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.315+0000 7efc1c467700 1 -- 172.21.15.102:0/971215856 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1418fb70 con 0x7efc14103500 2024-03-26T18:53:48.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.317+0000 7efc0b7fe700 1 -- 172.21.15.102:0/971215856 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7efc040105a0 con 0x7efc14103500 2024-03-26T18:53:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.317+0000 7efc0b7fe700 1 --2- 172.21.15.102:0/971215856 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7efc0006c680 0x7efc0006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.317+0000 7efc0b7fe700 1 -- 172.21.15.102:0/971215856 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7efc04014070 con 0x7efc14103500 2024-03-26T18:53:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.317+0000 7efc1a203700 1 --2- 172.21.15.102:0/971215856 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7efc0006c680 0x7efc0006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.318+0000 7efc1a203700 1 --2- 172.21.15.102:0/971215856 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7efc0006c680 0x7efc0006eb30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efc10009fd0 tx=0x7efc10005a90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.323+0000 7efc0b7fe700 1 -- 172.21.15.102:0/971215856 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7efc0405a2b0 con 0x7efc14103500 2024-03-26T18:53:48.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.561+0000 7efc1c467700 1 -- 172.21.15.102:0/971215856 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efc14061190 con 0x7efc0006c680 2024-03-26T18:53:48.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.562+0000 7efc0b7fe700 1 -- 172.21.15.102:0/971215856 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7efc14061190 con 0x7efc0006c680 2024-03-26T18:53:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.564+0000 7efc1c467700 1 -- 172.21.15.102:0/971215856 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7efc0006c680 msgr2=0x7efc0006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.564+0000 7efc1c467700 1 --2- 172.21.15.102:0/971215856 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7efc0006c680 0x7efc0006eb30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efc10009fd0 tx=0x7efc10005a90 comp rx=0 tx=0).stop 2024-03-26T18:53:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.564+0000 7efc1c467700 1 -- 172.21.15.102:0/971215856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc14103500 msgr2=0x7efc14195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.564+0000 7efc1c467700 1 --2- 172.21.15.102:0/971215856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc14103500 0x7efc14195e60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efc0400e970 tx=0x7efc0400ec80 comp rx=0 tx=0).stop 2024-03-26T18:53:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.564+0000 7efc1c467700 1 -- 172.21.15.102:0/971215856 shutdown_connections 2024-03-26T18:53:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.564+0000 7efc1c467700 1 --2- 172.21.15.102:0/971215856 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7efc0006c680 0x7efc0006eb30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.564+0000 7efc1c467700 1 --2- 172.21.15.102:0/971215856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc14100be0 0x7efc14195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.564+0000 7efc1c467700 1 --2- 172.21.15.102:0/971215856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc14103500 0x7efc14195e60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.564+0000 7efc1c467700 1 -- 172.21.15.102:0/971215856 >> 172.21.15.102:0/971215856 conn(0x7efc140fa5e0 msgr2=0x7efc140fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:48.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.564+0000 7efc1c467700 1 -- 172.21.15.102:0/971215856 shutdown_connections 2024-03-26T18:53:48.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.564+0000 7efc1c467700 1 -- 172.21.15.102:0/971215856 wait complete. 2024-03-26T18:53:48.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.681+0000 7fd1597db700 1 -- 172.21.15.102:0/3893636448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd154100be0 msgr2=0x7fd154102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:48.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.681+0000 7fd1597db700 1 --2- 172.21.15.102:0/3893636448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd154100be0 0x7fd154102fc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd144009910 tx=0x7fd144009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:48.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.681+0000 7fd1597db700 1 -- 172.21.15.102:0/3893636448 shutdown_connections 2024-03-26T18:53:48.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.681+0000 7fd1597db700 1 --2- 172.21.15.102:0/3893636448 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154103500 0x7fd1541058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.681+0000 7fd1597db700 1 --2- 172.21.15.102:0/3893636448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd154100be0 0x7fd154102fc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.681+0000 7fd1597db700 1 -- 172.21.15.102:0/3893636448 >> 172.21.15.102:0/3893636448 conn(0x7fd1540fa5e0 msgr2=0x7fd1540fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:48.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.681+0000 7fd1597db700 1 -- 172.21.15.102:0/3893636448 shutdown_connections 2024-03-26T18:53:48.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.681+0000 7fd1597db700 1 -- 172.21.15.102:0/3893636448 wait complete. 2024-03-26T18:53:48.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1597db700 1 Processor -- start 2024-03-26T18:53:48.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1597db700 1 -- start start 2024-03-26T18:53:48.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1597db700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154100be0 0x7fd154100540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:48.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1597db700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd154103500 0x7fd1540feb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:48.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1597db700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1540ff0d0 con 0x7fd154103500 2024-03-26T18:53:48.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1597db700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1540ff210 con 0x7fd154100be0 2024-03-26T18:53:48.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1527fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd154103500 0x7fd1540feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:48.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1527fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd154103500 0x7fd1540feb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56036/0 (socket says 172.21.15.102:56036) 2024-03-26T18:53:48.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1527fc700 1 -- 172.21.15.102:0/3150970342 learned_addr learned my addr 172.21.15.102:0/3150970342 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:48.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd152ffd700 1 --2- 172.21.15.102:0/3150970342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154100be0 0x7fd154100540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:48.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1527fc700 1 -- 172.21.15.102:0/3150970342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154100be0 msgr2=0x7fd154100540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:48.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1527fc700 1 --2- 172.21.15.102:0/3150970342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154100be0 0x7fd154100540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.682+0000 7fd1527fc700 1 -- 172.21.15.102:0/3150970342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1440095f0 con 0x7fd154103500 2024-03-26T18:53:48.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.683+0000 7fd1527fc700 1 --2- 172.21.15.102:0/3150970342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd154103500 0x7fd1540feb90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd14800e920 tx=0x7fd14800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:48.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.683+0000 7fd13bfff700 1 -- 172.21.15.102:0/3150970342 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd14800c9d0 con 0x7fd154103500 2024-03-26T18:53:48.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.683+0000 7fd13bfff700 1 -- 172.21.15.102:0/3150970342 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd148010430 con 0x7fd154103500 2024-03-26T18:53:48.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.683+0000 7fd13bfff700 1 -- 172.21.15.102:0/3150970342 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd148018650 con 0x7fd154103500 2024-03-26T18:53:48.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.683+0000 7fd1597db700 1 -- 172.21.15.102:0/3150970342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1540ff4f0 con 0x7fd154103500 2024-03-26T18:53:48.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.683+0000 7fd1597db700 1 -- 172.21.15.102:0/3150970342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1540ff9f0 con 0x7fd154103500 2024-03-26T18:53:48.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.684+0000 7fd1597db700 1 -- 172.21.15.102:0/3150970342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15418fb70 con 0x7fd154103500 2024-03-26T18:53:48.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.684+0000 7fd13bfff700 1 -- 172.21.15.102:0/3150970342 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fd1480105a0 con 0x7fd154103500 2024-03-26T18:53:48.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.685+0000 7fd13bfff700 1 --2- 172.21.15.102:0/3150970342 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd13c06c680 0x7fd13c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:48.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.685+0000 7fd13bfff700 1 -- 172.21.15.102:0/3150970342 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd148014070 con 0x7fd154103500 2024-03-26T18:53:48.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.685+0000 7fd152ffd700 1 --2- 172.21.15.102:0/3150970342 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd13c06c680 0x7fd13c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:48.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.685+0000 7fd152ffd700 1 --2- 172.21.15.102:0/3150970342 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd13c06c680 0x7fd13c06eb30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd144009fd0 tx=0x7fd144005940 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.692+0000 7fd13bfff700 1 -- 172.21.15.102:0/3150970342 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd148051720 con 0x7fd154103500 2024-03-26T18:53:48.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.929+0000 7fd1597db700 1 -- 172.21.15.102:0/3150970342 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd154061190 con 0x7fd13c06c680 2024-03-26T18:53:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.956+0000 7fd13bfff700 1 -- 172.21.15.102:0/3150970342 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7fd154061190 con 0x7fd13c06c680 2024-03-26T18:53:48.949 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:53:48.949 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (5m) 59s ago 6m 25.5M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:53:48.949 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (7m) 59s ago 7m 7457k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:53:48.949 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (6m) 64s ago 6m 7356k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:53:48.949 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (7m) 59s ago 7m 6665k - 18.2.0 dc2bc1663786 bfac00217c68 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (5m) 64s ago 5m 6656k - 18.2.0 dc2bc1663786 7379f1f6b746 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (5m) 59s ago 6m 85.5M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (79s) 59s ago 78s 15.9M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (85s) 59s ago 84s 11.5M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (76s) 64s ago 75s 12.1M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (82s) 64s ago 82s 14.9M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:9283,8765,8443 running (8m) 59s ago 8m 487M - 18.2.0 dc2bc1663786 61d2bd7101b6 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (5m) 64s ago 5m 440M - 18.2.0 dc2bc1663786 57e44bfc8ede 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (8m) 59s ago 8m 45.1M 2048M 18.2.0 dc2bc1663786 b5abd46c3973 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (5m) 64s ago 5m 35.3M 2048M 18.2.0 dc2bc1663786 6bd913d483f0 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (6m) 59s ago 6m 15.9M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (5m) 64s ago 5m 15.3M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:53:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (5m) 59s ago 5m 47.9M 3946M 18.2.0 dc2bc1663786 3e2065fb5fbc 2024-03-26T18:53:48.951 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (4m) 59s ago 4m 48.5M 3946M 18.2.0 dc2bc1663786 0bf773515cc1 2024-03-26T18:53:48.951 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (4m) 59s ago 4m 46.3M 3946M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:53:48.951 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (3m) 64s ago 3m 49.3M 4970M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:53:48.951 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (3m) 64s ago 3m 44.9M 4970M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:53:48.951 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (2m) 64s ago 2m 44.1M 4970M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:53:48.951 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (5m) 59s ago 6m 61.7M - 2.43.0 a07b618ecd1d d0dc2ca7f41f 2024-03-26T18:53:48.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.959+0000 7fd1597db700 1 -- 172.21.15.102:0/3150970342 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd13c06c680 msgr2=0x7fd13c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:48.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.959+0000 7fd1597db700 1 --2- 172.21.15.102:0/3150970342 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd13c06c680 0x7fd13c06eb30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd144009fd0 tx=0x7fd144005940 comp rx=0 tx=0).stop 2024-03-26T18:53:48.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.959+0000 7fd1597db700 1 -- 172.21.15.102:0/3150970342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd154103500 msgr2=0x7fd1540feb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:48.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.959+0000 7fd1597db700 1 --2- 172.21.15.102:0/3150970342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd154103500 0x7fd1540feb90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd14800e920 tx=0x7fd14800ec30 comp rx=0 tx=0).stop 2024-03-26T18:53:48.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.959+0000 7fd1597db700 1 -- 172.21.15.102:0/3150970342 shutdown_connections 2024-03-26T18:53:48.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.959+0000 7fd1597db700 1 --2- 172.21.15.102:0/3150970342 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd13c06c680 0x7fd13c06eb30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.959+0000 7fd1597db700 1 --2- 172.21.15.102:0/3150970342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154100be0 0x7fd154100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.959+0000 7fd1597db700 1 --2- 172.21.15.102:0/3150970342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd154103500 0x7fd1540feb90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:48.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.959+0000 7fd1597db700 1 -- 172.21.15.102:0/3150970342 >> 172.21.15.102:0/3150970342 conn(0x7fd1540fa5e0 msgr2=0x7fd1540fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:48.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.959+0000 7fd1597db700 1 -- 172.21.15.102:0/3150970342 shutdown_connections 2024-03-26T18:53:48.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:48.959+0000 7fd1597db700 1 -- 172.21.15.102:0/3150970342 wait complete. 2024-03-26T18:53:49.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.066+0000 7fcbf9f65700 1 -- 172.21.15.102:0/4133611120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbf4100c40 msgr2=0x7fcbf4103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:49.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.066+0000 7fcbf9f65700 1 --2- 172.21.15.102:0/4133611120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbf4100c40 0x7fcbf4103020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009910 tx=0x7fcbe4009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:49.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.066+0000 7fcbf9f65700 1 -- 172.21.15.102:0/4133611120 shutdown_connections 2024-03-26T18:53:49.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.066+0000 7fcbf9f65700 1 --2- 172.21.15.102:0/4133611120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf4103560 0x7fcbf4105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.066+0000 7fcbf9f65700 1 --2- 172.21.15.102:0/4133611120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbf4100c40 0x7fcbf4103020 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.066+0000 7fcbf9f65700 1 -- 172.21.15.102:0/4133611120 >> 172.21.15.102:0/4133611120 conn(0x7fcbf40fa5e0 msgr2=0x7fcbf40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:49.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.066+0000 7fcbf9f65700 1 -- 172.21.15.102:0/4133611120 shutdown_connections 2024-03-26T18:53:49.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.066+0000 7fcbf9f65700 1 -- 172.21.15.102:0/4133611120 wait complete. 2024-03-26T18:53:49.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.067+0000 7fcbf9f65700 1 Processor -- start 2024-03-26T18:53:49.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.067+0000 7fcbf9f65700 1 -- start start 2024-03-26T18:53:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.067+0000 7fcbf9f65700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbf4100c40 0x7fcbf4195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.067+0000 7fcbf9f65700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf4103560 0x7fcbf4195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.067+0000 7fcbf9f65700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf4196480 con 0x7fcbf4100c40 2024-03-26T18:53:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.067+0000 7fcbf9f65700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf41965c0 con 0x7fcbf4103560 2024-03-26T18:53:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbf4100c40 0x7fcbf4195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf2ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf4103560 0x7fcbf4195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbf4100c40 0x7fcbf4195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56062/0 (socket says 172.21.15.102:56062) 2024-03-26T18:53:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf37fe700 1 -- 172.21.15.102:0/167896939 learned_addr learned my addr 172.21.15.102:0/167896939 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf37fe700 1 -- 172.21.15.102:0/167896939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf4103560 msgr2=0x7fcbf4195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf37fe700 1 --2- 172.21.15.102:0/167896939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf4103560 0x7fcbf4195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf37fe700 1 -- 172.21.15.102:0/167896939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe40095f0 con 0x7fcbf4100c40 2024-03-26T18:53:49.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf37fe700 1 --2- 172.21.15.102:0/167896939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbf4100c40 0x7fcbf4195920 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4000c00 tx=0x7fcbe400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:49.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf0ff9700 1 -- 172.21.15.102:0/167896939 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbe401d070 con 0x7fcbf4100c40 2024-03-26T18:53:49.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf0ff9700 1 -- 172.21.15.102:0/167896939 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcbe4005dc0 con 0x7fcbf4100c40 2024-03-26T18:53:49.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf9f65700 1 -- 172.21.15.102:0/167896939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf419b010 con 0x7fcbf4100c40 2024-03-26T18:53:49.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.068+0000 7fcbf0ff9700 1 -- 172.21.15.102:0/167896939 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbe40226a0 con 0x7fcbf4100c40 2024-03-26T18:53:49.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.069+0000 7fcbf9f65700 1 -- 172.21.15.102:0/167896939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf419b500 con 0x7fcbf4100c40 2024-03-26T18:53:49.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.070+0000 7fcbf9f65700 1 -- 172.21.15.102:0/167896939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf418fb70 con 0x7fcbf4100c40 2024-03-26T18:53:49.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.070+0000 7fcbf0ff9700 1 -- 172.21.15.102:0/167896939 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fcbe40038f0 con 0x7fcbf4100c40 2024-03-26T18:53:49.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.071+0000 7fcbf0ff9700 1 --2- 172.21.15.102:0/167896939 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcbdc06c680 0x7fcbdc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:49.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.071+0000 7fcbf0ff9700 1 -- 172.21.15.102:0/167896939 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcbe408ce10 con 0x7fcbf4100c40 2024-03-26T18:53:49.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.071+0000 7fcbf2ffd700 1 --2- 172.21.15.102:0/167896939 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcbdc06c680 0x7fcbdc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:49.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.071+0000 7fcbf2ffd700 1 --2- 172.21.15.102:0/167896939 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcbdc06c680 0x7fcbdc06eb30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009c60 tx=0x7fcbe8009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:49.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.079+0000 7fcbf0ff9700 1 -- 172.21.15.102:0/167896939 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcbe405af20 con 0x7fcbf4100c40 2024-03-26T18:53:49.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.383+0000 7fcbf9f65700 1 -- 172.21.15.102:0/167896939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcbf402cc30 con 0x7fcbf4100c40 2024-03-26T18:53:49.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.383+0000 7fcbf0ff9700 1 -- 172.21.15.102:0/167896939 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fcbe4003680 con 0x7fcbf4100c40 2024-03-26T18:53:49.376 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:53:49.376 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:53:49.376 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-03-26T18:53:49.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:53:49.376 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:53:49.376 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-03-26T18:53:49.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:53:49.377 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:53:49.377 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-03-26T18:53:49.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:53:49.377 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:53:49.377 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:53:49.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:53:49.377 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:53:49.377 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 14 2024-03-26T18:53:49.377 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:53:49.377 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:53:49.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.386+0000 7fcbf9f65700 1 -- 172.21.15.102:0/167896939 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcbdc06c680 msgr2=0x7fcbdc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:49.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.386+0000 7fcbf9f65700 1 --2- 172.21.15.102:0/167896939 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcbdc06c680 0x7fcbdc06eb30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009c60 tx=0x7fcbe8009260 comp rx=0 tx=0).stop 2024-03-26T18:53:49.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.386+0000 7fcbf9f65700 1 -- 172.21.15.102:0/167896939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbf4100c40 msgr2=0x7fcbf4195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:49.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.386+0000 7fcbf9f65700 1 --2- 172.21.15.102:0/167896939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbf4100c40 0x7fcbf4195920 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4000c00 tx=0x7fcbe400f690 comp rx=0 tx=0).stop 2024-03-26T18:53:49.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.386+0000 7fcbf9f65700 1 -- 172.21.15.102:0/167896939 shutdown_connections 2024-03-26T18:53:49.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.386+0000 7fcbf9f65700 1 --2- 172.21.15.102:0/167896939 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fcbdc06c680 0x7fcbdc06eb30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.386+0000 7fcbf9f65700 1 --2- 172.21.15.102:0/167896939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbf4100c40 0x7fcbf4195920 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.386+0000 7fcbf9f65700 1 --2- 172.21.15.102:0/167896939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf4103560 0x7fcbf4195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.386+0000 7fcbf9f65700 1 -- 172.21.15.102:0/167896939 >> 172.21.15.102:0/167896939 conn(0x7fcbf40fa5e0 msgr2=0x7fcbf40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:49.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.386+0000 7fcbf9f65700 1 -- 172.21.15.102:0/167896939 shutdown_connections 2024-03-26T18:53:49.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.386+0000 7fcbf9f65700 1 -- 172.21.15.102:0/167896939 wait complete. 2024-03-26T18:53:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.496+0000 7f66c8d35700 1 -- 172.21.15.102:0/1480209795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c40ff050 msgr2=0x7f66c40ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.496+0000 7f66c8d35700 1 --2- 172.21.15.102:0/1480209795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c40ff050 0x7f66c40ff460 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009910 tx=0x7f66b4009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.496+0000 7f66c8d35700 1 -- 172.21.15.102:0/1480209795 shutdown_connections 2024-03-26T18:53:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.496+0000 7f66c8d35700 1 --2- 172.21.15.102:0/1480209795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66c4100250 0x7f66c41006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.496+0000 7f66c8d35700 1 --2- 172.21.15.102:0/1480209795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c40ff050 0x7f66c40ff460 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.496+0000 7f66c8d35700 1 -- 172.21.15.102:0/1480209795 >> 172.21.15.102:0/1480209795 conn(0x7f66c40fa5e0 msgr2=0x7f66c40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.496+0000 7f66c8d35700 1 -- 172.21.15.102:0/1480209795 shutdown_connections 2024-03-26T18:53:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.496+0000 7f66c8d35700 1 -- 172.21.15.102:0/1480209795 wait complete. 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c8d35700 1 Processor -- start 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c8d35700 1 -- start start 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c8d35700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c40ff050 0x7f66c41959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c8d35700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66c4100250 0x7f66c4195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c8d35700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c4196500 con 0x7f66c40ff050 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c8d35700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c4196640 con 0x7f66c4100250 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c40ff050 0x7f66c41959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c40ff050 0x7f66c41959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56078/0 (socket says 172.21.15.102:56078) 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c259c700 1 -- 172.21.15.102:0/669185465 learned_addr learned my addr 172.21.15.102:0/669185465 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c1d9b700 1 --2- 172.21.15.102:0/669185465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66c4100250 0x7f66c4195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c259c700 1 -- 172.21.15.102:0/669185465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66c4100250 msgr2=0x7f66c4195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c259c700 1 --2- 172.21.15.102:0/669185465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66c4100250 0x7f66c4195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.497+0000 7f66c259c700 1 -- 172.21.15.102:0/669185465 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b40095f0 con 0x7f66c40ff050 2024-03-26T18:53:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.498+0000 7f66c259c700 1 --2- 172.21.15.102:0/669185465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c40ff050 0x7f66c41959a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f66b400f6c0 tx=0x7f66b400f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.498+0000 7f66b37fe700 1 -- 172.21.15.102:0/669185465 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66b401d070 con 0x7f66c40ff050 2024-03-26T18:53:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.498+0000 7f66b37fe700 1 -- 172.21.15.102:0/669185465 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f66b4005dc0 con 0x7f66c40ff050 2024-03-26T18:53:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.498+0000 7f66c8d35700 1 -- 172.21.15.102:0/669185465 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c419b090 con 0x7f66c40ff050 2024-03-26T18:53:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.498+0000 7f66b37fe700 1 -- 172.21.15.102:0/669185465 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66b4017400 con 0x7f66c40ff050 2024-03-26T18:53:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.498+0000 7f66c8d35700 1 -- 172.21.15.102:0/669185465 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c419b580 con 0x7f66c40ff050 2024-03-26T18:53:49.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.499+0000 7f66c8d35700 1 -- 172.21.15.102:0/669185465 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c4107ba0 con 0x7f66c40ff050 2024-03-26T18:53:49.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.500+0000 7f66b37fe700 1 -- 172.21.15.102:0/669185465 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f66b40038a0 con 0x7f66c40ff050 2024-03-26T18:53:49.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.500+0000 7f66b37fe700 1 --2- 172.21.15.102:0/669185465 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f66ac06c680 0x7f66ac06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:49.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.500+0000 7f66b37fe700 1 -- 172.21.15.102:0/669185465 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f66b408cc60 con 0x7f66c40ff050 2024-03-26T18:53:49.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.500+0000 7f66c1d9b700 1 --2- 172.21.15.102:0/669185465 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f66ac06c680 0x7f66ac06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:49.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.501+0000 7f66c1d9b700 1 --2- 172.21.15.102:0/669185465 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f66ac06c680 0x7f66ac06eb30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009c60 tx=0x7f66b8009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:49.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.505+0000 7f66b37fe700 1 -- 172.21.15.102:0/669185465 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f66b405adf0 con 0x7f66c40ff050 2024-03-26T18:53:49.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:49 smithi190 ceph-mon[35043]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-26T18:53:49.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:49 smithi190 ceph-mon[35043]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:49.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:49 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:53:49.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:49 smithi102 ceph-mon[29002]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-26T18:53:49.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:49 smithi102 ceph-mon[29002]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:49.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:49 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:53:49.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.824+0000 7f66c8d35700 1 -- 172.21.15.102:0/669185465 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f66c4107d90 con 0x7f66c40ff050 2024-03-26T18:53:49.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.825+0000 7f66b37fe700 1 -- 172.21.15.102:0/669185465 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 17 v17) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f66b405a980 con 0x7f66c40ff050 2024-03-26T18:53:49.817 INFO:teuthology.orchestra.run.smithi102.stdout:e17 2024-03-26T18:53:49.817 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:53:49.818 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:53:49.818 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:53:49.818 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:53:49.818 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:53:49.818 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:53:49.818 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 17 2024-03-26T18:53:49.818 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:53:49.818 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:53:49.818 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:52:39.046175+0000 2024-03-26T18:53:49.819 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:53:49.819 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:53:49.819 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:53:49.819 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:53:49.819 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:53:49.819 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:53:49.819 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:53:49.819 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:53:49.819 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:53:49.819 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:53:49.820 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:53:49.820 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:53:49.820 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:53:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:53:49.822 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:53:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:53:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:53:49.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.828+0000 7f66c8d35700 1 -- 172.21.15.102:0/669185465 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f66ac06c680 msgr2=0x7f66ac06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:49.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.828+0000 7f66c8d35700 1 --2- 172.21.15.102:0/669185465 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f66ac06c680 0x7f66ac06eb30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009c60 tx=0x7f66b8009260 comp rx=0 tx=0).stop 2024-03-26T18:53:49.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.828+0000 7f66c8d35700 1 -- 172.21.15.102:0/669185465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c40ff050 msgr2=0x7f66c41959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:49.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.828+0000 7f66c8d35700 1 --2- 172.21.15.102:0/669185465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c40ff050 0x7f66c41959a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f66b400f6c0 tx=0x7f66b400f6f0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.828+0000 7f66c8d35700 1 -- 172.21.15.102:0/669185465 shutdown_connections 2024-03-26T18:53:49.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.828+0000 7f66c8d35700 1 --2- 172.21.15.102:0/669185465 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f66ac06c680 0x7f66ac06eb30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.828+0000 7f66c8d35700 1 --2- 172.21.15.102:0/669185465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c40ff050 0x7f66c41959a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.828+0000 7f66c8d35700 1 --2- 172.21.15.102:0/669185465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66c4100250 0x7f66c4195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.828+0000 7f66c8d35700 1 -- 172.21.15.102:0/669185465 >> 172.21.15.102:0/669185465 conn(0x7f66c40fa5e0 msgr2=0x7f66c40683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:49.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.828+0000 7f66c8d35700 1 -- 172.21.15.102:0/669185465 shutdown_connections 2024-03-26T18:53:49.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.828+0000 7f66c8d35700 1 -- 172.21.15.102:0/669185465 wait complete. 2024-03-26T18:53:49.823 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 17 2024-03-26T18:53:49.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.953+0000 7f391d273700 1 -- 172.21.15.102:0/979934507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3918103500 msgr2=0x7f39181058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.953+0000 7f391d273700 1 --2- 172.21.15.102:0/979934507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3918103500 0x7f39181058e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f390c009910 tx=0x7f390c009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.953+0000 7f391d273700 1 -- 172.21.15.102:0/979934507 shutdown_connections 2024-03-26T18:53:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.953+0000 7f391d273700 1 --2- 172.21.15.102:0/979934507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3918103500 0x7f39181058e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.953+0000 7f391d273700 1 --2- 172.21.15.102:0/979934507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3918100be0 0x7f3918102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.953+0000 7f391d273700 1 -- 172.21.15.102:0/979934507 >> 172.21.15.102:0/979934507 conn(0x7f39180fa5e0 msgr2=0x7f39180fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.954+0000 7f391d273700 1 -- 172.21.15.102:0/979934507 shutdown_connections 2024-03-26T18:53:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.954+0000 7f391d273700 1 -- 172.21.15.102:0/979934507 wait complete. 2024-03-26T18:53:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.954+0000 7f391d273700 1 Processor -- start 2024-03-26T18:53:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f391d273700 1 -- start start 2024-03-26T18:53:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f391d273700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3918100be0 0x7f3918195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f391d273700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3918103500 0x7f3918195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f391d273700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3918196490 con 0x7f3918103500 2024-03-26T18:53:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f391d273700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39181965d0 con 0x7f3918100be0 2024-03-26T18:53:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f39167fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3918103500 0x7f3918195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f39167fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3918103500 0x7f3918195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56094/0 (socket says 172.21.15.102:56094) 2024-03-26T18:53:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f39167fc700 1 -- 172.21.15.102:0/1408776808 learned_addr learned my addr 172.21.15.102:0/1408776808 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f3916ffd700 1 --2- 172.21.15.102:0/1408776808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3918100be0 0x7f3918195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f39167fc700 1 -- 172.21.15.102:0/1408776808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3918100be0 msgr2=0x7f3918195930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f39167fc700 1 --2- 172.21.15.102:0/1408776808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3918100be0 0x7f3918195930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.955+0000 7f39167fc700 1 -- 172.21.15.102:0/1408776808 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390c0095f0 con 0x7f3918103500 2024-03-26T18:53:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.956+0000 7f39167fc700 1 --2- 172.21.15.102:0/1408776808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3918103500 0x7f3918195e70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f390c0092e0 tx=0x7f390c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.956+0000 7f38fffff700 1 -- 172.21.15.102:0/1408776808 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f390c01d070 con 0x7f3918103500 2024-03-26T18:53:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.956+0000 7f391d273700 1 -- 172.21.15.102:0/1408776808 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391819b020 con 0x7f3918103500 2024-03-26T18:53:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.956+0000 7f391d273700 1 -- 172.21.15.102:0/1408776808 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391819b510 con 0x7f3918103500 2024-03-26T18:53:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.956+0000 7f38fffff700 1 -- 172.21.15.102:0/1408776808 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f390c005dc0 con 0x7f3918103500 2024-03-26T18:53:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.956+0000 7f38fffff700 1 -- 172.21.15.102:0/1408776808 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f390c017630 con 0x7f3918103500 2024-03-26T18:53:49.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.957+0000 7f391d273700 1 -- 172.21.15.102:0/1408776808 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391818fb70 con 0x7f3918103500 2024-03-26T18:53:49.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.958+0000 7f38fffff700 1 -- 172.21.15.102:0/1408776808 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f390c00faf0 con 0x7f3918103500 2024-03-26T18:53:49.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.958+0000 7f38fffff700 1 --2- 172.21.15.102:0/1408776808 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f390006c680 0x7f390006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.958+0000 7f38fffff700 1 -- 172.21.15.102:0/1408776808 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f390c08cc80 con 0x7f3918103500 2024-03-26T18:53:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.958+0000 7f3916ffd700 1 --2- 172.21.15.102:0/1408776808 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f390006c680 0x7f390006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.959+0000 7f3916ffd700 1 --2- 172.21.15.102:0/1408776808 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f390006c680 0x7f390006eb30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3908009c60 tx=0x7f3908009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:49.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:49.964+0000 7f38fffff700 1 -- 172.21.15.102:0/1408776808 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f390c05ae10 con 0x7f3918103500 2024-03-26T18:53:50.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.333+0000 7f391d273700 1 -- 172.21.15.102:0/1408776808 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3918061190 con 0x7f390006c680 2024-03-26T18:53:50.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.335+0000 7f38fffff700 1 -- 172.21.15.102:0/1408776808 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f3918061190 con 0x7f390006c680 2024-03-26T18:53:50.327 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:53:50.327 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:53:50.327 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:53:50.327 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:53:50.327 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-03-26T18:53:50.328 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "0/23 daemons upgraded", 2024-03-26T18:53:50.328 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 image on host smithi190", 2024-03-26T18:53:50.328 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:53:50.328 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:53:50.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.338+0000 7f391d273700 1 -- 172.21.15.102:0/1408776808 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f390006c680 msgr2=0x7f390006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:50.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.338+0000 7f391d273700 1 --2- 172.21.15.102:0/1408776808 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f390006c680 0x7f390006eb30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3908009c60 tx=0x7f3908009260 comp rx=0 tx=0).stop 2024-03-26T18:53:50.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.338+0000 7f391d273700 1 -- 172.21.15.102:0/1408776808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3918103500 msgr2=0x7f3918195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:50.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.338+0000 7f391d273700 1 --2- 172.21.15.102:0/1408776808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3918103500 0x7f3918195e70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f390c0092e0 tx=0x7f390c00f690 comp rx=0 tx=0).stop 2024-03-26T18:53:50.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.338+0000 7f391d273700 1 -- 172.21.15.102:0/1408776808 shutdown_connections 2024-03-26T18:53:50.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.338+0000 7f391d273700 1 --2- 172.21.15.102:0/1408776808 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f390006c680 0x7f390006eb30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:50.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.338+0000 7f391d273700 1 --2- 172.21.15.102:0/1408776808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3918100be0 0x7f3918195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:50.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.338+0000 7f391d273700 1 --2- 172.21.15.102:0/1408776808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3918103500 0x7f3918195e70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:50.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.338+0000 7f391d273700 1 -- 172.21.15.102:0/1408776808 >> 172.21.15.102:0/1408776808 conn(0x7f39180fa5e0 msgr2=0x7f39180fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:50.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.338+0000 7f391d273700 1 -- 172.21.15.102:0/1408776808 shutdown_connections 2024-03-26T18:53:50.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.338+0000 7f391d273700 1 -- 172.21.15.102:0/1408776808 wait complete. 2024-03-26T18:53:50.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.477+0000 7ff337893700 1 -- 172.21.15.102:0/4103841299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff330100be0 msgr2=0x7ff330102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:50.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.477+0000 7ff337893700 1 --2- 172.21.15.102:0/4103841299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff330100be0 0x7ff330102fc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009910 tx=0x7ff32c009c20 comp rx=0 tx=0).stop 2024-03-26T18:53:50.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.477+0000 7ff337893700 1 -- 172.21.15.102:0/4103841299 shutdown_connections 2024-03-26T18:53:50.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.477+0000 7ff337893700 1 --2- 172.21.15.102:0/4103841299 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff330103500 0x7ff3301058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:50.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.477+0000 7ff337893700 1 --2- 172.21.15.102:0/4103841299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff330100be0 0x7ff330102fc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:50.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.477+0000 7ff337893700 1 -- 172.21.15.102:0/4103841299 >> 172.21.15.102:0/4103841299 conn(0x7ff3300fa5e0 msgr2=0x7ff3300fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:50.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.477+0000 7ff337893700 1 -- 172.21.15.102:0/4103841299 shutdown_connections 2024-03-26T18:53:50.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.477+0000 7ff337893700 1 -- 172.21.15.102:0/4103841299 wait complete. 2024-03-26T18:53:50.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.478+0000 7ff337893700 1 Processor -- start 2024-03-26T18:53:50.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.478+0000 7ff337893700 1 -- start start 2024-03-26T18:53:50.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.478+0000 7ff337893700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff330100be0 0x7ff330100540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.478+0000 7ff337893700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff330103500 0x7ff3300feb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.478+0000 7ff337893700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3300ff0d0 con 0x7ff330100be0 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.478+0000 7ff337893700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3300ff210 con 0x7ff330103500 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.479+0000 7ff33562f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff330100be0 0x7ff330100540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.479+0000 7ff33562f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff330100be0 0x7ff330100540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56108/0 (socket says 172.21.15.102:56108) 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.479+0000 7ff33562f700 1 -- 172.21.15.102:0/1409866287 learned_addr learned my addr 172.21.15.102:0/1409866287 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.479+0000 7ff334e2e700 1 --2- 172.21.15.102:0/1409866287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff330103500 0x7ff3300feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.479+0000 7ff334e2e700 1 -- 172.21.15.102:0/1409866287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff330100be0 msgr2=0x7ff330100540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.479+0000 7ff334e2e700 1 --2- 172.21.15.102:0/1409866287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff330100be0 0x7ff330100540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.479+0000 7ff334e2e700 1 -- 172.21.15.102:0/1409866287 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c0095f0 con 0x7ff330103500 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.479+0000 7ff334e2e700 1 --2- 172.21.15.102:0/1409866287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff330103500 0x7ff3300feb90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff32000e970 tx=0x7ff32000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.479+0000 7ff3267fc700 1 -- 172.21.15.102:0/1409866287 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff32000c9d0 con 0x7ff330103500 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.479+0000 7ff337893700 1 -- 172.21.15.102:0/1409866287 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3300ff4f0 con 0x7ff330103500 2024-03-26T18:53:50.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.479+0000 7ff3267fc700 1 -- 172.21.15.102:0/1409866287 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff320010430 con 0x7ff330103500 2024-03-26T18:53:50.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.480+0000 7ff3267fc700 1 -- 172.21.15.102:0/1409866287 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff320018650 con 0x7ff330103500 2024-03-26T18:53:50.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.480+0000 7ff337893700 1 -- 172.21.15.102:0/1409866287 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3300ff980 con 0x7ff330103500 2024-03-26T18:53:50.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.480+0000 7ff337893700 1 -- 172.21.15.102:0/1409866287 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33018fb70 con 0x7ff330103500 2024-03-26T18:53:50.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.481+0000 7ff3267fc700 1 -- 172.21.15.102:0/1409866287 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7ff3200105a0 con 0x7ff330103500 2024-03-26T18:53:50.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.481+0000 7ff3267fc700 1 --2- 172.21.15.102:0/1409866287 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff31c06c680 0x7ff31c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:53:50.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.482+0000 7ff3267fc700 1 -- 172.21.15.102:0/1409866287 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff320014070 con 0x7ff330103500 2024-03-26T18:53:50.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.482+0000 7ff33562f700 1 --2- 172.21.15.102:0/1409866287 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff31c06c680 0x7ff31c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:53:50.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.482+0000 7ff33562f700 1 --2- 172.21.15.102:0/1409866287 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff31c06c680 0x7ff31c06eb30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009fd0 tx=0x7ff32c005940 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:53:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.490+0000 7ff3267fc700 1 -- 172.21.15.102:0/1409866287 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff320051720 con 0x7ff330103500 2024-03-26T18:53:50.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:50 smithi190 ceph-mon[35043]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:50 smithi190 ceph-mon[35043]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:50 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/167896939' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:53:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:50 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/669185465' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:53:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:50 smithi102 ceph-mon[29002]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:50 smithi102 ceph-mon[29002]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:50 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/167896939' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:53:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:50 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/669185465' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:53:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.848+0000 7ff337893700 1 -- 172.21.15.102:0/1409866287 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff33002cc80 con 0x7ff330103500 2024-03-26T18:53:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.849+0000 7ff3267fc700 1 -- 172.21.15.102:0/1409866287 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7ff320059d10 con 0x7ff330103500 2024-03-26T18:53:50.842 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T18:53:50.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.852+0000 7ff337893700 1 -- 172.21.15.102:0/1409866287 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff31c06c680 msgr2=0x7ff31c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:50.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.852+0000 7ff337893700 1 --2- 172.21.15.102:0/1409866287 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff31c06c680 0x7ff31c06eb30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009fd0 tx=0x7ff32c005940 comp rx=0 tx=0).stop 2024-03-26T18:53:50.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.852+0000 7ff337893700 1 -- 172.21.15.102:0/1409866287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff330103500 msgr2=0x7ff3300feb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:53:50.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.852+0000 7ff337893700 1 --2- 172.21.15.102:0/1409866287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff330103500 0x7ff3300feb90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff32000e970 tx=0x7ff32000ec80 comp rx=0 tx=0).stop 2024-03-26T18:53:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.853+0000 7ff337893700 1 -- 172.21.15.102:0/1409866287 shutdown_connections 2024-03-26T18:53:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.853+0000 7ff337893700 1 --2- 172.21.15.102:0/1409866287 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7ff31c06c680 0x7ff31c06eb30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.853+0000 7ff337893700 1 --2- 172.21.15.102:0/1409866287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff330100be0 0x7ff330100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.853+0000 7ff337893700 1 --2- 172.21.15.102:0/1409866287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff330103500 0x7ff3300feb90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:53:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.853+0000 7ff337893700 1 -- 172.21.15.102:0/1409866287 >> 172.21.15.102:0/1409866287 conn(0x7ff3300fa5e0 msgr2=0x7ff3300fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:53:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.853+0000 7ff337893700 1 -- 172.21.15.102:0/1409866287 shutdown_connections 2024-03-26T18:53:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:53:50.853+0000 7ff337893700 1 -- 172.21.15.102:0/1409866287 wait complete. 2024-03-26T18:53:51.030 INFO:tasks.workunit.client.0.smithi102.stderr:Updating files: 85% (10422/12167) Updating files: 86% (10464/12167) Updating files: 87% (10586/12167) Updating files: 88% (10707/12167) Updating files: 89% (10829/12167) Updating files: 90% (10951/12167) Updating files: 91% (11072/12167) Updating files: 92% (11194/12167) Updating files: 93% (11316/12167) Updating files: 94% (11437/12167) Updating files: 95% (11559/12167) Updating files: 96% (11681/12167) Updating files: 97% (11802/12167) Updating files: 98% (11924/12167) Updating files: 99% (12046/12167) Updating files: 100% (12167/12167) Updating files: 100% (12167/12167), done. 2024-03-26T18:53:51.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:51 smithi190 ceph-mon[35043]: pgmap v223: 65 pgs: 65 active+clean; 1.9 MiB data, 162 MiB used, 536 GiB / 536 GiB avail; 112 KiB/s wr, 0 op/s 2024-03-26T18:53:51.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:51 smithi190 ceph-mon[35043]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:51.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:51 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/1409866287' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:53:51.579 INFO:tasks.workunit.client.0.smithi102.stderr:Note: switching to '72e2437271dcafb9826f2f49bb1001426fa6de81'. 2024-03-26T18:53:51.579 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-26T18:53:51.579 INFO:tasks.workunit.client.0.smithi102.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-26T18:53:51.579 INFO:tasks.workunit.client.0.smithi102.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-26T18:53:51.579 INFO:tasks.workunit.client.0.smithi102.stderr:state without impacting any branches by switching back to a branch. 2024-03-26T18:53:51.579 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-26T18:53:51.579 INFO:tasks.workunit.client.0.smithi102.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-26T18:53:51.579 INFO:tasks.workunit.client.0.smithi102.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-26T18:53:51.580 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-26T18:53:51.580 INFO:tasks.workunit.client.0.smithi102.stderr: git switch -c 2024-03-26T18:53:51.580 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-26T18:53:51.580 INFO:tasks.workunit.client.0.smithi102.stderr:Or undo this operation with: 2024-03-26T18:53:51.580 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-26T18:53:51.580 INFO:tasks.workunit.client.0.smithi102.stderr: git switch - 2024-03-26T18:53:51.580 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-26T18:53:51.580 INFO:tasks.workunit.client.0.smithi102.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-26T18:53:51.580 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-26T18:53:51.580 INFO:tasks.workunit.client.0.smithi102.stderr:HEAD is now at 72e2437271d qa/cephfs/overrides: ignore expected PG_DEGRADED health warning 2024-03-26T18:53:51.583 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-26T18:53:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:51 smithi102 ceph-mon[29002]: pgmap v223: 65 pgs: 65 active+clean; 1.9 MiB data, 162 MiB used, 536 GiB / 536 GiB avail; 112 KiB/s wr, 0 op/s 2024-03-26T18:53:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:51 smithi102 ceph-mon[29002]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:53:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:51 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1409866287' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:53:51.725 INFO:tasks.workunit.client.0.smithi102.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-26T18:53:51.727 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T18:53:51.727 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-26T18:53:52.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:52 smithi102 ceph-mon[29002]: pgmap v224: 65 pgs: 65 active+clean; 4.8 MiB data, 170 MiB used, 536 GiB / 536 GiB avail; 382 KiB/s wr, 0 op/s 2024-03-26T18:53:52.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:52 smithi190 ceph-mon[35043]: pgmap v224: 65 pgs: 65 active+clean; 4.8 MiB data, 170 MiB used, 536 GiB / 536 GiB avail; 382 KiB/s wr, 0 op/s 2024-03-26T18:53:55.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:54 smithi190 ceph-mon[35043]: pgmap v225: 65 pgs: 65 active+clean; 17 MiB data, 230 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s wr, 114 op/s 2024-03-26T18:53:55.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:54 smithi102 ceph-mon[29002]: pgmap v225: 65 pgs: 65 active+clean; 17 MiB data, 230 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s wr, 114 op/s 2024-03-26T18:53:57.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:57 smithi190 ceph-mon[35043]: pgmap v226: 65 pgs: 65 active+clean; 50 MiB data, 354 MiB used, 536 GiB / 536 GiB avail; 4.4 MiB/s wr, 435 op/s 2024-03-26T18:53:57.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:57 smithi190 ceph-mon[35043]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:53:57.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:57 smithi102 ceph-mon[29002]: pgmap v226: 65 pgs: 65 active+clean; 50 MiB data, 354 MiB used, 536 GiB / 536 GiB avail; 4.4 MiB/s wr, 435 op/s 2024-03-26T18:53:57.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:57 smithi102 ceph-mon[29002]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:53:59.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:53:59 smithi190 ceph-mon[35043]: pgmap v227: 65 pgs: 65 active+clean; 50 MiB data, 354 MiB used, 536 GiB / 536 GiB avail; 4.4 MiB/s wr, 435 op/s 2024-03-26T18:53:59.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:53:59 smithi102 ceph-mon[29002]: pgmap v227: 65 pgs: 65 active+clean; 50 MiB data, 354 MiB used, 536 GiB / 536 GiB avail; 4.4 MiB/s wr, 435 op/s 2024-03-26T18:53:59.821 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-26T18:54:00.210 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-26T18:54:00.340 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T18:54:00.341 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T18:54:00.341 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-26T18:54:00.472 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T18:54:00.475 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T18:54:00.475 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-26T18:54:00.531 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-26T18:54:00.532 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-26T18:54:00.532 DEBUG:teuthology.orchestra.run.smithi102:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=72e2437271dcafb9826f2f49bb1001426fa6de81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-26T18:54:00.605 INFO:tasks.workunit.client.0.smithi102.stderr:+ mkdir -p fsstress 2024-03-26T18:54:00.609 INFO:tasks.workunit.client.0.smithi102.stderr:+ pushd fsstress 2024-03-26T18:54:00.609 INFO:tasks.workunit.client.0.smithi102.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-26T18:54:00.609 INFO:tasks.workunit.client.0.smithi102.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-26T18:54:01.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:01 smithi190 ceph-mon[35043]: pgmap v228: 65 pgs: 65 active+clean; 62 MiB data, 398 MiB used, 536 GiB / 536 GiB avail; 5.4 MiB/s wr, 456 op/s 2024-03-26T18:54:01.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:01 smithi102 ceph-mon[29002]: pgmap v228: 65 pgs: 65 active+clean; 62 MiB data, 398 MiB used, 536 GiB / 536 GiB avail; 5.4 MiB/s wr, 456 op/s 2024-03-26T18:54:02.175 INFO:tasks.workunit.client.1.smithi190.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-26T18:54:02.184 INFO:tasks.workunit.client.1.smithi190.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-26T18:54:02.185 INFO:tasks.workunit.client.1.smithi190.stderr:+ make 2024-03-26T18:54:02.209 INFO:tasks.workunit.client.0.smithi102.stderr:+ tar xzf ltp-full.tgz 2024-03-26T18:54:02.279 INFO:tasks.workunit.client.1.smithi190.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-26T18:54:02.540 INFO:tasks.workunit.client.1.smithi190.stderr:++ readlink -f fsstress 2024-03-26T18:54:02.540 INFO:tasks.workunit.client.1.smithi190.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-26T18:54:02.540 INFO:tasks.workunit.client.1.smithi190.stderr:+ popd 2024-03-26T18:54:02.541 INFO:tasks.workunit.client.1.smithi190.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-26T18:54:02.541 INFO:tasks.workunit.client.1.smithi190.stderr:+ popd 2024-03-26T18:54:02.541 INFO:tasks.workunit.client.1.smithi190.stdout:~/cephtest/mnt.1/client.1/tmp 2024-03-26T18:54:02.541 INFO:tasks.workunit.client.1.smithi190.stderr:++ mktemp -d -p . 2024-03-26T18:54:02.544 INFO:tasks.workunit.client.1.smithi190.stderr:+ T=./tmp.EA7GdPmEKz 2024-03-26T18:54:02.544 INFO:tasks.workunit.client.1.smithi190.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.EA7GdPmEKz -l 1 -n 1000 -p 10 -v 2024-03-26T18:54:02.546 INFO:tasks.workunit.client.1.smithi190.stdout:seed = 1712021870 2024-03-26T18:54:02.553 INFO:tasks.workunit.client.1.smithi190.stdout:1/0: link - no file 2024-03-26T18:54:02.553 INFO:tasks.workunit.client.1.smithi190.stdout:1/1: read - no filename 2024-03-26T18:54:02.554 INFO:tasks.workunit.client.1.smithi190.stdout:1/2: dread - no filename 2024-03-26T18:54:02.555 INFO:tasks.workunit.client.1.smithi190.stdout:0/0: mkdir d0 0 2024-03-26T18:54:02.555 INFO:tasks.workunit.client.1.smithi190.stdout:0/1: unlink - no file 2024-03-26T18:54:02.557 INFO:tasks.workunit.client.1.smithi190.stdout:2/0: rmdir - no directory 2024-03-26T18:54:02.557 INFO:tasks.workunit.client.1.smithi190.stdout:2/1: dread - no filename 2024-03-26T18:54:02.557 INFO:tasks.workunit.client.1.smithi190.stdout:2/2: readlink - no filename 2024-03-26T18:54:02.557 INFO:tasks.workunit.client.1.smithi190.stdout:2/3: stat - no entries 2024-03-26T18:54:02.557 INFO:tasks.workunit.client.1.smithi190.stdout:2/4: fsync - no filename 2024-03-26T18:54:02.558 INFO:tasks.workunit.client.1.smithi190.stdout:1/3: creat f0 x:0 0 0 2024-03-26T18:54:02.560 INFO:tasks.workunit.client.1.smithi190.stdout:0/2: symlink d0/l1 0 2024-03-26T18:54:02.561 INFO:tasks.workunit.client.1.smithi190.stdout:3/0: dread - no filename 2024-03-26T18:54:02.562 INFO:tasks.workunit.client.1.smithi190.stdout:3/1: stat - no entries 2024-03-26T18:54:02.562 INFO:tasks.workunit.client.1.smithi190.stdout:3/2: rmdir - no directory 2024-03-26T18:54:02.562 INFO:tasks.workunit.client.1.smithi190.stdout:3/3: rename - no filename 2024-03-26T18:54:02.563 INFO:tasks.workunit.client.1.smithi190.stdout:2/5: symlink l0 0 2024-03-26T18:54:02.565 INFO:tasks.workunit.client.1.smithi190.stdout:1/4: rename f0 to f1 0 2024-03-26T18:54:02.565 INFO:tasks.workunit.client.1.smithi190.stdout:1/5: creat f2 x:0 0 0 2024-03-26T18:54:02.567 INFO:tasks.workunit.client.1.smithi190.stdout:0/3: creat d0/f2 x:0 0 0 2024-03-26T18:54:02.568 INFO:tasks.workunit.client.1.smithi190.stdout:4/0: chown . 282222 1 2024-03-26T18:54:02.568 INFO:tasks.workunit.client.1.smithi190.stdout:4/1: link - no file 2024-03-26T18:54:02.570 INFO:tasks.workunit.client.1.smithi190.stdout:3/4: creat f0 x:0 0 0 2024-03-26T18:54:02.570 INFO:tasks.workunit.client.1.smithi190.stdout:3/5: truncate f0 211109 0 2024-03-26T18:54:02.571 INFO:tasks.workunit.client.1.smithi190.stdout:3/6: dread f0 [0,4194304] 0 2024-03-26T18:54:02.571 INFO:tasks.workunit.client.1.smithi190.stdout:3/7: rmdir - no directory 2024-03-26T18:54:02.572 INFO:tasks.workunit.client.1.smithi190.stdout:2/6: creat f1 x:0 0 0 2024-03-26T18:54:02.572 INFO:tasks.workunit.client.1.smithi190.stdout:2/7: chown l0 46413022 1 2024-03-26T18:54:02.575 INFO:tasks.workunit.client.1.smithi190.stdout:1/6: mkdir d3 0 2024-03-26T18:54:02.577 INFO:tasks.workunit.client.1.smithi190.stdout:0/4: rename d0/l1 to d0/l3 0 2024-03-26T18:54:02.577 INFO:tasks.workunit.client.1.smithi190.stdout:0/5: truncate d0/f2 665096 0 2024-03-26T18:54:02.578 INFO:tasks.workunit.client.1.smithi190.stdout:5/0: write - no filename 2024-03-26T18:54:02.578 INFO:tasks.workunit.client.1.smithi190.stdout:5/1: dwrite - no filename 2024-03-26T18:54:02.579 INFO:tasks.workunit.client.1.smithi190.stdout:5/2: write - no filename 2024-03-26T18:54:02.579 INFO:tasks.workunit.client.1.smithi190.stdout:5/3: dwrite - no filename 2024-03-26T18:54:02.579 INFO:tasks.workunit.client.1.smithi190.stdout:5/4: fsync - no filename 2024-03-26T18:54:02.580 INFO:tasks.workunit.client.1.smithi190.stdout:4/2: mknod c0 0 2024-03-26T18:54:02.580 INFO:tasks.workunit.client.1.smithi190.stdout:4/3: fdatasync - no filename 2024-03-26T18:54:02.582 INFO:tasks.workunit.client.1.smithi190.stdout:3/8: mkdir d1 0 2024-03-26T18:54:02.584 INFO:tasks.workunit.client.1.smithi190.stdout:2/8: rename f1 to f2 0 2024-03-26T18:54:02.586 INFO:tasks.workunit.client.1.smithi190.stdout:1/7: creat d3/f4 x:0 0 0 2024-03-26T18:54:02.586 INFO:tasks.workunit.client.1.smithi190.stdout:1/8: creat d3/f5 x:0 0 0 2024-03-26T18:54:02.586 INFO:tasks.workunit.client.1.smithi190.stdout:1/9: chown f1 115464223 1 2024-03-26T18:54:02.586 INFO:tasks.workunit.client.1.smithi190.stdout:1/10: write f1 [706971,42156] 0 2024-03-26T18:54:02.587 INFO:tasks.workunit.client.1.smithi190.stdout:1/11: chown d3/f4 1231 1 2024-03-26T18:54:02.587 INFO:tasks.workunit.client.1.smithi190.stdout:1/12: creat d3/f6 x:0 0 0 2024-03-26T18:54:02.587 INFO:tasks.workunit.client.1.smithi190.stdout:1/13: write f1 [1099754,52767] 0 2024-03-26T18:54:02.587 INFO:tasks.workunit.client.1.smithi190.stdout:6/0: stat - no entries 2024-03-26T18:54:02.588 INFO:tasks.workunit.client.1.smithi190.stdout:6/1: stat - no entries 2024-03-26T18:54:02.588 INFO:tasks.workunit.client.1.smithi190.stdout:5/5: creat f0 x:0 0 0 2024-03-26T18:54:02.597 INFO:tasks.workunit.client.1.smithi190.stdout:7/0: rename - no filename 2024-03-26T18:54:02.597 INFO:tasks.workunit.client.1.smithi190.stdout:7/1: chown . 2 1 2024-03-26T18:54:02.597 INFO:tasks.workunit.client.1.smithi190.stdout:3/9: dwrite f0 [0,4194304] 0 2024-03-26T18:54:02.597 INFO:tasks.workunit.client.1.smithi190.stdout:6/2: creat f0 x:0 0 0 2024-03-26T18:54:02.598 INFO:tasks.workunit.client.1.smithi190.stdout:6/3: dread - f0 zero size 2024-03-26T18:54:02.598 INFO:tasks.workunit.client.1.smithi190.stdout:5/6: symlink l1 0 2024-03-26T18:54:02.600 INFO:tasks.workunit.client.1.smithi190.stdout:8/0: dwrite - no filename 2024-03-26T18:54:02.600 INFO:tasks.workunit.client.1.smithi190.stdout:8/1: link - no file 2024-03-26T18:54:02.600 INFO:tasks.workunit.client.1.smithi190.stdout:8/2: dread - no filename 2024-03-26T18:54:02.604 INFO:tasks.workunit.client.1.smithi190.stdout:7/2: mknod c0 0 2024-03-26T18:54:02.604 INFO:tasks.workunit.client.1.smithi190.stdout:7/3: dread - no filename 2024-03-26T18:54:02.604 INFO:tasks.workunit.client.1.smithi190.stdout:7/4: write - no filename 2024-03-26T18:54:02.608 INFO:tasks.workunit.client.1.smithi190.stdout:6/4: unlink f0 0 2024-03-26T18:54:02.608 INFO:tasks.workunit.client.1.smithi190.stdout:6/5: creat f1 x:0 0 0 2024-03-26T18:54:02.608 INFO:tasks.workunit.client.1.smithi190.stdout:6/6: chown f1 753 1 2024-03-26T18:54:02.608 INFO:tasks.workunit.client.1.smithi190.stdout:6/7: creat f2 x:0 0 0 2024-03-26T18:54:02.610 INFO:tasks.workunit.client.1.smithi190.stdout:9/0: dwrite - no filename 2024-03-26T18:54:02.611 INFO:tasks.workunit.client.1.smithi190.stdout:8/3: mkdir d0 0 2024-03-26T18:54:02.611 INFO:tasks.workunit.client.1.smithi190.stdout:8/4: truncate - no filename 2024-03-26T18:54:02.613 INFO:tasks.workunit.client.1.smithi190.stdout:7/5: rename c0 to c1 0 2024-03-26T18:54:02.617 INFO:tasks.workunit.client.1.smithi190.stdout:9/1: mknod c0 0 2024-03-26T18:54:02.617 INFO:tasks.workunit.client.1.smithi190.stdout:9/2: write - no filename 2024-03-26T18:54:02.618 INFO:tasks.workunit.client.1.smithi190.stdout:8/5: rmdir d0 0 2024-03-26T18:54:02.618 INFO:tasks.workunit.client.1.smithi190.stdout:8/6: dread - no filename 2024-03-26T18:54:02.618 INFO:tasks.workunit.client.1.smithi190.stdout:8/7: write - no filename 2024-03-26T18:54:02.620 INFO:tasks.workunit.client.1.smithi190.stdout:7/6: symlink l2 0 2024-03-26T18:54:02.621 INFO:tasks.workunit.client.1.smithi190.stdout:9/3: creat f1 x:0 0 0 2024-03-26T18:54:02.621 INFO:tasks.workunit.client.1.smithi190.stdout:6/8: dwrite f2 [0,4194304] 0 2024-03-26T18:54:02.622 INFO:tasks.workunit.client.1.smithi190.stdout:8/8: symlink l1 0 2024-03-26T18:54:02.622 INFO:tasks.workunit.client.1.smithi190.stdout:8/9: dread - no filename 2024-03-26T18:54:02.623 INFO:tasks.workunit.client.1.smithi190.stdout:7/7: mkdir d3 0 2024-03-26T18:54:02.624 INFO:tasks.workunit.client.1.smithi190.stdout:7/8: truncate - no filename 2024-03-26T18:54:02.628 INFO:tasks.workunit.client.1.smithi190.stdout:6/9: mknod c3 0 2024-03-26T18:54:02.630 INFO:tasks.workunit.client.1.smithi190.stdout:7/9: creat d3/f4 x:0 0 0 2024-03-26T18:54:02.631 INFO:tasks.workunit.client.1.smithi190.stdout:7/10: dread - d3/f4 zero size 2024-03-26T18:54:02.632 INFO:tasks.workunit.client.1.smithi190.stdout:6/10: mknod c4 0 2024-03-26T18:54:02.632 INFO:tasks.workunit.client.1.smithi190.stdout:6/11: chown c4 56 1 2024-03-26T18:54:02.634 INFO:tasks.workunit.client.1.smithi190.stdout:7/11: rename d3/f4 to d3/f5 0 2024-03-26T18:54:02.637 INFO:tasks.workunit.client.1.smithi190.stdout:7/12: mkdir d3/d6 0 2024-03-26T18:54:02.637 INFO:tasks.workunit.client.1.smithi190.stdout:7/13: readlink l2 0 2024-03-26T18:54:02.637 INFO:tasks.workunit.client.1.smithi190.stdout:7/14: write d3/f5 [327724,91179] 0 2024-03-26T18:54:02.637 INFO:tasks.workunit.client.1.smithi190.stdout:7/15: rename d3 to d3/d6/d7 22 2024-03-26T18:54:02.658 INFO:tasks.workunit.client.1.smithi190.stdout:9/4: dwrite f1 [0,4194304] 0 2024-03-26T18:54:02.672 INFO:tasks.workunit.client.1.smithi190.stdout:9/5: symlink l2 0 2024-03-26T18:54:03.134 INFO:tasks.workunit.client.1.smithi190.stdout:3/10: fsync f0 0 2024-03-26T18:54:03.134 INFO:tasks.workunit.client.1.smithi190.stdout:3/11: rename d1 to d1/d2 22 2024-03-26T18:54:03.134 INFO:tasks.workunit.client.1.smithi190.stdout:3/12: fdatasync f0 0 2024-03-26T18:54:03.141 INFO:tasks.workunit.client.1.smithi190.stdout:4/4: getdents . 0 2024-03-26T18:54:03.149 INFO:tasks.workunit.client.1.smithi190.stdout:2/9: fsync f2 0 2024-03-26T18:54:03.149 INFO:tasks.workunit.client.1.smithi190.stdout:2/10: rmdir - no directory 2024-03-26T18:54:03.149 INFO:tasks.workunit.client.1.smithi190.stdout:2/11: rmdir - no directory 2024-03-26T18:54:03.149 INFO:tasks.workunit.client.1.smithi190.stdout:2/12: truncate f2 1027110 0 2024-03-26T18:54:03.158 INFO:tasks.workunit.client.1.smithi190.stdout:5/7: fsync f0 0 2024-03-26T18:54:03.159 INFO:tasks.workunit.client.1.smithi190.stdout:5/8: stat f0 0 2024-03-26T18:54:03.159 INFO:tasks.workunit.client.1.smithi190.stdout:5/9: dread - f0 zero size 2024-03-26T18:54:03.159 INFO:tasks.workunit.client.1.smithi190.stdout:5/10: fsync f0 0 2024-03-26T18:54:03.159 INFO:tasks.workunit.client.1.smithi190.stdout:5/11: rmdir - no directory 2024-03-26T18:54:03.159 INFO:tasks.workunit.client.1.smithi190.stdout:5/12: read - f0 zero size 2024-03-26T18:54:03.159 INFO:tasks.workunit.client.1.smithi190.stdout:5/13: dread - f0 zero size 2024-03-26T18:54:03.159 INFO:tasks.workunit.client.1.smithi190.stdout:5/14: write f0 [392891,59174] 0 2024-03-26T18:54:03.159 INFO:tasks.workunit.client.1.smithi190.stdout:5/15: truncate f0 1001177 0 2024-03-26T18:54:03.162 INFO:tasks.workunit.client.1.smithi190.stdout:1/14: rmdir d3 39 2024-03-26T18:54:03.162 INFO:tasks.workunit.client.1.smithi190.stdout:1/15: truncate f2 799840 0 2024-03-26T18:54:03.162 INFO:tasks.workunit.client.1.smithi190.stdout:1/16: dread - d3/f4 zero size 2024-03-26T18:54:03.162 INFO:tasks.workunit.client.1.smithi190.stdout:1/17: write f2 [1291137,117812] 0 2024-03-26T18:54:03.162 INFO:tasks.workunit.client.1.smithi190.stdout:1/18: chown d3/f5 665594475 1 2024-03-26T18:54:03.162 INFO:tasks.workunit.client.1.smithi190.stdout:1/19: truncate d3/f6 930625 0 2024-03-26T18:54:03.170 INFO:tasks.workunit.client.1.smithi190.stdout:0/6: truncate d0/f2 271723 0 2024-03-26T18:54:03.348 INFO:tasks.workunit.client.1.smithi190.stdout:6/12: dread f2 [0,4194304] 0 2024-03-26T18:54:03.508 INFO:tasks.workunit.client.1.smithi190.stdout:9/6: write f1 [1354036,15000] 0 2024-03-26T18:54:03.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:03 smithi190 ceph-mon[35043]: pgmap v229: 65 pgs: 65 active+clean; 63 MiB data, 416 MiB used, 536 GiB / 536 GiB avail; 5.4 MiB/s wr, 456 op/s 2024-03-26T18:54:03.563 INFO:tasks.workunit.client.1.smithi190.stdout:7/16: dwrite d3/f5 [0,4194304] 0 2024-03-26T18:54:03.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:03 smithi102 ceph-mon[29002]: pgmap v229: 65 pgs: 65 active+clean; 63 MiB data, 416 MiB used, 536 GiB / 536 GiB avail; 5.4 MiB/s wr, 456 op/s 2024-03-26T18:54:03.731 INFO:tasks.workunit.client.1.smithi190.stdout:2/13: dread f2 [0,4194304] 0 2024-03-26T18:54:03.731 INFO:tasks.workunit.client.1.smithi190.stdout:2/14: chown f2 56812822 1 2024-03-26T18:54:03.731 INFO:tasks.workunit.client.1.smithi190.stdout:2/15: creat f3 x:0 0 0 2024-03-26T18:54:03.740 INFO:tasks.workunit.client.1.smithi190.stdout:3/13: dwrite f0 [0,4194304] 0 2024-03-26T18:54:03.740 INFO:tasks.workunit.client.1.smithi190.stdout:3/14: chown d1 0 1 2024-03-26T18:54:03.898 INFO:tasks.workunit.client.1.smithi190.stdout:6/13: dwrite f2 [0,4194304] 0 2024-03-26T18:54:03.907 INFO:tasks.workunit.client.1.smithi190.stdout:6/14: write f2 [4126238,38910] 0 2024-03-26T18:54:04.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:04 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:54:04.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:04 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:54:04.929 INFO:tasks.workunit.client.1.smithi190.stdout:4/5: mkdir d1 0 2024-03-26T18:54:04.929 INFO:tasks.workunit.client.1.smithi190.stdout:4/6: readlink - no filename 2024-03-26T18:54:04.929 INFO:tasks.workunit.client.1.smithi190.stdout:4/7: dwrite - no filename 2024-03-26T18:54:04.929 INFO:tasks.workunit.client.1.smithi190.stdout:4/8: chown c0 13877053 1 2024-03-26T18:54:04.929 INFO:tasks.workunit.client.1.smithi190.stdout:4/9: chown d1 596748381 1 2024-03-26T18:54:04.956 INFO:tasks.workunit.client.1.smithi190.stdout:5/16: mkdir d2 0 2024-03-26T18:54:04.957 INFO:tasks.workunit.client.1.smithi190.stdout:1/20: creat d3/f7 x:0 0 0 2024-03-26T18:54:04.975 INFO:tasks.workunit.client.1.smithi190.stdout:5/17: dwrite f0 [0,4194304] 0 2024-03-26T18:54:05.498 INFO:tasks.workunit.client.1.smithi190.stdout:9/7: rename f1 to f3 0 2024-03-26T18:54:05.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:05 smithi190 ceph-mon[35043]: pgmap v230: 65 pgs: 65 active+clean; 91 MiB data, 440 MiB used, 536 GiB / 536 GiB avail; 7.5 MiB/s wr, 521 op/s 2024-03-26T18:54:05.585 INFO:tasks.workunit.client.1.smithi190.stdout:7/17: symlink d3/l8 0 2024-03-26T18:54:05.592 INFO:tasks.workunit.client.1.smithi190.stdout:7/18: write d3/f5 [735194,2189] 0 2024-03-26T18:54:05.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:05 smithi102 ceph-mon[29002]: pgmap v230: 65 pgs: 65 active+clean; 91 MiB data, 440 MiB used, 536 GiB / 536 GiB avail; 7.5 MiB/s wr, 521 op/s 2024-03-26T18:54:05.807 INFO:tasks.workunit.client.1.smithi190.stdout:2/16: rename f2 to f4 0 2024-03-26T18:54:05.818 INFO:tasks.workunit.client.1.smithi190.stdout:3/15: mkdir d1/d3 0 2024-03-26T18:54:06.213 INFO:tasks.workunit.client.1.smithi190.stdout:6/15: mknod c5 0 2024-03-26T18:54:06.214 INFO:tasks.workunit.client.1.smithi190.stdout:6/16: rmdir - no directory 2024-03-26T18:54:06.214 INFO:tasks.workunit.client.1.smithi190.stdout:6/17: chown f2 7 1 2024-03-26T18:54:06.215 INFO:tasks.workunit.client.1.smithi190.stdout:4/10: rmdir d1 0 2024-03-26T18:54:06.217 INFO:tasks.workunit.client.1.smithi190.stdout:1/21: symlink d3/l8 0 2024-03-26T18:54:06.218 INFO:tasks.workunit.client.1.smithi190.stdout:0/7: rmdir d0 39 2024-03-26T18:54:06.219 INFO:tasks.workunit.client.1.smithi190.stdout:1/22: fdatasync f2 0 2024-03-26T18:54:06.223 INFO:tasks.workunit.client.1.smithi190.stdout:5/18: rename f0 to d2/f3 0 2024-03-26T18:54:06.265 INFO:tasks.workunit.client.1.smithi190.stdout:7/19: mkdir d3/d6/d9 0 2024-03-26T18:54:06.268 INFO:tasks.workunit.client.1.smithi190.stdout:3/16: creat d1/f4 x:0 0 0 2024-03-26T18:54:06.268 INFO:tasks.workunit.client.1.smithi190.stdout:3/17: readlink - no filename 2024-03-26T18:54:06.268 INFO:tasks.workunit.client.1.smithi190.stdout:3/18: dread - d1/f4 zero size 2024-03-26T18:54:06.268 INFO:tasks.workunit.client.1.smithi190.stdout:3/19: write d1/f4 [578540,120989] 0 2024-03-26T18:54:06.269 INFO:tasks.workunit.client.1.smithi190.stdout:6/18: unlink c4 0 2024-03-26T18:54:06.271 INFO:tasks.workunit.client.1.smithi190.stdout:4/11: rename c0 to c2 0 2024-03-26T18:54:06.273 INFO:tasks.workunit.client.1.smithi190.stdout:0/8: symlink d0/l4 0 2024-03-26T18:54:06.275 INFO:tasks.workunit.client.1.smithi190.stdout:1/23: mkdir d3/d9 0 2024-03-26T18:54:06.275 INFO:tasks.workunit.client.1.smithi190.stdout:1/24: fsync d3/f6 0 2024-03-26T18:54:06.278 INFO:tasks.workunit.client.1.smithi190.stdout:5/19: creat d2/f4 x:0 0 0 2024-03-26T18:54:06.278 INFO:tasks.workunit.client.1.smithi190.stdout:5/20: dread - d2/f4 zero size 2024-03-26T18:54:06.280 INFO:tasks.workunit.client.1.smithi190.stdout:3/20: dread f0 [0,4194304] 0 2024-03-26T18:54:06.280 INFO:tasks.workunit.client.1.smithi190.stdout:3/21: creat d1/f5 x:0 0 0 2024-03-26T18:54:06.280 INFO:tasks.workunit.client.1.smithi190.stdout:9/8: truncate f3 948895 0 2024-03-26T18:54:06.282 INFO:tasks.workunit.client.1.smithi190.stdout:7/20: rename d3/l8 to d3/d6/la 0 2024-03-26T18:54:06.283 INFO:tasks.workunit.client.1.smithi190.stdout:3/22: dread f0 [0,4194304] 0 2024-03-26T18:54:06.284 INFO:tasks.workunit.client.1.smithi190.stdout:3/23: chown d1/d3 3 1 2024-03-26T18:54:06.285 INFO:tasks.workunit.client.1.smithi190.stdout:0/9: creat d0/f5 x:0 0 0 2024-03-26T18:54:06.288 INFO:tasks.workunit.client.1.smithi190.stdout:1/25: mkdir d3/da 0 2024-03-26T18:54:06.289 INFO:tasks.workunit.client.1.smithi190.stdout:6/19: dwrite f2 [0,4194304] 0 2024-03-26T18:54:06.289 INFO:tasks.workunit.client.1.smithi190.stdout:6/20: truncate f1 1034451 0 2024-03-26T18:54:06.294 INFO:tasks.workunit.client.1.smithi190.stdout:7/21: unlink d3/f5 0 2024-03-26T18:54:06.297 INFO:tasks.workunit.client.1.smithi190.stdout:3/24: symlink d1/d3/l6 0 2024-03-26T18:54:06.297 INFO:tasks.workunit.client.1.smithi190.stdout:3/25: write d1/f4 [31074,15354] 0 2024-03-26T18:54:06.300 INFO:tasks.workunit.client.1.smithi190.stdout:1/26: mkdir d3/d9/db 0 2024-03-26T18:54:06.300 INFO:tasks.workunit.client.1.smithi190.stdout:1/27: chown d3/l8 561182 1 2024-03-26T18:54:06.300 INFO:tasks.workunit.client.1.smithi190.stdout:1/28: fdatasync f2 0 2024-03-26T18:54:06.300 INFO:tasks.workunit.client.1.smithi190.stdout:1/29: getdents d3/da 0 2024-03-26T18:54:06.300 INFO:tasks.workunit.client.1.smithi190.stdout:1/30: write d3/f5 [404089,107744] 0 2024-03-26T18:54:06.302 INFO:tasks.workunit.client.1.smithi190.stdout:6/21: mknod c6 0 2024-03-26T18:54:06.305 INFO:tasks.workunit.client.1.smithi190.stdout:7/22: link c1 d3/d6/d9/cb 0 2024-03-26T18:54:06.305 INFO:tasks.workunit.client.1.smithi190.stdout:7/23: creat d3/fc x:0 0 0 2024-03-26T18:54:06.307 INFO:tasks.workunit.client.1.smithi190.stdout:3/26: unlink d1/d3/l6 0 2024-03-26T18:54:06.307 INFO:tasks.workunit.client.1.smithi190.stdout:3/27: readlink - no filename 2024-03-26T18:54:06.308 INFO:tasks.workunit.client.1.smithi190.stdout:1/31: mknod d3/d9/db/cc 0 2024-03-26T18:54:06.310 INFO:tasks.workunit.client.1.smithi190.stdout:6/22: link f1 f7 0 2024-03-26T18:54:06.310 INFO:tasks.workunit.client.1.smithi190.stdout:6/23: chown c6 12 1 2024-03-26T18:54:06.313 INFO:tasks.workunit.client.1.smithi190.stdout:1/32: dread d3/f5 [0,4194304] 0 2024-03-26T18:54:06.314 INFO:tasks.workunit.client.1.smithi190.stdout:7/24: creat d3/d6/fd x:0 0 0 2024-03-26T18:54:06.316 INFO:tasks.workunit.client.1.smithi190.stdout:6/24: rename f7 to f8 0 2024-03-26T18:54:06.316 INFO:tasks.workunit.client.1.smithi190.stdout:6/25: write f1 [1617373,108294] 0 2024-03-26T18:54:06.318 INFO:tasks.workunit.client.1.smithi190.stdout:1/33: mkdir d3/da/dd 0 2024-03-26T18:54:06.318 INFO:tasks.workunit.client.1.smithi190.stdout:1/34: write d3/f7 [155116,59853] 0 2024-03-26T18:54:06.318 INFO:tasks.workunit.client.1.smithi190.stdout:1/35: stat d3/da/dd 0 2024-03-26T18:54:06.318 INFO:tasks.workunit.client.1.smithi190.stdout:1/36: fdatasync f1 0 2024-03-26T18:54:06.319 INFO:tasks.workunit.client.1.smithi190.stdout:7/25: stat d3/d6/d9/cb 0 2024-03-26T18:54:06.322 INFO:tasks.workunit.client.1.smithi190.stdout:1/37: unlink d3/f4 0 2024-03-26T18:54:06.324 INFO:tasks.workunit.client.1.smithi190.stdout:6/26: fdatasync f1 0 2024-03-26T18:54:06.324 INFO:tasks.workunit.client.1.smithi190.stdout:6/27: write f1 [2653925,124053] 0 2024-03-26T18:54:06.325 INFO:tasks.workunit.client.1.smithi190.stdout:3/28: dwrite d1/f4 [0,4194304] 0 2024-03-26T18:54:06.325 INFO:tasks.workunit.client.1.smithi190.stdout:3/29: chown d1/f4 22695914 1 2024-03-26T18:54:06.326 INFO:tasks.workunit.client.1.smithi190.stdout:1/38: fdatasync d3/f7 0 2024-03-26T18:54:06.326 INFO:tasks.workunit.client.1.smithi190.stdout:1/39: write d3/f6 [1895424,92183] 0 2024-03-26T18:54:06.327 INFO:tasks.workunit.client.1.smithi190.stdout:1/40: write d3/f6 [355166,101828] 0 2024-03-26T18:54:06.331 INFO:tasks.workunit.client.1.smithi190.stdout:1/41: rename d3/f6 to d3/d9/fe 0 2024-03-26T18:54:06.332 INFO:tasks.workunit.client.1.smithi190.stdout:3/30: read f0 [3172192,92871] 0 2024-03-26T18:54:06.335 INFO:tasks.workunit.client.1.smithi190.stdout:3/31: unlink f0 0 2024-03-26T18:54:06.338 INFO:tasks.workunit.client.1.smithi190.stdout:3/32: mknod d1/d3/c7 0 2024-03-26T18:54:06.340 INFO:tasks.workunit.client.1.smithi190.stdout:3/33: mknod d1/d3/c8 0 2024-03-26T18:54:06.341 INFO:tasks.workunit.client.1.smithi190.stdout:6/28: dwrite f2 [0,4194304] 0 2024-03-26T18:54:06.341 INFO:tasks.workunit.client.1.smithi190.stdout:6/29: chown c5 21806449 1 2024-03-26T18:54:06.342 INFO:tasks.workunit.client.1.smithi190.stdout:3/34: mknod d1/d3/c9 0 2024-03-26T18:54:06.368 INFO:tasks.workunit.client.1.smithi190.stdout:6/30: dwrite f8 [0,4194304] 0 2024-03-26T18:54:06.368 INFO:tasks.workunit.client.1.smithi190.stdout:6/31: chown f1 4244 1 2024-03-26T18:54:06.369 INFO:tasks.workunit.client.1.smithi190.stdout:6/32: chown c5 15429 1 2024-03-26T18:54:06.369 INFO:tasks.workunit.client.1.smithi190.stdout:6/33: creat f9 x:0 0 0 2024-03-26T18:54:06.369 INFO:tasks.workunit.client.1.smithi190.stdout:6/34: dread - f9 zero size 2024-03-26T18:54:06.369 INFO:tasks.workunit.client.1.smithi190.stdout:6/35: truncate f9 410498 0 2024-03-26T18:54:06.526 INFO:tasks.workunit.client.1.smithi190.stdout:2/17: fsync f3 0 2024-03-26T18:54:06.526 INFO:tasks.workunit.client.1.smithi190.stdout:2/18: fdatasync f3 0 2024-03-26T18:54:06.527 INFO:tasks.workunit.client.1.smithi190.stdout:2/19: write f4 [818390,80683] 0 2024-03-26T18:54:06.660 INFO:tasks.workunit.client.1.smithi190.stdout:4/12: rename c2 to c3 0 2024-03-26T18:54:06.661 INFO:tasks.workunit.client.1.smithi190.stdout:4/13: symlink l4 0 2024-03-26T18:54:06.661 INFO:tasks.workunit.client.1.smithi190.stdout:4/14: dwrite - no filename 2024-03-26T18:54:06.662 INFO:tasks.workunit.client.1.smithi190.stdout:4/15: mknod c5 0 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/16: mknod c6 0 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/17: truncate - no filename 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/18: write - no filename 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/19: write - no filename 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/20: fdatasync - no filename 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/21: dwrite - no filename 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/22: dread - no filename 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/23: write - no filename 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/24: rmdir - no directory 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/25: dwrite - no filename 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/26: readlink l4 0 2024-03-26T18:54:06.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/27: dread - no filename 2024-03-26T18:54:06.664 INFO:tasks.workunit.client.1.smithi190.stdout:4/28: mkdir d7 0 2024-03-26T18:54:06.666 INFO:tasks.workunit.client.1.smithi190.stdout:0/10: fsync d0/f5 0 2024-03-26T18:54:06.668 INFO:tasks.workunit.client.1.smithi190.stdout:9/9: dread f3 [0,4194304] 0 2024-03-26T18:54:06.670 INFO:tasks.workunit.client.1.smithi190.stdout:9/10: unlink f3 0 2024-03-26T18:54:06.671 INFO:tasks.workunit.client.1.smithi190.stdout:9/11: rename l2 to l4 0 2024-03-26T18:54:06.671 INFO:tasks.workunit.client.1.smithi190.stdout:9/12: write - no filename 2024-03-26T18:54:06.671 INFO:tasks.workunit.client.1.smithi190.stdout:9/13: write - no filename 2024-03-26T18:54:06.671 INFO:tasks.workunit.client.1.smithi190.stdout:9/14: dread - no filename 2024-03-26T18:54:06.671 INFO:tasks.workunit.client.1.smithi190.stdout:9/15: chown l4 27305 1 2024-03-26T18:54:06.671 INFO:tasks.workunit.client.1.smithi190.stdout:9/16: dwrite - no filename 2024-03-26T18:54:06.671 INFO:tasks.workunit.client.1.smithi190.stdout:9/17: creat f5 x:0 0 0 2024-03-26T18:54:06.674 INFO:tasks.workunit.client.1.smithi190.stdout:9/18: link f5 f6 0 2024-03-26T18:54:06.679 INFO:tasks.workunit.client.1.smithi190.stdout:5/21: truncate d2/f3 3257540 0 2024-03-26T18:54:06.681 INFO:tasks.workunit.client.1.smithi190.stdout:5/22: link l1 d2/l5 0 2024-03-26T18:54:06.682 INFO:tasks.workunit.client.1.smithi190.stdout:5/23: link d2/f4 d2/f6 0 2024-03-26T18:54:06.682 INFO:tasks.workunit.client.1.smithi190.stdout:5/24: rename d2 to d2/d7 22 2024-03-26T18:54:06.685 INFO:tasks.workunit.client.1.smithi190.stdout:7/26: fsync d3/fc 0 2024-03-26T18:54:06.685 INFO:tasks.workunit.client.1.smithi190.stdout:7/27: creat d3/fe x:0 0 0 2024-03-26T18:54:06.687 INFO:tasks.workunit.client.1.smithi190.stdout:7/28: symlink d3/lf 0 2024-03-26T18:54:06.687 INFO:tasks.workunit.client.1.smithi190.stdout:7/29: dread - d3/d6/fd zero size 2024-03-26T18:54:06.688 INFO:tasks.workunit.client.1.smithi190.stdout:7/30: symlink d3/d6/d9/l10 0 2024-03-26T18:54:06.688 INFO:tasks.workunit.client.1.smithi190.stdout:7/31: truncate d3/fc 891107 0 2024-03-26T18:54:06.689 INFO:tasks.workunit.client.1.smithi190.stdout:7/32: getdents d3/d6 0 2024-03-26T18:54:06.690 INFO:tasks.workunit.client.1.smithi190.stdout:7/33: dread d3/fc [0,4194304] 0 2024-03-26T18:54:06.692 INFO:tasks.workunit.client.1.smithi190.stdout:7/34: mknod d3/d6/c11 0 2024-03-26T18:54:06.692 INFO:tasks.workunit.client.1.smithi190.stdout:7/35: chown d3/d6/d9 156694629 1 2024-03-26T18:54:06.693 INFO:tasks.workunit.client.1.smithi190.stdout:9/19: dwrite f6 [0,4194304] 0 2024-03-26T18:54:06.693 INFO:tasks.workunit.client.1.smithi190.stdout:7/36: link d3/d6/d9/l10 d3/d6/l12 0 2024-03-26T18:54:06.694 INFO:tasks.workunit.client.1.smithi190.stdout:0/11: dwrite d0/f2 [0,4194304] 0 2024-03-26T18:54:06.694 INFO:tasks.workunit.client.1.smithi190.stdout:7/37: dread d3/fc [0,4194304] 0 2024-03-26T18:54:06.698 INFO:tasks.workunit.client.1.smithi190.stdout:3/35: write d1/f4 [4836232,52335] 0 2024-03-26T18:54:06.699 INFO:tasks.workunit.client.1.smithi190.stdout:7/38: mknod d3/d6/d9/c13 0 2024-03-26T18:54:06.701 INFO:tasks.workunit.client.1.smithi190.stdout:1/42: rmdir d3 39 2024-03-26T18:54:06.702 INFO:tasks.workunit.client.1.smithi190.stdout:3/36: mknod d1/d3/ca 0 2024-03-26T18:54:06.704 INFO:tasks.workunit.client.1.smithi190.stdout:7/39: mkdir d3/d6/d9/d14 0 2024-03-26T18:54:06.704 INFO:tasks.workunit.client.1.smithi190.stdout:7/40: truncate d3/d6/fd 653073 0 2024-03-26T18:54:06.708 INFO:tasks.workunit.client.1.smithi190.stdout:7/41: rename d3/d6/d9/cb to d3/d6/d9/d14/c15 0 2024-03-26T18:54:06.708 INFO:tasks.workunit.client.1.smithi190.stdout:7/42: creat d3/f16 x:0 0 0 2024-03-26T18:54:06.708 INFO:tasks.workunit.client.1.smithi190.stdout:7/43: fdatasync d3/f16 0 2024-03-26T18:54:06.708 INFO:tasks.workunit.client.1.smithi190.stdout:7/44: dread - d3/f16 zero size 2024-03-26T18:54:06.708 INFO:tasks.workunit.client.1.smithi190.stdout:7/45: rename d3 to d3/d17 22 2024-03-26T18:54:06.711 INFO:tasks.workunit.client.1.smithi190.stdout:7/46: creat d3/d6/d9/d14/f18 x:0 0 0 2024-03-26T18:54:06.711 INFO:tasks.workunit.client.1.smithi190.stdout:7/47: creat d3/f19 x:0 0 0 2024-03-26T18:54:06.712 INFO:tasks.workunit.client.1.smithi190.stdout:0/12: dwrite d0/f2 [0,4194304] 0 2024-03-26T18:54:06.713 INFO:tasks.workunit.client.1.smithi190.stdout:7/48: rename d3/d6/l12 to d3/d6/l1a 0 2024-03-26T18:54:06.716 INFO:tasks.workunit.client.1.smithi190.stdout:0/13: getdents d0 0 2024-03-26T18:54:06.725 INFO:tasks.workunit.client.1.smithi190.stdout:0/14: read d0/f2 [1763590,8947] 0 2024-03-26T18:54:06.725 INFO:tasks.workunit.client.1.smithi190.stdout:0/15: fdatasync d0/f2 0 2024-03-26T18:54:06.727 INFO:tasks.workunit.client.1.smithi190.stdout:0/16: creat d0/f6 x:0 0 0 2024-03-26T18:54:06.727 INFO:tasks.workunit.client.1.smithi190.stdout:0/17: fdatasync d0/f6 0 2024-03-26T18:54:06.727 INFO:tasks.workunit.client.1.smithi190.stdout:0/18: truncate d0/f5 540718 0 2024-03-26T18:54:06.727 INFO:tasks.workunit.client.1.smithi190.stdout:0/19: stat d0/l3 0 2024-03-26T18:54:06.728 INFO:tasks.workunit.client.1.smithi190.stdout:0/20: mknod d0/c7 0 2024-03-26T18:54:06.730 INFO:tasks.workunit.client.1.smithi190.stdout:0/21: symlink d0/l8 0 2024-03-26T18:54:06.731 INFO:tasks.workunit.client.1.smithi190.stdout:0/22: symlink d0/l9 0 2024-03-26T18:54:06.733 INFO:tasks.workunit.client.1.smithi190.stdout:0/23: dread d0/f5 [0,4194304] 0 2024-03-26T18:54:06.733 INFO:tasks.workunit.client.1.smithi190.stdout:0/24: chown d0/l3 211 1 2024-03-26T18:54:06.733 INFO:tasks.workunit.client.1.smithi190.stdout:6/36: truncate f2 3572410 0 2024-03-26T18:54:06.736 INFO:tasks.workunit.client.1.smithi190.stdout:0/25: symlink d0/la 0 2024-03-26T18:54:06.738 INFO:tasks.workunit.client.1.smithi190.stdout:6/37: mknod ca 0 2024-03-26T18:54:06.758 INFO:tasks.workunit.client.1.smithi190.stdout:6/38: dwrite f9 [0,4194304] 0 2024-03-26T18:54:06.759 INFO:tasks.workunit.client.1.smithi190.stdout:0/26: dwrite d0/f6 [0,4194304] 0 2024-03-26T18:54:06.764 INFO:tasks.workunit.client.1.smithi190.stdout:0/27: dread d0/f2 [0,4194304] 0 2024-03-26T18:54:06.764 INFO:tasks.workunit.client.1.smithi190.stdout:0/28: chown d0/f2 28488785 1 2024-03-26T18:54:06.764 INFO:tasks.workunit.client.1.smithi190.stdout:0/29: stat d0/f2 0 2024-03-26T18:54:06.768 INFO:tasks.workunit.client.1.smithi190.stdout:0/30: dread d0/f5 [0,4194304] 0 2024-03-26T18:54:06.768 INFO:tasks.workunit.client.1.smithi190.stdout:6/39: dread f8 [0,4194304] 0 2024-03-26T18:54:06.768 INFO:tasks.workunit.client.1.smithi190.stdout:6/40: write f8 [4886543,74695] 0 2024-03-26T18:54:06.768 INFO:tasks.workunit.client.1.smithi190.stdout:6/41: fdatasync f2 0 2024-03-26T18:54:06.772 INFO:tasks.workunit.client.1.smithi190.stdout:6/42: link c6 cb 0 2024-03-26T18:54:06.772 INFO:tasks.workunit.client.1.smithi190.stdout:6/43: chown ca 31913821 1 2024-03-26T18:54:06.772 INFO:tasks.workunit.client.1.smithi190.stdout:6/44: fdatasync f2 0 2024-03-26T18:54:06.772 INFO:tasks.workunit.client.1.smithi190.stdout:6/45: chown f1 219623 1 2024-03-26T18:54:06.786 INFO:tasks.workunit.client.1.smithi190.stdout:0/31: dwrite d0/f6 [0,4194304] 0 2024-03-26T18:54:06.788 INFO:tasks.workunit.client.1.smithi190.stdout:0/32: mknod d0/cb 0 2024-03-26T18:54:06.789 INFO:tasks.workunit.client.1.smithi190.stdout:0/33: mkdir d0/dc 0 2024-03-26T18:54:06.802 INFO:tasks.workunit.client.1.smithi190.stdout:6/46: dwrite f1 [0,4194304] 0 2024-03-26T18:54:06.825 INFO:tasks.workunit.client.1.smithi190.stdout:9/20: getdents . 0 2024-03-26T18:54:06.829 INFO:tasks.workunit.client.1.smithi190.stdout:9/21: creat f7 x:0 0 0 2024-03-26T18:54:06.834 INFO:tasks.workunit.client.1.smithi190.stdout:5/25: dread d2/f3 [0,4194304] 0 2024-03-26T18:54:06.842 INFO:tasks.workunit.client.1.smithi190.stdout:9/22: dread f6 [0,4194304] 0 2024-03-26T18:54:06.849 INFO:tasks.workunit.client.1.smithi190.stdout:3/37: rmdir d1/d3 39 2024-03-26T18:54:06.849 INFO:tasks.workunit.client.1.smithi190.stdout:3/38: creat d1/fb x:0 0 0 2024-03-26T18:54:06.856 INFO:tasks.workunit.client.1.smithi190.stdout:9/23: truncate f6 637673 0 2024-03-26T18:54:06.864 INFO:tasks.workunit.client.1.smithi190.stdout:3/39: mknod d1/cc 0 2024-03-26T18:54:06.867 INFO:tasks.workunit.client.1.smithi190.stdout:0/34: fsync d0/f6 0 2024-03-26T18:54:06.868 INFO:tasks.workunit.client.1.smithi190.stdout:3/40: mknod d1/cd 0 2024-03-26T18:54:06.874 INFO:tasks.workunit.client.1.smithi190.stdout:3/41: mkdir d1/de 0 2024-03-26T18:54:06.879 INFO:tasks.workunit.client.1.smithi190.stdout:3/42: mkdir d1/d3/df 0 2024-03-26T18:54:06.887 INFO:tasks.workunit.client.1.smithi190.stdout:6/47: getdents . 0 2024-03-26T18:54:06.888 INFO:tasks.workunit.client.1.smithi190.stdout:3/43: creat d1/d3/f10 x:0 0 0 2024-03-26T18:54:06.888 INFO:tasks.workunit.client.1.smithi190.stdout:3/44: creat d1/f11 x:0 0 0 2024-03-26T18:54:07.073 INFO:tasks.workunit.client.1.smithi190.stdout:6/48: write f8 [2921173,22740] 0 2024-03-26T18:54:07.079 INFO:tasks.workunit.client.1.smithi190.stdout:6/49: creat fc x:0 0 0 2024-03-26T18:54:07.085 INFO:tasks.workunit.client.1.smithi190.stdout:6/50: mknod cd 0 2024-03-26T18:54:07.085 INFO:tasks.workunit.client.1.smithi190.stdout:6/51: rmdir - no directory 2024-03-26T18:54:07.086 INFO:tasks.workunit.client.1.smithi190.stdout:6/52: chown f1 391393890 1 2024-03-26T18:54:07.086 INFO:tasks.workunit.client.1.smithi190.stdout:6/53: read - fc zero size 2024-03-26T18:54:07.086 INFO:tasks.workunit.client.1.smithi190.stdout:6/54: fsync f2 0 2024-03-26T18:54:07.096 INFO:tasks.workunit.client.1.smithi190.stdout:5/26: dwrite d2/f4 [0,4194304] 0 2024-03-26T18:54:07.101 INFO:tasks.workunit.client.1.smithi190.stdout:3/45: fsync d1/fb 0 2024-03-26T18:54:07.101 INFO:tasks.workunit.client.1.smithi190.stdout:3/46: creat d1/d3/f12 x:0 0 0 2024-03-26T18:54:07.101 INFO:tasks.workunit.client.1.smithi190.stdout:3/47: fdatasync d1/d3/f10 0 2024-03-26T18:54:07.101 INFO:tasks.workunit.client.1.smithi190.stdout:3/48: dread - d1/d3/f10 zero size 2024-03-26T18:54:07.101 INFO:tasks.workunit.client.1.smithi190.stdout:3/49: getdents d1/d3/df 0 2024-03-26T18:54:07.101 INFO:tasks.workunit.client.1.smithi190.stdout:3/50: creat d1/d3/f13 x:0 0 0 2024-03-26T18:54:07.108 INFO:tasks.workunit.client.1.smithi190.stdout:9/24: write f5 [141564,28878] 0 2024-03-26T18:54:07.112 INFO:tasks.workunit.client.1.smithi190.stdout:0/35: dread d0/f6 [0,4194304] 0 2024-03-26T18:54:07.128 INFO:tasks.workunit.client.1.smithi190.stdout:3/51: link d1/d3/f10 d1/f14 0 2024-03-26T18:54:07.128 INFO:tasks.workunit.client.1.smithi190.stdout:3/52: creat d1/f15 x:0 0 0 2024-03-26T18:54:07.131 INFO:tasks.workunit.client.1.smithi190.stdout:9/25: unlink c0 0 2024-03-26T18:54:07.139 INFO:tasks.workunit.client.1.smithi190.stdout:3/53: mknod d1/c16 0 2024-03-26T18:54:07.139 INFO:tasks.workunit.client.1.smithi190.stdout:3/54: dread - d1/d3/f13 zero size 2024-03-26T18:54:07.143 INFO:tasks.workunit.client.1.smithi190.stdout:9/26: rename f7 to f8 0 2024-03-26T18:54:07.145 INFO:tasks.workunit.client.1.smithi190.stdout:3/55: symlink d1/d3/l17 0 2024-03-26T18:54:07.145 INFO:tasks.workunit.client.1.smithi190.stdout:3/56: truncate d1/d3/f13 371195 0 2024-03-26T18:54:07.145 INFO:tasks.workunit.client.1.smithi190.stdout:3/57: readlink d1/d3/l17 0 2024-03-26T18:54:07.145 INFO:tasks.workunit.client.1.smithi190.stdout:3/58: read - d1/fb zero size 2024-03-26T18:54:07.146 INFO:tasks.workunit.client.1.smithi190.stdout:9/27: creat f9 x:0 0 0 2024-03-26T18:54:07.155 INFO:tasks.workunit.client.1.smithi190.stdout:9/28: link f9 fa 0 2024-03-26T18:54:07.160 INFO:tasks.workunit.client.1.smithi190.stdout:9/29: creat fb x:0 0 0 2024-03-26T18:54:07.170 INFO:tasks.workunit.client.1.smithi190.stdout:0/36: dwrite d0/f5 [0,4194304] 0 2024-03-26T18:54:07.178 INFO:tasks.workunit.client.1.smithi190.stdout:0/37: mkdir d0/dd 0 2024-03-26T18:54:07.178 INFO:tasks.workunit.client.1.smithi190.stdout:0/38: read d0/f2 [1375521,71411] 0 2024-03-26T18:54:07.182 INFO:tasks.workunit.client.1.smithi190.stdout:0/39: creat d0/dd/fe x:0 0 0 2024-03-26T18:54:07.186 INFO:tasks.workunit.client.1.smithi190.stdout:0/40: symlink d0/dd/lf 0 2024-03-26T18:54:07.191 INFO:tasks.workunit.client.1.smithi190.stdout:0/41: truncate d0/f6 2647772 0 2024-03-26T18:54:07.191 INFO:tasks.workunit.client.1.smithi190.stdout:0/42: readlink d0/l3 0 2024-03-26T18:54:07.239 INFO:tasks.workunit.client.1.smithi190.stdout:9/30: dwrite fa [0,4194304] 0 2024-03-26T18:54:07.251 INFO:tasks.workunit.client.1.smithi190.stdout:9/31: link l4 lc 0 2024-03-26T18:54:07.254 INFO:tasks.workunit.client.1.smithi190.stdout:9/32: creat fd x:0 0 0 2024-03-26T18:54:07.256 INFO:tasks.workunit.client.1.smithi190.stdout:9/33: unlink fa 0 2024-03-26T18:54:07.256 INFO:tasks.workunit.client.1.smithi190.stdout:9/34: truncate fb 962397 0 2024-03-26T18:54:07.260 INFO:tasks.workunit.client.1.smithi190.stdout:6/55: dread f2 [0,4194304] 0 2024-03-26T18:54:07.264 INFO:tasks.workunit.client.1.smithi190.stdout:6/56: symlink le 0 2024-03-26T18:54:07.266 INFO:tasks.workunit.client.1.smithi190.stdout:6/57: rename le to lf 0 2024-03-26T18:54:07.268 INFO:tasks.workunit.client.1.smithi190.stdout:6/58: mknod c10 0 2024-03-26T18:54:07.286 INFO:tasks.workunit.client.1.smithi190.stdout:6/59: dwrite fc [0,4194304] 0 2024-03-26T18:54:07.301 INFO:tasks.workunit.client.1.smithi190.stdout:6/60: dwrite f9 [0,4194304] 0 2024-03-26T18:54:07.306 INFO:tasks.workunit.client.1.smithi190.stdout:6/61: dread f1 [0,4194304] 0 2024-03-26T18:54:07.310 INFO:tasks.workunit.client.1.smithi190.stdout:6/62: link c3 c11 0 2024-03-26T18:54:07.313 INFO:tasks.workunit.client.1.smithi190.stdout:6/63: rename cb to c12 0 2024-03-26T18:54:07.316 INFO:tasks.workunit.client.1.smithi190.stdout:6/64: rename ca to c13 0 2024-03-26T18:54:07.400 INFO:tasks.workunit.client.1.smithi190.stdout:0/43: write d0/f5 [4180216,76073] 0 2024-03-26T18:54:07.403 INFO:tasks.workunit.client.1.smithi190.stdout:0/44: unlink d0/l9 0 2024-03-26T18:54:07.403 INFO:tasks.workunit.client.1.smithi190.stdout:0/45: creat d0/f10 x:0 0 0 2024-03-26T18:54:07.424 INFO:tasks.workunit.client.1.smithi190.stdout:0/46: dwrite d0/f5 [0,4194304] 0 2024-03-26T18:54:07.424 INFO:tasks.workunit.client.1.smithi190.stdout:0/47: chown d0/cb 5910 1 2024-03-26T18:54:07.426 INFO:tasks.workunit.client.1.smithi190.stdout:0/48: mknod d0/c11 0 2024-03-26T18:54:07.437 INFO:tasks.workunit.client.1.smithi190.stdout:9/35: truncate f9 2431849 0 2024-03-26T18:54:07.444 INFO:tasks.workunit.client.1.smithi190.stdout:0/49: dwrite d0/f6 [0,4194304] 0 2024-03-26T18:54:07.446 INFO:tasks.workunit.client.1.smithi190.stdout:0/50: unlink d0/la 0 2024-03-26T18:54:07.455 INFO:tasks.workunit.client.1.smithi190.stdout:0/51: read d0/f6 [609117,105399] 0 2024-03-26T18:54:07.455 INFO:tasks.workunit.client.1.smithi190.stdout:0/52: write d0/f2 [777389,74354] 0 2024-03-26T18:54:07.457 INFO:tasks.workunit.client.1.smithi190.stdout:0/53: unlink d0/f10 0 2024-03-26T18:54:07.459 INFO:tasks.workunit.client.1.smithi190.stdout:0/54: rename d0/f2 to d0/dd/f12 0 2024-03-26T18:54:07.466 INFO:tasks.workunit.client.1.smithi190.stdout:0/55: dread d0/f6 [0,4194304] 0 2024-03-26T18:54:07.466 INFO:tasks.workunit.client.1.smithi190.stdout:0/56: creat d0/f13 x:0 0 0 2024-03-26T18:54:07.471 INFO:tasks.workunit.client.1.smithi190.stdout:0/57: dread d0/f6 [0,4194304] 0 2024-03-26T18:54:07.471 INFO:tasks.workunit.client.1.smithi190.stdout:0/58: read d0/f6 [3149224,29074] 0 2024-03-26T18:54:07.472 INFO:tasks.workunit.client.1.smithi190.stdout:0/59: mknod d0/c14 0 2024-03-26T18:54:07.473 INFO:tasks.workunit.client.1.smithi190.stdout:0/60: chown d0/dc 5 1 2024-03-26T18:54:07.474 INFO:tasks.workunit.client.1.smithi190.stdout:0/61: creat d0/dc/f15 x:0 0 0 2024-03-26T18:54:07.475 INFO:tasks.workunit.client.1.smithi190.stdout:0/62: write d0/f6 [48155,102394] 0 2024-03-26T18:54:07.475 INFO:tasks.workunit.client.1.smithi190.stdout:0/63: read - d0/dd/fe zero size 2024-03-26T18:54:07.475 INFO:tasks.workunit.client.1.smithi190.stdout:0/64: read d0/f6 [1201318,48933] 0 2024-03-26T18:54:07.476 INFO:tasks.workunit.client.1.smithi190.stdout:0/65: mknod d0/dc/c16 0 2024-03-26T18:54:07.476 INFO:tasks.workunit.client.1.smithi190.stdout:6/65: write fc [4865322,77680] 0 2024-03-26T18:54:07.478 INFO:tasks.workunit.client.1.smithi190.stdout:0/66: rename d0/dc/c16 to d0/c17 0 2024-03-26T18:54:07.480 INFO:tasks.workunit.client.1.smithi190.stdout:6/66: mknod c14 0 2024-03-26T18:54:07.480 INFO:tasks.workunit.client.1.smithi190.stdout:6/67: read f1 [708878,79590] 0 2024-03-26T18:54:07.480 INFO:tasks.workunit.client.1.smithi190.stdout:6/68: chown c12 2037 1 2024-03-26T18:54:07.480 INFO:tasks.workunit.client.1.smithi190.stdout:6/69: chown f1 1576 1 2024-03-26T18:54:07.513 INFO:tasks.workunit.client.1.smithi190.stdout:0/67: dwrite d0/f13 [0,4194304] 0 2024-03-26T18:54:07.513 INFO:tasks.workunit.client.1.smithi190.stdout:6/70: dwrite f8 [4194304,4194304] 0 2024-03-26T18:54:07.516 INFO:tasks.workunit.client.1.smithi190.stdout:6/71: fdatasync fc 0 2024-03-26T18:54:07.524 INFO:tasks.workunit.client.1.smithi190.stdout:6/72: write f9 [372150,120929] 0 2024-03-26T18:54:07.524 INFO:tasks.workunit.client.1.smithi190.stdout:0/68: dread d0/f5 [0,4194304] 0 2024-03-26T18:54:07.524 INFO:tasks.workunit.client.1.smithi190.stdout:0/69: write d0/dd/f12 [4007593,126383] 0 2024-03-26T18:54:07.524 INFO:tasks.workunit.client.1.smithi190.stdout:0/70: chown d0 43465 1 2024-03-26T18:54:07.526 INFO:tasks.workunit.client.1.smithi190.stdout:6/73: symlink l15 0 2024-03-26T18:54:07.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:07 smithi190 ceph-mon[35043]: pgmap v231: 65 pgs: 65 active+clean; 132 MiB data, 581 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s wr, 727 op/s 2024-03-26T18:54:07.550 INFO:tasks.workunit.client.1.smithi190.stdout:0/71: dwrite d0/dc/f15 [0,4194304] 0 2024-03-26T18:54:07.550 INFO:tasks.workunit.client.1.smithi190.stdout:6/74: dwrite f8 [8388608,4194304] 0 2024-03-26T18:54:07.563 INFO:tasks.workunit.client.1.smithi190.stdout:0/72: write d0/dc/f15 [1067590,108009] 0 2024-03-26T18:54:07.565 INFO:tasks.workunit.client.1.smithi190.stdout:0/73: mknod d0/dd/c18 0 2024-03-26T18:54:07.567 INFO:tasks.workunit.client.1.smithi190.stdout:0/74: mkdir d0/d19 0 2024-03-26T18:54:07.569 INFO:tasks.workunit.client.1.smithi190.stdout:0/75: mknod d0/dd/c1a 0 2024-03-26T18:54:07.570 INFO:tasks.workunit.client.1.smithi190.stdout:6/75: dwrite f8 [4194304,4194304] 0 2024-03-26T18:54:07.571 INFO:tasks.workunit.client.1.smithi190.stdout:0/76: symlink d0/dd/l1b 0 2024-03-26T18:54:07.572 INFO:tasks.workunit.client.1.smithi190.stdout:6/76: fsync f9 0 2024-03-26T18:54:07.572 INFO:tasks.workunit.client.1.smithi190.stdout:6/77: chown cd 211782 1 2024-03-26T18:54:07.572 INFO:tasks.workunit.client.1.smithi190.stdout:6/78: creat f16 x:0 0 0 2024-03-26T18:54:07.578 INFO:tasks.workunit.client.1.smithi190.stdout:6/79: mkdir d17 0 2024-03-26T18:54:07.581 INFO:tasks.workunit.client.1.smithi190.stdout:6/80: creat d17/f18 x:0 0 0 2024-03-26T18:54:07.603 INFO:tasks.workunit.client.1.smithi190.stdout:6/81: dwrite d17/f18 [0,4194304] 0 2024-03-26T18:54:07.608 INFO:tasks.workunit.client.1.smithi190.stdout:6/82: rename d17/f18 to d17/f19 0 2024-03-26T18:54:07.610 INFO:tasks.workunit.client.1.smithi190.stdout:6/83: link cd d17/c1a 0 2024-03-26T18:54:07.610 INFO:tasks.workunit.client.1.smithi190.stdout:6/84: chown fc 635 1 2024-03-26T18:54:07.610 INFO:tasks.workunit.client.1.smithi190.stdout:6/85: creat d17/f1b x:0 0 0 2024-03-26T18:54:07.615 INFO:tasks.workunit.client.1.smithi190.stdout:6/86: unlink f16 0 2024-03-26T18:54:07.615 INFO:tasks.workunit.client.1.smithi190.stdout:6/87: chown cd 155 1 2024-03-26T18:54:07.618 INFO:tasks.workunit.client.1.smithi190.stdout:6/88: mknod d17/c1c 0 2024-03-26T18:54:07.621 INFO:tasks.workunit.client.1.smithi190.stdout:6/89: mknod d17/c1d 0 2024-03-26T18:54:07.623 INFO:tasks.workunit.client.1.smithi190.stdout:6/90: mkdir d17/d1e 0 2024-03-26T18:54:07.623 INFO:tasks.workunit.client.1.smithi190.stdout:6/91: chown d17 78957 1 2024-03-26T18:54:07.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:07 smithi102 ceph-mon[29002]: pgmap v231: 65 pgs: 65 active+clean; 132 MiB data, 581 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s wr, 727 op/s 2024-03-26T18:54:07.625 INFO:tasks.workunit.client.1.smithi190.stdout:6/92: symlink d17/l1f 0 2024-03-26T18:54:07.645 INFO:tasks.workunit.client.1.smithi190.stdout:9/36: dwrite f9 [0,4194304] 0 2024-03-26T18:54:07.700 INFO:tasks.workunit.client.1.smithi190.stdout:0/77: truncate d0/f13 1462052 0 2024-03-26T18:54:07.700 INFO:tasks.workunit.client.1.smithi190.stdout:0/78: fsync d0/f5 0 2024-03-26T18:54:07.703 INFO:tasks.workunit.client.1.smithi190.stdout:0/79: mkdir d0/d19/d1c 0 2024-03-26T18:54:07.705 INFO:tasks.workunit.client.1.smithi190.stdout:0/80: mknod d0/dc/c1d 0 2024-03-26T18:54:07.705 INFO:tasks.workunit.client.1.smithi190.stdout:0/81: readlink d0/dd/lf 0 2024-03-26T18:54:07.724 INFO:tasks.workunit.client.1.smithi190.stdout:0/82: dwrite d0/dc/f15 [0,4194304] 0 2024-03-26T18:54:07.726 INFO:tasks.workunit.client.1.smithi190.stdout:0/83: rename d0/c14 to d0/dc/c1e 0 2024-03-26T18:54:07.726 INFO:tasks.workunit.client.1.smithi190.stdout:0/84: creat d0/f1f x:0 0 0 2024-03-26T18:54:07.726 INFO:tasks.workunit.client.1.smithi190.stdout:0/85: write d0/dd/f12 [2103715,55859] 0 2024-03-26T18:54:07.802 INFO:tasks.workunit.client.1.smithi190.stdout:6/93: getdents d17 0 2024-03-26T18:54:07.804 INFO:tasks.workunit.client.1.smithi190.stdout:6/94: symlink d17/l20 0 2024-03-26T18:54:07.808 INFO:tasks.workunit.client.1.smithi190.stdout:6/95: rmdir d17/d1e 0 2024-03-26T18:54:07.810 INFO:tasks.workunit.client.1.smithi190.stdout:6/96: truncate f9 2009783 0 2024-03-26T18:54:07.820 INFO:tasks.workunit.client.1.smithi190.stdout:9/37: truncate f9 3310466 0 2024-03-26T18:54:07.822 INFO:tasks.workunit.client.1.smithi190.stdout:9/38: dread fb [0,4194304] 0 2024-03-26T18:54:07.825 INFO:tasks.workunit.client.1.smithi190.stdout:9/39: symlink le 0 2024-03-26T18:54:07.829 INFO:tasks.workunit.client.1.smithi190.stdout:9/40: link fd ff 0 2024-03-26T18:54:07.901 INFO:tasks.workunit.client.1.smithi190.stdout:0/86: rmdir d0/dc 39 2024-03-26T18:54:07.924 INFO:tasks.workunit.client.1.smithi190.stdout:0/87: dwrite d0/f1f [0,4194304] 0 2024-03-26T18:54:07.924 INFO:tasks.workunit.client.1.smithi190.stdout:0/88: dread - d0/dd/fe zero size 2024-03-26T18:54:07.934 INFO:tasks.workunit.client.1.smithi190.stdout:0/89: dread d0/f6 [0,4194304] 0 2024-03-26T18:54:07.952 INFO:tasks.workunit.client.1.smithi190.stdout:0/90: dwrite d0/f6 [0,4194304] 0 2024-03-26T18:54:07.955 INFO:tasks.workunit.client.1.smithi190.stdout:0/91: rename d0/d19 to d0/dd/d20 0 2024-03-26T18:54:07.957 INFO:tasks.workunit.client.1.smithi190.stdout:0/92: symlink d0/dc/l21 0 2024-03-26T18:54:07.957 INFO:tasks.workunit.client.1.smithi190.stdout:0/93: creat d0/dd/f22 x:0 0 0 2024-03-26T18:54:07.967 INFO:tasks.workunit.client.1.smithi190.stdout:0/94: write d0/dc/f15 [608832,68189] 0 2024-03-26T18:54:07.973 INFO:tasks.workunit.client.1.smithi190.stdout:0/95: dread d0/f5 [0,4194304] 0 2024-03-26T18:54:07.973 INFO:tasks.workunit.client.1.smithi190.stdout:0/96: write d0/dc/f15 [1911436,106925] 0 2024-03-26T18:54:07.973 INFO:tasks.workunit.client.1.smithi190.stdout:0/97: readlink d0/dd/lf 0 2024-03-26T18:54:07.975 INFO:tasks.workunit.client.1.smithi190.stdout:0/98: symlink d0/dd/d20/d1c/l23 0 2024-03-26T18:54:07.977 INFO:tasks.workunit.client.1.smithi190.stdout:0/99: mkdir d0/dd/d24 0 2024-03-26T18:54:07.982 INFO:tasks.workunit.client.1.smithi190.stdout:0/100: dread d0/f5 [0,4194304] 0 2024-03-26T18:54:07.985 INFO:tasks.workunit.client.1.smithi190.stdout:0/101: unlink d0/l4 0 2024-03-26T18:54:07.986 INFO:tasks.workunit.client.1.smithi190.stdout:6/97: truncate fc 172364 0 2024-03-26T18:54:07.987 INFO:tasks.workunit.client.1.smithi190.stdout:0/102: rmdir d0/dc 39 2024-03-26T18:54:07.989 INFO:tasks.workunit.client.1.smithi190.stdout:6/98: chown d17/f19 0 1 2024-03-26T18:54:07.989 INFO:tasks.workunit.client.1.smithi190.stdout:6/99: truncate f2 3803315 0 2024-03-26T18:54:07.990 INFO:tasks.workunit.client.1.smithi190.stdout:0/103: symlink d0/dd/l25 0 2024-03-26T18:54:07.991 INFO:tasks.workunit.client.1.smithi190.stdout:6/100: dread - d17/f1b zero size 2024-03-26T18:54:07.993 INFO:tasks.workunit.client.1.smithi190.stdout:0/104: mkdir d0/dd/d26 0 2024-03-26T18:54:07.995 INFO:tasks.workunit.client.1.smithi190.stdout:0/105: mknod d0/dc/c27 0 2024-03-26T18:54:07.999 INFO:tasks.workunit.client.1.smithi190.stdout:0/106: creat d0/dd/d20/d1c/f28 x:0 0 0 2024-03-26T18:54:07.999 INFO:tasks.workunit.client.1.smithi190.stdout:6/101: dread d17/f19 [0,4194304] 0 2024-03-26T18:54:07.999 INFO:tasks.workunit.client.1.smithi190.stdout:6/102: fdatasync f1 0 2024-03-26T18:54:08.003 INFO:tasks.workunit.client.1.smithi190.stdout:6/103: creat d17/f21 x:0 0 0 2024-03-26T18:54:08.005 INFO:tasks.workunit.client.1.smithi190.stdout:0/107: read d0/f5 [485075,126126] 0 2024-03-26T18:54:08.005 INFO:tasks.workunit.client.1.smithi190.stdout:0/108: write d0/dd/f22 [751069,30582] 0 2024-03-26T18:54:08.006 INFO:tasks.workunit.client.1.smithi190.stdout:0/109: stat d0/cb 0 2024-03-26T18:54:08.006 INFO:tasks.workunit.client.1.smithi190.stdout:0/110: write d0/dd/fe [785750,52945] 0 2024-03-26T18:54:08.008 INFO:tasks.workunit.client.1.smithi190.stdout:9/41: dread f9 [0,4194304] 0 2024-03-26T18:54:08.009 INFO:tasks.workunit.client.1.smithi190.stdout:0/111: creat d0/dd/d20/d1c/f29 x:0 0 0 2024-03-26T18:54:08.009 INFO:tasks.workunit.client.1.smithi190.stdout:0/112: stat d0/dd/d20/d1c/f28 0 2024-03-26T18:54:08.009 INFO:tasks.workunit.client.1.smithi190.stdout:0/113: read d0/dd/f22 [103680,57013] 0 2024-03-26T18:54:08.010 INFO:tasks.workunit.client.1.smithi190.stdout:6/104: symlink d17/l22 0 2024-03-26T18:54:08.014 INFO:tasks.workunit.client.1.smithi190.stdout:6/105: creat d17/f23 x:0 0 0 2024-03-26T18:54:08.014 INFO:tasks.workunit.client.1.smithi190.stdout:6/106: truncate d17/f23 61838 0 2024-03-26T18:54:08.017 INFO:tasks.workunit.client.1.smithi190.stdout:9/42: link f8 f10 0 2024-03-26T18:54:08.017 INFO:tasks.workunit.client.1.smithi190.stdout:9/43: fsync f9 0 2024-03-26T18:54:08.028 INFO:tasks.workunit.client.1.smithi190.stdout:9/44: rename f9 to f11 0 2024-03-26T18:54:08.031 INFO:tasks.workunit.client.1.smithi190.stdout:9/45: mkdir d12 0 2024-03-26T18:54:08.031 INFO:tasks.workunit.client.1.smithi190.stdout:9/46: write fd [196489,90416] 0 2024-03-26T18:54:08.032 INFO:tasks.workunit.client.1.smithi190.stdout:9/47: read ff [163447,101461] 0 2024-03-26T18:54:08.034 INFO:tasks.workunit.client.1.smithi190.stdout:9/48: creat d12/f13 x:0 0 0 2024-03-26T18:54:08.034 INFO:tasks.workunit.client.1.smithi190.stdout:9/49: write d12/f13 [603559,118805] 0 2024-03-26T18:54:08.034 INFO:tasks.workunit.client.1.smithi190.stdout:9/50: creat d12/f14 x:0 0 0 2024-03-26T18:54:08.035 INFO:tasks.workunit.client.1.smithi190.stdout:9/51: creat d12/f15 x:0 0 0 2024-03-26T18:54:08.035 INFO:tasks.workunit.client.1.smithi190.stdout:9/52: chown d12 0 1 2024-03-26T18:54:08.035 INFO:tasks.workunit.client.1.smithi190.stdout:9/53: dread - f10 zero size 2024-03-26T18:54:08.035 INFO:tasks.workunit.client.1.smithi190.stdout:9/54: write f11 [4160352,86820] 0 2024-03-26T18:54:08.035 INFO:tasks.workunit.client.1.smithi190.stdout:9/55: readlink lc 0 2024-03-26T18:54:08.235 INFO:tasks.workunit.client.1.smithi190.stdout:9/56: fsync ff 0 2024-03-26T18:54:08.239 INFO:tasks.workunit.client.1.smithi190.stdout:0/114: dwrite d0/f5 [0,4194304] 0 2024-03-26T18:54:08.239 INFO:tasks.workunit.client.1.smithi190.stdout:0/115: readlink d0/dc/l21 0 2024-03-26T18:54:08.240 INFO:tasks.workunit.client.1.smithi190.stdout:9/57: rename fd to d12/f16 0 2024-03-26T18:54:08.240 INFO:tasks.workunit.client.1.smithi190.stdout:9/58: fsync f10 0 2024-03-26T18:54:08.242 INFO:tasks.workunit.client.1.smithi190.stdout:9/59: truncate f5 472807 0 2024-03-26T18:54:08.255 INFO:tasks.workunit.client.1.smithi190.stdout:0/116: dwrite d0/dd/f22 [0,4194304] 0 2024-03-26T18:54:08.260 INFO:tasks.workunit.client.1.smithi190.stdout:0/117: rename d0/dc/f15 to d0/dd/d24/f2a 0 2024-03-26T18:54:08.263 INFO:tasks.workunit.client.1.smithi190.stdout:0/118: creat d0/dd/d24/f2b x:0 0 0 2024-03-26T18:54:08.272 INFO:tasks.workunit.client.1.smithi190.stdout:0/119: write d0/f5 [3995666,79384] 0 2024-03-26T18:54:08.275 INFO:tasks.workunit.client.1.smithi190.stdout:0/120: symlink d0/dd/d20/d1c/l2c 0 2024-03-26T18:54:08.283 INFO:tasks.workunit.client.1.smithi190.stdout:0/121: dread d0/f5 [0,4194304] 0 2024-03-26T18:54:08.283 INFO:tasks.workunit.client.1.smithi190.stdout:0/122: chown d0 0 1 2024-03-26T18:54:08.301 INFO:tasks.workunit.client.1.smithi190.stdout:0/123: dwrite d0/dd/f22 [0,4194304] 0 2024-03-26T18:54:08.301 INFO:tasks.workunit.client.1.smithi190.stdout:0/124: write d0/dd/d20/d1c/f29 [192451,128167] 0 2024-03-26T18:54:08.301 INFO:tasks.workunit.client.1.smithi190.stdout:0/125: chown d0/f13 956195 1 2024-03-26T18:54:08.303 INFO:tasks.workunit.client.1.smithi190.stdout:0/126: mknod d0/dd/d24/c2d 0 2024-03-26T18:54:08.307 INFO:tasks.workunit.client.1.smithi190.stdout:0/127: rename d0/dd/c1a to d0/dd/d20/d1c/c2e 0 2024-03-26T18:54:08.307 INFO:tasks.workunit.client.1.smithi190.stdout:0/128: fsync d0/dd/fe 0 2024-03-26T18:54:08.309 INFO:tasks.workunit.client.1.smithi190.stdout:0/129: dread d0/dd/fe [0,4194304] 0 2024-03-26T18:54:08.309 INFO:tasks.workunit.client.1.smithi190.stdout:0/130: write d0/dd/f12 [3390147,85663] 0 2024-03-26T18:54:08.309 INFO:tasks.workunit.client.1.smithi190.stdout:0/131: fdatasync d0/dd/d20/d1c/f28 0 2024-03-26T18:54:08.309 INFO:tasks.workunit.client.1.smithi190.stdout:0/132: creat d0/dd/d24/f2f x:0 0 0 2024-03-26T18:54:08.436 INFO:tasks.workunit.client.1.smithi190.stdout:9/60: getdents d12 0 2024-03-26T18:54:08.493 INFO:tasks.workunit.client.1.smithi190.stdout:0/133: getdents d0/dd/d24 0 2024-03-26T18:54:08.493 INFO:tasks.workunit.client.1.smithi190.stdout:0/134: readlink d0/dd/lf 0 2024-03-26T18:54:08.622 INFO:tasks.workunit.client.1.smithi190.stdout:0/135: truncate d0/dd/d20/d1c/f29 317240 0 2024-03-26T18:54:08.624 INFO:tasks.workunit.client.1.smithi190.stdout:0/136: creat d0/dd/d20/f30 x:0 0 0 2024-03-26T18:54:09.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:09 smithi190 ceph-mon[35043]: pgmap v232: 65 pgs: 65 active+clean; 132 MiB data, 581 MiB used, 536 GiB / 536 GiB avail; 7.0 MiB/s wr, 406 op/s 2024-03-26T18:54:09.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:09 smithi102 ceph-mon[29002]: pgmap v232: 65 pgs: 65 active+clean; 132 MiB data, 581 MiB used, 536 GiB / 536 GiB avail; 7.0 MiB/s wr, 406 op/s 2024-03-26T18:54:11.335 INFO:tasks.workunit.client.1.smithi190.stdout:8/10: sync 2024-03-26T18:54:11.338 INFO:tasks.workunit.client.1.smithi190.stdout:8/11: mknod c2 0 2024-03-26T18:54:11.338 INFO:tasks.workunit.client.1.smithi190.stdout:8/12: write - no filename 2024-03-26T18:54:11.340 INFO:tasks.workunit.client.1.smithi190.stdout:8/13: symlink l3 0 2024-03-26T18:54:11.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:11 smithi190 ceph-mon[35043]: pgmap v233: 65 pgs: 65 active+clean; 191 MiB data, 827 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s rd, 16 MiB/s wr, 578 op/s 2024-03-26T18:54:11.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:11 smithi102 ceph-mon[29002]: pgmap v233: 65 pgs: 65 active+clean; 191 MiB data, 827 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s rd, 16 MiB/s wr, 578 op/s 2024-03-26T18:54:13.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:13 smithi190 ceph-mon[35043]: pgmap v234: 65 pgs: 65 active+clean; 228 MiB data, 929 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 20 MiB/s wr, 635 op/s 2024-03-26T18:54:13.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:13 smithi102 ceph-mon[29002]: pgmap v234: 65 pgs: 65 active+clean; 228 MiB data, 929 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 20 MiB/s wr, 635 op/s 2024-03-26T18:54:15.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:15 smithi190 ceph-mon[35043]: pgmap v235: 65 pgs: 65 active+clean; 228 MiB data, 971 MiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 21 MiB/s wr, 733 op/s 2024-03-26T18:54:15.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:15 smithi102 ceph-mon[29002]: pgmap v235: 65 pgs: 65 active+clean; 228 MiB data, 971 MiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 21 MiB/s wr, 733 op/s 2024-03-26T18:54:15.711 INFO:tasks.workunit.client.1.smithi190.stdout:2/20: sync 2024-03-26T18:54:15.711 INFO:tasks.workunit.client.1.smithi190.stdout:2/21: rmdir - no directory 2024-03-26T18:54:15.711 INFO:tasks.workunit.client.1.smithi190.stdout:2/22: creat f5 x:0 0 0 2024-03-26T18:54:15.715 INFO:tasks.workunit.client.1.smithi190.stdout:2/23: mkdir d6 0 2024-03-26T18:54:15.731 INFO:tasks.workunit.client.1.smithi190.stdout:2/24: dwrite f4 [0,4194304] 0 2024-03-26T18:54:15.731 INFO:tasks.workunit.client.1.smithi190.stdout:2/25: readlink l0 0 2024-03-26T18:54:15.739 INFO:tasks.workunit.client.1.smithi190.stdout:2/26: creat d6/f7 x:0 0 0 2024-03-26T18:54:15.742 INFO:tasks.workunit.client.1.smithi190.stdout:2/27: unlink f3 0 2024-03-26T18:54:15.758 INFO:tasks.workunit.client.1.smithi190.stdout:2/28: dwrite f4 [0,4194304] 0 2024-03-26T18:54:15.760 INFO:tasks.workunit.client.1.smithi190.stdout:2/29: mkdir d6/d8 0 2024-03-26T18:54:15.763 INFO:tasks.workunit.client.1.smithi190.stdout:2/30: unlink l0 0 2024-03-26T18:54:15.765 INFO:tasks.workunit.client.1.smithi190.stdout:2/31: mkdir d6/d9 0 2024-03-26T18:54:15.783 INFO:tasks.workunit.client.1.smithi190.stdout:2/32: dwrite f5 [0,4194304] 0 2024-03-26T18:54:15.785 INFO:tasks.workunit.client.1.smithi190.stdout:2/33: creat d6/fa x:0 0 0 2024-03-26T18:54:17.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:17 smithi190 ceph-mon[35043]: pgmap v236: 65 pgs: 65 active+clean; 254 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 21 MiB/s wr, 893 op/s 2024-03-26T18:54:17.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:17 smithi102 ceph-mon[29002]: pgmap v236: 65 pgs: 65 active+clean; 254 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 21 MiB/s wr, 893 op/s 2024-03-26T18:54:19.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:19 smithi190 ceph-mon[35043]: pgmap v237: 65 pgs: 65 active+clean; 254 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 18 MiB/s wr, 573 op/s 2024-03-26T18:54:19.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:19 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:54:19.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:19 smithi102 ceph-mon[29002]: pgmap v237: 65 pgs: 65 active+clean; 254 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 18 MiB/s wr, 573 op/s 2024-03-26T18:54:19.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:19 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:54:19.925 INFO:tasks.workunit.client.0.smithi102.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-26T18:54:19.925 INFO:tasks.workunit.client.0.smithi102.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-26T18:54:19.926 INFO:tasks.workunit.client.0.smithi102.stderr:+ make 2024-03-26T18:54:19.973 INFO:tasks.workunit.client.0.smithi102.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-26T18:54:20.271 INFO:tasks.workunit.client.1.smithi190.stdout:4/29: sync 2024-03-26T18:54:20.284 INFO:tasks.workunit.client.1.smithi190.stdout:4/30: mkdir d7/d8 0 2024-03-26T18:54:20.284 INFO:tasks.workunit.client.1.smithi190.stdout:4/31: dread - no filename 2024-03-26T18:54:20.289 INFO:tasks.workunit.client.1.smithi190.stdout:4/32: creat d7/f9 x:0 0 0 2024-03-26T18:54:20.477 INFO:tasks.workunit.client.1.smithi190.stdout:4/33: fsync d7/f9 0 2024-03-26T18:54:20.477 INFO:tasks.workunit.client.1.smithi190.stdout:4/34: creat d7/fa x:0 0 0 2024-03-26T18:54:20.480 INFO:tasks.workunit.client.1.smithi190.stdout:4/35: symlink d7/lb 0 2024-03-26T18:54:20.480 INFO:tasks.workunit.client.1.smithi190.stdout:4/36: creat d7/fc x:0 0 0 2024-03-26T18:54:20.483 INFO:tasks.workunit.client.1.smithi190.stdout:4/37: symlink d7/d8/ld 0 2024-03-26T18:54:20.484 INFO:tasks.workunit.client.1.smithi190.stdout:4/38: dread - d7/f9 zero size 2024-03-26T18:54:20.484 INFO:tasks.workunit.client.1.smithi190.stdout:4/39: truncate d7/fa 611680 0 2024-03-26T18:54:20.484 INFO:tasks.workunit.client.1.smithi190.stdout:4/40: chown d7 43377538 1 2024-03-26T18:54:20.485 INFO:tasks.workunit.client.1.smithi190.stdout:4/41: rmdir d7/d8 39 2024-03-26T18:54:20.487 INFO:tasks.workunit.client.1.smithi190.stdout:3/59: sync 2024-03-26T18:54:20.487 INFO:tasks.workunit.client.1.smithi190.stdout:8/14: sync 2024-03-26T18:54:20.487 INFO:tasks.workunit.client.1.smithi190.stdout:3/60: fdatasync d1/f15 0 2024-03-26T18:54:20.487 INFO:tasks.workunit.client.1.smithi190.stdout:7/49: sync 2024-03-26T18:54:20.487 INFO:tasks.workunit.client.1.smithi190.stdout:6/107: sync 2024-03-26T18:54:20.487 INFO:tasks.workunit.client.1.smithi190.stdout:1/43: sync 2024-03-26T18:54:20.487 INFO:tasks.workunit.client.1.smithi190.stdout:0/137: sync 2024-03-26T18:54:20.488 INFO:tasks.workunit.client.1.smithi190.stdout:5/27: sync 2024-03-26T18:54:20.488 INFO:tasks.workunit.client.1.smithi190.stdout:9/61: sync 2024-03-26T18:54:20.488 INFO:tasks.workunit.client.1.smithi190.stdout:2/34: sync 2024-03-26T18:54:20.488 INFO:tasks.workunit.client.1.smithi190.stdout:4/42: creat d7/d8/fe x:0 0 0 2024-03-26T18:54:20.488 INFO:tasks.workunit.client.1.smithi190.stdout:4/43: fdatasync d7/fa 0 2024-03-26T18:54:20.488 INFO:tasks.workunit.client.1.smithi190.stdout:3/61: dread d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:20.489 INFO:tasks.workunit.client.1.smithi190.stdout:8/15: symlink l4 0 2024-03-26T18:54:20.489 INFO:tasks.workunit.client.1.smithi190.stdout:8/16: write - no filename 2024-03-26T18:54:20.489 INFO:tasks.workunit.client.1.smithi190.stdout:8/17: dread - no filename 2024-03-26T18:54:20.492 INFO:tasks.workunit.client.1.smithi190.stdout:7/50: mkdir d3/d1b 0 2024-03-26T18:54:20.494 INFO:tasks.workunit.client.1.smithi190.stdout:5/28: symlink d2/l8 0 2024-03-26T18:54:20.494 INFO:tasks.workunit.client.1.smithi190.stdout:6/108: dread d17/f19 [0,4194304] 0 2024-03-26T18:54:20.494 INFO:tasks.workunit.client.1.smithi190.stdout:6/109: truncate f9 3018353 0 2024-03-26T18:54:20.496 INFO:tasks.workunit.client.1.smithi190.stdout:9/62: creat d12/f17 x:0 0 0 2024-03-26T18:54:20.497 INFO:tasks.workunit.client.1.smithi190.stdout:9/63: creat d12/f18 x:0 0 0 2024-03-26T18:54:20.499 INFO:tasks.workunit.client.1.smithi190.stdout:2/35: unlink d6/fa 0 2024-03-26T18:54:20.502 INFO:tasks.workunit.client.1.smithi190.stdout:4/44: symlink d7/lf 0 2024-03-26T18:54:20.502 INFO:tasks.workunit.client.1.smithi190.stdout:4/45: creat d7/f10 x:0 0 0 2024-03-26T18:54:20.504 INFO:tasks.workunit.client.1.smithi190.stdout:3/62: mkdir d1/d3/df/d18 0 2024-03-26T18:54:20.507 INFO:tasks.workunit.client.1.smithi190.stdout:8/18: link l4 l5 0 2024-03-26T18:54:20.508 INFO:tasks.workunit.client.1.smithi190.stdout:8/19: chown l5 1126538560 1 2024-03-26T18:54:20.508 INFO:tasks.workunit.client.1.smithi190.stdout:8/20: dread - no filename 2024-03-26T18:54:20.510 INFO:tasks.workunit.client.1.smithi190.stdout:5/29: write d2/f4 [1292898,2944] 0 2024-03-26T18:54:20.510 INFO:tasks.workunit.client.1.smithi190.stdout:7/51: creat d3/f1c x:0 0 0 2024-03-26T18:54:20.510 INFO:tasks.workunit.client.1.smithi190.stdout:7/52: fsync d3/d6/fd 0 2024-03-26T18:54:20.511 INFO:tasks.workunit.client.1.smithi190.stdout:7/53: write d3/fc [695393,88874] 0 2024-03-26T18:54:20.512 INFO:tasks.workunit.client.1.smithi190.stdout:6/110: creat d17/f24 x:0 0 0 2024-03-26T18:54:20.512 INFO:tasks.workunit.client.1.smithi190.stdout:0/138: dwrite d0/dd/d24/f2f [0,4194304] 0 2024-03-26T18:54:20.517 INFO:tasks.workunit.client.1.smithi190.stdout:0/139: dread d0/f13 [0,4194304] 0 2024-03-26T18:54:20.519 INFO:tasks.workunit.client.1.smithi190.stdout:2/36: dwrite f4 [0,4194304] 0 2024-03-26T18:54:20.519 INFO:tasks.workunit.client.1.smithi190.stdout:2/37: chown d6 0 1 2024-03-26T18:54:20.519 INFO:tasks.workunit.client.1.smithi190.stdout:4/46: rename c5 to d7/d8/c11 0 2024-03-26T18:54:20.519 INFO:tasks.workunit.client.1.smithi190.stdout:4/47: write d7/f10 [1007303,27793] 0 2024-03-26T18:54:20.521 INFO:tasks.workunit.client.1.smithi190.stdout:3/63: creat d1/d3/df/f19 x:0 0 0 2024-03-26T18:54:20.523 INFO:tasks.workunit.client.1.smithi190.stdout:1/44: dwrite f2 [0,4194304] 0 2024-03-26T18:54:20.524 INFO:tasks.workunit.client.1.smithi190.stdout:8/21: creat f6 x:0 0 0 2024-03-26T18:54:20.524 INFO:tasks.workunit.client.1.smithi190.stdout:8/22: creat f7 x:0 0 0 2024-03-26T18:54:20.524 INFO:tasks.workunit.client.1.smithi190.stdout:8/23: creat f8 x:0 0 0 2024-03-26T18:54:20.524 INFO:tasks.workunit.client.1.smithi190.stdout:8/24: readlink l4 0 2024-03-26T18:54:20.531 INFO:tasks.workunit.client.1.smithi190.stdout:6/111: truncate d17/f19 3125990 0 2024-03-26T18:54:20.532 INFO:tasks.workunit.client.1.smithi190.stdout:9/64: dwrite ff [0,4194304] 0 2024-03-26T18:54:20.535 INFO:tasks.workunit.client.1.smithi190.stdout:0/140: link d0/dd/d20/d1c/l23 d0/dc/l31 0 2024-03-26T18:54:20.535 INFO:tasks.workunit.client.1.smithi190.stdout:0/141: chown d0/dd/d20/d1c/l2c 353435 1 2024-03-26T18:54:20.537 INFO:tasks.workunit.client.1.smithi190.stdout:7/54: dwrite d3/f1c [0,4194304] 0 2024-03-26T18:54:20.537 INFO:tasks.workunit.client.1.smithi190.stdout:2/38: symlink d6/d9/lb 0 2024-03-26T18:54:20.539 INFO:tasks.workunit.client.1.smithi190.stdout:4/48: rename d7/d8/fe to d7/d8/f12 0 2024-03-26T18:54:20.539 INFO:tasks.workunit.client.1.smithi190.stdout:9/65: dread f11 [0,4194304] 0 2024-03-26T18:54:20.541 INFO:tasks.workunit.client.1.smithi190.stdout:3/64: mknod d1/de/c1a 0 2024-03-26T18:54:20.541 INFO:tasks.workunit.client.1.smithi190.stdout:3/65: readlink d1/d3/l17 0 2024-03-26T18:54:20.543 INFO:tasks.workunit.client.1.smithi190.stdout:2/39: dread f4 [0,4194304] 0 2024-03-26T18:54:20.544 INFO:tasks.workunit.client.1.smithi190.stdout:3/66: dread d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:20.547 INFO:tasks.workunit.client.1.smithi190.stdout:8/25: mkdir d9 0 2024-03-26T18:54:20.551 INFO:tasks.workunit.client.1.smithi190.stdout:6/112: creat d17/f25 x:0 0 0 2024-03-26T18:54:20.559 INFO:tasks.workunit.client.1.smithi190.stdout:0/142: truncate d0/f6 402782 0 2024-03-26T18:54:20.564 INFO:tasks.workunit.client.1.smithi190.stdout:7/55: unlink d3/d6/d9/c13 0 2024-03-26T18:54:20.564 INFO:tasks.workunit.client.1.smithi190.stdout:8/26: dwrite f7 [0,4194304] 0 2024-03-26T18:54:20.566 INFO:tasks.workunit.client.1.smithi190.stdout:4/49: unlink d7/fa 0 2024-03-26T18:54:20.566 INFO:tasks.workunit.client.1.smithi190.stdout:4/50: creat d7/f13 x:0 0 0 2024-03-26T18:54:20.569 INFO:tasks.workunit.client.1.smithi190.stdout:9/66: mkdir d12/d19 0 2024-03-26T18:54:20.569 INFO:tasks.workunit.client.1.smithi190.stdout:9/67: truncate d12/f13 1117401 0 2024-03-26T18:54:20.573 INFO:tasks.workunit.client.1.smithi190.stdout:2/40: creat d6/d9/fc x:0 0 0 2024-03-26T18:54:20.573 INFO:tasks.workunit.client.1.smithi190.stdout:2/41: getdents d6/d8 0 2024-03-26T18:54:20.575 INFO:tasks.workunit.client.1.smithi190.stdout:3/67: mkdir d1/d3/df/d18/d1b 0 2024-03-26T18:54:20.575 INFO:tasks.workunit.client.1.smithi190.stdout:3/68: dread - d1/d3/df/f19 zero size 2024-03-26T18:54:20.576 INFO:tasks.workunit.client.1.smithi190.stdout:7/56: dwrite d3/d6/fd [0,4194304] 0 2024-03-26T18:54:20.577 INFO:tasks.workunit.client.1.smithi190.stdout:7/57: write d3/d6/d9/d14/f18 [442027,61018] 0 2024-03-26T18:54:20.579 INFO:tasks.workunit.client.1.smithi190.stdout:0/143: rename d0/cb to d0/dd/c32 0 2024-03-26T18:54:20.585 INFO:tasks.workunit.client.1.smithi190.stdout:4/51: mkdir d7/d8/d14 0 2024-03-26T18:54:20.587 INFO:tasks.workunit.client.1.smithi190.stdout:9/68: unlink lc 0 2024-03-26T18:54:20.588 INFO:tasks.workunit.client.1.smithi190.stdout:9/69: write f10 [622855,117987] 0 2024-03-26T18:54:20.590 INFO:tasks.workunit.client.1.smithi190.stdout:3/69: mkdir d1/de/d1c 0 2024-03-26T18:54:20.592 INFO:tasks.workunit.client.1.smithi190.stdout:6/113: rename d17/l22 to d17/l26 0 2024-03-26T18:54:20.596 INFO:tasks.workunit.client.1.smithi190.stdout:9/70: fsync f8 0 2024-03-26T18:54:20.597 INFO:tasks.workunit.client.1.smithi190.stdout:7/58: link d3/f1c d3/d6/f1d 0 2024-03-26T18:54:20.599 INFO:tasks.workunit.client.1.smithi190.stdout:0/144: dwrite d0/dd/f12 [0,4194304] 0 2024-03-26T18:54:20.599 INFO:tasks.workunit.client.1.smithi190.stdout:2/42: rename f4 to d6/d9/fd 0 2024-03-26T18:54:20.600 INFO:tasks.workunit.client.1.smithi190.stdout:4/52: mknod d7/d8/d14/c15 0 2024-03-26T18:54:20.601 INFO:tasks.workunit.client.1.smithi190.stdout:6/114: creat d17/f27 x:0 0 0 2024-03-26T18:54:20.605 INFO:tasks.workunit.client.1.smithi190.stdout:9/71: symlink d12/d19/l1a 0 2024-03-26T18:54:20.606 INFO:tasks.workunit.client.1.smithi190.stdout:7/59: rmdir d3/d6 39 2024-03-26T18:54:20.606 INFO:tasks.workunit.client.1.smithi190.stdout:7/60: creat d3/d6/d9/d14/f1e x:0 0 0 2024-03-26T18:54:20.608 INFO:tasks.workunit.client.1.smithi190.stdout:0/145: mknod d0/dd/d26/c33 0 2024-03-26T18:54:20.608 INFO:tasks.workunit.client.1.smithi190.stdout:0/146: fdatasync d0/dd/d20/d1c/f28 0 2024-03-26T18:54:20.609 INFO:tasks.workunit.client.1.smithi190.stdout:2/43: write f5 [4042054,107452] 0 2024-03-26T18:54:20.611 INFO:tasks.workunit.client.1.smithi190.stdout:9/72: creat d12/d19/f1b x:0 0 0 2024-03-26T18:54:20.614 INFO:tasks.workunit.client.1.smithi190.stdout:0/147: creat d0/dd/d26/f34 x:0 0 0 2024-03-26T18:54:20.621 INFO:tasks.workunit.client.1.smithi190.stdout:0/148: symlink d0/dc/l35 0 2024-03-26T18:54:20.621 INFO:tasks.workunit.client.1.smithi190.stdout:0/149: read d0/f5 [3994479,118459] 0 2024-03-26T18:54:20.621 INFO:tasks.workunit.client.1.smithi190.stdout:0/150: fsync d0/f5 0 2024-03-26T18:54:20.621 INFO:tasks.workunit.client.1.smithi190.stdout:0/151: truncate d0/f1f 4782185 0 2024-03-26T18:54:20.621 INFO:tasks.workunit.client.1.smithi190.stdout:0/152: write d0/dd/d26/f34 [627880,41349] 0 2024-03-26T18:54:20.623 INFO:tasks.workunit.client.1.smithi190.stdout:0/153: rename d0/f5 to d0/dd/d26/f36 0 2024-03-26T18:54:20.625 INFO:tasks.workunit.client.1.smithi190.stdout:9/73: dwrite f6 [0,4194304] 0 2024-03-26T18:54:20.633 INFO:tasks.workunit.client.1.smithi190.stdout:9/74: dread f5 [0,4194304] 0 2024-03-26T18:54:20.633 INFO:tasks.workunit.client.1.smithi190.stdout:9/75: fdatasync f5 0 2024-03-26T18:54:20.636 INFO:tasks.workunit.client.1.smithi190.stdout:9/76: rename d12/f17 to d12/f1c 0 2024-03-26T18:54:20.639 INFO:tasks.workunit.client.1.smithi190.stdout:9/77: read f6 [2772947,56791] 0 2024-03-26T18:54:20.640 INFO:tasks.workunit.client.1.smithi190.stdout:9/78: mkdir d12/d19/d1d 0 2024-03-26T18:54:20.640 INFO:tasks.workunit.client.1.smithi190.stdout:9/79: rename d12/d19 to d12/d19/d1d/d1e 22 2024-03-26T18:54:20.644 INFO:tasks.workunit.client.1.smithi190.stdout:9/80: link d12/f16 d12/f1f 0 2024-03-26T18:54:20.644 INFO:tasks.workunit.client.1.smithi190.stdout:9/81: readlink l4 0 2024-03-26T18:54:20.661 INFO:tasks.workunit.client.1.smithi190.stdout:9/82: dwrite d12/f13 [0,4194304] 0 2024-03-26T18:54:20.666 INFO:tasks.workunit.client.1.smithi190.stdout:9/83: link d12/f18 d12/d19/d1d/f20 0 2024-03-26T18:54:20.666 INFO:tasks.workunit.client.1.smithi190.stdout:9/84: fsync f8 0 2024-03-26T18:54:20.694 INFO:tasks.workunit.client.1.smithi190.stdout:5/30: truncate d2/f6 1854524 0 2024-03-26T18:54:20.706 INFO:tasks.workunit.client.1.smithi190.stdout:4/53: fsync d7/d8/f12 0 2024-03-26T18:54:20.706 INFO:tasks.workunit.client.1.smithi190.stdout:4/54: read - d7/fc zero size 2024-03-26T18:54:20.706 INFO:tasks.workunit.client.1.smithi190.stdout:4/55: dread - d7/d8/f12 zero size 2024-03-26T18:54:20.706 INFO:tasks.workunit.client.1.smithi190.stdout:4/56: fdatasync d7/d8/f12 0 2024-03-26T18:54:20.706 INFO:tasks.workunit.client.1.smithi190.stdout:4/57: write d7/f10 [877971,102354] 0 2024-03-26T18:54:20.727 INFO:tasks.workunit.client.1.smithi190.stdout:9/85: truncate d12/f16 496590 0 2024-03-26T18:54:20.730 INFO:tasks.workunit.client.1.smithi190.stdout:9/86: symlink d12/d19/l21 0 2024-03-26T18:54:20.734 INFO:tasks.workunit.client.1.smithi190.stdout:9/87: rename d12/f1f to d12/d19/f22 0 2024-03-26T18:54:20.736 INFO:tasks.workunit.client.1.smithi190.stdout:9/88: mkdir d12/d19/d1d/d23 0 2024-03-26T18:54:20.736 INFO:tasks.workunit.client.1.smithi190.stdout:9/89: creat d12/f24 x:0 0 0 2024-03-26T18:54:20.736 INFO:tasks.workunit.client.1.smithi190.stdout:9/90: creat d12/f25 x:0 0 0 2024-03-26T18:54:20.749 INFO:tasks.workunit.client.1.smithi190.stdout:0/154: getdents d0/dd 0 2024-03-26T18:54:20.752 INFO:tasks.workunit.client.1.smithi190.stdout:0/155: link d0/dc/c1e d0/dd/d26/c37 0 2024-03-26T18:54:20.754 INFO:tasks.workunit.client.1.smithi190.stdout:0/156: mknod d0/dc/c38 0 2024-03-26T18:54:20.754 INFO:tasks.workunit.client.1.smithi190.stdout:9/91: dwrite d12/f1c [0,4194304] 0 2024-03-26T18:54:20.754 INFO:tasks.workunit.client.1.smithi190.stdout:0/157: write d0/dd/d26/f36 [5238296,106416] 0 2024-03-26T18:54:20.760 INFO:tasks.workunit.client.1.smithi190.stdout:9/92: symlink d12/d19/l26 0 2024-03-26T18:54:20.760 INFO:tasks.workunit.client.1.smithi190.stdout:9/93: dread - d12/f25 zero size 2024-03-26T18:54:20.760 INFO:tasks.workunit.client.1.smithi190.stdout:9/94: chown l4 7 1 2024-03-26T18:54:20.760 INFO:tasks.workunit.client.1.smithi190.stdout:9/95: chown d12/d19/d1d/f20 12062 1 2024-03-26T18:54:20.760 INFO:tasks.workunit.client.1.smithi190.stdout:9/96: creat d12/d19/f27 x:0 0 0 2024-03-26T18:54:20.760 INFO:tasks.workunit.client.1.smithi190.stdout:9/97: creat d12/f28 x:0 0 0 2024-03-26T18:54:20.767 INFO:tasks.workunit.client.1.smithi190.stdout:9/98: symlink d12/d19/l29 0 2024-03-26T18:54:20.767 INFO:tasks.workunit.client.1.smithi190.stdout:9/99: chown f11 311 1 2024-03-26T18:54:20.769 INFO:tasks.workunit.client.1.smithi190.stdout:0/158: dwrite d0/dd/d24/f2b [0,4194304] 0 2024-03-26T18:54:20.769 INFO:tasks.workunit.client.1.smithi190.stdout:0/159: read - d0/dd/d20/d1c/f28 zero size 2024-03-26T18:54:20.770 INFO:tasks.workunit.client.1.smithi190.stdout:2/44: rmdir d6/d9 39 2024-03-26T18:54:20.772 INFO:tasks.workunit.client.1.smithi190.stdout:4/58: getdents d7/d8/d14 0 2024-03-26T18:54:20.774 INFO:tasks.workunit.client.1.smithi190.stdout:9/100: dread f11 [0,4194304] 0 2024-03-26T18:54:20.774 INFO:tasks.workunit.client.1.smithi190.stdout:9/101: fdatasync f5 0 2024-03-26T18:54:20.774 INFO:tasks.workunit.client.1.smithi190.stdout:6/115: getdents d17 0 2024-03-26T18:54:20.779 INFO:tasks.workunit.client.1.smithi190.stdout:4/59: mknod d7/d8/d14/c16 0 2024-03-26T18:54:20.779 INFO:tasks.workunit.client.1.smithi190.stdout:4/60: creat d7/f17 x:0 0 0 2024-03-26T18:54:20.779 INFO:tasks.workunit.client.1.smithi190.stdout:4/61: dread - d7/f9 zero size 2024-03-26T18:54:20.779 INFO:tasks.workunit.client.1.smithi190.stdout:4/62: write d7/f17 [902477,109949] 0 2024-03-26T18:54:20.784 INFO:tasks.workunit.client.1.smithi190.stdout:6/116: mkdir d17/d28 0 2024-03-26T18:54:20.784 INFO:tasks.workunit.client.1.smithi190.stdout:9/102: mkdir d12/d19/d1d/d23/d2a 0 2024-03-26T18:54:20.789 INFO:tasks.workunit.client.1.smithi190.stdout:9/103: mkdir d12/d19/d1d/d23/d2b 0 2024-03-26T18:54:20.790 INFO:tasks.workunit.client.1.smithi190.stdout:9/104: readlink d12/d19/l21 0 2024-03-26T18:54:20.790 INFO:tasks.workunit.client.1.smithi190.stdout:9/105: read f5 [2624713,70915] 0 2024-03-26T18:54:20.791 INFO:tasks.workunit.client.1.smithi190.stdout:6/117: symlink d17/l29 0 2024-03-26T18:54:20.792 INFO:tasks.workunit.client.1.smithi190.stdout:9/106: symlink d12/d19/l2c 0 2024-03-26T18:54:20.792 INFO:tasks.workunit.client.1.smithi190.stdout:9/107: readlink d12/d19/l21 0 2024-03-26T18:54:20.798 INFO:tasks.workunit.client.1.smithi190.stdout:2/45: dwrite d6/d9/fc [0,4194304] 0 2024-03-26T18:54:20.798 INFO:tasks.workunit.client.1.smithi190.stdout:4/63: dwrite d7/f9 [0,4194304] 0 2024-03-26T18:54:20.798 INFO:tasks.workunit.client.1.smithi190.stdout:2/46: creat d6/fe x:0 0 0 2024-03-26T18:54:20.798 INFO:tasks.workunit.client.1.smithi190.stdout:4/64: creat d7/f18 x:0 0 0 2024-03-26T18:54:20.810 INFO:tasks.workunit.client.1.smithi190.stdout:6/118: dwrite d17/f21 [0,4194304] 0 2024-03-26T18:54:20.810 INFO:tasks.workunit.client.1.smithi190.stdout:6/119: write f1 [870770,48824] 0 2024-03-26T18:54:20.810 INFO:tasks.workunit.client.1.smithi190.stdout:2/47: mkdir d6/df 0 2024-03-26T18:54:20.812 INFO:tasks.workunit.client.1.smithi190.stdout:4/65: creat d7/d8/d14/f19 x:0 0 0 2024-03-26T18:54:20.812 INFO:tasks.workunit.client.1.smithi190.stdout:4/66: chown d7/lf 124207806 1 2024-03-26T18:54:20.817 INFO:tasks.workunit.client.1.smithi190.stdout:4/67: dread d7/f10 [0,4194304] 0 2024-03-26T18:54:20.817 INFO:tasks.workunit.client.1.smithi190.stdout:4/68: creat d7/d8/d14/f1a x:0 0 0 2024-03-26T18:54:20.817 INFO:tasks.workunit.client.1.smithi190.stdout:4/69: fsync d7/fc 0 2024-03-26T18:54:20.818 INFO:tasks.workunit.client.1.smithi190.stdout:2/48: write d6/d9/fc [959545,118765] 0 2024-03-26T18:54:20.823 INFO:tasks.workunit.client.1.smithi190.stdout:2/49: unlink d6/d9/fc 0 2024-03-26T18:54:20.834 INFO:tasks.workunit.client.1.smithi190.stdout:4/70: dwrite d7/d8/d14/f19 [0,4194304] 0 2024-03-26T18:54:20.871 INFO:tasks.workunit.client.1.smithi190.stdout:5/31: write d2/f4 [371480,19577] 0 2024-03-26T18:54:20.891 INFO:tasks.workunit.client.1.smithi190.stdout:5/32: dwrite d2/f6 [0,4194304] 0 2024-03-26T18:54:20.902 INFO:tasks.workunit.client.1.smithi190.stdout:5/33: dread d2/f6 [0,4194304] 0 2024-03-26T18:54:20.905 INFO:tasks.workunit.client.1.smithi190.stdout:8/27: sync 2024-03-26T18:54:20.905 INFO:tasks.workunit.client.1.smithi190.stdout:1/45: sync 2024-03-26T18:54:20.905 INFO:tasks.workunit.client.1.smithi190.stdout:7/61: sync 2024-03-26T18:54:20.905 INFO:tasks.workunit.client.1.smithi190.stdout:1/46: chown d3/da 1 1 2024-03-26T18:54:20.905 INFO:tasks.workunit.client.1.smithi190.stdout:3/70: sync 2024-03-26T18:54:20.905 INFO:tasks.workunit.client.1.smithi190.stdout:8/28: getdents d9 0 2024-03-26T18:54:20.905 INFO:tasks.workunit.client.1.smithi190.stdout:8/29: chown l1 7 1 2024-03-26T18:54:20.905 INFO:tasks.workunit.client.1.smithi190.stdout:8/30: truncate f8 361802 0 2024-03-26T18:54:20.910 INFO:tasks.workunit.client.1.smithi190.stdout:1/47: rmdir d3/d9/db 39 2024-03-26T18:54:20.913 INFO:tasks.workunit.client.1.smithi190.stdout:7/62: mkdir d3/d6/d9/d1f 0 2024-03-26T18:54:20.913 INFO:tasks.workunit.client.1.smithi190.stdout:7/63: stat d3/d6 0 2024-03-26T18:54:20.915 INFO:tasks.workunit.client.1.smithi190.stdout:8/31: creat d9/fa x:0 0 0 2024-03-26T18:54:20.915 INFO:tasks.workunit.client.1.smithi190.stdout:8/32: fdatasync d9/fa 0 2024-03-26T18:54:20.919 INFO:tasks.workunit.client.1.smithi190.stdout:0/160: dread d0/f6 [0,4194304] 0 2024-03-26T18:54:20.919 INFO:tasks.workunit.client.1.smithi190.stdout:0/161: chown d0/dd/c32 1 1 2024-03-26T18:54:20.919 INFO:tasks.workunit.client.1.smithi190.stdout:8/33: symlink d9/lb 0 2024-03-26T18:54:20.920 INFO:tasks.workunit.client.1.smithi190.stdout:8/34: truncate f6 600634 0 2024-03-26T18:54:20.920 INFO:tasks.workunit.client.1.smithi190.stdout:8/35: chown f7 49230 1 2024-03-26T18:54:20.920 INFO:tasks.workunit.client.1.smithi190.stdout:8/36: dread - d9/fa zero size 2024-03-26T18:54:20.924 INFO:tasks.workunit.client.1.smithi190.stdout:1/48: dread f2 [0,4194304] 0 2024-03-26T18:54:20.926 INFO:tasks.workunit.client.1.smithi190.stdout:7/64: dread d3/d6/fd [0,4194304] 0 2024-03-26T18:54:20.926 INFO:tasks.workunit.client.1.smithi190.stdout:3/71: dwrite d1/fb [0,4194304] 0 2024-03-26T18:54:20.929 INFO:tasks.workunit.client.1.smithi190.stdout:1/49: creat d3/d9/ff x:0 0 0 2024-03-26T18:54:20.929 INFO:tasks.workunit.client.1.smithi190.stdout:8/37: dread f7 [0,4194304] 0 2024-03-26T18:54:20.929 INFO:tasks.workunit.client.1.smithi190.stdout:8/38: chown l1 136520 1 2024-03-26T18:54:20.929 INFO:tasks.workunit.client.1.smithi190.stdout:8/39: fdatasync f6 0 2024-03-26T18:54:20.935 INFO:tasks.workunit.client.1.smithi190.stdout:7/65: rmdir d3/d6 39 2024-03-26T18:54:20.939 INFO:tasks.workunit.client.1.smithi190.stdout:3/72: creat d1/de/f1d x:0 0 0 2024-03-26T18:54:20.941 INFO:tasks.workunit.client.1.smithi190.stdout:1/50: symlink d3/d9/l10 0 2024-03-26T18:54:20.943 INFO:tasks.workunit.client.1.smithi190.stdout:7/66: rename d3/f1c to d3/d6/d9/d1f/f20 0 2024-03-26T18:54:20.943 INFO:tasks.workunit.client.1.smithi190.stdout:7/67: dread - d3/fe zero size 2024-03-26T18:54:20.943 INFO:tasks.workunit.client.1.smithi190.stdout:7/68: fsync d3/fc 0 2024-03-26T18:54:20.944 INFO:tasks.workunit.client.1.smithi190.stdout:7/69: stat d3/d6/d9/d14/f18 0 2024-03-26T18:54:20.944 INFO:tasks.workunit.client.1.smithi190.stdout:7/70: truncate d3/d6/d9/d14/f18 1338392 0 2024-03-26T18:54:20.944 INFO:tasks.workunit.client.1.smithi190.stdout:7/71: stat d3/d6/d9/d1f 0 2024-03-26T18:54:20.944 INFO:tasks.workunit.client.1.smithi190.stdout:7/72: write d3/fe [306720,54817] 0 2024-03-26T18:54:20.944 INFO:tasks.workunit.client.1.smithi190.stdout:0/162: rmdir d0/dd/d26 39 2024-03-26T18:54:20.947 INFO:tasks.workunit.client.1.smithi190.stdout:1/51: unlink d3/f5 0 2024-03-26T18:54:20.947 INFO:tasks.workunit.client.1.smithi190.stdout:1/52: fdatasync d3/d9/ff 0 2024-03-26T18:54:20.947 INFO:tasks.workunit.client.1.smithi190.stdout:1/53: truncate d3/f7 447050 0 2024-03-26T18:54:20.947 INFO:tasks.workunit.client.1.smithi190.stdout:1/54: chown d3/d9/db/cc 561063425 1 2024-03-26T18:54:20.947 INFO:tasks.workunit.client.1.smithi190.stdout:8/40: dwrite f7 [4194304,4194304] 0 2024-03-26T18:54:20.947 INFO:tasks.workunit.client.1.smithi190.stdout:8/41: fsync f6 0 2024-03-26T18:54:20.951 INFO:tasks.workunit.client.1.smithi190.stdout:7/73: mknod d3/c21 0 2024-03-26T18:54:20.953 INFO:tasks.workunit.client.1.smithi190.stdout:1/55: mknod d3/d9/c11 0 2024-03-26T18:54:20.963 INFO:tasks.workunit.client.1.smithi190.stdout:3/73: dwrite d1/d3/f12 [0,4194304] 0 2024-03-26T18:54:20.968 INFO:tasks.workunit.client.1.smithi190.stdout:0/163: dwrite d0/dd/d26/f34 [0,4194304] 0 2024-03-26T18:54:20.969 INFO:tasks.workunit.client.1.smithi190.stdout:8/42: mknod d9/cc 0 2024-03-26T18:54:20.970 INFO:tasks.workunit.client.1.smithi190.stdout:7/74: dwrite d3/d6/fd [0,4194304] 0 2024-03-26T18:54:20.973 INFO:tasks.workunit.client.1.smithi190.stdout:1/56: creat d3/da/dd/f12 x:0 0 0 2024-03-26T18:54:20.973 INFO:tasks.workunit.client.1.smithi190.stdout:1/57: creat d3/d9/f13 x:0 0 0 2024-03-26T18:54:20.973 INFO:tasks.workunit.client.1.smithi190.stdout:1/58: creat d3/da/dd/f14 x:0 0 0 2024-03-26T18:54:20.973 INFO:tasks.workunit.client.1.smithi190.stdout:1/59: dread - d3/d9/f13 zero size 2024-03-26T18:54:20.973 INFO:tasks.workunit.client.1.smithi190.stdout:1/60: creat d3/d9/f15 x:0 0 0 2024-03-26T18:54:20.973 INFO:tasks.workunit.client.1.smithi190.stdout:1/61: dread - d3/da/dd/f14 zero size 2024-03-26T18:54:20.973 INFO:tasks.workunit.client.1.smithi190.stdout:1/62: creat d3/d9/f16 x:0 0 0 2024-03-26T18:54:20.975 INFO:tasks.workunit.client.1.smithi190.stdout:1/63: read f2 [1459133,123181] 0 2024-03-26T18:54:20.979 INFO:tasks.workunit.client.1.smithi190.stdout:3/74: symlink d1/l1e 0 2024-03-26T18:54:20.984 INFO:tasks.workunit.client.1.smithi190.stdout:0/164: unlink d0/dd/l25 0 2024-03-26T18:54:20.987 INFO:tasks.workunit.client.1.smithi190.stdout:8/43: dwrite f7 [8388608,4194304] 0 2024-03-26T18:54:20.987 INFO:tasks.workunit.client.1.smithi190.stdout:7/75: creat d3/d6/d9/d1f/f22 x:0 0 0 2024-03-26T18:54:20.993 INFO:tasks.workunit.client.1.smithi190.stdout:1/64: symlink d3/d9/l17 0 2024-03-26T18:54:20.995 INFO:tasks.workunit.client.1.smithi190.stdout:8/44: mkdir d9/dd 0 2024-03-26T18:54:20.996 INFO:tasks.workunit.client.1.smithi190.stdout:9/108: rename d12/d19 to d12/d2d 0 2024-03-26T18:54:20.998 INFO:tasks.workunit.client.1.smithi190.stdout:8/45: symlink d9/dd/le 0 2024-03-26T18:54:20.999 INFO:tasks.workunit.client.1.smithi190.stdout:8/46: dread f8 [0,4194304] 0 2024-03-26T18:54:20.999 INFO:tasks.workunit.client.1.smithi190.stdout:8/47: fdatasync f8 0 2024-03-26T18:54:20.999 INFO:tasks.workunit.client.1.smithi190.stdout:8/48: truncate f6 1082513 0 2024-03-26T18:54:20.999 INFO:tasks.workunit.client.1.smithi190.stdout:8/49: creat d9/ff x:0 0 0 2024-03-26T18:54:21.000 INFO:tasks.workunit.client.1.smithi190.stdout:9/109: mknod d12/d2d/d1d/d23/d2b/c2e 0 2024-03-26T18:54:21.002 INFO:tasks.workunit.client.1.smithi190.stdout:9/110: mknod d12/c2f 0 2024-03-26T18:54:21.002 INFO:tasks.workunit.client.1.smithi190.stdout:9/111: stat d12/d2d/l26 0 2024-03-26T18:54:21.004 INFO:tasks.workunit.client.1.smithi190.stdout:9/112: rename d12/d2d/l2c to d12/l30 0 2024-03-26T18:54:21.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.010+0000 7f7098d10700 1 -- 172.21.15.102:0/4172011767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70940fe430 msgr2=0x7f70940fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:21.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.010+0000 7f7098d10700 1 --2- 172.21.15.102:0/4172011767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70940fe430 0x7f70940fe840 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7084009910 tx=0x7f7084009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:21.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.011+0000 7f7098d10700 1 -- 172.21.15.102:0/4172011767 shutdown_connections 2024-03-26T18:54:21.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.011+0000 7f7098d10700 1 --2- 172.21.15.102:0/4172011767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70941014b0 0x7f7094101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.011+0000 7f7098d10700 1 --2- 172.21.15.102:0/4172011767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70940fe430 0x7f70940fe840 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.011+0000 7f7098d10700 1 -- 172.21.15.102:0/4172011767 >> 172.21.15.102:0/4172011767 conn(0x7f70940f9d80 msgr2=0x7f70940fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:21.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.011+0000 7f7098d10700 1 -- 172.21.15.102:0/4172011767 shutdown_connections 2024-03-26T18:54:21.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.011+0000 7f7098d10700 1 -- 172.21.15.102:0/4172011767 wait complete. 2024-03-26T18:54:21.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.012+0000 7f7098d10700 1 Processor -- start 2024-03-26T18:54:21.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.013+0000 7f7098d10700 1 -- start start 2024-03-26T18:54:21.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.013+0000 7f7098d10700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70940fe430 0x7f7094195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:21.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.013+0000 7f7098d10700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70941014b0 0x7f7094195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:21.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.013+0000 7f7098d10700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70941964e0 con 0x7f70940fe430 2024-03-26T18:54:21.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.013+0000 7f7098d10700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7094196620 con 0x7f70941014b0 2024-03-26T18:54:21.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.013+0000 7f709259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70940fe430 0x7f7094195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:21.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.013+0000 7f709259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70940fe430 0x7f7094195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34244/0 (socket says 172.21.15.102:34244) 2024-03-26T18:54:21.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.013+0000 7f709259c700 1 -- 172.21.15.102:0/153384157 learned_addr learned my addr 172.21.15.102:0/153384157 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:21.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.013+0000 7f7091d9b700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70941014b0 0x7f7094195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:21.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.014+0000 7f709259c700 1 -- 172.21.15.102:0/153384157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70941014b0 msgr2=0x7f7094195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:21.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.014+0000 7f709259c700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70941014b0 0x7f7094195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.014+0000 7f709259c700 1 -- 172.21.15.102:0/153384157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70840095f0 con 0x7f70940fe430 2024-03-26T18:54:21.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.014+0000 7f7091d9b700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70941014b0 0x7f7094195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:54:21.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.014+0000 7f709259c700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70940fe430 0x7f7094195980 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f708400bda0 tx=0x7f708400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:21.009 INFO:tasks.workunit.client.1.smithi190.stdout:0/165: dwrite d0/dd/d24/f2a [0,4194304] 0 2024-03-26T18:54:21.009 INFO:tasks.workunit.client.1.smithi190.stdout:7/76: dwrite d3/d6/fd [0,4194304] 0 2024-03-26T18:54:21.009 INFO:tasks.workunit.client.1.smithi190.stdout:0/166: creat d0/dd/f39 x:0 0 0 2024-03-26T18:54:21.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.015+0000 7f70837fe700 1 -- 172.21.15.102:0/153384157 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f708401d070 con 0x7f70940fe430 2024-03-26T18:54:21.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.015+0000 7f7098d10700 1 -- 172.21.15.102:0/153384157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f709419b070 con 0x7f70940fe430 2024-03-26T18:54:21.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.015+0000 7f70837fe700 1 -- 172.21.15.102:0/153384157 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f708400f460 con 0x7f70940fe430 2024-03-26T18:54:21.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.016+0000 7f70837fe700 1 -- 172.21.15.102:0/153384157 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7084021640 con 0x7f70940fe430 2024-03-26T18:54:21.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.016+0000 7f7098d10700 1 -- 172.21.15.102:0/153384157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70940ff4c0 con 0x7f70940fe430 2024-03-26T18:54:21.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.017+0000 7f7098d10700 1 -- 172.21.15.102:0/153384157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7094108c90 con 0x7f70940fe430 2024-03-26T18:54:21.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.017+0000 7f70837fe700 1 -- 172.21.15.102:0/153384157 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f708400fa90 con 0x7f70940fe430 2024-03-26T18:54:21.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.018+0000 7f70837fe700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 0x7f707c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:21.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.018+0000 7f70837fe700 1 -- 172.21.15.102:0/153384157 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f708408ca40 con 0x7f70940fe430 2024-03-26T18:54:21.012 INFO:tasks.workunit.client.1.smithi190.stdout:7/77: symlink d3/l23 0 2024-03-26T18:54:21.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.019+0000 7f7091d9b700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 0x7f707c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:21.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.019+0000 7f7091d9b700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 0x7f707c06eb30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7088005950 tx=0x7f708800b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:21.013 INFO:tasks.workunit.client.1.smithi190.stdout:0/167: symlink d0/dd/d26/l3a 0 2024-03-26T18:54:21.013 INFO:tasks.workunit.client.1.smithi190.stdout:0/168: fdatasync d0/dd/d26/f34 0 2024-03-26T18:54:21.013 INFO:tasks.workunit.client.1.smithi190.stdout:0/169: rename d0/dd to d0/dd/d20/d3b 22 2024-03-26T18:54:21.014 INFO:tasks.workunit.client.1.smithi190.stdout:8/50: dwrite f8 [0,4194304] 0 2024-03-26T18:54:21.014 INFO:tasks.workunit.client.1.smithi190.stdout:0/170: dread d0/f6 [0,4194304] 0 2024-03-26T18:54:21.015 INFO:tasks.workunit.client.1.smithi190.stdout:6/120: truncate f1 12264461 0 2024-03-26T18:54:21.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.023+0000 7f70837fe700 1 -- 172.21.15.102:0/153384157 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f708405ac00 con 0x7f70940fe430 2024-03-26T18:54:21.017 INFO:tasks.workunit.client.1.smithi190.stdout:0/171: dread d0/f13 [0,4194304] 0 2024-03-26T18:54:21.017 INFO:tasks.workunit.client.1.smithi190.stdout:0/172: write d0/dd/fe [1270736,130594] 0 2024-03-26T18:54:21.019 INFO:tasks.workunit.client.1.smithi190.stdout:4/71: getdents d7/d8/d14 0 2024-03-26T18:54:21.022 INFO:tasks.workunit.client.1.smithi190.stdout:6/121: rename lf to d17/d28/l2a 0 2024-03-26T18:54:21.027 INFO:tasks.workunit.client.1.smithi190.stdout:4/72: write d7/f9 [2025887,37702] 0 2024-03-26T18:54:21.028 INFO:tasks.workunit.client.1.smithi190.stdout:4/73: dread - d7/f13 zero size 2024-03-26T18:54:21.028 INFO:tasks.workunit.client.1.smithi190.stdout:4/74: write d7/fc [368326,51903] 0 2024-03-26T18:54:21.030 INFO:tasks.workunit.client.1.smithi190.stdout:4/75: link l4 d7/d8/l1b 0 2024-03-26T18:54:21.032 INFO:tasks.workunit.client.1.smithi190.stdout:7/78: dwrite d3/f19 [0,4194304] 0 2024-03-26T18:54:21.032 INFO:tasks.workunit.client.1.smithi190.stdout:7/79: chown d3/d6/d9/d14 2393151 1 2024-03-26T18:54:21.034 INFO:tasks.workunit.client.1.smithi190.stdout:0/173: dwrite d0/dd/d20/d1c/f29 [0,4194304] 0 2024-03-26T18:54:21.034 INFO:tasks.workunit.client.1.smithi190.stdout:0/174: fsync d0/dd/f12 0 2024-03-26T18:54:21.039 INFO:tasks.workunit.client.1.smithi190.stdout:7/80: link d3/d6/d9/d14/f1e d3/d6/d9/f24 0 2024-03-26T18:54:21.041 INFO:tasks.workunit.client.1.smithi190.stdout:7/81: dread d3/d6/d9/d14/f18 [0,4194304] 0 2024-03-26T18:54:21.041 INFO:tasks.workunit.client.1.smithi190.stdout:7/82: write d3/d6/d9/d1f/f22 [906228,45715] 0 2024-03-26T18:54:21.041 INFO:tasks.workunit.client.1.smithi190.stdout:7/83: stat d3/d6/d9/f24 0 2024-03-26T18:54:21.042 INFO:tasks.workunit.client.1.smithi190.stdout:0/175: mknod d0/dd/d20/d1c/c3c 0 2024-03-26T18:54:21.056 INFO:tasks.workunit.client.1.smithi190.stdout:5/34: truncate d2/f4 1806575 0 2024-03-26T18:54:21.057 INFO:tasks.workunit.client.1.smithi190.stdout:7/84: dwrite d3/d6/d9/d1f/f20 [4194304,4194304] 0 2024-03-26T18:54:21.058 INFO:tasks.workunit.client.1.smithi190.stdout:5/35: unlink d2/l5 0 2024-03-26T18:54:21.072 INFO:tasks.workunit.client.1.smithi190.stdout:1/65: fsync d3/d9/f16 0 2024-03-26T18:54:21.072 INFO:tasks.workunit.client.1.smithi190.stdout:1/66: creat d3/d9/f18 x:0 0 0 2024-03-26T18:54:21.073 INFO:tasks.workunit.client.1.smithi190.stdout:7/85: dwrite d3/d6/fd [0,4194304] 0 2024-03-26T18:54:21.076 INFO:tasks.workunit.client.1.smithi190.stdout:1/67: creat d3/d9/db/f19 x:0 0 0 2024-03-26T18:54:21.076 INFO:tasks.workunit.client.1.smithi190.stdout:1/68: dread - d3/d9/f16 zero size 2024-03-26T18:54:21.076 INFO:tasks.workunit.client.1.smithi190.stdout:1/69: write d3/d9/ff [800388,68177] 0 2024-03-26T18:54:21.077 INFO:tasks.workunit.client.1.smithi190.stdout:1/70: chown d3/f7 106729888 1 2024-03-26T18:54:21.078 INFO:tasks.workunit.client.1.smithi190.stdout:8/51: getdents d9/dd 0 2024-03-26T18:54:21.078 INFO:tasks.workunit.client.1.smithi190.stdout:8/52: truncate d9/fa 100036 0 2024-03-26T18:54:21.083 INFO:tasks.workunit.client.1.smithi190.stdout:9/113: getdents d12 0 2024-03-26T18:54:21.086 INFO:tasks.workunit.client.1.smithi190.stdout:8/53: unlink c2 0 2024-03-26T18:54:21.087 INFO:tasks.workunit.client.1.smithi190.stdout:8/54: creat d9/f10 x:0 0 0 2024-03-26T18:54:21.088 INFO:tasks.workunit.client.1.smithi190.stdout:9/114: rename d12/d2d/d1d/f20 to d12/d2d/f31 0 2024-03-26T18:54:21.100 INFO:tasks.workunit.client.1.smithi190.stdout:7/86: dwrite d3/f19 [0,4194304] 0 2024-03-26T18:54:21.100 INFO:tasks.workunit.client.1.smithi190.stdout:9/115: dread d12/f13 [0,4194304] 0 2024-03-26T18:54:21.139 INFO:tasks.workunit.client.1.smithi190.stdout:8/55: symlink d9/dd/l11 0 2024-03-26T18:54:21.139 INFO:tasks.workunit.client.1.smithi190.stdout:8/56: write d9/fa [828098,129426] 0 2024-03-26T18:54:21.141 INFO:tasks.workunit.client.1.smithi190.stdout:7/87: link d3/f16 d3/d1b/f25 0 2024-03-26T18:54:21.142 INFO:tasks.workunit.client.1.smithi190.stdout:4/76: rmdir d7 39 2024-03-26T18:54:21.142 INFO:tasks.workunit.client.1.smithi190.stdout:4/77: chown d7/lb 1388 1 2024-03-26T18:54:21.143 INFO:tasks.workunit.client.1.smithi190.stdout:9/116: mknod d12/c32 0 2024-03-26T18:54:21.143 INFO:tasks.workunit.client.1.smithi190.stdout:9/117: chown d12/f18 137 1 2024-03-26T18:54:21.145 INFO:tasks.workunit.client.1.smithi190.stdout:7/88: truncate d3/fe 98343 0 2024-03-26T18:54:21.145 INFO:tasks.workunit.client.1.smithi190.stdout:7/89: dread - d3/d1b/f25 zero size 2024-03-26T18:54:21.146 INFO:tasks.workunit.client.1.smithi190.stdout:4/78: mkdir d7/d8/d1c 0 2024-03-26T18:54:21.147 INFO:tasks.workunit.client.1.smithi190.stdout:4/79: read - d7/f13 zero size 2024-03-26T18:54:21.147 INFO:tasks.workunit.client.1.smithi190.stdout:4/80: readlink d7/d8/ld 0 2024-03-26T18:54:21.147 INFO:tasks.workunit.client.1.smithi190.stdout:8/57: write f7 [4603174,4071] 0 2024-03-26T18:54:21.148 INFO:tasks.workunit.client.1.smithi190.stdout:7/90: symlink d3/d6/l26 0 2024-03-26T18:54:21.148 INFO:tasks.workunit.client.1.smithi190.stdout:0/176: write d0/f1f [5201532,54478] 0 2024-03-26T18:54:21.149 INFO:tasks.workunit.client.1.smithi190.stdout:5/36: fsync d2/f6 0 2024-03-26T18:54:21.151 INFO:tasks.workunit.client.1.smithi190.stdout:8/58: rename f7 to d9/f12 0 2024-03-26T18:54:21.157 INFO:tasks.workunit.client.1.smithi190.stdout:1/71: rmdir d3 39 2024-03-26T18:54:21.157 INFO:tasks.workunit.client.1.smithi190.stdout:1/72: read - d3/d9/f15 zero size 2024-03-26T18:54:21.161 INFO:tasks.workunit.client.1.smithi190.stdout:2/50: sync 2024-03-26T18:54:21.163 INFO:tasks.workunit.client.1.smithi190.stdout:1/73: mkdir d3/d9/d1a 0 2024-03-26T18:54:21.163 INFO:tasks.workunit.client.1.smithi190.stdout:1/74: truncate d3/d9/f13 909315 0 2024-03-26T18:54:21.163 INFO:tasks.workunit.client.1.smithi190.stdout:1/75: write f2 [2187896,15038] 0 2024-03-26T18:54:21.166 INFO:tasks.workunit.client.1.smithi190.stdout:2/51: dread f5 [0,4194304] 0 2024-03-26T18:54:21.167 INFO:tasks.workunit.client.1.smithi190.stdout:2/52: read - d6/fe zero size 2024-03-26T18:54:21.167 INFO:tasks.workunit.client.1.smithi190.stdout:2/53: write d6/fe [164976,30226] 0 2024-03-26T18:54:21.170 INFO:tasks.workunit.client.1.smithi190.stdout:2/54: rename d6/d8 to d6/d9/d10 0 2024-03-26T18:54:21.172 INFO:tasks.workunit.client.1.smithi190.stdout:2/55: symlink d6/df/l11 0 2024-03-26T18:54:21.174 INFO:tasks.workunit.client.1.smithi190.stdout:2/56: unlink d6/d9/fd 0 2024-03-26T18:54:21.177 INFO:tasks.workunit.client.1.smithi190.stdout:2/57: symlink d6/d9/l12 0 2024-03-26T18:54:21.177 INFO:tasks.workunit.client.1.smithi190.stdout:2/58: write d6/fe [345597,52385] 0 2024-03-26T18:54:21.193 INFO:tasks.workunit.client.1.smithi190.stdout:4/81: dwrite d7/f13 [0,4194304] 0 2024-03-26T18:54:21.196 INFO:tasks.workunit.client.1.smithi190.stdout:9/118: dwrite f11 [0,4194304] 0 2024-03-26T18:54:21.196 INFO:tasks.workunit.client.1.smithi190.stdout:9/119: fsync d12/f28 0 2024-03-26T18:54:21.197 INFO:tasks.workunit.client.1.smithi190.stdout:0/177: dwrite d0/dd/d24/f2b [0,4194304] 0 2024-03-26T18:54:21.197 INFO:tasks.workunit.client.1.smithi190.stdout:0/178: truncate d0/dd/d20/d1c/f28 820115 0 2024-03-26T18:54:21.197 INFO:tasks.workunit.client.1.smithi190.stdout:7/91: dwrite d3/d6/d9/d14/f1e [0,4194304] 0 2024-03-26T18:54:21.197 INFO:tasks.workunit.client.1.smithi190.stdout:7/92: write d3/fc [1261302,97908] 0 2024-03-26T18:54:21.197 INFO:tasks.workunit.client.1.smithi190.stdout:5/37: dwrite d2/f3 [0,4194304] 0 2024-03-26T18:54:21.197 INFO:tasks.workunit.client.1.smithi190.stdout:5/38: fdatasync d2/f3 0 2024-03-26T18:54:21.197 INFO:tasks.workunit.client.1.smithi190.stdout:1/76: dwrite d3/d9/f16 [0,4194304] 0 2024-03-26T18:54:21.197 INFO:tasks.workunit.client.1.smithi190.stdout:9/120: read d12/f16 [277082,81276] 0 2024-03-26T18:54:21.198 INFO:tasks.workunit.client.1.smithi190.stdout:9/121: chown d12/d2d/l1a 1085 1 2024-03-26T18:54:21.198 INFO:tasks.workunit.client.1.smithi190.stdout:9/122: stat d12/f25 0 2024-03-26T18:54:21.199 INFO:tasks.workunit.client.1.smithi190.stdout:2/59: dwrite f5 [0,4194304] 0 2024-03-26T18:54:21.210 INFO:tasks.workunit.client.1.smithi190.stdout:2/60: dread f5 [0,4194304] 0 2024-03-26T18:54:21.210 INFO:tasks.workunit.client.1.smithi190.stdout:0/179: dread d0/f1f [0,4194304] 0 2024-03-26T18:54:21.212 INFO:tasks.workunit.client.1.smithi190.stdout:7/93: dwrite d3/fe [0,4194304] 0 2024-03-26T18:54:21.212 INFO:tasks.workunit.client.1.smithi190.stdout:7/94: truncate d3/d6/fd 5111923 0 2024-03-26T18:54:21.213 INFO:tasks.workunit.client.1.smithi190.stdout:7/95: write d3/f16 [706550,112974] 0 2024-03-26T18:54:21.220 INFO:tasks.workunit.client.1.smithi190.stdout:7/96: dread d3/d6/d9/f24 [0,4194304] 0 2024-03-26T18:54:21.221 INFO:tasks.workunit.client.1.smithi190.stdout:7/97: read d3/fc [1076252,58399] 0 2024-03-26T18:54:21.257 INFO:tasks.workunit.client.1.smithi190.stdout:4/82: mkdir d7/d1d 0 2024-03-26T18:54:21.257 INFO:tasks.workunit.client.1.smithi190.stdout:4/83: readlink l4 0 2024-03-26T18:54:21.257 INFO:tasks.workunit.client.1.smithi190.stdout:4/84: fdatasync d7/fc 0 2024-03-26T18:54:21.257 INFO:tasks.workunit.client.1.smithi190.stdout:8/59: rmdir d9 39 2024-03-26T18:54:21.258 INFO:tasks.workunit.client.1.smithi190.stdout:4/85: dread d7/fc [0,4194304] 0 2024-03-26T18:54:21.262 INFO:tasks.workunit.client.1.smithi190.stdout:5/39: mknod d2/c9 0 2024-03-26T18:54:21.262 INFO:tasks.workunit.client.1.smithi190.stdout:5/40: readlink d2/l8 0 2024-03-26T18:54:21.263 INFO:tasks.workunit.client.1.smithi190.stdout:8/60: dread d9/fa [0,4194304] 0 2024-03-26T18:54:21.265 INFO:tasks.workunit.client.1.smithi190.stdout:1/77: creat d3/da/dd/f1b x:0 0 0 2024-03-26T18:54:21.265 INFO:tasks.workunit.client.1.smithi190.stdout:1/78: dread - d3/da/dd/f1b zero size 2024-03-26T18:54:21.265 INFO:tasks.workunit.client.1.smithi190.stdout:1/79: readlink d3/d9/l17 0 2024-03-26T18:54:21.266 INFO:tasks.workunit.client.1.smithi190.stdout:1/80: dread - d3/da/dd/f1b zero size 2024-03-26T18:54:21.266 INFO:tasks.workunit.client.1.smithi190.stdout:9/123: creat d12/d2d/d1d/d23/f33 x:0 0 0 2024-03-26T18:54:21.267 INFO:tasks.workunit.client.1.smithi190.stdout:2/61: creat d6/d9/d10/f13 x:0 0 0 2024-03-26T18:54:21.268 INFO:tasks.workunit.client.1.smithi190.stdout:0/180: symlink d0/dd/d20/l3d 0 2024-03-26T18:54:21.268 INFO:tasks.workunit.client.1.smithi190.stdout:0/181: chown d0/dd/d20/d1c 53 1 2024-03-26T18:54:21.284 INFO:tasks.workunit.client.1.smithi190.stdout:2/62: dwrite f5 [0,4194304] 0 2024-03-26T18:54:21.285 INFO:tasks.workunit.client.1.smithi190.stdout:2/63: chown d6/d9/d10/f13 14949284 1 2024-03-26T18:54:21.285 INFO:tasks.workunit.client.1.smithi190.stdout:2/64: fdatasync d6/d9/d10/f13 0 2024-03-26T18:54:21.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.358+0000 7f7098d10700 1 -- 172.21.15.102:0/153384157 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7094105e00 con 0x7f707c06c680 2024-03-26T18:54:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.360+0000 7f70837fe700 1 -- 172.21.15.102:0/153384157 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f7094105e00 con 0x7f707c06c680 2024-03-26T18:54:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.362+0000 7f7098d10700 1 -- 172.21.15.102:0/153384157 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 msgr2=0x7f707c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.362+0000 7f7098d10700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 0x7f707c06eb30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7088005950 tx=0x7f708800b410 comp rx=0 tx=0).stop 2024-03-26T18:54:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.362+0000 7f7098d10700 1 -- 172.21.15.102:0/153384157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70940fe430 msgr2=0x7f7094195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.362+0000 7f7098d10700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70940fe430 0x7f7094195980 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f708400bda0 tx=0x7f708400bdd0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.362+0000 7f7098d10700 1 -- 172.21.15.102:0/153384157 shutdown_connections 2024-03-26T18:54:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.362+0000 7f7098d10700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f707c06c680 0x7f707c06eb30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.362+0000 7f7098d10700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70940fe430 0x7f7094195980 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.362+0000 7f7098d10700 1 --2- 172.21.15.102:0/153384157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70941014b0 0x7f7094195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.362+0000 7f7098d10700 1 -- 172.21.15.102:0/153384157 >> 172.21.15.102:0/153384157 conn(0x7f70940f9d80 msgr2=0x7f70941046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.362+0000 7f7098d10700 1 -- 172.21.15.102:0/153384157 shutdown_connections 2024-03-26T18:54:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.362+0000 7f7098d10700 1 -- 172.21.15.102:0/153384157 wait complete. 2024-03-26T18:54:21.365 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:54:21.437 INFO:tasks.workunit.client.1.smithi190.stdout:4/86: link d7/d8/d14/f19 d7/d8/f1e 0 2024-03-26T18:54:21.440 INFO:tasks.workunit.client.1.smithi190.stdout:8/61: mkdir d9/dd/d13 0 2024-03-26T18:54:21.440 INFO:tasks.workunit.client.1.smithi190.stdout:8/62: read - d9/f10 zero size 2024-03-26T18:54:21.442 INFO:tasks.workunit.client.1.smithi190.stdout:1/81: rename f1 to d3/d9/f1c 0 2024-03-26T18:54:21.442 INFO:tasks.workunit.client.1.smithi190.stdout:9/124: rename d12/d2d to d12/d2d/d1d/d23/d2b/d34 22 2024-03-26T18:54:21.443 INFO:tasks.workunit.client.1.smithi190.stdout:0/182: rename d0/dd/d20/d1c/l2c to d0/dd/d20/d1c/l3e 0 2024-03-26T18:54:21.444 INFO:tasks.workunit.client.1.smithi190.stdout:2/65: symlink d6/df/l14 0 2024-03-26T18:54:21.444 INFO:tasks.workunit.client.1.smithi190.stdout:2/66: fsync d6/f7 0 2024-03-26T18:54:21.446 INFO:tasks.workunit.client.1.smithi190.stdout:7/98: getdents d3 0 2024-03-26T18:54:21.448 INFO:tasks.workunit.client.1.smithi190.stdout:4/87: unlink d7/d8/ld 0 2024-03-26T18:54:21.448 INFO:tasks.workunit.client.1.smithi190.stdout:4/88: dread - d7/d8/d14/f1a zero size 2024-03-26T18:54:21.450 INFO:tasks.workunit.client.1.smithi190.stdout:5/41: getdents d2 0 2024-03-26T18:54:21.452 INFO:tasks.workunit.client.1.smithi190.stdout:8/63: rmdir d9/dd/d13 0 2024-03-26T18:54:21.454 INFO:tasks.workunit.client.1.smithi190.stdout:0/183: write d0/dd/d24/f2f [1405367,84698] 0 2024-03-26T18:54:21.455 INFO:tasks.workunit.client.1.smithi190.stdout:9/125: creat d12/d2d/f35 x:0 0 0 2024-03-26T18:54:21.455 INFO:tasks.workunit.client.1.smithi190.stdout:9/126: truncate d12/f14 283600 0 2024-03-26T18:54:21.455 INFO:tasks.workunit.client.1.smithi190.stdout:5/42: dread d2/f4 [0,4194304] 0 2024-03-26T18:54:21.459 INFO:tasks.workunit.client.1.smithi190.stdout:4/89: creat d7/d8/d1c/f1f x:0 0 0 2024-03-26T18:54:21.459 INFO:tasks.workunit.client.1.smithi190.stdout:4/90: stat d7/d8/d1c 0 2024-03-26T18:54:21.459 INFO:tasks.workunit.client.1.smithi190.stdout:4/91: fsync d7/f17 0 2024-03-26T18:54:21.459 INFO:tasks.workunit.client.1.smithi190.stdout:4/92: fsync d7/d8/f12 0 2024-03-26T18:54:21.459 INFO:tasks.workunit.client.1.smithi190.stdout:4/93: write d7/f10 [1093321,44070] 0 2024-03-26T18:54:21.460 INFO:tasks.workunit.client.1.smithi190.stdout:1/82: dwrite d3/d9/f13 [0,4194304] 0 2024-03-26T18:54:21.461 INFO:tasks.workunit.client.1.smithi190.stdout:8/64: unlink d9/lb 0 2024-03-26T18:54:21.464 INFO:tasks.workunit.client.1.smithi190.stdout:0/184: read d0/dd/d26/f34 [239083,16395] 0 2024-03-26T18:54:21.464 INFO:tasks.workunit.client.1.smithi190.stdout:0/185: chown d0/dc/c38 2214 1 2024-03-26T18:54:21.464 INFO:tasks.workunit.client.1.smithi190.stdout:0/186: stat d0/dd/lf 0 2024-03-26T18:54:21.465 INFO:tasks.workunit.client.1.smithi190.stdout:9/127: getdents d12 0 2024-03-26T18:54:21.467 INFO:tasks.workunit.client.1.smithi190.stdout:5/43: truncate d2/f4 696104 0 2024-03-26T18:54:21.469 INFO:tasks.workunit.client.1.smithi190.stdout:7/99: dwrite d3/f19 [0,4194304] 0 2024-03-26T18:54:21.469 INFO:tasks.workunit.client.1.smithi190.stdout:2/67: rmdir d6/d9/d10 39 2024-03-26T18:54:21.475 INFO:tasks.workunit.client.1.smithi190.stdout:4/94: mkdir d7/d8/d1c/d20 0 2024-03-26T18:54:21.475 INFO:tasks.workunit.client.1.smithi190.stdout:4/95: fdatasync d7/d8/d1c/f1f 0 2024-03-26T18:54:21.482 INFO:tasks.workunit.client.1.smithi190.stdout:1/83: creat d3/d9/d1a/f1d x:0 0 0 2024-03-26T18:54:21.483 INFO:tasks.workunit.client.1.smithi190.stdout:8/65: creat d9/f14 x:0 0 0 2024-03-26T18:54:21.484 INFO:tasks.workunit.client.1.smithi190.stdout:8/66: dread - d9/f14 zero size 2024-03-26T18:54:21.485 INFO:tasks.workunit.client.1.smithi190.stdout:0/187: creat d0/dd/d26/f3f x:0 0 0 2024-03-26T18:54:21.485 INFO:tasks.workunit.client.1.smithi190.stdout:0/188: chown d0/dd/f39 15 1 2024-03-26T18:54:21.487 INFO:tasks.workunit.client.1.smithi190.stdout:5/44: creat d2/fa x:0 0 0 2024-03-26T18:54:21.488 INFO:tasks.workunit.client.1.smithi190.stdout:7/100: creat d3/d1b/f27 x:0 0 0 2024-03-26T18:54:21.489 INFO:tasks.workunit.client.1.smithi190.stdout:9/128: dwrite d12/f28 [0,4194304] 0 2024-03-26T18:54:21.489 INFO:tasks.workunit.client.1.smithi190.stdout:9/129: fdatasync f11 0 2024-03-26T18:54:21.489 INFO:tasks.workunit.client.1.smithi190.stdout:9/130: write ff [1178410,117960] 0 2024-03-26T18:54:21.489 INFO:tasks.workunit.client.1.smithi190.stdout:9/131: read ff [1046750,41598] 0 2024-03-26T18:54:21.489 INFO:tasks.workunit.client.1.smithi190.stdout:9/132: dread - d12/f15 zero size 2024-03-26T18:54:21.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.497+0000 7f69be1ad700 1 -- 172.21.15.102:0/1459534141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69b80fea30 msgr2=0x7f69b80fee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:21.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.497+0000 7f69be1ad700 1 --2- 172.21.15.102:0/1459534141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69b80fea30 0x7f69b80fee40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009910 tx=0x7f69a8009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:21.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.497+0000 7f69be1ad700 1 -- 172.21.15.102:0/1459534141 shutdown_connections 2024-03-26T18:54:21.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.497+0000 7f69be1ad700 1 --2- 172.21.15.102:0/1459534141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69b80ff380 0x7f69b80ff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.497+0000 7f69be1ad700 1 --2- 172.21.15.102:0/1459534141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69b80fea30 0x7f69b80fee40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.497+0000 7f69be1ad700 1 -- 172.21.15.102:0/1459534141 >> 172.21.15.102:0/1459534141 conn(0x7f69b80fa600 msgr2=0x7f69b80fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.497+0000 7f69be1ad700 1 -- 172.21.15.102:0/1459534141 shutdown_connections 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.497+0000 7f69be1ad700 1 -- 172.21.15.102:0/1459534141 wait complete. 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.498+0000 7f69be1ad700 1 Processor -- start 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.498+0000 7f69be1ad700 1 -- start start 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.498+0000 7f69be1ad700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69b80fea30 0x7f69b819bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.498+0000 7f69be1ad700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69b80ff380 0x7f69b819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.498+0000 7f69be1ad700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b819ca90 con 0x7f69b80ff380 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.498+0000 7f69be1ad700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b819cbd0 con 0x7f69b80fea30 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.498+0000 7f69b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69b80ff380 0x7f69b819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.498+0000 7f69b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69b80ff380 0x7f69b819c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34254/0 (socket says 172.21.15.102:34254) 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.498+0000 7f69b6ffd700 1 -- 172.21.15.102:0/3287531658 learned_addr learned my addr 172.21.15.102:0/3287531658 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.498+0000 7f69b77fe700 1 --2- 172.21.15.102:0/3287531658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69b80fea30 0x7f69b819bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.499+0000 7f69b6ffd700 1 -- 172.21.15.102:0/3287531658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69b80fea30 msgr2=0x7f69b819bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.499+0000 7f69b6ffd700 1 --2- 172.21.15.102:0/3287531658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69b80fea30 0x7f69b819bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.499+0000 7f69b6ffd700 1 -- 172.21.15.102:0/3287531658 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a80095f0 con 0x7f69b80ff380 2024-03-26T18:54:21.494 INFO:tasks.workunit.client.1.smithi190.stdout:2/68: rename d6/df/l14 to d6/d9/d10/l15 0 2024-03-26T18:54:21.494 INFO:tasks.workunit.client.1.smithi190.stdout:2/69: write d6/d9/d10/f13 [932020,75515] 0 2024-03-26T18:54:21.494 INFO:tasks.workunit.client.1.smithi190.stdout:4/96: symlink d7/d8/d1c/l21 0 2024-03-26T18:54:21.494 INFO:tasks.workunit.client.1.smithi190.stdout:4/97: write d7/f10 [1232024,28524] 0 2024-03-26T18:54:21.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.499+0000 7f69b6ffd700 1 --2- 172.21.15.102:0/3287531658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69b80ff380 0x7f69b819c470 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00e940 tx=0x7f69ac00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:21.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.499+0000 7f69b4ff9700 1 -- 172.21.15.102:0/3287531658 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69ac00cba0 con 0x7f69b80ff380 2024-03-26T18:54:21.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.499+0000 7f69b4ff9700 1 -- 172.21.15.102:0/3287531658 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f69ac00ee90 con 0x7f69b80ff380 2024-03-26T18:54:21.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.499+0000 7f69b4ff9700 1 -- 172.21.15.102:0/3287531658 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69ac005650 con 0x7f69b80ff380 2024-03-26T18:54:21.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.500+0000 7f69be1ad700 1 -- 172.21.15.102:0/3287531658 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b81a1680 con 0x7f69b80ff380 2024-03-26T18:54:21.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.500+0000 7f69be1ad700 1 -- 172.21.15.102:0/3287531658 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b81a1b80 con 0x7f69b80ff380 2024-03-26T18:54:21.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.501+0000 7f69b4ff9700 1 -- 172.21.15.102:0/3287531658 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f69ac021440 con 0x7f69b80ff380 2024-03-26T18:54:21.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.501+0000 7f69be1ad700 1 -- 172.21.15.102:0/3287531658 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b8105b10 con 0x7f69b80ff380 2024-03-26T18:54:21.495 INFO:tasks.workunit.client.1.smithi190.stdout:1/84: mkdir d3/da/d1e 0 2024-03-26T18:54:21.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.502+0000 7f69b4ff9700 1 --2- 172.21.15.102:0/3287531658 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f69a006c680 0x7f69a006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:21.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.502+0000 7f69b4ff9700 1 -- 172.21.15.102:0/3287531658 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f69ac014070 con 0x7f69b80ff380 2024-03-26T18:54:21.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.502+0000 7f69b77fe700 1 --2- 172.21.15.102:0/3287531658 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f69a006c680 0x7f69a006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:21.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.503+0000 7f69b77fe700 1 --2- 172.21.15.102:0/3287531658 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f69a006c680 0x7f69a006eb30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009fd0 tx=0x7f69a8005940 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:21.498 INFO:tasks.workunit.client.1.smithi190.stdout:8/67: mkdir d9/dd/d15 0 2024-03-26T18:54:21.498 INFO:tasks.workunit.client.1.smithi190.stdout:8/68: write d9/f14 [182677,131018] 0 2024-03-26T18:54:21.498 INFO:tasks.workunit.client.1.smithi190.stdout:7/101: dread d3/f19 [0,4194304] 0 2024-03-26T18:54:21.498 INFO:tasks.workunit.client.1.smithi190.stdout:7/102: creat d3/d1b/f28 x:0 0 0 2024-03-26T18:54:21.499 INFO:tasks.workunit.client.1.smithi190.stdout:2/70: dread f5 [0,4194304] 0 2024-03-26T18:54:21.500 INFO:tasks.workunit.client.1.smithi190.stdout:2/71: read d6/d9/d10/f13 [827571,43255] 0 2024-03-26T18:54:21.500 INFO:tasks.workunit.client.1.smithi190.stdout:2/72: chown d6/f7 0 1 2024-03-26T18:54:21.500 INFO:tasks.workunit.client.1.smithi190.stdout:2/73: write f5 [5129424,47361] 0 2024-03-26T18:54:21.500 INFO:tasks.workunit.client.1.smithi190.stdout:2/74: write d6/f7 [540853,73154] 0 2024-03-26T18:54:21.501 INFO:tasks.workunit.client.1.smithi190.stdout:0/189: rmdir d0/dd/d24 39 2024-03-26T18:54:21.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.509+0000 7f69b4ff9700 1 -- 172.21.15.102:0/3287531658 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f69ac059e60 con 0x7f69b80ff380 2024-03-26T18:54:21.504 INFO:tasks.workunit.client.1.smithi190.stdout:5/45: mknod d2/cb 0 2024-03-26T18:54:21.511 INFO:tasks.workunit.client.1.smithi190.stdout:2/75: dread d6/f7 [0,4194304] 0 2024-03-26T18:54:21.513 INFO:tasks.workunit.client.1.smithi190.stdout:9/133: symlink d12/d2d/l36 0 2024-03-26T18:54:21.513 INFO:tasks.workunit.client.1.smithi190.stdout:9/134: chown d12/d2d/l21 602 1 2024-03-26T18:54:21.513 INFO:tasks.workunit.client.1.smithi190.stdout:9/135: dread - d12/f24 zero size 2024-03-26T18:54:21.515 INFO:tasks.workunit.client.1.smithi190.stdout:1/85: link d3/da/dd/f14 d3/da/f1f 0 2024-03-26T18:54:21.515 INFO:tasks.workunit.client.1.smithi190.stdout:1/86: chown d3/d9/f16 236107 1 2024-03-26T18:54:21.515 INFO:tasks.workunit.client.1.smithi190.stdout:1/87: write d3/da/f1f [821273,46344] 0 2024-03-26T18:54:21.517 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:21 smithi190 ceph-mon[35043]: pgmap v238: 65 pgs: 65 active+clean; 268 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 19 MiB/s wr, 589 op/s 2024-03-26T18:54:21.518 INFO:tasks.workunit.client.1.smithi190.stdout:8/69: write d9/f12 [9204423,3144] 0 2024-03-26T18:54:21.518 INFO:tasks.workunit.client.1.smithi190.stdout:8/70: fdatasync d9/ff 0 2024-03-26T18:54:21.518 INFO:tasks.workunit.client.1.smithi190.stdout:8/71: write d9/f14 [1014057,74046] 0 2024-03-26T18:54:21.518 INFO:tasks.workunit.client.1.smithi190.stdout:8/72: chown d9/cc 8 1 2024-03-26T18:54:21.518 INFO:tasks.workunit.client.1.smithi190.stdout:4/98: dwrite d7/d8/d14/f1a [0,4194304] 0 2024-03-26T18:54:21.518 INFO:tasks.workunit.client.1.smithi190.stdout:2/76: write f5 [5056168,53634] 0 2024-03-26T18:54:21.519 INFO:tasks.workunit.client.1.smithi190.stdout:2/77: chown d6/df 1042 1 2024-03-26T18:54:21.519 INFO:tasks.workunit.client.1.smithi190.stdout:2/78: readlink d6/df/l11 0 2024-03-26T18:54:21.519 INFO:tasks.workunit.client.1.smithi190.stdout:7/103: rename d3/d6/d9/d14/f1e to d3/f29 0 2024-03-26T18:54:21.523 INFO:tasks.workunit.client.1.smithi190.stdout:0/190: mknod d0/dd/d20/c40 0 2024-03-26T18:54:21.530 INFO:tasks.workunit.client.1.smithi190.stdout:7/104: dread d3/d6/d9/f24 [0,4194304] 0 2024-03-26T18:54:21.530 INFO:tasks.workunit.client.1.smithi190.stdout:9/136: mknod d12/d2d/d1d/d23/d2a/c37 0 2024-03-26T18:54:21.535 INFO:tasks.workunit.client.1.smithi190.stdout:1/88: unlink d3/d9/d1a/f1d 0 2024-03-26T18:54:21.535 INFO:tasks.workunit.client.1.smithi190.stdout:1/89: write d3/d9/f15 [240209,114212] 0 2024-03-26T18:54:21.535 INFO:tasks.workunit.client.1.smithi190.stdout:1/90: readlink d3/l8 0 2024-03-26T18:54:21.535 INFO:tasks.workunit.client.1.smithi190.stdout:1/91: read - d3/da/dd/f1b zero size 2024-03-26T18:54:21.539 INFO:tasks.workunit.client.1.smithi190.stdout:1/92: fdatasync d3/d9/f15 0 2024-03-26T18:54:21.541 INFO:tasks.workunit.client.1.smithi190.stdout:4/99: dwrite d7/d8/d14/f1a [0,4194304] 0 2024-03-26T18:54:21.541 INFO:tasks.workunit.client.1.smithi190.stdout:4/100: write d7/f9 [2279569,103295] 0 2024-03-26T18:54:21.545 INFO:tasks.workunit.client.1.smithi190.stdout:1/93: dread d3/da/dd/f14 [0,4194304] 0 2024-03-26T18:54:21.545 INFO:tasks.workunit.client.1.smithi190.stdout:1/94: truncate d3/da/f1f 1531050 0 2024-03-26T18:54:21.555 INFO:tasks.workunit.client.1.smithi190.stdout:6/122: sync 2024-03-26T18:54:21.555 INFO:tasks.workunit.client.1.smithi190.stdout:3/75: sync 2024-03-26T18:54:21.556 INFO:tasks.workunit.client.1.smithi190.stdout:3/76: fsync d1/d3/f13 0 2024-03-26T18:54:21.556 INFO:tasks.workunit.client.1.smithi190.stdout:3/77: fdatasync d1/d3/df/f19 0 2024-03-26T18:54:21.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:21 smithi102 ceph-mon[29002]: pgmap v238: 65 pgs: 65 active+clean; 268 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 19 MiB/s wr, 589 op/s 2024-03-26T18:54:21.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.750+0000 7f69be1ad700 1 -- 172.21.15.102:0/3287531658 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f69b8061190 con 0x7f69a006c680 2024-03-26T18:54:21.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.751+0000 7f69b4ff9700 1 -- 172.21.15.102:0/3287531658 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f69b8061190 con 0x7f69a006c680 2024-03-26T18:54:21.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.753+0000 7f69be1ad700 1 -- 172.21.15.102:0/3287531658 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f69a006c680 msgr2=0x7f69a006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:21.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.753+0000 7f69be1ad700 1 --2- 172.21.15.102:0/3287531658 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f69a006c680 0x7f69a006eb30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009fd0 tx=0x7f69a8005940 comp rx=0 tx=0).stop 2024-03-26T18:54:21.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.753+0000 7f69be1ad700 1 -- 172.21.15.102:0/3287531658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69b80ff380 msgr2=0x7f69b819c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:21.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.753+0000 7f69be1ad700 1 --2- 172.21.15.102:0/3287531658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69b80ff380 0x7f69b819c470 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00e940 tx=0x7f69ac00ec50 comp rx=0 tx=0).stop 2024-03-26T18:54:21.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.753+0000 7f69be1ad700 1 -- 172.21.15.102:0/3287531658 shutdown_connections 2024-03-26T18:54:21.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.754+0000 7f69be1ad700 1 --2- 172.21.15.102:0/3287531658 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f69a006c680 0x7f69a006eb30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.754+0000 7f69be1ad700 1 --2- 172.21.15.102:0/3287531658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69b80fea30 0x7f69b819bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.754+0000 7f69be1ad700 1 --2- 172.21.15.102:0/3287531658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69b80ff380 0x7f69b819c470 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.754+0000 7f69be1ad700 1 -- 172.21.15.102:0/3287531658 >> 172.21.15.102:0/3287531658 conn(0x7f69b80fa600 msgr2=0x7f69b8102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:21.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.754+0000 7f69be1ad700 1 -- 172.21.15.102:0/3287531658 shutdown_connections 2024-03-26T18:54:21.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.754+0000 7f69be1ad700 1 -- 172.21.15.102:0/3287531658 wait complete. 2024-03-26T18:54:21.821 INFO:tasks.workunit.client.1.smithi190.stdout:8/73: mkdir d9/d16 0 2024-03-26T18:54:21.821 INFO:tasks.workunit.client.1.smithi190.stdout:8/74: fdatasync d9/f12 0 2024-03-26T18:54:21.823 INFO:tasks.workunit.client.1.smithi190.stdout:2/79: creat d6/df/f16 x:0 0 0 2024-03-26T18:54:21.824 INFO:tasks.workunit.client.1.smithi190.stdout:0/191: creat d0/dd/d20/d1c/f41 x:0 0 0 2024-03-26T18:54:21.825 INFO:tasks.workunit.client.1.smithi190.stdout:0/192: chown d0/dd/d24/f2b 5 1 2024-03-26T18:54:21.825 INFO:tasks.workunit.client.1.smithi190.stdout:2/80: dread d6/fe [0,4194304] 0 2024-03-26T18:54:21.826 INFO:tasks.workunit.client.1.smithi190.stdout:7/105: truncate d3/d6/d9/d1f/f22 903569 0 2024-03-26T18:54:21.828 INFO:tasks.workunit.client.1.smithi190.stdout:9/137: creat d12/f38 x:0 0 0 2024-03-26T18:54:21.828 INFO:tasks.workunit.client.1.smithi190.stdout:9/138: creat d12/d2d/f39 x:0 0 0 2024-03-26T18:54:21.828 INFO:tasks.workunit.client.1.smithi190.stdout:9/139: write d12/f24 [478853,106779] 0 2024-03-26T18:54:21.828 INFO:tasks.workunit.client.1.smithi190.stdout:9/140: truncate fb 1368871 0 2024-03-26T18:54:21.830 INFO:tasks.workunit.client.1.smithi190.stdout:9/141: dread f10 [0,4194304] 0 2024-03-26T18:54:21.831 INFO:tasks.workunit.client.1.smithi190.stdout:4/101: link d7/f13 d7/d8/f22 0 2024-03-26T18:54:21.835 INFO:tasks.workunit.client.1.smithi190.stdout:1/95: rename d3/da/dd/f14 to d3/d9/f20 0 2024-03-26T18:54:21.837 INFO:tasks.workunit.client.1.smithi190.stdout:6/123: mkdir d17/d28/d2b 0 2024-03-26T18:54:21.839 INFO:tasks.workunit.client.1.smithi190.stdout:6/124: dread fc [0,4194304] 0 2024-03-26T18:54:21.841 INFO:tasks.workunit.client.1.smithi190.stdout:4/102: dread d7/f13 [0,4194304] 0 2024-03-26T18:54:21.852 INFO:tasks.workunit.client.1.smithi190.stdout:8/75: rmdir d9/dd 39 2024-03-26T18:54:21.854 INFO:tasks.workunit.client.1.smithi190.stdout:0/193: mknod d0/dd/d24/c42 0 2024-03-26T18:54:21.855 INFO:tasks.workunit.client.1.smithi190.stdout:2/81: mknod d6/c17 0 2024-03-26T18:54:21.860 INFO:tasks.workunit.client.1.smithi190.stdout:1/96: symlink d3/l21 0 2024-03-26T18:54:21.860 INFO:tasks.workunit.client.1.smithi190.stdout:1/97: write d3/d9/f15 [929148,14332] 0 2024-03-26T18:54:21.861 INFO:tasks.workunit.client.1.smithi190.stdout:6/125: truncate f1 1747160 0 2024-03-26T18:54:21.863 INFO:tasks.workunit.client.1.smithi190.stdout:3/78: getdents d1/d3/df 0 2024-03-26T18:54:21.869 INFO:tasks.workunit.client.1.smithi190.stdout:8/76: link d9/ff d9/dd/f17 0 2024-03-26T18:54:21.869 INFO:tasks.workunit.client.1.smithi190.stdout:8/77: readlink l4 0 2024-03-26T18:54:21.869 INFO:tasks.workunit.client.1.smithi190.stdout:8/78: chown l1 7 1 2024-03-26T18:54:21.871 INFO:tasks.workunit.client.1.smithi190.stdout:0/194: mkdir d0/dd/d20/d43 0 2024-03-26T18:54:21.871 INFO:tasks.workunit.client.1.smithi190.stdout:0/195: truncate d0/dd/f39 164840 0 2024-03-26T18:54:21.872 INFO:tasks.workunit.client.1.smithi190.stdout:1/98: write d3/d9/f16 [296525,65373] 0 2024-03-26T18:54:21.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.880+0000 7fd68af22700 1 -- 172.21.15.102:0/3229298370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd684103500 msgr2=0x7fd6841058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:21.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.880+0000 7fd68af22700 1 --2- 172.21.15.102:0/3229298370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd684103500 0x7fd6841058e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd670009910 tx=0x7fd670009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:21.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.880+0000 7fd68af22700 1 -- 172.21.15.102:0/3229298370 shutdown_connections 2024-03-26T18:54:21.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.880+0000 7fd68af22700 1 --2- 172.21.15.102:0/3229298370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd684103500 0x7fd6841058e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.880+0000 7fd68af22700 1 --2- 172.21.15.102:0/3229298370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd684100be0 0x7fd684102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.880+0000 7fd68af22700 1 -- 172.21.15.102:0/3229298370 >> 172.21.15.102:0/3229298370 conn(0x7fd6840fa600 msgr2=0x7fd6840fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:21.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.880+0000 7fd68af22700 1 -- 172.21.15.102:0/3229298370 shutdown_connections 2024-03-26T18:54:21.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.880+0000 7fd68af22700 1 -- 172.21.15.102:0/3229298370 wait complete. 2024-03-26T18:54:21.874 INFO:tasks.workunit.client.1.smithi190.stdout:1/99: write d3/da/f1f [631338,12902] 0 2024-03-26T18:54:21.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.880+0000 7fd68af22700 1 Processor -- start 2024-03-26T18:54:21.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd68af22700 1 -- start start 2024-03-26T18:54:21.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd68af22700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd684100be0 0x7fd684100540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:21.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd68af22700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd684103500 0x7fd6840feb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:21.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd68af22700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6841995d0 con 0x7fd684103500 2024-03-26T18:54:21.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd68af22700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd684199710 con 0x7fd684100be0 2024-03-26T18:54:21.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd683fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd684103500 0x7fd6840feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:21.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd683fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd684103500 0x7fd6840feb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34272/0 (socket says 172.21.15.102:34272) 2024-03-26T18:54:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd683fff700 1 -- 172.21.15.102:0/3792927225 learned_addr learned my addr 172.21.15.102:0/3792927225 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd688cbe700 1 --2- 172.21.15.102:0/3792927225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd684100be0 0x7fd684100540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd683fff700 1 -- 172.21.15.102:0/3792927225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd684100be0 msgr2=0x7fd684100540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd683fff700 1 --2- 172.21.15.102:0/3792927225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd684100be0 0x7fd684100540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.881+0000 7fd683fff700 1 -- 172.21.15.102:0/3792927225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6700095f0 con 0x7fd684103500 2024-03-26T18:54:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.882+0000 7fd683fff700 1 --2- 172.21.15.102:0/3792927225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd684103500 0x7fd6840feb90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd670006010 tx=0x7fd67000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.882+0000 7fd681ffb700 1 -- 172.21.15.102:0/3792927225 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd67001d070 con 0x7fd684103500 2024-03-26T18:54:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.882+0000 7fd68af22700 1 -- 172.21.15.102:0/3792927225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6840ff0d0 con 0x7fd684103500 2024-03-26T18:54:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.882+0000 7fd681ffb700 1 -- 172.21.15.102:0/3792927225 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd67000f460 con 0x7fd684103500 2024-03-26T18:54:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.882+0000 7fd681ffb700 1 -- 172.21.15.102:0/3792927225 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd670021640 con 0x7fd684103500 2024-03-26T18:54:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.882+0000 7fd68af22700 1 -- 172.21.15.102:0/3792927225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6840ff590 con 0x7fd684103500 2024-03-26T18:54:21.877 INFO:tasks.workunit.client.1.smithi190.stdout:2/82: mknod d6/c18 0 2024-03-26T18:54:21.877 INFO:tasks.workunit.client.1.smithi190.stdout:2/83: dread d6/fe [0,4194304] 0 2024-03-26T18:54:21.878 INFO:tasks.workunit.client.1.smithi190.stdout:6/126: mkdir d17/d28/d2c 0 2024-03-26T18:54:21.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.883+0000 7fd68af22700 1 -- 172.21.15.102:0/3792927225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68418fbf0 con 0x7fd684103500 2024-03-26T18:54:21.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.884+0000 7fd681ffb700 1 -- 172.21.15.102:0/3792927225 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fd67000fab0 con 0x7fd684103500 2024-03-26T18:54:21.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.884+0000 7fd681ffb700 1 --2- 172.21.15.102:0/3792927225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd67406c680 0x7fd67406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:21.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.885+0000 7fd681ffb700 1 -- 172.21.15.102:0/3792927225 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd67008cdc0 con 0x7fd684103500 2024-03-26T18:54:21.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.885+0000 7fd688cbe700 1 --2- 172.21.15.102:0/3792927225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd67406c680 0x7fd67406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:21.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.885+0000 7fd688cbe700 1 --2- 172.21.15.102:0/3792927225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd67406c680 0x7fd67406eb30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd678006fd0 tx=0x7fd678008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:21.879 INFO:tasks.workunit.client.1.smithi190.stdout:3/79: symlink d1/l1f 0 2024-03-26T18:54:21.879 INFO:tasks.workunit.client.1.smithi190.stdout:3/80: write d1/d3/f10 [622847,14457] 0 2024-03-26T18:54:21.880 INFO:tasks.workunit.client.1.smithi190.stdout:4/103: getdents d7/d8 0 2024-03-26T18:54:21.882 INFO:tasks.workunit.client.1.smithi190.stdout:8/79: creat d9/dd/d15/f18 x:0 0 0 2024-03-26T18:54:21.884 INFO:tasks.workunit.client.1.smithi190.stdout:0/196: rmdir d0/dd/d20/d43 0 2024-03-26T18:54:21.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:21.893+0000 7fd681ffb700 1 -- 172.21.15.102:0/3792927225 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd67005aed0 con 0x7fd684103500 2024-03-26T18:54:21.892 INFO:tasks.workunit.client.1.smithi190.stdout:2/84: mknod d6/c19 0 2024-03-26T18:54:21.895 INFO:tasks.workunit.client.1.smithi190.stdout:6/127: creat d17/d28/f2d x:0 0 0 2024-03-26T18:54:21.899 INFO:tasks.workunit.client.1.smithi190.stdout:3/81: link d1/c16 d1/de/c20 0 2024-03-26T18:54:21.899 INFO:tasks.workunit.client.1.smithi190.stdout:3/82: readlink d1/d3/l17 0 2024-03-26T18:54:21.902 INFO:tasks.workunit.client.1.smithi190.stdout:4/104: mkdir d7/d8/d14/d23 0 2024-03-26T18:54:21.902 INFO:tasks.workunit.client.1.smithi190.stdout:4/105: dread - d7/d8/f12 zero size 2024-03-26T18:54:21.902 INFO:tasks.workunit.client.1.smithi190.stdout:4/106: chown d7/f18 794 1 2024-03-26T18:54:21.904 INFO:tasks.workunit.client.1.smithi190.stdout:0/197: symlink d0/dd/l44 0 2024-03-26T18:54:21.909 INFO:tasks.workunit.client.1.smithi190.stdout:6/128: write d17/f1b [344338,27067] 0 2024-03-26T18:54:21.909 INFO:tasks.workunit.client.1.smithi190.stdout:6/129: fsync f2 0 2024-03-26T18:54:21.911 INFO:tasks.workunit.client.1.smithi190.stdout:4/107: mknod d7/d8/d14/c24 0 2024-03-26T18:54:21.912 INFO:tasks.workunit.client.1.smithi190.stdout:8/80: dwrite f6 [0,4194304] 0 2024-03-26T18:54:21.912 INFO:tasks.workunit.client.1.smithi190.stdout:8/81: chown d9 334226 1 2024-03-26T18:54:21.912 INFO:tasks.workunit.client.1.smithi190.stdout:8/82: chown d9/ff 417 1 2024-03-26T18:54:21.912 INFO:tasks.workunit.client.1.smithi190.stdout:2/85: dwrite d6/d9/d10/f13 [0,4194304] 0 2024-03-26T18:54:21.915 INFO:tasks.workunit.client.1.smithi190.stdout:6/130: link f1 d17/d28/d2c/f2e 0 2024-03-26T18:54:21.915 INFO:tasks.workunit.client.1.smithi190.stdout:6/131: truncate f1 2572802 0 2024-03-26T18:54:21.917 INFO:tasks.workunit.client.1.smithi190.stdout:4/108: symlink d7/d8/d1c/l25 0 2024-03-26T18:54:21.918 INFO:tasks.workunit.client.1.smithi190.stdout:0/198: dread d0/dd/d24/f2b [0,4194304] 0 2024-03-26T18:54:21.918 INFO:tasks.workunit.client.1.smithi190.stdout:0/199: readlink d0/dd/l1b 0 2024-03-26T18:54:21.931 INFO:tasks.workunit.client.1.smithi190.stdout:6/132: write d17/f23 [608836,2574] 0 2024-03-26T18:54:21.933 INFO:tasks.workunit.client.1.smithi190.stdout:0/200: rmdir d0/dc 39 2024-03-26T18:54:21.933 INFO:tasks.workunit.client.1.smithi190.stdout:0/201: chown d0/dd/d20/c40 103 1 2024-03-26T18:54:21.934 INFO:tasks.workunit.client.1.smithi190.stdout:6/133: readlink d17/l1f 0 2024-03-26T18:54:21.936 INFO:tasks.workunit.client.1.smithi190.stdout:0/202: truncate d0/dd/d26/f34 2211756 0 2024-03-26T18:54:21.936 INFO:tasks.workunit.client.1.smithi190.stdout:0/203: creat d0/dd/f45 x:0 0 0 2024-03-26T18:54:21.938 INFO:tasks.workunit.client.1.smithi190.stdout:6/134: rename d17 to d17/d2f 22 2024-03-26T18:54:21.941 INFO:tasks.workunit.client.1.smithi190.stdout:0/204: mkdir d0/dd/d20/d1c/d46 0 2024-03-26T18:54:21.941 INFO:tasks.workunit.client.1.smithi190.stdout:0/205: chown d0/dd/fe 27296973 1 2024-03-26T18:54:21.941 INFO:tasks.workunit.client.1.smithi190.stdout:0/206: truncate d0/f1f 5692083 0 2024-03-26T18:54:21.941 INFO:tasks.workunit.client.1.smithi190.stdout:0/207: stat d0/f6 0 2024-03-26T18:54:21.944 INFO:tasks.workunit.client.1.smithi190.stdout:0/208: mknod d0/dd/d20/c47 0 2024-03-26T18:54:21.946 INFO:tasks.workunit.client.1.smithi190.stdout:6/135: symlink d17/l30 0 2024-03-26T18:54:21.946 INFO:tasks.workunit.client.1.smithi190.stdout:6/136: chown c14 3933079 1 2024-03-26T18:54:21.948 INFO:tasks.workunit.client.1.smithi190.stdout:0/209: link d0/dd/d24/c42 d0/dd/d24/c48 0 2024-03-26T18:54:21.948 INFO:tasks.workunit.client.1.smithi190.stdout:3/83: dwrite d1/f14 [0,4194304] 0 2024-03-26T18:54:21.949 INFO:tasks.workunit.client.1.smithi190.stdout:2/86: dwrite d6/f7 [0,4194304] 0 2024-03-26T18:54:21.949 INFO:tasks.workunit.client.1.smithi190.stdout:2/87: fsync d6/df/f16 0 2024-03-26T18:54:21.949 INFO:tasks.workunit.client.1.smithi190.stdout:4/109: dwrite d7/d8/d14/f19 [4194304,4194304] 0 2024-03-26T18:54:21.950 INFO:tasks.workunit.client.1.smithi190.stdout:1/100: dwrite d3/f7 [0,4194304] 0 2024-03-26T18:54:21.950 INFO:tasks.workunit.client.1.smithi190.stdout:1/101: chown d3/d9/fe 288437473 1 2024-03-26T18:54:21.950 INFO:tasks.workunit.client.1.smithi190.stdout:6/137: rename d17/l29 to d17/d28/d2b/l31 0 2024-03-26T18:54:21.968 INFO:tasks.workunit.client.1.smithi190.stdout:1/102: symlink d3/d9/db/l22 0 2024-03-26T18:54:21.975 INFO:tasks.workunit.client.1.smithi190.stdout:6/138: rename c5 to d17/d28/c32 0 2024-03-26T18:54:21.983 INFO:tasks.workunit.client.1.smithi190.stdout:1/103: rename d3/d9/ff to d3/d9/db/f23 0 2024-03-26T18:54:21.983 INFO:tasks.workunit.client.1.smithi190.stdout:1/104: chown d3/da 10668 1 2024-03-26T18:54:21.983 INFO:tasks.workunit.client.1.smithi190.stdout:1/105: write d3/da/f1f [2315774,82384] 0 2024-03-26T18:54:21.983 INFO:tasks.workunit.client.1.smithi190.stdout:4/110: dwrite d7/f17 [0,4194304] 0 2024-03-26T18:54:21.986 INFO:tasks.workunit.client.1.smithi190.stdout:6/139: symlink d17/d28/l33 0 2024-03-26T18:54:21.991 INFO:tasks.workunit.client.1.smithi190.stdout:6/140: chown d17/f19 66950 1 2024-03-26T18:54:21.991 INFO:tasks.workunit.client.1.smithi190.stdout:6/141: rename d17/d28 to d17/d28/d2c/d34 22 2024-03-26T18:54:21.991 INFO:tasks.workunit.client.1.smithi190.stdout:6/142: write d17/f1b [633060,31196] 0 2024-03-26T18:54:21.999 INFO:tasks.workunit.client.1.smithi190.stdout:2/88: dwrite d6/fe [0,4194304] 0 2024-03-26T18:54:21.999 INFO:tasks.workunit.client.1.smithi190.stdout:8/83: dwrite f6 [4194304,4194304] 0 2024-03-26T18:54:21.999 INFO:tasks.workunit.client.1.smithi190.stdout:8/84: fdatasync d9/ff 0 2024-03-26T18:54:21.999 INFO:tasks.workunit.client.1.smithi190.stdout:8/85: chown l3 8 1 2024-03-26T18:54:21.999 INFO:tasks.workunit.client.1.smithi190.stdout:8/86: getdents d9/d16 0 2024-03-26T18:54:21.999 INFO:tasks.workunit.client.1.smithi190.stdout:3/84: dwrite d1/f5 [0,4194304] 0 2024-03-26T18:54:21.999 INFO:tasks.workunit.client.1.smithi190.stdout:3/85: chown d1/d3/df 1249081 1 2024-03-26T18:54:21.999 INFO:tasks.workunit.client.1.smithi190.stdout:3/86: fsync d1/d3/f10 0 2024-03-26T18:54:22.000 INFO:tasks.workunit.client.1.smithi190.stdout:1/106: dread d3/d9/f15 [0,4194304] 0 2024-03-26T18:54:22.005 INFO:tasks.workunit.client.1.smithi190.stdout:2/89: mkdir d6/d9/d1a 0 2024-03-26T18:54:22.006 INFO:tasks.workunit.client.1.smithi190.stdout:3/87: dread d1/f4 [0,4194304] 0 2024-03-26T18:54:22.006 INFO:tasks.workunit.client.1.smithi190.stdout:0/210: truncate d0/f1f 5585530 0 2024-03-26T18:54:22.010 INFO:tasks.workunit.client.1.smithi190.stdout:8/87: mknod d9/d16/c19 0 2024-03-26T18:54:22.010 INFO:tasks.workunit.client.1.smithi190.stdout:8/88: fsync d9/dd/d15/f18 0 2024-03-26T18:54:22.010 INFO:tasks.workunit.client.1.smithi190.stdout:8/89: creat d9/f1a x:0 0 0 2024-03-26T18:54:22.010 INFO:tasks.workunit.client.1.smithi190.stdout:8/90: truncate d9/fa 1174864 0 2024-03-26T18:54:22.019 INFO:tasks.workunit.client.1.smithi190.stdout:8/91: dread d9/f12 [0,4194304] 0 2024-03-26T18:54:22.019 INFO:tasks.workunit.client.1.smithi190.stdout:8/92: dread - d9/f10 zero size 2024-03-26T18:54:22.022 INFO:tasks.workunit.client.1.smithi190.stdout:3/88: creat d1/d3/df/d18/d1b/f21 x:0 0 0 2024-03-26T18:54:22.023 INFO:tasks.workunit.client.1.smithi190.stdout:0/211: creat d0/dd/d24/f49 x:0 0 0 2024-03-26T18:54:22.024 INFO:tasks.workunit.client.1.smithi190.stdout:1/107: dwrite d3/d9/f13 [4194304,4194304] 0 2024-03-26T18:54:22.027 INFO:tasks.workunit.client.1.smithi190.stdout:8/93: rename d9/f10 to d9/d16/f1b 0 2024-03-26T18:54:22.030 INFO:tasks.workunit.client.1.smithi190.stdout:2/90: getdents d6/d9 0 2024-03-26T18:54:22.030 INFO:tasks.workunit.client.1.smithi190.stdout:2/91: fdatasync d6/df/f16 0 2024-03-26T18:54:22.030 INFO:tasks.workunit.client.1.smithi190.stdout:2/92: chown f5 5 1 2024-03-26T18:54:22.031 INFO:tasks.workunit.client.1.smithi190.stdout:0/212: mknod d0/c4a 0 2024-03-26T18:54:22.031 INFO:tasks.workunit.client.1.smithi190.stdout:0/213: fdatasync d0/dd/d26/f34 0 2024-03-26T18:54:22.037 INFO:tasks.workunit.client.1.smithi190.stdout:3/89: rename d1/cc to d1/d3/df/c22 0 2024-03-26T18:54:22.037 INFO:tasks.workunit.client.1.smithi190.stdout:3/90: truncate d1/f11 471063 0 2024-03-26T18:54:22.038 INFO:tasks.workunit.client.1.smithi190.stdout:1/108: dwrite d3/d9/f18 [0,4194304] 0 2024-03-26T18:54:22.039 INFO:tasks.workunit.client.1.smithi190.stdout:1/109: fdatasync f2 0 2024-03-26T18:54:22.039 INFO:tasks.workunit.client.1.smithi190.stdout:2/93: rmdir d6/d9/d10 39 2024-03-26T18:54:22.039 INFO:tasks.workunit.client.1.smithi190.stdout:2/94: write d6/df/f16 [529678,84981] 0 2024-03-26T18:54:22.052 INFO:tasks.workunit.client.1.smithi190.stdout:8/94: dwrite d9/fa [0,4194304] 0 2024-03-26T18:54:22.058 INFO:tasks.workunit.client.1.smithi190.stdout:1/110: rename d3/f7 to d3/da/f24 0 2024-03-26T18:54:22.060 INFO:tasks.workunit.client.1.smithi190.stdout:2/95: rename d6/d9/lb to d6/d9/l1b 0 2024-03-26T18:54:22.062 INFO:tasks.workunit.client.1.smithi190.stdout:3/91: getdents d1 0 2024-03-26T18:54:22.064 INFO:tasks.workunit.client.1.smithi190.stdout:0/214: truncate d0/f1f 4226055 0 2024-03-26T18:54:22.064 INFO:tasks.workunit.client.1.smithi190.stdout:0/215: truncate d0/dd/d20/f30 259679 0 2024-03-26T18:54:22.072 INFO:tasks.workunit.client.1.smithi190.stdout:1/111: link d3/l21 d3/d9/l25 0 2024-03-26T18:54:22.074 INFO:tasks.workunit.client.1.smithi190.stdout:2/96: mkdir d6/df/d1c 0 2024-03-26T18:54:22.076 INFO:tasks.workunit.client.1.smithi190.stdout:0/216: mkdir d0/dd/d24/d4b 0 2024-03-26T18:54:22.079 INFO:tasks.workunit.client.1.smithi190.stdout:8/95: dwrite d9/dd/f17 [0,4194304] 0 2024-03-26T18:54:22.079 INFO:tasks.workunit.client.1.smithi190.stdout:8/96: chown l5 9 1 2024-03-26T18:54:22.079 INFO:tasks.workunit.client.1.smithi190.stdout:8/97: readlink l5 0 2024-03-26T18:54:22.079 INFO:tasks.workunit.client.1.smithi190.stdout:2/97: creat d6/d9/f1d x:0 0 0 2024-03-26T18:54:22.079 INFO:tasks.workunit.client.1.smithi190.stdout:2/98: stat d6/d9/d10/f13 0 2024-03-26T18:54:22.079 INFO:tasks.workunit.client.1.smithi190.stdout:2/99: getdents d6/d9/d1a 0 2024-03-26T18:54:22.081 INFO:tasks.workunit.client.1.smithi190.stdout:0/217: link d0/dd/d24/c2d d0/dd/d26/c4c 0 2024-03-26T18:54:22.084 INFO:tasks.workunit.client.1.smithi190.stdout:1/112: write d3/d9/f13 [1589587,3656] 0 2024-03-26T18:54:22.084 INFO:tasks.workunit.client.1.smithi190.stdout:1/113: fdatasync f2 0 2024-03-26T18:54:22.085 INFO:tasks.workunit.client.1.smithi190.stdout:3/92: dwrite d1/f5 [0,4194304] 0 2024-03-26T18:54:22.085 INFO:tasks.workunit.client.1.smithi190.stdout:3/93: truncate d1/d3/df/f19 202170 0 2024-03-26T18:54:22.088 INFO:tasks.workunit.client.1.smithi190.stdout:8/98: rename d9/ff to d9/dd/d15/f1c 0 2024-03-26T18:54:22.089 INFO:tasks.workunit.client.1.smithi190.stdout:8/99: truncate d9/dd/d15/f18 400659 0 2024-03-26T18:54:22.090 INFO:tasks.workunit.client.1.smithi190.stdout:1/114: symlink d3/d9/db/l26 0 2024-03-26T18:54:22.096 INFO:tasks.workunit.client.1.smithi190.stdout:3/94: write d1/d3/f10 [3480790,64076] 0 2024-03-26T18:54:22.097 INFO:tasks.workunit.client.1.smithi190.stdout:3/95: dread d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:22.097 INFO:tasks.workunit.client.1.smithi190.stdout:3/96: readlink d1/l1f 0 2024-03-26T18:54:22.097 INFO:tasks.workunit.client.1.smithi190.stdout:3/97: chown d1/d3/ca 2564783 1 2024-03-26T18:54:22.097 INFO:tasks.workunit.client.1.smithi190.stdout:3/98: truncate d1/d3/f10 4534369 0 2024-03-26T18:54:22.097 INFO:tasks.workunit.client.1.smithi190.stdout:1/115: creat d3/f27 x:0 0 0 2024-03-26T18:54:22.100 INFO:tasks.workunit.client.1.smithi190.stdout:3/99: symlink d1/l23 0 2024-03-26T18:54:22.102 INFO:tasks.workunit.client.1.smithi190.stdout:2/100: dwrite d6/df/f16 [0,4194304] 0 2024-03-26T18:54:22.102 INFO:tasks.workunit.client.1.smithi190.stdout:2/101: creat d6/d9/f1e x:0 0 0 2024-03-26T18:54:22.102 INFO:tasks.workunit.client.1.smithi190.stdout:2/102: chown d6/fe 184054936 1 2024-03-26T18:54:22.102 INFO:tasks.workunit.client.1.smithi190.stdout:2/103: chown d6/d9/f1d 45 1 2024-03-26T18:54:22.102 INFO:tasks.workunit.client.1.smithi190.stdout:2/104: truncate d6/d9/f1e 110687 0 2024-03-26T18:54:22.103 INFO:tasks.workunit.client.1.smithi190.stdout:3/100: rename d1/d3/df/c22 to d1/c24 0 2024-03-26T18:54:22.103 INFO:tasks.workunit.client.1.smithi190.stdout:3/101: truncate d1/f15 1039620 0 2024-03-26T18:54:22.114 INFO:tasks.workunit.client.1.smithi190.stdout:3/102: symlink d1/d3/l25 0 2024-03-26T18:54:22.115 INFO:tasks.workunit.client.1.smithi190.stdout:2/105: dwrite d6/d9/f1d [0,4194304] 0 2024-03-26T18:54:22.115 INFO:tasks.workunit.client.1.smithi190.stdout:2/106: readlink d6/d9/d10/l15 0 2024-03-26T18:54:22.121 INFO:tasks.workunit.client.1.smithi190.stdout:8/100: getdents d9/dd/d15 0 2024-03-26T18:54:22.121 INFO:tasks.workunit.client.1.smithi190.stdout:3/103: dread d1/fb [0,4194304] 0 2024-03-26T18:54:22.135 INFO:tasks.workunit.client.1.smithi190.stdout:2/107: unlink d6/d9/f1e 0 2024-03-26T18:54:22.135 INFO:tasks.workunit.client.1.smithi190.stdout:2/108: chown d6/d9 147328 1 2024-03-26T18:54:22.138 INFO:tasks.workunit.client.1.smithi190.stdout:8/101: truncate f8 4018966 0 2024-03-26T18:54:22.138 INFO:tasks.workunit.client.1.smithi190.stdout:3/104: dread d1/f11 [0,4194304] 0 2024-03-26T18:54:22.138 INFO:tasks.workunit.client.1.smithi190.stdout:3/105: write d1/f15 [2032873,77859] 0 2024-03-26T18:54:22.139 INFO:tasks.workunit.client.1.smithi190.stdout:3/106: dread d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:22.139 INFO:tasks.workunit.client.1.smithi190.stdout:2/109: symlink d6/d9/d10/l1f 0 2024-03-26T18:54:22.140 INFO:tasks.workunit.client.1.smithi190.stdout:2/110: creat d6/f20 x:0 0 0 2024-03-26T18:54:22.141 INFO:tasks.workunit.client.1.smithi190.stdout:8/102: rmdir d9/d16 39 2024-03-26T18:54:22.142 INFO:tasks.workunit.client.1.smithi190.stdout:8/103: read d9/f14 [834952,88270] 0 2024-03-26T18:54:22.146 INFO:tasks.workunit.client.1.smithi190.stdout:3/107: creat d1/f26 x:0 0 0 2024-03-26T18:54:22.146 INFO:tasks.workunit.client.1.smithi190.stdout:3/108: stat d1/d3/l25 0 2024-03-26T18:54:22.146 INFO:tasks.workunit.client.1.smithi190.stdout:3/109: dread - d1/de/f1d zero size 2024-03-26T18:54:22.147 INFO:tasks.workunit.client.1.smithi190.stdout:3/110: dread - d1/f26 zero size 2024-03-26T18:54:22.147 INFO:tasks.workunit.client.1.smithi190.stdout:3/111: fsync d1/f4 0 2024-03-26T18:54:22.147 INFO:tasks.workunit.client.1.smithi190.stdout:3/112: fsync d1/fb 0 2024-03-26T18:54:22.147 INFO:tasks.workunit.client.1.smithi190.stdout:3/113: read - d1/de/f1d zero size 2024-03-26T18:54:22.147 INFO:tasks.workunit.client.1.smithi190.stdout:3/114: read d1/f15 [2038339,100752] 0 2024-03-26T18:54:22.147 INFO:tasks.workunit.client.1.smithi190.stdout:3/115: chown d1/fb 54149118 1 2024-03-26T18:54:22.149 INFO:tasks.workunit.client.1.smithi190.stdout:3/116: dread d1/fb [0,4194304] 0 2024-03-26T18:54:22.150 INFO:tasks.workunit.client.1.smithi190.stdout:8/104: mknod d9/dd/c1d 0 2024-03-26T18:54:22.153 INFO:tasks.workunit.client.1.smithi190.stdout:3/117: rename d1/d3/l25 to d1/d3/df/d18/d1b/l27 0 2024-03-26T18:54:22.153 INFO:tasks.workunit.client.1.smithi190.stdout:3/118: stat d1/f5 0 2024-03-26T18:54:22.153 INFO:tasks.workunit.client.1.smithi190.stdout:3/119: dread - d1/d3/df/d18/d1b/f21 zero size 2024-03-26T18:54:22.153 INFO:tasks.workunit.client.1.smithi190.stdout:3/120: stat d1/cd 0 2024-03-26T18:54:22.155 INFO:tasks.workunit.client.1.smithi190.stdout:8/105: creat d9/d16/f1e x:0 0 0 2024-03-26T18:54:22.155 INFO:tasks.workunit.client.1.smithi190.stdout:8/106: creat d9/d16/f1f x:0 0 0 2024-03-26T18:54:22.155 INFO:tasks.workunit.client.1.smithi190.stdout:2/111: dwrite d6/d9/d10/f13 [0,4194304] 0 2024-03-26T18:54:22.158 INFO:tasks.workunit.client.1.smithi190.stdout:8/107: unlink d9/cc 0 2024-03-26T18:54:22.159 INFO:tasks.workunit.client.1.smithi190.stdout:2/112: write f5 [578246,38894] 0 2024-03-26T18:54:22.160 INFO:tasks.workunit.client.1.smithi190.stdout:2/113: dread - d6/f20 zero size 2024-03-26T18:54:22.160 INFO:tasks.workunit.client.1.smithi190.stdout:3/121: dread d1/d3/f10 [0,4194304] 0 2024-03-26T18:54:22.160 INFO:tasks.workunit.client.1.smithi190.stdout:3/122: creat d1/d3/f28 x:0 0 0 2024-03-26T18:54:22.160 INFO:tasks.workunit.client.1.smithi190.stdout:3/123: creat d1/de/f29 x:0 0 0 2024-03-26T18:54:22.174 INFO:tasks.workunit.client.1.smithi190.stdout:2/114: write d6/f7 [2673619,80590] 0 2024-03-26T18:54:22.174 INFO:tasks.workunit.client.1.smithi190.stdout:2/115: fsync d6/fe 0 2024-03-26T18:54:22.175 INFO:tasks.workunit.client.1.smithi190.stdout:3/124: getdents d1/de 0 2024-03-26T18:54:22.175 INFO:tasks.workunit.client.1.smithi190.stdout:3/125: fsync d1/f26 0 2024-03-26T18:54:22.175 INFO:tasks.workunit.client.1.smithi190.stdout:5/46: sync 2024-03-26T18:54:22.178 INFO:tasks.workunit.client.1.smithi190.stdout:2/116: mkdir d6/d21 0 2024-03-26T18:54:22.179 INFO:tasks.workunit.client.1.smithi190.stdout:8/108: dwrite f6 [0,4194304] 0 2024-03-26T18:54:22.180 INFO:tasks.workunit.client.1.smithi190.stdout:3/126: dread d1/d3/f10 [0,4194304] 0 2024-03-26T18:54:22.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.209+0000 7fd68af22700 1 -- 172.21.15.102:0/3792927225 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd68402cfa0 con 0x7fd67406c680 2024-03-26T18:54:22.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.219+0000 7fd681ffb700 1 -- 172.21.15.102:0/3792927225 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3528 (secure 0 0 0) 0x7fd68402cfa0 con 0x7fd67406c680 2024-03-26T18:54:22.212 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:54:22.212 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (5m) 92s ago 7m 25.5M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:54:22.212 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (7m) 92s ago 7m 7457k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:54:22.212 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (6m) 98s ago 6m 7356k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:54:22.212 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (7m) 92s ago 7m 6665k - 18.2.0 dc2bc1663786 bfac00217c68 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (6m) 98s ago 6m 6656k - 18.2.0 dc2bc1663786 7379f1f6b746 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (5m) 92s ago 7m 85.5M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (112s) 92s ago 111s 15.9M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (118s) 92s ago 117s 11.5M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (109s) 98s ago 109s 12.1M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (115s) 98s ago 115s 14.9M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:9283,8765,8443 running (8m) 92s ago 8m 487M - 18.2.0 dc2bc1663786 61d2bd7101b6 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (6m) 98s ago 6m 440M - 18.2.0 dc2bc1663786 57e44bfc8ede 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (8m) 92s ago 8m 45.1M 2048M 18.2.0 dc2bc1663786 b5abd46c3973 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (6m) 98s ago 6m 35.3M 2048M 18.2.0 dc2bc1663786 6bd913d483f0 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (7m) 92s ago 7m 15.9M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (6m) 98s ago 6m 15.3M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (5m) 92s ago 5m 47.9M 3946M 18.2.0 dc2bc1663786 3e2065fb5fbc 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (5m) 92s ago 5m 48.5M 3946M 18.2.0 dc2bc1663786 0bf773515cc1 2024-03-26T18:54:22.213 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (4m) 92s ago 4m 46.3M 3946M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:54:22.214 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (4m) 98s ago 4m 49.3M 4970M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:54:22.214 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (3m) 98s ago 3m 44.9M 4970M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:54:22.214 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (3m) 98s ago 3m 44.1M 4970M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:54:22.214 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (5m) 92s ago 6m 61.7M - 2.43.0 a07b618ecd1d d0dc2ca7f41f 2024-03-26T18:54:22.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.221+0000 7fd68af22700 1 -- 172.21.15.102:0/3792927225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd67406c680 msgr2=0x7fd67406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:22.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.221+0000 7fd68af22700 1 --2- 172.21.15.102:0/3792927225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd67406c680 0x7fd67406eb30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd678006fd0 tx=0x7fd678008040 comp rx=0 tx=0).stop 2024-03-26T18:54:22.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.221+0000 7fd68af22700 1 -- 172.21.15.102:0/3792927225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd684103500 msgr2=0x7fd6840feb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:22.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.221+0000 7fd68af22700 1 --2- 172.21.15.102:0/3792927225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd684103500 0x7fd6840feb90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd670006010 tx=0x7fd67000be20 comp rx=0 tx=0).stop 2024-03-26T18:54:22.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.221+0000 7fd68af22700 1 -- 172.21.15.102:0/3792927225 shutdown_connections 2024-03-26T18:54:22.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.221+0000 7fd68af22700 1 --2- 172.21.15.102:0/3792927225 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd67406c680 0x7fd67406eb30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:22.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.221+0000 7fd68af22700 1 --2- 172.21.15.102:0/3792927225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd684100be0 0x7fd684100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:22.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.222+0000 7fd68af22700 1 --2- 172.21.15.102:0/3792927225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd684103500 0x7fd6840feb90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:22.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.222+0000 7fd68af22700 1 -- 172.21.15.102:0/3792927225 >> 172.21.15.102:0/3792927225 conn(0x7fd6840fa600 msgr2=0x7fd6840fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:22.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.222+0000 7fd68af22700 1 -- 172.21.15.102:0/3792927225 shutdown_connections 2024-03-26T18:54:22.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.222+0000 7fd68af22700 1 -- 172.21.15.102:0/3792927225 wait complete. 2024-03-26T18:54:22.229 INFO:tasks.workunit.client.1.smithi190.stdout:5/47: symlink d2/lc 0 2024-03-26T18:54:22.229 INFO:tasks.workunit.client.1.smithi190.stdout:5/48: creat d2/fd x:0 0 0 2024-03-26T18:54:22.230 INFO:tasks.workunit.client.1.smithi190.stdout:2/117: rename d6/f20 to d6/df/f22 0 2024-03-26T18:54:22.231 INFO:tasks.workunit.client.1.smithi190.stdout:8/109: truncate d9/dd/d15/f1c 755447 0 2024-03-26T18:54:22.232 INFO:tasks.workunit.client.1.smithi190.stdout:3/127: creat d1/d3/df/f2a x:0 0 0 2024-03-26T18:54:22.234 INFO:tasks.workunit.client.1.smithi190.stdout:5/49: rename d2/lc to d2/le 0 2024-03-26T18:54:22.234 INFO:tasks.workunit.client.1.smithi190.stdout:2/118: stat d6/d9/l1b 0 2024-03-26T18:54:22.236 INFO:tasks.workunit.client.1.smithi190.stdout:3/128: link d1/de/f1d d1/de/d1c/f2b 0 2024-03-26T18:54:22.237 INFO:tasks.workunit.client.1.smithi190.stdout:5/50: symlink d2/lf 0 2024-03-26T18:54:22.237 INFO:tasks.workunit.client.1.smithi190.stdout:5/51: creat d2/f10 x:0 0 0 2024-03-26T18:54:22.238 INFO:tasks.workunit.client.1.smithi190.stdout:2/119: mknod d6/d21/c23 0 2024-03-26T18:54:22.238 INFO:tasks.workunit.client.1.smithi190.stdout:2/120: readlink d6/d9/l1b 0 2024-03-26T18:54:22.242 INFO:tasks.workunit.client.1.smithi190.stdout:5/52: link d2/c9 d2/c11 0 2024-03-26T18:54:22.244 INFO:tasks.workunit.client.1.smithi190.stdout:2/121: mknod d6/d9/c24 0 2024-03-26T18:54:22.244 INFO:tasks.workunit.client.1.smithi190.stdout:8/110: dread d9/dd/f17 [0,4194304] 0 2024-03-26T18:54:22.245 INFO:tasks.workunit.client.1.smithi190.stdout:8/111: chown d9/dd/c1d 0 1 2024-03-26T18:54:22.247 INFO:tasks.workunit.client.1.smithi190.stdout:2/122: unlink d6/d9/c24 0 2024-03-26T18:54:22.249 INFO:tasks.workunit.client.1.smithi190.stdout:7/106: sync 2024-03-26T18:54:22.249 INFO:tasks.workunit.client.1.smithi190.stdout:0/218: sync 2024-03-26T18:54:22.249 INFO:tasks.workunit.client.1.smithi190.stdout:7/107: stat d3/d6/d9/d14/f18 0 2024-03-26T18:54:22.249 INFO:tasks.workunit.client.1.smithi190.stdout:7/108: fdatasync d3/d6/d9/d14/f18 0 2024-03-26T18:54:22.250 INFO:tasks.workunit.client.1.smithi190.stdout:7/109: creat d3/d6/d9/d14/f2a x:0 0 0 2024-03-26T18:54:22.250 INFO:tasks.workunit.client.1.smithi190.stdout:4/111: sync 2024-03-26T18:54:22.250 INFO:tasks.workunit.client.1.smithi190.stdout:1/116: sync 2024-03-26T18:54:22.250 INFO:tasks.workunit.client.1.smithi190.stdout:9/142: sync 2024-03-26T18:54:22.250 INFO:tasks.workunit.client.1.smithi190.stdout:1/117: stat d3/da/dd/f12 0 2024-03-26T18:54:22.250 INFO:tasks.workunit.client.1.smithi190.stdout:9/143: write d12/d2d/f22 [945152,6520] 0 2024-03-26T18:54:22.250 INFO:tasks.workunit.client.1.smithi190.stdout:6/143: sync 2024-03-26T18:54:22.250 INFO:tasks.workunit.client.1.smithi190.stdout:6/144: write f8 [1315915,46738] 0 2024-03-26T18:54:22.250 INFO:tasks.workunit.client.1.smithi190.stdout:8/112: rename d9/dd/f17 to d9/dd/f20 0 2024-03-26T18:54:22.253 INFO:tasks.workunit.client.1.smithi190.stdout:4/112: creat d7/d1d/f26 x:0 0 0 2024-03-26T18:54:22.253 INFO:tasks.workunit.client.1.smithi190.stdout:4/113: creat d7/d1d/f27 x:0 0 0 2024-03-26T18:54:22.253 INFO:tasks.workunit.client.1.smithi190.stdout:4/114: dread - d7/d1d/f27 zero size 2024-03-26T18:54:22.254 INFO:tasks.workunit.client.1.smithi190.stdout:5/53: getdents d2 0 2024-03-26T18:54:22.256 INFO:tasks.workunit.client.1.smithi190.stdout:9/144: creat d12/d2d/d1d/f3a x:0 0 0 2024-03-26T18:54:22.258 INFO:tasks.workunit.client.1.smithi190.stdout:8/113: symlink d9/l21 0 2024-03-26T18:54:22.261 INFO:tasks.workunit.client.1.smithi190.stdout:4/115: mkdir d7/d8/d14/d23/d28 0 2024-03-26T18:54:22.262 INFO:tasks.workunit.client.1.smithi190.stdout:4/116: fsync d7/f18 0 2024-03-26T18:54:22.262 INFO:tasks.workunit.client.1.smithi190.stdout:3/129: dwrite d1/d3/df/f2a [0,4194304] 0 2024-03-26T18:54:22.262 INFO:tasks.workunit.client.1.smithi190.stdout:3/130: dread - d1/f26 zero size 2024-03-26T18:54:22.263 INFO:tasks.workunit.client.1.smithi190.stdout:3/131: creat d1/f2c x:0 0 0 2024-03-26T18:54:22.263 INFO:tasks.workunit.client.1.smithi190.stdout:3/132: fdatasync d1/f2c 0 2024-03-26T18:54:22.263 INFO:tasks.workunit.client.1.smithi190.stdout:5/54: rename d2/fd to d2/f12 0 2024-03-26T18:54:22.265 INFO:tasks.workunit.client.1.smithi190.stdout:9/145: unlink d12/d2d/d1d/d23/d2b/c2e 0 2024-03-26T18:54:22.265 INFO:tasks.workunit.client.1.smithi190.stdout:9/146: write d12/d2d/d1d/d23/f33 [932833,9289] 0 2024-03-26T18:54:22.266 INFO:tasks.workunit.client.1.smithi190.stdout:8/114: rename d9/dd/l11 to d9/l22 0 2024-03-26T18:54:22.267 INFO:tasks.workunit.client.1.smithi190.stdout:4/117: creat d7/d8/d14/f29 x:0 0 0 2024-03-26T18:54:22.268 INFO:tasks.workunit.client.1.smithi190.stdout:4/118: dread d7/fc [0,4194304] 0 2024-03-26T18:54:22.269 INFO:tasks.workunit.client.1.smithi190.stdout:3/133: creat d1/de/f2d x:0 0 0 2024-03-26T18:54:22.269 INFO:tasks.workunit.client.1.smithi190.stdout:3/134: chown d1/d3/c7 0 1 2024-03-26T18:54:22.269 INFO:tasks.workunit.client.1.smithi190.stdout:3/135: truncate d1/f11 1167181 0 2024-03-26T18:54:22.271 INFO:tasks.workunit.client.1.smithi190.stdout:9/147: rename d12/f24 to d12/d2d/d1d/f3b 0 2024-03-26T18:54:22.272 INFO:tasks.workunit.client.1.smithi190.stdout:8/115: mkdir d9/dd/d23 0 2024-03-26T18:54:22.273 INFO:tasks.workunit.client.1.smithi190.stdout:8/116: symlink d9/l24 0 2024-03-26T18:54:22.273 INFO:tasks.workunit.client.1.smithi190.stdout:8/117: stat d9/f14 0 2024-03-26T18:54:22.273 INFO:tasks.workunit.client.1.smithi190.stdout:8/118: stat d9/l24 0 2024-03-26T18:54:22.275 INFO:tasks.workunit.client.1.smithi190.stdout:5/55: link d2/cb d2/c13 0 2024-03-26T18:54:22.275 INFO:tasks.workunit.client.1.smithi190.stdout:5/56: readlink l1 0 2024-03-26T18:54:22.275 INFO:tasks.workunit.client.1.smithi190.stdout:9/148: dread f8 [0,4194304] 0 2024-03-26T18:54:22.276 INFO:tasks.workunit.client.1.smithi190.stdout:5/57: creat d2/f14 x:0 0 0 2024-03-26T18:54:22.277 INFO:tasks.workunit.client.1.smithi190.stdout:5/58: chown d2/f14 409680 1 2024-03-26T18:54:22.277 INFO:tasks.workunit.client.1.smithi190.stdout:5/59: read - d2/f14 zero size 2024-03-26T18:54:22.283 INFO:tasks.workunit.client.1.smithi190.stdout:2/123: dwrite d6/fe [4194304,4194304] 0 2024-03-26T18:54:22.285 INFO:tasks.workunit.client.1.smithi190.stdout:2/124: symlink d6/df/d1c/l25 0 2024-03-26T18:54:22.285 INFO:tasks.workunit.client.1.smithi190.stdout:2/125: dread - d6/df/f22 zero size 2024-03-26T18:54:22.285 INFO:tasks.workunit.client.1.smithi190.stdout:9/149: dread f6 [0,4194304] 0 2024-03-26T18:54:22.286 INFO:tasks.workunit.client.1.smithi190.stdout:2/126: creat d6/d9/d1a/f26 x:0 0 0 2024-03-26T18:54:22.286 INFO:tasks.workunit.client.1.smithi190.stdout:2/127: creat d6/df/f27 x:0 0 0 2024-03-26T18:54:22.286 INFO:tasks.workunit.client.1.smithi190.stdout:2/128: chown d6/df/f27 7794763 1 2024-03-26T18:54:22.288 INFO:tasks.workunit.client.1.smithi190.stdout:2/129: creat d6/df/d1c/f28 x:0 0 0 2024-03-26T18:54:22.288 INFO:tasks.workunit.client.1.smithi190.stdout:2/130: truncate d6/f7 4206992 0 2024-03-26T18:54:22.288 INFO:tasks.workunit.client.1.smithi190.stdout:2/131: chown d6/d9/d10/f13 133 1 2024-03-26T18:54:22.290 INFO:tasks.workunit.client.1.smithi190.stdout:7/110: dwrite d3/d6/d9/d14/f2a [0,4194304] 0 2024-03-26T18:54:22.290 INFO:tasks.workunit.client.1.smithi190.stdout:6/145: dwrite d17/f1b [0,4194304] 0 2024-03-26T18:54:22.296 INFO:tasks.workunit.client.1.smithi190.stdout:6/146: creat d17/d28/d2c/f35 x:0 0 0 2024-03-26T18:54:22.296 INFO:tasks.workunit.client.1.smithi190.stdout:2/132: dread d6/f7 [0,4194304] 0 2024-03-26T18:54:22.296 INFO:tasks.workunit.client.1.smithi190.stdout:2/133: write d6/df/d1c/f28 [926822,50500] 0 2024-03-26T18:54:22.297 INFO:tasks.workunit.client.1.smithi190.stdout:6/147: truncate d17/f27 983064 0 2024-03-26T18:54:22.298 INFO:tasks.workunit.client.1.smithi190.stdout:7/111: dread d3/f19 [0,4194304] 0 2024-03-26T18:54:22.301 INFO:tasks.workunit.client.1.smithi190.stdout:2/134: dread f5 [0,4194304] 0 2024-03-26T18:54:22.301 INFO:tasks.workunit.client.1.smithi190.stdout:2/135: fsync f5 0 2024-03-26T18:54:22.302 INFO:tasks.workunit.client.1.smithi190.stdout:7/112: creat d3/f2b x:0 0 0 2024-03-26T18:54:22.303 INFO:tasks.workunit.client.1.smithi190.stdout:7/113: write d3/d6/d9/d14/f18 [1915158,42679] 0 2024-03-26T18:54:22.303 INFO:tasks.workunit.client.1.smithi190.stdout:3/136: dwrite d1/de/f1d [0,4194304] 0 2024-03-26T18:54:22.303 INFO:tasks.workunit.client.1.smithi190.stdout:4/119: dwrite d7/d8/d14/f19 [4194304,4194304] 0 2024-03-26T18:54:22.309 INFO:tasks.workunit.client.1.smithi190.stdout:5/60: dwrite d2/fa [0,4194304] 0 2024-03-26T18:54:22.309 INFO:tasks.workunit.client.1.smithi190.stdout:4/120: dread d7/f13 [0,4194304] 0 2024-03-26T18:54:22.309 INFO:tasks.workunit.client.1.smithi190.stdout:5/61: truncate d2/f12 1027690 0 2024-03-26T18:54:22.309 INFO:tasks.workunit.client.1.smithi190.stdout:5/62: write d2/f14 [400917,84288] 0 2024-03-26T18:54:22.309 INFO:tasks.workunit.client.1.smithi190.stdout:9/150: getdents d12/d2d/d1d 0 2024-03-26T18:54:22.309 INFO:tasks.workunit.client.1.smithi190.stdout:9/151: fdatasync d12/f13 0 2024-03-26T18:54:22.310 INFO:tasks.workunit.client.1.smithi190.stdout:9/152: dread - d12/d2d/f1b zero size 2024-03-26T18:54:22.310 INFO:tasks.workunit.client.1.smithi190.stdout:9/153: fsync d12/d2d/d1d/f3a 0 2024-03-26T18:54:22.313 INFO:tasks.workunit.client.1.smithi190.stdout:7/114: write d3/d6/f1d [936881,16752] 0 2024-03-26T18:54:22.314 INFO:tasks.workunit.client.1.smithi190.stdout:5/63: chown d2/f3 412002 1 2024-03-26T18:54:22.320 INFO:tasks.workunit.client.1.smithi190.stdout:9/154: mknod d12/d2d/c3c 0 2024-03-26T18:54:22.320 INFO:tasks.workunit.client.1.smithi190.stdout:9/155: chown d12/d2d/f39 7 1 2024-03-26T18:54:22.320 INFO:tasks.workunit.client.1.smithi190.stdout:9/156: fdatasync d12/f16 0 2024-03-26T18:54:22.322 INFO:tasks.workunit.client.1.smithi190.stdout:6/148: dwrite d17/f19 [0,4194304] 0 2024-03-26T18:54:22.324 INFO:tasks.workunit.client.1.smithi190.stdout:9/157: creat d12/d2d/d1d/d23/d2b/f3d x:0 0 0 2024-03-26T18:54:22.324 INFO:tasks.workunit.client.1.smithi190.stdout:9/158: creat d12/d2d/f3e x:0 0 0 2024-03-26T18:54:22.324 INFO:tasks.workunit.client.1.smithi190.stdout:9/159: chown d12/d2d/d1d/d23/d2a/c37 1290 1 2024-03-26T18:54:22.326 INFO:tasks.workunit.client.1.smithi190.stdout:6/149: symlink d17/d28/d2c/l36 0 2024-03-26T18:54:22.326 INFO:tasks.workunit.client.1.smithi190.stdout:6/150: fdatasync d17/f21 0 2024-03-26T18:54:22.326 INFO:tasks.workunit.client.1.smithi190.stdout:6/151: write d17/d28/d2c/f35 [119260,129283] 0 2024-03-26T18:54:22.343 INFO:tasks.workunit.client.1.smithi190.stdout:6/152: dwrite d17/f27 [0,4194304] 0 2024-03-26T18:54:22.343 INFO:tasks.workunit.client.1.smithi190.stdout:6/153: write d17/d28/d2c/f35 [435054,14061] 0 2024-03-26T18:54:22.345 INFO:tasks.workunit.client.1.smithi190.stdout:2/136: dwrite d6/fe [0,4194304] 0 2024-03-26T18:54:22.345 INFO:tasks.workunit.client.1.smithi190.stdout:3/137: dwrite d1/f2c [0,4194304] 0 2024-03-26T18:54:22.345 INFO:tasks.workunit.client.1.smithi190.stdout:3/138: write d1/de/f29 [443040,122073] 0 2024-03-26T18:54:22.345 INFO:tasks.workunit.client.1.smithi190.stdout:7/115: dwrite d3/d1b/f27 [0,4194304] 0 2024-03-26T18:54:22.345 INFO:tasks.workunit.client.1.smithi190.stdout:7/116: chown d3/d1b/f25 47612901 1 2024-03-26T18:54:22.346 INFO:tasks.workunit.client.1.smithi190.stdout:6/154: symlink d17/d28/d2b/l37 0 2024-03-26T18:54:22.346 INFO:tasks.workunit.client.1.smithi190.stdout:6/155: write fc [534024,122365] 0 2024-03-26T18:54:22.347 INFO:tasks.workunit.client.1.smithi190.stdout:6/156: read fc [368296,84865] 0 2024-03-26T18:54:22.349 INFO:tasks.workunit.client.1.smithi190.stdout:2/137: creat d6/d21/f29 x:0 0 0 2024-03-26T18:54:22.351 INFO:tasks.workunit.client.1.smithi190.stdout:3/139: rename d1/c16 to d1/c2e 0 2024-03-26T18:54:22.356 INFO:tasks.workunit.client.1.smithi190.stdout:7/117: write d3/fe [1977339,9989] 0 2024-03-26T18:54:22.356 INFO:tasks.workunit.client.1.smithi190.stdout:6/157: mkdir d17/d28/d38 0 2024-03-26T18:54:22.356 INFO:tasks.workunit.client.1.smithi190.stdout:7/118: write d3/d1b/f28 [118476,63287] 0 2024-03-26T18:54:22.356 INFO:tasks.workunit.client.1.smithi190.stdout:7/119: creat d3/d1b/f2c x:0 0 0 2024-03-26T18:54:22.357 INFO:tasks.workunit.client.1.smithi190.stdout:2/138: symlink d6/d9/d1a/l2a 0 2024-03-26T18:54:22.358 INFO:tasks.workunit.client.1.smithi190.stdout:3/140: truncate d1/d3/df/f2a 30025 0 2024-03-26T18:54:22.358 INFO:tasks.workunit.client.1.smithi190.stdout:3/141: creat d1/de/f2f x:0 0 0 2024-03-26T18:54:22.360 INFO:tasks.workunit.client.1.smithi190.stdout:2/139: mkdir d6/d9/d2b 0 2024-03-26T18:54:22.363 INFO:tasks.workunit.client.1.smithi190.stdout:7/120: dread d3/d6/d9/f24 [0,4194304] 0 2024-03-26T18:54:22.364 INFO:tasks.workunit.client.1.smithi190.stdout:3/142: creat d1/d3/df/d18/f30 x:0 0 0 2024-03-26T18:54:22.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.372+0000 7f8e65849700 1 -- 172.21.15.102:0/162055627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e601013f0 msgr2=0x7f8e60101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:22.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.372+0000 7f8e65849700 1 --2- 172.21.15.102:0/162055627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e601013f0 0x7f8e60101840 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009910 tx=0x7f8e54009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:22.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.372+0000 7f8e65849700 1 -- 172.21.15.102:0/162055627 shutdown_connections 2024-03-26T18:54:22.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.372+0000 7f8e65849700 1 --2- 172.21.15.102:0/162055627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e601013f0 0x7f8e60101840 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:22.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.372+0000 7f8e65849700 1 --2- 172.21.15.102:0/162055627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e600fe410 0x7f8e600fe820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:22.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.372+0000 7f8e65849700 1 -- 172.21.15.102:0/162055627 >> 172.21.15.102:0/162055627 conn(0x7f8e600f9d80 msgr2=0x7f8e600fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:22.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.373+0000 7f8e65849700 1 -- 172.21.15.102:0/162055627 shutdown_connections 2024-03-26T18:54:22.367 INFO:tasks.workunit.client.1.smithi190.stdout:2/140: creat d6/d9/d10/f2c x:0 0 0 2024-03-26T18:54:22.367 INFO:tasks.workunit.client.1.smithi190.stdout:7/121: dread d3/fc [0,4194304] 0 2024-03-26T18:54:22.367 INFO:tasks.workunit.client.1.smithi190.stdout:2/141: write d6/df/d1c/f28 [1591986,126908] 0 2024-03-26T18:54:22.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.373+0000 7f8e65849700 1 -- 172.21.15.102:0/162055627 wait complete. 2024-03-26T18:54:22.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.374+0000 7f8e65849700 1 Processor -- start 2024-03-26T18:54:22.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.374+0000 7f8e65849700 1 -- start start 2024-03-26T18:54:22.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.374+0000 7f8e65849700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e600fe410 0x7f8e6019c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:22.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.375+0000 7f8e65849700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e601013f0 0x7f8e6019c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:22.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.375+0000 7f8e65849700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6019cc20 con 0x7f8e600fe410 2024-03-26T18:54:22.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.375+0000 7f8e65849700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6019cd60 con 0x7f8e601013f0 2024-03-26T18:54:22.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.375+0000 7f8e5effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e600fe410 0x7f8e6019c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:22.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.375+0000 7f8e5effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e600fe410 0x7f8e6019c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34282/0 (socket says 172.21.15.102:34282) 2024-03-26T18:54:22.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.375+0000 7f8e5effd700 1 -- 172.21.15.102:0/3070867131 learned_addr learned my addr 172.21.15.102:0/3070867131 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:22.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.376+0000 7f8e5e7fc700 1 --2- 172.21.15.102:0/3070867131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e601013f0 0x7f8e6019c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:22.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.376+0000 7f8e5effd700 1 -- 172.21.15.102:0/3070867131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e601013f0 msgr2=0x7f8e6019c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:22.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.376+0000 7f8e5effd700 1 --2- 172.21.15.102:0/3070867131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e601013f0 0x7f8e6019c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:22.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.376+0000 7f8e5effd700 1 -- 172.21.15.102:0/3070867131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e540095f0 con 0x7f8e600fe410 2024-03-26T18:54:22.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.376+0000 7f8e5effd700 1 --2- 172.21.15.102:0/3070867131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e600fe410 0x7f8e6019c0c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000b480 tx=0x7f8e5000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:22.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.377+0000 7f8e64847700 1 -- 172.21.15.102:0/3070867131 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e500118c0 con 0x7f8e600fe410 2024-03-26T18:54:22.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.377+0000 7f8e65849700 1 -- 172.21.15.102:0/3070867131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e601a1810 con 0x7f8e600fe410 2024-03-26T18:54:22.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.377+0000 7f8e64847700 1 -- 172.21.15.102:0/3070867131 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8e50010380 con 0x7f8e600fe410 2024-03-26T18:54:22.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.377+0000 7f8e65849700 1 -- 172.21.15.102:0/3070867131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e601a1de0 con 0x7f8e600fe410 2024-03-26T18:54:22.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.378+0000 7f8e64847700 1 -- 172.21.15.102:0/3070867131 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e5000f4f0 con 0x7f8e600fe410 2024-03-26T18:54:22.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.378+0000 7f8e65849700 1 -- 172.21.15.102:0/3070867131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e60100d50 con 0x7f8e600fe410 2024-03-26T18:54:22.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.379+0000 7f8e64847700 1 -- 172.21.15.102:0/3070867131 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f8e50011a20 con 0x7f8e600fe410 2024-03-26T18:54:22.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.380+0000 7f8e64847700 1 --2- 172.21.15.102:0/3070867131 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8e4806c480 0x7f8e4806e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:22.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.380+0000 7f8e64847700 1 -- 172.21.15.102:0/3070867131 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f8e5008b980 con 0x7f8e600fe410 2024-03-26T18:54:22.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.380+0000 7f8e5e7fc700 1 --2- 172.21.15.102:0/3070867131 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8e4806c480 0x7f8e4806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:22.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.381+0000 7f8e5e7fc700 1 --2- 172.21.15.102:0/3070867131 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8e4806c480 0x7f8e4806e930 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8e54006010 tx=0x7f8e5401a040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:22.376 INFO:tasks.workunit.client.1.smithi190.stdout:6/158: dwrite d17/f27 [4194304,4194304] 0 2024-03-26T18:54:22.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.386+0000 7f8e64847700 1 -- 172.21.15.102:0/3070867131 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8e50059af0 con 0x7f8e600fe410 2024-03-26T18:54:22.385 INFO:tasks.workunit.client.1.smithi190.stdout:2/142: dread d6/df/d1c/f28 [0,4194304] 0 2024-03-26T18:54:22.387 INFO:tasks.workunit.client.1.smithi190.stdout:2/143: rename d6/df/d1c/f28 to d6/d9/d10/f2d 0 2024-03-26T18:54:22.388 INFO:tasks.workunit.client.1.smithi190.stdout:2/144: rename d6/d9/d2b to d6/df/d2e 0 2024-03-26T18:54:22.389 INFO:tasks.workunit.client.1.smithi190.stdout:2/145: truncate d6/d9/f1d 1913587 0 2024-03-26T18:54:22.389 INFO:tasks.workunit.client.1.smithi190.stdout:2/146: stat d6/fe 0 2024-03-26T18:54:22.391 INFO:tasks.workunit.client.1.smithi190.stdout:3/143: dwrite d1/f14 [4194304,4194304] 0 2024-03-26T18:54:22.391 INFO:tasks.workunit.client.1.smithi190.stdout:3/144: write d1/d3/df/d18/f30 [600626,71826] 0 2024-03-26T18:54:22.391 INFO:tasks.workunit.client.1.smithi190.stdout:7/122: dwrite d3/d1b/f25 [0,4194304] 0 2024-03-26T18:54:22.391 INFO:tasks.workunit.client.1.smithi190.stdout:7/123: stat d3/d6/d9/f24 0 2024-03-26T18:54:22.393 INFO:tasks.workunit.client.1.smithi190.stdout:3/145: mkdir d1/d3/df/d31 0 2024-03-26T18:54:22.395 INFO:tasks.workunit.client.1.smithi190.stdout:7/124: mknod d3/d6/c2d 0 2024-03-26T18:54:22.397 INFO:tasks.workunit.client.1.smithi190.stdout:3/146: mknod d1/d3/df/d18/d1b/c32 0 2024-03-26T18:54:22.421 INFO:tasks.workunit.client.1.smithi190.stdout:7/125: dwrite d3/fc [0,4194304] 0 2024-03-26T18:54:22.425 INFO:tasks.workunit.client.1.smithi190.stdout:3/147: dwrite d1/f11 [0,4194304] 0 2024-03-26T18:54:22.481 INFO:tasks.workunit.client.1.smithi190.stdout:3/148: dwrite d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:22.482 INFO:tasks.workunit.client.1.smithi190.stdout:3/149: dread d1/de/f29 [0,4194304] 0 2024-03-26T18:54:22.492 INFO:tasks.workunit.client.1.smithi190.stdout:3/150: write d1/f11 [1663143,98092] 0 2024-03-26T18:54:22.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.745+0000 7f8e65849700 1 -- 172.21.15.102:0/3070867131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8e601a28b0 con 0x7f8e600fe410 2024-03-26T18:54:22.741 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:54:22.741 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:54:22.741 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 14 2024-03-26T18:54:22.742 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:54:22.743 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:54:22.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.748+0000 7f8e64847700 1 -- 172.21.15.102:0/3070867131 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f8e50059680 con 0x7f8e600fe410 2024-03-26T18:54:22.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.753+0000 7f8e65849700 1 -- 172.21.15.102:0/3070867131 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8e4806c480 msgr2=0x7f8e4806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:22.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.753+0000 7f8e65849700 1 --2- 172.21.15.102:0/3070867131 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8e4806c480 0x7f8e4806e930 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8e54006010 tx=0x7f8e5401a040 comp rx=0 tx=0).stop 2024-03-26T18:54:22.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.753+0000 7f8e65849700 1 -- 172.21.15.102:0/3070867131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e600fe410 msgr2=0x7f8e6019c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:22.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.753+0000 7f8e65849700 1 --2- 172.21.15.102:0/3070867131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e600fe410 0x7f8e6019c0c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000b480 tx=0x7f8e5000b840 comp rx=0 tx=0).stop 2024-03-26T18:54:22.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.753+0000 7f8e65849700 1 -- 172.21.15.102:0/3070867131 shutdown_connections 2024-03-26T18:54:22.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.753+0000 7f8e65849700 1 --2- 172.21.15.102:0/3070867131 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f8e4806c480 0x7f8e4806e930 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8e54006010 tx=0x7f8e5401a040 comp rx=0 tx=0).stop 2024-03-26T18:54:22.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.753+0000 7f8e65849700 1 --2- 172.21.15.102:0/3070867131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e600fe410 0x7f8e6019c0c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:22.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.753+0000 7f8e65849700 1 --2- 172.21.15.102:0/3070867131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e601013f0 0x7f8e6019c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:22.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.753+0000 7f8e65849700 1 -- 172.21.15.102:0/3070867131 >> 172.21.15.102:0/3070867131 conn(0x7f8e600f9d80 msgr2=0x7f8e60104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:22.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.756+0000 7f8e65849700 1 -- 172.21.15.102:0/3070867131 shutdown_connections 2024-03-26T18:54:22.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.756+0000 7f8e65849700 1 -- 172.21.15.102:0/3070867131 wait complete. 2024-03-26T18:54:22.765 INFO:tasks.workunit.client.1.smithi190.stdout:3/151: unlink d1/de/c20 0 2024-03-26T18:54:22.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.952+0000 7fd243c97700 1 -- 172.21.15.102:0/1621392916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd23c1035a0 msgr2=0x7fd23c1039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:22.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.952+0000 7fd243c97700 1 --2- 172.21.15.102:0/1621392916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd23c1035a0 0x7fd23c1039f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd238009910 tx=0x7fd238009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:22.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.954+0000 7fd243c97700 1 -- 172.21.15.102:0/1621392916 shutdown_connections 2024-03-26T18:54:22.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.954+0000 7fd243c97700 1 --2- 172.21.15.102:0/1621392916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd23c1035a0 0x7fd23c1039f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:22.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.954+0000 7fd243c97700 1 --2- 172.21.15.102:0/1621392916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd23c1023a0 0x7fd23c1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:22.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.954+0000 7fd243c97700 1 -- 172.21.15.102:0/1621392916 >> 172.21.15.102:0/1621392916 conn(0x7fd23c0fd990 msgr2=0x7fd23c0ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:22.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.954+0000 7fd243c97700 1 -- 172.21.15.102:0/1621392916 shutdown_connections 2024-03-26T18:54:22.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.954+0000 7fd243c97700 1 -- 172.21.15.102:0/1621392916 wait complete. 2024-03-26T18:54:22.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.955+0000 7fd243c97700 1 Processor -- start 2024-03-26T18:54:22.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.955+0000 7fd243c97700 1 -- start start 2024-03-26T18:54:22.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.955+0000 7fd243c97700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd23c1023a0 0x7fd23c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:22.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.955+0000 7fd243c97700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd23c1035a0 0x7fd23c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:22.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.955+0000 7fd243c97700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c1986d0 con 0x7fd23c1035a0 2024-03-26T18:54:22.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.955+0000 7fd243c97700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c198810 con 0x7fd23c1023a0 2024-03-26T18:54:22.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.957+0000 7fd241232700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd23c1035a0 0x7fd23c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:22.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.957+0000 7fd241232700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd23c1035a0 0x7fd23c1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34306/0 (socket says 172.21.15.102:34306) 2024-03-26T18:54:22.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.957+0000 7fd241232700 1 -- 172.21.15.102:0/2138164389 learned_addr learned my addr 172.21.15.102:0/2138164389 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:22.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.958+0000 7fd241232700 1 -- 172.21.15.102:0/2138164389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd23c1023a0 msgr2=0x7fd23c197b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T18:54:22.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.958+0000 7fd241232700 1 --2- 172.21.15.102:0/2138164389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd23c1023a0 0x7fd23c197b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:22.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.958+0000 7fd241232700 1 -- 172.21.15.102:0/2138164389 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2380095f0 con 0x7fd23c1035a0 2024-03-26T18:54:22.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.958+0000 7fd241232700 1 --2- 172.21.15.102:0/2138164389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd23c1035a0 0x7fd23c1980b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd23800f690 tx=0x7fd23800b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:22.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.959+0000 7fd232ffd700 1 -- 172.21.15.102:0/2138164389 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd238025070 con 0x7fd23c1035a0 2024-03-26T18:54:22.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.959+0000 7fd243c97700 1 -- 172.21.15.102:0/2138164389 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23c075190 con 0x7fd23c1035a0 2024-03-26T18:54:22.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.959+0000 7fd243c97700 1 -- 172.21.15.102:0/2138164389 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23c075680 con 0x7fd23c1035a0 2024-03-26T18:54:22.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.959+0000 7fd232ffd700 1 -- 172.21.15.102:0/2138164389 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd2380052a0 con 0x7fd23c1035a0 2024-03-26T18:54:22.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.959+0000 7fd232ffd700 1 -- 172.21.15.102:0/2138164389 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd23802c470 con 0x7fd23c1035a0 2024-03-26T18:54:22.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.961+0000 7fd243c97700 1 -- 172.21.15.102:0/2138164389 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd220005320 con 0x7fd23c1035a0 2024-03-26T18:54:22.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.962+0000 7fd232ffd700 1 -- 172.21.15.102:0/2138164389 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fd238029070 con 0x7fd23c1035a0 2024-03-26T18:54:22.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.962+0000 7fd232ffd700 1 --2- 172.21.15.102:0/2138164389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd22806c720 0x7fd22806ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:22.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.962+0000 7fd232ffd700 1 -- 172.21.15.102:0/2138164389 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd238095880 con 0x7fd23c1035a0 2024-03-26T18:54:22.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.964+0000 7fd241a33700 1 --2- 172.21.15.102:0/2138164389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd22806c720 0x7fd22806ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:22.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.966+0000 7fd241a33700 1 --2- 172.21.15.102:0/2138164389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd22806c720 0x7fd22806ebd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd23c103400 tx=0x7fd23400b040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:22.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:22.972+0000 7fd232ffd700 1 -- 172.21.15.102:0/2138164389 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd238063970 con 0x7fd23c1035a0 2024-03-26T18:54:22.971 INFO:tasks.workunit.client.1.smithi190.stdout:3/152: dread d1/d3/f10 [0,4194304] 0 2024-03-26T18:54:23.078 INFO:tasks.workunit.client.1.smithi190.stdout:1/118: sync 2024-03-26T18:54:23.078 INFO:tasks.workunit.client.1.smithi190.stdout:1/119: dread - d3/da/dd/f12 zero size 2024-03-26T18:54:23.078 INFO:tasks.workunit.client.1.smithi190.stdout:9/160: sync 2024-03-26T18:54:23.078 INFO:tasks.workunit.client.1.smithi190.stdout:9/161: fsync d12/f38 0 2024-03-26T18:54:23.078 INFO:tasks.workunit.client.1.smithi190.stdout:7/126: sync 2024-03-26T18:54:23.078 INFO:tasks.workunit.client.1.smithi190.stdout:8/119: sync 2024-03-26T18:54:23.078 INFO:tasks.workunit.client.1.smithi190.stdout:5/64: sync 2024-03-26T18:54:23.078 INFO:tasks.workunit.client.1.smithi190.stdout:6/159: sync 2024-03-26T18:54:23.079 INFO:tasks.workunit.client.1.smithi190.stdout:0/219: sync 2024-03-26T18:54:23.079 INFO:tasks.workunit.client.1.smithi190.stdout:4/121: sync 2024-03-26T18:54:23.079 INFO:tasks.workunit.client.1.smithi190.stdout:2/147: sync 2024-03-26T18:54:23.082 INFO:tasks.workunit.client.1.smithi190.stdout:9/162: mkdir d12/d2d/d1d/d23/d2a/d3f 0 2024-03-26T18:54:23.083 INFO:tasks.workunit.client.1.smithi190.stdout:7/127: creat d3/d6/d9/f2e x:0 0 0 2024-03-26T18:54:23.084 INFO:tasks.workunit.client.1.smithi190.stdout:8/120: link d9/d16/c19 d9/dd/d15/c25 0 2024-03-26T18:54:23.087 INFO:tasks.workunit.client.1.smithi190.stdout:6/160: mknod d17/d28/d2c/c39 0 2024-03-26T18:54:23.091 INFO:tasks.workunit.client.1.smithi190.stdout:4/122: rename d7/d8/f12 to d7/d8/f2a 0 2024-03-26T18:54:23.091 INFO:tasks.workunit.client.1.smithi190.stdout:4/123: creat d7/d8/d14/f2b x:0 0 0 2024-03-26T18:54:23.091 INFO:tasks.workunit.client.1.smithi190.stdout:2/148: link d6/c19 d6/d9/d1a/c2f 0 2024-03-26T18:54:23.100 INFO:tasks.workunit.client.1.smithi190.stdout:9/163: dread f10 [0,4194304] 0 2024-03-26T18:54:23.101 INFO:tasks.workunit.client.1.smithi190.stdout:6/161: symlink d17/d28/l3a 0 2024-03-26T18:54:23.120 INFO:tasks.workunit.client.1.smithi190.stdout:9/164: rename f10 to d12/f40 0 2024-03-26T18:54:23.124 INFO:tasks.workunit.client.1.smithi190.stdout:6/162: mknod d17/d28/d2b/c3b 0 2024-03-26T18:54:23.137 INFO:tasks.workunit.client.1.smithi190.stdout:1/120: rmdir d3/d9/d1a 0 2024-03-26T18:54:23.142 INFO:tasks.workunit.client.1.smithi190.stdout:1/121: mknod d3/da/d1e/c28 0 2024-03-26T18:54:23.142 INFO:tasks.workunit.client.1.smithi190.stdout:1/122: creat d3/f29 x:0 0 0 2024-03-26T18:54:23.185 INFO:tasks.workunit.client.1.smithi190.stdout:2/149: rmdir d6/d9 39 2024-03-26T18:54:23.213 INFO:tasks.workunit.client.1.smithi190.stdout:3/153: dwrite d1/d3/f10 [8388608,4194304] 0 2024-03-26T18:54:23.218 INFO:tasks.workunit.client.1.smithi190.stdout:3/154: link d1/d3/c8 d1/de/c33 0 2024-03-26T18:54:23.218 INFO:tasks.workunit.client.1.smithi190.stdout:3/155: write d1/de/f29 [562027,102634] 0 2024-03-26T18:54:23.240 INFO:tasks.workunit.client.1.smithi190.stdout:8/121: dread d9/f12 [8388608,4194304] 0 2024-03-26T18:54:23.266 INFO:tasks.workunit.client.1.smithi190.stdout:6/163: write d17/f27 [4258094,15271] 0 2024-03-26T18:54:23.268 INFO:tasks.workunit.client.1.smithi190.stdout:6/164: truncate d17/d28/d2c/f2e 123819 0 2024-03-26T18:54:23.279 INFO:tasks.workunit.client.1.smithi190.stdout:1/123: dread d3/da/f1f [0,4194304] 0 2024-03-26T18:54:23.279 INFO:tasks.workunit.client.1.smithi190.stdout:1/124: chown d3/da/d1e/c28 644008 1 2024-03-26T18:54:23.282 INFO:tasks.workunit.client.1.smithi190.stdout:1/125: rmdir d3/da/dd 39 2024-03-26T18:54:23.282 INFO:tasks.workunit.client.1.smithi190.stdout:1/126: dread - d3/f27 zero size 2024-03-26T18:54:23.287 INFO:tasks.workunit.client.1.smithi190.stdout:9/165: dwrite f8 [0,4194304] 0 2024-03-26T18:54:23.287 INFO:tasks.workunit.client.1.smithi190.stdout:9/166: chown f8 148922725 1 2024-03-26T18:54:23.312 INFO:tasks.workunit.client.1.smithi190.stdout:7/128: write d3/d6/fd [1882179,34777] 0 2024-03-26T18:54:23.316 INFO:tasks.workunit.client.1.smithi190.stdout:7/129: getdents d3/d6/d9/d14 0 2024-03-26T18:54:23.316 INFO:tasks.workunit.client.1.smithi190.stdout:7/130: creat d3/f2f x:0 0 0 2024-03-26T18:54:23.339 INFO:tasks.workunit.client.1.smithi190.stdout:4/124: dwrite d7/d8/d1c/f1f [0,4194304] 0 2024-03-26T18:54:23.339 INFO:tasks.workunit.client.1.smithi190.stdout:4/125: chown d7/d8/d14/d23/d28 1437315 1 2024-03-26T18:54:23.339 INFO:tasks.workunit.client.1.smithi190.stdout:4/126: dread - d7/d1d/f27 zero size 2024-03-26T18:54:23.339 INFO:tasks.workunit.client.1.smithi190.stdout:4/127: rename d7 to d7/d8/d1c/d20/d2c 22 2024-03-26T18:54:23.340 INFO:tasks.workunit.client.1.smithi190.stdout:6/165: dwrite d17/d28/f2d [0,4194304] 0 2024-03-26T18:54:23.344 INFO:tasks.workunit.client.1.smithi190.stdout:4/128: rename d7/lf to d7/d8/l2d 0 2024-03-26T18:54:23.344 INFO:tasks.workunit.client.1.smithi190.stdout:4/129: readlink d7/d8/d1c/l21 0 2024-03-26T18:54:23.345 INFO:tasks.workunit.client.1.smithi190.stdout:4/130: read - d7/f18 zero size 2024-03-26T18:54:23.345 INFO:tasks.workunit.client.1.smithi190.stdout:4/131: dread - d7/d8/d14/f29 zero size 2024-03-26T18:54:23.345 INFO:tasks.workunit.client.1.smithi190.stdout:4/132: fsync d7/d1d/f27 0 2024-03-26T18:54:23.349 INFO:tasks.workunit.client.1.smithi190.stdout:6/166: rmdir d17 39 2024-03-26T18:54:23.349 INFO:tasks.workunit.client.1.smithi190.stdout:6/167: readlink d17/d28/d2c/l36 0 2024-03-26T18:54:23.356 INFO:tasks.workunit.client.1.smithi190.stdout:4/133: creat d7/d8/d14/d23/d28/f2e x:0 0 0 2024-03-26T18:54:23.356 INFO:tasks.workunit.client.1.smithi190.stdout:4/134: dread - d7/d1d/f27 zero size 2024-03-26T18:54:23.363 INFO:tasks.workunit.client.1.smithi190.stdout:6/168: creat d17/d28/d2b/f3c x:0 0 0 2024-03-26T18:54:23.388 INFO:tasks.workunit.client.1.smithi190.stdout:4/135: write d7/fc [161237,119553] 0 2024-03-26T18:54:23.388 INFO:tasks.workunit.client.1.smithi190.stdout:4/136: getdents d7/d8/d1c/d20 0 2024-03-26T18:54:23.388 INFO:tasks.workunit.client.1.smithi190.stdout:4/137: readlink d7/d8/d1c/l21 0 2024-03-26T18:54:23.388 INFO:tasks.workunit.client.1.smithi190.stdout:4/138: stat d7/d8/f2a 0 2024-03-26T18:54:23.388 INFO:tasks.workunit.client.1.smithi190.stdout:4/139: chown c3 25618 1 2024-03-26T18:54:23.390 INFO:tasks.workunit.client.1.smithi190.stdout:4/140: mknod d7/d8/d1c/c2f 0 2024-03-26T18:54:23.392 INFO:tasks.workunit.client.1.smithi190.stdout:4/141: unlink d7/d8/d14/c15 0 2024-03-26T18:54:23.394 INFO:tasks.workunit.client.1.smithi190.stdout:4/142: mknod d7/d8/d14/c30 0 2024-03-26T18:54:23.396 INFO:tasks.workunit.client.1.smithi190.stdout:4/143: mknod d7/d1d/c31 0 2024-03-26T18:54:23.397 INFO:tasks.workunit.client.1.smithi190.stdout:4/144: write d7/d1d/f27 [305391,86883] 0 2024-03-26T18:54:23.397 INFO:tasks.workunit.client.1.smithi190.stdout:4/145: chown d7/d8/d14/c16 0 1 2024-03-26T18:54:23.441 INFO:tasks.workunit.client.1.smithi190.stdout:4/146: dread d7/d8/d1c/f1f [0,4194304] 0 2024-03-26T18:54:23.441 INFO:tasks.workunit.client.1.smithi190.stdout:4/147: creat d7/d1d/f32 x:0 0 0 2024-03-26T18:54:23.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.469+0000 7fd243c97700 1 -- 172.21.15.102:0/2138164389 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd220006200 con 0x7fd23c1035a0 2024-03-26T18:54:23.465 INFO:teuthology.orchestra.run.smithi102.stdout:e18 2024-03-26T18:54:23.465 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:54:23.465 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:54:23.465 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:54:23.465 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:54:23.465 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:54:23.465 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:54:23.465 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 18 2024-03-26T18:54:23.465 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:54:23.465 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:53:56.228104+0000 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:54:23.466 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:54:23.467 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:54:23.468 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:54:23.468 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:54:23.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.472+0000 7fd232ffd700 1 -- 172.21.15.102:0/2138164389 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1854 (secure 0 0 0) 0x7fd238063500 con 0x7fd23c1035a0 2024-03-26T18:54:23.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.476+0000 7fd243c97700 1 -- 172.21.15.102:0/2138164389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd22806c720 msgr2=0x7fd22806ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:23.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.476+0000 7fd243c97700 1 --2- 172.21.15.102:0/2138164389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd22806c720 0x7fd22806ebd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd23c103400 tx=0x7fd23400b040 comp rx=0 tx=0).stop 2024-03-26T18:54:23.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.476+0000 7fd243c97700 1 -- 172.21.15.102:0/2138164389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd23c1035a0 msgr2=0x7fd23c1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:23.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.476+0000 7fd243c97700 1 --2- 172.21.15.102:0/2138164389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd23c1035a0 0x7fd23c1980b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd23800f690 tx=0x7fd23800b610 comp rx=0 tx=0).stop 2024-03-26T18:54:23.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.476+0000 7fd243c97700 1 -- 172.21.15.102:0/2138164389 shutdown_connections 2024-03-26T18:54:23.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.476+0000 7fd243c97700 1 --2- 172.21.15.102:0/2138164389 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fd22806c720 0x7fd22806ebd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:23.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.476+0000 7fd243c97700 1 --2- 172.21.15.102:0/2138164389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd23c1023a0 0x7fd23c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:23.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.476+0000 7fd243c97700 1 --2- 172.21.15.102:0/2138164389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd23c1035a0 0x7fd23c1980b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:23.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.476+0000 7fd243c97700 1 -- 172.21.15.102:0/2138164389 >> 172.21.15.102:0/2138164389 conn(0x7fd23c0fd990 msgr2=0x7fd23c1067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:23.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.477+0000 7fd243c97700 1 -- 172.21.15.102:0/2138164389 shutdown_connections 2024-03-26T18:54:23.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.477+0000 7fd243c97700 1 -- 172.21.15.102:0/2138164389 wait complete. 2024-03-26T18:54:23.471 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 18 2024-03-26T18:54:23.493 INFO:tasks.workunit.client.1.smithi190.stdout:5/65: dwrite d2/f6 [0,4194304] 0 2024-03-26T18:54:23.493 INFO:tasks.workunit.client.1.smithi190.stdout:4/148: write d7/d8/d14/f1a [1341040,119139] 0 2024-03-26T18:54:23.493 INFO:tasks.workunit.client.1.smithi190.stdout:4/149: write d7/d1d/f27 [272088,58746] 0 2024-03-26T18:54:23.509 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:23 smithi190 ceph-mon[35043]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:23.509 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:23 smithi190 ceph-mon[35043]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:23.509 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:23 smithi190 ceph-mon[35043]: pgmap v239: 65 pgs: 65 active+clean; 279 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 4.8 MiB/s rd, 11 MiB/s wr, 419 op/s 2024-03-26T18:54:23.509 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:23 smithi190 ceph-mon[35043]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:23.509 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:23 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3070867131' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:54:23.517 INFO:tasks.workunit.client.1.smithi190.stdout:5/66: dread d2/f3 [0,4194304] 0 2024-03-26T18:54:23.517 INFO:tasks.workunit.client.1.smithi190.stdout:5/67: stat d2/f6 0 2024-03-26T18:54:23.528 INFO:tasks.workunit.client.1.smithi190.stdout:5/68: getdents d2 0 2024-03-26T18:54:23.535 INFO:tasks.workunit.client.1.smithi190.stdout:5/69: creat d2/f15 x:0 0 0 2024-03-26T18:54:23.536 INFO:tasks.workunit.client.1.smithi190.stdout:5/70: readlink l1 0 2024-03-26T18:54:23.544 INFO:tasks.workunit.client.1.smithi190.stdout:5/71: creat d2/f16 x:0 0 0 2024-03-26T18:54:23.548 INFO:tasks.workunit.client.1.smithi190.stdout:1/127: read d3/d9/f13 [5959527,63189] 0 2024-03-26T18:54:23.561 INFO:tasks.workunit.client.1.smithi190.stdout:1/128: link d3/d9/db/f19 d3/d9/f2a 0 2024-03-26T18:54:23.574 INFO:tasks.workunit.client.1.smithi190.stdout:0/220: dwrite d0/dd/f45 [0,4194304] 0 2024-03-26T18:54:23.587 INFO:tasks.workunit.client.1.smithi190.stdout:5/72: dread d2/f6 [0,4194304] 0 2024-03-26T18:54:23.587 INFO:tasks.workunit.client.1.smithi190.stdout:5/73: readlink l1 0 2024-03-26T18:54:23.589 INFO:tasks.workunit.client.1.smithi190.stdout:1/129: dread d3/d9/f20 [0,4194304] 0 2024-03-26T18:54:23.589 INFO:tasks.workunit.client.1.smithi190.stdout:1/130: chown d3/d9 150045 1 2024-03-26T18:54:23.596 INFO:tasks.workunit.client.1.smithi190.stdout:0/221: rename d0/c11 to d0/dd/d24/c4d 0 2024-03-26T18:54:23.601 INFO:tasks.workunit.client.1.smithi190.stdout:5/74: chown d2/c13 69799839 1 2024-03-26T18:54:23.605 INFO:tasks.workunit.client.1.smithi190.stdout:0/222: stat d0/dd/d24/c42 0 2024-03-26T18:54:23.605 INFO:tasks.workunit.client.1.smithi190.stdout:0/223: write d0/dd/d24/f2f [2530196,61370] 0 2024-03-26T18:54:23.611 INFO:tasks.workunit.client.1.smithi190.stdout:5/75: write d2/f4 [4668980,107822] 0 2024-03-26T18:54:23.624 INFO:tasks.workunit.client.1.smithi190.stdout:0/224: rmdir d0/dd/d26 39 2024-03-26T18:54:23.624 INFO:tasks.workunit.client.1.smithi190.stdout:0/225: write d0/dd/d26/f3f [550056,51407] 0 2024-03-26T18:54:23.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:23 smithi102 ceph-mon[29002]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:23.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:23 smithi102 ceph-mon[29002]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:23.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:23 smithi102 ceph-mon[29002]: pgmap v239: 65 pgs: 65 active+clean; 279 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 4.8 MiB/s rd, 11 MiB/s wr, 419 op/s 2024-03-26T18:54:23.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:23 smithi102 ceph-mon[29002]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:23.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:23 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3070867131' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:54:23.636 INFO:tasks.workunit.client.1.smithi190.stdout:0/226: mkdir d0/dd/d20/d1c/d4e 0 2024-03-26T18:54:23.666 INFO:tasks.workunit.client.1.smithi190.stdout:5/76: getdents d2 0 2024-03-26T18:54:23.666 INFO:tasks.workunit.client.1.smithi190.stdout:5/77: readlink d2/le 0 2024-03-26T18:54:23.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.678+0000 7f7c74d00700 1 -- 172.21.15.102:0/1891993686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c700fed90 msgr2=0x7f7c700ff250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:23.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.678+0000 7f7c74d00700 1 --2- 172.21.15.102:0/1891993686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c700fed90 0x7f7c700ff250 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7c58009910 tx=0x7f7c58009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:23.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.678+0000 7f7c74d00700 1 -- 172.21.15.102:0/1891993686 shutdown_connections 2024-03-26T18:54:23.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.678+0000 7f7c74d00700 1 --2- 172.21.15.102:0/1891993686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c700fed90 0x7f7c700ff250 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7c58009910 tx=0x7f7c58009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:23.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.678+0000 7f7c74d00700 1 --2- 172.21.15.102:0/1891993686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c700fe440 0x7f7c700fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:23.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.678+0000 7f7c74d00700 1 -- 172.21.15.102:0/1891993686 >> 172.21.15.102:0/1891993686 conn(0x7f7c700f9e50 msgr2=0x7f7c700fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:23.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.680+0000 7f7c74d00700 1 -- 172.21.15.102:0/1891993686 shutdown_connections 2024-03-26T18:54:23.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.681+0000 7f7c74d00700 1 -- 172.21.15.102:0/1891993686 wait complete. 2024-03-26T18:54:23.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.682+0000 7f7c74d00700 1 Processor -- start 2024-03-26T18:54:23.675 INFO:tasks.workunit.client.1.smithi190.stdout:5/78: mkdir d2/d17 0 2024-03-26T18:54:23.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.683+0000 7f7c74d00700 1 -- start start 2024-03-26T18:54:23.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.683+0000 7f7c74d00700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c700fe440 0x7f7c7010ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:23.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.683+0000 7f7c74d00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7010f3d0 0x7f7c701aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:23.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.683+0000 7f7c74d00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7010f840 con 0x7f7c7010f3d0 2024-03-26T18:54:23.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.683+0000 7f7c74d00700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7010f9b0 con 0x7f7c700fe440 2024-03-26T18:54:23.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.690+0000 7f7c6dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7010f3d0 0x7f7c701aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:23.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.690+0000 7f7c6dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7010f3d0 0x7f7c701aa2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34326/0 (socket says 172.21.15.102:34326) 2024-03-26T18:54:23.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.690+0000 7f7c6dd9b700 1 -- 172.21.15.102:0/688505161 learned_addr learned my addr 172.21.15.102:0/688505161 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:23.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.694+0000 7f7c6dd9b700 1 -- 172.21.15.102:0/688505161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c700fe440 msgr2=0x7f7c7010ee90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-26T18:54:23.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.694+0000 7f7c6dd9b700 1 --2- 172.21.15.102:0/688505161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c700fe440 0x7f7c7010ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:23.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.694+0000 7f7c6dd9b700 1 -- 172.21.15.102:0/688505161 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c580095f0 con 0x7f7c7010f3d0 2024-03-26T18:54:23.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.695+0000 7f7c6dd9b700 1 --2- 172.21.15.102:0/688505161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7010f3d0 0x7f7c701aa2e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7c700ff180 tx=0x7f7c5800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:23.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.697+0000 7f7c677fe700 1 -- 172.21.15.102:0/688505161 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c5801d070 con 0x7f7c7010f3d0 2024-03-26T18:54:23.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.697+0000 7f7c677fe700 1 -- 172.21.15.102:0/688505161 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7c5800f460 con 0x7f7c7010f3d0 2024-03-26T18:54:23.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.697+0000 7f7c677fe700 1 -- 172.21.15.102:0/688505161 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c58005720 con 0x7f7c7010f3d0 2024-03-26T18:54:23.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.698+0000 7f7c74d00700 1 -- 172.21.15.102:0/688505161 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c701aa820 con 0x7f7c7010f3d0 2024-03-26T18:54:23.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.698+0000 7f7c74d00700 1 -- 172.21.15.102:0/688505161 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c701aab80 con 0x7f7c7010f3d0 2024-03-26T18:54:23.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.700+0000 7f7c74d00700 1 -- 172.21.15.102:0/688505161 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c701086a0 con 0x7f7c7010f3d0 2024-03-26T18:54:23.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.701+0000 7f7c677fe700 1 -- 172.21.15.102:0/688505161 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f7c5800fa90 con 0x7f7c7010f3d0 2024-03-26T18:54:23.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.702+0000 7f7c677fe700 1 --2- 172.21.15.102:0/688505161 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7c5c06c6d0 0x7f7c5c06eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:23.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.704+0000 7f7c677fe700 1 -- 172.21.15.102:0/688505161 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7c5805ad90 con 0x7f7c7010f3d0 2024-03-26T18:54:23.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.709+0000 7f7c6e59c700 1 --2- 172.21.15.102:0/688505161 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7c5c06c6d0 0x7f7c5c06eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:23.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.709+0000 7f7c6e59c700 1 --2- 172.21.15.102:0/688505161 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7c5c06c6d0 0x7f7c5c06eb80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009730 tx=0x7f7c60006cb0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:23.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:23.713+0000 7f7c677fe700 1 -- 172.21.15.102:0/688505161 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7c580bd720 con 0x7f7c7010f3d0 2024-03-26T18:54:23.713 INFO:tasks.workunit.client.1.smithi190.stdout:2/150: dwrite d6/fe [4194304,4194304] 0 2024-03-26T18:54:23.713 INFO:tasks.workunit.client.1.smithi190.stdout:2/151: fsync d6/fe 0 2024-03-26T18:54:23.715 INFO:tasks.workunit.client.1.smithi190.stdout:4/150: fsync d7/d8/d14/f1a 0 2024-03-26T18:54:23.754 INFO:tasks.workunit.client.1.smithi190.stdout:0/227: dwrite d0/dd/d20/d1c/f29 [0,4194304] 0 2024-03-26T18:54:23.770 INFO:tasks.workunit.client.1.smithi190.stdout:4/151: read d7/d8/f1e [7468853,116527] 0 2024-03-26T18:54:23.771 INFO:tasks.workunit.client.1.smithi190.stdout:0/228: rename d0/dd/d20/d1c/f41 to d0/dd/d20/d1c/d4e/f4f 0 2024-03-26T18:54:23.771 INFO:tasks.workunit.client.1.smithi190.stdout:0/229: fdatasync d0/dd/d20/d1c/f29 0 2024-03-26T18:54:23.771 INFO:tasks.workunit.client.1.smithi190.stdout:0/230: fdatasync d0/dd/f45 0 2024-03-26T18:54:23.782 INFO:tasks.workunit.client.1.smithi190.stdout:4/152: link d7/f13 d7/d8/f33 0 2024-03-26T18:54:23.788 INFO:tasks.workunit.client.1.smithi190.stdout:4/153: dread d7/d8/f22 [0,4194304] 0 2024-03-26T18:54:23.790 INFO:tasks.workunit.client.1.smithi190.stdout:0/231: link d0/dc/l31 d0/dd/d20/d1c/d4e/l50 0 2024-03-26T18:54:23.801 INFO:tasks.workunit.client.1.smithi190.stdout:4/154: getdents d7/d8/d14 0 2024-03-26T18:54:23.801 INFO:tasks.workunit.client.1.smithi190.stdout:4/155: dread - d7/d8/d14/d23/d28/f2e zero size 2024-03-26T18:54:23.805 INFO:tasks.workunit.client.1.smithi190.stdout:0/232: write d0/dd/f12 [3497448,77989] 0 2024-03-26T18:54:23.814 INFO:tasks.workunit.client.1.smithi190.stdout:4/156: symlink d7/d8/d1c/l34 0 2024-03-26T18:54:23.814 INFO:tasks.workunit.client.1.smithi190.stdout:0/233: rename d0/dc/l35 to d0/l51 0 2024-03-26T18:54:23.819 INFO:tasks.workunit.client.1.smithi190.stdout:4/157: creat d7/f35 x:0 0 0 2024-03-26T18:54:23.824 INFO:tasks.workunit.client.1.smithi190.stdout:0/234: truncate d0/dd/f22 3552170 0 2024-03-26T18:54:23.834 INFO:tasks.workunit.client.1.smithi190.stdout:4/158: rename d7/lb to d7/d8/d14/d23/d28/l36 0 2024-03-26T18:54:23.837 INFO:tasks.workunit.client.1.smithi190.stdout:3/156: write d1/de/f1d [3945491,66997] 0 2024-03-26T18:54:23.837 INFO:tasks.workunit.client.1.smithi190.stdout:3/157: readlink d1/l1f 0 2024-03-26T18:54:23.837 INFO:tasks.workunit.client.1.smithi190.stdout:3/158: write d1/d3/f10 [2468106,111643] 0 2024-03-26T18:54:23.837 INFO:tasks.workunit.client.1.smithi190.stdout:7/131: dwrite d3/d6/d9/f2e [0,4194304] 0 2024-03-26T18:54:23.838 INFO:tasks.workunit.client.1.smithi190.stdout:6/169: read d17/f23 [145444,86434] 0 2024-03-26T18:54:23.844 INFO:tasks.workunit.client.1.smithi190.stdout:9/167: dwrite f8 [4194304,4194304] 0 2024-03-26T18:54:23.858 INFO:tasks.workunit.client.1.smithi190.stdout:3/159: rename d1/d3/f28 to d1/d3/f34 0 2024-03-26T18:54:23.869 INFO:tasks.workunit.client.1.smithi190.stdout:3/160: getdents d1/de 0 2024-03-26T18:54:23.870 INFO:tasks.workunit.client.1.smithi190.stdout:9/168: dread ff [0,4194304] 0 2024-03-26T18:54:23.870 INFO:tasks.workunit.client.1.smithi190.stdout:9/169: chown d12/d2d/f1b 14976 1 2024-03-26T18:54:23.938 INFO:tasks.workunit.client.1.smithi190.stdout:5/79: truncate d2/f4 1873803 0 2024-03-26T18:54:23.939 INFO:tasks.workunit.client.1.smithi190.stdout:5/80: readlink d2/l8 0 2024-03-26T18:54:23.939 INFO:tasks.workunit.client.1.smithi190.stdout:5/81: chown d2/f3 50013 1 2024-03-26T18:54:23.972 INFO:tasks.workunit.client.1.smithi190.stdout:6/170: dwrite d17/d28/d2c/f2e [0,4194304] 0 2024-03-26T18:54:23.981 INFO:tasks.workunit.client.1.smithi190.stdout:6/171: mknod d17/d28/d38/c3d 0 2024-03-26T18:54:23.988 INFO:tasks.workunit.client.1.smithi190.stdout:0/235: rmdir d0/dd/d20/d1c 39 2024-03-26T18:54:23.993 INFO:tasks.workunit.client.1.smithi190.stdout:6/172: dread f8 [0,4194304] 0 2024-03-26T18:54:23.999 INFO:tasks.workunit.client.1.smithi190.stdout:6/173: mkdir d17/d28/d3e 0 2024-03-26T18:54:23.999 INFO:tasks.workunit.client.1.smithi190.stdout:6/174: truncate d17/d28/d2b/f3c 21333 0 2024-03-26T18:54:23.999 INFO:tasks.workunit.client.1.smithi190.stdout:6/175: readlink d17/l30 0 2024-03-26T18:54:24.003 INFO:tasks.workunit.client.1.smithi190.stdout:6/176: symlink d17/d28/d2b/l3f 0 2024-03-26T18:54:24.012 INFO:tasks.workunit.client.1.smithi190.stdout:6/177: mknod d17/c40 0 2024-03-26T18:54:24.024 INFO:tasks.workunit.client.1.smithi190.stdout:1/131: dwrite d3/da/dd/f12 [0,4194304] 0 2024-03-26T18:54:24.029 INFO:tasks.workunit.client.1.smithi190.stdout:8/122: dwrite f6 [0,4194304] 0 2024-03-26T18:54:24.030 INFO:tasks.workunit.client.1.smithi190.stdout:0/236: dwrite d0/dd/f22 [0,4194304] 0 2024-03-26T18:54:24.036 INFO:tasks.workunit.client.1.smithi190.stdout:8/123: rename d9/l22 to d9/l26 0 2024-03-26T18:54:24.036 INFO:tasks.workunit.client.1.smithi190.stdout:8/124: write d9/f14 [840687,25057] 0 2024-03-26T18:54:24.038 INFO:tasks.workunit.client.1.smithi190.stdout:0/237: mknod d0/dd/d24/d4b/c52 0 2024-03-26T18:54:24.039 INFO:tasks.workunit.client.1.smithi190.stdout:9/170: truncate f8 5149967 0 2024-03-26T18:54:24.043 INFO:tasks.workunit.client.1.smithi190.stdout:8/125: creat d9/dd/d23/f27 x:0 0 0 2024-03-26T18:54:24.045 INFO:tasks.workunit.client.1.smithi190.stdout:0/238: truncate d0/dd/d24/f2a 1279005 0 2024-03-26T18:54:24.050 INFO:tasks.workunit.client.1.smithi190.stdout:9/171: link d12/d2d/c3c d12/d2d/d1d/d23/d2a/c41 0 2024-03-26T18:54:24.050 INFO:tasks.workunit.client.1.smithi190.stdout:9/172: fdatasync fb 0 2024-03-26T18:54:24.055 INFO:tasks.workunit.client.1.smithi190.stdout:8/126: symlink d9/dd/l28 0 2024-03-26T18:54:24.057 INFO:tasks.workunit.client.1.smithi190.stdout:1/132: dwrite d3/d9/f2a [0,4194304] 0 2024-03-26T18:54:24.057 INFO:tasks.workunit.client.1.smithi190.stdout:1/133: creat d3/f2b x:0 0 0 2024-03-26T18:54:24.059 INFO:tasks.workunit.client.1.smithi190.stdout:4/159: dwrite d7/f17 [0,4194304] 0 2024-03-26T18:54:24.060 INFO:tasks.workunit.client.1.smithi190.stdout:9/173: creat d12/d2d/d1d/d23/d2a/f42 x:0 0 0 2024-03-26T18:54:24.060 INFO:tasks.workunit.client.1.smithi190.stdout:9/174: chown d12/d2d/l26 33 1 2024-03-26T18:54:24.063 INFO:tasks.workunit.client.1.smithi190.stdout:8/127: symlink d9/dd/l29 0 2024-03-26T18:54:24.074 INFO:tasks.workunit.client.1.smithi190.stdout:1/134: creat d3/d9/db/f2c x:0 0 0 2024-03-26T18:54:24.079 INFO:tasks.workunit.client.1.smithi190.stdout:9/175: unlink d12/d2d/f35 0 2024-03-26T18:54:24.081 INFO:tasks.workunit.client.1.smithi190.stdout:4/160: dwrite d7/f10 [0,4194304] 0 2024-03-26T18:54:24.082 INFO:tasks.workunit.client.1.smithi190.stdout:8/128: mknod d9/d16/c2a 0 2024-03-26T18:54:24.082 INFO:tasks.workunit.client.1.smithi190.stdout:8/129: readlink d9/dd/le 0 2024-03-26T18:54:24.082 INFO:tasks.workunit.client.1.smithi190.stdout:8/130: write d9/dd/d15/f1c [1264771,91916] 0 2024-03-26T18:54:24.084 INFO:tasks.workunit.client.1.smithi190.stdout:1/135: symlink d3/d9/db/l2d 0 2024-03-26T18:54:24.087 INFO:tasks.workunit.client.1.smithi190.stdout:9/176: symlink d12/d2d/d1d/l43 0 2024-03-26T18:54:24.088 INFO:tasks.workunit.client.1.smithi190.stdout:9/177: chown d12/f25 0 1 2024-03-26T18:54:24.095 INFO:tasks.workunit.client.1.smithi190.stdout:4/161: symlink d7/d8/d14/d23/d28/l37 0 2024-03-26T18:54:24.095 INFO:tasks.workunit.client.1.smithi190.stdout:4/162: write d7/d8/d14/f29 [1038152,120653] 0 2024-03-26T18:54:24.098 INFO:tasks.workunit.client.1.smithi190.stdout:1/136: rename d3/da/d1e to d3/da/d2e 0 2024-03-26T18:54:24.099 INFO:tasks.workunit.client.1.smithi190.stdout:9/178: symlink d12/d2d/l44 0 2024-03-26T18:54:24.101 INFO:tasks.workunit.client.1.smithi190.stdout:4/163: dread d7/d8/f1e [4194304,4194304] 0 2024-03-26T18:54:24.102 INFO:tasks.workunit.client.1.smithi190.stdout:1/137: dread d3/d9/f20 [0,4194304] 0 2024-03-26T18:54:24.102 INFO:tasks.workunit.client.1.smithi190.stdout:9/179: symlink d12/d2d/d1d/d23/l45 0 2024-03-26T18:54:24.103 INFO:tasks.workunit.client.1.smithi190.stdout:8/131: dwrite d9/dd/d15/f1c [0,4194304] 0 2024-03-26T18:54:24.103 INFO:tasks.workunit.client.1.smithi190.stdout:8/132: creat d9/d16/f2b x:0 0 0 2024-03-26T18:54:24.104 INFO:tasks.workunit.client.1.smithi190.stdout:8/133: stat d9/fa 0 2024-03-26T18:54:24.104 INFO:tasks.workunit.client.1.smithi190.stdout:8/134: dread - d9/d16/f1e zero size 2024-03-26T18:54:24.104 INFO:tasks.workunit.client.1.smithi190.stdout:8/135: chown d9/dd/f20 2046295724 1 2024-03-26T18:54:24.104 INFO:tasks.workunit.client.1.smithi190.stdout:8/136: write d9/f1a [177444,120825] 0 2024-03-26T18:54:24.104 INFO:tasks.workunit.client.1.smithi190.stdout:8/137: readlink d9/l26 0 2024-03-26T18:54:24.104 INFO:tasks.workunit.client.1.smithi190.stdout:4/164: mkdir d7/d8/d14/d23/d28/d38 0 2024-03-26T18:54:24.104 INFO:tasks.workunit.client.1.smithi190.stdout:4/165: creat d7/d8/d1c/f39 x:0 0 0 2024-03-26T18:54:24.109 INFO:tasks.workunit.client.1.smithi190.stdout:1/138: dread d3/d9/f16 [0,4194304] 0 2024-03-26T18:54:24.112 INFO:tasks.workunit.client.1.smithi190.stdout:4/166: link d7/f9 d7/d8/f3a 0 2024-03-26T18:54:24.127 INFO:tasks.workunit.client.1.smithi190.stdout:4/167: write d7/d8/f22 [1424808,88692] 0 2024-03-26T18:54:24.127 INFO:tasks.workunit.client.1.smithi190.stdout:4/168: truncate d7/f35 1026353 0 2024-03-26T18:54:24.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.136+0000 7f7c74d00700 1 -- 172.21.15.102:0/688505161 --> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c70061190 con 0x7f7c5c06c6d0 2024-03-26T18:54:24.129 INFO:tasks.workunit.client.1.smithi190.stdout:3/161: dwrite d1/de/f1d [0,4194304] 0 2024-03-26T18:54:24.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.138+0000 7f7c677fe700 1 -- 172.21.15.102:0/688505161 <== mgr.14211 v2:172.21.15.102:6800/748923615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7f7c70061190 con 0x7f7c5c06c6d0 2024-03-26T18:54:24.130 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:54:24.130 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:54:24.131 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:54:24.131 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:54:24.131 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-03-26T18:54:24.131 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "0/23 daemons upgraded", 2024-03-26T18:54:24.131 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading mgr daemons", 2024-03-26T18:54:24.131 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:54:24.131 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:54:24.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.140+0000 7f7c74d00700 1 -- 172.21.15.102:0/688505161 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7c5c06c6d0 msgr2=0x7f7c5c06eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:24.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.140+0000 7f7c74d00700 1 --2- 172.21.15.102:0/688505161 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7c5c06c6d0 0x7f7c5c06eb80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009730 tx=0x7f7c60006cb0 comp rx=0 tx=0).stop 2024-03-26T18:54:24.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.140+0000 7f7c74d00700 1 -- 172.21.15.102:0/688505161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7010f3d0 msgr2=0x7f7c701aa2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:24.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.140+0000 7f7c74d00700 1 --2- 172.21.15.102:0/688505161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7010f3d0 0x7f7c701aa2e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7c700ff180 tx=0x7f7c5800be20 comp rx=0 tx=0).stop 2024-03-26T18:54:24.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.140+0000 7f7c74d00700 1 -- 172.21.15.102:0/688505161 shutdown_connections 2024-03-26T18:54:24.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.140+0000 7f7c74d00700 1 --2- 172.21.15.102:0/688505161 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7f7c5c06c6d0 0x7f7c5c06eb80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:24.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.140+0000 7f7c74d00700 1 --2- 172.21.15.102:0/688505161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c700fe440 0x7f7c7010ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:24.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.140+0000 7f7c74d00700 1 --2- 172.21.15.102:0/688505161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7010f3d0 0x7f7c701aa2e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:24.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.140+0000 7f7c74d00700 1 -- 172.21.15.102:0/688505161 >> 172.21.15.102:0/688505161 conn(0x7f7c700f9e50 msgr2=0x7f7c701057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:24.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.140+0000 7f7c74d00700 1 -- 172.21.15.102:0/688505161 shutdown_connections 2024-03-26T18:54:24.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.140+0000 7f7c74d00700 1 -- 172.21.15.102:0/688505161 wait complete. 2024-03-26T18:54:24.134 INFO:tasks.workunit.client.1.smithi190.stdout:3/162: mkdir d1/d3/df/d18/d35 0 2024-03-26T18:54:24.134 INFO:tasks.workunit.client.1.smithi190.stdout:3/163: chown d1/d3/df/d18/d35 189 1 2024-03-26T18:54:24.134 INFO:tasks.workunit.client.1.smithi190.stdout:3/164: mkdir d1/de/d1c/d36 0 2024-03-26T18:54:24.135 INFO:tasks.workunit.client.1.smithi190.stdout:3/165: chown d1/de/f29 42899 1 2024-03-26T18:54:24.136 INFO:tasks.workunit.client.1.smithi190.stdout:3/166: truncate d1/d3/f13 4117411 0 2024-03-26T18:54:24.139 INFO:tasks.workunit.client.1.smithi190.stdout:1/139: dwrite d3/d9/db/f2c [0,4194304] 0 2024-03-26T18:54:24.200 INFO:tasks.workunit.client.1.smithi190.stdout:9/180: rename d12/f40 to d12/d2d/f46 0 2024-03-26T18:54:24.200 INFO:tasks.workunit.client.1.smithi190.stdout:9/181: chown d12/d2d/f31 1549191734 1 2024-03-26T18:54:24.200 INFO:tasks.workunit.client.1.smithi190.stdout:9/182: creat d12/d2d/f47 x:0 0 0 2024-03-26T18:54:24.200 INFO:tasks.workunit.client.1.smithi190.stdout:9/183: chown d12/c2f 49489335 1 2024-03-26T18:54:24.203 INFO:tasks.workunit.client.1.smithi190.stdout:8/138: rename d9/d16/f1b to d9/dd/d15/f2c 0 2024-03-26T18:54:24.210 INFO:tasks.workunit.client.1.smithi190.stdout:4/169: rename d7/d8/l1b to d7/d8/d14/d23/l3b 0 2024-03-26T18:54:24.211 INFO:tasks.workunit.client.1.smithi190.stdout:4/170: readlink d7/d8/d14/d23/d28/l36 0 2024-03-26T18:54:24.211 INFO:tasks.workunit.client.1.smithi190.stdout:4/171: creat d7/d1d/f3c x:0 0 0 2024-03-26T18:54:24.211 INFO:tasks.workunit.client.1.smithi190.stdout:8/139: dread f8 [0,4194304] 0 2024-03-26T18:54:24.212 INFO:tasks.workunit.client.1.smithi190.stdout:0/239: write d0/dd/d24/f2a [621344,109063] 0 2024-03-26T18:54:24.212 INFO:tasks.workunit.client.1.smithi190.stdout:0/240: write d0/dd/f22 [4204800,81769] 0 2024-03-26T18:54:24.215 INFO:tasks.workunit.client.1.smithi190.stdout:4/172: link d7/d8/d14/f19 d7/d8/d1c/d20/f3d 0 2024-03-26T18:54:24.217 INFO:tasks.workunit.client.1.smithi190.stdout:8/140: rename d9/dd/d15/f18 to d9/dd/f2d 0 2024-03-26T18:54:24.217 INFO:tasks.workunit.client.1.smithi190.stdout:8/141: readlink l1 0 2024-03-26T18:54:24.220 INFO:tasks.workunit.client.1.smithi190.stdout:0/241: symlink d0/dd/d24/l53 0 2024-03-26T18:54:24.221 INFO:tasks.workunit.client.1.smithi190.stdout:8/142: write f8 [2987589,88194] 0 2024-03-26T18:54:24.221 INFO:tasks.workunit.client.1.smithi190.stdout:8/143: chown d9/d16/f1f 19650 1 2024-03-26T18:54:24.221 INFO:tasks.workunit.client.1.smithi190.stdout:8/144: chown d9/f14 14780 1 2024-03-26T18:54:24.225 INFO:tasks.workunit.client.1.smithi190.stdout:0/242: dread d0/dd/d24/f2b [0,4194304] 0 2024-03-26T18:54:24.225 INFO:tasks.workunit.client.1.smithi190.stdout:8/145: mknod d9/dd/d23/c2e 0 2024-03-26T18:54:24.227 INFO:tasks.workunit.client.1.smithi190.stdout:4/173: getdents d7 0 2024-03-26T18:54:24.227 INFO:tasks.workunit.client.1.smithi190.stdout:4/174: readlink d7/d8/l2d 0 2024-03-26T18:54:24.231 INFO:tasks.workunit.client.1.smithi190.stdout:8/146: link d9/dd/d23/f27 d9/dd/d15/f2f 0 2024-03-26T18:54:24.234 INFO:tasks.workunit.client.1.smithi190.stdout:4/175: unlink d7/d8/d14/f29 0 2024-03-26T18:54:24.234 INFO:tasks.workunit.client.1.smithi190.stdout:4/176: chown d7/d8/d14 0 1 2024-03-26T18:54:24.234 INFO:tasks.workunit.client.1.smithi190.stdout:4/177: chown d7/d8/d1c/d20 98 1 2024-03-26T18:54:24.236 INFO:tasks.workunit.client.1.smithi190.stdout:8/147: creat d9/dd/d15/f30 x:0 0 0 2024-03-26T18:54:24.237 INFO:tasks.workunit.client.1.smithi190.stdout:8/148: dread d9/dd/f2d [0,4194304] 0 2024-03-26T18:54:24.237 INFO:tasks.workunit.client.1.smithi190.stdout:8/149: write d9/dd/f2d [737322,89517] 0 2024-03-26T18:54:24.237 INFO:tasks.workunit.client.1.smithi190.stdout:8/150: readlink d9/dd/l29 0 2024-03-26T18:54:24.238 INFO:tasks.workunit.client.1.smithi190.stdout:4/178: link d7/d8/d1c/l25 d7/d1d/l3e 0 2024-03-26T18:54:24.241 INFO:tasks.workunit.client.1.smithi190.stdout:8/151: rename f8 to d9/dd/d23/f31 0 2024-03-26T18:54:24.243 INFO:tasks.workunit.client.1.smithi190.stdout:8/152: mkdir d9/dd/d15/d32 0 2024-03-26T18:54:24.243 INFO:tasks.workunit.client.1.smithi190.stdout:8/153: stat f6 0 2024-03-26T18:54:24.257 INFO:tasks.workunit.client.1.smithi190.stdout:4/179: dwrite d7/d8/d1c/f39 [0,4194304] 0 2024-03-26T18:54:24.263 INFO:tasks.workunit.client.1.smithi190.stdout:4/180: dread d7/d8/f33 [0,4194304] 0 2024-03-26T18:54:24.263 INFO:tasks.workunit.client.1.smithi190.stdout:4/181: chown l4 79 1 2024-03-26T18:54:24.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.272+0000 7fc539bc3700 1 -- 172.21.15.102:0/1336835065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc534103500 msgr2=0x7fc5341058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:24.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.272+0000 7fc539bc3700 1 --2- 172.21.15.102:0/1336835065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc534103500 0x7fc5341058e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc528009910 tx=0x7fc528009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:24.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.272+0000 7fc539bc3700 1 -- 172.21.15.102:0/1336835065 shutdown_connections 2024-03-26T18:54:24.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.272+0000 7fc539bc3700 1 --2- 172.21.15.102:0/1336835065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc534103500 0x7fc5341058e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:24.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.272+0000 7fc539bc3700 1 --2- 172.21.15.102:0/1336835065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc534100be0 0x7fc534102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:24.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.272+0000 7fc539bc3700 1 -- 172.21.15.102:0/1336835065 >> 172.21.15.102:0/1336835065 conn(0x7fc5340fa600 msgr2=0x7fc5340fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:24.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.272+0000 7fc539bc3700 1 -- 172.21.15.102:0/1336835065 shutdown_connections 2024-03-26T18:54:24.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.272+0000 7fc539bc3700 1 -- 172.21.15.102:0/1336835065 wait complete. 2024-03-26T18:54:24.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.273+0000 7fc539bc3700 1 Processor -- start 2024-03-26T18:54:24.266 INFO:tasks.workunit.client.1.smithi190.stdout:4/182: link d7/d8/f3a d7/d8/d1c/d20/f3f 0 2024-03-26T18:54:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.273+0000 7fc539bc3700 1 -- start start 2024-03-26T18:54:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.273+0000 7fc539bc3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc534100be0 0x7fc534195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.273+0000 7fc539bc3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc534103500 0x7fc534195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.273+0000 7fc539bc3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5341964b0 con 0x7fc534103500 2024-03-26T18:54:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.273+0000 7fc539bc3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5341965f0 con 0x7fc534100be0 2024-03-26T18:54:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.274+0000 7fc5337fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc534100be0 0x7fc534195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.274+0000 7fc5337fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc534100be0 0x7fc534195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:54132/0 (socket says 172.21.15.102:54132) 2024-03-26T18:54:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.274+0000 7fc5337fe700 1 -- 172.21.15.102:0/3848143567 learned_addr learned my addr 172.21.15.102:0/3848143567 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.274+0000 7fc532ffd700 1 --2- 172.21.15.102:0/3848143567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc534103500 0x7fc534195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.274+0000 7fc5337fe700 1 -- 172.21.15.102:0/3848143567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc534103500 msgr2=0x7fc534195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.274+0000 7fc5337fe700 1 --2- 172.21.15.102:0/3848143567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc534103500 0x7fc534195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:24.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.274+0000 7fc5337fe700 1 -- 172.21.15.102:0/3848143567 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5280095f0 con 0x7fc534100be0 2024-03-26T18:54:24.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.274+0000 7fc5337fe700 1 --2- 172.21.15.102:0/3848143567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc534100be0 0x7fc534195950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc52400d900 tx=0x7fc52400dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:24.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.275+0000 7fc530ff9700 1 -- 172.21.15.102:0/3848143567 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5240041f0 con 0x7fc534100be0 2024-03-26T18:54:24.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.275+0000 7fc530ff9700 1 -- 172.21.15.102:0/3848143567 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc524004ce0 con 0x7fc534100be0 2024-03-26T18:54:24.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.275+0000 7fc539bc3700 1 -- 172.21.15.102:0/3848143567 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53419b0a0 con 0x7fc534100be0 2024-03-26T18:54:24.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.275+0000 7fc530ff9700 1 -- 172.21.15.102:0/3848143567 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc52400b750 con 0x7fc534100be0 2024-03-26T18:54:24.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.275+0000 7fc539bc3700 1 -- 172.21.15.102:0/3848143567 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53419b5f0 con 0x7fc534100be0 2024-03-26T18:54:24.270 INFO:tasks.workunit.client.1.smithi190.stdout:4/183: truncate d7/d1d/f27 318158 0 2024-03-26T18:54:24.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.276+0000 7fc539bc3700 1 -- 172.21.15.102:0/3848143567 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53418fb70 con 0x7fc534100be0 2024-03-26T18:54:24.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.277+0000 7fc530ff9700 1 -- 172.21.15.102:0/3848143567 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fc52400b8b0 con 0x7fc534100be0 2024-03-26T18:54:24.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.277+0000 7fc530ff9700 1 --2- 172.21.15.102:0/3848143567 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc51c06c680 0x7fc51c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:24.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.277+0000 7fc530ff9700 1 -- 172.21.15.102:0/3848143567 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc52408b760 con 0x7fc534100be0 2024-03-26T18:54:24.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.277+0000 7fc532ffd700 1 --2- 172.21.15.102:0/3848143567 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc51c06c680 0x7fc51c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:24.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.278+0000 7fc532ffd700 1 --2- 172.21.15.102:0/3848143567 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc51c06c680 0x7fc51c06eb30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc528006010 tx=0x7fc52800b560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:24.272 INFO:tasks.workunit.client.1.smithi190.stdout:4/184: rmdir d7/d8 39 2024-03-26T18:54:24.274 INFO:tasks.workunit.client.1.smithi190.stdout:4/185: mkdir d7/d8/d1c/d20/d40 0 2024-03-26T18:54:24.274 INFO:tasks.workunit.client.1.smithi190.stdout:4/186: chown l4 111 1 2024-03-26T18:54:24.274 INFO:tasks.workunit.client.1.smithi190.stdout:4/187: stat d7/d8/d1c 0 2024-03-26T18:54:24.276 INFO:tasks.workunit.client.1.smithi190.stdout:4/188: mkdir d7/d1d/d41 0 2024-03-26T18:54:24.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.285+0000 7fc530ff9700 1 -- 172.21.15.102:0/3848143567 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc5240598f0 con 0x7fc534100be0 2024-03-26T18:54:24.278 INFO:tasks.workunit.client.1.smithi190.stdout:4/189: rmdir d7/d8/d14/d23 39 2024-03-26T18:54:24.281 INFO:tasks.workunit.client.1.smithi190.stdout:4/190: truncate d7/f9 437318 0 2024-03-26T18:54:24.285 INFO:tasks.workunit.client.1.smithi190.stdout:4/191: mkdir d7/d1d/d42 0 2024-03-26T18:54:24.285 INFO:tasks.workunit.client.1.smithi190.stdout:4/192: truncate d7/d8/f2a 620754 0 2024-03-26T18:54:24.287 INFO:tasks.workunit.client.1.smithi190.stdout:4/193: rmdir d7/d8/d1c/d20 39 2024-03-26T18:54:24.290 INFO:tasks.workunit.client.1.smithi190.stdout:4/194: symlink d7/l43 0 2024-03-26T18:54:24.293 INFO:tasks.workunit.client.1.smithi190.stdout:4/195: unlink d7/d8/d14/c24 0 2024-03-26T18:54:24.301 INFO:tasks.workunit.client.1.smithi190.stdout:4/196: dread d7/d8/d14/f19 [0,4194304] 0 2024-03-26T18:54:24.301 INFO:tasks.workunit.client.1.smithi190.stdout:4/197: fdatasync d7/f9 0 2024-03-26T18:54:24.305 INFO:tasks.workunit.client.1.smithi190.stdout:4/198: getdents d7/d8/d14/d23/d28 0 2024-03-26T18:54:24.352 INFO:tasks.workunit.client.1.smithi190.stdout:9/184: getdents d12/d2d 0 2024-03-26T18:54:24.353 INFO:tasks.workunit.client.1.smithi190.stdout:9/185: mknod d12/d2d/d1d/d23/c48 0 2024-03-26T18:54:24.354 INFO:tasks.workunit.client.1.smithi190.stdout:9/186: write d12/d2d/d1d/d23/f33 [1065853,128670] 0 2024-03-26T18:54:24.356 INFO:tasks.workunit.client.1.smithi190.stdout:9/187: creat d12/d2d/f49 x:0 0 0 2024-03-26T18:54:24.376 INFO:tasks.workunit.client.1.smithi190.stdout:8/154: fsync d9/dd/d15/f30 0 2024-03-26T18:54:24.376 INFO:tasks.workunit.client.1.smithi190.stdout:8/155: stat d9/d16/c2a 0 2024-03-26T18:54:24.381 INFO:tasks.workunit.client.1.smithi190.stdout:9/188: dwrite d12/d2d/f22 [0,4194304] 0 2024-03-26T18:54:24.381 INFO:tasks.workunit.client.1.smithi190.stdout:9/189: write d12/d2d/f1b [652394,89297] 0 2024-03-26T18:54:24.381 INFO:tasks.workunit.client.1.smithi190.stdout:9/190: chown f8 6825734 1 2024-03-26T18:54:24.386 INFO:tasks.workunit.client.1.smithi190.stdout:0/243: dwrite d0/dd/d20/f30 [0,4194304] 0 2024-03-26T18:54:24.386 INFO:tasks.workunit.client.1.smithi190.stdout:0/244: creat d0/dd/d24/f54 x:0 0 0 2024-03-26T18:54:24.395 INFO:tasks.workunit.client.1.smithi190.stdout:0/245: mknod d0/dd/d20/d1c/d46/c55 0 2024-03-26T18:54:24.395 INFO:tasks.workunit.client.1.smithi190.stdout:0/246: truncate d0/dd/d24/f54 759860 0 2024-03-26T18:54:24.396 INFO:tasks.workunit.client.1.smithi190.stdout:0/247: rmdir d0/dd/d20/d1c 39 2024-03-26T18:54:24.397 INFO:tasks.workunit.client.1.smithi190.stdout:9/191: dwrite d12/d2d/f31 [0,4194304] 0 2024-03-26T18:54:24.398 INFO:tasks.workunit.client.1.smithi190.stdout:0/248: rename d0/dd/f39 to d0/dd/d20/d1c/f56 0 2024-03-26T18:54:24.402 INFO:tasks.workunit.client.1.smithi190.stdout:9/192: symlink d12/l4a 0 2024-03-26T18:54:24.402 INFO:tasks.workunit.client.1.smithi190.stdout:9/193: chown d12/f25 41702 1 2024-03-26T18:54:24.403 INFO:tasks.workunit.client.1.smithi190.stdout:0/249: mknod d0/dd/d26/c57 0 2024-03-26T18:54:24.405 INFO:tasks.workunit.client.1.smithi190.stdout:9/194: mkdir d12/d2d/d1d/d23/d2a/d3f/d4b 0 2024-03-26T18:54:24.405 INFO:tasks.workunit.client.1.smithi190.stdout:9/195: read d12/d2d/f1b [263983,127175] 0 2024-03-26T18:54:24.407 INFO:tasks.workunit.client.1.smithi190.stdout:0/250: mkdir d0/dd/d20/d1c/d46/d58 0 2024-03-26T18:54:24.407 INFO:tasks.workunit.client.1.smithi190.stdout:0/251: chown d0/f13 4940 1 2024-03-26T18:54:24.409 INFO:tasks.workunit.client.1.smithi190.stdout:0/252: dread d0/dd/d26/f3f [0,4194304] 0 2024-03-26T18:54:24.412 INFO:tasks.workunit.client.1.smithi190.stdout:4/199: dread d7/d1d/f27 [0,4194304] 0 2024-03-26T18:54:24.412 INFO:tasks.workunit.client.1.smithi190.stdout:0/253: truncate d0/f6 1345162 0 2024-03-26T18:54:24.416 INFO:tasks.workunit.client.1.smithi190.stdout:4/200: rename d7/d8/d14/d23/d28/f2e to d7/d8/d1c/f44 0 2024-03-26T18:54:24.418 INFO:tasks.workunit.client.1.smithi190.stdout:0/254: unlink d0/dd/d24/l53 0 2024-03-26T18:54:24.420 INFO:tasks.workunit.client.1.smithi190.stdout:4/201: link d7/d8/d14/c16 d7/d8/d1c/c45 0 2024-03-26T18:54:24.422 INFO:tasks.workunit.client.1.smithi190.stdout:0/255: link d0/dd/d20/d1c/c3c d0/dd/d24/d4b/c59 0 2024-03-26T18:54:24.422 INFO:tasks.workunit.client.1.smithi190.stdout:0/256: chown d0/dd/d20/d1c/c2e 2 1 2024-03-26T18:54:24.422 INFO:tasks.workunit.client.1.smithi190.stdout:0/257: truncate d0/dd/d26/f34 2307121 0 2024-03-26T18:54:24.424 INFO:tasks.workunit.client.1.smithi190.stdout:9/196: dwrite d12/f25 [0,4194304] 0 2024-03-26T18:54:24.424 INFO:tasks.workunit.client.1.smithi190.stdout:9/197: readlink d12/d2d/l29 0 2024-03-26T18:54:24.424 INFO:tasks.workunit.client.1.smithi190.stdout:9/198: write d12/f13 [4358497,97424] 0 2024-03-26T18:54:24.424 INFO:tasks.workunit.client.1.smithi190.stdout:9/199: creat d12/f4c x:0 0 0 2024-03-26T18:54:24.427 INFO:tasks.workunit.client.1.smithi190.stdout:0/258: rename d0/l8 to d0/dd/d26/l5a 0 2024-03-26T18:54:24.435 INFO:tasks.workunit.client.1.smithi190.stdout:9/200: link d12/d2d/f39 d12/d2d/d1d/d23/d2a/d3f/f4d 0 2024-03-26T18:54:24.437 INFO:tasks.workunit.client.1.smithi190.stdout:0/259: creat d0/f5b x:0 0 0 2024-03-26T18:54:24.437 INFO:tasks.workunit.client.1.smithi190.stdout:0/260: chown d0/dd/d20/d1c/c2e 4059805 1 2024-03-26T18:54:24.440 INFO:tasks.workunit.client.1.smithi190.stdout:0/261: rename d0/dd/d20/d1c/l3e to d0/dd/d20/l5c 0 2024-03-26T18:54:24.443 INFO:tasks.workunit.client.1.smithi190.stdout:0/262: creat d0/dd/d20/d1c/d46/f5d x:0 0 0 2024-03-26T18:54:24.443 INFO:tasks.workunit.client.1.smithi190.stdout:4/202: dwrite d7/d8/d14/f2b [0,4194304] 0 2024-03-26T18:54:24.445 INFO:tasks.workunit.client.1.smithi190.stdout:0/263: mkdir d0/dd/d24/d4b/d5e 0 2024-03-26T18:54:24.449 INFO:tasks.workunit.client.1.smithi190.stdout:4/203: dread d7/d8/d1c/d20/f3d [4194304,4194304] 0 2024-03-26T18:54:24.450 INFO:tasks.workunit.client.1.smithi190.stdout:0/264: unlink d0/dd/d20/d1c/c2e 0 2024-03-26T18:54:24.468 INFO:tasks.workunit.client.1.smithi190.stdout:0/265: dwrite d0/dd/d24/f2f [0,4194304] 0 2024-03-26T18:54:24.470 INFO:tasks.workunit.client.1.smithi190.stdout:0/266: creat d0/dd/d24/d4b/d5e/f5f x:0 0 0 2024-03-26T18:54:24.470 INFO:tasks.workunit.client.1.smithi190.stdout:0/267: chown d0/l51 238698831 1 2024-03-26T18:54:24.472 INFO:tasks.workunit.client.1.smithi190.stdout:0/268: symlink d0/l60 0 2024-03-26T18:54:24.474 INFO:tasks.workunit.client.1.smithi190.stdout:0/269: mkdir d0/dd/d20/d61 0 2024-03-26T18:54:24.475 INFO:tasks.workunit.client.1.smithi190.stdout:0/270: write d0/dd/d24/f2a [1682260,130290] 0 2024-03-26T18:54:24.477 INFO:tasks.workunit.client.1.smithi190.stdout:0/271: rename d0/dd/d20/d1c/d46/c55 to d0/dd/d26/c62 0 2024-03-26T18:54:24.480 INFO:tasks.workunit.client.1.smithi190.stdout:0/272: read d0/f13 [709883,120302] 0 2024-03-26T18:54:24.482 INFO:tasks.workunit.client.1.smithi190.stdout:0/273: symlink d0/dd/d20/l63 0 2024-03-26T18:54:24.484 INFO:tasks.workunit.client.1.smithi190.stdout:0/274: mkdir d0/dd/d24/d4b/d5e/d64 0 2024-03-26T18:54:24.484 INFO:tasks.workunit.client.1.smithi190.stdout:0/275: stat d0/dd/d20/l3d 0 2024-03-26T18:54:24.487 INFO:tasks.workunit.client.1.smithi190.stdout:0/276: link d0/l3 d0/dd/d24/d4b/d5e/l65 0 2024-03-26T18:54:24.489 INFO:tasks.workunit.client.1.smithi190.stdout:0/277: truncate d0/dd/d20/d1c/f29 905713 0 2024-03-26T18:54:24.491 INFO:tasks.workunit.client.1.smithi190.stdout:0/278: truncate d0/dd/f45 584814 0 2024-03-26T18:54:24.493 INFO:tasks.workunit.client.1.smithi190.stdout:0/279: rename d0/dd/d24/d4b/d5e/d64 to d0/dd/d24/d4b/d5e/d66 0 2024-03-26T18:54:24.496 INFO:tasks.workunit.client.1.smithi190.stdout:0/280: symlink d0/dd/d20/d1c/d46/l67 0 2024-03-26T18:54:24.496 INFO:tasks.workunit.client.1.smithi190.stdout:0/281: dread - d0/dd/d20/d1c/d4e/f4f zero size 2024-03-26T18:54:24.496 INFO:tasks.workunit.client.1.smithi190.stdout:0/282: fdatasync d0/dd/d24/f54 0 2024-03-26T18:54:24.498 INFO:tasks.workunit.client.1.smithi190.stdout:0/283: creat d0/dd/d20/f68 x:0 0 0 2024-03-26T18:54:24.500 INFO:tasks.workunit.client.1.smithi190.stdout:0/284: symlink d0/dd/d24/d4b/l69 0 2024-03-26T18:54:24.511 INFO:tasks.workunit.client.1.smithi190.stdout:8/156: truncate d9/dd/f20 2881273 0 2024-03-26T18:54:24.528 INFO:tasks.workunit.client.1.smithi190.stdout:9/201: write d12/f18 [5062749,121575] 0 2024-03-26T18:54:24.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:24 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2138164389' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:54:24.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:24 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:24.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:24 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.zrixos", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:54:24.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:24 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:54:24.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:24 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:54:24.536 INFO:tasks.workunit.client.1.smithi190.stdout:9/202: dread d12/f16 [0,4194304] 0 2024-03-26T18:54:24.537 INFO:tasks.workunit.client.1.smithi190.stdout:9/203: chown d12/d2d/l44 1792619 1 2024-03-26T18:54:24.546 INFO:tasks.workunit.client.1.smithi190.stdout:4/204: getdents d7/d8/d1c 0 2024-03-26T18:54:24.547 INFO:tasks.workunit.client.1.smithi190.stdout:4/205: stat d7/f13 0 2024-03-26T18:54:24.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:24 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2138164389' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:54:24.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:24 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:24.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:24 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.zrixos", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:54:24.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:24 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:54:24.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:24 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:54:24.635 INFO:tasks.workunit.client.1.smithi190.stdout:8/157: dread d9/dd/d15/f1c [0,4194304] 0 2024-03-26T18:54:24.635 INFO:tasks.workunit.client.1.smithi190.stdout:8/158: creat d9/f33 x:0 0 0 2024-03-26T18:54:24.645 INFO:tasks.workunit.client.1.smithi190.stdout:9/204: truncate d12/f18 3012959 0 2024-03-26T18:54:24.646 INFO:tasks.workunit.client.1.smithi190.stdout:9/205: symlink d12/d2d/d1d/d23/d2a/d3f/d4b/l4e 0 2024-03-26T18:54:24.648 INFO:tasks.workunit.client.1.smithi190.stdout:9/206: rename d12/d2d/d1d/l43 to d12/d2d/d1d/d23/l4f 0 2024-03-26T18:54:24.649 INFO:tasks.workunit.client.1.smithi190.stdout:9/207: creat d12/d2d/d1d/d23/d2a/d3f/f50 x:0 0 0 2024-03-26T18:54:24.649 INFO:tasks.workunit.client.1.smithi190.stdout:9/208: readlink d12/d2d/l26 0 2024-03-26T18:54:24.651 INFO:tasks.workunit.client.1.smithi190.stdout:9/209: truncate d12/f16 1130230 0 2024-03-26T18:54:24.654 INFO:tasks.workunit.client.1.smithi190.stdout:8/159: dwrite d9/dd/d15/f1c [0,4194304] 0 2024-03-26T18:54:24.655 INFO:tasks.workunit.client.1.smithi190.stdout:8/160: creat d9/dd/f34 x:0 0 0 2024-03-26T18:54:24.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.674+0000 7fc539bc3700 1 -- 172.21.15.102:0/3848143567 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc53402cc30 con 0x7fc534100be0 2024-03-26T18:54:24.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.675+0000 7fc530ff9700 1 -- 172.21.15.102:0/3848143567 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fc524059480 con 0x7fc534100be0 2024-03-26T18:54:24.668 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T18:54:24.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.677+0000 7fc539bc3700 1 -- 172.21.15.102:0/3848143567 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc51c06c680 msgr2=0x7fc51c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:24.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.677+0000 7fc539bc3700 1 --2- 172.21.15.102:0/3848143567 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc51c06c680 0x7fc51c06eb30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc528006010 tx=0x7fc52800b560 comp rx=0 tx=0).stop 2024-03-26T18:54:24.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.677+0000 7fc539bc3700 1 -- 172.21.15.102:0/3848143567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc534100be0 msgr2=0x7fc534195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:24.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.677+0000 7fc539bc3700 1 --2- 172.21.15.102:0/3848143567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc534100be0 0x7fc534195950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc52400d900 tx=0x7fc52400dc10 comp rx=0 tx=0).stop 2024-03-26T18:54:24.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.678+0000 7fc539bc3700 1 -- 172.21.15.102:0/3848143567 shutdown_connections 2024-03-26T18:54:24.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.678+0000 7fc539bc3700 1 --2- 172.21.15.102:0/3848143567 >> [v2:172.21.15.102:6800/748923615,v1:172.21.15.102:6801/748923615] conn(0x7fc51c06c680 0x7fc51c06eb30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:24.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.678+0000 7fc539bc3700 1 --2- 172.21.15.102:0/3848143567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc534100be0 0x7fc534195950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:24.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.678+0000 7fc539bc3700 1 --2- 172.21.15.102:0/3848143567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc534103500 0x7fc534195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:24.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.678+0000 7fc539bc3700 1 -- 172.21.15.102:0/3848143567 >> 172.21.15.102:0/3848143567 conn(0x7fc5340fa600 msgr2=0x7fc5340fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:24.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.678+0000 7fc539bc3700 1 -- 172.21.15.102:0/3848143567 shutdown_connections 2024-03-26T18:54:24.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:24.678+0000 7fc539bc3700 1 -- 172.21.15.102:0/3848143567 wait complete. 2024-03-26T18:54:24.673 INFO:tasks.workunit.client.1.smithi190.stdout:8/161: dwrite d9/dd/f34 [0,4194304] 0 2024-03-26T18:54:24.673 INFO:tasks.workunit.client.1.smithi190.stdout:8/162: read - d9/dd/d15/f30 zero size 2024-03-26T18:54:24.689 INFO:tasks.workunit.client.1.smithi190.stdout:8/163: dwrite d9/d16/f1f [0,4194304] 0 2024-03-26T18:54:24.705 INFO:tasks.workunit.client.1.smithi190.stdout:8/164: dread d9/dd/f34 [0,4194304] 0 2024-03-26T18:54:24.706 INFO:tasks.workunit.client.1.smithi190.stdout:8/165: creat d9/dd/d15/d32/f35 x:0 0 0 2024-03-26T18:54:24.714 INFO:tasks.workunit.client.1.smithi190.stdout:9/210: dwrite ff [0,4194304] 0 2024-03-26T18:54:24.714 INFO:tasks.workunit.client.1.smithi190.stdout:9/211: rmdir d12/d2d/d1d 39 2024-03-26T18:54:24.715 INFO:tasks.workunit.client.1.smithi190.stdout:9/212: rmdir d12/d2d/d1d/d23/d2a/d3f/d4b 39 2024-03-26T18:54:24.715 INFO:tasks.workunit.client.1.smithi190.stdout:9/213: fsync f6 0 2024-03-26T18:54:24.715 INFO:tasks.workunit.client.1.smithi190.stdout:9/214: chown d12/d2d/l21 1293 1 2024-03-26T18:54:24.715 INFO:tasks.workunit.client.1.smithi190.stdout:8/166: dread f6 [0,4194304] 0 2024-03-26T18:54:24.717 INFO:tasks.workunit.client.1.smithi190.stdout:8/167: mkdir d9/dd/d23/d36 0 2024-03-26T18:54:24.718 INFO:tasks.workunit.client.1.smithi190.stdout:9/215: rmdir d12/d2d/d1d/d23/d2a 39 2024-03-26T18:54:24.718 INFO:tasks.workunit.client.1.smithi190.stdout:9/216: creat d12/d2d/d1d/d23/f51 x:0 0 0 2024-03-26T18:54:24.719 INFO:tasks.workunit.client.1.smithi190.stdout:8/168: mkdir d9/dd/d23/d36/d37 0 2024-03-26T18:54:24.720 INFO:tasks.workunit.client.1.smithi190.stdout:9/217: rmdir d12/d2d/d1d/d23/d2a/d3f 39 2024-03-26T18:54:24.720 INFO:tasks.workunit.client.1.smithi190.stdout:9/218: unlink d12/d2d/l1a 0 2024-03-26T18:54:24.737 INFO:tasks.workunit.client.1.smithi190.stdout:9/219: dwrite d12/d2d/f49 [0,4194304] 0 2024-03-26T18:54:24.761 INFO:tasks.workunit.client.1.smithi190.stdout:9/220: dwrite d12/d2d/f22 [0,4194304] 0 2024-03-26T18:54:24.761 INFO:tasks.workunit.client.1.smithi190.stdout:9/221: creat d12/f52 x:0 0 0 2024-03-26T18:54:24.761 INFO:tasks.workunit.client.1.smithi190.stdout:9/222: write f6 [3684732,115348] 0 2024-03-26T18:54:24.761 INFO:tasks.workunit.client.1.smithi190.stdout:9/223: creat d12/d2d/f53 x:0 0 0 2024-03-26T18:54:24.761 INFO:tasks.workunit.client.1.smithi190.stdout:9/224: chown d12/d2d/d1d/d23/f33 298 1 2024-03-26T18:54:24.776 INFO:tasks.workunit.client.1.smithi190.stdout:9/225: dwrite d12/d2d/d1d/d23/f33 [0,4194304] 0 2024-03-26T18:54:24.778 INFO:tasks.workunit.client.1.smithi190.stdout:9/226: mkdir d12/d2d/d1d/d54 0 2024-03-26T18:54:24.779 INFO:tasks.workunit.client.1.smithi190.stdout:9/227: mkdir d12/d2d/d1d/d23/d2a/d3f/d55 0 2024-03-26T18:54:24.781 INFO:tasks.workunit.client.1.smithi190.stdout:9/228: dread d12/f14 [0,4194304] 0 2024-03-26T18:54:24.782 INFO:tasks.workunit.client.1.smithi190.stdout:9/229: creat d12/d2d/d1d/d23/d2a/d3f/d55/f56 x:0 0 0 2024-03-26T18:54:24.782 INFO:tasks.workunit.client.1.smithi190.stdout:9/230: dread - d12/d2d/d1d/d23/f51 zero size 2024-03-26T18:54:24.782 INFO:tasks.workunit.client.1.smithi190.stdout:9/231: write d12/f38 [892875,75249] 0 2024-03-26T18:54:24.782 INFO:tasks.workunit.client.1.smithi190.stdout:9/232: fdatasync d12/d2d/f46 0 2024-03-26T18:54:24.791 INFO:tasks.workunit.client.1.smithi190.stdout:9/233: write f8 [3021599,51829] 0 2024-03-26T18:54:24.792 INFO:tasks.workunit.client.1.smithi190.stdout:9/234: rename le to d12/d2d/d1d/d54/l57 0 2024-03-26T18:54:24.794 INFO:tasks.workunit.client.1.smithi190.stdout:9/235: rename d12/d2d/d1d/d23/d2a/d3f/d4b to d12/d2d/d1d/d54/d58 0 2024-03-26T18:54:24.794 INFO:tasks.workunit.client.1.smithi190.stdout:9/236: creat d12/d2d/d1d/d23/f59 x:0 0 0 2024-03-26T18:54:24.795 INFO:tasks.workunit.client.1.smithi190.stdout:9/237: mknod d12/d2d/d1d/d23/d2a/d3f/d55/c5a 0 2024-03-26T18:54:24.796 INFO:tasks.workunit.client.1.smithi190.stdout:9/238: getdents d12/d2d/d1d/d23/d2a 0 2024-03-26T18:54:24.803 INFO:tasks.workunit.client.1.smithi190.stdout:9/239: rmdir d12/d2d/d1d/d23 39 2024-03-26T18:54:24.804 INFO:tasks.workunit.client.1.smithi190.stdout:9/240: link d12/d2d/d1d/d23/d2a/c37 d12/d2d/d1d/c5b 0 2024-03-26T18:54:24.804 INFO:tasks.workunit.client.1.smithi190.stdout:9/241: stat d12/d2d/d1d/d23/d2a/d3f 0 2024-03-26T18:54:24.815 INFO:tasks.workunit.client.1.smithi190.stdout:9/242: dread ff [0,4194304] 0 2024-03-26T18:54:24.815 INFO:tasks.workunit.client.1.smithi190.stdout:9/243: chown d12/f25 441482583 1 2024-03-26T18:54:24.815 INFO:tasks.workunit.client.1.smithi190.stdout:9/244: write d12/d2d/d1d/d23/f33 [5143965,118570] 0 2024-03-26T18:54:24.815 INFO:tasks.workunit.client.1.smithi190.stdout:9/245: stat d12/f14 0 2024-03-26T18:54:24.815 INFO:tasks.workunit.client.1.smithi190.stdout:9/246: dread - d12/d2d/f27 zero size 2024-03-26T18:54:24.815 INFO:tasks.workunit.client.1.smithi190.stdout:9/247: dread - d12/d2d/d1d/d23/d2b/f3d zero size 2024-03-26T18:54:24.831 INFO:tasks.workunit.client.1.smithi190.stdout:9/248: dwrite d12/d2d/f47 [0,4194304] 0 2024-03-26T18:54:24.855 INFO:tasks.workunit.client.1.smithi190.stdout:9/249: dwrite d12/d2d/d1d/d23/d2a/f42 [0,4194304] 0 2024-03-26T18:54:24.855 INFO:tasks.workunit.client.1.smithi190.stdout:9/250: chown d12/d2d/f31 633 1 2024-03-26T18:54:24.855 INFO:tasks.workunit.client.1.smithi190.stdout:9/251: readlink d12/l30 0 2024-03-26T18:54:24.856 INFO:tasks.workunit.client.1.smithi190.stdout:9/252: mkdir d12/d2d/d1d/d23/d2a/d3f/d55/d5c 0 2024-03-26T18:54:24.856 INFO:tasks.workunit.client.1.smithi190.stdout:9/253: readlink d12/l30 0 2024-03-26T18:54:24.858 INFO:tasks.workunit.client.1.smithi190.stdout:9/254: symlink d12/d2d/d1d/d23/d2b/l5d 0 2024-03-26T18:54:24.858 INFO:tasks.workunit.client.1.smithi190.stdout:9/255: creat d12/d2d/f5e x:0 0 0 2024-03-26T18:54:24.858 INFO:tasks.workunit.client.1.smithi190.stdout:9/256: dread - d12/d2d/d1d/d23/d2b/f3d zero size 2024-03-26T18:54:24.858 INFO:tasks.workunit.client.1.smithi190.stdout:9/257: chown d12/d2d/d1d/d23/l45 911 1 2024-03-26T18:54:24.858 INFO:tasks.workunit.client.1.smithi190.stdout:9/258: chown d12/d2d/d1d/f3a 64961 1 2024-03-26T18:54:24.867 INFO:tasks.workunit.client.1.smithi190.stdout:9/259: dread d12/f18 [0,4194304] 0 2024-03-26T18:54:24.867 INFO:tasks.workunit.client.1.smithi190.stdout:9/260: fdatasync d12/d2d/d1d/d23/f33 0 2024-03-26T18:54:24.867 INFO:tasks.workunit.client.1.smithi190.stdout:9/261: chown d12/d2d/d1d/f3b 9496 1 2024-03-26T18:54:24.868 INFO:tasks.workunit.client.1.smithi190.stdout:9/262: dread fb [0,4194304] 0 2024-03-26T18:54:24.868 INFO:tasks.workunit.client.1.smithi190.stdout:9/263: write f6 [4924966,79427] 0 2024-03-26T18:54:24.871 INFO:tasks.workunit.client.1.smithi190.stdout:9/264: write d12/f14 [1273641,101939] 0 2024-03-26T18:54:24.873 INFO:tasks.workunit.client.1.smithi190.stdout:9/265: fdatasync d12/d2d/f46 0 2024-03-26T18:54:24.873 INFO:tasks.workunit.client.1.smithi190.stdout:9/266: write d12/d2d/f3e [469043,29966] 0 2024-03-26T18:54:24.887 INFO:tasks.workunit.client.1.smithi190.stdout:9/267: dwrite d12/d2d/d1d/d23/f59 [0,4194304] 0 2024-03-26T18:54:24.887 INFO:tasks.workunit.client.1.smithi190.stdout:9/268: chown d12/f13 165 1 2024-03-26T18:54:24.888 INFO:tasks.workunit.client.1.smithi190.stdout:9/269: creat d12/d2d/d1d/d23/f5f x:0 0 0 2024-03-26T18:54:24.889 INFO:tasks.workunit.client.1.smithi190.stdout:9/270: mknod d12/d2d/d1d/d54/c60 0 2024-03-26T18:54:24.889 INFO:tasks.workunit.client.1.smithi190.stdout:9/271: chown d12/d2d/d1d/d23/l45 14 1 2024-03-26T18:54:24.890 INFO:tasks.workunit.client.1.smithi190.stdout:9/272: mknod d12/d2d/d1d/d23/d2a/c61 0 2024-03-26T18:54:24.894 INFO:tasks.workunit.client.1.smithi190.stdout:9/273: truncate d12/d2d/d1d/d23/f33 3188260 0 2024-03-26T18:54:24.900 INFO:tasks.workunit.client.1.smithi190.stdout:9/274: truncate d12/d2d/d1d/d23/f33 1066831 0 2024-03-26T18:54:24.900 INFO:tasks.workunit.client.1.smithi190.stdout:9/275: chown d12/c32 3622547 1 2024-03-26T18:54:24.905 INFO:tasks.workunit.client.1.smithi190.stdout:9/276: dread d12/f16 [0,4194304] 0 2024-03-26T18:54:24.906 INFO:tasks.workunit.client.1.smithi190.stdout:9/277: link d12/d2d/f53 d12/d2d/d1d/d23/d2a/d3f/d55/f62 0 2024-03-26T18:54:24.907 INFO:tasks.workunit.client.1.smithi190.stdout:9/278: mknod d12/d2d/c63 0 2024-03-26T18:54:24.916 INFO:tasks.workunit.client.1.smithi190.stdout:9/279: write d12/d2d/d1d/d23/d2a/f42 [3223303,61226] 0 2024-03-26T18:54:24.916 INFO:tasks.workunit.client.1.smithi190.stdout:9/280: fdatasync d12/d2d/d1d/d23/d2a/d3f/d55/f56 0 2024-03-26T18:54:24.917 INFO:tasks.workunit.client.1.smithi190.stdout:9/281: truncate d12/d2d/f49 3370800 0 2024-03-26T18:54:24.940 INFO:tasks.workunit.client.1.smithi190.stdout:9/282: dwrite d12/d2d/f39 [0,4194304] 0 2024-03-26T18:54:24.940 INFO:tasks.workunit.client.1.smithi190.stdout:9/283: fsync d12/d2d/f31 0 2024-03-26T18:54:24.945 INFO:tasks.workunit.client.1.smithi190.stdout:9/284: dread d12/f16 [0,4194304] 0 2024-03-26T18:54:24.945 INFO:tasks.workunit.client.1.smithi190.stdout:9/285: read d12/d2d/f3e [251271,128146] 0 2024-03-26T18:54:24.946 INFO:tasks.workunit.client.1.smithi190.stdout:9/286: mknod d12/d2d/d1d/d23/d2b/c64 0 2024-03-26T18:54:24.962 INFO:tasks.workunit.client.1.smithi190.stdout:9/287: dwrite d12/f13 [0,4194304] 0 2024-03-26T18:54:24.963 INFO:tasks.workunit.client.1.smithi190.stdout:9/288: symlink d12/d2d/d1d/d23/l65 0 2024-03-26T18:54:24.978 INFO:tasks.workunit.client.1.smithi190.stdout:9/289: dwrite d12/d2d/d1d/d23/d2a/d3f/d55/f62 [0,4194304] 0 2024-03-26T18:54:25.134 INFO:tasks.workunit.client.1.smithi190.stdout:9/290: dwrite ff [0,4194304] 0 2024-03-26T18:54:25.135 INFO:tasks.workunit.client.1.smithi190.stdout:9/291: symlink d12/d2d/d1d/d23/d2a/d3f/d55/l66 0 2024-03-26T18:54:25.136 INFO:tasks.workunit.client.1.smithi190.stdout:9/292: rmdir d12/d2d/d1d/d23/d2a/d3f/d55/d5c 0 2024-03-26T18:54:25.136 INFO:tasks.workunit.client.1.smithi190.stdout:9/293: write d12/d2d/d1d/d23/d2a/f42 [2272428,35951] 0 2024-03-26T18:54:25.143 INFO:tasks.workunit.client.1.smithi190.stdout:9/294: dread f5 [0,4194304] 0 2024-03-26T18:54:25.143 INFO:tasks.workunit.client.1.smithi190.stdout:9/295: chown fb 258720 1 2024-03-26T18:54:25.144 INFO:tasks.workunit.client.1.smithi190.stdout:9/296: truncate d12/d2d/f31 1190123 0 2024-03-26T18:54:25.145 INFO:tasks.workunit.client.1.smithi190.stdout:9/297: mknod d12/d2d/c67 0 2024-03-26T18:54:25.146 INFO:tasks.workunit.client.1.smithi190.stdout:9/298: rename d12/d2d/d1d/d23/d2b/f3d to d12/d2d/d1d/d23/d2b/f68 0 2024-03-26T18:54:25.158 INFO:tasks.workunit.client.1.smithi190.stdout:9/299: write f11 [3812600,27224] 0 2024-03-26T18:54:25.159 INFO:tasks.workunit.client.1.smithi190.stdout:9/300: rename d12/d2d/d1d/d23/f51 to d12/d2d/d1d/d23/f69 0 2024-03-26T18:54:25.167 INFO:tasks.workunit.client.1.smithi190.stdout:9/301: read d12/f13 [403602,102447] 0 2024-03-26T18:54:25.167 INFO:tasks.workunit.client.1.smithi190.stdout:9/302: dread - d12/d2d/f27 zero size 2024-03-26T18:54:25.168 INFO:tasks.workunit.client.1.smithi190.stdout:9/303: mkdir d12/d2d/d1d/d23/d6a 0 2024-03-26T18:54:25.170 INFO:tasks.workunit.client.1.smithi190.stdout:9/304: rename d12/d2d/d1d/d23/d2a/c61 to d12/d2d/d1d/d23/d2a/d3f/c6b 0 2024-03-26T18:54:25.170 INFO:tasks.workunit.client.1.smithi190.stdout:9/305: getdents d12/d2d/d1d/d54 0 2024-03-26T18:54:25.188 INFO:tasks.workunit.client.1.smithi190.stdout:9/306: dwrite d12/f28 [0,4194304] 0 2024-03-26T18:54:25.188 INFO:tasks.workunit.client.1.smithi190.stdout:9/307: truncate d12/f16 4989585 0 2024-03-26T18:54:25.188 INFO:tasks.workunit.client.1.smithi190.stdout:9/308: truncate d12/d2d/f5e 326854 0 2024-03-26T18:54:25.197 INFO:tasks.workunit.client.1.smithi190.stdout:9/309: dread d12/d2d/d1d/d23/d2a/d3f/f4d [0,4194304] 0 2024-03-26T18:54:25.199 INFO:tasks.workunit.client.1.smithi190.stdout:9/310: creat d12/d2d/d1d/d23/d2a/f6c x:0 0 0 2024-03-26T18:54:25.199 INFO:tasks.workunit.client.1.smithi190.stdout:9/311: mkdir d12/d6d 0 2024-03-26T18:54:25.201 INFO:tasks.workunit.client.1.smithi190.stdout:9/312: creat d12/d2d/d1d/d23/d2b/f6e x:0 0 0 2024-03-26T18:54:25.201 INFO:tasks.workunit.client.1.smithi190.stdout:9/313: write d12/d2d/f5e [546600,67527] 0 2024-03-26T18:54:25.201 INFO:tasks.workunit.client.1.smithi190.stdout:9/314: write d12/d2d/d1d/f3a [37805,104496] 0 2024-03-26T18:54:25.218 INFO:tasks.workunit.client.1.smithi190.stdout:9/315: dwrite d12/f14 [0,4194304] 0 2024-03-26T18:54:25.219 INFO:tasks.workunit.client.1.smithi190.stdout:9/316: creat d12/d2d/d1d/f6f x:0 0 0 2024-03-26T18:54:25.227 INFO:tasks.workunit.client.1.smithi190.stdout:9/317: dread d12/d2d/d1d/d23/f59 [0,4194304] 0 2024-03-26T18:54:25.228 INFO:tasks.workunit.client.1.smithi190.stdout:9/318: symlink d12/d2d/d1d/d23/d2a/d3f/d55/l70 0 2024-03-26T18:54:25.228 INFO:tasks.workunit.client.1.smithi190.stdout:9/319: write d12/f1c [5156281,92523] 0 2024-03-26T18:54:25.237 INFO:tasks.workunit.client.1.smithi190.stdout:9/320: read d12/f16 [3054178,66495] 0 2024-03-26T18:54:25.238 INFO:tasks.workunit.client.1.smithi190.stdout:9/321: mkdir d12/d2d/d1d/d23/d2a/d71 0 2024-03-26T18:54:25.238 INFO:tasks.workunit.client.1.smithi190.stdout:9/322: read f6 [3068330,57163] 0 2024-03-26T18:54:25.240 INFO:tasks.workunit.client.1.smithi190.stdout:9/323: rename d12/d2d/d1d/d23/d2a/c41 to d12/d6d/c72 0 2024-03-26T18:54:25.240 INFO:tasks.workunit.client.1.smithi190.stdout:9/324: creat d12/d2d/d1d/d23/d2a/f73 x:0 0 0 2024-03-26T18:54:25.241 INFO:tasks.workunit.client.1.smithi190.stdout:9/325: symlink d12/d2d/d1d/d23/d2b/l74 0 2024-03-26T18:54:25.241 INFO:tasks.workunit.client.1.smithi190.stdout:9/326: dread - d12/d2d/d1d/f6f zero size 2024-03-26T18:54:25.241 INFO:tasks.workunit.client.1.smithi190.stdout:9/327: dread - d12/d2d/d1d/f6f zero size 2024-03-26T18:54:25.241 INFO:tasks.workunit.client.1.smithi190.stdout:9/328: chown d12/d2d/d1d/d23/d2a/d3f/f50 0 1 2024-03-26T18:54:25.249 INFO:tasks.workunit.client.1.smithi190.stdout:9/329: write d12/f28 [1938762,24842] 0 2024-03-26T18:54:25.249 INFO:tasks.workunit.client.1.smithi190.stdout:9/330: write d12/d2d/f22 [3524011,533] 0 2024-03-26T18:54:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:25 smithi190 ceph-mon[35043]: Upgrade: Updating mgr.smithi190.zrixos 2024-03-26T18:54:25.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:25 smithi190 ceph-mon[35043]: Deploying daemon mgr.smithi190.zrixos on smithi190 2024-03-26T18:54:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:25 smithi190 ceph-mon[35043]: pgmap v240: 65 pgs: 65 active+clean; 372 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 5.2 MiB/s rd, 16 MiB/s wr, 439 op/s 2024-03-26T18:54:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:25 smithi190 ceph-mon[35043]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:25 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3848143567' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:54:25.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:25 smithi102 ceph-mon[29002]: Upgrade: Updating mgr.smithi190.zrixos 2024-03-26T18:54:25.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:25 smithi102 ceph-mon[29002]: Deploying daemon mgr.smithi190.zrixos on smithi190 2024-03-26T18:54:25.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:25 smithi102 ceph-mon[29002]: pgmap v240: 65 pgs: 65 active+clean; 372 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 5.2 MiB/s rd, 16 MiB/s wr, 439 op/s 2024-03-26T18:54:25.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:25 smithi102 ceph-mon[29002]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:25.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:25 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3848143567' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:54:25.892 INFO:tasks.workunit.client.1.smithi190.stdout:3/167: sync 2024-03-26T18:54:25.892 INFO:tasks.workunit.client.1.smithi190.stdout:4/206: sync 2024-03-26T18:54:25.892 INFO:tasks.workunit.client.1.smithi190.stdout:1/140: sync 2024-03-26T18:54:25.892 INFO:tasks.workunit.client.1.smithi190.stdout:7/132: sync 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:0/285: sync 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:7/133: fsync d3/d6/d9/d14/f2a 0 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:0/286: chown d0/dc/l31 13282 1 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:0/287: write d0/dd/d26/f34 [1016067,29121] 0 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:3/168: write d1/f11 [2782635,56389] 0 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:3/169: fsync d1/f15 0 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:6/178: sync 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:5/82: sync 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:8/169: sync 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:9/331: sync 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:9/332: write d12/f13 [1887720,11780] 0 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:9/333: truncate d12/d2d/d1d/d23/f5f 965107 0 2024-03-26T18:54:25.893 INFO:tasks.workunit.client.1.smithi190.stdout:2/152: sync 2024-03-26T18:54:25.894 INFO:tasks.workunit.client.1.smithi190.stdout:4/207: unlink d7/fc 0 2024-03-26T18:54:25.894 INFO:tasks.workunit.client.1.smithi190.stdout:4/208: truncate d7/d1d/f26 825235 0 2024-03-26T18:54:25.895 INFO:tasks.workunit.client.1.smithi190.stdout:5/83: dread d2/f14 [0,4194304] 0 2024-03-26T18:54:25.895 INFO:tasks.workunit.client.1.smithi190.stdout:5/84: chown d2/f6 926683942 1 2024-03-26T18:54:25.895 INFO:tasks.workunit.client.1.smithi190.stdout:5/85: readlink d2/lf 0 2024-03-26T18:54:25.895 INFO:tasks.workunit.client.1.smithi190.stdout:1/141: mkdir d3/da/dd/d2f 0 2024-03-26T18:54:25.898 INFO:tasks.workunit.client.1.smithi190.stdout:7/134: creat d3/d6/f30 x:0 0 0 2024-03-26T18:54:25.898 INFO:tasks.workunit.client.1.smithi190.stdout:0/288: rename d0/dd/d24/c42 to d0/dd/d26/c6a 0 2024-03-26T18:54:25.901 INFO:tasks.workunit.client.1.smithi190.stdout:8/170: truncate d9/dd/f34 3261351 0 2024-03-26T18:54:25.901 INFO:tasks.workunit.client.1.smithi190.stdout:8/171: truncate d9/f1a 569683 0 2024-03-26T18:54:25.903 INFO:tasks.workunit.client.1.smithi190.stdout:2/153: mknod d6/c30 0 2024-03-26T18:54:25.904 INFO:tasks.workunit.client.1.smithi190.stdout:5/86: truncate d2/f12 726853 0 2024-03-26T18:54:25.907 INFO:tasks.workunit.client.1.smithi190.stdout:0/289: mkdir d0/dd/d24/d4b/d5e/d6b 0 2024-03-26T18:54:25.907 INFO:tasks.workunit.client.1.smithi190.stdout:0/290: fsync d0/f6 0 2024-03-26T18:54:25.908 INFO:tasks.workunit.client.1.smithi190.stdout:2/154: mkdir d6/df/d2e/d31 0 2024-03-26T18:54:25.910 INFO:tasks.workunit.client.1.smithi190.stdout:5/87: dread d2/f3 [0,4194304] 0 2024-03-26T18:54:25.910 INFO:tasks.workunit.client.1.smithi190.stdout:2/155: rmdir d6/d9 39 2024-03-26T18:54:25.912 INFO:tasks.workunit.client.1.smithi190.stdout:2/156: mknod d6/df/c32 0 2024-03-26T18:54:25.913 INFO:tasks.workunit.client.1.smithi190.stdout:7/135: write d3/d1b/f27 [798463,118430] 0 2024-03-26T18:54:25.913 INFO:tasks.workunit.client.1.smithi190.stdout:2/157: creat d6/df/d2e/d31/f33 x:0 0 0 2024-03-26T18:54:25.914 INFO:tasks.workunit.client.1.smithi190.stdout:2/158: truncate d6/d9/d10/f2c 808010 0 2024-03-26T18:54:25.920 INFO:tasks.workunit.client.1.smithi190.stdout:9/334: dwrite d12/d2d/d1d/d23/d2a/f73 [0,4194304] 0 2024-03-26T18:54:25.926 INFO:tasks.workunit.client.1.smithi190.stdout:9/335: read d12/d2d/d1d/d23/f59 [3671086,89405] 0 2024-03-26T18:54:25.930 INFO:tasks.workunit.client.1.smithi190.stdout:9/336: creat d12/d2d/d1d/d23/d2a/d71/f75 x:0 0 0 2024-03-26T18:54:25.933 INFO:tasks.workunit.client.1.smithi190.stdout:9/337: rename d12/d2d/d1d/d23/l4f to d12/d2d/d1d/d23/d2a/d3f/l76 0 2024-03-26T18:54:25.933 INFO:tasks.workunit.client.1.smithi190.stdout:9/338: readlink l4 0 2024-03-26T18:54:25.933 INFO:tasks.workunit.client.1.smithi190.stdout:9/339: fdatasync ff 0 2024-03-26T18:54:25.933 INFO:tasks.workunit.client.1.smithi190.stdout:9/340: write d12/d2d/f49 [4288653,28946] 0 2024-03-26T18:54:25.934 INFO:tasks.workunit.client.1.smithi190.stdout:9/341: write d12/d2d/d1d/d23/f59 [4498108,116121] 0 2024-03-26T18:54:25.935 INFO:tasks.workunit.client.1.smithi190.stdout:9/342: unlink d12/d2d/c63 0 2024-03-26T18:54:25.935 INFO:tasks.workunit.client.1.smithi190.stdout:9/343: read - d12/d2d/d1d/d23/d2b/f6e zero size 2024-03-26T18:54:25.937 INFO:tasks.workunit.client.1.smithi190.stdout:9/344: rename d12/d2d/d1d/d23/d2a/d3f/d55/f56 to d12/d2d/d1d/f77 0 2024-03-26T18:54:25.937 INFO:tasks.workunit.client.1.smithi190.stdout:9/345: write d12/d2d/d1d/d23/d2a/f6c [47316,124408] 0 2024-03-26T18:54:25.938 INFO:tasks.workunit.client.1.smithi190.stdout:9/346: symlink d12/d6d/l78 0 2024-03-26T18:54:25.938 INFO:tasks.workunit.client.1.smithi190.stdout:9/347: write d12/d2d/d1d/d23/d2a/d71/f75 [310152,31841] 0 2024-03-26T18:54:25.938 INFO:tasks.workunit.client.1.smithi190.stdout:9/348: readlink d12/d2d/d1d/d54/d58/l4e 0 2024-03-26T18:54:25.938 INFO:tasks.workunit.client.1.smithi190.stdout:9/349: fsync fb 0 2024-03-26T18:54:25.953 INFO:tasks.workunit.client.1.smithi190.stdout:1/142: dwrite d3/f29 [0,4194304] 0 2024-03-26T18:54:25.954 INFO:tasks.workunit.client.1.smithi190.stdout:1/143: stat d3/d9/db/cc 0 2024-03-26T18:54:25.956 INFO:tasks.workunit.client.1.smithi190.stdout:1/144: creat d3/da/d2e/f30 x:0 0 0 2024-03-26T18:54:25.956 INFO:tasks.workunit.client.1.smithi190.stdout:1/145: mknod d3/da/dd/d2f/c31 0 2024-03-26T18:54:25.956 INFO:tasks.workunit.client.1.smithi190.stdout:1/146: creat d3/d9/db/f32 x:0 0 0 2024-03-26T18:54:25.957 INFO:tasks.workunit.client.1.smithi190.stdout:9/350: write d12/f1c [3790514,120321] 0 2024-03-26T18:54:25.957 INFO:tasks.workunit.client.1.smithi190.stdout:9/351: chown d12/f1c 1252959489 1 2024-03-26T18:54:25.965 INFO:tasks.workunit.client.1.smithi190.stdout:7/136: dwrite d3/f2b [0,4194304] 0 2024-03-26T18:54:25.965 INFO:tasks.workunit.client.1.smithi190.stdout:8/172: dwrite d9/dd/d23/f27 [0,4194304] 0 2024-03-26T18:54:25.965 INFO:tasks.workunit.client.1.smithi190.stdout:8/173: chown d9/dd/d15/f2c 345585 1 2024-03-26T18:54:25.965 INFO:tasks.workunit.client.1.smithi190.stdout:8/174: stat d9/dd/d23/c2e 0 2024-03-26T18:54:25.965 INFO:tasks.workunit.client.1.smithi190.stdout:8/175: stat d9/l21 0 2024-03-26T18:54:25.965 INFO:tasks.workunit.client.1.smithi190.stdout:8/176: fsync d9/dd/f34 0 2024-03-26T18:54:25.965 INFO:tasks.workunit.client.1.smithi190.stdout:8/177: rename d9/dd to d9/dd/d38 22 2024-03-26T18:54:25.965 INFO:tasks.workunit.client.1.smithi190.stdout:5/88: dwrite d2/fa [0,4194304] 0 2024-03-26T18:54:25.966 INFO:tasks.workunit.client.1.smithi190.stdout:2/159: dwrite d6/f7 [0,4194304] 0 2024-03-26T18:54:25.966 INFO:tasks.workunit.client.1.smithi190.stdout:2/160: read - d6/d9/d1a/f26 zero size 2024-03-26T18:54:25.966 INFO:tasks.workunit.client.1.smithi190.stdout:2/161: fsync d6/d9/d10/f2d 0 2024-03-26T18:54:25.966 INFO:tasks.workunit.client.1.smithi190.stdout:0/291: dwrite d0/f13 [0,4194304] 0 2024-03-26T18:54:25.966 INFO:tasks.workunit.client.1.smithi190.stdout:0/292: write d0/dd/d20/d1c/d46/f5d [551635,38680] 0 2024-03-26T18:54:25.966 INFO:tasks.workunit.client.1.smithi190.stdout:6/179: dwrite f9 [0,4194304] 0 2024-03-26T18:54:25.967 INFO:tasks.workunit.client.1.smithi190.stdout:7/137: mknod d3/d1b/c31 0 2024-03-26T18:54:25.972 INFO:tasks.workunit.client.1.smithi190.stdout:8/178: mkdir d9/d39 0 2024-03-26T18:54:25.972 INFO:tasks.workunit.client.1.smithi190.stdout:9/352: rmdir d12/d2d/d1d/d23/d2a 39 2024-03-26T18:54:25.974 INFO:tasks.workunit.client.1.smithi190.stdout:7/138: dread d3/fe [0,4194304] 0 2024-03-26T18:54:25.975 INFO:tasks.workunit.client.1.smithi190.stdout:2/162: creat d6/d9/d10/f34 x:0 0 0 2024-03-26T18:54:25.977 INFO:tasks.workunit.client.1.smithi190.stdout:1/147: dwrite f2 [0,4194304] 0 2024-03-26T18:54:25.977 INFO:tasks.workunit.client.1.smithi190.stdout:0/293: creat d0/dd/d20/d1c/f6c x:0 0 0 2024-03-26T18:54:25.977 INFO:tasks.workunit.client.1.smithi190.stdout:9/353: dread f5 [4194304,4194304] 0 2024-03-26T18:54:25.977 INFO:tasks.workunit.client.1.smithi190.stdout:0/294: dread - d0/dd/d20/f68 zero size 2024-03-26T18:54:25.979 INFO:tasks.workunit.client.1.smithi190.stdout:8/179: mkdir d9/dd/d23/d36/d37/d3a 0 2024-03-26T18:54:25.980 INFO:tasks.workunit.client.1.smithi190.stdout:5/89: symlink d2/l18 0 2024-03-26T18:54:25.982 INFO:tasks.workunit.client.1.smithi190.stdout:9/354: unlink f5 0 2024-03-26T18:54:25.982 INFO:tasks.workunit.client.1.smithi190.stdout:9/355: chown d12/d2d/d1d/d23/d2a/d3f/f4d 702 1 2024-03-26T18:54:25.982 INFO:tasks.workunit.client.1.smithi190.stdout:0/295: mknod d0/dd/d24/d4b/d5e/d6b/c6d 0 2024-03-26T18:54:25.983 INFO:tasks.workunit.client.1.smithi190.stdout:0/296: creat d0/dd/d20/d1c/f6e x:0 0 0 2024-03-26T18:54:25.983 INFO:tasks.workunit.client.1.smithi190.stdout:6/180: getdents d17/d28/d2b 0 2024-03-26T18:54:25.985 INFO:tasks.workunit.client.1.smithi190.stdout:5/90: symlink d2/d17/l19 0 2024-03-26T18:54:25.985 INFO:tasks.workunit.client.1.smithi190.stdout:7/139: dread d3/d1b/f27 [0,4194304] 0 2024-03-26T18:54:25.985 INFO:tasks.workunit.client.1.smithi190.stdout:7/140: fsync d3/d1b/f25 0 2024-03-26T18:54:25.986 INFO:tasks.workunit.client.1.smithi190.stdout:9/356: mknod d12/d2d/d1d/d23/d6a/c79 0 2024-03-26T18:54:25.989 INFO:tasks.workunit.client.1.smithi190.stdout:2/163: write d6/fe [5013420,106107] 0 2024-03-26T18:54:25.989 INFO:tasks.workunit.client.1.smithi190.stdout:2/164: truncate d6/d9/d10/f34 68832 0 2024-03-26T18:54:25.990 INFO:tasks.workunit.client.1.smithi190.stdout:8/180: dread d9/fa [0,4194304] 0 2024-03-26T18:54:25.990 INFO:tasks.workunit.client.1.smithi190.stdout:8/181: fsync d9/dd/d15/f1c 0 2024-03-26T18:54:25.990 INFO:tasks.workunit.client.1.smithi190.stdout:8/182: fsync d9/dd/d15/f2f 0 2024-03-26T18:54:25.990 INFO:tasks.workunit.client.1.smithi190.stdout:8/183: write d9/f14 [860022,28480] 0 2024-03-26T18:54:25.990 INFO:tasks.workunit.client.1.smithi190.stdout:6/181: rename d17/f27 to d17/d28/d2c/f41 0 2024-03-26T18:54:25.991 INFO:tasks.workunit.client.1.smithi190.stdout:6/182: creat d17/d28/d2c/f42 x:0 0 0 2024-03-26T18:54:25.991 INFO:tasks.workunit.client.1.smithi190.stdout:5/91: rmdir d2/d17 39 2024-03-26T18:54:25.991 INFO:tasks.workunit.client.1.smithi190.stdout:1/148: write d3/f29 [1737105,85932] 0 2024-03-26T18:54:25.991 INFO:tasks.workunit.client.1.smithi190.stdout:1/149: chown d3/d9 18 1 2024-03-26T18:54:25.991 INFO:tasks.workunit.client.1.smithi190.stdout:1/150: chown d3/d9/f20 7820490 1 2024-03-26T18:54:25.991 INFO:tasks.workunit.client.1.smithi190.stdout:0/297: getdents d0/dd/d20/d1c/d46 0 2024-03-26T18:54:25.994 INFO:tasks.workunit.client.1.smithi190.stdout:7/141: write d3/f2b [3676757,65082] 0 2024-03-26T18:54:25.994 INFO:tasks.workunit.client.1.smithi190.stdout:7/142: dread - d3/d1b/f2c zero size 2024-03-26T18:54:25.994 INFO:tasks.workunit.client.1.smithi190.stdout:8/184: mknod d9/dd/d15/c3b 0 2024-03-26T18:54:25.994 INFO:tasks.workunit.client.1.smithi190.stdout:8/185: fdatasync d9/d16/f1e 0 2024-03-26T18:54:25.994 INFO:tasks.workunit.client.1.smithi190.stdout:8/186: write d9/f33 [236381,49910] 0 2024-03-26T18:54:25.994 INFO:tasks.workunit.client.1.smithi190.stdout:8/187: dread - d9/dd/d15/d32/f35 zero size 2024-03-26T18:54:25.995 INFO:tasks.workunit.client.1.smithi190.stdout:2/165: dread d6/d9/d10/f2d [0,4194304] 0 2024-03-26T18:54:25.997 INFO:tasks.workunit.client.1.smithi190.stdout:8/188: dread d9/f1a [0,4194304] 0 2024-03-26T18:54:25.998 INFO:tasks.workunit.client.1.smithi190.stdout:6/183: unlink d17/d28/d2c/c39 0 2024-03-26T18:54:26.000 INFO:tasks.workunit.client.1.smithi190.stdout:1/151: truncate d3/da/dd/f12 1523893 0 2024-03-26T18:54:26.002 INFO:tasks.workunit.client.1.smithi190.stdout:0/298: mknod d0/dd/c6f 0 2024-03-26T18:54:26.002 INFO:tasks.workunit.client.1.smithi190.stdout:0/299: fsync d0/dd/d24/f2a 0 2024-03-26T18:54:26.003 INFO:tasks.workunit.client.1.smithi190.stdout:8/189: write d9/fa [1159540,101135] 0 2024-03-26T18:54:26.004 INFO:tasks.workunit.client.1.smithi190.stdout:7/143: rename d3/lf to d3/d6/d9/d1f/l32 0 2024-03-26T18:54:26.007 INFO:tasks.workunit.client.1.smithi190.stdout:5/92: symlink d2/l1a 0 2024-03-26T18:54:26.008 INFO:tasks.workunit.client.1.smithi190.stdout:7/144: truncate d3/d6/d9/d1f/f20 7713339 0 2024-03-26T18:54:26.008 INFO:tasks.workunit.client.1.smithi190.stdout:7/145: write d3/d6/f30 [614707,59928] 0 2024-03-26T18:54:26.008 INFO:tasks.workunit.client.1.smithi190.stdout:7/146: write d3/f2f [188413,68444] 0 2024-03-26T18:54:26.010 INFO:tasks.workunit.client.1.smithi190.stdout:6/184: rename l15 to d17/d28/d38/l43 0 2024-03-26T18:54:26.010 INFO:tasks.workunit.client.1.smithi190.stdout:2/166: write d6/f7 [639449,124312] 0 2024-03-26T18:54:26.010 INFO:tasks.workunit.client.1.smithi190.stdout:1/152: dread d3/da/f24 [0,4194304] 0 2024-03-26T18:54:26.011 INFO:tasks.workunit.client.1.smithi190.stdout:9/357: dwrite d12/d2d/d1d/f3a [0,4194304] 0 2024-03-26T18:54:26.014 INFO:tasks.workunit.client.1.smithi190.stdout:5/93: mkdir d2/d17/d1b 0 2024-03-26T18:54:26.014 INFO:tasks.workunit.client.1.smithi190.stdout:5/94: chown d2/f6 3 1 2024-03-26T18:54:26.014 INFO:tasks.workunit.client.1.smithi190.stdout:0/300: dread d0/dd/d20/d1c/d46/f5d [0,4194304] 0 2024-03-26T18:54:26.015 INFO:tasks.workunit.client.1.smithi190.stdout:2/167: write d6/d9/f1d [1280001,61015] 0 2024-03-26T18:54:26.015 INFO:tasks.workunit.client.1.smithi190.stdout:2/168: read d6/d9/f1d [1805867,22970] 0 2024-03-26T18:54:26.015 INFO:tasks.workunit.client.1.smithi190.stdout:7/147: creat d3/d6/d9/d14/f33 x:0 0 0 2024-03-26T18:54:26.016 INFO:tasks.workunit.client.1.smithi190.stdout:7/148: chown d3/d6/d9 5282 1 2024-03-26T18:54:26.017 INFO:tasks.workunit.client.1.smithi190.stdout:6/185: rmdir d17/d28/d3e 0 2024-03-26T18:54:26.019 INFO:tasks.workunit.client.1.smithi190.stdout:1/153: mkdir d3/da/dd/d2f/d33 0 2024-03-26T18:54:26.024 INFO:tasks.workunit.client.1.smithi190.stdout:7/149: dread d3/d6/d9/f24 [0,4194304] 0 2024-03-26T18:54:26.024 INFO:tasks.workunit.client.1.smithi190.stdout:7/150: chown d3/f16 39089 1 2024-03-26T18:54:26.039 INFO:tasks.workunit.client.1.smithi190.stdout:5/95: link d2/l1a d2/d17/l1c 0 2024-03-26T18:54:26.039 INFO:tasks.workunit.client.1.smithi190.stdout:0/301: symlink d0/dd/d26/l70 0 2024-03-26T18:54:26.040 INFO:tasks.workunit.client.1.smithi190.stdout:0/302: readlink d0/dd/d20/d1c/l23 0 2024-03-26T18:54:26.040 INFO:tasks.workunit.client.1.smithi190.stdout:0/303: write d0/dd/f45 [1053193,38292] 0 2024-03-26T18:54:26.045 INFO:tasks.workunit.client.1.smithi190.stdout:1/154: dwrite d3/d9/db/f19 [0,4194304] 0 2024-03-26T18:54:26.045 INFO:tasks.workunit.client.1.smithi190.stdout:9/358: dwrite d12/d2d/f31 [0,4194304] 0 2024-03-26T18:54:26.048 INFO:tasks.workunit.client.1.smithi190.stdout:2/169: rename d6/d21/f29 to d6/d9/d10/f35 0 2024-03-26T18:54:26.048 INFO:tasks.workunit.client.1.smithi190.stdout:2/170: creat d6/f36 x:0 0 0 2024-03-26T18:54:26.048 INFO:tasks.workunit.client.1.smithi190.stdout:2/171: stat d6/d9/d10/l1f 0 2024-03-26T18:54:26.050 INFO:tasks.workunit.client.1.smithi190.stdout:6/186: unlink d17/d28/d38/l43 0 2024-03-26T18:54:26.050 INFO:tasks.workunit.client.1.smithi190.stdout:7/151: stat d3/d6/d9/d1f/l32 0 2024-03-26T18:54:26.051 INFO:tasks.workunit.client.1.smithi190.stdout:7/152: dread d3/d6/fd [4194304,4194304] 0 2024-03-26T18:54:26.051 INFO:tasks.workunit.client.1.smithi190.stdout:7/153: write d3/fe [3386093,97404] 0 2024-03-26T18:54:26.053 INFO:tasks.workunit.client.1.smithi190.stdout:1/155: mkdir d3/da/dd/d2f/d34 0 2024-03-26T18:54:26.053 INFO:tasks.workunit.client.1.smithi190.stdout:5/96: dwrite d2/f3 [0,4194304] 0 2024-03-26T18:54:26.053 INFO:tasks.workunit.client.1.smithi190.stdout:5/97: chown d2/d17/l1c 628633931 1 2024-03-26T18:54:26.053 INFO:tasks.workunit.client.1.smithi190.stdout:5/98: read - d2/f16 zero size 2024-03-26T18:54:26.056 INFO:tasks.workunit.client.1.smithi190.stdout:9/359: mknod d12/d2d/d1d/d23/d2b/c7a 0 2024-03-26T18:54:26.057 INFO:tasks.workunit.client.1.smithi190.stdout:2/172: link d6/df/d1c/l25 d6/df/d2e/d31/l37 0 2024-03-26T18:54:26.058 INFO:tasks.workunit.client.1.smithi190.stdout:6/187: mknod d17/d28/d2c/c44 0 2024-03-26T18:54:26.059 INFO:tasks.workunit.client.1.smithi190.stdout:0/304: getdents d0/dd 0 2024-03-26T18:54:26.061 INFO:tasks.workunit.client.1.smithi190.stdout:1/156: creat d3/da/dd/d2f/d34/f35 x:0 0 0 2024-03-26T18:54:26.061 INFO:tasks.workunit.client.1.smithi190.stdout:1/157: chown d3/d9/f1c 91285589 1 2024-03-26T18:54:26.061 INFO:tasks.workunit.client.1.smithi190.stdout:2/173: dread d6/d9/d10/f2d [0,4194304] 0 2024-03-26T18:54:26.061 INFO:tasks.workunit.client.1.smithi190.stdout:2/174: chown d6/df/d2e 212669934 1 2024-03-26T18:54:26.062 INFO:tasks.workunit.client.1.smithi190.stdout:9/360: symlink d12/d2d/d1d/d23/d2a/d71/l7b 0 2024-03-26T18:54:26.063 INFO:tasks.workunit.client.1.smithi190.stdout:9/361: readlink d12/d2d/l26 0 2024-03-26T18:54:26.064 INFO:tasks.workunit.client.1.smithi190.stdout:1/158: dread d3/da/f24 [0,4194304] 0 2024-03-26T18:54:26.065 INFO:tasks.workunit.client.1.smithi190.stdout:6/188: creat d17/d28/d38/f45 x:0 0 0 2024-03-26T18:54:26.067 INFO:tasks.workunit.client.1.smithi190.stdout:0/305: mknod d0/dd/d24/d4b/c71 0 2024-03-26T18:54:26.069 INFO:tasks.workunit.client.1.smithi190.stdout:1/159: rename d3/d9/l25 to d3/l36 0 2024-03-26T18:54:26.071 INFO:tasks.workunit.client.1.smithi190.stdout:0/306: unlink d0/dd/d20/c40 0 2024-03-26T18:54:26.072 INFO:tasks.workunit.client.1.smithi190.stdout:9/362: dread d12/f28 [0,4194304] 0 2024-03-26T18:54:26.073 INFO:tasks.workunit.client.1.smithi190.stdout:0/307: creat d0/dd/d20/d1c/d46/f72 x:0 0 0 2024-03-26T18:54:26.074 INFO:tasks.workunit.client.1.smithi190.stdout:7/154: dwrite d3/f2f [0,4194304] 0 2024-03-26T18:54:26.078 INFO:tasks.workunit.client.1.smithi190.stdout:5/99: dwrite d2/f6 [0,4194304] 0 2024-03-26T18:54:26.083 INFO:tasks.workunit.client.1.smithi190.stdout:2/175: rmdir d6/df/d2e/d31 39 2024-03-26T18:54:26.083 INFO:tasks.workunit.client.1.smithi190.stdout:2/176: stat d6/df/d2e/d31/f33 0 2024-03-26T18:54:26.086 INFO:tasks.workunit.client.1.smithi190.stdout:5/100: link d2/le d2/d17/l1d 0 2024-03-26T18:54:26.086 INFO:tasks.workunit.client.1.smithi190.stdout:5/101: write d2/f3 [5039120,7767] 0 2024-03-26T18:54:26.086 INFO:tasks.workunit.client.1.smithi190.stdout:5/102: chown d2/d17/d1b 717482138 1 2024-03-26T18:54:26.087 INFO:tasks.workunit.client.1.smithi190.stdout:7/155: write d3/f19 [2862000,51423] 0 2024-03-26T18:54:26.088 INFO:tasks.workunit.client.1.smithi190.stdout:2/177: creat d6/d9/f38 x:0 0 0 2024-03-26T18:54:26.089 INFO:tasks.workunit.client.1.smithi190.stdout:1/160: dwrite d3/d9/db/f19 [0,4194304] 0 2024-03-26T18:54:26.089 INFO:tasks.workunit.client.1.smithi190.stdout:1/161: chown d3/da/f24 3396983 1 2024-03-26T18:54:26.091 INFO:tasks.workunit.client.1.smithi190.stdout:5/103: mkdir d2/d17/d1b/d1e 0 2024-03-26T18:54:26.092 INFO:tasks.workunit.client.1.smithi190.stdout:2/178: truncate f5 3128342 0 2024-03-26T18:54:26.094 INFO:tasks.workunit.client.1.smithi190.stdout:1/162: creat d3/da/f37 x:0 0 0 2024-03-26T18:54:26.094 INFO:tasks.workunit.client.1.smithi190.stdout:1/163: readlink d3/l8 0 2024-03-26T18:54:26.094 INFO:tasks.workunit.client.1.smithi190.stdout:1/164: readlink d3/l36 0 2024-03-26T18:54:26.095 INFO:tasks.workunit.client.1.smithi190.stdout:5/104: symlink d2/d17/d1b/l1f 0 2024-03-26T18:54:26.096 INFO:tasks.workunit.client.1.smithi190.stdout:2/179: link d6/c18 d6/d9/d1a/c39 0 2024-03-26T18:54:26.097 INFO:tasks.workunit.client.1.smithi190.stdout:1/165: symlink d3/da/dd/d2f/d33/l38 0 2024-03-26T18:54:26.099 INFO:tasks.workunit.client.1.smithi190.stdout:5/105: symlink d2/d17/d1b/l20 0 2024-03-26T18:54:26.100 INFO:tasks.workunit.client.1.smithi190.stdout:2/180: unlink d6/df/f16 0 2024-03-26T18:54:26.100 INFO:tasks.workunit.client.1.smithi190.stdout:2/181: readlink d6/d9/d10/l15 0 2024-03-26T18:54:26.101 INFO:tasks.workunit.client.1.smithi190.stdout:5/106: truncate d2/fa 3037741 0 2024-03-26T18:54:26.112 INFO:tasks.workunit.client.1.smithi190.stdout:9/363: dwrite f8 [0,4194304] 0 2024-03-26T18:54:26.112 INFO:tasks.workunit.client.1.smithi190.stdout:9/364: chown f8 79888 1 2024-03-26T18:54:26.112 INFO:tasks.workunit.client.1.smithi190.stdout:9/365: readlink d12/d6d/l78 0 2024-03-26T18:54:26.114 INFO:tasks.workunit.client.1.smithi190.stdout:9/366: creat d12/d2d/d1d/d54/d58/f7c x:0 0 0 2024-03-26T18:54:26.114 INFO:tasks.workunit.client.1.smithi190.stdout:0/308: dwrite d0/dd/f22 [4194304,4194304] 0 2024-03-26T18:54:26.116 INFO:tasks.workunit.client.1.smithi190.stdout:9/367: rename d12/d2d/d1d/d23/d2a/d3f/f50 to d12/d6d/f7d 0 2024-03-26T18:54:26.117 INFO:tasks.workunit.client.1.smithi190.stdout:0/309: symlink d0/dd/d26/l73 0 2024-03-26T18:54:26.118 INFO:tasks.workunit.client.1.smithi190.stdout:9/368: creat d12/d2d/d1d/d23/d6a/f7e x:0 0 0 2024-03-26T18:54:26.120 INFO:tasks.workunit.client.1.smithi190.stdout:0/310: rmdir d0/dd/d26 39 2024-03-26T18:54:26.120 INFO:tasks.workunit.client.1.smithi190.stdout:0/311: write d0/dd/d24/f49 [339506,75974] 0 2024-03-26T18:54:26.121 INFO:tasks.workunit.client.1.smithi190.stdout:0/312: creat d0/dd/d20/d1c/d4e/f74 x:0 0 0 2024-03-26T18:54:26.121 INFO:tasks.workunit.client.1.smithi190.stdout:0/313: fsync d0/dd/d20/d1c/f28 0 2024-03-26T18:54:26.122 INFO:tasks.workunit.client.1.smithi190.stdout:0/314: mkdir d0/dd/d24/d4b/d5e/d75 0 2024-03-26T18:54:26.126 INFO:tasks.workunit.client.1.smithi190.stdout:7/156: dwrite d3/fe [0,4194304] 0 2024-03-26T18:54:26.131 INFO:tasks.workunit.client.1.smithi190.stdout:1/166: dwrite f2 [4194304,4194304] 0 2024-03-26T18:54:26.134 INFO:tasks.workunit.client.1.smithi190.stdout:7/157: dread d3/d6/fd [0,4194304] 0 2024-03-26T18:54:26.134 INFO:tasks.workunit.client.1.smithi190.stdout:1/167: unlink d3/f27 0 2024-03-26T18:54:26.135 INFO:tasks.workunit.client.1.smithi190.stdout:2/182: dwrite d6/d9/f1d [0,4194304] 0 2024-03-26T18:54:26.135 INFO:tasks.workunit.client.1.smithi190.stdout:2/183: write d6/d9/d1a/f26 [570483,82679] 0 2024-03-26T18:54:26.135 INFO:tasks.workunit.client.1.smithi190.stdout:2/184: chown d6/df/f22 796285156 1 2024-03-26T18:54:26.136 INFO:tasks.workunit.client.1.smithi190.stdout:9/369: write d12/d2d/f53 [2142487,114303] 0 2024-03-26T18:54:26.136 INFO:tasks.workunit.client.1.smithi190.stdout:5/107: dwrite d2/f16 [0,4194304] 0 2024-03-26T18:54:26.136 INFO:tasks.workunit.client.1.smithi190.stdout:1/168: rename d3/l8 to d3/d9/db/l39 0 2024-03-26T18:54:26.137 INFO:tasks.workunit.client.1.smithi190.stdout:1/169: chown d3/d9/db/f2c 86674026 1 2024-03-26T18:54:26.137 INFO:tasks.workunit.client.1.smithi190.stdout:1/170: truncate d3/da/dd/d2f/d34/f35 1043916 0 2024-03-26T18:54:26.137 INFO:tasks.workunit.client.1.smithi190.stdout:1/171: stat d3/da/d2e 0 2024-03-26T18:54:26.137 INFO:tasks.workunit.client.1.smithi190.stdout:2/185: mkdir d6/d9/d3a 0 2024-03-26T18:54:26.138 INFO:tasks.workunit.client.1.smithi190.stdout:2/186: dread d6/d9/d10/f2c [0,4194304] 0 2024-03-26T18:54:26.142 INFO:tasks.workunit.client.1.smithi190.stdout:9/370: creat d12/d2d/d1d/d23/d2a/d3f/f7f x:0 0 0 2024-03-26T18:54:26.144 INFO:tasks.workunit.client.1.smithi190.stdout:0/315: dwrite d0/dd/d20/f30 [4194304,4194304] 0 2024-03-26T18:54:26.144 INFO:tasks.workunit.client.1.smithi190.stdout:5/108: creat d2/d17/f21 x:0 0 0 2024-03-26T18:54:26.144 INFO:tasks.workunit.client.1.smithi190.stdout:0/316: chown d0/dd/d24/d4b/c71 2580 1 2024-03-26T18:54:26.144 INFO:tasks.workunit.client.1.smithi190.stdout:0/317: chown d0/dd/d20 151128 1 2024-03-26T18:54:26.147 INFO:tasks.workunit.client.1.smithi190.stdout:1/172: mkdir d3/da/d3a 0 2024-03-26T18:54:26.153 INFO:tasks.workunit.client.1.smithi190.stdout:5/109: mkdir d2/d17/d1b/d22 0 2024-03-26T18:54:26.154 INFO:tasks.workunit.client.1.smithi190.stdout:1/173: mknod d3/da/c3b 0 2024-03-26T18:54:26.161 INFO:tasks.workunit.client.1.smithi190.stdout:7/158: dwrite d3/f16 [0,4194304] 0 2024-03-26T18:54:26.162 INFO:tasks.workunit.client.1.smithi190.stdout:7/159: link d3/d1b/f27 d3/d6/d9/d1f/f34 0 2024-03-26T18:54:26.165 INFO:tasks.workunit.client.1.smithi190.stdout:5/110: write d2/f3 [1918430,21647] 0 2024-03-26T18:54:26.165 INFO:tasks.workunit.client.1.smithi190.stdout:7/160: link d3/d6/d9/d1f/f34 d3/f35 0 2024-03-26T18:54:26.166 INFO:tasks.workunit.client.1.smithi190.stdout:5/111: link d2/c13 d2/d17/d1b/c23 0 2024-03-26T18:54:26.173 INFO:tasks.workunit.client.1.smithi190.stdout:9/371: dwrite d12/d2d/d1d/f3a [0,4194304] 0 2024-03-26T18:54:26.173 INFO:tasks.workunit.client.1.smithi190.stdout:2/187: dwrite d6/d9/d10/f2c [0,4194304] 0 2024-03-26T18:54:26.173 INFO:tasks.workunit.client.1.smithi190.stdout:2/188: write d6/df/d2e/d31/f33 [593829,16619] 0 2024-03-26T18:54:26.173 INFO:tasks.workunit.client.1.smithi190.stdout:2/189: chown d6/d9/d1a/f26 11 1 2024-03-26T18:54:26.173 INFO:tasks.workunit.client.1.smithi190.stdout:2/190: readlink d6/df/d1c/l25 0 2024-03-26T18:54:26.174 INFO:tasks.workunit.client.1.smithi190.stdout:1/174: dwrite d3/d9/f16 [4194304,4194304] 0 2024-03-26T18:54:26.174 INFO:tasks.workunit.client.1.smithi190.stdout:1/175: write d3/d9/f15 [1001234,73218] 0 2024-03-26T18:54:26.175 INFO:tasks.workunit.client.1.smithi190.stdout:9/372: mkdir d12/d80 0 2024-03-26T18:54:26.175 INFO:tasks.workunit.client.1.smithi190.stdout:9/373: write d12/f38 [1152056,20746] 0 2024-03-26T18:54:26.175 INFO:tasks.workunit.client.1.smithi190.stdout:9/374: creat d12/d2d/d1d/d23/d2a/d3f/f81 x:0 0 0 2024-03-26T18:54:26.178 INFO:tasks.workunit.client.1.smithi190.stdout:2/191: creat d6/d9/d3a/f3b x:0 0 0 2024-03-26T18:54:26.179 INFO:tasks.workunit.client.1.smithi190.stdout:2/192: truncate d6/d9/d1a/f26 946182 0 2024-03-26T18:54:26.179 INFO:tasks.workunit.client.1.smithi190.stdout:2/193: write d6/df/f27 [291012,113297] 0 2024-03-26T18:54:26.179 INFO:tasks.workunit.client.1.smithi190.stdout:2/194: write d6/d9/d1a/f26 [1378140,8638] 0 2024-03-26T18:54:26.179 INFO:tasks.workunit.client.1.smithi190.stdout:2/195: write d6/df/d2e/d31/f33 [1411844,29372] 0 2024-03-26T18:54:26.179 INFO:tasks.workunit.client.1.smithi190.stdout:2/196: creat d6/f3c x:0 0 0 2024-03-26T18:54:26.179 INFO:tasks.workunit.client.1.smithi190.stdout:1/176: creat d3/d9/f3c x:0 0 0 2024-03-26T18:54:26.180 INFO:tasks.workunit.client.1.smithi190.stdout:1/177: chown d3/f29 47 1 2024-03-26T18:54:26.181 INFO:tasks.workunit.client.1.smithi190.stdout:5/112: dwrite d2/f14 [0,4194304] 0 2024-03-26T18:54:26.183 INFO:tasks.workunit.client.1.smithi190.stdout:9/375: creat d12/d2d/d1d/d23/d2a/d3f/d55/f82 x:0 0 0 2024-03-26T18:54:26.190 INFO:tasks.workunit.client.1.smithi190.stdout:1/178: link d3/d9/c11 d3/d9/c3d 0 2024-03-26T18:54:26.190 INFO:tasks.workunit.client.1.smithi190.stdout:1/179: fdatasync d3/da/f1f 0 2024-03-26T18:54:26.191 INFO:tasks.workunit.client.1.smithi190.stdout:7/161: truncate d3/f16 3061040 0 2024-03-26T18:54:26.192 INFO:tasks.workunit.client.1.smithi190.stdout:5/113: mkdir d2/d17/d1b/d22/d24 0 2024-03-26T18:54:26.194 INFO:tasks.workunit.client.1.smithi190.stdout:1/180: mkdir d3/da/dd/d2f/d34/d3e 0 2024-03-26T18:54:26.194 INFO:tasks.workunit.client.1.smithi190.stdout:1/181: write d3/da/dd/f12 [1592319,27324] 0 2024-03-26T18:54:26.195 INFO:tasks.workunit.client.1.smithi190.stdout:7/162: unlink d3/d6/l1a 0 2024-03-26T18:54:26.195 INFO:tasks.workunit.client.1.smithi190.stdout:7/163: read d3/f19 [3421247,113480] 0 2024-03-26T18:54:26.195 INFO:tasks.workunit.client.1.smithi190.stdout:7/164: write d3/d6/d9/d1f/f22 [1669005,25677] 0 2024-03-26T18:54:26.196 INFO:tasks.workunit.client.1.smithi190.stdout:5/114: symlink d2/d17/d1b/d22/l25 0 2024-03-26T18:54:26.197 INFO:tasks.workunit.client.1.smithi190.stdout:2/197: rename d6/df to d6/d9/d3a/d3d 0 2024-03-26T18:54:26.198 INFO:tasks.workunit.client.1.smithi190.stdout:7/165: mkdir d3/d6/d9/d14/d36 0 2024-03-26T18:54:26.200 INFO:tasks.workunit.client.1.smithi190.stdout:7/166: symlink d3/d1b/l37 0 2024-03-26T18:54:26.202 INFO:tasks.workunit.client.1.smithi190.stdout:1/182: dread f2 [4194304,4194304] 0 2024-03-26T18:54:26.202 INFO:tasks.workunit.client.1.smithi190.stdout:1/183: chown d3/d9/db 1264390 1 2024-03-26T18:54:26.202 INFO:tasks.workunit.client.1.smithi190.stdout:1/184: read - d3/da/f37 zero size 2024-03-26T18:54:26.202 INFO:tasks.workunit.client.1.smithi190.stdout:7/167: symlink d3/d6/d9/d14/l38 0 2024-03-26T18:54:26.203 INFO:tasks.workunit.client.1.smithi190.stdout:7/168: creat d3/d6/f39 x:0 0 0 2024-03-26T18:54:26.203 INFO:tasks.workunit.client.1.smithi190.stdout:7/169: truncate d3/fe 4856392 0 2024-03-26T18:54:26.205 INFO:tasks.workunit.client.1.smithi190.stdout:9/376: rename d12/d2d/d1d/d54/d58 to d12/d2d/d1d/d23/d83 0 2024-03-26T18:54:26.205 INFO:tasks.workunit.client.1.smithi190.stdout:9/377: chown d12/d6d/l78 378088377 1 2024-03-26T18:54:26.207 INFO:tasks.workunit.client.1.smithi190.stdout:7/170: symlink d3/d6/l3a 0 2024-03-26T18:54:26.207 INFO:tasks.workunit.client.1.smithi190.stdout:7/171: creat d3/d1b/f3b x:0 0 0 2024-03-26T18:54:26.208 INFO:tasks.workunit.client.1.smithi190.stdout:9/378: link d12/d2d/d1d/d23/d2a/c37 d12/d2d/d1d/d23/d2a/c84 0 2024-03-26T18:54:26.208 INFO:tasks.workunit.client.1.smithi190.stdout:5/115: read d2/f6 [3244712,76834] 0 2024-03-26T18:54:26.209 INFO:tasks.workunit.client.1.smithi190.stdout:2/198: rename d6/d9/d3a/d3d/c32 to d6/c3e 0 2024-03-26T18:54:26.209 INFO:tasks.workunit.client.1.smithi190.stdout:2/199: chown d6/d9/d10/f13 173879 1 2024-03-26T18:54:26.209 INFO:tasks.workunit.client.1.smithi190.stdout:2/200: read f5 [1174057,34570] 0 2024-03-26T18:54:26.210 INFO:tasks.workunit.client.1.smithi190.stdout:2/201: creat d6/f3f x:0 0 0 2024-03-26T18:54:26.211 INFO:tasks.workunit.client.1.smithi190.stdout:7/172: dread d3/d1b/f27 [0,4194304] 0 2024-03-26T18:54:26.214 INFO:tasks.workunit.client.1.smithi190.stdout:9/379: link d12/d2d/d1d/d23/f5f d12/d2d/d1d/d23/d2a/d71/f85 0 2024-03-26T18:54:26.216 INFO:tasks.workunit.client.1.smithi190.stdout:5/116: creat d2/d17/d1b/f26 x:0 0 0 2024-03-26T18:54:26.218 INFO:tasks.workunit.client.1.smithi190.stdout:2/202: rename d6/d9/d10/f35 to d6/d9/d10/f40 0 2024-03-26T18:54:26.218 INFO:tasks.workunit.client.1.smithi190.stdout:2/203: readlink d6/d9/l1b 0 2024-03-26T18:54:26.218 INFO:tasks.workunit.client.1.smithi190.stdout:2/204: creat d6/d9/f41 x:0 0 0 2024-03-26T18:54:26.219 INFO:tasks.workunit.client.1.smithi190.stdout:2/205: chown d6/d9/d3a/d3d/d1c/l25 1814495 1 2024-03-26T18:54:26.219 INFO:tasks.workunit.client.1.smithi190.stdout:7/173: creat d3/d6/d9/d14/d36/f3c x:0 0 0 2024-03-26T18:54:26.220 INFO:tasks.workunit.client.1.smithi190.stdout:7/174: creat d3/d6/d9/d1f/f3d x:0 0 0 2024-03-26T18:54:26.220 INFO:tasks.workunit.client.1.smithi190.stdout:7/175: write d3/d6/d9/d14/f18 [2942251,101638] 0 2024-03-26T18:54:26.220 INFO:tasks.workunit.client.1.smithi190.stdout:9/380: mknod d12/d2d/d1d/d54/c86 0 2024-03-26T18:54:26.222 INFO:tasks.workunit.client.1.smithi190.stdout:5/117: creat d2/d17/d1b/d22/f27 x:0 0 0 2024-03-26T18:54:26.222 INFO:tasks.workunit.client.1.smithi190.stdout:5/118: write d2/d17/f21 [335922,81550] 0 2024-03-26T18:54:26.226 INFO:tasks.workunit.client.1.smithi190.stdout:2/206: rename d6/d9/d3a/d3d/d2e/d31/l37 to d6/d9/d1a/l42 0 2024-03-26T18:54:26.231 INFO:tasks.workunit.client.1.smithi190.stdout:9/381: symlink d12/d2d/d1d/l87 0 2024-03-26T18:54:26.235 INFO:tasks.workunit.client.1.smithi190.stdout:9/382: creat d12/d2d/d1d/d54/f88 x:0 0 0 2024-03-26T18:54:26.235 INFO:tasks.workunit.client.1.smithi190.stdout:9/383: stat d12/d2d/l21 0 2024-03-26T18:54:26.235 INFO:tasks.workunit.client.1.smithi190.stdout:9/384: symlink d12/d2d/l89 0 2024-03-26T18:54:26.243 INFO:tasks.workunit.client.1.smithi190.stdout:2/207: dwrite d6/d9/f38 [0,4194304] 0 2024-03-26T18:54:26.244 INFO:tasks.workunit.client.1.smithi190.stdout:7/176: dwrite d3/d6/f30 [0,4194304] 0 2024-03-26T18:54:26.244 INFO:tasks.workunit.client.1.smithi190.stdout:7/177: creat d3/d6/f3e x:0 0 0 2024-03-26T18:54:26.246 INFO:tasks.workunit.client.1.smithi190.stdout:9/385: dread d12/d2d/f31 [0,4194304] 0 2024-03-26T18:54:26.246 INFO:tasks.workunit.client.1.smithi190.stdout:9/386: creat d12/d2d/d1d/d54/f8a x:0 0 0 2024-03-26T18:54:26.246 INFO:tasks.workunit.client.1.smithi190.stdout:2/208: creat d6/d9/d1a/f43 x:0 0 0 2024-03-26T18:54:26.247 INFO:tasks.workunit.client.1.smithi190.stdout:2/209: dread d6/d9/d10/f34 [0,4194304] 0 2024-03-26T18:54:26.251 INFO:tasks.workunit.client.1.smithi190.stdout:9/387: dread d12/f13 [0,4194304] 0 2024-03-26T18:54:26.252 INFO:tasks.workunit.client.1.smithi190.stdout:7/178: symlink d3/d6/d9/d14/d36/l3f 0 2024-03-26T18:54:26.252 INFO:tasks.workunit.client.1.smithi190.stdout:7/179: truncate d3/d1b/f28 988813 0 2024-03-26T18:54:26.254 INFO:tasks.workunit.client.1.smithi190.stdout:9/388: creat d12/d2d/d1d/d23/d2a/d71/f8b x:0 0 0 2024-03-26T18:54:26.256 INFO:tasks.workunit.client.1.smithi190.stdout:7/180: link d3/d6/d9/d14/d36/f3c d3/d6/d9/f40 0 2024-03-26T18:54:26.258 INFO:tasks.workunit.client.1.smithi190.stdout:9/389: symlink d12/l8c 0 2024-03-26T18:54:26.258 INFO:tasks.workunit.client.1.smithi190.stdout:9/390: chown d12/d2d/f3e 454714 1 2024-03-26T18:54:26.261 INFO:tasks.workunit.client.1.smithi190.stdout:2/210: rmdir d6/d9/d3a 39 2024-03-26T18:54:26.264 INFO:tasks.workunit.client.1.smithi190.stdout:2/211: unlink d6/d9/d3a/d3d/d2e/d31/f33 0 2024-03-26T18:54:26.264 INFO:tasks.workunit.client.1.smithi190.stdout:9/391: dread d12/d2d/f31 [0,4194304] 0 2024-03-26T18:54:26.264 INFO:tasks.workunit.client.1.smithi190.stdout:9/392: creat d12/d2d/d1d/f8d x:0 0 0 2024-03-26T18:54:26.265 INFO:tasks.workunit.client.1.smithi190.stdout:7/181: rmdir d3/d6/d9/d1f 39 2024-03-26T18:54:26.271 INFO:tasks.workunit.client.1.smithi190.stdout:3/170: sync 2024-03-26T18:54:26.271 INFO:tasks.workunit.client.1.smithi190.stdout:7/182: dread d3/d6/d9/f24 [0,4194304] 0 2024-03-26T18:54:26.273 INFO:tasks.workunit.client.1.smithi190.stdout:7/183: mkdir d3/d6/d9/d1f/d41 0 2024-03-26T18:54:26.275 INFO:tasks.workunit.client.1.smithi190.stdout:7/184: symlink d3/d6/d9/d14/d36/l42 0 2024-03-26T18:54:26.279 INFO:tasks.workunit.client.1.smithi190.stdout:3/171: dread d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:26.281 INFO:tasks.workunit.client.1.smithi190.stdout:3/172: link d1/l23 d1/d3/df/d18/d1b/l37 0 2024-03-26T18:54:26.281 INFO:tasks.workunit.client.1.smithi190.stdout:3/173: write d1/de/f1d [4264442,66456] 0 2024-03-26T18:54:26.281 INFO:tasks.workunit.client.1.smithi190.stdout:3/174: stat d1/d3/l17 0 2024-03-26T18:54:26.286 INFO:tasks.workunit.client.1.smithi190.stdout:2/212: dwrite d6/d9/d10/f13 [4194304,4194304] 0 2024-03-26T18:54:26.286 INFO:tasks.workunit.client.1.smithi190.stdout:9/393: dwrite d12/d2d/d1d/d23/d2a/d71/f75 [0,4194304] 0 2024-03-26T18:54:26.288 INFO:tasks.workunit.client.1.smithi190.stdout:2/213: mkdir d6/d9/d3a/d3d/d1c/d44 0 2024-03-26T18:54:26.290 INFO:tasks.workunit.client.1.smithi190.stdout:2/214: mkdir d6/d9/d45 0 2024-03-26T18:54:26.290 INFO:tasks.workunit.client.1.smithi190.stdout:7/185: write d3/f2f [4043627,64131] 0 2024-03-26T18:54:26.291 INFO:tasks.workunit.client.1.smithi190.stdout:7/186: fsync d3/d6/fd 0 2024-03-26T18:54:26.291 INFO:tasks.workunit.client.1.smithi190.stdout:7/187: creat d3/f43 x:0 0 0 2024-03-26T18:54:26.292 INFO:tasks.workunit.client.1.smithi190.stdout:2/215: mknod d6/d9/d3a/c46 0 2024-03-26T18:54:26.297 INFO:tasks.workunit.client.1.smithi190.stdout:2/216: mknod d6/d9/d3a/d3d/d1c/c47 0 2024-03-26T18:54:26.300 INFO:tasks.workunit.client.1.smithi190.stdout:7/188: write d3/fe [889412,41475] 0 2024-03-26T18:54:26.301 INFO:tasks.workunit.client.1.smithi190.stdout:7/189: link d3/d6/d9/f40 d3/d6/d9/d1f/d41/f44 0 2024-03-26T18:54:26.302 INFO:tasks.workunit.client.1.smithi190.stdout:7/190: write d3/f35 [5197074,73833] 0 2024-03-26T18:54:26.303 INFO:tasks.workunit.client.1.smithi190.stdout:7/191: rmdir d3/d6/d9/d14 39 2024-03-26T18:54:26.303 INFO:tasks.workunit.client.1.smithi190.stdout:7/192: write d3/d6/d9/d1f/f3d [363201,44260] 0 2024-03-26T18:54:26.305 INFO:tasks.workunit.client.1.smithi190.stdout:7/193: link d3/d6/f30 d3/d6/d9/f45 0 2024-03-26T18:54:26.305 INFO:tasks.workunit.client.1.smithi190.stdout:9/394: dwrite d12/d2d/f46 [0,4194304] 0 2024-03-26T18:54:26.307 INFO:tasks.workunit.client.1.smithi190.stdout:9/395: mkdir d12/d2d/d1d/d23/d8e 0 2024-03-26T18:54:26.307 INFO:tasks.workunit.client.1.smithi190.stdout:9/396: readlink d12/l30 0 2024-03-26T18:54:26.307 INFO:tasks.workunit.client.1.smithi190.stdout:9/397: creat d12/d2d/d1d/d23/f8f x:0 0 0 2024-03-26T18:54:26.310 INFO:tasks.workunit.client.1.smithi190.stdout:9/398: link d12/d2d/f27 d12/d2d/d1d/d54/f90 0 2024-03-26T18:54:26.312 INFO:tasks.workunit.client.1.smithi190.stdout:9/399: mkdir d12/d2d/d1d/d23/d83/d91 0 2024-03-26T18:54:26.313 INFO:tasks.workunit.client.1.smithi190.stdout:9/400: mkdir d12/d2d/d1d/d54/d92 0 2024-03-26T18:54:26.314 INFO:tasks.workunit.client.1.smithi190.stdout:9/401: mknod d12/d2d/d1d/d54/d92/c93 0 2024-03-26T18:54:26.314 INFO:tasks.workunit.client.1.smithi190.stdout:9/402: chown d12/d2d/d1d/d23/d2a/d3f/d55/f62 2255073 1 2024-03-26T18:54:26.315 INFO:tasks.workunit.client.1.smithi190.stdout:9/403: symlink d12/d2d/l94 0 2024-03-26T18:54:26.316 INFO:tasks.workunit.client.1.smithi190.stdout:9/404: getdents d12/d2d/d1d/d23/d6a 0 2024-03-26T18:54:26.316 INFO:tasks.workunit.client.1.smithi190.stdout:9/405: write d12/d2d/d1d/f3b [698800,7764] 0 2024-03-26T18:54:26.318 INFO:tasks.workunit.client.1.smithi190.stdout:9/406: rename d12/f15 to d12/d2d/d1d/d23/d2b/f95 0 2024-03-26T18:54:26.318 INFO:tasks.workunit.client.1.smithi190.stdout:9/407: chown d12/d2d/d1d/d54/c86 420210 1 2024-03-26T18:54:26.319 INFO:tasks.workunit.client.1.smithi190.stdout:9/408: creat d12/d2d/d1d/d23/d83/f96 x:0 0 0 2024-03-26T18:54:26.319 INFO:tasks.workunit.client.1.smithi190.stdout:9/409: stat d12/d80 0 2024-03-26T18:54:26.323 INFO:tasks.workunit.client.1.smithi190.stdout:9/410: dread d12/d2d/f3e [0,4194304] 0 2024-03-26T18:54:26.323 INFO:tasks.workunit.client.1.smithi190.stdout:7/194: dwrite d3/d6/f30 [0,4194304] 0 2024-03-26T18:54:26.323 INFO:tasks.workunit.client.1.smithi190.stdout:7/195: creat d3/f46 x:0 0 0 2024-03-26T18:54:26.326 INFO:tasks.workunit.client.1.smithi190.stdout:7/196: truncate d3/d1b/f25 847581 0 2024-03-26T18:54:26.328 INFO:tasks.workunit.client.1.smithi190.stdout:7/197: symlink d3/d6/d9/d1f/d41/l47 0 2024-03-26T18:54:26.328 INFO:tasks.workunit.client.1.smithi190.stdout:7/198: write d3/f2f [71665,43178] 0 2024-03-26T18:54:26.333 INFO:tasks.workunit.client.1.smithi190.stdout:9/411: dread d12/d2d/d1d/d23/d2a/d71/f75 [0,4194304] 0 2024-03-26T18:54:26.333 INFO:tasks.workunit.client.1.smithi190.stdout:9/412: chown d12/d80 734636287 1 2024-03-26T18:54:26.333 INFO:tasks.workunit.client.1.smithi190.stdout:9/413: creat d12/d2d/f97 x:0 0 0 2024-03-26T18:54:26.336 INFO:tasks.workunit.client.1.smithi190.stdout:7/199: write d3/fc [2930000,63430] 0 2024-03-26T18:54:26.336 INFO:tasks.workunit.client.1.smithi190.stdout:9/414: rename l4 to d12/d2d/d1d/d23/d2a/d3f/l98 0 2024-03-26T18:54:26.336 INFO:tasks.workunit.client.1.smithi190.stdout:9/415: creat d12/d2d/f99 x:0 0 0 2024-03-26T18:54:26.336 INFO:tasks.workunit.client.1.smithi190.stdout:9/416: chown d12/d2d/d1d/d54/f8a 0 1 2024-03-26T18:54:26.336 INFO:tasks.workunit.client.1.smithi190.stdout:9/417: fdatasync d12/d2d/d1d/d23/d2a/f73 0 2024-03-26T18:54:26.338 INFO:tasks.workunit.client.1.smithi190.stdout:7/200: creat d3/d6/d9/d14/d36/f48 x:0 0 0 2024-03-26T18:54:26.338 INFO:tasks.workunit.client.1.smithi190.stdout:7/201: read d3/f2b [1599802,129404] 0 2024-03-26T18:54:26.339 INFO:tasks.workunit.client.1.smithi190.stdout:7/202: mkdir d3/d6/d9/d1f/d49 0 2024-03-26T18:54:26.346 INFO:tasks.workunit.client.1.smithi190.stdout:9/418: read d12/d2d/d1d/d23/d2a/d3f/f4d [1110217,111076] 0 2024-03-26T18:54:26.346 INFO:tasks.workunit.client.1.smithi190.stdout:9/419: chown d12/d2d/f5e 1058875 1 2024-03-26T18:54:26.347 INFO:tasks.workunit.client.1.smithi190.stdout:9/420: rename d12/d2d/d1d/d23/d2a/c37 to d12/d2d/d1d/d23/d83/c9a 0 2024-03-26T18:54:26.348 INFO:tasks.workunit.client.1.smithi190.stdout:9/421: mkdir d12/d2d/d1d/d23/d8e/d9b 0 2024-03-26T18:54:26.350 INFO:tasks.workunit.client.1.smithi190.stdout:9/422: fdatasync d12/d2d/d1d/f3b 0 2024-03-26T18:54:26.352 INFO:tasks.workunit.client.1.smithi190.stdout:9/423: link d12/d2d/d1d/d23/c48 d12/d2d/d1d/d23/d83/d91/c9c 0 2024-03-26T18:54:26.352 INFO:tasks.workunit.client.1.smithi190.stdout:9/424: chown d12/d2d/d1d/d23/d2a/d3f/f7f 9915670 1 2024-03-26T18:54:26.358 INFO:tasks.workunit.client.1.smithi190.stdout:7/203: dwrite d3/fc [0,4194304] 0 2024-03-26T18:54:26.358 INFO:tasks.workunit.client.1.smithi190.stdout:7/204: rename d3/d6 to d3/d6/d9/d1f/d41/d4a 22 2024-03-26T18:54:26.368 INFO:tasks.workunit.client.1.smithi190.stdout:9/425: dwrite d12/d2d/d1d/d23/d2a/d71/f85 [0,4194304] 0 2024-03-26T18:54:26.368 INFO:tasks.workunit.client.1.smithi190.stdout:9/426: truncate d12/d2d/d1d/d54/f8a 814620 0 2024-03-26T18:54:26.370 INFO:tasks.workunit.client.1.smithi190.stdout:9/427: symlink d12/d2d/d1d/d54/l9d 0 2024-03-26T18:54:26.372 INFO:tasks.workunit.client.1.smithi190.stdout:9/428: rename d12/d2d/d1d/d23/d2a/d3f/f7f to d12/d2d/f9e 0 2024-03-26T18:54:26.381 INFO:tasks.workunit.client.1.smithi190.stdout:7/205: dwrite d3/d1b/f28 [0,4194304] 0 2024-03-26T18:54:26.383 INFO:tasks.workunit.client.1.smithi190.stdout:7/206: symlink d3/d6/d9/l4b 0 2024-03-26T18:54:26.384 INFO:tasks.workunit.client.1.smithi190.stdout:7/207: rmdir d3/d6/d9/d1f 39 2024-03-26T18:54:26.385 INFO:tasks.workunit.client.1.smithi190.stdout:7/208: mkdir d3/d6/d9/d1f/d4c 0 2024-03-26T18:54:26.386 INFO:tasks.workunit.client.1.smithi190.stdout:7/209: symlink d3/d6/d9/d14/l4d 0 2024-03-26T18:54:26.387 INFO:tasks.workunit.client.1.smithi190.stdout:7/210: link d3/d6/d9/d14/f18 d3/d6/f4e 0 2024-03-26T18:54:26.387 INFO:tasks.workunit.client.1.smithi190.stdout:7/211: readlink d3/l23 0 2024-03-26T18:54:26.389 INFO:tasks.workunit.client.1.smithi190.stdout:9/429: dwrite d12/d2d/d1d/f8d [0,4194304] 0 2024-03-26T18:54:26.389 INFO:tasks.workunit.client.1.smithi190.stdout:9/430: readlink d12/d2d/d1d/l87 0 2024-03-26T18:54:26.397 INFO:tasks.workunit.client.1.smithi190.stdout:9/431: rename d12/f13 to d12/d2d/f9f 0 2024-03-26T18:54:26.397 INFO:tasks.workunit.client.1.smithi190.stdout:9/432: write d12/d2d/d1d/d23/d83/f96 [156109,96988] 0 2024-03-26T18:54:26.397 INFO:tasks.workunit.client.1.smithi190.stdout:9/433: chown d12/l4a 571 1 2024-03-26T18:54:26.397 INFO:tasks.workunit.client.1.smithi190.stdout:9/434: write d12/d2d/d1d/d23/d2b/f68 [806333,5748] 0 2024-03-26T18:54:26.399 INFO:tasks.workunit.client.1.smithi190.stdout:9/435: mkdir d12/d2d/d1d/d23/d2b/da0 0 2024-03-26T18:54:26.402 INFO:tasks.workunit.client.1.smithi190.stdout:4/209: sync 2024-03-26T18:54:26.402 INFO:tasks.workunit.client.1.smithi190.stdout:4/210: fsync d7/d8/d1c/d20/f3d 0 2024-03-26T18:54:26.402 INFO:tasks.workunit.client.1.smithi190.stdout:4/211: write d7/d8/f33 [923546,28647] 0 2024-03-26T18:54:26.403 INFO:tasks.workunit.client.1.smithi190.stdout:7/212: dwrite d3/fc [0,4194304] 0 2024-03-26T18:54:26.403 INFO:tasks.workunit.client.1.smithi190.stdout:4/212: write d7/d1d/f26 [301432,70726] 0 2024-03-26T18:54:26.403 INFO:tasks.workunit.client.1.smithi190.stdout:4/213: readlink d7/d8/d14/d23/l3b 0 2024-03-26T18:54:26.403 INFO:tasks.workunit.client.1.smithi190.stdout:4/214: write d7/d8/f3a [541674,68439] 0 2024-03-26T18:54:26.404 INFO:tasks.workunit.client.1.smithi190.stdout:4/215: symlink d7/d1d/l46 0 2024-03-26T18:54:26.405 INFO:tasks.workunit.client.1.smithi190.stdout:9/436: dread d12/d2d/f39 [0,4194304] 0 2024-03-26T18:54:26.405 INFO:tasks.workunit.client.1.smithi190.stdout:9/437: chown d12/d2d/d1d/l87 7758740 1 2024-03-26T18:54:26.405 INFO:tasks.workunit.client.1.smithi190.stdout:9/438: fdatasync d12/d2d/f97 0 2024-03-26T18:54:26.408 INFO:tasks.workunit.client.1.smithi190.stdout:4/216: creat d7/d8/d1c/d20/f47 x:0 0 0 2024-03-26T18:54:26.408 INFO:tasks.workunit.client.1.smithi190.stdout:4/217: creat d7/d8/d14/f48 x:0 0 0 2024-03-26T18:54:26.416 INFO:tasks.workunit.client.1.smithi190.stdout:4/218: write d7/d8/d1c/f1f [3445719,35828] 0 2024-03-26T18:54:26.416 INFO:tasks.workunit.client.1.smithi190.stdout:4/219: readlink l4 0 2024-03-26T18:54:26.417 INFO:tasks.workunit.client.1.smithi190.stdout:4/220: creat d7/f49 x:0 0 0 2024-03-26T18:54:26.418 INFO:tasks.workunit.client.1.smithi190.stdout:4/221: mkdir d7/d1d/d42/d4a 0 2024-03-26T18:54:26.429 INFO:tasks.workunit.client.1.smithi190.stdout:7/213: dwrite d3/d6/f30 [0,4194304] 0 2024-03-26T18:54:26.430 INFO:tasks.workunit.client.1.smithi190.stdout:9/439: dwrite d12/d2d/d1d/f3b [0,4194304] 0 2024-03-26T18:54:26.430 INFO:tasks.workunit.client.1.smithi190.stdout:9/440: dread - d12/d2d/d1d/d54/f90 zero size 2024-03-26T18:54:26.431 INFO:tasks.workunit.client.1.smithi190.stdout:7/214: rename d3/c21 to d3/d6/d9/d14/c4f 0 2024-03-26T18:54:26.438 INFO:tasks.workunit.client.1.smithi190.stdout:9/441: mkdir d12/d2d/d1d/da1 0 2024-03-26T18:54:26.438 INFO:tasks.workunit.client.1.smithi190.stdout:9/442: stat d12/d2d/d1d/d23/d6a/c79 0 2024-03-26T18:54:26.438 INFO:tasks.workunit.client.1.smithi190.stdout:4/222: dread d7/f13 [0,4194304] 0 2024-03-26T18:54:26.438 INFO:tasks.workunit.client.1.smithi190.stdout:4/223: write d7/d8/d14/f48 [964158,52740] 0 2024-03-26T18:54:26.438 INFO:tasks.workunit.client.1.smithi190.stdout:4/224: fdatasync d7/d8/f33 0 2024-03-26T18:54:26.438 INFO:tasks.workunit.client.1.smithi190.stdout:4/225: readlink d7/d8/d14/d23/d28/l37 0 2024-03-26T18:54:26.438 INFO:tasks.workunit.client.1.smithi190.stdout:4/226: truncate d7/d8/d1c/f44 1769 0 2024-03-26T18:54:26.438 INFO:tasks.workunit.client.1.smithi190.stdout:7/215: mkdir d3/d6/d9/d1f/d41/d50 0 2024-03-26T18:54:26.438 INFO:tasks.workunit.client.1.smithi190.stdout:4/227: mkdir d7/d8/d14/d4b 0 2024-03-26T18:54:26.439 INFO:tasks.workunit.client.1.smithi190.stdout:7/216: mknod d3/d6/d9/d1f/d41/c51 0 2024-03-26T18:54:26.439 INFO:tasks.workunit.client.1.smithi190.stdout:4/228: rmdir d7/d8 39 2024-03-26T18:54:26.439 INFO:tasks.workunit.client.1.smithi190.stdout:7/217: creat d3/d6/d9/d1f/d41/d50/f52 x:0 0 0 2024-03-26T18:54:26.441 INFO:tasks.workunit.client.1.smithi190.stdout:7/218: rename d3/d6/d9/d1f/d49 to d3/d6/d9/d1f/d41/d50/d53 0 2024-03-26T18:54:26.443 INFO:tasks.workunit.client.1.smithi190.stdout:7/219: link d3/d6/d9/d14/f33 d3/d6/d9/d1f/d4c/f54 0 2024-03-26T18:54:26.443 INFO:tasks.workunit.client.1.smithi190.stdout:7/220: fdatasync d3/d6/d9/f2e 0 2024-03-26T18:54:26.444 INFO:tasks.workunit.client.1.smithi190.stdout:4/229: dread d7/f13 [0,4194304] 0 2024-03-26T18:54:26.445 INFO:tasks.workunit.client.1.smithi190.stdout:4/230: rename d7/f17 to d7/d8/d14/d4b/f4c 0 2024-03-26T18:54:26.450 INFO:tasks.workunit.client.1.smithi190.stdout:9/443: dwrite d12/d2d/d1d/d23/f59 [0,4194304] 0 2024-03-26T18:54:26.450 INFO:tasks.workunit.client.1.smithi190.stdout:4/231: dread d7/d8/d1c/f44 [0,4194304] 0 2024-03-26T18:54:26.450 INFO:tasks.workunit.client.1.smithi190.stdout:7/221: dread d3/f29 [0,4194304] 0 2024-03-26T18:54:26.450 INFO:tasks.workunit.client.1.smithi190.stdout:4/232: write d7/d8/d1c/d20/f3f [918457,106005] 0 2024-03-26T18:54:26.450 INFO:tasks.workunit.client.1.smithi190.stdout:7/222: write d3/d6/d9/f40 [182114,50416] 0 2024-03-26T18:54:26.451 INFO:tasks.workunit.client.1.smithi190.stdout:9/444: mknod d12/d2d/d1d/d54/d92/ca2 0 2024-03-26T18:54:26.451 INFO:tasks.workunit.client.1.smithi190.stdout:9/445: fdatasync d12/d2d/d1d/d23/d6a/f7e 0 2024-03-26T18:54:26.451 INFO:tasks.workunit.client.1.smithi190.stdout:9/446: truncate d12/f4c 320878 0 2024-03-26T18:54:26.451 INFO:tasks.workunit.client.1.smithi190.stdout:4/233: symlink d7/d1d/d41/l4d 0 2024-03-26T18:54:26.452 INFO:tasks.workunit.client.1.smithi190.stdout:9/447: write f6 [2430859,121621] 0 2024-03-26T18:54:26.454 INFO:tasks.workunit.client.1.smithi190.stdout:4/234: creat d7/d8/d14/d23/d28/d38/f4e x:0 0 0 2024-03-26T18:54:26.454 INFO:tasks.workunit.client.1.smithi190.stdout:4/235: creat d7/f4f x:0 0 0 2024-03-26T18:54:26.454 INFO:tasks.workunit.client.1.smithi190.stdout:9/448: mknod d12/d2d/d1d/d23/d6a/ca3 0 2024-03-26T18:54:26.455 INFO:tasks.workunit.client.1.smithi190.stdout:9/449: write d12/d2d/d1d/d23/d2b/f68 [132027,15975] 0 2024-03-26T18:54:26.456 INFO:tasks.workunit.client.1.smithi190.stdout:4/236: creat d7/d1d/d42/f50 x:0 0 0 2024-03-26T18:54:26.457 INFO:tasks.workunit.client.1.smithi190.stdout:9/450: mknod d12/d2d/d1d/ca4 0 2024-03-26T18:54:26.458 INFO:tasks.workunit.client.1.smithi190.stdout:7/223: getdents d3/d6/d9/d1f/d4c 0 2024-03-26T18:54:26.458 INFO:tasks.workunit.client.1.smithi190.stdout:7/224: chown d3/d6/fd 299991424 1 2024-03-26T18:54:26.459 INFO:tasks.workunit.client.1.smithi190.stdout:4/237: symlink d7/d8/d14/d4b/l51 0 2024-03-26T18:54:26.459 INFO:tasks.workunit.client.1.smithi190.stdout:4/238: creat d7/d8/d14/f52 x:0 0 0 2024-03-26T18:54:26.461 INFO:tasks.workunit.client.1.smithi190.stdout:9/451: rename d12/d2d/d1d/c5b to d12/d2d/d1d/d23/d2b/ca5 0 2024-03-26T18:54:26.461 INFO:tasks.workunit.client.1.smithi190.stdout:9/452: stat d12/d2d/d1d/f8d 0 2024-03-26T18:54:26.461 INFO:tasks.workunit.client.1.smithi190.stdout:9/453: readlink d12/d2d/l26 0 2024-03-26T18:54:26.461 INFO:tasks.workunit.client.1.smithi190.stdout:9/454: creat d12/d2d/d1d/d23/d2a/d71/fa6 x:0 0 0 2024-03-26T18:54:26.461 INFO:tasks.workunit.client.1.smithi190.stdout:9/455: chown d12/d2d/d1d/d23/d8e 1694749354 1 2024-03-26T18:54:26.462 INFO:tasks.workunit.client.1.smithi190.stdout:7/225: creat d3/d6/d9/d1f/d41/f55 x:0 0 0 2024-03-26T18:54:26.465 INFO:tasks.workunit.client.1.smithi190.stdout:9/456: mkdir d12/d2d/d1d/d23/d2a/d71/da7 0 2024-03-26T18:54:26.465 INFO:tasks.workunit.client.1.smithi190.stdout:9/457: mknod d12/d2d/d1d/ca8 0 2024-03-26T18:54:26.467 INFO:tasks.workunit.client.1.smithi190.stdout:9/458: rename d12/f1c to d12/d2d/d1d/d23/d8e/fa9 0 2024-03-26T18:54:26.473 INFO:tasks.workunit.client.1.smithi190.stdout:4/239: getdents d7/d8/d14 0 2024-03-26T18:54:26.473 INFO:tasks.workunit.client.1.smithi190.stdout:7/226: dread d3/d1b/f28 [0,4194304] 0 2024-03-26T18:54:26.474 INFO:tasks.workunit.client.1.smithi190.stdout:7/227: chown d3/d6/d9/f2e 15159 1 2024-03-26T18:54:26.477 INFO:tasks.workunit.client.1.smithi190.stdout:4/240: symlink d7/d8/d1c/d20/d40/l53 0 2024-03-26T18:54:26.477 INFO:tasks.workunit.client.1.smithi190.stdout:4/241: read d7/d1d/f27 [266692,124850] 0 2024-03-26T18:54:26.477 INFO:tasks.workunit.client.1.smithi190.stdout:4/242: fsync d7/d8/f22 0 2024-03-26T18:54:26.491 INFO:tasks.workunit.client.1.smithi190.stdout:7/228: dwrite d3/d1b/f25 [0,4194304] 0 2024-03-26T18:54:26.576 INFO:tasks.workunit.client.1.smithi190.stdout:8/190: sync 2024-03-26T18:54:26.576 INFO:tasks.workunit.client.1.smithi190.stdout:8/191: fsync d9/d16/f1e 0 2024-03-26T18:54:26.585 INFO:tasks.workunit.client.1.smithi190.stdout:8/192: write d9/dd/f20 [1130646,27624] 0 2024-03-26T18:54:26.586 INFO:tasks.workunit.client.1.smithi190.stdout:8/193: mknod d9/dd/d15/d32/c3c 0 2024-03-26T18:54:26.586 INFO:tasks.workunit.client.1.smithi190.stdout:8/194: write d9/d16/f1e [466254,122905] 0 2024-03-26T18:54:26.588 INFO:tasks.workunit.client.1.smithi190.stdout:8/195: rmdir d9/dd 39 2024-03-26T18:54:26.588 INFO:tasks.workunit.client.1.smithi190.stdout:8/196: stat d9/dd/c1d 0 2024-03-26T18:54:26.589 INFO:tasks.workunit.client.1.smithi190.stdout:8/197: mknod d9/dd/d15/d32/c3d 0 2024-03-26T18:54:26.589 INFO:tasks.workunit.client.1.smithi190.stdout:8/198: chown l1 59504 1 2024-03-26T18:54:26.590 INFO:tasks.workunit.client.1.smithi190.stdout:8/199: creat d9/dd/d23/f3e x:0 0 0 2024-03-26T18:54:26.591 INFO:tasks.workunit.client.1.smithi190.stdout:8/200: link d9/dd/d15/d32/c3c d9/d39/c3f 0 2024-03-26T18:54:26.591 INFO:tasks.workunit.client.1.smithi190.stdout:8/201: dread - d9/dd/d15/f2c zero size 2024-03-26T18:54:26.591 INFO:tasks.workunit.client.1.smithi190.stdout:8/202: chown d9/f1a 773175 1 2024-03-26T18:54:26.591 INFO:tasks.workunit.client.1.smithi190.stdout:8/203: chown d9/dd/l29 3979260 1 2024-03-26T18:54:26.606 INFO:tasks.workunit.client.1.smithi190.stdout:8/204: dwrite d9/f14 [0,4194304] 0 2024-03-26T18:54:26.612 INFO:tasks.workunit.client.1.smithi190.stdout:8/205: truncate d9/dd/f20 1898238 0 2024-03-26T18:54:26.613 INFO:tasks.workunit.client.1.smithi190.stdout:8/206: link l4 d9/dd/l40 0 2024-03-26T18:54:26.615 INFO:tasks.workunit.client.1.smithi190.stdout:8/207: mkdir d9/dd/d15/d41 0 2024-03-26T18:54:26.615 INFO:tasks.workunit.client.1.smithi190.stdout:8/208: chown d9/d16/c19 29416218 1 2024-03-26T18:54:26.615 INFO:tasks.workunit.client.1.smithi190.stdout:8/209: write d9/f1a [380731,23134] 0 2024-03-26T18:54:26.625 INFO:tasks.workunit.client.1.smithi190.stdout:8/210: write d9/d16/f1f [557496,104719] 0 2024-03-26T18:54:26.650 INFO:tasks.workunit.client.1.smithi190.stdout:8/211: dwrite d9/f33 [0,4194304] 0 2024-03-26T18:54:26.653 INFO:tasks.workunit.client.1.smithi190.stdout:8/212: rename d9/l24 to d9/dd/d23/d36/l42 0 2024-03-26T18:54:26.654 INFO:tasks.workunit.client.1.smithi190.stdout:8/213: mkdir d9/dd/d23/d36/d37/d3a/d43 0 2024-03-26T18:54:26.662 INFO:tasks.workunit.client.1.smithi190.stdout:8/214: rmdir d9 39 2024-03-26T18:54:26.665 INFO:tasks.workunit.client.1.smithi190.stdout:8/215: rename d9/l21 to d9/dd/d23/d36/d37/d3a/l44 0 2024-03-26T18:54:26.665 INFO:tasks.workunit.client.1.smithi190.stdout:8/216: write d9/dd/d15/f2c [6297,20184] 0 2024-03-26T18:54:26.665 INFO:tasks.workunit.client.1.smithi190.stdout:8/217: readlink d9/dd/l28 0 2024-03-26T18:54:26.666 INFO:tasks.workunit.client.1.smithi190.stdout:8/218: creat d9/dd/d15/f45 x:0 0 0 2024-03-26T18:54:26.667 INFO:tasks.workunit.client.1.smithi190.stdout:8/219: fdatasync d9/f1a 0 2024-03-26T18:54:26.667 INFO:tasks.workunit.client.1.smithi190.stdout:8/220: mknod d9/dd/d15/d32/c46 0 2024-03-26T18:54:26.668 INFO:tasks.workunit.client.1.smithi190.stdout:8/221: rmdir d9/dd/d23/d36/d37/d3a 39 2024-03-26T18:54:26.678 INFO:tasks.workunit.client.1.smithi190.stdout:8/222: dread d9/f12 [8388608,4194304] 0 2024-03-26T18:54:26.680 INFO:tasks.workunit.client.1.smithi190.stdout:8/223: rename d9/dd/l40 to d9/dd/l47 0 2024-03-26T18:54:26.680 INFO:tasks.workunit.client.1.smithi190.stdout:8/224: chown d9/dd/l47 2963063 1 2024-03-26T18:54:26.695 INFO:tasks.workunit.client.1.smithi190.stdout:8/225: dwrite d9/dd/f2d [0,4194304] 0 2024-03-26T18:54:26.698 INFO:tasks.workunit.client.1.smithi190.stdout:8/226: mkdir d9/d48 0 2024-03-26T18:54:26.700 INFO:tasks.workunit.client.1.smithi190.stdout:8/227: mkdir d9/d48/d49 0 2024-03-26T18:54:26.700 INFO:tasks.workunit.client.1.smithi190.stdout:8/228: fdatasync d9/dd/f2d 0 2024-03-26T18:54:26.700 INFO:tasks.workunit.client.1.smithi190.stdout:8/229: truncate d9/dd/d15/f30 456068 0 2024-03-26T18:54:26.701 INFO:tasks.workunit.client.1.smithi190.stdout:8/230: mknod d9/dd/d23/d36/d37/d3a/c4a 0 2024-03-26T18:54:26.701 INFO:tasks.workunit.client.1.smithi190.stdout:8/231: creat d9/dd/d15/f4b x:0 0 0 2024-03-26T18:54:26.710 INFO:tasks.workunit.client.1.smithi190.stdout:8/232: dread d9/f12 [4194304,4194304] 0 2024-03-26T18:54:26.717 INFO:tasks.workunit.client.1.smithi190.stdout:8/233: dread f6 [4194304,4194304] 0 2024-03-26T18:54:26.718 INFO:tasks.workunit.client.1.smithi190.stdout:8/234: rmdir d9/dd/d15/d32 39 2024-03-26T18:54:26.719 INFO:tasks.workunit.client.1.smithi190.stdout:8/235: mknod d9/dd/d23/d36/d37/c4c 0 2024-03-26T18:54:26.727 INFO:tasks.workunit.client.1.smithi190.stdout:6/189: sync 2024-03-26T18:54:26.727 INFO:tasks.workunit.client.1.smithi190.stdout:6/190: read - d17/d28/d2c/f42 zero size 2024-03-26T18:54:26.727 INFO:tasks.workunit.client.1.smithi190.stdout:8/236: write d9/f12 [2566882,129516] 0 2024-03-26T18:54:26.728 INFO:tasks.workunit.client.1.smithi190.stdout:8/237: dread d9/dd/d15/f30 [0,4194304] 0 2024-03-26T18:54:26.728 INFO:tasks.workunit.client.1.smithi190.stdout:8/238: write d9/d16/f1e [1407006,38700] 0 2024-03-26T18:54:26.728 INFO:tasks.workunit.client.1.smithi190.stdout:8/239: dread - d9/dd/d23/f3e zero size 2024-03-26T18:54:26.733 INFO:tasks.workunit.client.1.smithi190.stdout:6/191: dread d17/d28/d2c/f41 [4194304,4194304] 0 2024-03-26T18:54:26.734 INFO:tasks.workunit.client.1.smithi190.stdout:6/192: mkdir d17/d28/d46 0 2024-03-26T18:54:26.735 INFO:tasks.workunit.client.1.smithi190.stdout:6/193: fdatasync d17/f23 0 2024-03-26T18:54:26.736 INFO:tasks.workunit.client.1.smithi190.stdout:8/240: dread d9/f14 [0,4194304] 0 2024-03-26T18:54:26.737 INFO:tasks.workunit.client.1.smithi190.stdout:8/241: truncate d9/dd/f34 3869651 0 2024-03-26T18:54:26.737 INFO:tasks.workunit.client.1.smithi190.stdout:8/242: symlink d9/dd/d23/d36/d37/d3a/l4d 0 2024-03-26T18:54:26.739 INFO:tasks.workunit.client.1.smithi190.stdout:6/194: getdents d17/d28 0 2024-03-26T18:54:26.739 INFO:tasks.workunit.client.1.smithi190.stdout:6/195: creat d17/d28/d2c/f47 x:0 0 0 2024-03-26T18:54:26.739 INFO:tasks.workunit.client.1.smithi190.stdout:6/196: write d17/f23 [619654,113228] 0 2024-03-26T18:54:26.741 INFO:tasks.workunit.client.1.smithi190.stdout:6/197: rename d17/d28/d2b/c3b to d17/d28/d46/c48 0 2024-03-26T18:54:26.742 INFO:tasks.workunit.client.1.smithi190.stdout:6/198: mknod d17/d28/d46/c49 0 2024-03-26T18:54:26.758 INFO:tasks.workunit.client.1.smithi190.stdout:8/243: dwrite d9/f14 [0,4194304] 0 2024-03-26T18:54:26.759 INFO:tasks.workunit.client.1.smithi190.stdout:6/199: dwrite f9 [4194304,4194304] 0 2024-03-26T18:54:26.764 INFO:tasks.workunit.client.1.smithi190.stdout:6/200: mknod d17/c4a 0 2024-03-26T18:54:26.767 INFO:tasks.workunit.client.1.smithi190.stdout:6/201: symlink d17/d28/d46/l4b 0 2024-03-26T18:54:26.778 INFO:tasks.workunit.client.1.smithi190.stdout:8/244: dwrite d9/dd/d15/f1c [0,4194304] 0 2024-03-26T18:54:26.778 INFO:tasks.workunit.client.1.smithi190.stdout:8/245: chown d9/dd/d23/d36/d37/d3a/d43 211876833 1 2024-03-26T18:54:26.780 INFO:tasks.workunit.client.1.smithi190.stdout:3/175: sync 2024-03-26T18:54:26.780 INFO:tasks.workunit.client.1.smithi190.stdout:3/176: fdatasync d1/de/f2f 0 2024-03-26T18:54:26.780 INFO:tasks.workunit.client.1.smithi190.stdout:0/318: sync 2024-03-26T18:54:26.780 INFO:tasks.workunit.client.1.smithi190.stdout:1/185: sync 2024-03-26T18:54:26.781 INFO:tasks.workunit.client.1.smithi190.stdout:4/243: sync 2024-03-26T18:54:26.781 INFO:tasks.workunit.client.1.smithi190.stdout:5/119: sync 2024-03-26T18:54:26.781 INFO:tasks.workunit.client.1.smithi190.stdout:2/217: sync 2024-03-26T18:54:26.781 INFO:tasks.workunit.client.1.smithi190.stdout:9/459: sync 2024-03-26T18:54:26.781 INFO:tasks.workunit.client.1.smithi190.stdout:5/120: creat d2/d17/d1b/d22/f28 x:0 0 0 2024-03-26T18:54:26.781 INFO:tasks.workunit.client.1.smithi190.stdout:9/460: dread - d12/d2d/d1d/d23/d2a/d71/fa6 zero size 2024-03-26T18:54:26.782 INFO:tasks.workunit.client.1.smithi190.stdout:7/229: sync 2024-03-26T18:54:26.782 INFO:tasks.workunit.client.1.smithi190.stdout:3/177: link d1/d3/df/d18/d1b/l37 d1/d3/df/l38 0 2024-03-26T18:54:26.782 INFO:tasks.workunit.client.1.smithi190.stdout:3/178: fsync d1/de/f2f 0 2024-03-26T18:54:26.782 INFO:tasks.workunit.client.1.smithi190.stdout:3/179: chown d1/d3/df/d18/d1b/c32 1153645 1 2024-03-26T18:54:26.782 INFO:tasks.workunit.client.1.smithi190.stdout:0/319: symlink d0/dd/d24/d4b/d5e/l76 0 2024-03-26T18:54:26.783 INFO:tasks.workunit.client.1.smithi190.stdout:0/320: dread - d0/dd/d20/d1c/d4e/f74 zero size 2024-03-26T18:54:26.783 INFO:tasks.workunit.client.1.smithi190.stdout:0/321: readlink d0/dd/d24/d4b/d5e/l76 0 2024-03-26T18:54:26.787 INFO:tasks.workunit.client.1.smithi190.stdout:4/244: truncate d7/d8/d14/f2b 191257 0 2024-03-26T18:54:26.787 INFO:tasks.workunit.client.1.smithi190.stdout:2/218: mkdir d6/d9/d48 0 2024-03-26T18:54:26.787 INFO:tasks.workunit.client.1.smithi190.stdout:9/461: mkdir d12/d2d/d1d/d23/d2a/d71/daa 0 2024-03-26T18:54:26.787 INFO:tasks.workunit.client.1.smithi190.stdout:9/462: creat d12/d2d/d1d/d23/d2b/fab x:0 0 0 2024-03-26T18:54:26.788 INFO:tasks.workunit.client.1.smithi190.stdout:5/121: symlink d2/d17/l29 0 2024-03-26T18:54:26.790 INFO:tasks.workunit.client.1.smithi190.stdout:7/230: rename d3/d6/d9/f2e to d3/d6/f56 0 2024-03-26T18:54:26.790 INFO:tasks.workunit.client.1.smithi190.stdout:1/186: dread d3/da/dd/f12 [0,4194304] 0 2024-03-26T18:54:26.790 INFO:tasks.workunit.client.1.smithi190.stdout:8/246: dread d9/f33 [0,4194304] 0 2024-03-26T18:54:26.793 INFO:tasks.workunit.client.1.smithi190.stdout:2/219: creat d6/d9/d3a/d3d/d1c/d44/f49 x:0 0 0 2024-03-26T18:54:26.793 INFO:tasks.workunit.client.1.smithi190.stdout:2/220: chown d6/d9/d3a/c46 772518 1 2024-03-26T18:54:26.793 INFO:tasks.workunit.client.1.smithi190.stdout:2/221: dread - d6/d9/f41 zero size 2024-03-26T18:54:26.794 INFO:tasks.workunit.client.1.smithi190.stdout:5/122: truncate d2/f6 1194319 0 2024-03-26T18:54:26.794 INFO:tasks.workunit.client.1.smithi190.stdout:6/202: dwrite d17/f23 [0,4194304] 0 2024-03-26T18:54:26.795 INFO:tasks.workunit.client.1.smithi190.stdout:7/231: mkdir d3/d1b/d57 0 2024-03-26T18:54:26.797 INFO:tasks.workunit.client.1.smithi190.stdout:3/180: getdents d1/d3/df/d18 0 2024-03-26T18:54:26.797 INFO:tasks.workunit.client.1.smithi190.stdout:3/181: readlink d1/d3/df/d18/d1b/l27 0 2024-03-26T18:54:26.800 INFO:tasks.workunit.client.1.smithi190.stdout:8/247: rename d9/dd/d15/d32/c3c to d9/d48/d49/c4e 0 2024-03-26T18:54:26.801 INFO:tasks.workunit.client.1.smithi190.stdout:4/245: rmdir d7/d1d/d42 39 2024-03-26T18:54:26.803 INFO:tasks.workunit.client.1.smithi190.stdout:2/222: creat d6/d9/d3a/d3d/d1c/f4a x:0 0 0 2024-03-26T18:54:26.804 INFO:tasks.workunit.client.1.smithi190.stdout:5/123: symlink d2/d17/d1b/d22/d24/l2a 0 2024-03-26T18:54:26.807 INFO:tasks.workunit.client.1.smithi190.stdout:9/463: dwrite f8 [0,4194304] 0 2024-03-26T18:54:26.807 INFO:tasks.workunit.client.1.smithi190.stdout:9/464: write d12/d2d/d1d/d23/d2b/f68 [1294950,35259] 0 2024-03-26T18:54:26.807 INFO:tasks.workunit.client.1.smithi190.stdout:9/465: chown d12/d2d/d1d/d54/f90 0 1 2024-03-26T18:54:26.807 INFO:tasks.workunit.client.1.smithi190.stdout:2/223: dread f5 [0,4194304] 0 2024-03-26T18:54:26.808 INFO:tasks.workunit.client.1.smithi190.stdout:6/203: getdents d17/d28/d46 0 2024-03-26T18:54:26.809 INFO:tasks.workunit.client.1.smithi190.stdout:6/204: fdatasync d17/f23 0 2024-03-26T18:54:26.809 INFO:tasks.workunit.client.1.smithi190.stdout:7/232: creat d3/d6/d9/d1f/d41/d50/d53/f58 x:0 0 0 2024-03-26T18:54:26.811 INFO:tasks.workunit.client.1.smithi190.stdout:3/182: creat d1/d3/df/d18/d35/f39 x:0 0 0 2024-03-26T18:54:26.811 INFO:tasks.workunit.client.1.smithi190.stdout:3/183: write d1/d3/df/f19 [1126347,88140] 0 2024-03-26T18:54:26.812 INFO:tasks.workunit.client.1.smithi190.stdout:4/246: link d7/f10 d7/d8/d1c/d20/d40/f54 0 2024-03-26T18:54:26.812 INFO:tasks.workunit.client.1.smithi190.stdout:4/247: chown d7/d8/d14/d23/d28/l37 20 1 2024-03-26T18:54:26.813 INFO:tasks.workunit.client.1.smithi190.stdout:5/124: dread d2/f4 [0,4194304] 0 2024-03-26T18:54:26.815 INFO:tasks.workunit.client.1.smithi190.stdout:9/466: unlink d12/d2d/f53 0 2024-03-26T18:54:26.815 INFO:tasks.workunit.client.1.smithi190.stdout:9/467: write d12/d2d/f99 [152071,83127] 0 2024-03-26T18:54:26.816 INFO:tasks.workunit.client.1.smithi190.stdout:6/205: rmdir d17/d28/d2b 39 2024-03-26T18:54:26.818 INFO:tasks.workunit.client.1.smithi190.stdout:2/224: write d6/d9/d10/f2c [1335326,111234] 0 2024-03-26T18:54:26.818 INFO:tasks.workunit.client.1.smithi190.stdout:2/225: creat d6/d9/d10/f4b x:0 0 0 2024-03-26T18:54:26.818 INFO:tasks.workunit.client.1.smithi190.stdout:2/226: write f5 [1494552,78125] 0 2024-03-26T18:54:26.818 INFO:tasks.workunit.client.1.smithi190.stdout:2/227: creat d6/d9/d3a/d3d/d1c/d44/f4c x:0 0 0 2024-03-26T18:54:26.819 INFO:tasks.workunit.client.1.smithi190.stdout:2/228: creat d6/f4d x:0 0 0 2024-03-26T18:54:26.819 INFO:tasks.workunit.client.1.smithi190.stdout:2/229: creat d6/d9/f4e x:0 0 0 2024-03-26T18:54:26.819 INFO:tasks.workunit.client.1.smithi190.stdout:3/184: creat d1/de/d1c/d36/f3a x:0 0 0 2024-03-26T18:54:26.819 INFO:tasks.workunit.client.1.smithi190.stdout:3/185: stat d1/fb 0 2024-03-26T18:54:26.820 INFO:tasks.workunit.client.1.smithi190.stdout:4/248: link d7/d8/d14/d23/d28/d38/f4e d7/d8/d14/d23/f55 0 2024-03-26T18:54:26.823 INFO:tasks.workunit.client.1.smithi190.stdout:9/468: creat d12/d2d/d1d/da1/fac x:0 0 0 2024-03-26T18:54:26.824 INFO:tasks.workunit.client.1.smithi190.stdout:6/206: unlink d17/d28/d38/c3d 0 2024-03-26T18:54:26.829 INFO:tasks.workunit.client.1.smithi190.stdout:7/233: dwrite d3/d6/f39 [0,4194304] 0 2024-03-26T18:54:26.832 INFO:tasks.workunit.client.1.smithi190.stdout:5/125: symlink d2/l2b 0 2024-03-26T18:54:26.833 INFO:tasks.workunit.client.1.smithi190.stdout:9/469: symlink d12/d80/lad 0 2024-03-26T18:54:26.833 INFO:tasks.workunit.client.1.smithi190.stdout:9/470: dread - d12/d2d/d1d/d23/f69 zero size 2024-03-26T18:54:26.833 INFO:tasks.workunit.client.1.smithi190.stdout:9/471: readlink d12/d2d/l89 0 2024-03-26T18:54:26.833 INFO:tasks.workunit.client.1.smithi190.stdout:9/472: chown d12/d2d/d1d/d23/d2a/d3f/f4d 677551029 1 2024-03-26T18:54:26.836 INFO:tasks.workunit.client.1.smithi190.stdout:7/234: mkdir d3/d6/d9/d14/d59 0 2024-03-26T18:54:26.838 INFO:tasks.workunit.client.1.smithi190.stdout:5/126: mknod d2/d17/c2c 0 2024-03-26T18:54:26.838 INFO:tasks.workunit.client.1.smithi190.stdout:9/473: stat f11 0 2024-03-26T18:54:26.840 INFO:tasks.workunit.client.1.smithi190.stdout:9/474: creat d12/d2d/d1d/d23/d83/d91/fae x:0 0 0 2024-03-26T18:54:26.840 INFO:tasks.workunit.client.1.smithi190.stdout:7/235: getdents d3/d1b 0 2024-03-26T18:54:26.840 INFO:tasks.workunit.client.1.smithi190.stdout:7/236: readlink d3/d6/d9/d14/l4d 0 2024-03-26T18:54:26.846 INFO:tasks.workunit.client.1.smithi190.stdout:3/186: dwrite d1/de/f29 [0,4194304] 0 2024-03-26T18:54:26.847 INFO:tasks.workunit.client.1.smithi190.stdout:4/249: dwrite d7/d1d/f3c [0,4194304] 0 2024-03-26T18:54:26.852 INFO:tasks.workunit.client.1.smithi190.stdout:3/187: truncate d1/f2c 2546859 0 2024-03-26T18:54:26.857 INFO:tasks.workunit.client.1.smithi190.stdout:4/250: symlink d7/d8/d14/d23/l56 0 2024-03-26T18:54:26.857 INFO:tasks.workunit.client.1.smithi190.stdout:2/230: rmdir d6/d9/d10 39 2024-03-26T18:54:26.862 INFO:tasks.workunit.client.1.smithi190.stdout:3/188: link d1/f15 d1/f3b 0 2024-03-26T18:54:26.864 INFO:tasks.workunit.client.1.smithi190.stdout:4/251: creat d7/d8/d14/f57 x:0 0 0 2024-03-26T18:54:26.865 INFO:tasks.workunit.client.1.smithi190.stdout:2/231: creat d6/d9/d3a/f4f x:0 0 0 2024-03-26T18:54:26.867 INFO:tasks.workunit.client.1.smithi190.stdout:3/189: mkdir d1/de/d1c/d36/d3c 0 2024-03-26T18:54:26.872 INFO:tasks.workunit.client.1.smithi190.stdout:3/190: mknod d1/d3/df/d18/d1b/c3d 0 2024-03-26T18:54:26.876 INFO:tasks.workunit.client.1.smithi190.stdout:7/237: dwrite d3/d1b/f3b [0,4194304] 0 2024-03-26T18:54:26.876 INFO:tasks.workunit.client.1.smithi190.stdout:7/238: creat d3/f5a x:0 0 0 2024-03-26T18:54:26.876 INFO:tasks.workunit.client.1.smithi190.stdout:7/239: fsync d3/d6/d9/d1f/f34 0 2024-03-26T18:54:26.878 INFO:tasks.workunit.client.1.smithi190.stdout:7/240: getdents d3/d6/d9/d14 0 2024-03-26T18:54:26.879 INFO:tasks.workunit.client.1.smithi190.stdout:7/241: mknod d3/d6/d9/d14/d59/c5b 0 2024-03-26T18:54:26.879 INFO:tasks.workunit.client.1.smithi190.stdout:5/127: dwrite d2/fa [0,4194304] 0 2024-03-26T18:54:26.879 INFO:tasks.workunit.client.1.smithi190.stdout:5/128: chown d2/d17/d1b/d22/f27 35 1 2024-03-26T18:54:26.880 INFO:tasks.workunit.client.1.smithi190.stdout:7/242: creat d3/d6/d9/d1f/d4c/f5c x:0 0 0 2024-03-26T18:54:26.881 INFO:tasks.workunit.client.1.smithi190.stdout:9/475: dwrite d12/d2d/d1d/d23/d2a/d3f/d55/f62 [0,4194304] 0 2024-03-26T18:54:26.881 INFO:tasks.workunit.client.1.smithi190.stdout:9/476: chown d12/d2d/d1d/d23/d83 2214 1 2024-03-26T18:54:26.881 INFO:tasks.workunit.client.1.smithi190.stdout:9/477: fdatasync d12/f16 0 2024-03-26T18:54:26.881 INFO:tasks.workunit.client.1.smithi190.stdout:9/478: chown d12/d2d/f27 4746 1 2024-03-26T18:54:26.881 INFO:tasks.workunit.client.1.smithi190.stdout:9/479: stat d12/d2d/d1d/d23/d2a/d3f 0 2024-03-26T18:54:26.881 INFO:tasks.workunit.client.1.smithi190.stdout:7/243: truncate d3/d6/f4e 1837992 0 2024-03-26T18:54:26.882 INFO:tasks.workunit.client.1.smithi190.stdout:7/244: write d3/d6/f30 [4894253,120014] 0 2024-03-26T18:54:26.886 INFO:tasks.workunit.client.1.smithi190.stdout:3/191: write d1/d3/f12 [906855,4648] 0 2024-03-26T18:54:26.886 INFO:tasks.workunit.client.1.smithi190.stdout:3/192: chown d1/d3/f34 93147 1 2024-03-26T18:54:26.886 INFO:tasks.workunit.client.1.smithi190.stdout:3/193: write d1/de/d1c/d36/f3a [749967,29809] 0 2024-03-26T18:54:26.889 INFO:tasks.workunit.client.1.smithi190.stdout:3/194: rmdir d1/de/d1c/d36/d3c 0 2024-03-26T18:54:26.890 INFO:tasks.workunit.client.1.smithi190.stdout:9/480: dread d12/d2d/d1d/d23/d2a/d71/f75 [0,4194304] 0 2024-03-26T18:54:26.896 INFO:tasks.workunit.client.1.smithi190.stdout:2/232: dwrite d6/d9/d10/f2c [0,4194304] 0 2024-03-26T18:54:26.899 INFO:tasks.workunit.client.1.smithi190.stdout:2/233: mknod d6/d9/d3a/d3d/d2e/d31/c50 0 2024-03-26T18:54:26.899 INFO:tasks.workunit.client.1.smithi190.stdout:2/234: write d6/d9/d10/f4b [989633,94018] 0 2024-03-26T18:54:26.899 INFO:tasks.workunit.client.1.smithi190.stdout:2/235: truncate d6/d9/d10/f34 565677 0 2024-03-26T18:54:26.900 INFO:tasks.workunit.client.1.smithi190.stdout:2/236: readlink d6/d9/l12 0 2024-03-26T18:54:26.919 INFO:tasks.workunit.client.1.smithi190.stdout:5/129: dwrite d2/f3 [0,4194304] 0 2024-03-26T18:54:26.920 INFO:tasks.workunit.client.1.smithi190.stdout:9/481: dwrite d12/d2d/d1d/d54/f90 [0,4194304] 0 2024-03-26T18:54:26.920 INFO:tasks.workunit.client.1.smithi190.stdout:7/245: dwrite d3/d6/d9/d14/f33 [0,4194304] 0 2024-03-26T18:54:26.920 INFO:tasks.workunit.client.1.smithi190.stdout:7/246: fsync d3/f46 0 2024-03-26T18:54:26.920 INFO:tasks.workunit.client.1.smithi190.stdout:7/247: creat d3/d6/f5d x:0 0 0 2024-03-26T18:54:26.922 INFO:tasks.workunit.client.1.smithi190.stdout:3/195: dwrite d1/d3/df/d18/f30 [0,4194304] 0 2024-03-26T18:54:26.925 INFO:tasks.workunit.client.1.smithi190.stdout:9/482: dread d12/d2d/f31 [0,4194304] 0 2024-03-26T18:54:26.925 INFO:tasks.workunit.client.1.smithi190.stdout:9/483: chown d12/f4c 0 1 2024-03-26T18:54:26.926 INFO:tasks.workunit.client.1.smithi190.stdout:9/484: creat d12/faf x:0 0 0 2024-03-26T18:54:26.926 INFO:tasks.workunit.client.1.smithi190.stdout:2/237: rmdir d6/d9/d3a/d3d/d2e/d31 39 2024-03-26T18:54:26.926 INFO:tasks.workunit.client.1.smithi190.stdout:2/238: stat d6/d9/f1d 0 2024-03-26T18:54:26.928 INFO:tasks.workunit.client.1.smithi190.stdout:3/196: write d1/f2c [2402325,129554] 0 2024-03-26T18:54:26.928 INFO:tasks.workunit.client.1.smithi190.stdout:3/197: stat d1/d3/f10 0 2024-03-26T18:54:26.928 INFO:tasks.workunit.client.1.smithi190.stdout:7/248: creat d3/d1b/f5e x:0 0 0 2024-03-26T18:54:26.932 INFO:tasks.workunit.client.1.smithi190.stdout:9/485: truncate d12/d2d/f9f 1303032 0 2024-03-26T18:54:26.934 INFO:tasks.workunit.client.1.smithi190.stdout:2/239: creat d6/d9/d3a/d3d/d2e/f51 x:0 0 0 2024-03-26T18:54:26.936 INFO:tasks.workunit.client.1.smithi190.stdout:7/249: creat d3/d1b/d57/f5f x:0 0 0 2024-03-26T18:54:26.936 INFO:tasks.workunit.client.1.smithi190.stdout:7/250: creat d3/d6/d9/d1f/d41/d50/f60 x:0 0 0 2024-03-26T18:54:26.936 INFO:tasks.workunit.client.1.smithi190.stdout:7/251: chown d3/d6/d9/d14/d36/l42 24 1 2024-03-26T18:54:26.936 INFO:tasks.workunit.client.1.smithi190.stdout:7/252: read d3/d6/d9/d1f/d41/f44 [104733,66852] 0 2024-03-26T18:54:26.938 INFO:tasks.workunit.client.1.smithi190.stdout:3/198: write d1/f14 [7780469,84078] 0 2024-03-26T18:54:26.939 INFO:tasks.workunit.client.1.smithi190.stdout:3/199: symlink d1/l3e 0 2024-03-26T18:54:26.943 INFO:tasks.workunit.client.1.smithi190.stdout:5/130: dwrite d2/d17/d1b/d22/f27 [0,4194304] 0 2024-03-26T18:54:26.953 INFO:tasks.workunit.client.1.smithi190.stdout:2/240: dread d6/f7 [0,4194304] 0 2024-03-26T18:54:26.953 INFO:tasks.workunit.client.1.smithi190.stdout:2/241: stat d6/d21/c23 0 2024-03-26T18:54:26.982 INFO:tasks.workunit.client.1.smithi190.stdout:5/131: dwrite d2/f6 [0,4194304] 0 2024-03-26T18:54:26.982 INFO:tasks.workunit.client.1.smithi190.stdout:5/132: write d2/f15 [79375,24166] 0 2024-03-26T18:54:26.982 INFO:tasks.workunit.client.1.smithi190.stdout:5/133: creat d2/d17/d1b/d22/f2d x:0 0 0 2024-03-26T18:54:26.982 INFO:tasks.workunit.client.1.smithi190.stdout:2/242: dwrite d6/d9/d3a/d3d/f27 [0,4194304] 0 2024-03-26T18:54:26.982 INFO:tasks.workunit.client.1.smithi190.stdout:2/243: fsync d6/f36 0 2024-03-26T18:54:26.982 INFO:tasks.workunit.client.1.smithi190.stdout:2/244: truncate d6/d9/d10/f40 413385 0 2024-03-26T18:54:26.983 INFO:tasks.workunit.client.1.smithi190.stdout:2/245: chown d6/c30 72459 1 2024-03-26T18:54:26.983 INFO:tasks.workunit.client.1.smithi190.stdout:5/134: creat d2/d17/d1b/d1e/f2e x:0 0 0 2024-03-26T18:54:26.983 INFO:tasks.workunit.client.1.smithi190.stdout:5/135: truncate d2/f10 510917 0 2024-03-26T18:54:26.984 INFO:tasks.workunit.client.1.smithi190.stdout:2/246: mknod d6/d9/d3a/d3d/d2e/d31/c52 0 2024-03-26T18:54:26.984 INFO:tasks.workunit.client.1.smithi190.stdout:2/247: fsync d6/f3c 0 2024-03-26T18:54:26.985 INFO:tasks.workunit.client.1.smithi190.stdout:5/136: link d2/d17/c2c d2/d17/c2f 0 2024-03-26T18:54:26.986 INFO:tasks.workunit.client.1.smithi190.stdout:2/248: symlink d6/d9/d3a/d3d/d1c/l53 0 2024-03-26T18:54:26.986 INFO:tasks.workunit.client.1.smithi190.stdout:2/249: readlink d6/d9/d1a/l2a 0 2024-03-26T18:54:26.987 INFO:tasks.workunit.client.1.smithi190.stdout:2/250: chown d6/d9/d45 671 1 2024-03-26T18:54:26.987 INFO:tasks.workunit.client.1.smithi190.stdout:5/137: mkdir d2/d17/d1b/d30 0 2024-03-26T18:54:26.989 INFO:tasks.workunit.client.1.smithi190.stdout:2/251: rename d6/f3f to d6/d9/d3a/d3d/d1c/f54 0 2024-03-26T18:54:26.989 INFO:tasks.workunit.client.1.smithi190.stdout:2/252: chown d6/d9/d3a/d3d/l11 21050348 1 2024-03-26T18:54:26.989 INFO:tasks.workunit.client.1.smithi190.stdout:2/253: readlink d6/d9/d10/l1f 0 2024-03-26T18:54:26.989 INFO:tasks.workunit.client.1.smithi190.stdout:5/138: truncate d2/f14 50850 0 2024-03-26T18:54:26.989 INFO:tasks.workunit.client.1.smithi190.stdout:5/139: chown d2/d17/d1b/c23 186 1 2024-03-26T18:54:26.989 INFO:tasks.workunit.client.1.smithi190.stdout:5/140: truncate d2/d17/f21 642604 0 2024-03-26T18:54:26.990 INFO:tasks.workunit.client.1.smithi190.stdout:2/254: rmdir d6/d21 39 2024-03-26T18:54:26.994 INFO:tasks.workunit.client.1.smithi190.stdout:2/255: dread f5 [0,4194304] 0 2024-03-26T18:54:26.996 INFO:tasks.workunit.client.1.smithi190.stdout:2/256: mkdir d6/d9/d3a/d3d/d55 0 2024-03-26T18:54:26.996 INFO:tasks.workunit.client.1.smithi190.stdout:2/257: rename d6/d9 to d6/d9/d10/d56 22 2024-03-26T18:54:26.998 INFO:tasks.workunit.client.1.smithi190.stdout:5/141: dread d2/f3 [0,4194304] 0 2024-03-26T18:54:26.999 INFO:tasks.workunit.client.1.smithi190.stdout:5/142: rename d2/c9 to d2/d17/d1b/d1e/c31 0 2024-03-26T18:54:27.000 INFO:tasks.workunit.client.1.smithi190.stdout:5/143: symlink d2/d17/d1b/d22/l32 0 2024-03-26T18:54:27.001 INFO:tasks.workunit.client.1.smithi190.stdout:5/144: chown d2/d17/l1d 168930342 1 2024-03-26T18:54:27.001 INFO:tasks.workunit.client.1.smithi190.stdout:5/145: chown d2/c11 42 1 2024-03-26T18:54:27.003 INFO:tasks.workunit.client.1.smithi190.stdout:5/146: unlink d2/d17/l1d 0 2024-03-26T18:54:27.004 INFO:tasks.workunit.client.1.smithi190.stdout:5/147: symlink d2/d17/d1b/d22/l33 0 2024-03-26T18:54:27.006 INFO:tasks.workunit.client.1.smithi190.stdout:2/258: write d6/fe [3676166,73050] 0 2024-03-26T18:54:27.006 INFO:tasks.workunit.client.1.smithi190.stdout:2/259: write d6/d9/d1a/f43 [927140,50795] 0 2024-03-26T18:54:27.008 INFO:tasks.workunit.client.1.smithi190.stdout:2/260: creat d6/d9/d3a/d3d/f57 x:0 0 0 2024-03-26T18:54:27.008 INFO:tasks.workunit.client.1.smithi190.stdout:2/261: creat d6/d9/d1a/f58 x:0 0 0 2024-03-26T18:54:27.014 INFO:tasks.workunit.client.1.smithi190.stdout:2/262: fsync d6/d9/d1a/f58 0 2024-03-26T18:54:27.015 INFO:tasks.workunit.client.1.smithi190.stdout:2/263: creat d6/d9/d10/f59 x:0 0 0 2024-03-26T18:54:27.015 INFO:tasks.workunit.client.1.smithi190.stdout:2/264: symlink d6/d9/d45/l5a 0 2024-03-26T18:54:27.015 INFO:tasks.workunit.client.1.smithi190.stdout:2/265: write f5 [3324595,65816] 0 2024-03-26T18:54:27.016 INFO:tasks.workunit.client.1.smithi190.stdout:2/266: write d6/f7 [3237063,122966] 0 2024-03-26T18:54:27.017 INFO:tasks.workunit.client.1.smithi190.stdout:5/148: dwrite d2/d17/d1b/d22/f28 [0,4194304] 0 2024-03-26T18:54:27.039 INFO:tasks.workunit.client.1.smithi190.stdout:5/149: dwrite d2/f16 [0,4194304] 0 2024-03-26T18:54:27.039 INFO:tasks.workunit.client.1.smithi190.stdout:5/150: stat d2/f3 0 2024-03-26T18:54:27.039 INFO:tasks.workunit.client.1.smithi190.stdout:5/151: stat d2/f16 0 2024-03-26T18:54:27.039 INFO:tasks.workunit.client.1.smithi190.stdout:2/267: dwrite d6/d9/d10/f13 [4194304,4194304] 0 2024-03-26T18:54:27.040 INFO:tasks.workunit.client.1.smithi190.stdout:2/268: symlink d6/d9/d3a/d3d/d1c/d44/l5b 0 2024-03-26T18:54:27.041 INFO:tasks.workunit.client.1.smithi190.stdout:2/269: stat d6/d21 0 2024-03-26T18:54:27.041 INFO:tasks.workunit.client.1.smithi190.stdout:2/270: truncate d6/f36 402978 0 2024-03-26T18:54:27.041 INFO:tasks.workunit.client.1.smithi190.stdout:2/271: creat d6/d9/d3a/d3d/d2e/f5c x:0 0 0 2024-03-26T18:54:27.042 INFO:tasks.workunit.client.1.smithi190.stdout:2/272: mknod d6/d9/d3a/c5d 0 2024-03-26T18:54:27.043 INFO:tasks.workunit.client.1.smithi190.stdout:2/273: dread d6/f36 [0,4194304] 0 2024-03-26T18:54:27.043 INFO:tasks.workunit.client.1.smithi190.stdout:2/274: stat d6/d9/d10/f2c 0 2024-03-26T18:54:27.050 INFO:tasks.workunit.client.1.smithi190.stdout:5/152: write d2/d17/d1b/d22/f27 [279717,92307] 0 2024-03-26T18:54:27.050 INFO:tasks.workunit.client.1.smithi190.stdout:5/153: readlink d2/d17/d1b/d22/l25 0 2024-03-26T18:54:27.051 INFO:tasks.workunit.client.1.smithi190.stdout:5/154: dread d2/f14 [0,4194304] 0 2024-03-26T18:54:27.067 INFO:tasks.workunit.client.1.smithi190.stdout:2/275: dwrite d6/f3c [0,4194304] 0 2024-03-26T18:54:27.069 INFO:tasks.workunit.client.1.smithi190.stdout:2/276: rename d6/d9/f1d to d6/d9/d3a/d3d/d1c/f5e 0 2024-03-26T18:54:27.070 INFO:tasks.workunit.client.1.smithi190.stdout:2/277: mkdir d6/d9/d10/d5f 0 2024-03-26T18:54:27.072 INFO:tasks.workunit.client.1.smithi190.stdout:2/278: unlink d6/d9/d1a/c2f 0 2024-03-26T18:54:27.108 INFO:tasks.workunit.client.0.smithi102.stderr:++ readlink -f fsstress 2024-03-26T18:54:27.108 INFO:tasks.workunit.client.0.smithi102.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-26T18:54:27.108 INFO:tasks.workunit.client.0.smithi102.stderr:+ popd 2024-03-26T18:54:27.109 INFO:tasks.workunit.client.0.smithi102.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-26T18:54:27.109 INFO:tasks.workunit.client.0.smithi102.stderr:+ popd 2024-03-26T18:54:27.109 INFO:tasks.workunit.client.0.smithi102.stdout:~/cephtest/mnt.0/client.0/tmp 2024-03-26T18:54:27.109 INFO:tasks.workunit.client.0.smithi102.stderr:++ mktemp -d -p . 2024-03-26T18:54:27.110 INFO:tasks.workunit.client.0.smithi102.stderr:+ T=./tmp.yZZUd38di3 2024-03-26T18:54:27.110 INFO:tasks.workunit.client.0.smithi102.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.yZZUd38di3 -l 1 -n 1000 -p 10 -v 2024-03-26T18:54:27.111 INFO:tasks.workunit.client.0.smithi102.stdout:seed = 1711460342 2024-03-26T18:54:27.114 INFO:tasks.workunit.client.0.smithi102.stdout:1/0: dwrite - no filename 2024-03-26T18:54:27.114 INFO:tasks.workunit.client.0.smithi102.stdout:1/1: readlink - no filename 2024-03-26T18:54:27.114 INFO:tasks.workunit.client.0.smithi102.stdout:0/0: creat f0 x:0 0 0 2024-03-26T18:54:27.114 INFO:tasks.workunit.client.0.smithi102.stdout:0/1: stat f0 0 2024-03-26T18:54:27.115 INFO:tasks.workunit.client.0.smithi102.stdout:1/2: symlink l0 0 2024-03-26T18:54:27.115 INFO:tasks.workunit.client.0.smithi102.stdout:1/3: write - no filename 2024-03-26T18:54:27.115 INFO:tasks.workunit.client.0.smithi102.stdout:1/4: dwrite - no filename 2024-03-26T18:54:27.115 INFO:tasks.workunit.client.0.smithi102.stdout:1/5: write - no filename 2024-03-26T18:54:27.115 INFO:tasks.workunit.client.0.smithi102.stdout:1/6: write - no filename 2024-03-26T18:54:27.116 INFO:tasks.workunit.client.0.smithi102.stdout:0/2: rename f0 to f1 0 2024-03-26T18:54:27.116 INFO:tasks.workunit.client.0.smithi102.stdout:0/3: dread - f1 zero size 2024-03-26T18:54:27.116 INFO:tasks.workunit.client.0.smithi102.stdout:0/4: dread - f1 zero size 2024-03-26T18:54:27.117 INFO:tasks.workunit.client.0.smithi102.stdout:2/0: mknod c0 0 2024-03-26T18:54:27.117 INFO:tasks.workunit.client.0.smithi102.stdout:2/1: chown c0 4944090 1 2024-03-26T18:54:27.117 INFO:tasks.workunit.client.0.smithi102.stdout:2/2: write - no filename 2024-03-26T18:54:27.117 INFO:tasks.workunit.client.0.smithi102.stdout:2/3: stat c0 0 2024-03-26T18:54:27.118 INFO:tasks.workunit.client.0.smithi102.stdout:1/7: creat f1 x:0 0 0 2024-03-26T18:54:27.118 INFO:tasks.workunit.client.0.smithi102.stdout:1/8: creat f2 x:0 0 0 2024-03-26T18:54:27.120 INFO:tasks.workunit.client.0.smithi102.stdout:3/0: creat f0 x:0 0 0 2024-03-26T18:54:27.120 INFO:tasks.workunit.client.0.smithi102.stdout:3/1: dread - f0 zero size 2024-03-26T18:54:27.120 INFO:tasks.workunit.client.0.smithi102.stdout:4/0: write - no filename 2024-03-26T18:54:27.120 INFO:tasks.workunit.client.0.smithi102.stdout:2/4: mknod c1 0 2024-03-26T18:54:27.120 INFO:tasks.workunit.client.0.smithi102.stdout:2/5: dread - no filename 2024-03-26T18:54:27.120 INFO:tasks.workunit.client.0.smithi102.stdout:2/6: chown c1 0 1 2024-03-26T18:54:27.122 INFO:tasks.workunit.client.0.smithi102.stdout:5/0: write - no filename 2024-03-26T18:54:27.123 INFO:tasks.workunit.client.0.smithi102.stdout:5/1: truncate - no filename 2024-03-26T18:54:27.123 INFO:tasks.workunit.client.0.smithi102.stdout:5/2: dread - no filename 2024-03-26T18:54:27.123 INFO:tasks.workunit.client.0.smithi102.stdout:5/3: truncate - no filename 2024-03-26T18:54:27.124 INFO:tasks.workunit.client.0.smithi102.stdout:3/2: mknod c1 0 2024-03-26T18:54:27.124 INFO:tasks.workunit.client.0.smithi102.stdout:3/3: chown c1 455992 1 2024-03-26T18:54:27.124 INFO:tasks.workunit.client.0.smithi102.stdout:3/4: readlink - no filename 2024-03-26T18:54:27.124 INFO:tasks.workunit.client.0.smithi102.stdout:3/5: chown c1 3092 1 2024-03-26T18:54:27.124 INFO:tasks.workunit.client.0.smithi102.stdout:3/6: creat f2 x:0 0 0 2024-03-26T18:54:27.125 INFO:tasks.workunit.client.0.smithi102.stdout:3/7: creat f3 x:0 0 0 2024-03-26T18:54:27.125 INFO:tasks.workunit.client.0.smithi102.stdout:4/1: mkdir d0 0 2024-03-26T18:54:27.125 INFO:tasks.workunit.client.0.smithi102.stdout:4/2: write - no filename 2024-03-26T18:54:27.125 INFO:tasks.workunit.client.0.smithi102.stdout:4/3: chown d0 258674546 1 2024-03-26T18:54:27.127 INFO:tasks.workunit.client.0.smithi102.stdout:2/7: link c1 c2 0 2024-03-26T18:54:27.127 INFO:tasks.workunit.client.0.smithi102.stdout:2/8: dread - no filename 2024-03-26T18:54:27.127 INFO:tasks.workunit.client.0.smithi102.stdout:2/9: truncate - no filename 2024-03-26T18:54:27.127 INFO:tasks.workunit.client.0.smithi102.stdout:2/10: truncate - no filename 2024-03-26T18:54:27.127 INFO:tasks.workunit.client.0.smithi102.stdout:2/11: dread - no filename 2024-03-26T18:54:27.128 INFO:tasks.workunit.client.0.smithi102.stdout:2/12: dwrite - no filename 2024-03-26T18:54:27.129 INFO:tasks.workunit.client.0.smithi102.stdout:6/0: dwrite - no filename 2024-03-26T18:54:27.129 INFO:tasks.workunit.client.0.smithi102.stdout:6/1: getdents . 0 2024-03-26T18:54:27.129 INFO:tasks.workunit.client.0.smithi102.stdout:6/2: unlink - no file 2024-03-26T18:54:27.131 INFO:tasks.workunit.client.0.smithi102.stdout:5/4: creat f0 x:0 0 0 2024-03-26T18:54:27.131 INFO:tasks.workunit.client.0.smithi102.stdout:5/5: creat f1 x:0 0 0 2024-03-26T18:54:27.131 INFO:tasks.workunit.client.0.smithi102.stdout:5/6: truncate f0 508528 0 2024-03-26T18:54:27.132 INFO:tasks.workunit.client.0.smithi102.stdout:5/7: dread f0 [0,4194304] 0 2024-03-26T18:54:27.133 INFO:tasks.workunit.client.0.smithi102.stdout:0/5: rename f1 to f2 0 2024-03-26T18:54:27.133 INFO:tasks.workunit.client.0.smithi102.stdout:0/6: chown f2 42275 1 2024-03-26T18:54:27.133 INFO:tasks.workunit.client.0.smithi102.stdout:0/7: chown f2 44260 1 2024-03-26T18:54:27.133 INFO:tasks.workunit.client.0.smithi102.stdout:0/8: creat f3 x:0 0 0 2024-03-26T18:54:27.136 INFO:tasks.workunit.client.0.smithi102.stdout:4/4: rmdir d0 0 2024-03-26T18:54:27.137 INFO:tasks.workunit.client.0.smithi102.stdout:2/13: mkdir d3 0 2024-03-26T18:54:27.137 INFO:tasks.workunit.client.0.smithi102.stdout:2/14: write - no filename 2024-03-26T18:54:27.138 INFO:tasks.workunit.client.0.smithi102.stdout:7/0: chown . 241 1 2024-03-26T18:54:27.139 INFO:tasks.workunit.client.0.smithi102.stdout:6/3: creat f0 x:0 0 0 2024-03-26T18:54:27.139 INFO:tasks.workunit.client.0.smithi102.stdout:6/4: write f0 [344055,80070] 0 2024-03-26T18:54:27.141 INFO:tasks.workunit.client.0.smithi102.stdout:5/8: mkdir d2 0 2024-03-26T18:54:27.141 INFO:tasks.workunit.client.0.smithi102.stdout:5/9: read f0 [36827,129796] 0 2024-03-26T18:54:27.142 INFO:tasks.workunit.client.0.smithi102.stdout:0/9: symlink l4 0 2024-03-26T18:54:27.142 INFO:tasks.workunit.client.0.smithi102.stdout:0/10: write f2 [58659,55261] 0 2024-03-26T18:54:27.142 INFO:tasks.workunit.client.0.smithi102.stdout:0/11: write f2 [550645,35057] 0 2024-03-26T18:54:27.143 INFO:tasks.workunit.client.0.smithi102.stdout:4/5: mkdir d1 0 2024-03-26T18:54:27.143 INFO:tasks.workunit.client.0.smithi102.stdout:4/6: truncate - no filename 2024-03-26T18:54:27.143 INFO:tasks.workunit.client.0.smithi102.stdout:4/7: fdatasync - no filename 2024-03-26T18:54:27.143 INFO:tasks.workunit.client.0.smithi102.stdout:4/8: fsync - no filename 2024-03-26T18:54:27.143 INFO:tasks.workunit.client.0.smithi102.stdout:2/15: creat d3/f4 x:0 0 0 2024-03-26T18:54:27.144 INFO:tasks.workunit.client.0.smithi102.stdout:3/8: dwrite f0 [0,4194304] 0 2024-03-26T18:54:27.144 INFO:tasks.workunit.client.0.smithi102.stdout:3/9: creat f4 x:0 0 0 2024-03-26T18:54:27.144 INFO:tasks.workunit.client.0.smithi102.stdout:7/1: creat f0 x:0 0 0 2024-03-26T18:54:27.146 INFO:tasks.workunit.client.0.smithi102.stdout:5/10: symlink d2/l3 0 2024-03-26T18:54:27.148 INFO:tasks.workunit.client.0.smithi102.stdout:0/12: unlink l4 0 2024-03-26T18:54:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:4/9: symlink d1/l2 0 2024-03-26T18:54:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:4/10: truncate - no filename 2024-03-26T18:54:27.152 INFO:tasks.workunit.client.0.smithi102.stdout:0/13: dread f2 [0,4194304] 0 2024-03-26T18:54:27.152 INFO:tasks.workunit.client.0.smithi102.stdout:2/16: unlink c2 0 2024-03-26T18:54:27.153 INFO:tasks.workunit.client.0.smithi102.stdout:9/0: dwrite - no filename 2024-03-26T18:54:27.153 INFO:tasks.workunit.client.0.smithi102.stdout:9/1: chown . 467 1 2024-03-26T18:54:27.153 INFO:tasks.workunit.client.0.smithi102.stdout:9/2: chown . 0 1 2024-03-26T18:54:27.153 INFO:tasks.workunit.client.0.smithi102.stdout:9/3: dread - no filename 2024-03-26T18:54:27.154 INFO:tasks.workunit.client.0.smithi102.stdout:9/4: dwrite - no filename 2024-03-26T18:54:27.154 INFO:tasks.workunit.client.0.smithi102.stdout:9/5: dwrite - no filename 2024-03-26T18:54:27.154 INFO:tasks.workunit.client.0.smithi102.stdout:9/6: dread - no filename 2024-03-26T18:54:27.154 INFO:tasks.workunit.client.0.smithi102.stdout:9/7: rmdir - no directory 2024-03-26T18:54:27.154 INFO:tasks.workunit.client.0.smithi102.stdout:9/8: dread - no filename 2024-03-26T18:54:27.154 INFO:tasks.workunit.client.0.smithi102.stdout:9/9: chown . 98859296 1 2024-03-26T18:54:27.154 INFO:tasks.workunit.client.0.smithi102.stdout:9/10: dwrite - no filename 2024-03-26T18:54:27.154 INFO:tasks.workunit.client.0.smithi102.stdout:9/11: stat - no entries 2024-03-26T18:54:27.154 INFO:tasks.workunit.client.0.smithi102.stdout:9/12: chown . 367543 1 2024-03-26T18:54:27.154 INFO:tasks.workunit.client.0.smithi102.stdout:9/13: write - no filename 2024-03-26T18:54:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:8/0: mkdir d0 0 2024-03-26T18:54:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:8/1: fsync - no filename 2024-03-26T18:54:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:8/2: dread - no filename 2024-03-26T18:54:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:8/3: write - no filename 2024-03-26T18:54:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:8/4: truncate - no filename 2024-03-26T18:54:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:8/5: link - no file 2024-03-26T18:54:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:8/6: fsync - no filename 2024-03-26T18:54:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:8/7: chown d0 0 1 2024-03-26T18:54:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:8/8: dwrite - no filename 2024-03-26T18:54:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:8/9: fdatasync - no filename 2024-03-26T18:54:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:8/10: write - no filename 2024-03-26T18:54:27.156 INFO:tasks.workunit.client.0.smithi102.stdout:8/11: write - no filename 2024-03-26T18:54:27.156 INFO:tasks.workunit.client.0.smithi102.stdout:8/12: truncate - no filename 2024-03-26T18:54:27.156 INFO:tasks.workunit.client.0.smithi102.stdout:8/13: dread - no filename 2024-03-26T18:54:27.160 INFO:tasks.workunit.client.0.smithi102.stdout:4/11: mknod d1/c3 0 2024-03-26T18:54:27.161 INFO:tasks.workunit.client.0.smithi102.stdout:0/14: mknod c5 0 2024-03-26T18:54:27.162 INFO:tasks.workunit.client.0.smithi102.stdout:2/17: link d3/f4 d3/f5 0 2024-03-26T18:54:27.162 INFO:tasks.workunit.client.0.smithi102.stdout:2/18: stat c0 0 2024-03-26T18:54:27.164 INFO:tasks.workunit.client.0.smithi102.stdout:9/14: mkdir d0 0 2024-03-26T18:54:27.164 INFO:tasks.workunit.client.0.smithi102.stdout:9/15: chown d0 173396459 1 2024-03-26T18:54:27.164 INFO:tasks.workunit.client.0.smithi102.stdout:9/16: rename d0 to d0/d1 22 2024-03-26T18:54:27.164 INFO:tasks.workunit.client.0.smithi102.stdout:9/17: chown d0 488 1 2024-03-26T18:54:27.165 INFO:tasks.workunit.client.0.smithi102.stdout:8/14: mknod d0/c1 0 2024-03-26T18:54:27.166 INFO:tasks.workunit.client.0.smithi102.stdout:3/10: dwrite f0 [0,4194304] 0 2024-03-26T18:54:27.166 INFO:tasks.workunit.client.0.smithi102.stdout:3/11: write f2 [151730,64582] 0 2024-03-26T18:54:27.168 INFO:tasks.workunit.client.0.smithi102.stdout:4/12: creat d1/f4 x:0 0 0 2024-03-26T18:54:27.168 INFO:tasks.workunit.client.0.smithi102.stdout:4/13: chown d1 0 1 2024-03-26T18:54:27.168 INFO:tasks.workunit.client.0.smithi102.stdout:4/14: write d1/f4 [122672,106716] 0 2024-03-26T18:54:27.168 INFO:tasks.workunit.client.0.smithi102.stdout:4/15: chown d1/c3 24 1 2024-03-26T18:54:27.168 INFO:tasks.workunit.client.0.smithi102.stdout:4/16: chown d1/c3 26556 1 2024-03-26T18:54:27.169 INFO:tasks.workunit.client.0.smithi102.stdout:6/5: getdents . 0 2024-03-26T18:54:27.171 INFO:tasks.workunit.client.0.smithi102.stdout:9/18: rmdir d0 0 2024-03-26T18:54:27.174 INFO:tasks.workunit.client.0.smithi102.stdout:8/15: symlink d0/l2 0 2024-03-26T18:54:27.174 INFO:tasks.workunit.client.0.smithi102.stdout:8/16: stat d0/c1 0 2024-03-26T18:54:27.174 INFO:tasks.workunit.client.0.smithi102.stdout:8/17: chown d0/l2 3 1 2024-03-26T18:54:27.174 INFO:tasks.workunit.client.0.smithi102.stdout:8/18: chown d0/l2 427243 1 2024-03-26T18:54:27.174 INFO:tasks.workunit.client.0.smithi102.stdout:8/19: dwrite - no filename 2024-03-26T18:54:27.174 INFO:tasks.workunit.client.0.smithi102.stdout:8/20: dread - no filename 2024-03-26T18:54:27.177 INFO:tasks.workunit.client.0.smithi102.stdout:3/12: mknod c5 0 2024-03-26T18:54:27.177 INFO:tasks.workunit.client.0.smithi102.stdout:3/13: dread - f4 zero size 2024-03-26T18:54:27.177 INFO:tasks.workunit.client.0.smithi102.stdout:3/14: write f4 [967244,76108] 0 2024-03-26T18:54:27.178 INFO:tasks.workunit.client.0.smithi102.stdout:9/19: mknod c2 0 2024-03-26T18:54:27.179 INFO:tasks.workunit.client.0.smithi102.stdout:9/20: write - no filename 2024-03-26T18:54:27.179 INFO:tasks.workunit.client.0.smithi102.stdout:9/21: chown c2 68698384 1 2024-03-26T18:54:27.180 INFO:tasks.workunit.client.0.smithi102.stdout:8/21: mkdir d0/d3 0 2024-03-26T18:54:27.180 INFO:tasks.workunit.client.0.smithi102.stdout:8/22: dwrite - no filename 2024-03-26T18:54:27.180 INFO:tasks.workunit.client.0.smithi102.stdout:2/19: dwrite d3/f5 [0,4194304] 0 2024-03-26T18:54:27.183 INFO:tasks.workunit.client.0.smithi102.stdout:9/22: rename c2 to c3 0 2024-03-26T18:54:27.183 INFO:tasks.workunit.client.0.smithi102.stdout:8/23: creat d0/d3/f4 x:0 0 0 2024-03-26T18:54:27.188 INFO:tasks.workunit.client.0.smithi102.stdout:5/11: getdents d2 0 2024-03-26T18:54:27.188 INFO:tasks.workunit.client.0.smithi102.stdout:5/12: dread - f1 zero size 2024-03-26T18:54:27.188 INFO:tasks.workunit.client.0.smithi102.stdout:5/13: truncate f1 788945 0 2024-03-26T18:54:27.191 INFO:tasks.workunit.client.0.smithi102.stdout:9/23: creat f4 x:0 0 0 2024-03-26T18:54:27.193 INFO:tasks.workunit.client.0.smithi102.stdout:5/14: rmdir d2 39 2024-03-26T18:54:27.194 INFO:tasks.workunit.client.1.smithi190.stdout:0/322: sync 2024-03-26T18:54:27.195 INFO:tasks.workunit.client.1.smithi190.stdout:0/323: rename d0/dd/d24/d4b to d0/dd/d24/d4b/d5e/d77 22 2024-03-26T18:54:27.195 INFO:tasks.workunit.client.1.smithi190.stdout:0/324: creat d0/dd/d20/f78 x:0 0 0 2024-03-26T18:54:27.195 INFO:tasks.workunit.client.0.smithi102.stdout:9/24: symlink l5 0 2024-03-26T18:54:27.196 INFO:tasks.workunit.client.1.smithi190.stdout:1/187: sync 2024-03-26T18:54:27.196 INFO:tasks.workunit.client.1.smithi190.stdout:1/188: stat d3/d9/l17 0 2024-03-26T18:54:27.198 INFO:tasks.workunit.client.0.smithi102.stdout:5/15: link f1 d2/f4 0 2024-03-26T18:54:27.198 INFO:tasks.workunit.client.0.smithi102.stdout:5/16: chown d2 24 1 2024-03-26T18:54:27.198 INFO:tasks.workunit.client.0.smithi102.stdout:5/17: stat f0 0 2024-03-26T18:54:27.198 INFO:tasks.workunit.client.0.smithi102.stdout:5/18: fdatasync d2/f4 0 2024-03-26T18:54:27.202 INFO:tasks.workunit.client.1.smithi190.stdout:1/189: symlink d3/da/l3f 0 2024-03-26T18:54:27.203 INFO:tasks.workunit.client.0.smithi102.stdout:5/19: mknod d2/c5 0 2024-03-26T18:54:27.204 INFO:tasks.workunit.client.0.smithi102.stdout:5/20: write d2/f4 [1650808,86690] 0 2024-03-26T18:54:27.205 INFO:tasks.workunit.client.1.smithi190.stdout:1/190: mknod d3/d9/c40 0 2024-03-26T18:54:27.205 INFO:tasks.workunit.client.1.smithi190.stdout:1/191: creat d3/d9/f41 x:0 0 0 2024-03-26T18:54:27.205 INFO:tasks.workunit.client.1.smithi190.stdout:1/192: creat d3/d9/f42 x:0 0 0 2024-03-26T18:54:27.205 INFO:tasks.workunit.client.1.smithi190.stdout:1/193: chown d3/da/f37 375431 1 2024-03-26T18:54:27.205 INFO:tasks.workunit.client.1.smithi190.stdout:1/194: fdatasync d3/d9/f15 0 2024-03-26T18:54:27.205 INFO:tasks.workunit.client.1.smithi190.stdout:1/195: fsync d3/da/d2e/f30 0 2024-03-26T18:54:27.206 INFO:tasks.workunit.client.0.smithi102.stdout:5/21: mkdir d2/d6 0 2024-03-26T18:54:27.209 INFO:tasks.workunit.client.1.smithi190.stdout:1/196: symlink d3/da/l43 0 2024-03-26T18:54:27.210 INFO:tasks.workunit.client.1.smithi190.stdout:1/197: mknod d3/da/dd/d2f/d33/c44 0 2024-03-26T18:54:27.211 INFO:tasks.workunit.client.1.smithi190.stdout:1/198: mkdir d3/da/d45 0 2024-03-26T18:54:27.223 INFO:tasks.workunit.client.0.smithi102.stdout:6/6: dwrite f0 [0,4194304] 0 2024-03-26T18:54:27.224 INFO:tasks.workunit.client.0.smithi102.stdout:6/7: creat f1 x:0 0 0 2024-03-26T18:54:27.224 INFO:tasks.workunit.client.0.smithi102.stdout:6/8: dread - f1 zero size 2024-03-26T18:54:27.224 INFO:tasks.workunit.client.0.smithi102.stdout:6/9: rmdir - no directory 2024-03-26T18:54:27.224 INFO:tasks.workunit.client.0.smithi102.stdout:6/10: write f1 [766744,124221] 0 2024-03-26T18:54:27.244 INFO:tasks.workunit.client.0.smithi102.stdout:6/11: dread f0 [0,4194304] 0 2024-03-26T18:54:27.247 INFO:tasks.workunit.client.0.smithi102.stdout:2/20: dwrite d3/f4 [0,4194304] 0 2024-03-26T18:54:27.247 INFO:tasks.workunit.client.0.smithi102.stdout:3/15: dwrite f0 [0,4194304] 0 2024-03-26T18:54:27.248 INFO:tasks.workunit.client.0.smithi102.stdout:4/17: dwrite d1/f4 [0,4194304] 0 2024-03-26T18:54:27.248 INFO:tasks.workunit.client.0.smithi102.stdout:9/25: dwrite f4 [0,4194304] 0 2024-03-26T18:54:27.249 INFO:tasks.workunit.client.1.smithi190.stdout:0/325: dwrite d0/dd/d20/d1c/d46/f72 [0,4194304] 0 2024-03-26T18:54:27.249 INFO:tasks.workunit.client.0.smithi102.stdout:5/22: dwrite f0 [0,4194304] 0 2024-03-26T18:54:27.250 INFO:tasks.workunit.client.0.smithi102.stdout:0/15: dwrite f3 [0,4194304] 0 2024-03-26T18:54:27.250 INFO:tasks.workunit.client.0.smithi102.stdout:3/16: mknod c6 0 2024-03-26T18:54:27.250 INFO:tasks.workunit.client.0.smithi102.stdout:8/24: dwrite d0/d3/f4 [0,4194304] 0 2024-03-26T18:54:27.251 INFO:tasks.workunit.client.0.smithi102.stdout:3/17: creat f7 x:0 0 0 2024-03-26T18:54:27.252 INFO:tasks.workunit.client.1.smithi190.stdout:1/199: dwrite d3/d9/f20 [0,4194304] 0 2024-03-26T18:54:27.253 INFO:tasks.workunit.client.0.smithi102.stdout:0/16: dread f2 [0,4194304] 0 2024-03-26T18:54:27.258 INFO:tasks.workunit.client.0.smithi102.stdout:5/23: creat d2/f7 x:0 0 0 2024-03-26T18:54:27.264 INFO:tasks.workunit.client.0.smithi102.stdout:3/18: rename c1 to c8 0 2024-03-26T18:54:27.265 INFO:tasks.workunit.client.0.smithi102.stdout:0/17: mkdir d6 0 2024-03-26T18:54:27.267 INFO:tasks.workunit.client.0.smithi102.stdout:5/24: mknod d2/c8 0 2024-03-26T18:54:27.267 INFO:tasks.workunit.client.0.smithi102.stdout:2/21: write d3/f4 [23875,72194] 0 2024-03-26T18:54:27.267 INFO:tasks.workunit.client.0.smithi102.stdout:3/19: symlink l9 0 2024-03-26T18:54:27.267 INFO:tasks.workunit.client.0.smithi102.stdout:3/20: creat fa x:0 0 0 2024-03-26T18:54:27.268 INFO:tasks.workunit.client.0.smithi102.stdout:3/21: chown l9 19579134 1 2024-03-26T18:54:27.268 INFO:tasks.workunit.client.0.smithi102.stdout:3/22: creat fb x:0 0 0 2024-03-26T18:54:27.268 INFO:tasks.workunit.client.0.smithi102.stdout:4/18: dread d1/f4 [0,4194304] 0 2024-03-26T18:54:27.269 INFO:tasks.workunit.client.0.smithi102.stdout:0/18: unlink f3 0 2024-03-26T18:54:27.270 INFO:tasks.workunit.client.0.smithi102.stdout:0/19: readlink - no filename 2024-03-26T18:54:27.272 INFO:tasks.workunit.client.0.smithi102.stdout:0/20: dread f2 [0,4194304] 0 2024-03-26T18:54:27.272 INFO:tasks.workunit.client.0.smithi102.stdout:2/22: mkdir d3/d6 0 2024-03-26T18:54:27.272 INFO:tasks.workunit.client.0.smithi102.stdout:2/23: truncate d3/f4 5031596 0 2024-03-26T18:54:27.272 INFO:tasks.workunit.client.0.smithi102.stdout:4/19: write d1/f4 [3438180,117456] 0 2024-03-26T18:54:27.272 INFO:tasks.workunit.client.0.smithi102.stdout:4/20: write d1/f4 [355117,16266] 0 2024-03-26T18:54:27.274 INFO:tasks.workunit.client.0.smithi102.stdout:2/24: write d3/f4 [4527739,108432] 0 2024-03-26T18:54:27.274 INFO:tasks.workunit.client.0.smithi102.stdout:0/21: creat d6/f7 x:0 0 0 2024-03-26T18:54:27.274 INFO:tasks.workunit.client.0.smithi102.stdout:0/22: creat d6/f8 x:0 0 0 2024-03-26T18:54:27.276 INFO:tasks.workunit.client.0.smithi102.stdout:4/21: mknod d1/c5 0 2024-03-26T18:54:27.276 INFO:tasks.workunit.client.0.smithi102.stdout:4/22: read d1/f4 [154141,22508] 0 2024-03-26T18:54:27.285 INFO:tasks.workunit.client.0.smithi102.stdout:5/25: write d2/f4 [644523,42063] 0 2024-03-26T18:54:27.285 INFO:tasks.workunit.client.0.smithi102.stdout:5/26: readlink d2/l3 0 2024-03-26T18:54:27.287 INFO:tasks.workunit.client.0.smithi102.stdout:6/12: dwrite f0 [0,4194304] 0 2024-03-26T18:54:27.287 INFO:tasks.workunit.client.0.smithi102.stdout:5/27: rename f0 to d2/f9 0 2024-03-26T18:54:27.288 INFO:tasks.workunit.client.1.smithi190.stdout:0/326: dwrite d0/dd/d26/f3f [0,4194304] 0 2024-03-26T18:54:27.288 INFO:tasks.workunit.client.0.smithi102.stdout:6/13: read f1 [158668,124466] 0 2024-03-26T18:54:27.290 INFO:tasks.workunit.client.1.smithi190.stdout:1/200: dwrite d3/d9/db/f32 [0,4194304] 0 2024-03-26T18:54:27.294 INFO:tasks.workunit.client.0.smithi102.stdout:5/28: rmdir d2/d6 0 2024-03-26T18:54:27.297 INFO:tasks.workunit.client.0.smithi102.stdout:3/23: dwrite f4 [0,4194304] 0 2024-03-26T18:54:27.297 INFO:tasks.workunit.client.0.smithi102.stdout:3/24: truncate f3 371749 0 2024-03-26T18:54:27.297 INFO:tasks.workunit.client.0.smithi102.stdout:6/14: mkdir d2 0 2024-03-26T18:54:27.298 INFO:tasks.workunit.client.1.smithi190.stdout:0/327: dread d0/f1f [0,4194304] 0 2024-03-26T18:54:27.298 INFO:tasks.workunit.client.1.smithi190.stdout:0/328: chown d0/dd/d24/d4b/l69 139 1 2024-03-26T18:54:27.298 INFO:tasks.workunit.client.1.smithi190.stdout:0/329: truncate d0/f6 1778264 0 2024-03-26T18:54:27.302 INFO:tasks.workunit.client.0.smithi102.stdout:3/25: rename f0 to fc 0 2024-03-26T18:54:27.302 INFO:tasks.workunit.client.0.smithi102.stdout:3/26: creat fd x:0 0 0 2024-03-26T18:54:27.302 INFO:tasks.workunit.client.0.smithi102.stdout:3/27: creat fe x:0 0 0 2024-03-26T18:54:27.306 INFO:tasks.workunit.client.1.smithi190.stdout:0/330: truncate d0/dd/d26/f34 844940 0 2024-03-26T18:54:27.306 INFO:tasks.workunit.client.1.smithi190.stdout:1/201: write d3/d9/db/f2c [948673,18543] 0 2024-03-26T18:54:27.308 INFO:tasks.workunit.client.0.smithi102.stdout:4/23: dwrite d1/f4 [0,4194304] 0 2024-03-26T18:54:27.311 INFO:tasks.workunit.client.0.smithi102.stdout:3/28: mkdir df 0 2024-03-26T18:54:27.312 INFO:tasks.workunit.client.1.smithi190.stdout:0/331: symlink d0/dd/d24/d4b/d5e/d66/l79 0 2024-03-26T18:54:27.313 INFO:tasks.workunit.client.0.smithi102.stdout:4/24: unlink d1/c3 0 2024-03-26T18:54:27.316 INFO:tasks.workunit.client.0.smithi102.stdout:3/29: mkdir df/d10 0 2024-03-26T18:54:27.316 INFO:tasks.workunit.client.0.smithi102.stdout:3/30: chown df/d10 608 1 2024-03-26T18:54:27.318 INFO:tasks.workunit.client.1.smithi190.stdout:0/332: link d0/dd/d20/d1c/f6e d0/dd/d24/f7a 0 2024-03-26T18:54:27.319 INFO:tasks.workunit.client.1.smithi190.stdout:1/202: read d3/d9/db/f32 [1050559,82607] 0 2024-03-26T18:54:27.319 INFO:tasks.workunit.client.1.smithi190.stdout:1/203: chown d3/da/dd/d2f/d34 3565816 1 2024-03-26T18:54:27.320 INFO:tasks.workunit.client.1.smithi190.stdout:0/333: link d0/f13 d0/dd/d20/d61/f7b 0 2024-03-26T18:54:27.321 INFO:tasks.workunit.client.0.smithi102.stdout:8/25: write d0/d3/f4 [5188830,95428] 0 2024-03-26T18:54:27.322 INFO:tasks.workunit.client.1.smithi190.stdout:1/204: creat d3/da/dd/d2f/d34/d3e/f46 x:0 0 0 2024-03-26T18:54:27.323 INFO:tasks.workunit.client.1.smithi190.stdout:1/205: dread - d3/da/f37 zero size 2024-03-26T18:54:27.325 INFO:tasks.workunit.client.0.smithi102.stdout:6/15: dwrite f1 [0,4194304] 0 2024-03-26T18:54:27.325 INFO:tasks.workunit.client.0.smithi102.stdout:3/31: dread f4 [0,4194304] 0 2024-03-26T18:54:27.325 INFO:tasks.workunit.client.0.smithi102.stdout:5/29: dwrite f1 [0,4194304] 0 2024-03-26T18:54:27.326 INFO:tasks.workunit.client.1.smithi190.stdout:0/334: mkdir d0/dd/d20/d1c/d46/d7c 0 2024-03-26T18:54:27.326 INFO:tasks.workunit.client.1.smithi190.stdout:0/335: creat d0/dd/d24/f7d x:0 0 0 2024-03-26T18:54:27.326 INFO:tasks.workunit.client.1.smithi190.stdout:0/336: creat d0/dd/d20/d1c/d4e/f7e x:0 0 0 2024-03-26T18:54:27.327 INFO:tasks.workunit.client.0.smithi102.stdout:3/32: dread f3 [0,4194304] 0 2024-03-26T18:54:27.327 INFO:tasks.workunit.client.0.smithi102.stdout:3/33: chown f4 1275 1 2024-03-26T18:54:27.327 INFO:tasks.workunit.client.0.smithi102.stdout:3/34: write fb [616189,108613] 0 2024-03-26T18:54:27.330 INFO:tasks.workunit.client.1.smithi190.stdout:1/206: dread f2 [4194304,4194304] 0 2024-03-26T18:54:27.331 INFO:tasks.workunit.client.0.smithi102.stdout:3/35: unlink c8 0 2024-03-26T18:54:27.331 INFO:tasks.workunit.client.0.smithi102.stdout:3/36: write fe [704075,75257] 0 2024-03-26T18:54:27.335 INFO:tasks.workunit.client.0.smithi102.stdout:3/37: symlink df/l11 0 2024-03-26T18:54:27.336 INFO:tasks.workunit.client.0.smithi102.stdout:2/25: rmdir d3 39 2024-03-26T18:54:27.337 INFO:tasks.workunit.client.0.smithi102.stdout:8/26: dread d0/d3/f4 [0,4194304] 0 2024-03-26T18:54:27.338 INFO:tasks.workunit.client.0.smithi102.stdout:5/30: dread d2/f4 [0,4194304] 0 2024-03-26T18:54:27.338 INFO:tasks.workunit.client.0.smithi102.stdout:2/26: creat d3/f7 x:0 0 0 2024-03-26T18:54:27.340 INFO:tasks.workunit.client.0.smithi102.stdout:5/31: link d2/f4 d2/fa 0 2024-03-26T18:54:27.341 INFO:tasks.workunit.client.0.smithi102.stdout:5/32: creat d2/fb x:0 0 0 2024-03-26T18:54:27.341 INFO:tasks.workunit.client.0.smithi102.stdout:5/33: dread - d2/fb zero size 2024-03-26T18:54:27.343 INFO:tasks.workunit.client.0.smithi102.stdout:2/27: link d3/f5 d3/f8 0 2024-03-26T18:54:27.343 INFO:tasks.workunit.client.0.smithi102.stdout:2/28: write d3/f8 [1476787,5225] 0 2024-03-26T18:54:27.343 INFO:tasks.workunit.client.0.smithi102.stdout:2/29: write d3/f7 [747449,76079] 0 2024-03-26T18:54:27.343 INFO:tasks.workunit.client.0.smithi102.stdout:2/30: creat d3/f9 x:0 0 0 2024-03-26T18:54:27.343 INFO:tasks.workunit.client.0.smithi102.stdout:8/27: getdents d0/d3 0 2024-03-26T18:54:27.345 INFO:tasks.workunit.client.0.smithi102.stdout:0/23: fsync d6/f8 0 2024-03-26T18:54:27.347 INFO:tasks.workunit.client.0.smithi102.stdout:6/16: dwrite f0 [0,4194304] 0 2024-03-26T18:54:27.347 INFO:tasks.workunit.client.0.smithi102.stdout:6/17: chown f1 906195470 1 2024-03-26T18:54:27.348 INFO:tasks.workunit.client.1.smithi190.stdout:0/337: dwrite d0/dd/d20/d1c/f6e [0,4194304] 0 2024-03-26T18:54:27.348 INFO:tasks.workunit.client.1.smithi190.stdout:0/338: stat d0/dd/d24/d4b/l69 0 2024-03-26T18:54:27.348 INFO:tasks.workunit.client.1.smithi190.stdout:1/207: fsync d3/d9/db/f2c 0 2024-03-26T18:54:27.348 INFO:tasks.workunit.client.1.smithi190.stdout:1/208: write d3/da/d2e/f30 [1042957,121416] 0 2024-03-26T18:54:27.350 INFO:tasks.workunit.client.0.smithi102.stdout:2/31: creat d3/d6/fa x:0 0 0 2024-03-26T18:54:27.350 INFO:tasks.workunit.client.0.smithi102.stdout:2/32: truncate d3/f7 1014569 0 2024-03-26T18:54:27.350 INFO:tasks.workunit.client.0.smithi102.stdout:5/34: fsync d2/f9 0 2024-03-26T18:54:27.350 INFO:tasks.workunit.client.0.smithi102.stdout:5/35: write d2/f9 [4945733,123185] 0 2024-03-26T18:54:27.350 INFO:tasks.workunit.client.0.smithi102.stdout:5/36: readlink d2/l3 0 2024-03-26T18:54:27.350 INFO:tasks.workunit.client.0.smithi102.stdout:5/37: write d2/f7 [305118,112959] 0 2024-03-26T18:54:27.350 INFO:tasks.workunit.client.0.smithi102.stdout:5/38: chown d2/f9 50085647 1 2024-03-26T18:54:27.350 INFO:tasks.workunit.client.0.smithi102.stdout:8/28: dread d0/d3/f4 [4194304,4194304] 0 2024-03-26T18:54:27.350 INFO:tasks.workunit.client.0.smithi102.stdout:8/29: chown d0/c1 104171 1 2024-03-26T18:54:27.350 INFO:tasks.workunit.client.0.smithi102.stdout:8/30: chown d0/c1 51 1 2024-03-26T18:54:27.351 INFO:tasks.workunit.client.0.smithi102.stdout:6/18: mkdir d2/d3 0 2024-03-26T18:54:27.351 INFO:tasks.workunit.client.0.smithi102.stdout:6/19: getdents d2/d3 0 2024-03-26T18:54:27.351 INFO:tasks.workunit.client.0.smithi102.stdout:5/39: symlink d2/lc 0 2024-03-26T18:54:27.353 INFO:tasks.workunit.client.0.smithi102.stdout:2/33: dread d3/f7 [0,4194304] 0 2024-03-26T18:54:27.354 INFO:tasks.workunit.client.0.smithi102.stdout:8/31: mknod d0/c5 0 2024-03-26T18:54:27.357 INFO:tasks.workunit.client.0.smithi102.stdout:2/34: link d3/f8 d3/fb 0 2024-03-26T18:54:27.357 INFO:tasks.workunit.client.0.smithi102.stdout:2/35: readlink - no filename 2024-03-26T18:54:27.358 INFO:tasks.workunit.client.0.smithi102.stdout:6/20: dread f1 [0,4194304] 0 2024-03-26T18:54:27.359 INFO:tasks.workunit.client.0.smithi102.stdout:5/40: dread d2/fa [0,4194304] 0 2024-03-26T18:54:27.359 INFO:tasks.workunit.client.0.smithi102.stdout:5/41: creat d2/fd x:0 0 0 2024-03-26T18:54:27.359 INFO:tasks.workunit.client.0.smithi102.stdout:5/42: truncate d2/fd 806359 0 2024-03-26T18:54:27.360 INFO:tasks.workunit.client.0.smithi102.stdout:8/32: rename d0/d3/f4 to d0/d3/f6 0 2024-03-26T18:54:27.361 INFO:tasks.workunit.client.1.smithi190.stdout:1/209: truncate d3/da/dd/d2f/d34/f35 442838 0 2024-03-26T18:54:27.368 INFO:tasks.workunit.client.0.smithi102.stdout:0/24: dwrite d6/f7 [0,4194304] 0 2024-03-26T18:54:27.370 INFO:tasks.workunit.client.0.smithi102.stdout:4/25: rmdir d1 39 2024-03-26T18:54:27.372 INFO:tasks.workunit.client.0.smithi102.stdout:8/33: rename d0/l2 to d0/d3/l7 0 2024-03-26T18:54:27.375 INFO:tasks.workunit.client.0.smithi102.stdout:2/36: dread d3/f8 [0,4194304] 0 2024-03-26T18:54:27.376 INFO:tasks.workunit.client.0.smithi102.stdout:6/21: dwrite f0 [0,4194304] 0 2024-03-26T18:54:27.379 INFO:tasks.workunit.client.0.smithi102.stdout:2/37: dread d3/f8 [0,4194304] 0 2024-03-26T18:54:27.380 INFO:tasks.workunit.client.1.smithi190.stdout:1/210: rename d3/d9/db/f19 to d3/da/f47 0 2024-03-26T18:54:27.380 INFO:tasks.workunit.client.0.smithi102.stdout:3/38: getdents df 0 2024-03-26T18:54:27.381 INFO:tasks.workunit.client.0.smithi102.stdout:4/26: rename d1/l2 to d1/l6 0 2024-03-26T18:54:27.383 INFO:tasks.workunit.client.1.smithi190.stdout:0/339: unlink d0/dd/d26/f34 0 2024-03-26T18:54:27.385 INFO:tasks.workunit.client.0.smithi102.stdout:0/25: rename f2 to d6/f9 0 2024-03-26T18:54:27.386 INFO:tasks.workunit.client.1.smithi190.stdout:8/248: sync 2024-03-26T18:54:27.387 INFO:tasks.workunit.client.1.smithi190.stdout:1/211: creat d3/da/d45/f48 x:0 0 0 2024-03-26T18:54:27.387 INFO:tasks.workunit.client.1.smithi190.stdout:1/212: write d3/d9/f41 [989934,95293] 0 2024-03-26T18:54:27.387 INFO:tasks.workunit.client.1.smithi190.stdout:1/213: dread - d3/da/f37 zero size 2024-03-26T18:54:27.389 INFO:tasks.workunit.client.0.smithi102.stdout:6/22: dread f0 [0,4194304] 0 2024-03-26T18:54:27.389 INFO:tasks.workunit.client.0.smithi102.stdout:6/23: rename d2 to d2/d3/d4 22 2024-03-26T18:54:27.390 INFO:tasks.workunit.client.0.smithi102.stdout:8/34: symlink d0/d3/l8 0 2024-03-26T18:54:27.390 INFO:tasks.workunit.client.0.smithi102.stdout:8/35: stat d0/d3 0 2024-03-26T18:54:27.390 INFO:tasks.workunit.client.0.smithi102.stdout:8/36: write d0/d3/f6 [5408954,94093] 0 2024-03-26T18:54:27.393 INFO:tasks.workunit.client.0.smithi102.stdout:3/39: creat df/f12 x:0 0 0 2024-03-26T18:54:27.393 INFO:tasks.workunit.client.0.smithi102.stdout:3/40: dread - f7 zero size 2024-03-26T18:54:27.399 INFO:tasks.workunit.client.0.smithi102.stdout:0/26: mknod d6/ca 0 2024-03-26T18:54:27.401 INFO:tasks.workunit.client.1.smithi190.stdout:8/249: dread d9/f12 [4194304,4194304] 0 2024-03-26T18:54:27.403 INFO:tasks.workunit.client.0.smithi102.stdout:3/41: write fc [3476077,66194] 0 2024-03-26T18:54:27.404 INFO:tasks.workunit.client.0.smithi102.stdout:6/24: write f1 [5042063,31280] 0 2024-03-26T18:54:27.404 INFO:tasks.workunit.client.0.smithi102.stdout:6/25: chown d2/d3 6 1 2024-03-26T18:54:27.405 INFO:tasks.workunit.client.1.smithi190.stdout:8/250: rmdir d9/dd/d23/d36 39 2024-03-26T18:54:27.405 INFO:tasks.workunit.client.0.smithi102.stdout:6/26: read f1 [4992158,70174] 0 2024-03-26T18:54:27.405 INFO:tasks.workunit.client.0.smithi102.stdout:6/27: chown f1 427647913 1 2024-03-26T18:54:27.406 INFO:tasks.workunit.client.0.smithi102.stdout:3/42: getdents df/d10 0 2024-03-26T18:54:27.407 INFO:tasks.workunit.client.0.smithi102.stdout:3/43: chown fd 3 1 2024-03-26T18:54:27.407 INFO:tasks.workunit.client.1.smithi190.stdout:1/214: dwrite d3/da/f47 [4194304,4194304] 0 2024-03-26T18:54:27.407 INFO:tasks.workunit.client.1.smithi190.stdout:1/215: chown d3/d9/f42 131638 1 2024-03-26T18:54:27.409 INFO:tasks.workunit.client.0.smithi102.stdout:6/28: unlink f1 0 2024-03-26T18:54:27.409 INFO:tasks.workunit.client.0.smithi102.stdout:6/29: fdatasync f0 0 2024-03-26T18:54:27.410 INFO:tasks.workunit.client.0.smithi102.stdout:0/27: write d6/f7 [3625748,31677] 0 2024-03-26T18:54:27.410 INFO:tasks.workunit.client.0.smithi102.stdout:5/43: getdents d2 0 2024-03-26T18:54:27.412 INFO:tasks.workunit.client.0.smithi102.stdout:0/28: dread d6/f9 [0,4194304] 0 2024-03-26T18:54:27.414 INFO:tasks.workunit.client.0.smithi102.stdout:6/30: creat d2/d3/f5 x:0 0 0 2024-03-26T18:54:27.415 INFO:tasks.workunit.client.1.smithi190.stdout:8/251: dread f6 [0,4194304] 0 2024-03-26T18:54:27.419 INFO:tasks.workunit.client.0.smithi102.stdout:5/44: link d2/f4 d2/fe 0 2024-03-26T18:54:27.421 INFO:tasks.workunit.client.0.smithi102.stdout:0/29: rename d6/f8 to d6/fb 0 2024-03-26T18:54:27.421 INFO:tasks.workunit.client.0.smithi102.stdout:0/30: creat d6/fc x:0 0 0 2024-03-26T18:54:27.421 INFO:tasks.workunit.client.0.smithi102.stdout:0/31: creat d6/fd x:0 0 0 2024-03-26T18:54:27.422 INFO:tasks.workunit.client.1.smithi190.stdout:5/155: sync 2024-03-26T18:54:27.422 INFO:tasks.workunit.client.1.smithi190.stdout:6/207: sync 2024-03-26T18:54:27.422 INFO:tasks.workunit.client.1.smithi190.stdout:4/252: sync 2024-03-26T18:54:27.422 INFO:tasks.workunit.client.1.smithi190.stdout:3/200: sync 2024-03-26T18:54:27.422 INFO:tasks.workunit.client.1.smithi190.stdout:7/253: sync 2024-03-26T18:54:27.422 INFO:tasks.workunit.client.1.smithi190.stdout:9/486: sync 2024-03-26T18:54:27.422 INFO:tasks.workunit.client.1.smithi190.stdout:9/487: chown d12/d2d/f9f 15 1 2024-03-26T18:54:27.422 INFO:tasks.workunit.client.1.smithi190.stdout:9/488: read - d12/d2d/d1d/d23/d2b/fab zero size 2024-03-26T18:54:27.422 INFO:tasks.workunit.client.1.smithi190.stdout:2/279: sync 2024-03-26T18:54:27.423 INFO:tasks.workunit.client.1.smithi190.stdout:8/252: unlink d9/d39/c3f 0 2024-03-26T18:54:27.426 INFO:tasks.workunit.client.1.smithi190.stdout:3/201: unlink d1/cd 0 2024-03-26T18:54:27.426 INFO:tasks.workunit.client.1.smithi190.stdout:3/202: fdatasync d1/d3/df/d18/d35/f39 0 2024-03-26T18:54:27.427 INFO:tasks.workunit.client.0.smithi102.stdout:5/45: stat d2/f9 0 2024-03-26T18:54:27.427 INFO:tasks.workunit.client.0.smithi102.stdout:5/46: fdatasync d2/fe 0 2024-03-26T18:54:27.431 INFO:tasks.workunit.client.1.smithi190.stdout:6/208: rename d17/d28/d2c/f2e to d17/d28/d2c/f4c 0 2024-03-26T18:54:27.432 INFO:tasks.workunit.client.1.smithi190.stdout:6/209: getdents d17/d28/d46 0 2024-03-26T18:54:27.435 INFO:tasks.workunit.client.1.smithi190.stdout:7/254: creat d3/d6/d9/d14/d59/f61 x:0 0 0 2024-03-26T18:54:27.437 INFO:tasks.workunit.client.1.smithi190.stdout:5/156: symlink d2/d17/d1b/d30/l34 0 2024-03-26T18:54:27.437 INFO:tasks.workunit.client.1.smithi190.stdout:5/157: chown d2/d17/f21 8322601 1 2024-03-26T18:54:27.437 INFO:tasks.workunit.client.1.smithi190.stdout:5/158: readlink d2/d17/l1c 0 2024-03-26T18:54:27.438 INFO:tasks.workunit.client.1.smithi190.stdout:5/159: fsync d2/d17/d1b/d1e/f2e 0 2024-03-26T18:54:27.439 INFO:tasks.workunit.client.0.smithi102.stdout:0/32: mkdir d6/de 0 2024-03-26T18:54:27.439 INFO:tasks.workunit.client.0.smithi102.stdout:0/33: truncate d6/fb 1037087 0 2024-03-26T18:54:27.439 INFO:tasks.workunit.client.0.smithi102.stdout:0/34: chown d6/de 8715 1 2024-03-26T18:54:27.439 INFO:tasks.workunit.client.1.smithi190.stdout:1/216: dwrite d3/da/dd/d2f/d34/d3e/f46 [0,4194304] 0 2024-03-26T18:54:27.440 INFO:tasks.workunit.client.0.smithi102.stdout:0/35: read d6/fb [555858,49098] 0 2024-03-26T18:54:27.440 INFO:tasks.workunit.client.1.smithi190.stdout:2/280: symlink d6/d9/d3a/d3d/d1c/d44/l60 0 2024-03-26T18:54:27.441 INFO:tasks.workunit.client.0.smithi102.stdout:6/31: truncate f0 2398784 0 2024-03-26T18:54:27.443 INFO:tasks.workunit.client.0.smithi102.stdout:2/38: truncate d3/fb 4179271 0 2024-03-26T18:54:27.444 INFO:tasks.workunit.client.1.smithi190.stdout:2/281: dread d6/d9/d10/f2d [0,4194304] 0 2024-03-26T18:54:27.445 INFO:tasks.workunit.client.1.smithi190.stdout:8/253: link d9/f1a d9/dd/d15/d32/f4f 0 2024-03-26T18:54:27.449 INFO:tasks.workunit.client.0.smithi102.stdout:0/36: truncate d6/f9 50937 0 2024-03-26T18:54:27.449 INFO:tasks.workunit.client.0.smithi102.stdout:0/37: dread - d6/fc zero size 2024-03-26T18:54:27.450 INFO:tasks.workunit.client.1.smithi190.stdout:6/210: mkdir d17/d28/d46/d4d 0 2024-03-26T18:54:27.450 INFO:tasks.workunit.client.1.smithi190.stdout:5/160: truncate d2/f12 538026 0 2024-03-26T18:54:27.451 INFO:tasks.workunit.client.1.smithi190.stdout:6/211: dread d17/d28/d2b/f3c [0,4194304] 0 2024-03-26T18:54:27.451 INFO:tasks.workunit.client.1.smithi190.stdout:6/212: stat c14 0 2024-03-26T18:54:27.452 INFO:tasks.workunit.client.0.smithi102.stdout:6/32: symlink d2/d3/l6 0 2024-03-26T18:54:27.454 INFO:tasks.workunit.client.0.smithi102.stdout:5/47: rename f1 to d2/ff 0 2024-03-26T18:54:27.455 INFO:tasks.workunit.client.0.smithi102.stdout:0/38: creat d6/de/ff x:0 0 0 2024-03-26T18:54:27.456 INFO:tasks.workunit.client.1.smithi190.stdout:1/217: dread d3/da/f1f [0,4194304] 0 2024-03-26T18:54:27.456 INFO:tasks.workunit.client.1.smithi190.stdout:1/218: creat d3/da/d45/f49 x:0 0 0 2024-03-26T18:54:27.456 INFO:tasks.workunit.client.1.smithi190.stdout:1/219: chown d3/da/dd/d2f/c31 9346 1 2024-03-26T18:54:27.457 INFO:tasks.workunit.client.0.smithi102.stdout:5/48: truncate d2/f9 3926556 0 2024-03-26T18:54:27.458 INFO:tasks.workunit.client.1.smithi190.stdout:6/213: creat d17/f4e x:0 0 0 2024-03-26T18:54:27.459 INFO:tasks.workunit.client.0.smithi102.stdout:5/49: stat d2/fd 0 2024-03-26T18:54:27.460 INFO:tasks.workunit.client.0.smithi102.stdout:5/50: creat d2/f10 x:0 0 0 2024-03-26T18:54:27.461 INFO:tasks.workunit.client.0.smithi102.stdout:5/51: creat d2/f11 x:0 0 0 2024-03-26T18:54:27.462 INFO:tasks.workunit.client.1.smithi190.stdout:8/254: dread d9/f14 [0,4194304] 0 2024-03-26T18:54:27.464 INFO:tasks.workunit.client.0.smithi102.stdout:5/52: rename d2/fb to d2/f12 0 2024-03-26T18:54:27.464 INFO:tasks.workunit.client.0.smithi102.stdout:3/44: dwrite fa [0,4194304] 0 2024-03-26T18:54:27.467 INFO:tasks.workunit.client.0.smithi102.stdout:5/53: symlink d2/l13 0 2024-03-26T18:54:27.467 INFO:tasks.workunit.client.0.smithi102.stdout:5/54: truncate d2/f11 202469 0 2024-03-26T18:54:27.468 INFO:tasks.workunit.client.1.smithi190.stdout:8/255: dread d9/d16/f1e [0,4194304] 0 2024-03-26T18:54:27.468 INFO:tasks.workunit.client.0.smithi102.stdout:3/45: mknod df/c13 0 2024-03-26T18:54:27.469 INFO:tasks.workunit.client.1.smithi190.stdout:9/489: dwrite d12/d2d/d1d/d23/d2a/f73 [0,4194304] 0 2024-03-26T18:54:27.469 INFO:tasks.workunit.client.1.smithi190.stdout:4/253: dwrite d7/d8/d14/f57 [0,4194304] 0 2024-03-26T18:54:27.470 INFO:tasks.workunit.client.1.smithi190.stdout:7/255: dwrite d3/d6/d9/d1f/d4c/f54 [0,4194304] 0 2024-03-26T18:54:27.470 INFO:tasks.workunit.client.0.smithi102.stdout:5/55: mkdir d2/d14 0 2024-03-26T18:54:27.472 INFO:tasks.workunit.client.0.smithi102.stdout:3/46: symlink df/l14 0 2024-03-26T18:54:27.472 INFO:tasks.workunit.client.0.smithi102.stdout:3/47: truncate f7 397319 0 2024-03-26T18:54:27.475 INFO:tasks.workunit.client.1.smithi190.stdout:6/214: dread d17/f23 [0,4194304] 0 2024-03-26T18:54:27.475 INFO:tasks.workunit.client.1.smithi190.stdout:3/203: dwrite d1/de/d1c/f2b [0,4194304] 0 2024-03-26T18:54:27.478 INFO:tasks.workunit.client.1.smithi190.stdout:1/220: dwrite d3/d9/f16 [4194304,4194304] 0 2024-03-26T18:54:27.478 INFO:tasks.workunit.client.1.smithi190.stdout:1/221: creat d3/da/d2e/f4a x:0 0 0 2024-03-26T18:54:27.481 INFO:tasks.workunit.client.1.smithi190.stdout:7/256: rename d3/d6/f39 to d3/d6/d9/d1f/d41/d50/d53/f62 0 2024-03-26T18:54:27.483 INFO:tasks.workunit.client.0.smithi102.stdout:3/48: rename df/c13 to df/c15 0 2024-03-26T18:54:27.485 INFO:tasks.workunit.client.0.smithi102.stdout:6/33: fsync f0 0 2024-03-26T18:54:27.487 INFO:tasks.workunit.client.1.smithi190.stdout:9/490: dread ff [0,4194304] 0 2024-03-26T18:54:27.487 INFO:tasks.workunit.client.1.smithi190.stdout:9/491: write d12/d2d/d1d/d23/d2a/f6c [95846,55899] 0 2024-03-26T18:54:27.490 INFO:tasks.workunit.client.0.smithi102.stdout:6/34: getdents d2 0 2024-03-26T18:54:27.493 INFO:tasks.workunit.client.1.smithi190.stdout:3/204: write d1/de/f29 [2784238,114406] 0 2024-03-26T18:54:27.493 INFO:tasks.workunit.client.1.smithi190.stdout:3/205: write d1/d3/df/f2a [115297,23737] 0 2024-03-26T18:54:27.495 INFO:tasks.workunit.client.1.smithi190.stdout:5/161: write d2/f12 [1018736,36782] 0 2024-03-26T18:54:27.496 INFO:tasks.workunit.client.1.smithi190.stdout:5/162: chown d2/d17/d1b/d22/l32 4820 1 2024-03-26T18:54:27.499 INFO:tasks.workunit.client.1.smithi190.stdout:7/257: creat d3/d6/d9/d14/f63 x:0 0 0 2024-03-26T18:54:27.502 INFO:tasks.workunit.client.0.smithi102.stdout:3/49: dread fa [0,4194304] 0 2024-03-26T18:54:27.504 INFO:tasks.workunit.client.1.smithi190.stdout:4/254: dwrite d7/d8/d1c/d20/f3d [4194304,4194304] 0 2024-03-26T18:54:27.504 INFO:tasks.workunit.client.1.smithi190.stdout:4/255: chown d7/d8/f22 13458 1 2024-03-26T18:54:27.504 INFO:tasks.workunit.client.1.smithi190.stdout:6/215: dwrite d17/f19 [4194304,4194304] 0 2024-03-26T18:54:27.507 INFO:tasks.workunit.client.1.smithi190.stdout:5/163: rename d2/d17/d1b/c23 to d2/d17/d1b/d22/c35 0 2024-03-26T18:54:27.508 INFO:tasks.workunit.client.0.smithi102.stdout:0/39: rmdir d6/de 39 2024-03-26T18:54:27.510 INFO:tasks.workunit.client.1.smithi190.stdout:7/258: symlink d3/d1b/l64 0 2024-03-26T18:54:27.510 INFO:tasks.workunit.client.1.smithi190.stdout:7/259: write d3/d1b/f2c [989876,26526] 0 2024-03-26T18:54:27.511 INFO:tasks.workunit.client.0.smithi102.stdout:3/50: creat df/f16 x:0 0 0 2024-03-26T18:54:27.515 INFO:tasks.workunit.client.0.smithi102.stdout:2/39: dread d3/fb [0,4194304] 0 2024-03-26T18:54:27.515 INFO:tasks.workunit.client.0.smithi102.stdout:2/40: chown d3/f5 21841189 1 2024-03-26T18:54:27.515 INFO:tasks.workunit.client.0.smithi102.stdout:2/41: write d3/f5 [3719365,37159] 0 2024-03-26T18:54:27.516 INFO:tasks.workunit.client.0.smithi102.stdout:0/40: unlink d6/de/ff 0 2024-03-26T18:54:27.516 INFO:tasks.workunit.client.0.smithi102.stdout:0/41: readlink - no filename 2024-03-26T18:54:27.523 INFO:tasks.workunit.client.0.smithi102.stdout:2/42: unlink d3/f5 0 2024-03-26T18:54:27.525 INFO:tasks.workunit.client.0.smithi102.stdout:0/42: symlink d6/l10 0 2024-03-26T18:54:27.529 INFO:tasks.workunit.client.0.smithi102.stdout:2/43: rename c1 to d3/d6/cc 0 2024-03-26T18:54:27.531 INFO:tasks.workunit.client.0.smithi102.stdout:3/51: rmdir df/d10 0 2024-03-26T18:54:27.533 INFO:tasks.workunit.client.0.smithi102.stdout:2/44: symlink d3/d6/ld 0 2024-03-26T18:54:27.534 INFO:tasks.workunit.client.0.smithi102.stdout:5/56: getdents d2 0 2024-03-26T18:54:27.535 INFO:tasks.workunit.client.0.smithi102.stdout:3/52: creat df/f17 x:0 0 0 2024-03-26T18:54:27.535 INFO:tasks.workunit.client.0.smithi102.stdout:3/53: stat fc 0 2024-03-26T18:54:27.536 INFO:tasks.workunit.client.0.smithi102.stdout:2/45: write d3/f4 [4128782,46894] 0 2024-03-26T18:54:27.538 INFO:tasks.workunit.client.0.smithi102.stdout:5/57: link d2/f10 d2/d14/f15 0 2024-03-26T18:54:27.538 INFO:tasks.workunit.client.0.smithi102.stdout:5/58: truncate d2/f11 405406 0 2024-03-26T18:54:27.538 INFO:tasks.workunit.client.1.smithi190.stdout:1/222: dwrite d3/da/d45/f48 [0,4194304] 0 2024-03-26T18:54:27.540 INFO:tasks.workunit.client.0.smithi102.stdout:5/59: rmdir d2/d14 39 2024-03-26T18:54:27.542 INFO:tasks.workunit.client.0.smithi102.stdout:6/35: dwrite f0 [0,4194304] 0 2024-03-26T18:54:27.542 INFO:tasks.workunit.client.0.smithi102.stdout:6/36: chown d2/d3/f5 156 1 2024-03-26T18:54:27.544 INFO:tasks.workunit.client.0.smithi102.stdout:3/54: link df/l11 df/l18 0 2024-03-26T18:54:27.544 INFO:tasks.workunit.client.0.smithi102.stdout:3/55: chown f3 0 1 2024-03-26T18:54:27.545 INFO:tasks.workunit.client.0.smithi102.stdout:6/37: mkdir d2/d7 0 2024-03-26T18:54:27.548 INFO:tasks.workunit.client.0.smithi102.stdout:6/38: creat d2/d7/f8 x:0 0 0 2024-03-26T18:54:27.549 INFO:tasks.workunit.client.0.smithi102.stdout:3/56: mknod df/c19 0 2024-03-26T18:54:27.549 INFO:tasks.workunit.client.0.smithi102.stdout:3/57: chown df/l14 3 1 2024-03-26T18:54:27.550 INFO:tasks.workunit.client.0.smithi102.stdout:6/39: symlink d2/d7/l9 0 2024-03-26T18:54:27.552 INFO:tasks.workunit.client.0.smithi102.stdout:3/58: creat df/f1a x:0 0 0 2024-03-26T18:54:27.552 INFO:tasks.workunit.client.0.smithi102.stdout:3/59: write f7 [575954,11873] 0 2024-03-26T18:54:27.552 INFO:tasks.workunit.client.0.smithi102.stdout:3/60: chown f7 1 1 2024-03-26T18:54:27.552 INFO:tasks.workunit.client.0.smithi102.stdout:3/61: truncate df/f1a 113043 0 2024-03-26T18:54:27.553 INFO:tasks.workunit.client.0.smithi102.stdout:6/40: unlink f0 0 2024-03-26T18:54:27.555 INFO:tasks.workunit.client.0.smithi102.stdout:3/62: creat df/f1b x:0 0 0 2024-03-26T18:54:27.555 INFO:tasks.workunit.client.0.smithi102.stdout:3/63: chown c6 55939 1 2024-03-26T18:54:27.556 INFO:tasks.workunit.client.1.smithi190.stdout:9/492: dwrite d12/d2d/f31 [0,4194304] 0 2024-03-26T18:54:27.556 INFO:tasks.workunit.client.1.smithi190.stdout:4/256: dwrite d7/d8/d14/f1a [4194304,4194304] 0 2024-03-26T18:54:27.557 INFO:tasks.workunit.client.0.smithi102.stdout:6/41: rename d2/d7/l9 to d2/d3/la 0 2024-03-26T18:54:27.559 INFO:tasks.workunit.client.0.smithi102.stdout:6/42: symlink d2/lb 0 2024-03-26T18:54:27.560 INFO:tasks.workunit.client.1.smithi190.stdout:4/257: getdents d7/d8/d14/d23/d28 0 2024-03-26T18:54:27.560 INFO:tasks.workunit.client.1.smithi190.stdout:4/258: creat d7/d8/d1c/d20/f58 x:0 0 0 2024-03-26T18:54:27.560 INFO:tasks.workunit.client.0.smithi102.stdout:6/43: mknod d2/d7/cc 0 2024-03-26T18:54:27.560 INFO:tasks.workunit.client.0.smithi102.stdout:6/44: chown d2/d7/f8 828 1 2024-03-26T18:54:27.561 INFO:tasks.workunit.client.0.smithi102.stdout:6/45: chown d2/d7/cc 96 1 2024-03-26T18:54:27.561 INFO:tasks.workunit.client.0.smithi102.stdout:6/46: stat d2/d7 0 2024-03-26T18:54:27.562 INFO:tasks.workunit.client.0.smithi102.stdout:6/47: mkdir d2/dd 0 2024-03-26T18:54:27.562 INFO:tasks.workunit.client.0.smithi102.stdout:6/48: dread - d2/d3/f5 zero size 2024-03-26T18:54:27.562 INFO:tasks.workunit.client.0.smithi102.stdout:6/49: getdents d2/dd 0 2024-03-26T18:54:27.562 INFO:tasks.workunit.client.0.smithi102.stdout:6/50: dread - d2/d3/f5 zero size 2024-03-26T18:54:27.562 INFO:tasks.workunit.client.0.smithi102.stdout:6/51: write d2/d3/f5 [22622,33030] 0 2024-03-26T18:54:27.562 INFO:tasks.workunit.client.0.smithi102.stdout:6/52: write d2/d3/f5 [149601,129451] 0 2024-03-26T18:54:27.565 INFO:tasks.workunit.client.1.smithi190.stdout:1/223: dwrite d3/d9/f42 [0,4194304] 0 2024-03-26T18:54:27.566 INFO:tasks.workunit.client.1.smithi190.stdout:6/216: dwrite d17/d28/d2c/f35 [0,4194304] 0 2024-03-26T18:54:27.567 INFO:tasks.workunit.client.1.smithi190.stdout:5/164: dwrite d2/d17/d1b/d22/f27 [0,4194304] 0 2024-03-26T18:54:27.567 INFO:tasks.workunit.client.1.smithi190.stdout:5/165: creat d2/d17/d1b/f36 x:0 0 0 2024-03-26T18:54:27.567 INFO:tasks.workunit.client.1.smithi190.stdout:5/166: write d2/f12 [1668909,48854] 0 2024-03-26T18:54:27.567 INFO:tasks.workunit.client.0.smithi102.stdout:0/43: dwrite d6/fb [0,4194304] 0 2024-03-26T18:54:27.567 INFO:tasks.workunit.client.1.smithi190.stdout:7/260: dwrite d3/d6/d9/f40 [0,4194304] 0 2024-03-26T18:54:27.568 INFO:tasks.workunit.client.1.smithi190.stdout:3/206: truncate d1/de/f29 1827588 0 2024-03-26T18:54:27.569 INFO:tasks.workunit.client.0.smithi102.stdout:5/60: dwrite d2/fd [0,4194304] 0 2024-03-26T18:54:27.571 INFO:tasks.workunit.client.1.smithi190.stdout:1/224: truncate d3/d9/db/f2c 3337286 0 2024-03-26T18:54:27.578 INFO:tasks.workunit.client.0.smithi102.stdout:5/61: dread d2/fa [0,4194304] 0 2024-03-26T18:54:27.578 INFO:tasks.workunit.client.0.smithi102.stdout:5/62: chown d2/d14 84140 1 2024-03-26T18:54:27.579 INFO:tasks.workunit.client.1.smithi190.stdout:9/493: dwrite d12/d2d/d1d/d23/d83/f96 [0,4194304] 0 2024-03-26T18:54:27.579 INFO:tasks.workunit.client.1.smithi190.stdout:9/494: creat d12/fb0 x:0 0 0 2024-03-26T18:54:27.579 INFO:tasks.workunit.client.1.smithi190.stdout:9/495: chown d12/d2d/d1d/d23/d2b/fab 106 1 2024-03-26T18:54:27.581 INFO:tasks.workunit.client.1.smithi190.stdout:6/217: chown d17/l20 6788 1 2024-03-26T18:54:27.581 INFO:tasks.workunit.client.1.smithi190.stdout:6/218: dread - d17/d28/d38/f45 zero size 2024-03-26T18:54:27.581 INFO:tasks.workunit.client.1.smithi190.stdout:6/219: write d17/d28/d2b/f3c [190113,128984] 0 2024-03-26T18:54:27.583 INFO:tasks.workunit.client.0.smithi102.stdout:5/63: dread d2/f7 [0,4194304] 0 2024-03-26T18:54:27.583 INFO:tasks.workunit.client.0.smithi102.stdout:5/64: write d2/f10 [106054,102671] 0 2024-03-26T18:54:27.584 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:27 smithi102 ceph-mon[29002]: pgmap v241: 65 pgs: 65 active+clean; 606 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 27 MiB/s rd, 45 MiB/s wr, 681 op/s 2024-03-26T18:54:27.586 INFO:tasks.workunit.client.1.smithi190.stdout:1/225: dread d3/da/f1f [0,4194304] 0 2024-03-26T18:54:27.586 INFO:tasks.workunit.client.0.smithi102.stdout:0/44: dread d6/fb [0,4194304] 0 2024-03-26T18:54:27.586 INFO:tasks.workunit.client.0.smithi102.stdout:0/45: write d6/fd [18806,45886] 0 2024-03-26T18:54:27.589 INFO:tasks.workunit.client.0.smithi102.stdout:5/65: rename d2/fe to d2/f16 0 2024-03-26T18:54:27.592 INFO:tasks.workunit.client.0.smithi102.stdout:0/46: link d6/fd d6/f11 0 2024-03-26T18:54:27.594 INFO:tasks.workunit.client.1.smithi190.stdout:4/259: dwrite d7/d8/f33 [0,4194304] 0 2024-03-26T18:54:27.596 INFO:tasks.workunit.client.1.smithi190.stdout:6/220: write d17/f1b [1156753,21216] 0 2024-03-26T18:54:27.596 INFO:tasks.workunit.client.0.smithi102.stdout:5/66: creat d2/f17 x:0 0 0 2024-03-26T18:54:27.599 INFO:tasks.workunit.client.0.smithi102.stdout:2/46: dwrite d3/f8 [0,4194304] 0 2024-03-26T18:54:27.599 INFO:tasks.workunit.client.0.smithi102.stdout:2/47: fdatasync d3/f7 0 2024-03-26T18:54:27.600 INFO:tasks.workunit.client.0.smithi102.stdout:5/67: truncate d2/f16 2539665 0 2024-03-26T18:54:27.600 INFO:tasks.workunit.client.0.smithi102.stdout:5/68: readlink d2/l3 0 2024-03-26T18:54:27.603 INFO:tasks.workunit.client.1.smithi190.stdout:5/167: symlink d2/d17/d1b/l37 0 2024-03-26T18:54:27.604 INFO:tasks.workunit.client.0.smithi102.stdout:5/69: symlink d2/l18 0 2024-03-26T18:54:27.612 INFO:tasks.workunit.client.0.smithi102.stdout:2/48: dread d3/f4 [0,4194304] 0 2024-03-26T18:54:27.613 INFO:tasks.workunit.client.1.smithi190.stdout:3/207: mknod d1/d3/df/d18/c3f 0 2024-03-26T18:54:27.613 INFO:tasks.workunit.client.1.smithi190.stdout:3/208: write d1/de/f2d [689670,109235] 0 2024-03-26T18:54:27.616 INFO:tasks.workunit.client.0.smithi102.stdout:2/49: dread d3/fb [0,4194304] 0 2024-03-26T18:54:27.620 INFO:tasks.workunit.client.1.smithi190.stdout:5/168: dwrite d2/f10 [0,4194304] 0 2024-03-26T18:54:27.620 INFO:tasks.workunit.client.0.smithi102.stdout:6/53: dwrite d2/d3/f5 [0,4194304] 0 2024-03-26T18:54:27.623 INFO:tasks.workunit.client.1.smithi190.stdout:9/496: link d12/d2d/d1d/d54/d92/c93 d12/d2d/d1d/d54/d92/cb1 0 2024-03-26T18:54:27.624 INFO:tasks.workunit.client.0.smithi102.stdout:3/64: dwrite f7 [0,4194304] 0 2024-03-26T18:54:27.625 INFO:tasks.workunit.client.1.smithi190.stdout:1/226: truncate d3/da/f47 940377 0 2024-03-26T18:54:27.626 INFO:tasks.workunit.client.1.smithi190.stdout:3/209: write d1/d3/df/d18/f30 [4012326,55052] 0 2024-03-26T18:54:27.627 INFO:tasks.workunit.client.1.smithi190.stdout:4/260: symlink d7/d1d/l59 0 2024-03-26T18:54:27.627 INFO:tasks.workunit.client.1.smithi190.stdout:4/261: chown c3 89235610 1 2024-03-26T18:54:27.628 INFO:tasks.workunit.client.1.smithi190.stdout:3/210: write d1/de/d1c/d36/f3a [388564,124115] 0 2024-03-26T18:54:27.628 INFO:tasks.workunit.client.1.smithi190.stdout:3/211: fsync d1/de/f1d 0 2024-03-26T18:54:27.628 INFO:tasks.workunit.client.1.smithi190.stdout:3/212: creat d1/de/d1c/d36/f40 x:0 0 0 2024-03-26T18:54:27.631 INFO:tasks.workunit.client.1.smithi190.stdout:6/221: fdatasync d17/f24 0 2024-03-26T18:54:27.631 INFO:tasks.workunit.client.1.smithi190.stdout:6/222: write d17/d28/d2b/f3c [802105,46439] 0 2024-03-26T18:54:27.634 INFO:tasks.workunit.client.0.smithi102.stdout:3/65: mkdir df/d1c 0 2024-03-26T18:54:27.634 INFO:tasks.workunit.client.0.smithi102.stdout:6/54: dread d2/d3/f5 [0,4194304] 0 2024-03-26T18:54:27.634 INFO:tasks.workunit.client.0.smithi102.stdout:3/66: chown df/f1b 255455649 1 2024-03-26T18:54:27.634 INFO:tasks.workunit.client.0.smithi102.stdout:6/55: dread - d2/d7/f8 zero size 2024-03-26T18:54:27.636 INFO:tasks.workunit.client.1.smithi190.stdout:4/262: read d7/f10 [1926985,99856] 0 2024-03-26T18:54:27.636 INFO:tasks.workunit.client.1.smithi190.stdout:4/263: stat d7/f10 0 2024-03-26T18:54:27.636 INFO:tasks.workunit.client.0.smithi102.stdout:2/50: dwrite d3/fb [0,4194304] 0 2024-03-26T18:54:27.636 INFO:tasks.workunit.client.0.smithi102.stdout:3/67: creat df/f1d x:0 0 0 2024-03-26T18:54:27.637 INFO:tasks.workunit.client.0.smithi102.stdout:3/68: stat fb 0 2024-03-26T18:54:27.640 INFO:tasks.workunit.client.1.smithi190.stdout:3/213: read d1/fb [3044654,69059] 0 2024-03-26T18:54:27.640 INFO:tasks.workunit.client.1.smithi190.stdout:3/214: dread - d1/de/f2f zero size 2024-03-26T18:54:27.640 INFO:tasks.workunit.client.0.smithi102.stdout:3/69: mknod df/d1c/c1e 0 2024-03-26T18:54:27.641 INFO:tasks.workunit.client.0.smithi102.stdout:0/47: fsync d6/fd 0 2024-03-26T18:54:27.641 INFO:tasks.workunit.client.0.smithi102.stdout:0/48: readlink d6/l10 0 2024-03-26T18:54:27.645 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:27 smithi190 ceph-mon[35043]: pgmap v241: 65 pgs: 65 active+clean; 606 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 27 MiB/s rd, 45 MiB/s wr, 681 op/s 2024-03-26T18:54:27.647 INFO:tasks.workunit.client.0.smithi102.stdout:0/49: unlink d6/ca 0 2024-03-26T18:54:27.647 INFO:tasks.workunit.client.0.smithi102.stdout:0/50: chown d6/de 244460 1 2024-03-26T18:54:27.647 INFO:tasks.workunit.client.0.smithi102.stdout:0/51: chown d6 191 1 2024-03-26T18:54:27.649 INFO:tasks.workunit.client.0.smithi102.stdout:3/70: symlink df/l1f 0 2024-03-26T18:54:27.649 INFO:tasks.workunit.client.0.smithi102.stdout:3/71: readlink l9 0 2024-03-26T18:54:27.652 INFO:tasks.workunit.client.0.smithi102.stdout:0/52: symlink d6/de/l12 0 2024-03-26T18:54:27.652 INFO:tasks.workunit.client.0.smithi102.stdout:6/56: dwrite d2/d7/f8 [0,4194304] 0 2024-03-26T18:54:27.652 INFO:tasks.workunit.client.0.smithi102.stdout:6/57: readlink d2/d3/la 0 2024-03-26T18:54:27.652 INFO:tasks.workunit.client.0.smithi102.stdout:6/58: chown d2/d7/f8 589 1 2024-03-26T18:54:27.653 INFO:tasks.workunit.client.1.smithi190.stdout:7/261: rmdir d3/d6 39 2024-03-26T18:54:27.653 INFO:tasks.workunit.client.1.smithi190.stdout:7/262: stat d3/d6/d9/d1f/l32 0 2024-03-26T18:54:27.653 INFO:tasks.workunit.client.0.smithi102.stdout:3/72: creat df/f20 x:0 0 0 2024-03-26T18:54:27.655 INFO:tasks.workunit.client.0.smithi102.stdout:0/53: mkdir d6/de/d13 0 2024-03-26T18:54:27.659 INFO:tasks.workunit.client.1.smithi190.stdout:5/169: rename d2/d17/d1b/f26 to d2/d17/d1b/d22/f38 0 2024-03-26T18:54:27.661 INFO:tasks.workunit.client.1.smithi190.stdout:9/497: readlink d12/d2d/d1d/d23/d2a/d3f/d55/l66 0 2024-03-26T18:54:27.663 INFO:tasks.workunit.client.0.smithi102.stdout:6/59: rename d2/lb to d2/d7/le 0 2024-03-26T18:54:27.664 INFO:tasks.workunit.client.1.smithi190.stdout:1/227: creat d3/da/dd/f4b x:0 0 0 2024-03-26T18:54:27.665 INFO:tasks.workunit.client.0.smithi102.stdout:3/73: unlink c5 0 2024-03-26T18:54:27.665 INFO:tasks.workunit.client.0.smithi102.stdout:3/74: readlink df/l1f 0 2024-03-26T18:54:27.666 INFO:tasks.workunit.client.1.smithi190.stdout:2/282: sync 2024-03-26T18:54:27.667 INFO:tasks.workunit.client.1.smithi190.stdout:0/340: sync 2024-03-26T18:54:27.668 INFO:tasks.workunit.client.1.smithi190.stdout:0/341: dread d0/dd/d20/d1c/f56 [0,4194304] 0 2024-03-26T18:54:27.668 INFO:tasks.workunit.client.0.smithi102.stdout:0/54: link d6/f7 d6/f14 0 2024-03-26T18:54:27.671 INFO:tasks.workunit.client.0.smithi102.stdout:6/60: rename d2/dd to d2/df 0 2024-03-26T18:54:27.671 INFO:tasks.workunit.client.0.smithi102.stdout:6/61: fsync d2/d3/f5 0 2024-03-26T18:54:27.671 INFO:tasks.workunit.client.1.smithi190.stdout:8/256: sync 2024-03-26T18:54:27.671 INFO:tasks.workunit.client.0.smithi102.stdout:3/75: dread fc [0,4194304] 0 2024-03-26T18:54:27.672 INFO:tasks.workunit.client.0.smithi102.stdout:3/76: dread f3 [0,4194304] 0 2024-03-26T18:54:27.672 INFO:tasks.workunit.client.0.smithi102.stdout:3/77: fdatasync df/f17 0 2024-03-26T18:54:27.672 INFO:tasks.workunit.client.0.smithi102.stdout:3/78: stat df/d1c 0 2024-03-26T18:54:27.672 INFO:tasks.workunit.client.0.smithi102.stdout:3/79: chown df 0 1 2024-03-26T18:54:27.673 INFO:tasks.workunit.client.0.smithi102.stdout:3/80: readlink df/l1f 0 2024-03-26T18:54:27.673 INFO:tasks.workunit.client.1.smithi190.stdout:4/264: unlink d7/d8/d14/f2b 0 2024-03-26T18:54:27.673 INFO:tasks.workunit.client.1.smithi190.stdout:8/257: dread d9/dd/d15/d32/f4f [0,4194304] 0 2024-03-26T18:54:27.674 INFO:tasks.workunit.client.0.smithi102.stdout:0/55: rename d6/fc to d6/de/f15 0 2024-03-26T18:54:27.675 INFO:tasks.workunit.client.1.smithi190.stdout:3/215: mkdir d1/d3/df/d18/d35/d41 0 2024-03-26T18:54:27.675 INFO:tasks.workunit.client.1.smithi190.stdout:3/216: chown d1/d3/df/d18 0 1 2024-03-26T18:54:27.675 INFO:tasks.workunit.client.0.smithi102.stdout:2/51: truncate d3/f8 1793801 0 2024-03-26T18:54:27.677 INFO:tasks.workunit.client.1.smithi190.stdout:3/217: fsync d1/de/d1c/d36/f3a 0 2024-03-26T18:54:27.678 INFO:tasks.workunit.client.1.smithi190.stdout:7/263: mkdir d3/d6/d9/d1f/d41/d65 0 2024-03-26T18:54:27.681 INFO:tasks.workunit.client.1.smithi190.stdout:6/223: rename d17/d28/d2b/l37 to d17/d28/d38/l4f 0 2024-03-26T18:54:27.683 INFO:tasks.workunit.client.0.smithi102.stdout:6/62: mkdir d2/d7/d10 0 2024-03-26T18:54:27.684 INFO:tasks.workunit.client.1.smithi190.stdout:5/170: mknod d2/d17/d1b/d1e/c39 0 2024-03-26T18:54:27.687 INFO:tasks.workunit.client.1.smithi190.stdout:9/498: unlink d12/d2d/c67 0 2024-03-26T18:54:27.687 INFO:tasks.workunit.client.1.smithi190.stdout:9/499: write d12/d2d/f46 [5429198,31730] 0 2024-03-26T18:54:27.689 INFO:tasks.workunit.client.0.smithi102.stdout:3/81: unlink df/f12 0 2024-03-26T18:54:27.689 INFO:tasks.workunit.client.0.smithi102.stdout:6/63: dread d2/d3/f5 [0,4194304] 0 2024-03-26T18:54:27.691 INFO:tasks.workunit.client.0.smithi102.stdout:0/56: creat d6/de/f16 x:0 0 0 2024-03-26T18:54:27.692 INFO:tasks.workunit.client.1.smithi190.stdout:2/283: mkdir d6/d21/d61 0 2024-03-26T18:54:27.693 INFO:tasks.workunit.client.1.smithi190.stdout:0/342: mkdir d0/dd/d20/d61/d7f 0 2024-03-26T18:54:27.695 INFO:tasks.workunit.client.0.smithi102.stdout:0/57: mknod d6/de/d13/c17 0 2024-03-26T18:54:27.695 INFO:tasks.workunit.client.0.smithi102.stdout:0/58: chown d6/de/f15 22 1 2024-03-26T18:54:27.695 INFO:tasks.workunit.client.0.smithi102.stdout:0/59: write d6/f11 [204872,43628] 0 2024-03-26T18:54:27.696 INFO:tasks.workunit.client.1.smithi190.stdout:4/265: mknod d7/d8/d14/d23/d28/c5a 0 2024-03-26T18:54:27.698 INFO:tasks.workunit.client.0.smithi102.stdout:3/82: rename df/l1f to df/l21 0 2024-03-26T18:54:27.699 INFO:tasks.workunit.client.0.smithi102.stdout:2/52: dwrite d3/f7 [0,4194304] 0 2024-03-26T18:54:27.699 INFO:tasks.workunit.client.0.smithi102.stdout:2/53: creat d3/d6/fe x:0 0 0 2024-03-26T18:54:27.700 INFO:tasks.workunit.client.0.smithi102.stdout:5/70: truncate d2/f16 1541270 0 2024-03-26T18:54:27.700 INFO:tasks.workunit.client.0.smithi102.stdout:5/71: truncate d2/f7 771256 0 2024-03-26T18:54:27.702 INFO:tasks.workunit.client.1.smithi190.stdout:7/264: unlink d3/d6/d9/d14/f63 0 2024-03-26T18:54:27.703 INFO:tasks.workunit.client.1.smithi190.stdout:3/218: dwrite d1/de/f1d [0,4194304] 0 2024-03-26T18:54:27.703 INFO:tasks.workunit.client.1.smithi190.stdout:3/219: creat d1/f42 x:0 0 0 2024-03-26T18:54:27.705 INFO:tasks.workunit.client.1.smithi190.stdout:8/258: rename d9/dd/d23/f31 to d9/d48/f50 0 2024-03-26T18:54:27.709 INFO:tasks.workunit.client.0.smithi102.stdout:6/64: read d2/d7/f8 [2291730,54040] 0 2024-03-26T18:54:27.709 INFO:tasks.workunit.client.0.smithi102.stdout:6/65: write d2/d7/f8 [3513345,88064] 0 2024-03-26T18:54:27.709 INFO:tasks.workunit.client.0.smithi102.stdout:6/66: chown d2/d3 325 1 2024-03-26T18:54:27.710 INFO:tasks.workunit.client.1.smithi190.stdout:4/266: dread d7/d1d/f3c [0,4194304] 0 2024-03-26T18:54:27.711 INFO:tasks.workunit.client.1.smithi190.stdout:5/171: mkdir d2/d17/d1b/d1e/d3a 0 2024-03-26T18:54:27.711 INFO:tasks.workunit.client.1.smithi190.stdout:5/172: chown d2/d17/d1b 109006 1 2024-03-26T18:54:27.711 INFO:tasks.workunit.client.1.smithi190.stdout:5/173: truncate d2/d17/d1b/d1e/f2e 811569 0 2024-03-26T18:54:27.712 INFO:tasks.workunit.client.1.smithi190.stdout:5/174: chown d2/d17/l1c 74 1 2024-03-26T18:54:27.712 INFO:tasks.workunit.client.1.smithi190.stdout:7/265: read d3/d6/f1d [4657886,65249] 0 2024-03-26T18:54:27.712 INFO:tasks.workunit.client.0.smithi102.stdout:5/72: mknod d2/c19 0 2024-03-26T18:54:27.713 INFO:tasks.workunit.client.0.smithi102.stdout:5/73: dread d2/f7 [0,4194304] 0 2024-03-26T18:54:27.714 INFO:tasks.workunit.client.1.smithi190.stdout:9/500: mkdir d12/d80/db2 0 2024-03-26T18:54:27.714 INFO:tasks.workunit.client.1.smithi190.stdout:9/501: write d12/d2d/d1d/d23/d2a/d71/fa6 [901109,88615] 0 2024-03-26T18:54:27.715 INFO:tasks.workunit.client.0.smithi102.stdout:3/83: link df/l14 df/l22 0 2024-03-26T18:54:27.717 INFO:tasks.workunit.client.1.smithi190.stdout:1/228: getdents d3/da/dd/d2f/d34 0 2024-03-26T18:54:27.720 INFO:tasks.workunit.client.0.smithi102.stdout:6/67: rmdir d2/d7/d10 0 2024-03-26T18:54:27.720 INFO:tasks.workunit.client.0.smithi102.stdout:6/68: truncate d2/d7/f8 4572949 0 2024-03-26T18:54:27.721 INFO:tasks.workunit.client.1.smithi190.stdout:2/284: mknod d6/c62 0 2024-03-26T18:54:27.725 INFO:tasks.workunit.client.1.smithi190.stdout:0/343: mkdir d0/dd/d20/d61/d7f/d80 0 2024-03-26T18:54:27.729 INFO:tasks.workunit.client.0.smithi102.stdout:6/69: creat d2/f11 x:0 0 0 2024-03-26T18:54:27.731 INFO:tasks.workunit.client.1.smithi190.stdout:3/220: mknod d1/d3/df/d18/c43 0 2024-03-26T18:54:27.732 INFO:tasks.workunit.client.1.smithi190.stdout:6/224: rmdir d17/d28/d2c 39 2024-03-26T18:54:27.736 INFO:tasks.workunit.client.1.smithi190.stdout:5/175: mkdir d2/d17/d1b/d22/d24/d3b 0 2024-03-26T18:54:27.737 INFO:tasks.workunit.client.1.smithi190.stdout:7/266: link d3/d6/d9/d1f/f3d d3/d1b/f66 0 2024-03-26T18:54:27.737 INFO:tasks.workunit.client.0.smithi102.stdout:5/74: unlink d2/f4 0 2024-03-26T18:54:27.737 INFO:tasks.workunit.client.0.smithi102.stdout:5/75: readlink d2/l18 0 2024-03-26T18:54:27.738 INFO:tasks.workunit.client.1.smithi190.stdout:9/502: mkdir d12/d2d/d1d/d23/d2a/d71/daa/db3 0 2024-03-26T18:54:27.739 INFO:tasks.workunit.client.1.smithi190.stdout:1/229: creat d3/da/dd/d2f/d34/f4c x:0 0 0 2024-03-26T18:54:27.740 INFO:tasks.workunit.client.1.smithi190.stdout:2/285: creat d6/d9/d48/f63 x:0 0 0 2024-03-26T18:54:27.740 INFO:tasks.workunit.client.0.smithi102.stdout:5/76: mkdir d2/d14/d1a 0 2024-03-26T18:54:27.741 INFO:tasks.workunit.client.0.smithi102.stdout:5/77: creat d2/d14/d1a/f1b x:0 0 0 2024-03-26T18:54:27.741 INFO:tasks.workunit.client.0.smithi102.stdout:5/78: fsync d2/f9 0 2024-03-26T18:54:27.742 INFO:tasks.workunit.client.1.smithi190.stdout:3/221: rmdir d1/de/d1c 39 2024-03-26T18:54:27.742 INFO:tasks.workunit.client.1.smithi190.stdout:3/222: creat d1/f44 x:0 0 0 2024-03-26T18:54:27.743 INFO:tasks.workunit.client.1.smithi190.stdout:0/344: read d0/dd/d24/f2f [148659,91880] 0 2024-03-26T18:54:27.744 INFO:tasks.workunit.client.0.smithi102.stdout:5/79: creat d2/d14/f1c x:0 0 0 2024-03-26T18:54:27.744 INFO:tasks.workunit.client.0.smithi102.stdout:5/80: creat d2/f1d x:0 0 0 2024-03-26T18:54:27.744 INFO:tasks.workunit.client.0.smithi102.stdout:5/81: readlink d2/lc 0 2024-03-26T18:54:27.744 INFO:tasks.workunit.client.1.smithi190.stdout:6/225: mkdir d17/d28/d2c/d50 0 2024-03-26T18:54:27.744 INFO:tasks.workunit.client.1.smithi190.stdout:5/176: mkdir d2/d17/d1b/d1e/d3a/d3c 0 2024-03-26T18:54:27.744 INFO:tasks.workunit.client.1.smithi190.stdout:5/177: read - d2/d17/d1b/d22/f2d zero size 2024-03-26T18:54:27.744 INFO:tasks.workunit.client.1.smithi190.stdout:5/178: truncate d2/f14 949281 0 2024-03-26T18:54:27.744 INFO:tasks.workunit.client.1.smithi190.stdout:7/267: mknod d3/d6/d9/c67 0 2024-03-26T18:54:27.745 INFO:tasks.workunit.client.1.smithi190.stdout:1/230: unlink d3/d9/f18 0 2024-03-26T18:54:27.748 INFO:tasks.workunit.client.1.smithi190.stdout:8/259: dwrite d9/d48/f50 [0,4194304] 0 2024-03-26T18:54:27.748 INFO:tasks.workunit.client.1.smithi190.stdout:8/260: truncate d9/dd/d15/f45 468463 0 2024-03-26T18:54:27.749 INFO:tasks.workunit.client.1.smithi190.stdout:3/223: getdents d1/de/d1c 0 2024-03-26T18:54:27.750 INFO:tasks.workunit.client.1.smithi190.stdout:3/224: fdatasync d1/f4 0 2024-03-26T18:54:27.750 INFO:tasks.workunit.client.1.smithi190.stdout:3/225: chown d1/f42 146112 1 2024-03-26T18:54:27.750 INFO:tasks.workunit.client.1.smithi190.stdout:3/226: fdatasync d1/d3/df/f19 0 2024-03-26T18:54:27.751 INFO:tasks.workunit.client.1.smithi190.stdout:9/503: write d12/d2d/d1d/d23/f5f [3950297,46622] 0 2024-03-26T18:54:27.752 INFO:tasks.workunit.client.0.smithi102.stdout:0/60: dwrite d6/fd [0,4194304] 0 2024-03-26T18:54:27.752 INFO:tasks.workunit.client.0.smithi102.stdout:3/84: dwrite fd [0,4194304] 0 2024-03-26T18:54:27.752 INFO:tasks.workunit.client.0.smithi102.stdout:3/85: write df/f17 [114485,130005] 0 2024-03-26T18:54:27.754 INFO:tasks.workunit.client.1.smithi190.stdout:5/179: getdents d2/d17/d1b/d30 0 2024-03-26T18:54:27.755 INFO:tasks.workunit.client.1.smithi190.stdout:1/231: symlink d3/da/l4d 0 2024-03-26T18:54:27.756 INFO:tasks.workunit.client.1.smithi190.stdout:3/227: mknod d1/d3/df/d31/c45 0 2024-03-26T18:54:27.756 INFO:tasks.workunit.client.1.smithi190.stdout:4/267: dwrite d7/f13 [0,4194304] 0 2024-03-26T18:54:27.756 INFO:tasks.workunit.client.1.smithi190.stdout:4/268: write d7/d8/d1c/d20/d40/f54 [380930,23037] 0 2024-03-26T18:54:27.759 INFO:tasks.workunit.client.0.smithi102.stdout:6/70: dwrite d2/f11 [0,4194304] 0 2024-03-26T18:54:27.760 INFO:tasks.workunit.client.1.smithi190.stdout:5/180: dread d2/d17/f21 [0,4194304] 0 2024-03-26T18:54:27.767 INFO:tasks.workunit.client.0.smithi102.stdout:3/86: symlink df/d1c/l23 0 2024-03-26T18:54:27.767 INFO:tasks.workunit.client.0.smithi102.stdout:3/87: write f3 [1042261,28929] 0 2024-03-26T18:54:27.769 INFO:tasks.workunit.client.1.smithi190.stdout:2/286: dwrite d6/fe [4194304,4194304] 0 2024-03-26T18:54:27.769 INFO:tasks.workunit.client.1.smithi190.stdout:2/287: creat d6/d9/d1a/f64 x:0 0 0 2024-03-26T18:54:27.775 INFO:tasks.workunit.client.0.smithi102.stdout:3/88: symlink df/d1c/l24 0 2024-03-26T18:54:27.775 INFO:tasks.workunit.client.0.smithi102.stdout:3/89: fsync fb 0 2024-03-26T18:54:27.776 INFO:tasks.workunit.client.1.smithi190.stdout:3/228: symlink d1/de/d1c/l46 0 2024-03-26T18:54:27.779 INFO:tasks.workunit.client.1.smithi190.stdout:0/345: rename d0/dd/d26 to d0/dd/d24/d4b/d5e/d81 0 2024-03-26T18:54:27.779 INFO:tasks.workunit.client.1.smithi190.stdout:0/346: creat d0/dd/d24/d4b/d5e/f82 x:0 0 0 2024-03-26T18:54:27.781 INFO:tasks.workunit.client.0.smithi102.stdout:3/90: rename fe to df/f25 0 2024-03-26T18:54:27.783 INFO:tasks.workunit.client.1.smithi190.stdout:5/181: rename d2/d17/d1b/d22/f27 to d2/d17/d1b/d22/f3d 0 2024-03-26T18:54:27.785 INFO:tasks.workunit.client.1.smithi190.stdout:9/504: getdents d12 0 2024-03-26T18:54:27.785 INFO:tasks.workunit.client.1.smithi190.stdout:9/505: readlink d12/d2d/d1d/d23/l65 0 2024-03-26T18:54:27.785 INFO:tasks.workunit.client.1.smithi190.stdout:9/506: dread - d12/d2d/f9e zero size 2024-03-26T18:54:27.789 INFO:tasks.workunit.client.1.smithi190.stdout:3/229: mkdir d1/d47 0 2024-03-26T18:54:27.789 INFO:tasks.workunit.client.1.smithi190.stdout:6/226: rmdir d17 39 2024-03-26T18:54:27.790 INFO:tasks.workunit.client.1.smithi190.stdout:6/227: chown d17/f21 16093 1 2024-03-26T18:54:27.791 INFO:tasks.workunit.client.0.smithi102.stdout:0/61: dwrite d6/fb [0,4194304] 0 2024-03-26T18:54:27.791 INFO:tasks.workunit.client.1.smithi190.stdout:7/268: dwrite d3/d6/d9/d1f/d4c/f5c [0,4194304] 0 2024-03-26T18:54:27.792 INFO:tasks.workunit.client.0.smithi102.stdout:6/71: dwrite d2/d3/f5 [0,4194304] 0 2024-03-26T18:54:27.792 INFO:tasks.workunit.client.1.smithi190.stdout:2/288: rename d6/d9/d3a/d3d/f27 to d6/d9/d48/f65 0 2024-03-26T18:54:27.795 INFO:tasks.workunit.client.1.smithi190.stdout:4/269: dwrite d7/d8/d14/f48 [0,4194304] 0 2024-03-26T18:54:27.795 INFO:tasks.workunit.client.1.smithi190.stdout:4/270: dread - d7/d1d/d42/f50 zero size 2024-03-26T18:54:27.795 INFO:tasks.workunit.client.1.smithi190.stdout:4/271: write d7/d8/d14/d23/d28/d38/f4e [685698,69560] 0 2024-03-26T18:54:27.795 INFO:tasks.workunit.client.1.smithi190.stdout:4/272: creat d7/d8/d14/f5b x:0 0 0 2024-03-26T18:54:27.796 INFO:tasks.workunit.client.0.smithi102.stdout:0/62: link d6/fb d6/f18 0 2024-03-26T18:54:27.796 INFO:tasks.workunit.client.0.smithi102.stdout:0/63: chown d6/f7 30137 1 2024-03-26T18:54:27.796 INFO:tasks.workunit.client.0.smithi102.stdout:0/64: chown d6/f18 4631 1 2024-03-26T18:54:27.796 INFO:tasks.workunit.client.0.smithi102.stdout:0/65: chown d6/f9 8660506 1 2024-03-26T18:54:27.796 INFO:tasks.workunit.client.0.smithi102.stdout:5/82: getdents d2/d14 0 2024-03-26T18:54:27.797 INFO:tasks.workunit.client.0.smithi102.stdout:5/83: write d2/f17 [76009,19522] 0 2024-03-26T18:54:27.797 INFO:tasks.workunit.client.0.smithi102.stdout:5/84: stat d2/f10 0 2024-03-26T18:54:27.799 INFO:tasks.workunit.client.1.smithi190.stdout:0/347: mknod d0/c83 0 2024-03-26T18:54:27.801 INFO:tasks.workunit.client.0.smithi102.stdout:3/91: write fd [3003273,53065] 0 2024-03-26T18:54:27.802 INFO:tasks.workunit.client.1.smithi190.stdout:5/182: mknod d2/d17/d1b/d22/d24/d3b/c3e 0 2024-03-26T18:54:27.802 INFO:tasks.workunit.client.1.smithi190.stdout:6/228: dread d17/d28/d2c/f4c [0,4194304] 0 2024-03-26T18:54:27.802 INFO:tasks.workunit.client.1.smithi190.stdout:6/229: readlink d17/d28/l33 0 2024-03-26T18:54:27.802 INFO:tasks.workunit.client.1.smithi190.stdout:6/230: truncate d17/d28/d2c/f42 540251 0 2024-03-26T18:54:27.803 INFO:tasks.workunit.client.0.smithi102.stdout:6/72: mkdir d2/d12 0 2024-03-26T18:54:27.803 INFO:tasks.workunit.client.0.smithi102.stdout:6/73: getdents d2/d12 0 2024-03-26T18:54:27.805 INFO:tasks.workunit.client.0.smithi102.stdout:0/66: truncate d6/f14 2073390 0 2024-03-26T18:54:27.807 INFO:tasks.workunit.client.0.smithi102.stdout:5/85: mkdir d2/d14/d1e 0 2024-03-26T18:54:27.809 INFO:tasks.workunit.client.1.smithi190.stdout:7/269: mknod d3/d6/d9/d1f/d41/c68 0 2024-03-26T18:54:27.809 INFO:tasks.workunit.client.1.smithi190.stdout:7/270: readlink d3/d6/d9/l10 0 2024-03-26T18:54:27.809 INFO:tasks.workunit.client.1.smithi190.stdout:7/271: write d3/d6/d9/d1f/f3d [1065267,26245] 0 2024-03-26T18:54:27.811 INFO:tasks.workunit.client.1.smithi190.stdout:1/232: rename d3/da/dd/f4b to d3/da/dd/d2f/d34/d3e/f4e 0 2024-03-26T18:54:27.815 INFO:tasks.workunit.client.0.smithi102.stdout:3/92: rename df/f1d to df/f26 0 2024-03-26T18:54:27.815 INFO:tasks.workunit.client.0.smithi102.stdout:3/93: write df/f1a [1090263,543] 0 2024-03-26T18:54:27.817 INFO:tasks.workunit.client.1.smithi190.stdout:9/507: dwrite d12/f28 [0,4194304] 0 2024-03-26T18:54:27.817 INFO:tasks.workunit.client.1.smithi190.stdout:2/289: dwrite d6/d9/d1a/f58 [0,4194304] 0 2024-03-26T18:54:27.818 INFO:tasks.workunit.client.0.smithi102.stdout:5/86: mknod d2/d14/c1f 0 2024-03-26T18:54:27.819 INFO:tasks.workunit.client.1.smithi190.stdout:2/290: dread d6/d9/d10/f40 [0,4194304] 0 2024-03-26T18:54:27.819 INFO:tasks.workunit.client.1.smithi190.stdout:2/291: stat d6/d9/d3a/c46 0 2024-03-26T18:54:27.822 INFO:tasks.workunit.client.1.smithi190.stdout:0/348: dwrite d0/dd/f45 [0,4194304] 0 2024-03-26T18:54:27.822 INFO:tasks.workunit.client.1.smithi190.stdout:0/349: readlink d0/dd/d20/d1c/d46/l67 0 2024-03-26T18:54:27.829 INFO:tasks.workunit.client.1.smithi190.stdout:4/273: mkdir d7/d8/d14/d5c 0 2024-03-26T18:54:27.831 INFO:tasks.workunit.client.0.smithi102.stdout:5/87: dread d2/f9 [0,4194304] 0 2024-03-26T18:54:27.831 INFO:tasks.workunit.client.0.smithi102.stdout:5/88: fdatasync d2/f9 0 2024-03-26T18:54:27.831 INFO:tasks.workunit.client.1.smithi190.stdout:5/183: creat d2/d17/d1b/d30/f3f x:0 0 0 2024-03-26T18:54:27.832 INFO:tasks.workunit.client.1.smithi190.stdout:6/231: mkdir d17/d28/d46/d4d/d51 0 2024-03-26T18:54:27.835 INFO:tasks.workunit.client.1.smithi190.stdout:3/230: rename d1/d3/df/d18/c43 to d1/de/c48 0 2024-03-26T18:54:27.835 INFO:tasks.workunit.client.0.smithi102.stdout:5/89: dread d2/f16 [0,4194304] 0 2024-03-26T18:54:27.836 INFO:tasks.workunit.client.1.smithi190.stdout:1/233: mknod d3/c4f 0 2024-03-26T18:54:27.838 INFO:tasks.workunit.client.0.smithi102.stdout:5/90: creat d2/d14/d1a/f20 x:0 0 0 2024-03-26T18:54:27.838 INFO:tasks.workunit.client.0.smithi102.stdout:5/91: fdatasync d2/ff 0 2024-03-26T18:54:27.838 INFO:tasks.workunit.client.0.smithi102.stdout:5/92: chown d2/f17 940 1 2024-03-26T18:54:27.841 INFO:tasks.workunit.client.1.smithi190.stdout:2/292: symlink d6/d9/l66 0 2024-03-26T18:54:27.841 INFO:tasks.workunit.client.1.smithi190.stdout:2/293: chown d6/d9/d10 14384669 1 2024-03-26T18:54:27.841 INFO:tasks.workunit.client.1.smithi190.stdout:2/294: stat d6/d9/d3a/d3d/d1c/d44/f49 0 2024-03-26T18:54:27.842 INFO:tasks.workunit.client.0.smithi102.stdout:5/93: rmdir d2/d14 39 2024-03-26T18:54:27.843 INFO:tasks.workunit.client.1.smithi190.stdout:0/350: symlink d0/dd/d24/d4b/d5e/d6b/l84 0 2024-03-26T18:54:27.843 INFO:tasks.workunit.client.1.smithi190.stdout:0/351: write d0/dd/d24/d4b/d5e/f5f [50816,75604] 0 2024-03-26T18:54:27.844 INFO:tasks.workunit.client.0.smithi102.stdout:6/74: truncate d2/d3/f5 1053459 0 2024-03-26T18:54:27.844 INFO:tasks.workunit.client.1.smithi190.stdout:5/184: dread d2/f16 [0,4194304] 0 2024-03-26T18:54:27.844 INFO:tasks.workunit.client.1.smithi190.stdout:5/185: read - d2/d17/d1b/d30/f3f zero size 2024-03-26T18:54:27.844 INFO:tasks.workunit.client.1.smithi190.stdout:4/274: symlink d7/d8/l5d 0 2024-03-26T18:54:27.846 INFO:tasks.workunit.client.0.smithi102.stdout:0/67: dwrite d6/fb [0,4194304] 0 2024-03-26T18:54:27.846 INFO:tasks.workunit.client.0.smithi102.stdout:0/68: chown d6/de 14 1 2024-03-26T18:54:27.846 INFO:tasks.workunit.client.0.smithi102.stdout:5/94: symlink d2/d14/d1a/l21 0 2024-03-26T18:54:27.846 INFO:tasks.workunit.client.0.smithi102.stdout:5/95: dread - d2/f12 zero size 2024-03-26T18:54:27.846 INFO:tasks.workunit.client.0.smithi102.stdout:5/96: truncate d2/f10 1045012 0 2024-03-26T18:54:27.846 INFO:tasks.workunit.client.0.smithi102.stdout:3/94: dwrite fb [0,4194304] 0 2024-03-26T18:54:27.847 INFO:tasks.workunit.client.1.smithi190.stdout:7/272: dwrite d3/d6/d9/d1f/f20 [0,4194304] 0 2024-03-26T18:54:27.847 INFO:tasks.workunit.client.1.smithi190.stdout:7/273: chown d3/d6/d9/d1f/f3d 15487 1 2024-03-26T18:54:27.848 INFO:tasks.workunit.client.1.smithi190.stdout:9/508: rename d12/d80/lad to d12/d2d/d1d/da1/lb4 0 2024-03-26T18:54:27.849 INFO:tasks.workunit.client.0.smithi102.stdout:6/75: symlink d2/l13 0 2024-03-26T18:54:27.850 INFO:tasks.workunit.client.0.smithi102.stdout:0/69: rmdir d6/de/d13 39 2024-03-26T18:54:27.852 INFO:tasks.workunit.client.1.smithi190.stdout:1/234: creat d3/da/d3a/f50 x:0 0 0 2024-03-26T18:54:27.852 INFO:tasks.workunit.client.1.smithi190.stdout:1/235: chown d3 33938 1 2024-03-26T18:54:27.853 INFO:tasks.workunit.client.0.smithi102.stdout:5/97: creat d2/d14/d1e/f22 x:0 0 0 2024-03-26T18:54:27.853 INFO:tasks.workunit.client.0.smithi102.stdout:5/98: stat d2/d14/d1a/f1b 0 2024-03-26T18:54:27.855 INFO:tasks.workunit.client.1.smithi190.stdout:2/295: mkdir d6/d9/d3a/d3d/d67 0 2024-03-26T18:54:27.855 INFO:tasks.workunit.client.0.smithi102.stdout:5/99: dread d2/f16 [0,4194304] 0 2024-03-26T18:54:27.857 INFO:tasks.workunit.client.1.smithi190.stdout:0/352: creat d0/dd/d20/d1c/d46/d7c/f85 x:0 0 0 2024-03-26T18:54:27.857 INFO:tasks.workunit.client.1.smithi190.stdout:9/509: dread d12/d2d/d1d/d54/f90 [0,4194304] 0 2024-03-26T18:54:27.857 INFO:tasks.workunit.client.1.smithi190.stdout:9/510: write d12/d2d/f39 [4953876,47166] 0 2024-03-26T18:54:27.860 INFO:tasks.workunit.client.1.smithi190.stdout:0/353: dread d0/dd/fe [0,4194304] 0 2024-03-26T18:54:27.860 INFO:tasks.workunit.client.1.smithi190.stdout:4/275: mkdir d7/d1d/d5e 0 2024-03-26T18:54:27.862 INFO:tasks.workunit.client.1.smithi190.stdout:7/274: truncate d3/d6/f4e 307922 0 2024-03-26T18:54:27.865 INFO:tasks.workunit.client.0.smithi102.stdout:0/70: mkdir d6/de/d19 0 2024-03-26T18:54:27.867 INFO:tasks.workunit.client.0.smithi102.stdout:3/95: mknod df/c27 0 2024-03-26T18:54:27.873 INFO:tasks.workunit.client.0.smithi102.stdout:5/100: dwrite d2/f11 [0,4194304] 0 2024-03-26T18:54:27.874 INFO:tasks.workunit.client.0.smithi102.stdout:5/101: fsync d2/fd 0 2024-03-26T18:54:27.874 INFO:tasks.workunit.client.0.smithi102.stdout:5/102: fdatasync d2/d14/d1e/f22 0 2024-03-26T18:54:27.874 INFO:tasks.workunit.client.0.smithi102.stdout:3/96: symlink df/d1c/l28 0 2024-03-26T18:54:27.874 INFO:tasks.workunit.client.0.smithi102.stdout:0/71: write d6/f18 [2810089,26538] 0 2024-03-26T18:54:27.881 INFO:tasks.workunit.client.0.smithi102.stdout:5/103: dread d2/fd [0,4194304] 0 2024-03-26T18:54:27.882 INFO:tasks.workunit.client.0.smithi102.stdout:3/97: mkdir df/d29 0 2024-03-26T18:54:27.884 INFO:tasks.workunit.client.1.smithi190.stdout:3/231: rmdir d1/de/d1c/d36 39 2024-03-26T18:54:27.884 INFO:tasks.workunit.client.1.smithi190.stdout:3/232: chown d1/d3/df/d18/d1b/l37 233 1 2024-03-26T18:54:27.885 INFO:tasks.workunit.client.0.smithi102.stdout:3/98: unlink fb 0 2024-03-26T18:54:27.885 INFO:tasks.workunit.client.1.smithi190.stdout:1/236: mknod d3/da/dd/c51 0 2024-03-26T18:54:27.887 INFO:tasks.workunit.client.0.smithi102.stdout:5/104: rename d2/d14/f15 to d2/d14/d1e/f23 0 2024-03-26T18:54:27.887 INFO:tasks.workunit.client.0.smithi102.stdout:5/105: fdatasync d2/f1d 0 2024-03-26T18:54:27.888 INFO:tasks.workunit.client.1.smithi190.stdout:0/354: rename d0/l60 to d0/dd/d20/d1c/l86 0 2024-03-26T18:54:27.888 INFO:tasks.workunit.client.0.smithi102.stdout:3/99: creat df/d1c/f2a x:0 0 0 2024-03-26T18:54:27.889 INFO:tasks.workunit.client.1.smithi190.stdout:4/276: truncate d7/d8/d14/f1a 2489478 0 2024-03-26T18:54:27.889 INFO:tasks.workunit.client.1.smithi190.stdout:4/277: truncate d7/f18 87147 0 2024-03-26T18:54:27.889 INFO:tasks.workunit.client.1.smithi190.stdout:4/278: read d7/d8/d1c/d20/f3f [105305,85007] 0 2024-03-26T18:54:27.890 INFO:tasks.workunit.client.1.smithi190.stdout:4/279: write d7/d8/d14/f5b [447573,24811] 0 2024-03-26T18:54:27.890 INFO:tasks.workunit.client.0.smithi102.stdout:5/106: truncate d2/f16 847387 0 2024-03-26T18:54:27.890 INFO:tasks.workunit.client.0.smithi102.stdout:5/107: fdatasync d2/f11 0 2024-03-26T18:54:27.890 INFO:tasks.workunit.client.0.smithi102.stdout:5/108: dread - d2/f1d zero size 2024-03-26T18:54:27.890 INFO:tasks.workunit.client.0.smithi102.stdout:5/109: readlink d2/l13 0 2024-03-26T18:54:27.891 INFO:tasks.workunit.client.1.smithi190.stdout:3/233: mknod d1/de/c49 0 2024-03-26T18:54:27.891 INFO:tasks.workunit.client.1.smithi190.stdout:9/511: dwrite d12/f25 [0,4194304] 0 2024-03-26T18:54:27.891 INFO:tasks.workunit.client.0.smithi102.stdout:3/100: symlink df/d29/l2b 0 2024-03-26T18:54:27.892 INFO:tasks.workunit.client.1.smithi190.stdout:1/237: getdents d3/da/dd/d2f 0 2024-03-26T18:54:27.892 INFO:tasks.workunit.client.1.smithi190.stdout:7/275: dwrite d3/d6/d9/d1f/d41/d50/d53/f58 [0,4194304] 0 2024-03-26T18:54:27.892 INFO:tasks.workunit.client.1.smithi190.stdout:1/238: chown d3/da/d45 1216915842 1 2024-03-26T18:54:27.892 INFO:tasks.workunit.client.1.smithi190.stdout:1/239: stat d3/da/l4d 0 2024-03-26T18:54:27.892 INFO:tasks.workunit.client.1.smithi190.stdout:4/280: dread d7/d1d/f26 [0,4194304] 0 2024-03-26T18:54:27.892 INFO:tasks.workunit.client.1.smithi190.stdout:7/276: chown d3/d6/d9/d14/d36 15876 1 2024-03-26T18:54:27.893 INFO:tasks.workunit.client.0.smithi102.stdout:5/110: mknod d2/d14/d1a/c24 0 2024-03-26T18:54:27.893 INFO:tasks.workunit.client.0.smithi102.stdout:5/111: fsync d2/fd 0 2024-03-26T18:54:27.893 INFO:tasks.workunit.client.1.smithi190.stdout:0/355: mkdir d0/dd/d24/d87 0 2024-03-26T18:54:27.894 INFO:tasks.workunit.client.1.smithi190.stdout:3/234: mknod d1/de/d1c/c4a 0 2024-03-26T18:54:27.894 INFO:tasks.workunit.client.1.smithi190.stdout:3/235: write d1/d3/df/f2a [1051195,16412] 0 2024-03-26T18:54:27.894 INFO:tasks.workunit.client.1.smithi190.stdout:3/236: stat d1/fb 0 2024-03-26T18:54:27.897 INFO:tasks.workunit.client.1.smithi190.stdout:9/512: rename d12/d2d/d1d/d23/d2a/d3f/f4d to d12/d2d/d1d/d23/d2b/da0/fb5 0 2024-03-26T18:54:27.897 INFO:tasks.workunit.client.1.smithi190.stdout:3/237: dread d1/d3/df/f19 [0,4194304] 0 2024-03-26T18:54:27.898 INFO:tasks.workunit.client.0.smithi102.stdout:5/112: rmdir d2/d14 39 2024-03-26T18:54:27.901 INFO:tasks.workunit.client.1.smithi190.stdout:1/240: mkdir d3/da/dd/d2f/d34/d3e/d52 0 2024-03-26T18:54:27.901 INFO:tasks.workunit.client.1.smithi190.stdout:1/241: creat d3/f53 x:0 0 0 2024-03-26T18:54:27.902 INFO:tasks.workunit.client.1.smithi190.stdout:0/356: mkdir d0/dd/d20/d1c/d46/d7c/d88 0 2024-03-26T18:54:27.902 INFO:tasks.workunit.client.1.smithi190.stdout:0/357: chown d0/c83 246733 1 2024-03-26T18:54:27.902 INFO:tasks.workunit.client.1.smithi190.stdout:0/358: chown d0/dd/d24/c4d 71903 1 2024-03-26T18:54:27.902 INFO:tasks.workunit.client.1.smithi190.stdout:3/238: dread d1/f15 [0,4194304] 0 2024-03-26T18:54:27.902 INFO:tasks.workunit.client.1.smithi190.stdout:3/239: read d1/de/f29 [1793551,117171] 0 2024-03-26T18:54:27.902 INFO:tasks.workunit.client.1.smithi190.stdout:3/240: chown d1/de/d1c/c4a 16448 1 2024-03-26T18:54:27.903 INFO:tasks.workunit.client.0.smithi102.stdout:5/113: mkdir d2/d25 0 2024-03-26T18:54:27.903 INFO:tasks.workunit.client.0.smithi102.stdout:5/114: write d2/f1d [741002,84836] 0 2024-03-26T18:54:27.904 INFO:tasks.workunit.client.1.smithi190.stdout:4/281: rename d7/d8/d14/d23/d28/d38/f4e to d7/d8/d14/d4b/f5f 0 2024-03-26T18:54:27.905 INFO:tasks.workunit.client.1.smithi190.stdout:9/513: mknod d12/d2d/d1d/cb6 0 2024-03-26T18:54:27.906 INFO:tasks.workunit.client.1.smithi190.stdout:1/242: mknod d3/d9/db/c54 0 2024-03-26T18:54:27.906 INFO:tasks.workunit.client.1.smithi190.stdout:7/277: dwrite d3/f43 [0,4194304] 0 2024-03-26T18:54:27.907 INFO:tasks.workunit.client.1.smithi190.stdout:7/278: write d3/f46 [409851,34953] 0 2024-03-26T18:54:27.907 INFO:tasks.workunit.client.1.smithi190.stdout:0/359: unlink d0/dd/d24/f2a 0 2024-03-26T18:54:27.909 INFO:tasks.workunit.client.1.smithi190.stdout:3/241: link d1/d3/df/f2a d1/d3/f4b 0 2024-03-26T18:54:27.909 INFO:tasks.workunit.client.1.smithi190.stdout:4/282: creat d7/d1d/d5e/f60 x:0 0 0 2024-03-26T18:54:27.912 INFO:tasks.workunit.client.1.smithi190.stdout:0/360: truncate d0/dd/d20/f30 2221051 0 2024-03-26T18:54:27.912 INFO:tasks.workunit.client.1.smithi190.stdout:3/242: symlink d1/d3/df/d18/d35/d41/l4c 0 2024-03-26T18:54:27.913 INFO:tasks.workunit.client.1.smithi190.stdout:9/514: truncate d12/d2d/d1d/d23/d2a/d3f/d55/f62 545795 0 2024-03-26T18:54:27.913 INFO:tasks.workunit.client.1.smithi190.stdout:9/515: fdatasync d12/d2d/f3e 0 2024-03-26T18:54:27.914 INFO:tasks.workunit.client.1.smithi190.stdout:9/516: creat d12/d2d/d1d/d23/d83/d91/fb7 x:0 0 0 2024-03-26T18:54:27.914 INFO:tasks.workunit.client.1.smithi190.stdout:0/361: creat d0/dd/d24/d4b/f89 x:0 0 0 2024-03-26T18:54:27.915 INFO:tasks.workunit.client.1.smithi190.stdout:3/243: symlink d1/de/d1c/d36/l4d 0 2024-03-26T18:54:27.916 INFO:tasks.workunit.client.1.smithi190.stdout:3/244: mkdir d1/de/d4e 0 2024-03-26T18:54:27.918 INFO:tasks.workunit.client.1.smithi190.stdout:3/245: creat d1/de/f4f x:0 0 0 2024-03-26T18:54:27.919 INFO:tasks.workunit.client.1.smithi190.stdout:3/246: mknod d1/d3/df/d18/d35/c50 0 2024-03-26T18:54:27.942 INFO:tasks.workunit.client.1.smithi190.stdout:1/243: dwrite f2 [4194304,4194304] 0 2024-03-26T18:54:27.954 INFO:tasks.workunit.client.0.smithi102.stdout:3/101: rmdir df 39 2024-03-26T18:54:27.955 INFO:tasks.workunit.client.0.smithi102.stdout:3/102: chown f7 31070206 1 2024-03-26T18:54:27.956 INFO:tasks.workunit.client.1.smithi190.stdout:8/261: sync 2024-03-26T18:54:27.956 INFO:tasks.workunit.client.1.smithi190.stdout:8/262: write d9/fa [2942844,15852] 0 2024-03-26T18:54:27.956 INFO:tasks.workunit.client.0.smithi102.stdout:3/103: rename fc to df/d29/f2c 0 2024-03-26T18:54:27.957 INFO:tasks.workunit.client.1.smithi190.stdout:8/263: creat d9/d16/f51 x:0 0 0 2024-03-26T18:54:27.959 INFO:tasks.workunit.client.0.smithi102.stdout:3/104: creat df/f2d x:0 0 0 2024-03-26T18:54:27.959 INFO:tasks.workunit.client.1.smithi190.stdout:7/279: dwrite d3/d6/f3e [0,4194304] 0 2024-03-26T18:54:27.961 INFO:tasks.workunit.client.1.smithi190.stdout:8/264: stat d9/d48/d49/c4e 0 2024-03-26T18:54:27.961 INFO:tasks.workunit.client.0.smithi102.stdout:3/105: unlink df/c27 0 2024-03-26T18:54:27.963 INFO:tasks.workunit.client.1.smithi190.stdout:9/517: dwrite d12/d2d/d1d/d23/f33 [0,4194304] 0 2024-03-26T18:54:27.963 INFO:tasks.workunit.client.1.smithi190.stdout:9/518: truncate d12/d2d/f49 5056059 0 2024-03-26T18:54:27.964 INFO:tasks.workunit.client.1.smithi190.stdout:0/362: dwrite d0/dd/d20/d1c/d46/f5d [0,4194304] 0 2024-03-26T18:54:27.964 INFO:tasks.workunit.client.1.smithi190.stdout:4/283: dwrite d7/f35 [0,4194304] 0 2024-03-26T18:54:27.965 INFO:tasks.workunit.client.1.smithi190.stdout:4/284: write d7/d8/d14/d23/f55 [1184581,43168] 0 2024-03-26T18:54:27.965 INFO:tasks.workunit.client.1.smithi190.stdout:4/285: dread - d7/d8/d1c/d20/f47 zero size 2024-03-26T18:54:27.965 INFO:tasks.workunit.client.1.smithi190.stdout:3/247: dwrite d1/f14 [4194304,4194304] 0 2024-03-26T18:54:27.965 INFO:tasks.workunit.client.1.smithi190.stdout:3/248: write d1/f14 [1290556,123684] 0 2024-03-26T18:54:27.967 INFO:tasks.workunit.client.0.smithi102.stdout:3/106: creat df/f2e x:0 0 0 2024-03-26T18:54:27.968 INFO:tasks.workunit.client.1.smithi190.stdout:9/519: symlink d12/d2d/d1d/d23/d83/d91/lb8 0 2024-03-26T18:54:27.973 INFO:tasks.workunit.client.1.smithi190.stdout:1/244: dwrite d3/da/f47 [0,4194304] 0 2024-03-26T18:54:27.973 INFO:tasks.workunit.client.1.smithi190.stdout:8/265: dread d9/dd/f34 [0,4194304] 0 2024-03-26T18:54:27.974 INFO:tasks.workunit.client.1.smithi190.stdout:1/245: creat d3/da/d3a/f55 x:0 0 0 2024-03-26T18:54:27.974 INFO:tasks.workunit.client.1.smithi190.stdout:1/246: write d3/da/dd/f12 [1933746,24411] 0 2024-03-26T18:54:27.975 INFO:tasks.workunit.client.0.smithi102.stdout:3/107: symlink df/l2f 0 2024-03-26T18:54:27.977 INFO:tasks.workunit.client.0.smithi102.stdout:6/76: write d2/d3/f5 [142705,53530] 0 2024-03-26T18:54:27.977 INFO:tasks.workunit.client.0.smithi102.stdout:6/77: write d2/f11 [5094433,106569] 0 2024-03-26T18:54:27.977 INFO:tasks.workunit.client.0.smithi102.stdout:6/78: truncate d2/d7/f8 4724213 0 2024-03-26T18:54:27.977 INFO:tasks.workunit.client.0.smithi102.stdout:5/115: dwrite d2/f16 [0,4194304] 0 2024-03-26T18:54:27.977 INFO:tasks.workunit.client.0.smithi102.stdout:5/116: write d2/d14/d1a/f1b [774102,25462] 0 2024-03-26T18:54:27.983 INFO:tasks.workunit.client.0.smithi102.stdout:6/79: unlink d2/d7/cc 0 2024-03-26T18:54:27.985 INFO:tasks.workunit.client.1.smithi190.stdout:3/249: mkdir d1/d3/df/d18/d35/d41/d51 0 2024-03-26T18:54:27.985 INFO:tasks.workunit.client.1.smithi190.stdout:3/250: write d1/de/f2f [947799,66097] 0 2024-03-26T18:54:27.987 INFO:tasks.workunit.client.0.smithi102.stdout:5/117: rename d2/d14/d1a/f20 to d2/d25/f26 0 2024-03-26T18:54:27.987 INFO:tasks.workunit.client.0.smithi102.stdout:5/118: stat d2/f10 0 2024-03-26T18:54:27.987 INFO:tasks.workunit.client.0.smithi102.stdout:5/119: dread - d2/d14/d1e/f22 zero size 2024-03-26T18:54:27.989 INFO:tasks.workunit.client.0.smithi102.stdout:3/108: mknod df/c30 0 2024-03-26T18:54:27.990 INFO:tasks.workunit.client.1.smithi190.stdout:8/266: rmdir d9/dd/d15/d32 39 2024-03-26T18:54:27.991 INFO:tasks.workunit.client.1.smithi190.stdout:4/286: dwrite d7/f10 [0,4194304] 0 2024-03-26T18:54:27.992 INFO:tasks.workunit.client.1.smithi190.stdout:3/251: dread d1/d3/f4b [0,4194304] 0 2024-03-26T18:54:27.994 INFO:tasks.workunit.client.1.smithi190.stdout:7/280: dwrite d3/d6/d9/d1f/f20 [4194304,4194304] 0 2024-03-26T18:54:27.994 INFO:tasks.workunit.client.1.smithi190.stdout:7/281: write d3/d6/d9/d14/d36/f48 [579233,5380] 0 2024-03-26T18:54:27.995 INFO:tasks.workunit.client.1.smithi190.stdout:9/520: dwrite d12/d2d/d1d/d23/d2a/d71/fa6 [0,4194304] 0 2024-03-26T18:54:27.996 INFO:tasks.workunit.client.1.smithi190.stdout:0/363: getdents d0/dd/d24/d4b/d5e 0 2024-03-26T18:54:27.999 INFO:tasks.workunit.client.0.smithi102.stdout:5/120: link d2/ff d2/d14/f27 0 2024-03-26T18:54:27.999 INFO:tasks.workunit.client.0.smithi102.stdout:5/121: write d2/f9 [3980660,93325] 0 2024-03-26T18:54:28.001 INFO:tasks.workunit.client.1.smithi190.stdout:0/364: dread d0/dd/d24/f49 [0,4194304] 0 2024-03-26T18:54:28.001 INFO:tasks.workunit.client.1.smithi190.stdout:0/365: dread - d0/dd/d20/f68 zero size 2024-03-26T18:54:28.001 INFO:tasks.workunit.client.1.smithi190.stdout:0/366: chown d0/dc/l21 110 1 2024-03-26T18:54:28.001 INFO:tasks.workunit.client.1.smithi190.stdout:0/367: chown d0/dd/d24/d4b/d5e/d81/l70 88 1 2024-03-26T18:54:28.001 INFO:tasks.workunit.client.0.smithi102.stdout:6/80: fdatasync d2/d3/f5 0 2024-03-26T18:54:28.001 INFO:tasks.workunit.client.0.smithi102.stdout:6/81: chown d2/d7/le 0 1 2024-03-26T18:54:28.002 INFO:tasks.workunit.client.0.smithi102.stdout:5/122: symlink d2/d14/d1e/l28 0 2024-03-26T18:54:28.004 INFO:tasks.workunit.client.1.smithi190.stdout:1/247: dread d3/da/dd/f12 [0,4194304] 0 2024-03-26T18:54:28.004 INFO:tasks.workunit.client.1.smithi190.stdout:1/248: creat d3/da/dd/d2f/d34/d3e/f56 x:0 0 0 2024-03-26T18:54:28.004 INFO:tasks.workunit.client.1.smithi190.stdout:1/249: dread - d3/da/f37 zero size 2024-03-26T18:54:28.006 INFO:tasks.workunit.client.0.smithi102.stdout:5/123: link d2/lc d2/l29 0 2024-03-26T18:54:28.009 INFO:tasks.workunit.client.0.smithi102.stdout:5/124: mkdir d2/d25/d2a 0 2024-03-26T18:54:28.009 INFO:tasks.workunit.client.0.smithi102.stdout:5/125: stat d2/d14/d1e/l28 0 2024-03-26T18:54:28.012 INFO:tasks.workunit.client.0.smithi102.stdout:5/126: rename d2/f16 to d2/d25/f2b 0 2024-03-26T18:54:28.013 INFO:tasks.workunit.client.0.smithi102.stdout:5/127: write d2/d25/f26 [408563,60315] 0 2024-03-26T18:54:28.013 INFO:tasks.workunit.client.1.smithi190.stdout:0/368: dread d0/dd/d20/d61/f7b [0,4194304] 0 2024-03-26T18:54:28.013 INFO:tasks.workunit.client.1.smithi190.stdout:0/369: write d0/dd/d20/d1c/f6c [510996,9953] 0 2024-03-26T18:54:28.015 INFO:tasks.workunit.client.0.smithi102.stdout:5/128: mkdir d2/d2c 0 2024-03-26T18:54:28.016 INFO:tasks.workunit.client.1.smithi190.stdout:4/287: dwrite d7/d8/d14/f5b [0,4194304] 0 2024-03-26T18:54:28.017 INFO:tasks.workunit.client.1.smithi190.stdout:4/288: truncate d7/d1d/f27 973179 0 2024-03-26T18:54:28.017 INFO:tasks.workunit.client.1.smithi190.stdout:4/289: fsync d7/f13 0 2024-03-26T18:54:28.017 INFO:tasks.workunit.client.0.smithi102.stdout:3/109: dwrite df/f2d [0,4194304] 0 2024-03-26T18:54:28.017 INFO:tasks.workunit.client.0.smithi102.stdout:3/110: dread - df/d1c/f2a zero size 2024-03-26T18:54:28.018 INFO:tasks.workunit.client.1.smithi190.stdout:8/267: mknod d9/dd/d15/d32/c52 0 2024-03-26T18:54:28.018 INFO:tasks.workunit.client.1.smithi190.stdout:0/370: dread d0/dd/d24/d4b/d5e/d81/f36 [4194304,4194304] 0 2024-03-26T18:54:28.019 INFO:tasks.workunit.client.0.smithi102.stdout:5/129: creat d2/d14/f2d x:0 0 0 2024-03-26T18:54:28.020 INFO:tasks.workunit.client.1.smithi190.stdout:3/252: stat d1/d3/c8 0 2024-03-26T18:54:28.023 INFO:tasks.workunit.client.1.smithi190.stdout:0/371: dread d0/f13 [0,4194304] 0 2024-03-26T18:54:28.023 INFO:tasks.workunit.client.1.smithi190.stdout:7/282: creat d3/d6/d9/d1f/d41/d50/f69 x:0 0 0 2024-03-26T18:54:28.024 INFO:tasks.workunit.client.1.smithi190.stdout:4/290: dread d7/d8/d1c/f39 [0,4194304] 0 2024-03-26T18:54:28.031 INFO:tasks.workunit.client.1.smithi190.stdout:9/521: rename d12/d2d/f49 to d12/d2d/d1d/d23/d83/d91/fb9 0 2024-03-26T18:54:28.035 INFO:tasks.workunit.client.0.smithi102.stdout:3/111: mkdir df/d29/d31 0 2024-03-26T18:54:28.037 INFO:tasks.workunit.client.0.smithi102.stdout:5/130: creat d2/d25/d2a/f2e x:0 0 0 2024-03-26T18:54:28.039 INFO:tasks.workunit.client.1.smithi190.stdout:1/250: truncate d3/da/dd/d2f/d34/d3e/f46 2531643 0 2024-03-26T18:54:28.040 INFO:tasks.workunit.client.0.smithi102.stdout:3/112: creat df/d29/d31/f32 x:0 0 0 2024-03-26T18:54:28.042 INFO:tasks.workunit.client.1.smithi190.stdout:8/268: creat d9/dd/d15/d32/f53 x:0 0 0 2024-03-26T18:54:28.042 INFO:tasks.workunit.client.1.smithi190.stdout:8/269: fdatasync d9/dd/d15/f2c 0 2024-03-26T18:54:28.042 INFO:tasks.workunit.client.1.smithi190.stdout:8/270: chown d9/f14 241555 1 2024-03-26T18:54:28.042 INFO:tasks.workunit.client.1.smithi190.stdout:8/271: chown d9/dd/c1d 15269 1 2024-03-26T18:54:28.042 INFO:tasks.workunit.client.1.smithi190.stdout:8/272: chown d9/dd/l47 696934 1 2024-03-26T18:54:28.043 INFO:tasks.workunit.client.0.smithi102.stdout:3/113: creat df/d29/f33 x:0 0 0 2024-03-26T18:54:28.046 INFO:tasks.workunit.client.0.smithi102.stdout:3/114: mkdir df/d1c/d34 0 2024-03-26T18:54:28.046 INFO:tasks.workunit.client.0.smithi102.stdout:3/115: chown df/d29/f2c 0 1 2024-03-26T18:54:28.050 INFO:tasks.workunit.client.0.smithi102.stdout:5/131: dread d2/fa [0,4194304] 0 2024-03-26T18:54:28.055 INFO:tasks.workunit.client.0.smithi102.stdout:5/132: creat d2/d14/f2f x:0 0 0 2024-03-26T18:54:28.055 INFO:tasks.workunit.client.0.smithi102.stdout:5/133: chown d2/d14/c1f 6454614 1 2024-03-26T18:54:28.055 INFO:tasks.workunit.client.0.smithi102.stdout:5/134: readlink d2/d14/d1e/l28 0 2024-03-26T18:54:28.055 INFO:tasks.workunit.client.1.smithi190.stdout:4/291: symlink d7/d8/d14/d5c/l61 0 2024-03-26T18:54:28.058 INFO:tasks.workunit.client.1.smithi190.stdout:0/372: rename d0/dd/d24/d4b/d5e/d81/c57 to d0/dd/d24/d4b/d5e/c8a 0 2024-03-26T18:54:28.058 INFO:tasks.workunit.client.1.smithi190.stdout:0/373: stat d0/dd/d24/f2f 0 2024-03-26T18:54:28.058 INFO:tasks.workunit.client.1.smithi190.stdout:0/374: write d0/dd/d20/d1c/f56 [473007,66727] 0 2024-03-26T18:54:28.058 INFO:tasks.workunit.client.1.smithi190.stdout:0/375: stat d0/dc/c1d 0 2024-03-26T18:54:28.059 INFO:tasks.workunit.client.0.smithi102.stdout:5/135: link d2/c19 d2/d25/c30 0 2024-03-26T18:54:28.059 INFO:tasks.workunit.client.1.smithi190.stdout:9/522: mkdir d12/d2d/d1d/da1/dba 0 2024-03-26T18:54:28.059 INFO:tasks.workunit.client.1.smithi190.stdout:9/523: creat d12/d2d/d1d/d23/d2a/d3f/fbb x:0 0 0 2024-03-26T18:54:28.059 INFO:tasks.workunit.client.1.smithi190.stdout:9/524: stat d12/d2d/d1d/d23 0 2024-03-26T18:54:28.059 INFO:tasks.workunit.client.1.smithi190.stdout:1/251: write d3/da/f47 [860244,106513] 0 2024-03-26T18:54:28.060 INFO:tasks.workunit.client.1.smithi190.stdout:3/253: dwrite d1/de/d1c/d36/f3a [0,4194304] 0 2024-03-26T18:54:28.060 INFO:tasks.workunit.client.1.smithi190.stdout:7/283: dwrite d3/d1b/f3b [0,4194304] 0 2024-03-26T18:54:28.060 INFO:tasks.workunit.client.1.smithi190.stdout:1/252: creat d3/d9/f57 x:0 0 0 2024-03-26T18:54:28.061 INFO:tasks.workunit.client.1.smithi190.stdout:4/292: link d7/f4f d7/d8/d14/d23/d28/d38/f62 0 2024-03-26T18:54:28.061 INFO:tasks.workunit.client.1.smithi190.stdout:0/376: truncate d0/dd/d20/d1c/f6e 2794753 0 2024-03-26T18:54:28.061 INFO:tasks.workunit.client.1.smithi190.stdout:0/377: chown d0/dc 1 1 2024-03-26T18:54:28.065 INFO:tasks.workunit.client.0.smithi102.stdout:5/136: getdents d2/d14/d1e 0 2024-03-26T18:54:28.065 INFO:tasks.workunit.client.0.smithi102.stdout:5/137: creat d2/d25/d2a/f31 x:0 0 0 2024-03-26T18:54:28.065 INFO:tasks.workunit.client.0.smithi102.stdout:5/138: chown d2/d14/d1e/f22 8708578 1 2024-03-26T18:54:28.065 INFO:tasks.workunit.client.0.smithi102.stdout:5/139: readlink d2/l13 0 2024-03-26T18:54:28.065 INFO:tasks.workunit.client.1.smithi190.stdout:0/378: write d0/f6 [1213266,73955] 0 2024-03-26T18:54:28.066 INFO:tasks.workunit.client.1.smithi190.stdout:0/379: write d0/dd/d24/f2b [5088391,122120] 0 2024-03-26T18:54:28.066 INFO:tasks.workunit.client.1.smithi190.stdout:8/273: write d9/f14 [2202181,80869] 0 2024-03-26T18:54:28.068 INFO:tasks.workunit.client.1.smithi190.stdout:3/254: dread d1/f3b [0,4194304] 0 2024-03-26T18:54:28.069 INFO:tasks.workunit.client.0.smithi102.stdout:5/140: link d2/lc d2/d25/l32 0 2024-03-26T18:54:28.070 INFO:tasks.workunit.client.0.smithi102.stdout:5/141: readlink d2/l13 0 2024-03-26T18:54:28.070 INFO:tasks.workunit.client.0.smithi102.stdout:5/142: chown d2/d25/d2a/f31 2 1 2024-03-26T18:54:28.070 INFO:tasks.workunit.client.1.smithi190.stdout:9/525: write d12/d2d/d1d/d23/d2a/d71/fa6 [2548869,20348] 0 2024-03-26T18:54:28.070 INFO:tasks.workunit.client.1.smithi190.stdout:7/284: mknod d3/d6/c6a 0 2024-03-26T18:54:28.072 INFO:tasks.workunit.client.1.smithi190.stdout:4/293: creat d7/d8/d14/d5c/f63 x:0 0 0 2024-03-26T18:54:28.078 INFO:tasks.workunit.client.1.smithi190.stdout:8/274: symlink d9/dd/d15/d32/l54 0 2024-03-26T18:54:28.078 INFO:tasks.workunit.client.1.smithi190.stdout:3/255: mkdir d1/d3/d52 0 2024-03-26T18:54:28.079 INFO:tasks.workunit.client.1.smithi190.stdout:9/526: mkdir d12/d2d/d1d/da1/dba/dbc 0 2024-03-26T18:54:28.081 INFO:tasks.workunit.client.1.smithi190.stdout:4/294: dread d7/d1d/f3c [0,4194304] 0 2024-03-26T18:54:28.081 INFO:tasks.workunit.client.1.smithi190.stdout:7/285: creat d3/d6/d9/d14/d36/f6b x:0 0 0 2024-03-26T18:54:28.082 INFO:tasks.workunit.client.1.smithi190.stdout:8/275: mknod d9/d48/d49/c55 0 2024-03-26T18:54:28.083 INFO:tasks.workunit.client.0.smithi102.stdout:3/116: dwrite df/f2d [0,4194304] 0 2024-03-26T18:54:28.084 INFO:tasks.workunit.client.1.smithi190.stdout:1/253: dwrite d3/d9/f13 [0,4194304] 0 2024-03-26T18:54:28.084 INFO:tasks.workunit.client.1.smithi190.stdout:3/256: rename d1/l1f to d1/de/l53 0 2024-03-26T18:54:28.085 INFO:tasks.workunit.client.1.smithi190.stdout:4/295: symlink d7/d8/d14/d23/l64 0 2024-03-26T18:54:28.085 INFO:tasks.workunit.client.1.smithi190.stdout:4/296: readlink d7/d1d/l59 0 2024-03-26T18:54:28.086 INFO:tasks.workunit.client.1.smithi190.stdout:4/297: chown d7/d8/d1c/d20 43770949 1 2024-03-26T18:54:28.087 INFO:tasks.workunit.client.1.smithi190.stdout:7/286: creat d3/d6/d9/d1f/f6c x:0 0 0 2024-03-26T18:54:28.087 INFO:tasks.workunit.client.1.smithi190.stdout:4/298: dread d7/d8/f2a [0,4194304] 0 2024-03-26T18:54:28.087 INFO:tasks.workunit.client.0.smithi102.stdout:3/117: dread df/f17 [0,4194304] 0 2024-03-26T18:54:28.088 INFO:tasks.workunit.client.1.smithi190.stdout:1/254: mknod d3/da/d2e/c58 0 2024-03-26T18:54:28.089 INFO:tasks.workunit.client.1.smithi190.stdout:3/257: getdents d1/d3/df/d18 0 2024-03-26T18:54:28.089 INFO:tasks.workunit.client.1.smithi190.stdout:3/258: truncate d1/d3/df/f2a 1382572 0 2024-03-26T18:54:28.089 INFO:tasks.workunit.client.1.smithi190.stdout:3/259: chown d1/d3/df/d18/d1b/l37 45240 1 2024-03-26T18:54:28.090 INFO:tasks.workunit.client.1.smithi190.stdout:3/260: write d1/d3/f4b [1855225,61218] 0 2024-03-26T18:54:28.091 INFO:tasks.workunit.client.1.smithi190.stdout:7/287: symlink d3/d6/l6d 0 2024-03-26T18:54:28.091 INFO:tasks.workunit.client.1.smithi190.stdout:7/288: creat d3/d6/d9/d1f/f6e x:0 0 0 2024-03-26T18:54:28.091 INFO:tasks.workunit.client.1.smithi190.stdout:7/289: chown d3/d6/d9/d1f/d41/f44 1900 1 2024-03-26T18:54:28.091 INFO:tasks.workunit.client.1.smithi190.stdout:7/290: creat d3/d6/d9/d1f/d41/d50/f6f x:0 0 0 2024-03-26T18:54:28.091 INFO:tasks.workunit.client.1.smithi190.stdout:7/291: creat d3/d6/d9/d14/d36/f70 x:0 0 0 2024-03-26T18:54:28.093 INFO:tasks.workunit.client.1.smithi190.stdout:3/261: creat d1/de/d4e/f54 x:0 0 0 2024-03-26T18:54:28.094 INFO:tasks.workunit.client.1.smithi190.stdout:8/276: dread f6 [4194304,4194304] 0 2024-03-26T18:54:28.095 INFO:tasks.workunit.client.1.smithi190.stdout:2/296: sync 2024-03-26T18:54:28.095 INFO:tasks.workunit.client.1.smithi190.stdout:6/232: sync 2024-03-26T18:54:28.095 INFO:tasks.workunit.client.1.smithi190.stdout:5/186: sync 2024-03-26T18:54:28.096 INFO:tasks.workunit.client.1.smithi190.stdout:6/233: chown d17 3205862 1 2024-03-26T18:54:28.096 INFO:tasks.workunit.client.1.smithi190.stdout:2/297: stat d6/d9/d3a/d3d/d2e 0 2024-03-26T18:54:28.096 INFO:tasks.workunit.client.1.smithi190.stdout:2/298: creat d6/d9/d48/f68 x:0 0 0 2024-03-26T18:54:28.096 INFO:tasks.workunit.client.0.smithi102.stdout:5/143: dwrite d2/f1d [0,4194304] 0 2024-03-26T18:54:28.096 INFO:tasks.workunit.client.0.smithi102.stdout:5/144: fdatasync d2/ff 0 2024-03-26T18:54:28.097 INFO:tasks.workunit.client.1.smithi190.stdout:4/299: dread d7/d8/d14/f5b [0,4194304] 0 2024-03-26T18:54:28.099 INFO:tasks.workunit.client.1.smithi190.stdout:7/292: mkdir d3/d6/d9/d1f/d4c/d71 0 2024-03-26T18:54:28.099 INFO:tasks.workunit.client.1.smithi190.stdout:7/293: creat d3/d6/d9/d14/d36/f72 x:0 0 0 2024-03-26T18:54:28.099 INFO:tasks.workunit.client.1.smithi190.stdout:7/294: fsync d3/d6/d9/d14/d59/f61 0 2024-03-26T18:54:28.104 INFO:tasks.workunit.client.1.smithi190.stdout:3/262: symlink d1/d3/df/d18/d35/d41/d51/l55 0 2024-03-26T18:54:28.105 INFO:tasks.workunit.client.1.smithi190.stdout:8/277: link d9/dd/d23/d36/d37/c4c d9/d39/c56 0 2024-03-26T18:54:28.107 INFO:tasks.workunit.client.1.smithi190.stdout:6/234: symlink d17/d28/d46/d4d/d51/l52 0 2024-03-26T18:54:28.112 INFO:tasks.workunit.client.1.smithi190.stdout:7/295: mkdir d3/d6/d9/d14/d59/d73 0 2024-03-26T18:54:28.113 INFO:tasks.workunit.client.1.smithi190.stdout:3/263: creat d1/d3/df/d31/f56 x:0 0 0 2024-03-26T18:54:28.115 INFO:tasks.workunit.client.1.smithi190.stdout:8/278: rmdir d9/dd/d15/d41 0 2024-03-26T18:54:28.116 INFO:tasks.workunit.client.1.smithi190.stdout:8/279: write d9/dd/d15/f30 [199090,4032] 0 2024-03-26T18:54:28.117 INFO:tasks.workunit.client.1.smithi190.stdout:2/299: getdents d6 0 2024-03-26T18:54:28.120 INFO:tasks.workunit.client.1.smithi190.stdout:7/296: mknod d3/d6/d9/d1f/d41/d50/c74 0 2024-03-26T18:54:28.122 INFO:tasks.workunit.client.1.smithi190.stdout:8/280: symlink d9/dd/d23/d36/d37/d3a/d43/l57 0 2024-03-26T18:54:28.124 INFO:tasks.workunit.client.1.smithi190.stdout:2/300: creat d6/d21/d61/f69 x:0 0 0 2024-03-26T18:54:28.124 INFO:tasks.workunit.client.1.smithi190.stdout:2/301: stat d6/d9/d3a/d3d/d1c/d44/f4c 0 2024-03-26T18:54:28.126 INFO:tasks.workunit.client.1.smithi190.stdout:2/302: mknod d6/d9/d3a/d3d/c6a 0 2024-03-26T18:54:28.126 INFO:tasks.workunit.client.1.smithi190.stdout:2/303: readlink d6/d9/d10/l1f 0 2024-03-26T18:54:28.126 INFO:tasks.workunit.client.1.smithi190.stdout:2/304: fsync d6/d9/f41 0 2024-03-26T18:54:28.126 INFO:tasks.workunit.client.1.smithi190.stdout:2/305: dread - d6/f4d zero size 2024-03-26T18:54:28.141 INFO:tasks.workunit.client.1.smithi190.stdout:3/264: dread d1/de/f2d [0,4194304] 0 2024-03-26T18:54:28.225 INFO:tasks.workunit.client.1.smithi190.stdout:3/265: truncate d1/de/f4f 633651 0 2024-03-26T18:54:28.225 INFO:tasks.workunit.client.1.smithi190.stdout:1/255: dread d3/da/f24 [0,4194304] 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/145: dwrite d2/d25/f26 [0,4194304] 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/146: mknod d2/d14/c33 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/147: fdatasync d2/d14/d1a/f1b 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:3/118: dwrite df/f1b [0,4194304] 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/148: symlink d2/d25/l34 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/149: fsync d2/fa 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:3/119: dread df/d29/f2c [0,4194304] 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/150: symlink d2/d14/l35 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/151: mknod d2/c36 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/152: creat d2/d25/d2a/f37 x:0 0 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/153: write d2/d25/d2a/f2e [1029197,15355] 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/154: mkdir d2/d2c/d38 0 2024-03-26T18:54:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/155: symlink d2/d2c/d38/l39 0 2024-03-26T18:54:28.231 INFO:tasks.workunit.client.0.smithi102.stdout:5/156: creat d2/f3a x:0 0 0 2024-03-26T18:54:28.231 INFO:tasks.workunit.client.0.smithi102.stdout:5/157: creat d2/d2c/d38/f3b x:0 0 0 2024-03-26T18:54:28.231 INFO:tasks.workunit.client.0.smithi102.stdout:5/158: dwrite d2/d25/d2a/f31 [0,4194304] 0 2024-03-26T18:54:28.231 INFO:tasks.workunit.client.0.smithi102.stdout:5/159: chown d2/d25/d2a/f2e 479520 1 2024-03-26T18:54:28.231 INFO:tasks.workunit.client.0.smithi102.stdout:5/160: creat d2/f3c x:0 0 0 2024-03-26T18:54:28.231 INFO:tasks.workunit.client.0.smithi102.stdout:5/161: symlink d2/d25/d2a/l3d 0 2024-03-26T18:54:28.231 INFO:tasks.workunit.client.0.smithi102.stdout:5/162: chown d2/d2c/d38/f3b 5 1 2024-03-26T18:54:28.231 INFO:tasks.workunit.client.0.smithi102.stdout:5/163: dread d2/d14/d1a/f1b [0,4194304] 0 2024-03-26T18:54:28.231 INFO:tasks.workunit.client.0.smithi102.stdout:5/164: unlink d2/d14/l35 0 2024-03-26T18:54:28.232 INFO:tasks.workunit.client.0.smithi102.stdout:5/165: dread d2/d14/d1e/f23 [0,4194304] 0 2024-03-26T18:54:28.235 INFO:tasks.workunit.client.1.smithi190.stdout:4/300: dread d7/d8/d14/f57 [0,4194304] 0 2024-03-26T18:54:28.235 INFO:tasks.workunit.client.1.smithi190.stdout:1/256: dread d3/d9/f41 [0,4194304] 0 2024-03-26T18:54:28.235 INFO:tasks.workunit.client.1.smithi190.stdout:1/257: chown d3/da/d45 106 1 2024-03-26T18:54:28.235 INFO:tasks.workunit.client.1.smithi190.stdout:4/301: mknod d7/d8/d14/d23/d28/d38/c65 0 2024-03-26T18:54:28.235 INFO:tasks.workunit.client.1.smithi190.stdout:1/258: mkdir d3/da/d2e/d59 0 2024-03-26T18:54:28.235 INFO:tasks.workunit.client.1.smithi190.stdout:7/297: dread d3/f35 [0,4194304] 0 2024-03-26T18:54:28.235 INFO:tasks.workunit.client.1.smithi190.stdout:3/266: dread d1/de/f2f [0,4194304] 0 2024-03-26T18:54:28.235 INFO:tasks.workunit.client.1.smithi190.stdout:7/298: rename d3/d6/d9/d1f/f6e to d3/d6/d9/f75 0 2024-03-26T18:54:28.235 INFO:tasks.workunit.client.1.smithi190.stdout:7/299: fsync d3/d1b/f5e 0 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:7/300: stat d3/d6/d9/d1f/d41/f55 0 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:7/301: dread - d3/d6/d9/d14/d36/f6b zero size 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:7/302: dread - d3/d6/d9/d1f/f6c zero size 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:7/303: write d3/d6/f5d [880651,59802] 0 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:7/304: chown d3/f2b 272 1 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:1/259: dread d3/d9/db/f32 [0,4194304] 0 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:5/187: dwrite d2/d17/d1b/d22/f28 [0,4194304] 0 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:5/188: readlink d2/d17/d1b/d22/l32 0 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:2/306: dwrite d6/d21/d61/f69 [0,4194304] 0 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:6/235: dwrite d17/f21 [0,4194304] 0 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:3/267: rmdir d1/d3/df/d18 39 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:1/260: creat d3/da/dd/d2f/d34/d3e/d52/f5a x:0 0 0 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:7/305: write d3/f16 [811371,125907] 0 2024-03-26T18:54:28.236 INFO:tasks.workunit.client.1.smithi190.stdout:5/189: creat d2/d17/d1b/d1e/d3a/f40 x:0 0 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:4/302: dwrite d7/d1d/f26 [0,4194304] 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:6/236: creat d17/d28/d2c/f53 x:0 0 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:2/307: dwrite d6/d9/d10/f13 [0,4194304] 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:1/261: mkdir d3/da/dd/d2f/d5b 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:1/262: chown d3/d9/db/f32 3087 1 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:5/190: symlink d2/d17/d1b/d1e/l41 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:4/303: creat d7/d1d/d42/d4a/f66 x:0 0 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:6/237: creat d17/d28/d46/d4d/d51/f54 x:0 0 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:6/238: fsync d17/f21 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:3/268: truncate d1/d3/df/f2a 457024 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:5/191: rename d2/d17/c2f to d2/d17/c42 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:4/304: truncate d7/d8/d14/f57 4102987 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:6/239: creat d17/d28/d2c/d50/f55 x:0 0 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:3/269: mkdir d1/d3/df/d18/d57 0 2024-03-26T18:54:28.237 INFO:tasks.workunit.client.1.smithi190.stdout:4/305: rename d7/d8/d14/d23/d28/l37 to d7/d8/d1c/d20/l67 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:6/240: creat d17/d28/d46/f56 x:0 0 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:4/306: unlink d7/f10 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:6/241: mknod d17/d28/d46/d4d/c57 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:6/242: truncate fc 1540677 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:6/243: stat d17/d28/d2c/f47 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:6/244: unlink c11 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:6/245: creat d17/d28/f58 x:0 0 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:6/246: truncate f8 2569967 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:3/270: write d1/f4 [1157936,124498] 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:3/271: readlink d1/l23 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:3/272: write d1/f3b [3152327,80028] 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:3/273: chown d1/d3 0 1 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:1/263: dwrite d3/d9/f42 [0,4194304] 0 2024-03-26T18:54:28.238 INFO:tasks.workunit.client.1.smithi190.stdout:1/264: write d3/d9/f57 [354178,28612] 0 2024-03-26T18:54:28.239 INFO:tasks.workunit.client.1.smithi190.stdout:1/265: truncate d3/da/f37 188226 0 2024-03-26T18:54:28.239 INFO:tasks.workunit.client.1.smithi190.stdout:2/308: dwrite d6/f4d [0,4194304] 0 2024-03-26T18:54:28.239 INFO:tasks.workunit.client.1.smithi190.stdout:5/192: dwrite d2/d17/d1b/d30/f3f [0,4194304] 0 2024-03-26T18:54:28.239 INFO:tasks.workunit.client.1.smithi190.stdout:6/247: rename d17/d28/d46/c48 to d17/d28/d46/c59 0 2024-03-26T18:54:28.239 INFO:tasks.workunit.client.1.smithi190.stdout:7/306: truncate d3/d1b/f3b 1467944 0 2024-03-26T18:54:28.239 INFO:tasks.workunit.client.1.smithi190.stdout:4/307: dwrite d7/f35 [0,4194304] 0 2024-03-26T18:54:28.239 INFO:tasks.workunit.client.1.smithi190.stdout:2/309: mkdir d6/d9/d45/d6b 0 2024-03-26T18:54:28.239 INFO:tasks.workunit.client.1.smithi190.stdout:4/308: write d7/d8/d14/f19 [262995,104366] 0 2024-03-26T18:54:28.240 INFO:tasks.workunit.client.0.smithi102.stdout:5/166: creat d2/d14/d1e/f3e x:0 0 0 2024-03-26T18:54:28.240 INFO:tasks.workunit.client.1.smithi190.stdout:4/309: dread d7/d8/f2a [0,4194304] 0 2024-03-26T18:54:28.240 INFO:tasks.workunit.client.1.smithi190.stdout:5/193: creat d2/d17/d1b/d1e/d3a/d3c/f43 x:0 0 0 2024-03-26T18:54:28.240 INFO:tasks.workunit.client.1.smithi190.stdout:5/194: creat d2/d17/f44 x:0 0 0 2024-03-26T18:54:28.240 INFO:tasks.workunit.client.1.smithi190.stdout:6/248: getdents d17/d28/d2b 0 2024-03-26T18:54:28.240 INFO:tasks.workunit.client.1.smithi190.stdout:7/307: symlink d3/d6/d9/d1f/l76 0 2024-03-26T18:54:28.241 INFO:tasks.workunit.client.1.smithi190.stdout:7/308: chown d3/d6/d9/d1f/d41/f44 191327 1 2024-03-26T18:54:28.242 INFO:tasks.workunit.client.1.smithi190.stdout:3/274: dwrite d1/de/d1c/d36/f3a [0,4194304] 0 2024-03-26T18:54:28.243 INFO:tasks.workunit.client.0.smithi102.stdout:3/120: dwrite df/f1b [4194304,4194304] 0 2024-03-26T18:54:28.243 INFO:tasks.workunit.client.0.smithi102.stdout:3/121: chown fd 50304352 1 2024-03-26T18:54:28.244 INFO:tasks.workunit.client.1.smithi190.stdout:0/380: truncate d0/dd/d20/d1c/f6e 1370106 0 2024-03-26T18:54:28.244 INFO:tasks.workunit.client.1.smithi190.stdout:0/381: fsync d0/dd/d20/d1c/d4e/f4f 0 2024-03-26T18:54:28.244 INFO:tasks.workunit.client.1.smithi190.stdout:0/382: dread - d0/dd/d20/f68 zero size 2024-03-26T18:54:28.244 INFO:tasks.workunit.client.1.smithi190.stdout:0/383: write d0/dd/d24/f2f [3878441,60847] 0 2024-03-26T18:54:28.251 INFO:tasks.workunit.client.1.smithi190.stdout:2/310: mkdir d6/d21/d61/d6c 0 2024-03-26T18:54:28.251 INFO:tasks.workunit.client.1.smithi190.stdout:2/311: chown d6/d9/d3a/d3d/d1c/d44/f4c 11924 1 2024-03-26T18:54:28.251 INFO:tasks.workunit.client.1.smithi190.stdout:2/312: rename d6/d21/d61 to d6/d21/d61/d6c/d6d 22 2024-03-26T18:54:28.252 INFO:tasks.workunit.client.1.smithi190.stdout:4/310: link d7/d8/d1c/l34 d7/d1d/l68 0 2024-03-26T18:54:28.257 INFO:tasks.workunit.client.1.smithi190.stdout:5/195: creat d2/f45 x:0 0 0 2024-03-26T18:54:28.260 INFO:tasks.workunit.client.0.smithi102.stdout:3/122: dwrite df/f25 [0,4194304] 0 2024-03-26T18:54:28.260 INFO:tasks.workunit.client.1.smithi190.stdout:3/275: creat d1/de/d1c/d36/f58 x:0 0 0 2024-03-26T18:54:28.263 INFO:tasks.workunit.client.1.smithi190.stdout:2/313: creat d6/d9/d3a/d3d/d2e/f6e x:0 0 0 2024-03-26T18:54:28.269 INFO:tasks.workunit.client.1.smithi190.stdout:6/249: read d17/d28/d2c/f4c [1681210,117095] 0 2024-03-26T18:54:28.269 INFO:tasks.workunit.client.1.smithi190.stdout:6/250: write f1 [56922,56668] 0 2024-03-26T18:54:28.269 INFO:tasks.workunit.client.1.smithi190.stdout:5/196: creat d2/d17/d1b/d22/d24/d3b/f46 x:0 0 0 2024-03-26T18:54:28.271 INFO:tasks.workunit.client.1.smithi190.stdout:7/309: rmdir d3/d1b/d57 39 2024-03-26T18:54:28.275 INFO:tasks.workunit.client.1.smithi190.stdout:3/276: unlink d1/de/f4f 0 2024-03-26T18:54:28.275 INFO:tasks.workunit.client.1.smithi190.stdout:3/277: readlink d1/d3/df/d18/d1b/l27 0 2024-03-26T18:54:28.278 INFO:tasks.workunit.client.0.smithi102.stdout:3/123: dwrite df/f26 [0,4194304] 0 2024-03-26T18:54:28.279 INFO:tasks.workunit.client.1.smithi190.stdout:6/251: symlink d17/d28/d46/l5a 0 2024-03-26T18:54:28.279 INFO:tasks.workunit.client.1.smithi190.stdout:5/197: dread d2/f10 [0,4194304] 0 2024-03-26T18:54:28.279 INFO:tasks.workunit.client.1.smithi190.stdout:2/314: unlink d6/fe 0 2024-03-26T18:54:28.279 INFO:tasks.workunit.client.1.smithi190.stdout:6/252: dread - d17/f24 zero size 2024-03-26T18:54:28.279 INFO:tasks.workunit.client.1.smithi190.stdout:5/198: creat d2/d17/d1b/d22/d24/d3b/f47 x:0 0 0 2024-03-26T18:54:28.279 INFO:tasks.workunit.client.1.smithi190.stdout:6/253: write d17/d28/d46/f56 [107041,41234] 0 2024-03-26T18:54:28.279 INFO:tasks.workunit.client.1.smithi190.stdout:5/199: chown d2/d17/f21 17373078 1 2024-03-26T18:54:28.280 INFO:tasks.workunit.client.1.smithi190.stdout:6/254: readlink d17/l30 0 2024-03-26T18:54:28.281 INFO:tasks.workunit.client.1.smithi190.stdout:6/255: write d17/d28/d46/f56 [397210,46592] 0 2024-03-26T18:54:28.281 INFO:tasks.workunit.client.1.smithi190.stdout:3/278: mknod d1/de/d1c/c59 0 2024-03-26T18:54:28.283 INFO:tasks.workunit.client.1.smithi190.stdout:5/200: dread d2/f12 [0,4194304] 0 2024-03-26T18:54:28.284 INFO:tasks.workunit.client.1.smithi190.stdout:7/310: dread d3/d6/d9/d14/f33 [0,4194304] 0 2024-03-26T18:54:28.284 INFO:tasks.workunit.client.0.smithi102.stdout:5/167: rmdir d2 39 2024-03-26T18:54:28.285 INFO:tasks.workunit.client.1.smithi190.stdout:3/279: write d1/de/f2f [845128,5424] 0 2024-03-26T18:54:28.285 INFO:tasks.workunit.client.0.smithi102.stdout:3/124: rename df/d1c/c1e to df/d1c/c35 0 2024-03-26T18:54:28.285 INFO:tasks.workunit.client.0.smithi102.stdout:3/125: truncate f7 4249670 0 2024-03-26T18:54:28.287 INFO:tasks.workunit.client.0.smithi102.stdout:5/168: symlink d2/l3f 0 2024-03-26T18:54:28.287 INFO:tasks.workunit.client.0.smithi102.stdout:5/169: dread - d2/f3c zero size 2024-03-26T18:54:28.289 INFO:tasks.workunit.client.1.smithi190.stdout:2/315: rename d6/d9/d3a/d3d/d1c/d44/l60 to d6/d9/d3a/d3d/d67/l6f 0 2024-03-26T18:54:28.289 INFO:tasks.workunit.client.0.smithi102.stdout:5/170: read d2/d14/d1a/f1b [186194,11243] 0 2024-03-26T18:54:28.289 INFO:tasks.workunit.client.0.smithi102.stdout:5/171: fsync d2/d14/f2d 0 2024-03-26T18:54:28.289 INFO:tasks.workunit.client.0.smithi102.stdout:5/172: fsync d2/d25/f2b 0 2024-03-26T18:54:28.290 INFO:tasks.workunit.client.0.smithi102.stdout:3/126: mkdir df/d29/d31/d36 0 2024-03-26T18:54:28.291 INFO:tasks.workunit.client.0.smithi102.stdout:3/127: dread df/f17 [0,4194304] 0 2024-03-26T18:54:28.291 INFO:tasks.workunit.client.1.smithi190.stdout:9/527: sync 2024-03-26T18:54:28.291 INFO:tasks.workunit.client.1.smithi190.stdout:8/281: sync 2024-03-26T18:54:28.292 INFO:tasks.workunit.client.0.smithi102.stdout:5/173: symlink d2/d2c/d38/l40 0 2024-03-26T18:54:28.293 INFO:tasks.workunit.client.1.smithi190.stdout:5/201: link d2/d17/f21 d2/d17/d1b/d1e/d3a/f48 0 2024-03-26T18:54:28.295 INFO:tasks.workunit.client.0.smithi102.stdout:3/128: truncate fd 1825228 0 2024-03-26T18:54:28.297 INFO:tasks.workunit.client.1.smithi190.stdout:7/311: mknod d3/d1b/d57/c77 0 2024-03-26T18:54:28.297 INFO:tasks.workunit.client.1.smithi190.stdout:5/202: dread d2/d17/f21 [0,4194304] 0 2024-03-26T18:54:28.297 INFO:tasks.workunit.client.1.smithi190.stdout:5/203: chown d2/f16 138966 1 2024-03-26T18:54:28.298 INFO:tasks.workunit.client.0.smithi102.stdout:5/174: rmdir d2/d25/d2a 39 2024-03-26T18:54:28.298 INFO:tasks.workunit.client.0.smithi102.stdout:5/175: truncate d2/f3c 401149 0 2024-03-26T18:54:28.298 INFO:tasks.workunit.client.0.smithi102.stdout:5/176: truncate d2/d25/d2a/f37 224759 0 2024-03-26T18:54:28.299 INFO:tasks.workunit.client.1.smithi190.stdout:3/280: mkdir d1/d3/d5a 0 2024-03-26T18:54:28.299 INFO:tasks.workunit.client.1.smithi190.stdout:3/281: write d1/de/d1c/d36/f40 [636007,82390] 0 2024-03-26T18:54:28.299 INFO:tasks.workunit.client.1.smithi190.stdout:3/282: stat d1/f3b 0 2024-03-26T18:54:28.301 INFO:tasks.workunit.client.0.smithi102.stdout:3/129: creat df/d1c/d34/f37 x:0 0 0 2024-03-26T18:54:28.303 INFO:tasks.workunit.client.1.smithi190.stdout:2/316: unlink d6/c17 0 2024-03-26T18:54:28.303 INFO:tasks.workunit.client.1.smithi190.stdout:0/384: dread d0/dd/d24/f7a [0,4194304] 0 2024-03-26T18:54:28.303 INFO:tasks.workunit.client.1.smithi190.stdout:6/256: mkdir d17/d5b 0 2024-03-26T18:54:28.303 INFO:tasks.workunit.client.1.smithi190.stdout:6/257: creat d17/d28/d46/f5c x:0 0 0 2024-03-26T18:54:28.305 INFO:tasks.workunit.client.1.smithi190.stdout:8/282: link d9/d16/f2b d9/d48/f58 0 2024-03-26T18:54:28.305 INFO:tasks.workunit.client.1.smithi190.stdout:8/283: truncate d9/d48/f58 268294 0 2024-03-26T18:54:28.305 INFO:tasks.workunit.client.1.smithi190.stdout:9/528: symlink d12/lbd 0 2024-03-26T18:54:28.306 INFO:tasks.workunit.client.1.smithi190.stdout:5/204: write d2/fa [2701300,122048] 0 2024-03-26T18:54:28.313 INFO:tasks.workunit.client.1.smithi190.stdout:7/312: mknod d3/d6/d9/d1f/d41/d50/c78 0 2024-03-26T18:54:28.314 INFO:tasks.workunit.client.1.smithi190.stdout:7/313: creat d3/d6/d9/d14/d36/f79 x:0 0 0 2024-03-26T18:54:28.319 INFO:tasks.workunit.client.1.smithi190.stdout:6/258: rmdir d17/d5b 0 2024-03-26T18:54:28.319 INFO:tasks.workunit.client.1.smithi190.stdout:6/259: fsync f2 0 2024-03-26T18:54:28.319 INFO:tasks.workunit.client.1.smithi190.stdout:6/260: chown d17/d28/f58 5185415 1 2024-03-26T18:54:28.319 INFO:tasks.workunit.client.1.smithi190.stdout:2/317: dread d6/d9/d10/f2c [0,4194304] 0 2024-03-26T18:54:28.320 INFO:tasks.workunit.client.1.smithi190.stdout:8/284: creat d9/dd/f59 x:0 0 0 2024-03-26T18:54:28.320 INFO:tasks.workunit.client.1.smithi190.stdout:8/285: creat d9/dd/f5a x:0 0 0 2024-03-26T18:54:28.320 INFO:tasks.workunit.client.1.smithi190.stdout:8/286: chown d9/dd/d15/f2f 450129760 1 2024-03-26T18:54:28.324 INFO:tasks.workunit.client.1.smithi190.stdout:5/205: symlink d2/d17/d1b/d1e/l49 0 2024-03-26T18:54:28.326 INFO:tasks.workunit.client.1.smithi190.stdout:7/314: read d3/d6/f1d [5917973,90639] 0 2024-03-26T18:54:28.327 INFO:tasks.workunit.client.1.smithi190.stdout:0/385: getdents d0/dd/d20/d1c 0 2024-03-26T18:54:28.330 INFO:tasks.workunit.client.1.smithi190.stdout:2/318: link d6/d9/d10/f34 d6/d9/d10/f70 0 2024-03-26T18:54:28.331 INFO:tasks.workunit.client.1.smithi190.stdout:3/283: dwrite d1/f42 [0,4194304] 0 2024-03-26T18:54:28.332 INFO:tasks.workunit.client.1.smithi190.stdout:3/284: truncate d1/f26 383291 0 2024-03-26T18:54:28.333 INFO:tasks.workunit.client.0.smithi102.stdout:3/130: dwrite df/f20 [0,4194304] 0 2024-03-26T18:54:28.333 INFO:tasks.workunit.client.0.smithi102.stdout:3/131: chown df/l14 460764103 1 2024-03-26T18:54:28.334 INFO:tasks.workunit.client.1.smithi190.stdout:5/206: truncate d2/d17/d1b/d22/f3d 4183836 0 2024-03-26T18:54:28.336 INFO:tasks.workunit.client.1.smithi190.stdout:0/386: rename d0/dc to d0/dd/d20/d1c/d8b 0 2024-03-26T18:54:28.336 INFO:tasks.workunit.client.1.smithi190.stdout:0/387: chown d0/dd/d24/d4b/d5e/d6b/l84 14 1 2024-03-26T18:54:28.336 INFO:tasks.workunit.client.1.smithi190.stdout:0/388: fdatasync d0/dd/f22 0 2024-03-26T18:54:28.337 INFO:tasks.workunit.client.0.smithi102.stdout:5/177: dwrite d2/d25/f2b [0,4194304] 0 2024-03-26T18:54:28.338 INFO:tasks.workunit.client.1.smithi190.stdout:2/319: mknod d6/d21/d61/c71 0 2024-03-26T18:54:28.339 INFO:tasks.workunit.client.1.smithi190.stdout:7/315: dread d3/f2f [0,4194304] 0 2024-03-26T18:54:28.340 INFO:tasks.workunit.client.0.smithi102.stdout:5/178: dread d2/f10 [0,4194304] 0 2024-03-26T18:54:28.340 INFO:tasks.workunit.client.0.smithi102.stdout:5/179: chown d2/c8 253 1 2024-03-26T18:54:28.341 INFO:tasks.workunit.client.0.smithi102.stdout:3/132: getdents df/d1c 0 2024-03-26T18:54:28.343 INFO:tasks.workunit.client.0.smithi102.stdout:5/180: creat d2/d14/d1a/f41 x:0 0 0 2024-03-26T18:54:28.343 INFO:tasks.workunit.client.0.smithi102.stdout:5/181: read - d2/f3a zero size 2024-03-26T18:54:28.345 INFO:tasks.workunit.client.0.smithi102.stdout:3/133: mknod df/d1c/d34/c38 0 2024-03-26T18:54:28.347 INFO:tasks.workunit.client.0.smithi102.stdout:3/134: creat df/f39 x:0 0 0 2024-03-26T18:54:28.350 INFO:tasks.workunit.client.1.smithi190.stdout:8/287: dwrite d9/fa [0,4194304] 0 2024-03-26T18:54:28.350 INFO:tasks.workunit.client.1.smithi190.stdout:3/285: unlink d1/de/f2f 0 2024-03-26T18:54:28.351 INFO:tasks.workunit.client.1.smithi190.stdout:1/266: sync 2024-03-26T18:54:28.352 INFO:tasks.workunit.client.1.smithi190.stdout:5/207: link d2/d17/d1b/d1e/l49 d2/d17/l4a 0 2024-03-26T18:54:28.353 INFO:tasks.workunit.client.0.smithi102.stdout:5/182: read d2/f9 [860040,95429] 0 2024-03-26T18:54:28.353 INFO:tasks.workunit.client.0.smithi102.stdout:5/183: readlink d2/l3f 0 2024-03-26T18:54:28.353 INFO:tasks.workunit.client.0.smithi102.stdout:5/184: truncate d2/d25/d2a/f2e 1600022 0 2024-03-26T18:54:28.353 INFO:tasks.workunit.client.1.smithi190.stdout:0/389: write d0/dd/d20/d1c/f29 [1477560,76110] 0 2024-03-26T18:54:28.355 INFO:tasks.workunit.client.1.smithi190.stdout:2/320: mkdir d6/d9/d1a/d72 0 2024-03-26T18:54:28.355 INFO:tasks.workunit.client.1.smithi190.stdout:2/321: chown d6/d9/d1a/f26 4015 1 2024-03-26T18:54:28.355 INFO:tasks.workunit.client.1.smithi190.stdout:2/322: chown d6/d9/f38 91 1 2024-03-26T18:54:28.356 INFO:tasks.workunit.client.0.smithi102.stdout:5/185: creat d2/d14/d1e/f42 x:0 0 0 2024-03-26T18:54:28.359 INFO:tasks.workunit.client.1.smithi190.stdout:7/316: mknod d3/d1b/d57/c7a 0 2024-03-26T18:54:28.360 INFO:tasks.workunit.client.1.smithi190.stdout:8/288: symlink d9/d16/l5b 0 2024-03-26T18:54:28.360 INFO:tasks.workunit.client.1.smithi190.stdout:8/289: creat d9/dd/d15/d32/f5c x:0 0 0 2024-03-26T18:54:28.363 INFO:tasks.workunit.client.1.smithi190.stdout:6/261: write fc [2367728,116483] 0 2024-03-26T18:54:28.363 INFO:tasks.workunit.client.0.smithi102.stdout:5/186: mkdir d2/d2c/d38/d43 0 2024-03-26T18:54:28.365 INFO:tasks.workunit.client.1.smithi190.stdout:5/208: symlink d2/d17/d1b/d22/d24/d3b/l4b 0 2024-03-26T18:54:28.367 INFO:tasks.workunit.client.1.smithi190.stdout:3/286: dwrite d1/d3/df/f19 [0,4194304] 0 2024-03-26T18:54:28.367 INFO:tasks.workunit.client.1.smithi190.stdout:3/287: fsync d1/fb 0 2024-03-26T18:54:28.368 INFO:tasks.workunit.client.0.smithi102.stdout:3/135: dwrite df/f17 [0,4194304] 0 2024-03-26T18:54:28.368 INFO:tasks.workunit.client.1.smithi190.stdout:0/390: truncate d0/dd/d20/d1c/d46/f5d 4796679 0 2024-03-26T18:54:28.368 INFO:tasks.workunit.client.1.smithi190.stdout:0/391: fdatasync d0/dd/d24/f54 0 2024-03-26T18:54:28.372 INFO:tasks.workunit.client.1.smithi190.stdout:2/323: stat d6/d9/d10/l15 0 2024-03-26T18:54:28.372 INFO:tasks.workunit.client.1.smithi190.stdout:2/324: dread - d6/d9/f4e zero size 2024-03-26T18:54:28.374 INFO:tasks.workunit.client.1.smithi190.stdout:7/317: symlink d3/d6/d9/d1f/d41/d50/d53/l7b 0 2024-03-26T18:54:28.374 INFO:tasks.workunit.client.1.smithi190.stdout:7/318: stat d3/d6/d9/d1f/d41/d50/d53 0 2024-03-26T18:54:28.378 INFO:tasks.workunit.client.1.smithi190.stdout:1/267: getdents d3/da 0 2024-03-26T18:54:28.379 INFO:tasks.workunit.client.0.smithi102.stdout:3/136: mkdir df/d29/d3a 0 2024-03-26T18:54:28.383 INFO:tasks.workunit.client.0.smithi102.stdout:3/137: unlink fa 0 2024-03-26T18:54:28.383 INFO:tasks.workunit.client.0.smithi102.stdout:3/138: chown df/d1c/l28 0 1 2024-03-26T18:54:28.383 INFO:tasks.workunit.client.0.smithi102.stdout:3/139: chown df/d29/f2c 0 1 2024-03-26T18:54:28.383 INFO:tasks.workunit.client.0.smithi102.stdout:3/140: rename df/d29 to df/d29/d31/d36/d3b 22 2024-03-26T18:54:28.383 INFO:tasks.workunit.client.0.smithi102.stdout:3/141: write f2 [1103876,47331] 0 2024-03-26T18:54:28.384 INFO:tasks.workunit.client.1.smithi190.stdout:4/311: sync 2024-03-26T18:54:28.384 INFO:tasks.workunit.client.1.smithi190.stdout:0/392: creat d0/dd/d20/d1c/f8c x:0 0 0 2024-03-26T18:54:28.384 INFO:tasks.workunit.client.0.smithi102.stdout:3/142: mkdir df/d29/d31/d36/d3c 0 2024-03-26T18:54:28.386 INFO:tasks.workunit.client.1.smithi190.stdout:2/325: creat d6/d9/d3a/d3d/d55/f73 x:0 0 0 2024-03-26T18:54:28.387 INFO:tasks.workunit.client.1.smithi190.stdout:7/319: unlink d3/f46 0 2024-03-26T18:54:28.388 INFO:tasks.workunit.client.1.smithi190.stdout:7/320: creat d3/f7c x:0 0 0 2024-03-26T18:54:28.388 INFO:tasks.workunit.client.0.smithi102.stdout:3/143: mknod df/d29/d31/c3d 0 2024-03-26T18:54:28.389 INFO:tasks.workunit.client.1.smithi190.stdout:6/262: rename d17/c1d to d17/d28/d46/d4d/c5d 0 2024-03-26T18:54:28.390 INFO:tasks.workunit.client.1.smithi190.stdout:2/326: creat d6/d9/d45/f74 x:0 0 0 2024-03-26T18:54:28.392 INFO:tasks.workunit.client.1.smithi190.stdout:5/209: rename d2/cb to d2/d17/d1b/d30/c4c 0 2024-03-26T18:54:28.392 INFO:tasks.workunit.client.1.smithi190.stdout:5/210: write d2/d17/f44 [611726,95670] 0 2024-03-26T18:54:28.393 INFO:tasks.workunit.client.1.smithi190.stdout:8/290: dwrite d9/dd/d15/d32/f35 [0,4194304] 0 2024-03-26T18:54:28.393 INFO:tasks.workunit.client.1.smithi190.stdout:4/312: symlink d7/d8/d1c/d20/d40/l69 0 2024-03-26T18:54:28.393 INFO:tasks.workunit.client.1.smithi190.stdout:9/529: dwrite d12/d2d/d1d/d23/d2a/d3f/d55/f62 [0,4194304] 0 2024-03-26T18:54:28.393 INFO:tasks.workunit.client.1.smithi190.stdout:9/530: chown d12/d2d/d1d/f6f 6 1 2024-03-26T18:54:28.394 INFO:tasks.workunit.client.1.smithi190.stdout:3/288: dwrite d1/d3/f34 [0,4194304] 0 2024-03-26T18:54:28.394 INFO:tasks.workunit.client.1.smithi190.stdout:3/289: write d1/fb [2783809,30781] 0 2024-03-26T18:54:28.395 INFO:tasks.workunit.client.1.smithi190.stdout:3/290: fsync d1/d3/f13 0 2024-03-26T18:54:28.395 INFO:tasks.workunit.client.1.smithi190.stdout:3/291: readlink d1/l1e 0 2024-03-26T18:54:28.395 INFO:tasks.workunit.client.0.smithi102.stdout:5/187: dwrite d2/f1d [0,4194304] 0 2024-03-26T18:54:28.396 INFO:tasks.workunit.client.1.smithi190.stdout:1/268: rename d3/da/dd/d2f/d34/d3e/f4e to d3/d9/f5c 0 2024-03-26T18:54:28.396 INFO:tasks.workunit.client.1.smithi190.stdout:6/263: rename d17 to d17/d28/d5e 22 2024-03-26T18:54:28.397 INFO:tasks.workunit.client.0.smithi102.stdout:5/188: symlink d2/d14/d1a/l44 0 2024-03-26T18:54:28.398 INFO:tasks.workunit.client.1.smithi190.stdout:8/291: getdents d9/dd/d23/d36/d37 0 2024-03-26T18:54:28.401 INFO:tasks.workunit.client.0.smithi102.stdout:5/189: symlink d2/d2c/d38/l45 0 2024-03-26T18:54:28.404 INFO:tasks.workunit.client.1.smithi190.stdout:9/531: creat d12/d80/fbe x:0 0 0 2024-03-26T18:54:28.404 INFO:tasks.workunit.client.1.smithi190.stdout:9/532: dread - d12/d2d/d1d/d23/d2a/d3f/f81 zero size 2024-03-26T18:54:28.408 INFO:tasks.workunit.client.1.smithi190.stdout:6/264: symlink d17/d28/d2b/l5f 0 2024-03-26T18:54:28.409 INFO:tasks.workunit.client.1.smithi190.stdout:8/292: mknod d9/dd/d23/c5d 0 2024-03-26T18:54:28.410 INFO:tasks.workunit.client.0.smithi102.stdout:3/144: dread f7 [0,4194304] 0 2024-03-26T18:54:28.411 INFO:tasks.workunit.client.1.smithi190.stdout:9/533: link d12/d2d/c3c d12/d2d/d1d/d23/d8e/cbf 0 2024-03-26T18:54:28.414 INFO:tasks.workunit.client.1.smithi190.stdout:6/265: link d17/f24 d17/d28/d2b/f60 0 2024-03-26T18:54:28.414 INFO:tasks.workunit.client.1.smithi190.stdout:6/266: creat d17/d28/d2c/f61 x:0 0 0 2024-03-26T18:54:28.414 INFO:tasks.workunit.client.1.smithi190.stdout:6/267: write d17/d28/f58 [803532,103079] 0 2024-03-26T18:54:28.416 INFO:tasks.workunit.client.1.smithi190.stdout:8/293: rmdir d9/d39 39 2024-03-26T18:54:28.416 INFO:tasks.workunit.client.0.smithi102.stdout:5/190: write d2/ff [1519018,92896] 0 2024-03-26T18:54:28.418 INFO:tasks.workunit.client.1.smithi190.stdout:9/534: mknod d12/d2d/d1d/d23/d2a/d71/daa/cc0 0 2024-03-26T18:54:28.421 INFO:tasks.workunit.client.1.smithi190.stdout:6/268: dread f8 [0,4194304] 0 2024-03-26T18:54:28.421 INFO:tasks.workunit.client.1.smithi190.stdout:6/269: truncate d17/d28/d2b/f60 432345 0 2024-03-26T18:54:28.421 INFO:tasks.workunit.client.0.smithi102.stdout:5/191: rename d2/d25/l32 to d2/d2c/d38/l46 0 2024-03-26T18:54:28.423 INFO:tasks.workunit.client.1.smithi190.stdout:8/294: mknod d9/d48/c5e 0 2024-03-26T18:54:28.423 INFO:tasks.workunit.client.1.smithi190.stdout:8/295: chown d9/dd/d23/c2e 506762769 1 2024-03-26T18:54:28.425 INFO:tasks.workunit.client.0.smithi102.stdout:3/145: rmdir df 39 2024-03-26T18:54:28.426 INFO:tasks.workunit.client.0.smithi102.stdout:3/146: write f2 [387661,103858] 0 2024-03-26T18:54:28.427 INFO:tasks.workunit.client.0.smithi102.stdout:5/192: rename d2/c5 to d2/d2c/d38/c47 0 2024-03-26T18:54:28.429 INFO:tasks.workunit.client.0.smithi102.stdout:3/147: creat df/d1c/f3e x:0 0 0 2024-03-26T18:54:28.429 INFO:tasks.workunit.client.0.smithi102.stdout:3/148: chown df/l2f 6048042 1 2024-03-26T18:54:28.429 INFO:tasks.workunit.client.0.smithi102.stdout:3/149: write f7 [5116185,8118] 0 2024-03-26T18:54:28.430 INFO:tasks.workunit.client.1.smithi190.stdout:7/321: dwrite d3/d6/d9/d14/d36/f6b [0,4194304] 0 2024-03-26T18:54:28.431 INFO:tasks.workunit.client.0.smithi102.stdout:3/150: truncate fd 316665 0 2024-03-26T18:54:28.432 INFO:tasks.workunit.client.0.smithi102.stdout:3/151: creat df/d29/d31/d36/d3c/f3f x:0 0 0 2024-03-26T18:54:28.433 INFO:tasks.workunit.client.0.smithi102.stdout:3/152: creat df/d29/f40 x:0 0 0 2024-03-26T18:54:28.434 INFO:tasks.workunit.client.0.smithi102.stdout:3/153: creat df/d29/d31/d36/f41 x:0 0 0 2024-03-26T18:54:28.434 INFO:tasks.workunit.client.0.smithi102.stdout:3/154: dread - df/d1c/f3e zero size 2024-03-26T18:54:28.434 INFO:tasks.workunit.client.0.smithi102.stdout:3/155: dread - df/f2e zero size 2024-03-26T18:54:28.434 INFO:tasks.workunit.client.0.smithi102.stdout:3/156: fdatasync df/d29/f33 0 2024-03-26T18:54:28.435 INFO:tasks.workunit.client.1.smithi190.stdout:6/270: write d17/f23 [2984207,60121] 0 2024-03-26T18:54:28.436 INFO:tasks.workunit.client.1.smithi190.stdout:6/271: chown d17/d28/d46/d4d 1 1 2024-03-26T18:54:28.437 INFO:tasks.workunit.client.1.smithi190.stdout:6/272: link d17/d28/d2b/l5f d17/l62 0 2024-03-26T18:54:28.438 INFO:tasks.workunit.client.1.smithi190.stdout:6/273: write d17/f24 [1158724,112088] 0 2024-03-26T18:54:28.438 INFO:tasks.workunit.client.1.smithi190.stdout:6/274: stat d17/d28/f2d 0 2024-03-26T18:54:28.439 INFO:tasks.workunit.client.1.smithi190.stdout:6/275: mkdir d17/d28/d2c/d63 0 2024-03-26T18:54:28.440 INFO:tasks.workunit.client.0.smithi102.stdout:3/157: dread df/f1a [0,4194304] 0 2024-03-26T18:54:28.440 INFO:tasks.workunit.client.0.smithi102.stdout:3/158: stat df/d29/d31/f32 0 2024-03-26T18:54:28.440 INFO:tasks.workunit.client.0.smithi102.stdout:3/159: truncate df/d29/d31/d36/f41 829468 0 2024-03-26T18:54:28.441 INFO:tasks.workunit.client.1.smithi190.stdout:6/276: unlink d17/d28/d2c/d50/f55 0 2024-03-26T18:54:28.441 INFO:tasks.workunit.client.1.smithi190.stdout:6/277: write d17/f4e [200455,4884] 0 2024-03-26T18:54:28.441 INFO:tasks.workunit.client.1.smithi190.stdout:6/278: readlink d17/d28/d2b/l31 0 2024-03-26T18:54:28.442 INFO:tasks.workunit.client.1.smithi190.stdout:7/322: read d3/d6/f56 [393382,78049] 0 2024-03-26T18:54:28.443 INFO:tasks.workunit.client.1.smithi190.stdout:6/279: stat d17/c1c 0 2024-03-26T18:54:28.443 INFO:tasks.workunit.client.1.smithi190.stdout:6/280: fdatasync d17/d28/d2c/f42 0 2024-03-26T18:54:28.443 INFO:tasks.workunit.client.1.smithi190.stdout:6/281: read d17/d28/d2c/f4c [1051907,71951] 0 2024-03-26T18:54:28.444 INFO:tasks.workunit.client.1.smithi190.stdout:6/282: mknod d17/d28/c64 0 2024-03-26T18:54:28.444 INFO:tasks.workunit.client.1.smithi190.stdout:6/283: dread - d17/d28/d38/f45 zero size 2024-03-26T18:54:28.444 INFO:tasks.workunit.client.1.smithi190.stdout:6/284: creat d17/d28/d2c/d50/f65 x:0 0 0 2024-03-26T18:54:28.446 INFO:tasks.workunit.client.1.smithi190.stdout:6/285: rename d17/f1b to d17/d28/d46/d4d/f66 0 2024-03-26T18:54:28.446 INFO:tasks.workunit.client.1.smithi190.stdout:6/286: fdatasync f9 0 2024-03-26T18:54:28.446 INFO:tasks.workunit.client.1.smithi190.stdout:6/287: dread - d17/d28/d46/d4d/d51/f54 zero size 2024-03-26T18:54:28.446 INFO:tasks.workunit.client.1.smithi190.stdout:6/288: creat d17/d28/d2c/d50/f67 x:0 0 0 2024-03-26T18:54:28.454 INFO:tasks.workunit.client.0.smithi102.stdout:3/160: dread df/f1b [0,4194304] 0 2024-03-26T18:54:28.454 INFO:tasks.workunit.client.1.smithi190.stdout:7/323: read d3/d6/d9/d14/d36/f3c [3947045,14287] 0 2024-03-26T18:54:28.455 INFO:tasks.workunit.client.1.smithi190.stdout:7/324: write d3/d6/d9/d14/d36/f70 [577903,46059] 0 2024-03-26T18:54:28.455 INFO:tasks.workunit.client.1.smithi190.stdout:6/289: write d17/f24 [880722,58293] 0 2024-03-26T18:54:28.455 INFO:tasks.workunit.client.0.smithi102.stdout:3/161: symlink df/d29/d31/d36/d3c/l42 0 2024-03-26T18:54:28.456 INFO:tasks.workunit.client.1.smithi190.stdout:4/313: dwrite d7/d8/d14/d23/d28/d38/f62 [0,4194304] 0 2024-03-26T18:54:28.457 INFO:tasks.workunit.client.1.smithi190.stdout:5/211: dwrite d2/f3 [0,4194304] 0 2024-03-26T18:54:28.457 INFO:tasks.workunit.client.1.smithi190.stdout:8/296: dwrite d9/d16/f1f [0,4194304] 0 2024-03-26T18:54:28.457 INFO:tasks.workunit.client.1.smithi190.stdout:7/325: rename d3/d6/d9/d1f/d41/d50/f6f to d3/d6/d9/d1f/d41/d65/f7d 0 2024-03-26T18:54:28.457 INFO:tasks.workunit.client.1.smithi190.stdout:7/326: stat d3/d6/d9/d1f/d4c 0 2024-03-26T18:54:28.457 INFO:tasks.workunit.client.1.smithi190.stdout:4/314: write d7/d1d/f27 [887695,108469] 0 2024-03-26T18:54:28.457 INFO:tasks.workunit.client.1.smithi190.stdout:4/315: readlink d7/d1d/l46 0 2024-03-26T18:54:28.457 INFO:tasks.workunit.client.1.smithi190.stdout:4/316: read d7/d8/d1c/f1f [937428,27277] 0 2024-03-26T18:54:28.458 INFO:tasks.workunit.client.0.smithi102.stdout:3/162: unlink f4 0 2024-03-26T18:54:28.458 INFO:tasks.workunit.client.0.smithi102.stdout:3/163: chown df/d29/d31/f32 4596317 1 2024-03-26T18:54:28.458 INFO:tasks.workunit.client.0.smithi102.stdout:3/164: write df/d29/f33 [1047311,59614] 0 2024-03-26T18:54:28.458 INFO:tasks.workunit.client.0.smithi102.stdout:3/165: write df/d1c/f3e [443258,46206] 0 2024-03-26T18:54:28.458 INFO:tasks.workunit.client.0.smithi102.stdout:5/193: rmdir d2/d14/d1a 39 2024-03-26T18:54:28.458 INFO:tasks.workunit.client.0.smithi102.stdout:5/194: read d2/f9 [4037332,29671] 0 2024-03-26T18:54:28.459 INFO:tasks.workunit.client.1.smithi190.stdout:1/269: dwrite d3/da/dd/f12 [0,4194304] 0 2024-03-26T18:54:28.459 INFO:tasks.workunit.client.1.smithi190.stdout:1/270: fsync d3/da/f47 0 2024-03-26T18:54:28.459 INFO:tasks.workunit.client.1.smithi190.stdout:3/292: dwrite d1/fb [4194304,4194304] 0 2024-03-26T18:54:28.463 INFO:tasks.workunit.client.0.smithi102.stdout:5/195: rename d2/d25/c30 to d2/d2c/c48 0 2024-03-26T18:54:28.464 INFO:tasks.workunit.client.1.smithi190.stdout:8/297: creat d9/dd/d23/d36/d37/f5f x:0 0 0 2024-03-26T18:54:28.464 INFO:tasks.workunit.client.1.smithi190.stdout:8/298: creat d9/dd/d15/f60 x:0 0 0 2024-03-26T18:54:28.467 INFO:tasks.workunit.client.1.smithi190.stdout:6/290: rename d17/d28/l3a to d17/d28/d46/d4d/d51/l68 0 2024-03-26T18:54:28.467 INFO:tasks.workunit.client.1.smithi190.stdout:6/291: write d17/f24 [2051070,53279] 0 2024-03-26T18:54:28.470 INFO:tasks.workunit.client.0.smithi102.stdout:5/196: rename d2/fd to d2/d2c/d38/f49 0 2024-03-26T18:54:28.471 INFO:tasks.workunit.client.1.smithi190.stdout:7/327: creat d3/d6/d9/d14/f7e x:0 0 0 2024-03-26T18:54:28.474 INFO:tasks.workunit.client.1.smithi190.stdout:5/212: dwrite d2/f3 [4194304,4194304] 0 2024-03-26T18:54:28.474 INFO:tasks.workunit.client.1.smithi190.stdout:5/213: creat d2/d17/d1b/d1e/d3a/f4d x:0 0 0 2024-03-26T18:54:28.475 INFO:tasks.workunit.client.0.smithi102.stdout:5/197: link d2/d25/l34 d2/d2c/d38/l4a 0 2024-03-26T18:54:28.476 INFO:tasks.workunit.client.0.smithi102.stdout:5/198: dread d2/d14/d1a/f1b [0,4194304] 0 2024-03-26T18:54:28.476 INFO:tasks.workunit.client.0.smithi102.stdout:5/199: getdents d2/d2c/d38/d43 0 2024-03-26T18:54:28.477 INFO:tasks.workunit.client.1.smithi190.stdout:1/271: mkdir d3/da/dd/d2f/d5b/d5d 0 2024-03-26T18:54:28.479 INFO:tasks.workunit.client.1.smithi190.stdout:8/299: dwrite d9/d16/f1e [0,4194304] 0 2024-03-26T18:54:28.479 INFO:tasks.workunit.client.1.smithi190.stdout:8/300: chown d9/dd/d15/f30 1787312546 1 2024-03-26T18:54:28.479 INFO:tasks.workunit.client.1.smithi190.stdout:8/301: write d9/d16/f2b [1263258,108140] 0 2024-03-26T18:54:28.479 INFO:tasks.workunit.client.0.smithi102.stdout:5/200: rename d2/d2c/d38/l39 to d2/l4b 0 2024-03-26T18:54:28.486 INFO:tasks.workunit.client.0.smithi102.stdout:5/201: rename d2/d25/d2a/f37 to d2/d2c/f4c 0 2024-03-26T18:54:28.486 INFO:tasks.workunit.client.0.smithi102.stdout:5/202: read d2/f10 [374240,20677] 0 2024-03-26T18:54:28.486 INFO:tasks.workunit.client.0.smithi102.stdout:5/203: chown d2 0 1 2024-03-26T18:54:28.487 INFO:tasks.workunit.client.0.smithi102.stdout:5/204: dread - d2/d14/f1c zero size 2024-03-26T18:54:28.487 INFO:tasks.workunit.client.0.smithi102.stdout:5/205: chown d2/d14/d1e/l28 1335039 1 2024-03-26T18:54:28.487 INFO:tasks.workunit.client.0.smithi102.stdout:5/206: chown d2/d14/f1c 1466 1 2024-03-26T18:54:28.487 INFO:tasks.workunit.client.0.smithi102.stdout:5/207: fsync d2/f3a 0 2024-03-26T18:54:28.487 INFO:tasks.workunit.client.0.smithi102.stdout:5/208: dread - d2/d14/d1a/f41 zero size 2024-03-26T18:54:28.491 INFO:tasks.workunit.client.0.smithi102.stdout:5/209: symlink d2/d2c/d38/l4d 0 2024-03-26T18:54:28.494 INFO:tasks.workunit.client.0.smithi102.stdout:5/210: rename d2/ff to d2/d2c/d38/d43/f4e 0 2024-03-26T18:54:28.498 INFO:tasks.workunit.client.1.smithi190.stdout:3/293: mkdir d1/d3/df/d31/d5b 0 2024-03-26T18:54:28.498 INFO:tasks.workunit.client.1.smithi190.stdout:3/294: readlink d1/d3/df/d18/d1b/l27 0 2024-03-26T18:54:28.498 INFO:tasks.workunit.client.1.smithi190.stdout:3/295: stat d1/l1e 0 2024-03-26T18:54:28.498 INFO:tasks.workunit.client.1.smithi190.stdout:8/302: dwrite f6 [0,4194304] 0 2024-03-26T18:54:28.499 INFO:tasks.workunit.client.0.smithi102.stdout:5/211: mkdir d2/d25/d2a/d4f 0 2024-03-26T18:54:28.499 INFO:tasks.workunit.client.0.smithi102.stdout:5/212: dread - d2/f3a zero size 2024-03-26T18:54:28.501 INFO:tasks.workunit.client.1.smithi190.stdout:6/292: rmdir d17 39 2024-03-26T18:54:28.502 INFO:tasks.workunit.client.0.smithi102.stdout:5/213: link d2/d14/f2d d2/d14/d1a/f50 0 2024-03-26T18:54:28.504 INFO:tasks.workunit.client.1.smithi190.stdout:7/328: creat d3/d6/d9/d1f/d4c/f7f x:0 0 0 2024-03-26T18:54:28.508 INFO:tasks.workunit.client.0.smithi102.stdout:5/214: mkdir d2/d25/d2a/d51 0 2024-03-26T18:54:28.512 INFO:tasks.workunit.client.1.smithi190.stdout:4/317: link d7/d1d/d42/f50 d7/d8/d1c/d20/d40/f6a 0 2024-03-26T18:54:28.514 INFO:tasks.workunit.client.1.smithi190.stdout:3/296: read d1/f42 [316194,49668] 0 2024-03-26T18:54:28.514 INFO:tasks.workunit.client.1.smithi190.stdout:3/297: fdatasync d1/f11 0 2024-03-26T18:54:28.514 INFO:tasks.workunit.client.1.smithi190.stdout:3/298: fdatasync d1/f3b 0 2024-03-26T18:54:28.514 INFO:tasks.workunit.client.1.smithi190.stdout:3/299: chown d1/d3/df/d18/d1b/l37 619328049 1 2024-03-26T18:54:28.515 INFO:tasks.workunit.client.1.smithi190.stdout:1/272: creat d3/da/dd/d2f/f5e x:0 0 0 2024-03-26T18:54:28.516 INFO:tasks.workunit.client.0.smithi102.stdout:3/166: truncate f2 173815 0 2024-03-26T18:54:28.519 INFO:tasks.workunit.client.1.smithi190.stdout:7/329: getdents d3/d6/d9/d1f/d41/d50/d53 0 2024-03-26T18:54:28.521 INFO:tasks.workunit.client.1.smithi190.stdout:5/214: getdents d2/d17/d1b/d22/d24/d3b 0 2024-03-26T18:54:28.523 INFO:tasks.workunit.client.0.smithi102.stdout:5/215: dread d2/f11 [0,4194304] 0 2024-03-26T18:54:28.523 INFO:tasks.workunit.client.0.smithi102.stdout:5/216: read d2/d25/f2b [4050917,129566] 0 2024-03-26T18:54:28.525 INFO:tasks.workunit.client.1.smithi190.stdout:3/300: rename d1/d3/df/d18/d1b/c3d to d1/de/d1c/c5c 0 2024-03-26T18:54:28.526 INFO:tasks.workunit.client.1.smithi190.stdout:3/301: creat d1/de/d1c/d36/f5d x:0 0 0 2024-03-26T18:54:28.526 INFO:tasks.workunit.client.0.smithi102.stdout:5/217: creat d2/d25/d2a/d4f/f52 x:0 0 0 2024-03-26T18:54:28.526 INFO:tasks.workunit.client.0.smithi102.stdout:5/218: write d2/d14/d1a/f1b [1157508,36637] 0 2024-03-26T18:54:28.527 INFO:tasks.workunit.client.1.smithi190.stdout:1/273: mkdir d3/d5f 0 2024-03-26T18:54:28.528 INFO:tasks.workunit.client.1.smithi190.stdout:7/330: mknod d3/d6/d9/d1f/d41/c80 0 2024-03-26T18:54:28.529 INFO:tasks.workunit.client.1.smithi190.stdout:7/331: fsync d3/f5a 0 2024-03-26T18:54:28.529 INFO:tasks.workunit.client.1.smithi190.stdout:7/332: creat d3/d6/d9/d1f/d4c/f81 x:0 0 0 2024-03-26T18:54:28.529 INFO:tasks.workunit.client.1.smithi190.stdout:1/274: write d3/d9/f57 [66892,20415] 0 2024-03-26T18:54:28.529 INFO:tasks.workunit.client.1.smithi190.stdout:1/275: write d3/d9/f2a [3723449,71347] 0 2024-03-26T18:54:28.529 INFO:tasks.workunit.client.0.smithi102.stdout:5/219: rmdir d2/d25/d2a/d51 0 2024-03-26T18:54:28.530 INFO:tasks.workunit.client.1.smithi190.stdout:5/215: stat d2/d17/d1b/d22/c35 0 2024-03-26T18:54:28.533 INFO:tasks.workunit.client.0.smithi102.stdout:5/220: mknod d2/d25/d2a/d4f/c53 0 2024-03-26T18:54:28.533 INFO:tasks.workunit.client.0.smithi102.stdout:5/221: write d2/f12 [413245,108576] 0 2024-03-26T18:54:28.533 INFO:tasks.workunit.client.1.smithi190.stdout:3/302: write d1/de/f1d [1236443,76094] 0 2024-03-26T18:54:28.534 INFO:tasks.workunit.client.1.smithi190.stdout:3/303: chown d1/de/f2d 0 1 2024-03-26T18:54:28.536 INFO:tasks.workunit.client.0.smithi102.stdout:5/222: link d2/d2c/d38/f49 d2/f54 0 2024-03-26T18:54:28.536 INFO:tasks.workunit.client.0.smithi102.stdout:5/223: write d2/d14/f1c [579295,9552] 0 2024-03-26T18:54:28.537 INFO:tasks.workunit.client.1.smithi190.stdout:3/304: dread d1/f42 [0,4194304] 0 2024-03-26T18:54:28.537 INFO:tasks.workunit.client.1.smithi190.stdout:7/333: mknod d3/d6/d9/d1f/d4c/d71/c82 0 2024-03-26T18:54:28.538 INFO:tasks.workunit.client.1.smithi190.stdout:1/276: symlink d3/l60 0 2024-03-26T18:54:28.538 INFO:tasks.workunit.client.0.smithi102.stdout:5/224: truncate d2/d2c/d38/f49 64548 0 2024-03-26T18:54:28.538 INFO:tasks.workunit.client.0.smithi102.stdout:5/225: readlink d2/l18 0 2024-03-26T18:54:28.540 INFO:tasks.workunit.client.1.smithi190.stdout:5/216: write d2/d17/d1b/d30/f3f [4177410,121437] 0 2024-03-26T18:54:28.540 INFO:tasks.workunit.client.1.smithi190.stdout:5/217: chown d2/d17/d1b/d22/f2d 248 1 2024-03-26T18:54:28.540 INFO:tasks.workunit.client.1.smithi190.stdout:1/277: dread d3/d9/f41 [0,4194304] 0 2024-03-26T18:54:28.540 INFO:tasks.workunit.client.1.smithi190.stdout:3/305: mknod d1/d47/c5e 0 2024-03-26T18:54:28.541 INFO:tasks.workunit.client.0.smithi102.stdout:5/226: truncate d2/d14/d1e/f23 515646 0 2024-03-26T18:54:28.541 INFO:tasks.workunit.client.0.smithi102.stdout:5/227: write d2/f7 [688560,88674] 0 2024-03-26T18:54:28.541 INFO:tasks.workunit.client.0.smithi102.stdout:5/228: readlink d2/l13 0 2024-03-26T18:54:28.542 INFO:tasks.workunit.client.0.smithi102.stdout:5/229: read - d2/d14/d1e/f22 zero size 2024-03-26T18:54:28.542 INFO:tasks.workunit.client.0.smithi102.stdout:5/230: read - d2/f3a zero size 2024-03-26T18:54:28.542 INFO:tasks.workunit.client.1.smithi190.stdout:7/334: link d3/d1b/d57/f5f d3/d6/d9/d1f/d4c/d71/f83 0 2024-03-26T18:54:28.542 INFO:tasks.workunit.client.1.smithi190.stdout:7/335: stat d3/d6/d9/d1f/d4c 0 2024-03-26T18:54:28.542 INFO:tasks.workunit.client.1.smithi190.stdout:7/336: truncate d3/d6/d9/d14/d36/f72 303392 0 2024-03-26T18:54:28.542 INFO:tasks.workunit.client.1.smithi190.stdout:7/337: stat d3/d6/d9/d14/d59 0 2024-03-26T18:54:28.544 INFO:tasks.workunit.client.0.smithi102.stdout:5/231: unlink d2/d25/d2a/f31 0 2024-03-26T18:54:28.544 INFO:tasks.workunit.client.0.smithi102.stdout:5/232: creat d2/d14/d1e/f55 x:0 0 0 2024-03-26T18:54:28.544 INFO:tasks.workunit.client.1.smithi190.stdout:1/278: rmdir d3/da/dd/d2f/d33 39 2024-03-26T18:54:28.545 INFO:tasks.workunit.client.1.smithi190.stdout:1/279: readlink d3/da/dd/d2f/d33/l38 0 2024-03-26T18:54:28.545 INFO:tasks.workunit.client.1.smithi190.stdout:7/338: mkdir d3/d6/d9/d1f/d84 0 2024-03-26T18:54:28.545 INFO:tasks.workunit.client.1.smithi190.stdout:7/339: fdatasync d3/d1b/f27 0 2024-03-26T18:54:28.546 INFO:tasks.workunit.client.1.smithi190.stdout:5/218: mknod d2/c4e 0 2024-03-26T18:54:28.548 INFO:tasks.workunit.client.1.smithi190.stdout:7/340: truncate d3/f19 156673 0 2024-03-26T18:54:28.549 INFO:tasks.workunit.client.1.smithi190.stdout:7/341: symlink d3/d6/d9/d1f/l85 0 2024-03-26T18:54:28.549 INFO:tasks.workunit.client.1.smithi190.stdout:7/342: fsync d3/f43 0 2024-03-26T18:54:28.549 INFO:tasks.workunit.client.1.smithi190.stdout:7/343: stat d3/d6/d9/d1f/d41/d50/d53 0 2024-03-26T18:54:28.566 INFO:tasks.workunit.client.1.smithi190.stdout:7/344: dread d3/d1b/f27 [0,4194304] 0 2024-03-26T18:54:28.567 INFO:tasks.workunit.client.1.smithi190.stdout:7/345: chown d3/d6/d9/d1f/d41/d50/d53 262115 1 2024-03-26T18:54:28.568 INFO:tasks.workunit.client.1.smithi190.stdout:8/303: dwrite d9/dd/d23/d36/d37/f5f [0,4194304] 0 2024-03-26T18:54:28.568 INFO:tasks.workunit.client.1.smithi190.stdout:5/219: write d2/f6 [1480882,105640] 0 2024-03-26T18:54:28.569 INFO:tasks.workunit.client.1.smithi190.stdout:4/318: dwrite d7/d8/d14/d23/f55 [0,4194304] 0 2024-03-26T18:54:28.570 INFO:tasks.workunit.client.1.smithi190.stdout:7/346: link c1 d3/d6/d9/d14/d59/d73/c86 0 2024-03-26T18:54:28.570 INFO:tasks.workunit.client.1.smithi190.stdout:7/347: dread - d3/d6/d9/d14/d59/f61 zero size 2024-03-26T18:54:28.574 INFO:tasks.workunit.client.1.smithi190.stdout:8/304: mkdir d9/dd/d23/d36/d37/d3a/d61 0 2024-03-26T18:54:28.579 INFO:tasks.workunit.client.1.smithi190.stdout:4/319: rename d7/d1d/d5e/f60 to d7/d1d/d5e/f6b 0 2024-03-26T18:54:28.579 INFO:tasks.workunit.client.0.smithi102.stdout:3/167: dwrite df/d29/f2c [0,4194304] 0 2024-03-26T18:54:28.580 INFO:tasks.workunit.client.1.smithi190.stdout:3/306: dwrite d1/de/f1d [0,4194304] 0 2024-03-26T18:54:28.581 INFO:tasks.workunit.client.1.smithi190.stdout:7/348: mkdir d3/d6/d9/d1f/d41/d50/d53/d87 0 2024-03-26T18:54:28.581 INFO:tasks.workunit.client.1.smithi190.stdout:7/349: write d3/d1b/f5e [756955,102379] 0 2024-03-26T18:54:28.582 INFO:tasks.workunit.client.1.smithi190.stdout:8/305: creat d9/d48/f62 x:0 0 0 2024-03-26T18:54:28.582 INFO:tasks.workunit.client.1.smithi190.stdout:8/306: creat d9/d16/f63 x:0 0 0 2024-03-26T18:54:28.583 INFO:tasks.workunit.client.1.smithi190.stdout:4/320: getdents d7/d8/d14/d4b 0 2024-03-26T18:54:28.583 INFO:tasks.workunit.client.1.smithi190.stdout:1/280: dwrite d3/d9/f20 [0,4194304] 0 2024-03-26T18:54:28.585 INFO:tasks.workunit.client.1.smithi190.stdout:3/307: truncate d1/d3/f12 1327440 0 2024-03-26T18:54:28.588 INFO:tasks.workunit.client.1.smithi190.stdout:7/350: rename d3/d1b/f27 to d3/d6/d9/d1f/f88 0 2024-03-26T18:54:28.588 INFO:tasks.workunit.client.1.smithi190.stdout:7/351: creat d3/d6/d9/d1f/d41/d50/f89 x:0 0 0 2024-03-26T18:54:28.590 INFO:tasks.workunit.client.1.smithi190.stdout:8/307: symlink d9/dd/d23/d36/d37/d3a/d43/l64 0 2024-03-26T18:54:28.591 INFO:tasks.workunit.client.0.smithi102.stdout:3/168: write df/f26 [74726,26534] 0 2024-03-26T18:54:28.591 INFO:tasks.workunit.client.0.smithi102.stdout:3/169: write df/f26 [898332,129632] 0 2024-03-26T18:54:28.591 INFO:tasks.workunit.client.1.smithi190.stdout:4/321: symlink d7/d1d/d42/l6c 0 2024-03-26T18:54:28.594 INFO:tasks.workunit.client.0.smithi102.stdout:3/170: mknod df/d29/d31/d36/c43 0 2024-03-26T18:54:28.595 INFO:tasks.workunit.client.1.smithi190.stdout:5/220: dwrite d2/d17/f21 [0,4194304] 0 2024-03-26T18:54:28.595 INFO:tasks.workunit.client.1.smithi190.stdout:5/221: creat d2/d17/f4f x:0 0 0 2024-03-26T18:54:28.595 INFO:tasks.workunit.client.0.smithi102.stdout:5/233: dwrite d2/d14/d1a/f1b [0,4194304] 0 2024-03-26T18:54:28.595 INFO:tasks.workunit.client.0.smithi102.stdout:3/171: dread df/d29/d31/d36/f41 [0,4194304] 0 2024-03-26T18:54:28.602 INFO:tasks.workunit.client.0.smithi102.stdout:3/172: rename df/c19 to df/d1c/c44 0 2024-03-26T18:54:28.605 INFO:tasks.workunit.client.0.smithi102.stdout:5/234: mknod d2/d2c/d38/d43/c56 0 2024-03-26T18:54:28.606 INFO:tasks.workunit.client.0.smithi102.stdout:5/235: stat d2/f3c 0 2024-03-26T18:54:28.609 INFO:tasks.workunit.client.1.smithi190.stdout:1/281: mknod d3/da/dd/d2f/d34/d3e/d52/c61 0 2024-03-26T18:54:28.611 INFO:tasks.workunit.client.1.smithi190.stdout:3/308: symlink d1/de/d4e/l5f 0 2024-03-26T18:54:28.612 INFO:tasks.workunit.client.1.smithi190.stdout:7/352: getdents d3/d6/d9/d14/d59 0 2024-03-26T18:54:28.613 INFO:tasks.workunit.client.1.smithi190.stdout:8/308: stat d9/l26 0 2024-03-26T18:54:28.614 INFO:tasks.workunit.client.1.smithi190.stdout:4/322: dwrite d7/d1d/f3c [0,4194304] 0 2024-03-26T18:54:28.620 INFO:tasks.workunit.client.1.smithi190.stdout:5/222: symlink d2/d17/d1b/d1e/d3a/d3c/l50 0 2024-03-26T18:54:28.622 INFO:tasks.workunit.client.1.smithi190.stdout:7/353: truncate d3/f2f 739753 0 2024-03-26T18:54:28.622 INFO:tasks.workunit.client.1.smithi190.stdout:7/354: stat d3/d6/d9/d1f/f20 0 2024-03-26T18:54:28.622 INFO:tasks.workunit.client.1.smithi190.stdout:7/355: chown d3/d6/d9/d1f/d4c/f81 345 1 2024-03-26T18:54:28.623 INFO:tasks.workunit.client.1.smithi190.stdout:8/309: mknod d9/dd/d23/d36/d37/d3a/d61/c65 0 2024-03-26T18:54:28.623 INFO:tasks.workunit.client.1.smithi190.stdout:8/310: creat d9/dd/d15/d32/f66 x:0 0 0 2024-03-26T18:54:28.624 INFO:tasks.workunit.client.1.smithi190.stdout:4/323: rename d7/d8/d1c/c2f to d7/d8/d1c/d20/c6d 0 2024-03-26T18:54:28.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:28 smithi102 ceph-mon[29002]: pgmap v242: 65 pgs: 65 active+clean; 606 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 27 MiB/s rd, 42 MiB/s wr, 455 op/s 2024-03-26T18:54:28.626 INFO:tasks.workunit.client.1.smithi190.stdout:7/356: getdents d3/d6/d9/d1f/d41/d65 0 2024-03-26T18:54:28.626 INFO:tasks.workunit.client.1.smithi190.stdout:7/357: truncate d3/d6/d9/d1f/d41/f55 660739 0 2024-03-26T18:54:28.627 INFO:tasks.workunit.client.1.smithi190.stdout:8/311: mkdir d9/d39/d67 0 2024-03-26T18:54:28.629 INFO:tasks.workunit.client.1.smithi190.stdout:8/312: creat d9/d39/d67/f68 x:0 0 0 2024-03-26T18:54:28.629 INFO:tasks.workunit.client.1.smithi190.stdout:8/313: dread - d9/d16/f63 zero size 2024-03-26T18:54:28.631 INFO:tasks.workunit.client.1.smithi190.stdout:8/314: getdents d9/dd 0 2024-03-26T18:54:28.633 INFO:tasks.workunit.client.1.smithi190.stdout:8/315: mknod d9/d48/c69 0 2024-03-26T18:54:28.633 INFO:tasks.workunit.client.1.smithi190.stdout:8/316: read d9/dd/d15/f2c [14834,22775] 0 2024-03-26T18:54:28.635 INFO:tasks.workunit.client.1.smithi190.stdout:8/317: symlink d9/d39/l6a 0 2024-03-26T18:54:28.635 INFO:tasks.workunit.client.1.smithi190.stdout:2/327: sync 2024-03-26T18:54:28.635 INFO:tasks.workunit.client.1.smithi190.stdout:0/393: sync 2024-03-26T18:54:28.635 INFO:tasks.workunit.client.1.smithi190.stdout:9/535: sync 2024-03-26T18:54:28.635 INFO:tasks.workunit.client.1.smithi190.stdout:9/536: chown d12/d2d/d1d/f77 139625931 1 2024-03-26T18:54:28.636 INFO:tasks.workunit.client.0.smithi102.stdout:5/236: dwrite d2/f12 [0,4194304] 0 2024-03-26T18:54:28.637 INFO:tasks.workunit.client.0.smithi102.stdout:5/237: creat d2/d14/d1e/f57 x:0 0 0 2024-03-26T18:54:28.637 INFO:tasks.workunit.client.1.smithi190.stdout:8/318: mknod d9/d39/d67/c6b 0 2024-03-26T18:54:28.637 INFO:tasks.workunit.client.1.smithi190.stdout:8/319: chown d9/d48/f62 210 1 2024-03-26T18:54:28.637 INFO:tasks.workunit.client.1.smithi190.stdout:0/394: write d0/dd/d20/d1c/d4e/f74 [213219,54648] 0 2024-03-26T18:54:28.638 INFO:tasks.workunit.client.1.smithi190.stdout:9/537: link d12/d2d/f97 d12/d2d/d1d/d23/d2a/d71/daa/db3/fc1 0 2024-03-26T18:54:28.638 INFO:tasks.workunit.client.1.smithi190.stdout:9/538: read d12/d2d/d1d/d23/d2a/f6c [91276,64074] 0 2024-03-26T18:54:28.639 INFO:tasks.workunit.client.1.smithi190.stdout:0/395: mkdir d0/dd/d20/d1c/d8b/d8d 0 2024-03-26T18:54:28.640 INFO:tasks.workunit.client.1.smithi190.stdout:0/396: chown d0 181 1 2024-03-26T18:54:28.640 INFO:tasks.workunit.client.1.smithi190.stdout:0/397: chown d0/dd/d24/d4b/d5e/d81/c37 256653 1 2024-03-26T18:54:28.640 INFO:tasks.workunit.client.0.smithi102.stdout:5/238: link d2/d14/f2f d2/d2c/f58 0 2024-03-26T18:54:28.640 INFO:tasks.workunit.client.0.smithi102.stdout:5/239: write d2/f7 [631064,51140] 0 2024-03-26T18:54:28.640 INFO:tasks.workunit.client.0.smithi102.stdout:5/240: write d2/f3a [457563,112804] 0 2024-03-26T18:54:28.641 INFO:tasks.workunit.client.1.smithi190.stdout:9/539: mknod d12/d2d/d1d/d23/d2a/d3f/cc2 0 2024-03-26T18:54:28.644 INFO:tasks.workunit.client.1.smithi190.stdout:0/398: symlink d0/dd/d20/d1c/d8b/l8e 0 2024-03-26T18:54:28.645 INFO:tasks.workunit.client.1.smithi190.stdout:9/540: link d12/d2d/d1d/d23/d2a/d3f/d55/l66 d12/d2d/d1d/da1/lc3 0 2024-03-26T18:54:28.646 INFO:tasks.workunit.client.1.smithi190.stdout:9/541: mknod d12/d2d/d1d/d23/d8e/d9b/cc4 0 2024-03-26T18:54:28.647 INFO:tasks.workunit.client.1.smithi190.stdout:9/542: truncate d12/d2d/f27 5003220 0 2024-03-26T18:54:28.648 INFO:tasks.workunit.client.1.smithi190.stdout:2/328: write d6/f4d [3146921,76118] 0 2024-03-26T18:54:28.648 INFO:tasks.workunit.client.1.smithi190.stdout:7/358: dwrite d3/d6/d9/d1f/d4c/f54 [0,4194304] 0 2024-03-26T18:54:28.648 INFO:tasks.workunit.client.1.smithi190.stdout:2/329: chown d6/d9/d48/f68 806105 1 2024-03-26T18:54:28.649 INFO:tasks.workunit.client.1.smithi190.stdout:7/359: chown d3/d6/d9/d1f/d41/d50/f89 2594677 1 2024-03-26T18:54:28.649 INFO:tasks.workunit.client.1.smithi190.stdout:2/330: creat d6/d9/f75 x:0 0 0 2024-03-26T18:54:28.649 INFO:tasks.workunit.client.1.smithi190.stdout:2/331: fdatasync d6/d9/d10/f2c 0 2024-03-26T18:54:28.649 INFO:tasks.workunit.client.1.smithi190.stdout:7/360: write d3/f19 [594891,110855] 0 2024-03-26T18:54:28.649 INFO:tasks.workunit.client.1.smithi190.stdout:7/361: fdatasync d3/d6/d9/d14/f18 0 2024-03-26T18:54:28.656 INFO:tasks.workunit.client.1.smithi190.stdout:2/332: write d6/d9/d10/f13 [4658075,45226] 0 2024-03-26T18:54:28.667 INFO:tasks.workunit.client.1.smithi190.stdout:9/543: write d12/f28 [2404997,44365] 0 2024-03-26T18:54:28.667 INFO:tasks.workunit.client.1.smithi190.stdout:9/544: dread - d12/d2d/d1d/d23/d6a/f7e zero size 2024-03-26T18:54:28.667 INFO:tasks.workunit.client.1.smithi190.stdout:9/545: write f11 [3135750,109400] 0 2024-03-26T18:54:28.667 INFO:tasks.workunit.client.1.smithi190.stdout:9/546: dread - d12/d2d/d1d/d54/f88 zero size 2024-03-26T18:54:28.668 INFO:tasks.workunit.client.1.smithi190.stdout:9/547: symlink d12/d2d/d1d/d23/d2a/d3f/lc5 0 2024-03-26T18:54:28.670 INFO:tasks.workunit.client.1.smithi190.stdout:9/548: creat d12/fc6 x:0 0 0 2024-03-26T18:54:28.670 INFO:tasks.workunit.client.1.smithi190.stdout:9/549: fdatasync d12/d2d/f31 0 2024-03-26T18:54:28.671 INFO:tasks.workunit.client.1.smithi190.stdout:8/320: dwrite d9/d48/f62 [0,4194304] 0 2024-03-26T18:54:28.671 INFO:tasks.workunit.client.1.smithi190.stdout:8/321: write d9/dd/d15/d32/f5c [798450,122838] 0 2024-03-26T18:54:28.671 INFO:tasks.workunit.client.1.smithi190.stdout:9/550: truncate ff 196947 0 2024-03-26T18:54:28.672 INFO:tasks.workunit.client.1.smithi190.stdout:8/322: symlink d9/d48/l6c 0 2024-03-26T18:54:28.672 INFO:tasks.workunit.client.1.smithi190.stdout:8/323: write d9/dd/d15/f60 [934110,98160] 0 2024-03-26T18:54:28.673 INFO:tasks.workunit.client.1.smithi190.stdout:9/551: getdents d12/d2d/d1d/d23/d83/d91 0 2024-03-26T18:54:28.680 INFO:tasks.workunit.client.1.smithi190.stdout:9/552: dread d12/d2d/d1d/d23/d2a/d71/fa6 [0,4194304] 0 2024-03-26T18:54:28.690 INFO:tasks.workunit.client.1.smithi190.stdout:7/362: dwrite d3/d6/d9/d1f/f88 [0,4194304] 0 2024-03-26T18:54:28.691 INFO:tasks.workunit.client.1.smithi190.stdout:7/363: creat d3/d6/d9/d1f/d41/d50/d53/f8a x:0 0 0 2024-03-26T18:54:28.699 INFO:tasks.workunit.client.1.smithi190.stdout:2/333: dwrite d6/d9/d48/f65 [0,4194304] 0 2024-03-26T18:54:28.700 INFO:tasks.workunit.client.1.smithi190.stdout:2/334: dread - d6/d9/f41 zero size 2024-03-26T18:54:28.700 INFO:tasks.workunit.client.1.smithi190.stdout:2/335: truncate d6/d9/d10/f4b 1612798 0 2024-03-26T18:54:28.700 INFO:tasks.workunit.client.1.smithi190.stdout:8/324: dwrite d9/dd/d15/d32/f4f [0,4194304] 0 2024-03-26T18:54:28.702 INFO:tasks.workunit.client.1.smithi190.stdout:8/325: creat d9/d39/f6d x:0 0 0 2024-03-26T18:54:28.715 INFO:tasks.workunit.client.1.smithi190.stdout:4/324: dread d7/d8/d14/d4b/f5f [0,4194304] 0 2024-03-26T18:54:28.717 INFO:tasks.workunit.client.1.smithi190.stdout:4/325: unlink d7/d8/f3a 0 2024-03-26T18:54:28.717 INFO:tasks.workunit.client.1.smithi190.stdout:4/326: rmdir d7/d1d/d41 39 2024-03-26T18:54:28.728 INFO:tasks.workunit.client.0.smithi102.stdout:3/173: dwrite df/d29/f2c [0,4194304] 0 2024-03-26T18:54:28.728 INFO:tasks.workunit.client.0.smithi102.stdout:3/174: stat df/f26 0 2024-03-26T18:54:28.728 INFO:tasks.workunit.client.0.smithi102.stdout:3/175: fsync fd 0 2024-03-26T18:54:28.729 INFO:tasks.workunit.client.1.smithi190.stdout:4/327: dread d7/d8/d1c/f39 [0,4194304] 0 2024-03-26T18:54:28.731 INFO:tasks.workunit.client.1.smithi190.stdout:4/328: mkdir d7/d8/d14/d6e 0 2024-03-26T18:54:28.731 INFO:tasks.workunit.client.1.smithi190.stdout:4/329: dread - d7/d8/d1c/d20/f58 zero size 2024-03-26T18:54:28.732 INFO:tasks.workunit.client.1.smithi190.stdout:2/336: dwrite d6/d9/d1a/f64 [0,4194304] 0 2024-03-26T18:54:28.732 INFO:tasks.workunit.client.1.smithi190.stdout:2/337: chown d6/d9/d1a/l42 322879 1 2024-03-26T18:54:28.732 INFO:tasks.workunit.client.1.smithi190.stdout:9/553: dwrite d12/d2d/d1d/d23/d2a/f42 [0,4194304] 0 2024-03-26T18:54:28.732 INFO:tasks.workunit.client.1.smithi190.stdout:9/554: read - d12/d2d/d1d/f77 zero size 2024-03-26T18:54:28.732 INFO:tasks.workunit.client.1.smithi190.stdout:1/282: dwrite d3/da/f37 [0,4194304] 0 2024-03-26T18:54:28.733 INFO:tasks.workunit.client.1.smithi190.stdout:4/330: mkdir d7/d8/d14/d6e/d6f 0 2024-03-26T18:54:28.733 INFO:tasks.workunit.client.1.smithi190.stdout:0/399: dwrite d0/dd/d20/d1c/f6e [0,4194304] 0 2024-03-26T18:54:28.733 INFO:tasks.workunit.client.1.smithi190.stdout:0/400: write d0/dd/d24/d4b/f89 [1033973,109191] 0 2024-03-26T18:54:28.738 INFO:tasks.workunit.client.1.smithi190.stdout:2/338: mknod d6/d9/d45/d6b/c76 0 2024-03-26T18:54:28.738 INFO:tasks.workunit.client.1.smithi190.stdout:8/326: getdents d9/dd/d15/d32 0 2024-03-26T18:54:28.743 INFO:tasks.workunit.client.1.smithi190.stdout:4/331: mknod d7/d8/c70 0 2024-03-26T18:54:28.743 INFO:tasks.workunit.client.1.smithi190.stdout:4/332: creat d7/d8/d14/d23/d28/d38/f71 x:0 0 0 2024-03-26T18:54:28.745 INFO:tasks.workunit.client.1.smithi190.stdout:9/555: dread d12/d2d/d1d/d23/d2a/f73 [0,4194304] 0 2024-03-26T18:54:28.745 INFO:tasks.workunit.client.1.smithi190.stdout:9/556: creat d12/d2d/d1d/d23/d2a/d3f/fc7 x:0 0 0 2024-03-26T18:54:28.749 INFO:tasks.workunit.client.1.smithi190.stdout:0/401: write d0/dd/f22 [5178519,75529] 0 2024-03-26T18:54:28.753 INFO:tasks.workunit.client.1.smithi190.stdout:8/327: link d9/d48/c5e d9/dd/d23/c6e 0 2024-03-26T18:54:28.754 INFO:tasks.workunit.client.0.smithi102.stdout:3/176: dwrite df/d1c/f2a [0,4194304] 0 2024-03-26T18:54:28.755 INFO:tasks.workunit.client.1.smithi190.stdout:1/283: getdents d3/d9 0 2024-03-26T18:54:28.755 INFO:tasks.workunit.client.0.smithi102.stdout:5/241: dwrite d2/d2c/d38/d43/f4e [0,4194304] 0 2024-03-26T18:54:28.757 INFO:tasks.workunit.client.1.smithi190.stdout:4/333: symlink d7/d8/d1c/l72 0 2024-03-26T18:54:28.758 INFO:tasks.workunit.client.1.smithi190.stdout:0/402: dread d0/dd/d20/d1c/d46/f5d [0,4194304] 0 2024-03-26T18:54:28.760 INFO:tasks.workunit.client.0.smithi102.stdout:5/242: creat d2/d25/d2a/f59 x:0 0 0 2024-03-26T18:54:28.760 INFO:tasks.workunit.client.0.smithi102.stdout:5/243: write d2/d14/d1a/f41 [136618,5621] 0 2024-03-26T18:54:28.763 INFO:tasks.workunit.client.1.smithi190.stdout:9/557: creat d12/d2d/d1d/d23/d2b/da0/fc8 x:0 0 0 2024-03-26T18:54:28.764 INFO:tasks.workunit.client.1.smithi190.stdout:8/328: rename d9/dd/f20 to d9/d39/f6f 0 2024-03-26T18:54:28.765 INFO:tasks.workunit.client.0.smithi102.stdout:3/177: dread f7 [0,4194304] 0 2024-03-26T18:54:28.765 INFO:tasks.workunit.client.0.smithi102.stdout:3/178: write fd [1266712,28183] 0 2024-03-26T18:54:28.766 INFO:tasks.workunit.client.1.smithi190.stdout:1/284: creat d3/da/dd/d2f/d5b/f62 x:0 0 0 2024-03-26T18:54:28.766 INFO:tasks.workunit.client.1.smithi190.stdout:1/285: dread - d3/da/dd/d2f/d34/d3e/f56 zero size 2024-03-26T18:54:28.767 INFO:tasks.workunit.client.1.smithi190.stdout:4/334: creat d7/d8/d1c/f73 x:0 0 0 2024-03-26T18:54:28.769 INFO:tasks.workunit.client.0.smithi102.stdout:3/179: mkdir df/d29/d31/d36/d45 0 2024-03-26T18:54:28.769 INFO:tasks.workunit.client.0.smithi102.stdout:3/180: fdatasync f3 0 2024-03-26T18:54:28.770 INFO:tasks.workunit.client.1.smithi190.stdout:0/403: mkdir d0/dd/d20/d1c/d8b/d8d/d8f 0 2024-03-26T18:54:28.772 INFO:tasks.workunit.client.0.smithi102.stdout:3/181: rename df/d29/d31/d36/c43 to df/d29/d31/d36/d45/c46 0 2024-03-26T18:54:28.772 INFO:tasks.workunit.client.0.smithi102.stdout:3/182: chown l9 13620 1 2024-03-26T18:54:28.777 INFO:tasks.workunit.client.0.smithi102.stdout:3/183: mkdir df/d29/d3a/d47 0 2024-03-26T18:54:28.777 INFO:tasks.workunit.client.0.smithi102.stdout:3/184: truncate df/f1a 1875789 0 2024-03-26T18:54:28.780 INFO:tasks.workunit.client.1.smithi190.stdout:0/404: rmdir d0/dd/d20/d1c/d46/d7c/d88 0 2024-03-26T18:54:28.780 INFO:tasks.workunit.client.1.smithi190.stdout:0/405: chown d0/dd/d20/l3d 150896741 1 2024-03-26T18:54:28.780 INFO:tasks.workunit.client.1.smithi190.stdout:0/406: creat d0/dd/d20/d1c/d4e/f90 x:0 0 0 2024-03-26T18:54:28.781 INFO:tasks.workunit.client.1.smithi190.stdout:0/407: readlink d0/dd/d20/d1c/d46/l67 0 2024-03-26T18:54:28.781 INFO:tasks.workunit.client.1.smithi190.stdout:0/408: chown d0/dd/d24/f54 3 1 2024-03-26T18:54:28.781 INFO:tasks.workunit.client.1.smithi190.stdout:0/409: stat d0/dd/d20/d1c/d8b/d8d/d8f 0 2024-03-26T18:54:28.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:28 smithi190 ceph-mon[35043]: pgmap v242: 65 pgs: 65 active+clean; 606 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 27 MiB/s rd, 42 MiB/s wr, 455 op/s 2024-03-26T18:54:28.782 INFO:tasks.workunit.client.0.smithi102.stdout:3/185: unlink df/f16 0 2024-03-26T18:54:28.796 INFO:tasks.workunit.client.1.smithi190.stdout:0/410: dread d0/f13 [0,4194304] 0 2024-03-26T18:54:28.797 INFO:tasks.workunit.client.1.smithi190.stdout:8/329: getdents d9/dd 0 2024-03-26T18:54:28.798 INFO:tasks.workunit.client.1.smithi190.stdout:8/330: mknod d9/d48/d49/c70 0 2024-03-26T18:54:28.799 INFO:tasks.workunit.client.1.smithi190.stdout:8/331: creat d9/dd/d23/d36/f71 x:0 0 0 2024-03-26T18:54:28.799 INFO:tasks.workunit.client.1.smithi190.stdout:8/332: stat d9/dd/d23/d36/d37/d3a/d43 0 2024-03-26T18:54:28.805 INFO:tasks.workunit.client.0.smithi102.stdout:5/244: dwrite d2/d14/f1c [0,4194304] 0 2024-03-26T18:54:28.809 INFO:tasks.workunit.client.1.smithi190.stdout:8/333: write d9/d48/f50 [3490665,5101] 0 2024-03-26T18:54:28.811 INFO:tasks.workunit.client.1.smithi190.stdout:8/334: creat d9/dd/d23/d36/d37/d3a/d61/f72 x:0 0 0 2024-03-26T18:54:28.813 INFO:tasks.workunit.client.1.smithi190.stdout:9/558: dwrite d12/d2d/d1d/d23/d2a/d3f/d55/f82 [0,4194304] 0 2024-03-26T18:54:28.813 INFO:tasks.workunit.client.1.smithi190.stdout:9/559: stat d12/d2d/d1d/l87 0 2024-03-26T18:54:28.813 INFO:tasks.workunit.client.1.smithi190.stdout:8/335: mkdir d9/d16/d73 0 2024-03-26T18:54:28.813 INFO:tasks.workunit.client.1.smithi190.stdout:4/335: dwrite d7/d8/d1c/f44 [0,4194304] 0 2024-03-26T18:54:28.816 INFO:tasks.workunit.client.1.smithi190.stdout:8/336: symlink d9/d16/l74 0 2024-03-26T18:54:28.819 INFO:tasks.workunit.client.1.smithi190.stdout:4/336: mkdir d7/d1d/d41/d74 0 2024-03-26T18:54:28.819 INFO:tasks.workunit.client.1.smithi190.stdout:4/337: dread - d7/d1d/d42/f50 zero size 2024-03-26T18:54:28.819 INFO:tasks.workunit.client.1.smithi190.stdout:4/338: stat d7/d8/d14/d23/l64 0 2024-03-26T18:54:28.819 INFO:tasks.workunit.client.1.smithi190.stdout:4/339: write d7/f18 [1099612,83851] 0 2024-03-26T18:54:28.829 INFO:tasks.workunit.client.1.smithi190.stdout:6/293: sync 2024-03-26T18:54:28.829 INFO:tasks.workunit.client.1.smithi190.stdout:6/294: chown d17/d28/d46/d4d/d51/l52 59961861 1 2024-03-26T18:54:28.829 INFO:tasks.workunit.client.1.smithi190.stdout:6/295: write d17/d28/d46/d4d/f66 [346702,45902] 0 2024-03-26T18:54:28.829 INFO:tasks.workunit.client.1.smithi190.stdout:6/296: dread - d17/d28/d2c/f47 zero size 2024-03-26T18:54:28.830 INFO:tasks.workunit.client.1.smithi190.stdout:6/297: rename d17/f4e to d17/d28/d46/d4d/f69 0 2024-03-26T18:54:28.831 INFO:tasks.workunit.client.1.smithi190.stdout:8/337: write d9/d16/f1f [284350,85275] 0 2024-03-26T18:54:28.836 INFO:tasks.workunit.client.1.smithi190.stdout:0/411: dwrite d0/f13 [4194304,4194304] 0 2024-03-26T18:54:28.836 INFO:tasks.workunit.client.1.smithi190.stdout:4/340: dwrite d7/d8/d14/f5b [0,4194304] 0 2024-03-26T18:54:28.836 INFO:tasks.workunit.client.1.smithi190.stdout:4/341: stat d7/d1d/d41/l4d 0 2024-03-26T18:54:28.836 INFO:tasks.workunit.client.1.smithi190.stdout:4/342: truncate d7/d8/d14/f57 5029236 0 2024-03-26T18:54:28.837 INFO:tasks.workunit.client.0.smithi102.stdout:5/245: dwrite d2/d2c/f58 [0,4194304] 0 2024-03-26T18:54:28.838 INFO:tasks.workunit.client.1.smithi190.stdout:0/412: truncate d0/dd/f12 1914915 0 2024-03-26T18:54:28.847 INFO:tasks.workunit.client.0.smithi102.stdout:5/246: dread d2/f11 [0,4194304] 0 2024-03-26T18:54:28.847 INFO:tasks.workunit.client.0.smithi102.stdout:5/247: chown d2/f7 818416 1 2024-03-26T18:54:28.847 INFO:tasks.workunit.client.1.smithi190.stdout:4/343: link d7/d8/d14/d5c/f63 d7/d8/d14/d4b/f75 0 2024-03-26T18:54:28.848 INFO:tasks.workunit.client.0.smithi102.stdout:5/248: rename d2/d2c/d38/f3b to d2/d2c/f5a 0 2024-03-26T18:54:28.849 INFO:tasks.workunit.client.1.smithi190.stdout:0/413: chown d0/dd/d20/d1c/f56 1531 1 2024-03-26T18:54:28.850 INFO:tasks.workunit.client.0.smithi102.stdout:5/249: mkdir d2/d25/d5b 0 2024-03-26T18:54:28.850 INFO:tasks.workunit.client.0.smithi102.stdout:5/250: dread - d2/d14/d1e/f42 zero size 2024-03-26T18:54:28.850 INFO:tasks.workunit.client.0.smithi102.stdout:5/251: dread - d2/d14/d1e/f55 zero size 2024-03-26T18:54:28.851 INFO:tasks.workunit.client.1.smithi190.stdout:4/344: link d7/d8/d14/d23/d28/d38/f71 d7/d1d/d5e/f76 0 2024-03-26T18:54:28.852 INFO:tasks.workunit.client.0.smithi102.stdout:5/252: creat d2/d2c/d38/f5c x:0 0 0 2024-03-26T18:54:28.852 INFO:tasks.workunit.client.1.smithi190.stdout:0/414: dread d0/dd/d24/f49 [0,4194304] 0 2024-03-26T18:54:28.853 INFO:tasks.workunit.client.1.smithi190.stdout:4/345: getdents d7/d1d/d41 0 2024-03-26T18:54:28.855 INFO:tasks.workunit.client.1.smithi190.stdout:0/415: read d0/dd/f12 [1660885,92104] 0 2024-03-26T18:54:28.855 INFO:tasks.workunit.client.1.smithi190.stdout:0/416: creat d0/dd/d20/d1c/d46/f91 x:0 0 0 2024-03-26T18:54:28.855 INFO:tasks.workunit.client.1.smithi190.stdout:8/338: dwrite d9/d48/f50 [4194304,4194304] 0 2024-03-26T18:54:28.856 INFO:tasks.workunit.client.1.smithi190.stdout:6/298: dwrite d17/f23 [0,4194304] 0 2024-03-26T18:54:28.856 INFO:tasks.workunit.client.1.smithi190.stdout:6/299: fdatasync d17/d28/d46/d4d/f66 0 2024-03-26T18:54:28.859 INFO:tasks.workunit.client.1.smithi190.stdout:0/417: mknod d0/dd/d20/d61/d7f/c92 0 2024-03-26T18:54:28.859 INFO:tasks.workunit.client.1.smithi190.stdout:0/418: read - d0/dd/d20/f78 zero size 2024-03-26T18:54:28.872 INFO:tasks.workunit.client.1.smithi190.stdout:0/419: dread d0/dd/d20/d1c/d46/f72 [0,4194304] 0 2024-03-26T18:54:28.874 INFO:tasks.workunit.client.1.smithi190.stdout:0/420: mkdir d0/dd/d24/d87/d93 0 2024-03-26T18:54:28.876 INFO:tasks.workunit.client.1.smithi190.stdout:0/421: creat d0/dd/d24/d4b/d5e/d66/f94 x:0 0 0 2024-03-26T18:54:28.876 INFO:tasks.workunit.client.1.smithi190.stdout:0/422: write d0/dd/d20/d1c/d4e/f90 [264642,42991] 0 2024-03-26T18:54:28.876 INFO:tasks.workunit.client.1.smithi190.stdout:0/423: truncate d0/dd/d20/d1c/f29 1834642 0 2024-03-26T18:54:28.888 INFO:tasks.workunit.client.1.smithi190.stdout:4/346: dwrite d7/d1d/f27 [0,4194304] 0 2024-03-26T18:54:28.889 INFO:tasks.workunit.client.1.smithi190.stdout:6/300: dwrite f9 [4194304,4194304] 0 2024-03-26T18:54:28.889 INFO:tasks.workunit.client.1.smithi190.stdout:6/301: write d17/d28/d2c/f61 [187328,94111] 0 2024-03-26T18:54:28.890 INFO:tasks.workunit.client.1.smithi190.stdout:6/302: write d17/d28/f2d [5219785,126319] 0 2024-03-26T18:54:28.890 INFO:tasks.workunit.client.1.smithi190.stdout:8/339: dwrite d9/dd/d23/d36/d37/d3a/d61/f72 [0,4194304] 0 2024-03-26T18:54:28.890 INFO:tasks.workunit.client.1.smithi190.stdout:8/340: write d9/dd/d15/f30 [1294550,58524] 0 2024-03-26T18:54:28.891 INFO:tasks.workunit.client.0.smithi102.stdout:5/253: dwrite d2/d2c/d38/f5c [0,4194304] 0 2024-03-26T18:54:28.891 INFO:tasks.workunit.client.1.smithi190.stdout:4/347: link d7/d8/d14/f5b d7/d8/d14/d4b/f77 0 2024-03-26T18:54:28.894 INFO:tasks.workunit.client.1.smithi190.stdout:6/303: truncate d17/d28/d2b/f3c 74284 0 2024-03-26T18:54:28.896 INFO:tasks.workunit.client.0.smithi102.stdout:5/254: mkdir d2/d25/d2a/d5d 0 2024-03-26T18:54:28.900 INFO:tasks.workunit.client.0.smithi102.stdout:5/255: link d2/f12 d2/d14/d1a/f5e 0 2024-03-26T18:54:28.901 INFO:tasks.workunit.client.1.smithi190.stdout:8/341: write d9/d48/f62 [2399125,61092] 0 2024-03-26T18:54:28.901 INFO:tasks.workunit.client.1.smithi190.stdout:8/342: dread - d9/d16/f63 zero size 2024-03-26T18:54:28.904 INFO:tasks.workunit.client.1.smithi190.stdout:6/304: mkdir d17/d28/d46/d6a 0 2024-03-26T18:54:28.904 INFO:tasks.workunit.client.1.smithi190.stdout:6/305: readlink d17/d28/d2c/l36 0 2024-03-26T18:54:28.906 INFO:tasks.workunit.client.1.smithi190.stdout:8/343: mknod d9/dd/c75 0 2024-03-26T18:54:28.908 INFO:tasks.workunit.client.1.smithi190.stdout:6/306: symlink d17/d28/d2c/l6b 0 2024-03-26T18:54:28.909 INFO:tasks.workunit.client.1.smithi190.stdout:8/344: creat d9/dd/d15/d32/f76 x:0 0 0 2024-03-26T18:54:28.911 INFO:tasks.workunit.client.1.smithi190.stdout:6/307: link d17/d28/c32 d17/d28/d2b/c6c 0 2024-03-26T18:54:28.911 INFO:tasks.workunit.client.1.smithi190.stdout:4/348: dwrite d7/d8/d1c/d20/f58 [0,4194304] 0 2024-03-26T18:54:28.911 INFO:tasks.workunit.client.1.smithi190.stdout:4/349: chown d7/d8/d14/d23/d28/d38/c65 23 1 2024-03-26T18:54:28.914 INFO:tasks.workunit.client.1.smithi190.stdout:6/308: mknod d17/d28/d46/d4d/d51/c6d 0 2024-03-26T18:54:28.915 INFO:tasks.workunit.client.1.smithi190.stdout:6/309: getdents d17/d28/d2c/d50 0 2024-03-26T18:54:28.915 INFO:tasks.workunit.client.1.smithi190.stdout:6/310: dread - d17/d28/d2c/d50/f67 zero size 2024-03-26T18:54:28.921 INFO:tasks.workunit.client.1.smithi190.stdout:4/350: dread d7/d1d/f27 [0,4194304] 0 2024-03-26T18:54:28.921 INFO:tasks.workunit.client.1.smithi190.stdout:4/351: creat d7/d8/d1c/d20/f78 x:0 0 0 2024-03-26T18:54:28.924 INFO:tasks.workunit.client.1.smithi190.stdout:4/352: unlink d7/d8/d1c/f73 0 2024-03-26T18:54:28.926 INFO:tasks.workunit.client.0.smithi102.stdout:5/256: dwrite d2/d14/f2f [0,4194304] 0 2024-03-26T18:54:28.926 INFO:tasks.workunit.client.0.smithi102.stdout:5/257: chown d2/d25/d2a/l3d 1 1 2024-03-26T18:54:28.927 INFO:tasks.workunit.client.0.smithi102.stdout:5/258: rmdir d2 39 2024-03-26T18:54:28.927 INFO:tasks.workunit.client.0.smithi102.stdout:5/259: write d2/d25/f2b [4788933,75449] 0 2024-03-26T18:54:28.928 INFO:tasks.workunit.client.0.smithi102.stdout:5/260: getdents d2/d14/d1e 0 2024-03-26T18:54:28.928 INFO:tasks.workunit.client.0.smithi102.stdout:5/261: chown d2/d14/c33 0 1 2024-03-26T18:54:28.929 INFO:tasks.workunit.client.1.smithi190.stdout:6/311: write d17/d28/d2c/f41 [2184368,110128] 0 2024-03-26T18:54:28.930 INFO:tasks.workunit.client.1.smithi190.stdout:8/345: dwrite d9/dd/d15/d32/f35 [0,4194304] 0 2024-03-26T18:54:28.931 INFO:tasks.workunit.client.1.smithi190.stdout:8/346: chown d9/dd/d23/d36/d37/f5f 482800628 1 2024-03-26T18:54:28.931 INFO:tasks.workunit.client.1.smithi190.stdout:8/347: dread - d9/d16/f51 zero size 2024-03-26T18:54:28.931 INFO:tasks.workunit.client.1.smithi190.stdout:8/348: creat d9/d16/f77 x:0 0 0 2024-03-26T18:54:28.931 INFO:tasks.workunit.client.0.smithi102.stdout:5/262: rename d2/d14/d1e to d2/d25/d2a/d5f 0 2024-03-26T18:54:28.931 INFO:tasks.workunit.client.1.smithi190.stdout:6/312: creat d17/d28/d2b/f6e x:0 0 0 2024-03-26T18:54:28.933 INFO:tasks.workunit.client.0.smithi102.stdout:5/263: symlink d2/d25/d2a/d5f/l60 0 2024-03-26T18:54:28.933 INFO:tasks.workunit.client.0.smithi102.stdout:5/264: write d2/f54 [964687,1120] 0 2024-03-26T18:54:28.934 INFO:tasks.workunit.client.1.smithi190.stdout:8/349: creat d9/dd/f78 x:0 0 0 2024-03-26T18:54:28.934 INFO:tasks.workunit.client.1.smithi190.stdout:8/350: readlink l4 0 2024-03-26T18:54:28.934 INFO:tasks.workunit.client.1.smithi190.stdout:8/351: readlink d9/dd/d23/d36/d37/d3a/d43/l64 0 2024-03-26T18:54:28.935 INFO:tasks.workunit.client.0.smithi102.stdout:5/265: creat d2/d14/d1a/f61 x:0 0 0 2024-03-26T18:54:28.936 INFO:tasks.workunit.client.1.smithi190.stdout:9/560: sync 2024-03-26T18:54:28.936 INFO:tasks.workunit.client.1.smithi190.stdout:1/286: sync 2024-03-26T18:54:28.936 INFO:tasks.workunit.client.1.smithi190.stdout:2/339: sync 2024-03-26T18:54:28.936 INFO:tasks.workunit.client.1.smithi190.stdout:5/223: sync 2024-03-26T18:54:28.936 INFO:tasks.workunit.client.1.smithi190.stdout:3/309: sync 2024-03-26T18:54:28.936 INFO:tasks.workunit.client.1.smithi190.stdout:7/364: sync 2024-03-26T18:54:28.936 INFO:tasks.workunit.client.1.smithi190.stdout:2/340: fsync d6/d9/d3a/d3d/d2e/f6e 0 2024-03-26T18:54:28.936 INFO:tasks.workunit.client.1.smithi190.stdout:7/365: fsync d3/f7c 0 2024-03-26T18:54:28.937 INFO:tasks.workunit.client.1.smithi190.stdout:1/287: creat d3/da/dd/d2f/d5b/f63 x:0 0 0 2024-03-26T18:54:28.937 INFO:tasks.workunit.client.1.smithi190.stdout:5/224: creat d2/d17/d1b/d30/f51 x:0 0 0 2024-03-26T18:54:28.937 INFO:tasks.workunit.client.1.smithi190.stdout:5/225: read - d2/d17/d1b/d22/f38 zero size 2024-03-26T18:54:28.937 INFO:tasks.workunit.client.1.smithi190.stdout:5/226: creat d2/d17/d1b/d1e/d3a/f52 x:0 0 0 2024-03-26T18:54:28.937 INFO:tasks.workunit.client.1.smithi190.stdout:5/227: readlink d2/d17/d1b/l1f 0 2024-03-26T18:54:28.937 INFO:tasks.workunit.client.1.smithi190.stdout:5/228: stat d2/lf 0 2024-03-26T18:54:28.937 INFO:tasks.workunit.client.1.smithi190.stdout:6/313: rename d17/d28/d2c/f42 to d17/d28/d46/d4d/f6f 0 2024-03-26T18:54:28.937 INFO:tasks.workunit.client.1.smithi190.stdout:8/352: unlink d9/d16/f77 0 2024-03-26T18:54:28.938 INFO:tasks.workunit.client.0.smithi102.stdout:5/266: creat d2/f62 x:0 0 0 2024-03-26T18:54:28.938 INFO:tasks.workunit.client.1.smithi190.stdout:9/561: getdents d12/d2d/d1d/d23/d8e/d9b 0 2024-03-26T18:54:28.940 INFO:tasks.workunit.client.0.smithi102.stdout:5/267: symlink d2/d2c/l63 0 2024-03-26T18:54:28.941 INFO:tasks.workunit.client.1.smithi190.stdout:4/353: dwrite d7/d8/d14/d5c/f63 [0,4194304] 0 2024-03-26T18:54:28.942 INFO:tasks.workunit.client.1.smithi190.stdout:2/341: symlink d6/d9/d3a/d3d/l77 0 2024-03-26T18:54:28.944 INFO:tasks.workunit.client.0.smithi102.stdout:5/268: link d2/d25/f2b d2/f64 0 2024-03-26T18:54:28.945 INFO:tasks.workunit.client.1.smithi190.stdout:3/310: dread d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:28.948 INFO:tasks.workunit.client.1.smithi190.stdout:1/288: truncate d3/da/dd/d2f/d34/d3e/f46 1225004 0 2024-03-26T18:54:28.949 INFO:tasks.workunit.client.1.smithi190.stdout:1/289: read - d3/f53 zero size 2024-03-26T18:54:28.949 INFO:tasks.workunit.client.1.smithi190.stdout:1/290: fsync d3/d9/f15 0 2024-03-26T18:54:28.952 INFO:tasks.workunit.client.1.smithi190.stdout:5/229: mknod d2/d17/d1b/c53 0 2024-03-26T18:54:28.952 INFO:tasks.workunit.client.1.smithi190.stdout:5/230: creat d2/d17/d1b/d22/f54 x:0 0 0 2024-03-26T18:54:28.953 INFO:tasks.workunit.client.1.smithi190.stdout:9/562: getdents d12/d2d/d1d/d23/d6a 0 2024-03-26T18:54:28.953 INFO:tasks.workunit.client.1.smithi190.stdout:9/563: read d12/d2d/f39 [4175280,9271] 0 2024-03-26T18:54:28.955 INFO:tasks.workunit.client.1.smithi190.stdout:4/354: rename d7/d1d/d41/l4d to d7/d1d/d41/d74/l79 0 2024-03-26T18:54:28.955 INFO:tasks.workunit.client.1.smithi190.stdout:4/355: creat d7/d8/d1c/f7a x:0 0 0 2024-03-26T18:54:28.955 INFO:tasks.workunit.client.1.smithi190.stdout:4/356: write d7/d1d/d5e/f76 [750649,65575] 0 2024-03-26T18:54:28.955 INFO:tasks.workunit.client.1.smithi190.stdout:4/357: chown d7/d8/d14/d23/d28/d38/f71 0 1 2024-03-26T18:54:28.956 INFO:tasks.workunit.client.1.smithi190.stdout:2/342: creat d6/d21/f78 x:0 0 0 2024-03-26T18:54:28.960 INFO:tasks.workunit.client.1.smithi190.stdout:1/291: link d3/d9/l17 d3/da/dd/d2f/d34/l64 0 2024-03-26T18:54:28.960 INFO:tasks.workunit.client.1.smithi190.stdout:1/292: write d3/da/dd/d2f/d34/f4c [1008083,47295] 0 2024-03-26T18:54:28.962 INFO:tasks.workunit.client.1.smithi190.stdout:1/293: creat d3/da/d2e/d59/f65 x:0 0 0 2024-03-26T18:54:28.965 INFO:tasks.workunit.client.1.smithi190.stdout:8/353: dread d9/fa [0,4194304] 0 2024-03-26T18:54:28.972 INFO:tasks.workunit.client.1.smithi190.stdout:4/358: dread d7/d8/d1c/f44 [0,4194304] 0 2024-03-26T18:54:28.973 INFO:tasks.workunit.client.1.smithi190.stdout:4/359: truncate d7/d8/d1c/d20/f47 552264 0 2024-03-26T18:54:28.976 INFO:tasks.workunit.client.1.smithi190.stdout:5/231: read d2/d17/f21 [3527892,3721] 0 2024-03-26T18:54:28.981 INFO:tasks.workunit.client.1.smithi190.stdout:3/311: dwrite d1/d3/df/f19 [0,4194304] 0 2024-03-26T18:54:29.002 INFO:tasks.workunit.client.1.smithi190.stdout:6/314: dwrite fc [0,4194304] 0 2024-03-26T18:54:29.003 INFO:tasks.workunit.client.1.smithi190.stdout:6/315: creat d17/d28/d2c/d63/f70 x:0 0 0 2024-03-26T18:54:29.003 INFO:tasks.workunit.client.1.smithi190.stdout:6/316: truncate d17/d28/d46/d4d/d51/f54 535283 0 2024-03-26T18:54:29.003 INFO:tasks.workunit.client.1.smithi190.stdout:6/317: truncate d17/d28/d46/d4d/f66 2562847 0 2024-03-26T18:54:29.008 INFO:tasks.workunit.client.1.smithi190.stdout:8/354: dwrite d9/d48/f62 [0,4194304] 0 2024-03-26T18:54:29.010 INFO:tasks.workunit.client.1.smithi190.stdout:8/355: mkdir d9/d39/d79 0 2024-03-26T18:54:29.010 INFO:tasks.workunit.client.1.smithi190.stdout:8/356: creat d9/dd/d15/f7a x:0 0 0 2024-03-26T18:54:29.010 INFO:tasks.workunit.client.1.smithi190.stdout:8/357: fsync d9/d48/f62 0 2024-03-26T18:54:29.013 INFO:tasks.workunit.client.1.smithi190.stdout:8/358: rename d9/dd/d23/c6e to d9/dd/d23/d36/d37/d3a/d43/c7b 0 2024-03-26T18:54:29.014 INFO:tasks.workunit.client.1.smithi190.stdout:8/359: mknod d9/dd/d23/d36/d37/d3a/c7c 0 2024-03-26T18:54:29.014 INFO:tasks.workunit.client.1.smithi190.stdout:8/360: stat d9/d39/d79 0 2024-03-26T18:54:29.014 INFO:tasks.workunit.client.1.smithi190.stdout:2/343: dwrite d6/d9/d48/f65 [4194304,4194304] 0 2024-03-26T18:54:29.015 INFO:tasks.workunit.client.1.smithi190.stdout:8/361: creat d9/d39/d79/f7d x:0 0 0 2024-03-26T18:54:29.015 INFO:tasks.workunit.client.1.smithi190.stdout:8/362: readlink d9/d48/l6c 0 2024-03-26T18:54:29.015 INFO:tasks.workunit.client.1.smithi190.stdout:8/363: creat d9/d39/d79/f7e x:0 0 0 2024-03-26T18:54:29.015 INFO:tasks.workunit.client.1.smithi190.stdout:8/364: write d9/dd/d15/f45 [1165180,72707] 0 2024-03-26T18:54:29.016 INFO:tasks.workunit.client.1.smithi190.stdout:7/366: dwrite d3/d6/d9/d1f/d41/d50/f52 [0,4194304] 0 2024-03-26T18:54:29.016 INFO:tasks.workunit.client.1.smithi190.stdout:7/367: chown d3/d1b/l37 1240157 1 2024-03-26T18:54:29.017 INFO:tasks.workunit.client.1.smithi190.stdout:8/365: unlink d9/dd/f34 0 2024-03-26T18:54:29.018 INFO:tasks.workunit.client.1.smithi190.stdout:2/344: write d6/d9/d10/f2c [2377341,32704] 0 2024-03-26T18:54:29.018 INFO:tasks.workunit.client.1.smithi190.stdout:2/345: read - d6/d9/f41 zero size 2024-03-26T18:54:29.018 INFO:tasks.workunit.client.1.smithi190.stdout:2/346: stat d6/d9/f41 0 2024-03-26T18:54:29.018 INFO:tasks.workunit.client.1.smithi190.stdout:2/347: readlink d6/d9/d3a/d3d/l77 0 2024-03-26T18:54:29.018 INFO:tasks.workunit.client.1.smithi190.stdout:7/368: mkdir d3/d6/d9/d14/d59/d73/d8b 0 2024-03-26T18:54:29.021 INFO:tasks.workunit.client.1.smithi190.stdout:8/366: symlink d9/dd/d23/d36/d37/d3a/d43/l7f 0 2024-03-26T18:54:29.021 INFO:tasks.workunit.client.1.smithi190.stdout:8/367: read - d9/d39/d79/f7e zero size 2024-03-26T18:54:29.021 INFO:tasks.workunit.client.1.smithi190.stdout:8/368: read - d9/d39/d67/f68 zero size 2024-03-26T18:54:29.021 INFO:tasks.workunit.client.1.smithi190.stdout:2/348: dread d6/d9/d1a/f43 [0,4194304] 0 2024-03-26T18:54:29.021 INFO:tasks.workunit.client.1.smithi190.stdout:2/349: fsync d6/d9/d3a/f3b 0 2024-03-26T18:54:29.021 INFO:tasks.workunit.client.1.smithi190.stdout:8/369: unlink d9/d16/f1f 0 2024-03-26T18:54:29.022 INFO:tasks.workunit.client.1.smithi190.stdout:2/350: creat d6/d9/d45/d6b/f79 x:0 0 0 2024-03-26T18:54:29.023 INFO:tasks.workunit.client.1.smithi190.stdout:2/351: creat d6/f7a x:0 0 0 2024-03-26T18:54:29.024 INFO:tasks.workunit.client.1.smithi190.stdout:2/352: symlink d6/l7b 0 2024-03-26T18:54:29.025 INFO:tasks.workunit.client.1.smithi190.stdout:2/353: mkdir d6/d9/d1a/d72/d7c 0 2024-03-26T18:54:29.032 INFO:tasks.workunit.client.1.smithi190.stdout:7/369: dread d3/fc [0,4194304] 0 2024-03-26T18:54:29.032 INFO:tasks.workunit.client.1.smithi190.stdout:9/564: dwrite d12/fb0 [0,4194304] 0 2024-03-26T18:54:29.034 INFO:tasks.workunit.client.1.smithi190.stdout:9/565: mkdir d12/d2d/d1d/d23/d83/dc9 0 2024-03-26T18:54:29.034 INFO:tasks.workunit.client.1.smithi190.stdout:9/566: dread - d12/d2d/d1d/d23/f69 zero size 2024-03-26T18:54:29.035 INFO:tasks.workunit.client.1.smithi190.stdout:9/567: creat d12/d2d/d1d/d23/d2a/d71/daa/db3/fca x:0 0 0 2024-03-26T18:54:29.035 INFO:tasks.workunit.client.1.smithi190.stdout:9/568: write d12/d2d/f1b [47940,13123] 0 2024-03-26T18:54:29.035 INFO:tasks.workunit.client.1.smithi190.stdout:9/569: truncate d12/d2d/f5e 1295665 0 2024-03-26T18:54:29.036 INFO:tasks.workunit.client.1.smithi190.stdout:9/570: rmdir d12/d2d/d1d/d23/d6a 39 2024-03-26T18:54:29.036 INFO:tasks.workunit.client.1.smithi190.stdout:9/571: symlink d12/d2d/d1d/d23/d2a/d71/daa/db3/lcb 0 2024-03-26T18:54:29.037 INFO:tasks.workunit.client.1.smithi190.stdout:9/572: chown d12/d2d/d1d/d23/l45 645 1 2024-03-26T18:54:29.037 INFO:tasks.workunit.client.1.smithi190.stdout:9/573: chown d12/d2d/d1d/d23/d6a/f7e 104902737 1 2024-03-26T18:54:29.037 INFO:tasks.workunit.client.1.smithi190.stdout:5/232: dwrite d2/d17/d1b/d1e/d3a/f4d [0,4194304] 0 2024-03-26T18:54:29.037 INFO:tasks.workunit.client.1.smithi190.stdout:4/360: dwrite d7/f35 [4194304,4194304] 0 2024-03-26T18:54:29.037 INFO:tasks.workunit.client.1.smithi190.stdout:5/233: fsync d2/d17/d1b/d1e/d3a/f40 0 2024-03-26T18:54:29.038 INFO:tasks.workunit.client.1.smithi190.stdout:5/234: chown d2/f6 1003933 1 2024-03-26T18:54:29.038 INFO:tasks.workunit.client.0.smithi102.stdout:5/269: dwrite d2/d2c/d38/f5c [0,4194304] 0 2024-03-26T18:54:29.040 INFO:tasks.workunit.client.1.smithi190.stdout:2/354: dread d6/d9/f38 [0,4194304] 0 2024-03-26T18:54:29.040 INFO:tasks.workunit.client.1.smithi190.stdout:5/235: write d2/f14 [383769,22597] 0 2024-03-26T18:54:29.040 INFO:tasks.workunit.client.1.smithi190.stdout:5/236: read d2/f4 [2855699,111868] 0 2024-03-26T18:54:29.040 INFO:tasks.workunit.client.1.smithi190.stdout:5/237: read - d2/d17/d1b/d1e/d3a/f40 zero size 2024-03-26T18:54:29.041 INFO:tasks.workunit.client.1.smithi190.stdout:5/238: chown d2/d17/d1b/d22/c35 0 1 2024-03-26T18:54:29.041 INFO:tasks.workunit.client.1.smithi190.stdout:5/239: stat d2/d17/d1b/d22/d24/d3b/f46 0 2024-03-26T18:54:29.041 INFO:tasks.workunit.client.1.smithi190.stdout:4/361: rename d7/d1d/d42/l6c to d7/d8/d14/d5c/l7b 0 2024-03-26T18:54:29.041 INFO:tasks.workunit.client.1.smithi190.stdout:4/362: chown d7/d8/d1c/d20/f3f 1609 1 2024-03-26T18:54:29.041 INFO:tasks.workunit.client.1.smithi190.stdout:4/363: chown d7/d8/d14/d5c 34984116 1 2024-03-26T18:54:29.042 INFO:tasks.workunit.client.1.smithi190.stdout:3/312: dwrite d1/de/d1c/d36/f5d [0,4194304] 0 2024-03-26T18:54:29.043 INFO:tasks.workunit.client.1.smithi190.stdout:8/370: dwrite d9/dd/f2d [0,4194304] 0 2024-03-26T18:54:29.044 INFO:tasks.workunit.client.1.smithi190.stdout:3/313: read d1/de/d1c/d36/f40 [429942,109036] 0 2024-03-26T18:54:29.044 INFO:tasks.workunit.client.1.smithi190.stdout:3/314: read d1/d3/df/d18/f30 [2747816,49768] 0 2024-03-26T18:54:29.044 INFO:tasks.workunit.client.1.smithi190.stdout:3/315: chown d1/d3/df 278728568 1 2024-03-26T18:54:29.044 INFO:tasks.workunit.client.1.smithi190.stdout:3/316: chown d1/d3/df/d31/d5b 327 1 2024-03-26T18:54:29.044 INFO:tasks.workunit.client.1.smithi190.stdout:3/317: stat d1/d3 0 2024-03-26T18:54:29.044 INFO:tasks.workunit.client.1.smithi190.stdout:5/240: link d2/d17/d1b/d22/f38 d2/d17/d1b/d1e/d3a/f55 0 2024-03-26T18:54:29.045 INFO:tasks.workunit.client.1.smithi190.stdout:7/370: dread d3/f35 [0,4194304] 0 2024-03-26T18:54:29.045 INFO:tasks.workunit.client.1.smithi190.stdout:6/318: dwrite d17/d28/d2b/f60 [0,4194304] 0 2024-03-26T18:54:29.047 INFO:tasks.workunit.client.1.smithi190.stdout:4/364: mknod d7/c7c 0 2024-03-26T18:54:29.047 INFO:tasks.workunit.client.1.smithi190.stdout:6/319: dread d17/d28/d2b/f3c [0,4194304] 0 2024-03-26T18:54:29.054 INFO:tasks.workunit.client.1.smithi190.stdout:8/371: creat d9/dd/d23/d36/d37/d3a/d43/f80 x:0 0 0 2024-03-26T18:54:29.057 INFO:tasks.workunit.client.1.smithi190.stdout:5/241: creat d2/d17/d1b/d22/d24/d3b/f56 x:0 0 0 2024-03-26T18:54:29.057 INFO:tasks.workunit.client.1.smithi190.stdout:5/242: stat d2/d17/d1b/d1e/d3a/f4d 0 2024-03-26T18:54:29.057 INFO:tasks.workunit.client.1.smithi190.stdout:5/243: chown d2/d17/d1b/d22/f54 0 1 2024-03-26T18:54:29.059 INFO:tasks.workunit.client.1.smithi190.stdout:7/371: creat d3/d6/f8c x:0 0 0 2024-03-26T18:54:29.061 INFO:tasks.workunit.client.1.smithi190.stdout:9/574: dwrite d12/d2d/f46 [4194304,4194304] 0 2024-03-26T18:54:29.061 INFO:tasks.workunit.client.1.smithi190.stdout:3/318: dread d1/d3/f10 [8388608,4194304] 0 2024-03-26T18:54:29.061 INFO:tasks.workunit.client.1.smithi190.stdout:9/575: chown d12/d2d/d1d/d23/d83/d91/fb9 12990 1 2024-03-26T18:54:29.061 INFO:tasks.workunit.client.1.smithi190.stdout:3/319: chown d1/f14 447459277 1 2024-03-26T18:54:29.062 INFO:tasks.workunit.client.1.smithi190.stdout:3/320: chown d1/de/f29 16376859 1 2024-03-26T18:54:29.062 INFO:tasks.workunit.client.1.smithi190.stdout:3/321: creat d1/d3/df/d18/d35/f60 x:0 0 0 2024-03-26T18:54:29.062 INFO:tasks.workunit.client.1.smithi190.stdout:3/322: dread - d1/d3/df/d31/f56 zero size 2024-03-26T18:54:29.062 INFO:tasks.workunit.client.1.smithi190.stdout:3/323: stat d1/de/d4e/f54 0 2024-03-26T18:54:29.063 INFO:tasks.workunit.client.1.smithi190.stdout:4/365: mkdir d7/d8/d14/d4b/d7d 0 2024-03-26T18:54:29.064 INFO:tasks.workunit.client.1.smithi190.stdout:6/320: creat d17/d28/d46/d4d/f71 x:0 0 0 2024-03-26T18:54:29.065 INFO:tasks.workunit.client.1.smithi190.stdout:6/321: chown d17/d28/d46/d4d/d51 25540 1 2024-03-26T18:54:29.072 INFO:tasks.workunit.client.1.smithi190.stdout:5/244: dread d2/d17/d1b/d1e/d3a/f4d [0,4194304] 0 2024-03-26T18:54:29.077 INFO:tasks.workunit.client.1.smithi190.stdout:9/576: dread d12/f14 [0,4194304] 0 2024-03-26T18:54:29.077 INFO:tasks.workunit.client.1.smithi190.stdout:5/245: write d2/d17/d1b/d1e/d3a/f4d [2100658,49698] 0 2024-03-26T18:54:29.078 INFO:tasks.workunit.client.1.smithi190.stdout:3/324: dread d1/de/d1c/f2b [0,4194304] 0 2024-03-26T18:54:29.083 INFO:tasks.workunit.client.1.smithi190.stdout:3/325: dread d1/de/d1c/d36/f40 [0,4194304] 0 2024-03-26T18:54:29.085 INFO:tasks.workunit.client.1.smithi190.stdout:8/372: link d9/dd/f78 d9/d16/d73/f81 0 2024-03-26T18:54:29.085 INFO:tasks.workunit.client.1.smithi190.stdout:8/373: write d9/dd/d23/d36/d37/d3a/d61/f72 [4401197,57140] 0 2024-03-26T18:54:29.085 INFO:tasks.workunit.client.1.smithi190.stdout:8/374: creat d9/d39/f82 x:0 0 0 2024-03-26T18:54:29.085 INFO:tasks.workunit.client.1.smithi190.stdout:8/375: truncate d9/d39/f6d 16188 0 2024-03-26T18:54:29.088 INFO:tasks.workunit.client.1.smithi190.stdout:6/322: mknod d17/d28/d2b/c72 0 2024-03-26T18:54:29.089 INFO:tasks.workunit.client.1.smithi190.stdout:7/372: dwrite d3/d6/d9/d1f/f3d [0,4194304] 0 2024-03-26T18:54:29.089 INFO:tasks.workunit.client.1.smithi190.stdout:2/355: dwrite d6/d9/d10/f2c [0,4194304] 0 2024-03-26T18:54:29.090 INFO:tasks.workunit.client.1.smithi190.stdout:4/366: dwrite d7/d1d/f32 [0,4194304] 0 2024-03-26T18:54:29.091 INFO:tasks.workunit.client.1.smithi190.stdout:9/577: symlink d12/d2d/d1d/da1/lcc 0 2024-03-26T18:54:29.091 INFO:tasks.workunit.client.1.smithi190.stdout:9/578: fdatasync d12/d2d/d1d/d54/f90 0 2024-03-26T18:54:29.094 INFO:tasks.workunit.client.1.smithi190.stdout:8/376: dread d9/f33 [0,4194304] 0 2024-03-26T18:54:29.094 INFO:tasks.workunit.client.1.smithi190.stdout:5/246: link d2/d17/l4a d2/d17/d1b/l57 0 2024-03-26T18:54:29.095 INFO:tasks.workunit.client.1.smithi190.stdout:5/247: creat d2/d17/d1b/f58 x:0 0 0 2024-03-26T18:54:29.097 INFO:tasks.workunit.client.1.smithi190.stdout:3/326: mkdir d1/d3/df/d18/d35/d41/d61 0 2024-03-26T18:54:29.102 INFO:tasks.workunit.client.1.smithi190.stdout:9/579: dread f6 [0,4194304] 0 2024-03-26T18:54:29.102 INFO:tasks.workunit.client.1.smithi190.stdout:9/580: creat d12/d2d/d1d/d23/d2a/d3f/fcd x:0 0 0 2024-03-26T18:54:29.102 INFO:tasks.workunit.client.1.smithi190.stdout:9/581: dread - d12/d2d/d1d/f77 zero size 2024-03-26T18:54:29.102 INFO:tasks.workunit.client.1.smithi190.stdout:9/582: creat d12/d80/fce x:0 0 0 2024-03-26T18:54:29.102 INFO:tasks.workunit.client.1.smithi190.stdout:9/583: fsync d12/d2d/f3e 0 2024-03-26T18:54:29.103 INFO:tasks.workunit.client.1.smithi190.stdout:3/327: dread d1/de/d1c/f2b [0,4194304] 0 2024-03-26T18:54:29.103 INFO:tasks.workunit.client.1.smithi190.stdout:3/328: chown d1/d3/df/d18/d35/f39 1 1 2024-03-26T18:54:29.103 INFO:tasks.workunit.client.1.smithi190.stdout:6/323: rename c6 to d17/d28/d2c/d63/c73 0 2024-03-26T18:54:29.110 INFO:tasks.workunit.client.1.smithi190.stdout:7/373: mknod d3/d6/d9/d1f/c8d 0 2024-03-26T18:54:29.111 INFO:tasks.workunit.client.1.smithi190.stdout:8/377: dread d9/dd/d15/f1c [0,4194304] 0 2024-03-26T18:54:29.116 INFO:tasks.workunit.client.1.smithi190.stdout:3/329: dread d1/d3/df/f19 [0,4194304] 0 2024-03-26T18:54:29.116 INFO:tasks.workunit.client.1.smithi190.stdout:3/330: fdatasync d1/f2c 0 2024-03-26T18:54:29.116 INFO:tasks.workunit.client.1.smithi190.stdout:2/356: dwrite d6/d9/d3a/d3d/d1c/f4a [0,4194304] 0 2024-03-26T18:54:29.116 INFO:tasks.workunit.client.1.smithi190.stdout:3/331: readlink d1/d3/df/d18/d1b/l37 0 2024-03-26T18:54:29.116 INFO:tasks.workunit.client.1.smithi190.stdout:5/248: link d2/d17/d1b/d22/c35 d2/d17/d1b/d22/d24/d3b/c59 0 2024-03-26T18:54:29.121 INFO:tasks.workunit.client.1.smithi190.stdout:9/584: unlink d12/d2d/d1d/d23/d2a/d3f/d55/l66 0 2024-03-26T18:54:29.121 INFO:tasks.workunit.client.1.smithi190.stdout:9/585: readlink d12/d2d/d1d/d23/d2b/l74 0 2024-03-26T18:54:29.121 INFO:tasks.workunit.client.1.smithi190.stdout:9/586: chown d12/d2d/l21 26389597 1 2024-03-26T18:54:29.122 INFO:tasks.workunit.client.1.smithi190.stdout:9/587: write d12/d2d/d1d/f77 [563629,27555] 0 2024-03-26T18:54:29.123 INFO:tasks.workunit.client.1.smithi190.stdout:6/324: mkdir d17/d28/d74 0 2024-03-26T18:54:29.123 INFO:tasks.workunit.client.1.smithi190.stdout:6/325: write f8 [672141,124394] 0 2024-03-26T18:54:29.124 INFO:tasks.workunit.client.1.smithi190.stdout:4/367: dwrite d7/d8/d14/d4b/f4c [0,4194304] 0 2024-03-26T18:54:29.124 INFO:tasks.workunit.client.1.smithi190.stdout:7/374: rename d3/d6/d9/d1f/f34 to d3/d6/d9/d14/f8e 0 2024-03-26T18:54:29.125 INFO:tasks.workunit.client.1.smithi190.stdout:5/249: dread d2/f10 [0,4194304] 0 2024-03-26T18:54:29.126 INFO:tasks.workunit.client.1.smithi190.stdout:8/378: mknod d9/d16/d73/c83 0 2024-03-26T18:54:29.130 INFO:tasks.workunit.client.1.smithi190.stdout:3/332: unlink d1/d3/f12 0 2024-03-26T18:54:29.133 INFO:tasks.workunit.client.1.smithi190.stdout:2/357: creat d6/d9/d1a/d72/d7c/f7d x:0 0 0 2024-03-26T18:54:29.133 INFO:tasks.workunit.client.1.smithi190.stdout:2/358: dread - d6/d9/d1a/d72/d7c/f7d zero size 2024-03-26T18:54:29.133 INFO:tasks.workunit.client.1.smithi190.stdout:2/359: creat d6/d9/d48/f7e x:0 0 0 2024-03-26T18:54:29.133 INFO:tasks.workunit.client.1.smithi190.stdout:2/360: write d6/d9/d45/d6b/f79 [8281,8522] 0 2024-03-26T18:54:29.134 INFO:tasks.workunit.client.1.smithi190.stdout:9/588: mknod d12/d80/db2/ccf 0 2024-03-26T18:54:29.136 INFO:tasks.workunit.client.1.smithi190.stdout:6/326: symlink d17/d28/d38/l75 0 2024-03-26T18:54:29.139 INFO:tasks.workunit.client.1.smithi190.stdout:4/368: creat d7/d8/d14/d6e/d6f/f7e x:0 0 0 2024-03-26T18:54:29.140 INFO:tasks.workunit.client.1.smithi190.stdout:3/333: dread d1/de/d1c/d36/f3a [0,4194304] 0 2024-03-26T18:54:29.141 INFO:tasks.workunit.client.1.smithi190.stdout:3/334: stat d1/f14 0 2024-03-26T18:54:29.141 INFO:tasks.workunit.client.1.smithi190.stdout:7/375: mkdir d3/d6/d9/d1f/d41/d50/d53/d8f 0 2024-03-26T18:54:29.146 INFO:tasks.workunit.client.1.smithi190.stdout:0/424: sync 2024-03-26T18:54:29.146 INFO:tasks.workunit.client.1.smithi190.stdout:0/425: fdatasync d0/dd/d20/d1c/d4e/f4f 0 2024-03-26T18:54:29.146 INFO:tasks.workunit.client.1.smithi190.stdout:8/379: symlink d9/dd/d23/d36/d37/d3a/l84 0 2024-03-26T18:54:29.155 INFO:tasks.workunit.client.1.smithi190.stdout:9/589: dwrite d12/d2d/d1d/d23/f5f [0,4194304] 0 2024-03-26T18:54:29.163 INFO:tasks.workunit.client.1.smithi190.stdout:2/361: creat d6/d9/d10/d5f/f7f x:0 0 0 2024-03-26T18:54:29.163 INFO:tasks.workunit.client.1.smithi190.stdout:2/362: fdatasync d6/f4d 0 2024-03-26T18:54:29.163 INFO:tasks.workunit.client.1.smithi190.stdout:2/363: creat d6/d9/d45/d6b/f80 x:0 0 0 2024-03-26T18:54:29.164 INFO:tasks.workunit.client.1.smithi190.stdout:6/327: symlink d17/d28/d2c/l76 0 2024-03-26T18:54:29.164 INFO:tasks.workunit.client.1.smithi190.stdout:2/364: write d6/d9/d10/f2d [26989,129441] 0 2024-03-26T18:54:29.165 INFO:tasks.workunit.client.1.smithi190.stdout:3/335: dwrite d1/de/d1c/d36/f3a [0,4194304] 0 2024-03-26T18:54:29.166 INFO:tasks.workunit.client.1.smithi190.stdout:4/369: readlink d7/d8/d1c/d20/d40/l53 0 2024-03-26T18:54:29.168 INFO:tasks.workunit.client.1.smithi190.stdout:7/376: mkdir d3/d6/d9/d1f/d41/d65/d90 0 2024-03-26T18:54:29.168 INFO:tasks.workunit.client.1.smithi190.stdout:7/377: creat d3/d6/f91 x:0 0 0 2024-03-26T18:54:29.168 INFO:tasks.workunit.client.1.smithi190.stdout:7/378: stat d3/d6/d9/d14/d36/f6b 0 2024-03-26T18:54:29.182 INFO:tasks.workunit.client.1.smithi190.stdout:5/250: rmdir d2/d17 39 2024-03-26T18:54:29.182 INFO:tasks.workunit.client.1.smithi190.stdout:5/251: write d2/f15 [285289,113251] 0 2024-03-26T18:54:29.183 INFO:tasks.workunit.client.1.smithi190.stdout:5/252: write d2/d17/d1b/d1e/d3a/f40 [314797,123657] 0 2024-03-26T18:54:29.183 INFO:tasks.workunit.client.1.smithi190.stdout:5/253: truncate d2/d17/d1b/f58 691038 0 2024-03-26T18:54:29.186 INFO:tasks.workunit.client.1.smithi190.stdout:0/426: write d0/dd/d20/d1c/f28 [642988,128059] 0 2024-03-26T18:54:29.188 INFO:tasks.workunit.client.1.smithi190.stdout:8/380: unlink d9/dd/d15/d32/c46 0 2024-03-26T18:54:29.197 INFO:tasks.workunit.client.1.smithi190.stdout:3/336: creat d1/d3/d52/f62 x:0 0 0 2024-03-26T18:54:29.197 INFO:tasks.workunit.client.1.smithi190.stdout:3/337: fsync d1/d3/df/f19 0 2024-03-26T18:54:29.197 INFO:tasks.workunit.client.1.smithi190.stdout:3/338: write d1/f44 [594215,78641] 0 2024-03-26T18:54:29.201 INFO:tasks.workunit.client.1.smithi190.stdout:7/379: mkdir d3/d6/d9/d1f/d41/d65/d90/d92 0 2024-03-26T18:54:29.206 INFO:tasks.workunit.client.1.smithi190.stdout:5/254: truncate d2/f10 3430494 0 2024-03-26T18:54:29.206 INFO:tasks.workunit.client.1.smithi190.stdout:5/255: dread - d2/d17/d1b/d1e/d3a/d3c/f43 zero size 2024-03-26T18:54:29.209 INFO:tasks.workunit.client.1.smithi190.stdout:0/427: unlink d0/dd/d20/d1c/f56 0 2024-03-26T18:54:29.210 INFO:tasks.workunit.client.1.smithi190.stdout:1/294: sync 2024-03-26T18:54:29.212 INFO:tasks.workunit.client.1.smithi190.stdout:8/381: mknod d9/d16/d73/c85 0 2024-03-26T18:54:29.212 INFO:tasks.workunit.client.1.smithi190.stdout:8/382: fdatasync d9/d48/f62 0 2024-03-26T18:54:29.212 INFO:tasks.workunit.client.1.smithi190.stdout:8/383: dread - d9/d16/f63 zero size 2024-03-26T18:54:29.213 INFO:tasks.workunit.client.1.smithi190.stdout:9/590: rename d12/d2d/d1d/d23/d2a/d3f/fcd to d12/d2d/d1d/da1/dba/fd0 0 2024-03-26T18:54:29.213 INFO:tasks.workunit.client.1.smithi190.stdout:9/591: getdents d12/d2d/d1d/d23/d6a 0 2024-03-26T18:54:29.214 INFO:tasks.workunit.client.1.smithi190.stdout:9/592: stat d12/d2d/d1d/d23/d8e 0 2024-03-26T18:54:29.214 INFO:tasks.workunit.client.1.smithi190.stdout:9/593: stat d12/d2d/d1d/d23/d2b/f68 0 2024-03-26T18:54:29.215 INFO:tasks.workunit.client.1.smithi190.stdout:6/328: getdents d17/d28/d46/d4d 0 2024-03-26T18:54:29.217 INFO:tasks.workunit.client.1.smithi190.stdout:4/370: getdents d7/d8/d14/d23/d28/d38 0 2024-03-26T18:54:29.217 INFO:tasks.workunit.client.1.smithi190.stdout:4/371: read - d7/d1d/d42/d4a/f66 zero size 2024-03-26T18:54:29.218 INFO:tasks.workunit.client.1.smithi190.stdout:0/428: unlink d0/f6 0 2024-03-26T18:54:29.220 INFO:tasks.workunit.client.1.smithi190.stdout:1/295: symlink d3/da/dd/d2f/l66 0 2024-03-26T18:54:29.222 INFO:tasks.workunit.client.1.smithi190.stdout:1/296: read d3/d9/db/f23 [293194,60340] 0 2024-03-26T18:54:29.224 INFO:tasks.workunit.client.1.smithi190.stdout:2/365: rename d6/d9/d10/l1f to d6/d9/d3a/d3d/d2e/l81 0 2024-03-26T18:54:29.224 INFO:tasks.workunit.client.1.smithi190.stdout:0/429: creat d0/dd/d24/d4b/d5e/f95 x:0 0 0 2024-03-26T18:54:29.226 INFO:tasks.workunit.client.1.smithi190.stdout:3/339: dwrite d1/d3/f10 [4194304,4194304] 0 2024-03-26T18:54:29.230 INFO:tasks.workunit.client.1.smithi190.stdout:5/256: rename d2/d17/d1b/l57 to d2/d17/l5a 0 2024-03-26T18:54:29.230 INFO:tasks.workunit.client.1.smithi190.stdout:8/384: rename d9 to d9/d16/d86 22 2024-03-26T18:54:29.231 INFO:tasks.workunit.client.1.smithi190.stdout:7/380: dwrite d3/d1b/f25 [0,4194304] 0 2024-03-26T18:54:29.231 INFO:tasks.workunit.client.1.smithi190.stdout:7/381: stat d3/d1b/f28 0 2024-03-26T18:54:29.232 INFO:tasks.workunit.client.1.smithi190.stdout:6/329: dread d17/f19 [0,4194304] 0 2024-03-26T18:54:29.232 INFO:tasks.workunit.client.1.smithi190.stdout:2/366: rmdir d6/d9/d3a/d3d/d67 39 2024-03-26T18:54:29.235 INFO:tasks.workunit.client.1.smithi190.stdout:0/430: mknod d0/dd/d24/d4b/d5e/d6b/c96 0 2024-03-26T18:54:29.237 INFO:tasks.workunit.client.1.smithi190.stdout:9/594: rename d12/d80/fce to d12/d2d/d1d/d23/d6a/fd1 0 2024-03-26T18:54:29.237 INFO:tasks.workunit.client.1.smithi190.stdout:9/595: write d12/d2d/d1d/d54/f88 [909442,110931] 0 2024-03-26T18:54:29.237 INFO:tasks.workunit.client.1.smithi190.stdout:4/372: read d7/d8/d1c/d20/f58 [331365,39999] 0 2024-03-26T18:54:29.238 INFO:tasks.workunit.client.1.smithi190.stdout:8/385: mkdir d9/d16/d87 0 2024-03-26T18:54:29.240 INFO:tasks.workunit.client.1.smithi190.stdout:5/257: symlink d2/d17/d1b/d1e/d3a/l5b 0 2024-03-26T18:54:29.243 INFO:tasks.workunit.client.1.smithi190.stdout:1/297: dwrite d3/d9/db/f32 [0,4194304] 0 2024-03-26T18:54:29.244 INFO:tasks.workunit.client.1.smithi190.stdout:2/367: symlink d6/d9/d3a/l82 0 2024-03-26T18:54:29.248 INFO:tasks.workunit.client.1.smithi190.stdout:0/431: symlink d0/dd/l97 0 2024-03-26T18:54:29.254 INFO:tasks.workunit.client.1.smithi190.stdout:3/340: rename d1/f44 to d1/d3/d5a/f63 0 2024-03-26T18:54:29.257 INFO:tasks.workunit.client.1.smithi190.stdout:0/432: write d0/dd/d20/d61/f7b [1443960,59457] 0 2024-03-26T18:54:29.257 INFO:tasks.workunit.client.1.smithi190.stdout:4/373: unlink d7/d8/d14/f1a 0 2024-03-26T18:54:29.263 INFO:tasks.workunit.client.1.smithi190.stdout:4/374: dread d7/d8/d1c/d20/f58 [0,4194304] 0 2024-03-26T18:54:29.263 INFO:tasks.workunit.client.1.smithi190.stdout:4/375: write d7/f18 [1608373,31170] 0 2024-03-26T18:54:29.269 INFO:tasks.workunit.client.1.smithi190.stdout:3/341: dread d1/f11 [0,4194304] 0 2024-03-26T18:54:29.271 INFO:tasks.workunit.client.1.smithi190.stdout:9/596: mkdir d12/d2d/d1d/da1/dba/dd2 0 2024-03-26T18:54:29.272 INFO:tasks.workunit.client.1.smithi190.stdout:9/597: dread d12/d2d/d1d/d54/f8a [0,4194304] 0 2024-03-26T18:54:29.272 INFO:tasks.workunit.client.1.smithi190.stdout:9/598: creat d12/d2d/d1d/fd3 x:0 0 0 2024-03-26T18:54:29.274 INFO:tasks.workunit.client.1.smithi190.stdout:0/433: fdatasync d0/dd/d20/d1c/f6c 0 2024-03-26T18:54:29.275 INFO:tasks.workunit.client.1.smithi190.stdout:8/386: rmdir d9/d39/d79 39 2024-03-26T18:54:29.276 INFO:tasks.workunit.client.1.smithi190.stdout:2/368: dwrite d6/d9/d1a/f26 [0,4194304] 0 2024-03-26T18:54:29.276 INFO:tasks.workunit.client.1.smithi190.stdout:2/369: write d6/d9/d10/f34 [1329839,124444] 0 2024-03-26T18:54:29.276 INFO:tasks.workunit.client.1.smithi190.stdout:5/258: dwrite d2/d17/d1b/d22/f54 [0,4194304] 0 2024-03-26T18:54:29.276 INFO:tasks.workunit.client.1.smithi190.stdout:2/370: chown d6/c3e 37 1 2024-03-26T18:54:29.276 INFO:tasks.workunit.client.1.smithi190.stdout:7/382: dwrite d3/f29 [0,4194304] 0 2024-03-26T18:54:29.277 INFO:tasks.workunit.client.1.smithi190.stdout:3/342: mkdir d1/d3/df/d31/d5b/d64 0 2024-03-26T18:54:29.277 INFO:tasks.workunit.client.1.smithi190.stdout:3/343: stat d1/d3/df/d18/f30 0 2024-03-26T18:54:29.279 INFO:tasks.workunit.client.1.smithi190.stdout:6/330: rename d17/d28/d46/d4d to d17/d28/d2c/d77 0 2024-03-26T18:54:29.279 INFO:tasks.workunit.client.1.smithi190.stdout:6/331: write d17/d28/d2b/f3c [1113645,17745] 0 2024-03-26T18:54:29.285 INFO:tasks.workunit.client.1.smithi190.stdout:0/434: mknod d0/dd/d20/d1c/d8b/c98 0 2024-03-26T18:54:29.286 INFO:tasks.workunit.client.1.smithi190.stdout:3/344: dread d1/de/d1c/d36/f3a [0,4194304] 0 2024-03-26T18:54:29.287 INFO:tasks.workunit.client.1.smithi190.stdout:2/371: write d6/d9/d10/f13 [2840862,34366] 0 2024-03-26T18:54:29.287 INFO:tasks.workunit.client.1.smithi190.stdout:2/372: stat d6/d9/d3a/d3d/d2e/d31/c50 0 2024-03-26T18:54:29.287 INFO:tasks.workunit.client.1.smithi190.stdout:8/387: mkdir d9/dd/d23/d88 0 2024-03-26T18:54:29.287 INFO:tasks.workunit.client.1.smithi190.stdout:8/388: read - d9/dd/f78 zero size 2024-03-26T18:54:29.296 INFO:tasks.workunit.client.1.smithi190.stdout:5/259: rmdir d2/d17/d1b/d1e/d3a/d3c 39 2024-03-26T18:54:29.297 INFO:tasks.workunit.client.1.smithi190.stdout:5/260: dread - d2/d17/d1b/d22/f2d zero size 2024-03-26T18:54:29.302 INFO:tasks.workunit.client.1.smithi190.stdout:7/383: mkdir d3/d6/d9/d1f/d93 0 2024-03-26T18:54:29.302 INFO:tasks.workunit.client.1.smithi190.stdout:3/345: dwrite d1/d3/f4b [0,4194304] 0 2024-03-26T18:54:29.312 INFO:tasks.workunit.client.1.smithi190.stdout:6/332: symlink d17/d28/d2c/d77/d51/l78 0 2024-03-26T18:54:29.313 INFO:tasks.workunit.client.1.smithi190.stdout:9/599: getdents d12/d2d/d1d/d23/d2a/d71 0 2024-03-26T18:54:29.315 INFO:tasks.workunit.client.1.smithi190.stdout:0/435: creat d0/f99 x:0 0 0 2024-03-26T18:54:29.315 INFO:tasks.workunit.client.1.smithi190.stdout:0/436: readlink d0/l3 0 2024-03-26T18:54:29.319 INFO:tasks.workunit.client.1.smithi190.stdout:2/373: link d6/d9/d3a/f3b d6/d9/d3a/f83 0 2024-03-26T18:54:29.319 INFO:tasks.workunit.client.1.smithi190.stdout:6/333: write d17/f23 [2755658,114500] 0 2024-03-26T18:54:29.320 INFO:tasks.workunit.client.1.smithi190.stdout:6/334: chown f1 16 1 2024-03-26T18:54:29.320 INFO:tasks.workunit.client.1.smithi190.stdout:6/335: write f1 [1124242,70225] 0 2024-03-26T18:54:29.320 INFO:tasks.workunit.client.1.smithi190.stdout:6/336: dread - d17/d28/d2c/d50/f67 zero size 2024-03-26T18:54:29.320 INFO:tasks.workunit.client.1.smithi190.stdout:8/389: mknod d9/d39/d67/c89 0 2024-03-26T18:54:29.322 INFO:tasks.workunit.client.1.smithi190.stdout:5/261: symlink d2/d17/d1b/l5c 0 2024-03-26T18:54:29.324 INFO:tasks.workunit.client.1.smithi190.stdout:7/384: mknod d3/d6/d9/d1f/d41/d50/d53/d87/c94 0 2024-03-26T18:54:29.328 INFO:tasks.workunit.client.1.smithi190.stdout:0/437: getdents d0/dd/d20/d1c/d8b/d8d 0 2024-03-26T18:54:29.331 INFO:tasks.workunit.client.1.smithi190.stdout:8/390: symlink d9/d48/l8a 0 2024-03-26T18:54:29.331 INFO:tasks.workunit.client.1.smithi190.stdout:8/391: readlink d9/dd/l28 0 2024-03-26T18:54:29.331 INFO:tasks.workunit.client.1.smithi190.stdout:8/392: chown d9/d16/c2a 241761233 1 2024-03-26T18:54:29.333 INFO:tasks.workunit.client.1.smithi190.stdout:5/262: mknod d2/d17/d1b/d1e/c5d 0 2024-03-26T18:54:29.335 INFO:tasks.workunit.client.1.smithi190.stdout:0/438: dread d0/dd/d24/f2b [4194304,4194304] 0 2024-03-26T18:54:29.337 INFO:tasks.workunit.client.1.smithi190.stdout:6/337: dwrite d17/d28/d2b/f3c [0,4194304] 0 2024-03-26T18:54:29.339 INFO:tasks.workunit.client.1.smithi190.stdout:9/600: rename d12/d2d/d1d/d23/d2a to d12/d2d/d1d/d23/d83/dc9/dd4 0 2024-03-26T18:54:29.340 INFO:tasks.workunit.client.1.smithi190.stdout:9/601: chown d12/c32 104 1 2024-03-26T18:54:29.341 INFO:tasks.workunit.client.1.smithi190.stdout:2/374: truncate d6/d9/d10/f2d 1001367 0 2024-03-26T18:54:29.343 INFO:tasks.workunit.client.1.smithi190.stdout:5/263: creat d2/f5e x:0 0 0 2024-03-26T18:54:29.343 INFO:tasks.workunit.client.1.smithi190.stdout:5/264: chown d2/d17/d1b/d1e/d3a/d3c 51143 1 2024-03-26T18:54:29.348 INFO:tasks.workunit.client.1.smithi190.stdout:7/385: rename d3/d6/d9/d14/d36/l3f to d3/d6/l95 0 2024-03-26T18:54:29.348 INFO:tasks.workunit.client.1.smithi190.stdout:7/386: dread - d3/d6/d9/d14/f7e zero size 2024-03-26T18:54:29.348 INFO:tasks.workunit.client.1.smithi190.stdout:7/387: truncate d3/d6/d9/d1f/d41/d65/f7d 424749 0 2024-03-26T18:54:29.350 INFO:tasks.workunit.client.1.smithi190.stdout:9/602: rmdir d12/d80/db2 39 2024-03-26T18:54:29.352 INFO:tasks.workunit.client.1.smithi190.stdout:2/375: symlink d6/d9/d3a/d3d/d2e/l84 0 2024-03-26T18:54:29.352 INFO:tasks.workunit.client.1.smithi190.stdout:2/376: creat d6/d9/d10/f85 x:0 0 0 2024-03-26T18:54:29.353 INFO:tasks.workunit.client.1.smithi190.stdout:5/265: stat l1 0 2024-03-26T18:54:29.353 INFO:tasks.workunit.client.1.smithi190.stdout:5/266: dread - d2/d17/d1b/d22/d24/d3b/f47 zero size 2024-03-26T18:54:29.353 INFO:tasks.workunit.client.1.smithi190.stdout:5/267: write d2/f12 [2762855,2387] 0 2024-03-26T18:54:29.354 INFO:tasks.workunit.client.1.smithi190.stdout:8/393: dwrite d9/d39/d79/f7e [0,4194304] 0 2024-03-26T18:54:29.354 INFO:tasks.workunit.client.1.smithi190.stdout:9/603: rmdir d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa 39 2024-03-26T18:54:29.358 INFO:tasks.workunit.client.1.smithi190.stdout:2/377: unlink d6/d21/d61/f69 0 2024-03-26T18:54:29.358 INFO:tasks.workunit.client.1.smithi190.stdout:2/378: read f5 [3307576,66925] 0 2024-03-26T18:54:29.360 INFO:tasks.workunit.client.1.smithi190.stdout:0/439: dwrite d0/dd/fe [0,4194304] 0 2024-03-26T18:54:29.360 INFO:tasks.workunit.client.1.smithi190.stdout:0/440: write d0/f1f [5255243,99047] 0 2024-03-26T18:54:29.360 INFO:tasks.workunit.client.1.smithi190.stdout:6/338: dwrite d17/d28/d2c/f61 [0,4194304] 0 2024-03-26T18:54:29.361 INFO:tasks.workunit.client.1.smithi190.stdout:5/268: rename d2/fa to d2/d17/d1b/d1e/f5f 0 2024-03-26T18:54:29.365 INFO:tasks.workunit.client.1.smithi190.stdout:9/604: mkdir d12/d2d/d1d/d23/d83/dd5 0 2024-03-26T18:54:29.368 INFO:tasks.workunit.client.1.smithi190.stdout:0/441: read d0/dd/d20/d61/f7b [4312437,36141] 0 2024-03-26T18:54:29.368 INFO:tasks.workunit.client.1.smithi190.stdout:0/442: fdatasync d0/f99 0 2024-03-26T18:54:29.376 INFO:tasks.workunit.client.1.smithi190.stdout:2/379: dwrite d6/d9/d10/f85 [0,4194304] 0 2024-03-26T18:54:29.381 INFO:tasks.workunit.client.1.smithi190.stdout:6/339: mkdir d17/d28/d38/d79 0 2024-03-26T18:54:29.383 INFO:tasks.workunit.client.1.smithi190.stdout:8/394: rename d9/dd/c1d to d9/dd/d23/d36/d37/c8b 0 2024-03-26T18:54:29.383 INFO:tasks.workunit.client.1.smithi190.stdout:0/443: dread d0/dd/d20/d1c/f6e [0,4194304] 0 2024-03-26T18:54:29.383 INFO:tasks.workunit.client.1.smithi190.stdout:8/395: write d9/d16/f63 [260665,22330] 0 2024-03-26T18:54:29.384 INFO:tasks.workunit.client.1.smithi190.stdout:5/269: mkdir d2/d17/d1b/d60 0 2024-03-26T18:54:29.386 INFO:tasks.workunit.client.1.smithi190.stdout:9/605: rmdir d12/d2d/d1d/d23/d83/dc9 39 2024-03-26T18:54:29.386 INFO:tasks.workunit.client.1.smithi190.stdout:9/606: write d12/d2d/d1d/d23/d2b/f68 [289338,91847] 0 2024-03-26T18:54:29.386 INFO:tasks.workunit.client.1.smithi190.stdout:9/607: chown d12/d2d/d1d/d23/d6a/f7e 163537 1 2024-03-26T18:54:29.394 INFO:tasks.workunit.client.1.smithi190.stdout:6/340: mknod d17/d28/d2c/d77/c7a 0 2024-03-26T18:54:29.394 INFO:tasks.workunit.client.1.smithi190.stdout:6/341: chown d17/d28/d2b/l31 2006 1 2024-03-26T18:54:29.394 INFO:tasks.workunit.client.1.smithi190.stdout:5/270: dread d2/d17/d1b/d30/f3f [0,4194304] 0 2024-03-26T18:54:29.397 INFO:tasks.workunit.client.1.smithi190.stdout:8/396: symlink d9/dd/d15/d32/l8c 0 2024-03-26T18:54:29.397 INFO:tasks.workunit.client.1.smithi190.stdout:8/397: write d9/dd/d23/d36/d37/f5f [4461611,103970] 0 2024-03-26T18:54:29.397 INFO:tasks.workunit.client.1.smithi190.stdout:8/398: write d9/dd/f2d [4444539,9963] 0 2024-03-26T18:54:29.397 INFO:tasks.workunit.client.1.smithi190.stdout:8/399: readlink d9/l26 0 2024-03-26T18:54:29.398 INFO:tasks.workunit.client.1.smithi190.stdout:8/400: creat d9/dd/d23/d36/d37/f8d x:0 0 0 2024-03-26T18:54:29.398 INFO:tasks.workunit.client.1.smithi190.stdout:0/444: symlink d0/dd/d20/d1c/d46/l9a 0 2024-03-26T18:54:29.399 INFO:tasks.workunit.client.1.smithi190.stdout:0/445: write d0/dd/d20/d1c/d4e/f74 [210081,94857] 0 2024-03-26T18:54:29.399 INFO:tasks.workunit.client.1.smithi190.stdout:2/380: rmdir d6/d9/d10/d5f 39 2024-03-26T18:54:29.400 INFO:tasks.workunit.client.1.smithi190.stdout:2/381: write d6/d9/d3a/d3d/d1c/f5e [5064069,101645] 0 2024-03-26T18:54:29.401 INFO:tasks.workunit.client.1.smithi190.stdout:6/342: mkdir d17/d28/d74/d7b 0 2024-03-26T18:54:29.402 INFO:tasks.workunit.client.1.smithi190.stdout:5/271: getdents d2/d17/d1b/d1e/d3a/d3c 0 2024-03-26T18:54:29.403 INFO:tasks.workunit.client.1.smithi190.stdout:5/272: write d2/f14 [897524,39944] 0 2024-03-26T18:54:29.406 INFO:tasks.workunit.client.1.smithi190.stdout:9/608: dwrite d12/d2d/d1d/f8d [0,4194304] 0 2024-03-26T18:54:29.407 INFO:tasks.workunit.client.1.smithi190.stdout:8/401: dread d9/f33 [0,4194304] 0 2024-03-26T18:54:29.408 INFO:tasks.workunit.client.1.smithi190.stdout:8/402: fdatasync d9/f1a 0 2024-03-26T18:54:29.409 INFO:tasks.workunit.client.1.smithi190.stdout:0/446: rename d0/dd/d20/f68 to d0/dd/d24/d4b/d5e/d81/f9b 0 2024-03-26T18:54:29.412 INFO:tasks.workunit.client.1.smithi190.stdout:2/382: mkdir d6/d9/d3a/d3d/d2e/d86 0 2024-03-26T18:54:29.413 INFO:tasks.workunit.client.1.smithi190.stdout:0/447: dread d0/dd/d20/d1c/f29 [0,4194304] 0 2024-03-26T18:54:29.415 INFO:tasks.workunit.client.1.smithi190.stdout:5/273: unlink d2/d17/d1b/d22/d24/l2a 0 2024-03-26T18:54:29.418 INFO:tasks.workunit.client.1.smithi190.stdout:9/609: mknod d12/d2d/d1d/d54/d92/cd6 0 2024-03-26T18:54:29.418 INFO:tasks.workunit.client.1.smithi190.stdout:9/610: write d12/d2d/d1d/d23/d2b/fab [162084,32474] 0 2024-03-26T18:54:29.419 INFO:tasks.workunit.client.1.smithi190.stdout:8/403: getdents d9/d48/d49 0 2024-03-26T18:54:29.421 INFO:tasks.workunit.client.1.smithi190.stdout:0/448: creat d0/dd/d20/d61/f9c x:0 0 0 2024-03-26T18:54:29.421 INFO:tasks.workunit.client.1.smithi190.stdout:0/449: chown d0/dd/d24/f7d 17356 1 2024-03-26T18:54:29.424 INFO:tasks.workunit.client.1.smithi190.stdout:8/404: mknod d9/dd/d23/d88/c8e 0 2024-03-26T18:54:29.433 INFO:tasks.workunit.client.1.smithi190.stdout:8/405: getdents d9/d16 0 2024-03-26T18:54:29.439 INFO:tasks.workunit.client.1.smithi190.stdout:8/406: mkdir d9/dd/d23/d36/d37/d3a/d43/d8f 0 2024-03-26T18:54:29.439 INFO:tasks.workunit.client.1.smithi190.stdout:8/407: dread - d9/dd/f78 zero size 2024-03-26T18:54:29.442 INFO:tasks.workunit.client.1.smithi190.stdout:8/408: creat d9/d39/d79/f90 x:0 0 0 2024-03-26T18:54:29.442 INFO:tasks.workunit.client.1.smithi190.stdout:8/409: creat d9/dd/d23/d36/d37/d3a/d61/f91 x:0 0 0 2024-03-26T18:54:29.442 INFO:tasks.workunit.client.1.smithi190.stdout:8/410: chown d9/dd/d23 1820382 1 2024-03-26T18:54:29.455 INFO:tasks.workunit.client.0.smithi102.stdout:5/270: unlink d2/d25/d2a/d5f/f23 0 2024-03-26T18:54:29.460 INFO:tasks.workunit.client.0.smithi102.stdout:5/271: mknod d2/c65 0 2024-03-26T18:54:29.461 INFO:tasks.workunit.client.1.smithi190.stdout:8/411: write d9/d48/f62 [618819,96248] 0 2024-03-26T18:54:29.462 INFO:tasks.workunit.client.0.smithi102.stdout:5/272: creat d2/d25/d2a/d5f/f66 x:0 0 0 2024-03-26T18:54:29.463 INFO:tasks.workunit.client.1.smithi190.stdout:9/611: dwrite d12/d2d/f46 [0,4194304] 0 2024-03-26T18:54:29.463 INFO:tasks.workunit.client.1.smithi190.stdout:5/274: dwrite d2/f15 [0,4194304] 0 2024-03-26T18:54:29.464 INFO:tasks.workunit.client.1.smithi190.stdout:2/383: dwrite d6/f36 [0,4194304] 0 2024-03-26T18:54:29.464 INFO:tasks.workunit.client.1.smithi190.stdout:2/384: creat d6/d9/d10/f87 x:0 0 0 2024-03-26T18:54:29.465 INFO:tasks.workunit.client.0.smithi102.stdout:5/273: mknod d2/d25/d2a/d5d/c67 0 2024-03-26T18:54:29.466 INFO:tasks.workunit.client.1.smithi190.stdout:0/450: dwrite d0/dd/d24/f7a [4194304,4194304] 0 2024-03-26T18:54:29.467 INFO:tasks.workunit.client.0.smithi102.stdout:5/274: rmdir d2/d25 39 2024-03-26T18:54:29.468 INFO:tasks.workunit.client.0.smithi102.stdout:5/275: dread d2/d14/d1a/f41 [0,4194304] 0 2024-03-26T18:54:29.470 INFO:tasks.workunit.client.1.smithi190.stdout:5/275: creat d2/d17/d1b/d22/d24/f61 x:0 0 0 2024-03-26T18:54:29.470 INFO:tasks.workunit.client.1.smithi190.stdout:0/451: dread d0/dd/d20/d61/f7b [4194304,4194304] 0 2024-03-26T18:54:29.470 INFO:tasks.workunit.client.1.smithi190.stdout:0/452: dread - d0/dd/d20/d1c/d4e/f4f zero size 2024-03-26T18:54:29.470 INFO:tasks.workunit.client.1.smithi190.stdout:0/453: creat d0/dd/d20/d1c/d4e/f9d x:0 0 0 2024-03-26T18:54:29.470 INFO:tasks.workunit.client.1.smithi190.stdout:0/454: write d0/dd/f22 [6346784,64074] 0 2024-03-26T18:54:29.470 INFO:tasks.workunit.client.1.smithi190.stdout:0/455: chown d0/dd/d20 14400060 1 2024-03-26T18:54:29.470 INFO:tasks.workunit.client.1.smithi190.stdout:0/456: fsync d0/dd/d24/f49 0 2024-03-26T18:54:29.472 INFO:tasks.workunit.client.0.smithi102.stdout:5/276: link d2/d25/d2a/d5f/f42 d2/d25/d2a/d5d/f68 0 2024-03-26T18:54:29.472 INFO:tasks.workunit.client.0.smithi102.stdout:5/277: readlink d2/d2c/d38/l40 0 2024-03-26T18:54:29.472 INFO:tasks.workunit.client.0.smithi102.stdout:5/278: dread - d2/d25/d2a/d5f/f22 zero size 2024-03-26T18:54:29.473 INFO:tasks.workunit.client.1.smithi190.stdout:6/343: dwrite d17/f25 [0,4194304] 0 2024-03-26T18:54:29.476 INFO:tasks.workunit.client.0.smithi102.stdout:5/279: dread d2/f10 [0,4194304] 0 2024-03-26T18:54:29.479 INFO:tasks.workunit.client.1.smithi190.stdout:2/385: rename d6/d9/d3a/d3d/l11 to d6/d9/d3a/d3d/d2e/d31/l88 0 2024-03-26T18:54:29.486 INFO:tasks.workunit.client.1.smithi190.stdout:5/276: mkdir d2/d17/d62 0 2024-03-26T18:54:29.490 INFO:tasks.workunit.client.1.smithi190.stdout:6/344: creat d17/d28/d2c/d77/d51/f7c x:0 0 0 2024-03-26T18:54:29.491 INFO:tasks.workunit.client.1.smithi190.stdout:2/386: mknod d6/d21/d61/d6c/c89 0 2024-03-26T18:54:29.491 INFO:tasks.workunit.client.1.smithi190.stdout:5/277: mkdir d2/d17/d1b/d60/d63 0 2024-03-26T18:54:29.493 INFO:tasks.workunit.client.1.smithi190.stdout:2/387: rename d6/d9/d3a/f4f to d6/d9/d3a/d3d/d2e/d86/f8a 0 2024-03-26T18:54:29.494 INFO:tasks.workunit.client.1.smithi190.stdout:0/457: dwrite d0/dd/d24/f7d [0,4194304] 0 2024-03-26T18:54:29.494 INFO:tasks.workunit.client.0.smithi102.stdout:5/280: dwrite d2/d14/f2d [0,4194304] 0 2024-03-26T18:54:29.499 INFO:tasks.workunit.client.1.smithi190.stdout:0/458: unlink d0/dd/d20/d1c/d46/d7c/f85 0 2024-03-26T18:54:29.502 INFO:tasks.workunit.client.0.smithi102.stdout:5/281: truncate d2/f12 2825990 0 2024-03-26T18:54:29.502 INFO:tasks.workunit.client.1.smithi190.stdout:2/388: dread d6/d9/d10/f2c [0,4194304] 0 2024-03-26T18:54:29.502 INFO:tasks.workunit.client.1.smithi190.stdout:2/389: dread - d6/d9/d48/f63 zero size 2024-03-26T18:54:29.503 INFO:tasks.workunit.client.0.smithi102.stdout:5/282: truncate d2/d14/d1a/f1b 336660 0 2024-03-26T18:54:29.503 INFO:tasks.workunit.client.0.smithi102.stdout:5/283: chown d2/l13 45 1 2024-03-26T18:54:29.505 INFO:tasks.workunit.client.1.smithi190.stdout:6/345: dwrite d17/d28/f2d [4194304,4194304] 0 2024-03-26T18:54:29.505 INFO:tasks.workunit.client.1.smithi190.stdout:2/390: mkdir d6/d9/d1a/d72/d8b 0 2024-03-26T18:54:29.507 INFO:tasks.workunit.client.0.smithi102.stdout:5/284: link d2/d25/d2a/d5d/c67 d2/d2c/d38/d43/c69 0 2024-03-26T18:54:29.509 INFO:tasks.workunit.client.1.smithi190.stdout:6/346: getdents d17/d28/d2c/d63 0 2024-03-26T18:54:29.509 INFO:tasks.workunit.client.1.smithi190.stdout:6/347: fsync d17/f25 0 2024-03-26T18:54:29.510 INFO:tasks.workunit.client.0.smithi102.stdout:5/285: dread d2/f10 [0,4194304] 0 2024-03-26T18:54:29.512 INFO:tasks.workunit.client.0.smithi102.stdout:5/286: link d2/d14/d1a/f50 d2/f6a 0 2024-03-26T18:54:29.512 INFO:tasks.workunit.client.0.smithi102.stdout:5/287: write d2/d25/d2a/d5f/f22 [523564,84711] 0 2024-03-26T18:54:29.514 INFO:tasks.workunit.client.1.smithi190.stdout:6/348: rename d17/c1a to d17/d28/d46/d6a/c7d 0 2024-03-26T18:54:29.514 INFO:tasks.workunit.client.1.smithi190.stdout:6/349: chown cd 0 1 2024-03-26T18:54:29.517 INFO:tasks.workunit.client.1.smithi190.stdout:6/350: creat d17/d28/d2c/d50/f7e x:0 0 0 2024-03-26T18:54:29.517 INFO:tasks.workunit.client.1.smithi190.stdout:6/351: creat d17/d28/d46/f7f x:0 0 0 2024-03-26T18:54:29.524 INFO:tasks.workunit.client.1.smithi190.stdout:1/298: sync 2024-03-26T18:54:29.530 INFO:tasks.workunit.client.1.smithi190.stdout:6/352: dread d17/d28/f2d [0,4194304] 0 2024-03-26T18:54:29.532 INFO:tasks.workunit.client.1.smithi190.stdout:0/459: dwrite d0/dd/d24/f49 [0,4194304] 0 2024-03-26T18:54:29.532 INFO:tasks.workunit.client.1.smithi190.stdout:2/391: dwrite d6/f7a [0,4194304] 0 2024-03-26T18:54:29.533 INFO:tasks.workunit.client.0.smithi102.stdout:5/288: dwrite d2/f3a [0,4194304] 0 2024-03-26T18:54:29.533 INFO:tasks.workunit.client.0.smithi102.stdout:5/289: write d2/d25/d2a/f2e [2574498,29049] 0 2024-03-26T18:54:29.533 INFO:tasks.workunit.client.0.smithi102.stdout:5/290: readlink d2/l3 0 2024-03-26T18:54:29.536 INFO:tasks.workunit.client.1.smithi190.stdout:0/460: truncate d0/dd/d24/f7a 6011831 0 2024-03-26T18:54:29.537 INFO:tasks.workunit.client.0.smithi102.stdout:5/291: symlink d2/d2c/d38/l6b 0 2024-03-26T18:54:29.538 INFO:tasks.workunit.client.1.smithi190.stdout:0/461: stat d0/dd/d24/c2d 0 2024-03-26T18:54:29.538 INFO:tasks.workunit.client.1.smithi190.stdout:0/462: dread - d0/dd/d20/d1c/f8c zero size 2024-03-26T18:54:29.539 INFO:tasks.workunit.client.1.smithi190.stdout:0/463: creat d0/dd/d20/d61/d7f/f9e x:0 0 0 2024-03-26T18:54:29.539 INFO:tasks.workunit.client.1.smithi190.stdout:0/464: write d0/dd/d20/d61/f9c [741911,72069] 0 2024-03-26T18:54:29.542 INFO:tasks.workunit.client.1.smithi190.stdout:6/353: dread f2 [0,4194304] 0 2024-03-26T18:54:29.543 INFO:tasks.workunit.client.1.smithi190.stdout:6/354: mkdir d17/d28/d80 0 2024-03-26T18:54:29.545 INFO:tasks.workunit.client.1.smithi190.stdout:1/299: dwrite d3/d9/f2a [0,4194304] 0 2024-03-26T18:54:29.546 INFO:tasks.workunit.client.1.smithi190.stdout:2/392: dread d6/d9/d3a/d3d/d1c/f5e [0,4194304] 0 2024-03-26T18:54:29.546 INFO:tasks.workunit.client.1.smithi190.stdout:2/393: stat d6/d9/d10/f59 0 2024-03-26T18:54:29.546 INFO:tasks.workunit.client.1.smithi190.stdout:2/394: creat d6/d9/d10/f8c x:0 0 0 2024-03-26T18:54:29.546 INFO:tasks.workunit.client.1.smithi190.stdout:2/395: chown d6/d9/d1a 0 1 2024-03-26T18:54:29.546 INFO:tasks.workunit.client.1.smithi190.stdout:2/396: dread - d6/d9/d3a/d3d/d1c/d44/f49 zero size 2024-03-26T18:54:29.546 INFO:tasks.workunit.client.1.smithi190.stdout:2/397: dread - d6/d21/f78 zero size 2024-03-26T18:54:29.548 INFO:tasks.workunit.client.0.smithi102.stdout:5/292: write d2/d14/d1a/f50 [2355006,18310] 0 2024-03-26T18:54:29.554 INFO:tasks.workunit.client.1.smithi190.stdout:6/355: write d17/f25 [1013225,19369] 0 2024-03-26T18:54:29.554 INFO:tasks.workunit.client.1.smithi190.stdout:6/356: chown f8 9 1 2024-03-26T18:54:29.554 INFO:tasks.workunit.client.1.smithi190.stdout:6/357: read - d17/d28/d2c/f53 zero size 2024-03-26T18:54:29.559 INFO:tasks.workunit.client.1.smithi190.stdout:2/398: dread d6/d9/d3a/d3d/d1c/f4a [0,4194304] 0 2024-03-26T18:54:29.560 INFO:tasks.workunit.client.1.smithi190.stdout:2/399: truncate d6/d9/d10/f87 1003912 0 2024-03-26T18:54:29.560 INFO:tasks.workunit.client.1.smithi190.stdout:0/465: dwrite d0/dd/d24/d4b/f89 [0,4194304] 0 2024-03-26T18:54:29.560 INFO:tasks.workunit.client.1.smithi190.stdout:0/466: truncate d0/dd/d20/d1c/d4e/f7e 556262 0 2024-03-26T18:54:29.560 INFO:tasks.workunit.client.1.smithi190.stdout:0/467: creat d0/dd/d20/f9f x:0 0 0 2024-03-26T18:54:29.564 INFO:tasks.workunit.client.1.smithi190.stdout:2/400: mkdir d6/d9/d3a/d3d/d1c/d8d 0 2024-03-26T18:54:29.567 INFO:tasks.workunit.client.1.smithi190.stdout:2/401: write d6/d9/d3a/d3d/d1c/f4a [2212171,40320] 0 2024-03-26T18:54:29.571 INFO:tasks.workunit.client.1.smithi190.stdout:4/376: sync 2024-03-26T18:54:29.571 INFO:tasks.workunit.client.1.smithi190.stdout:4/377: chown d7/d1d/f26 6 1 2024-03-26T18:54:29.571 INFO:tasks.workunit.client.1.smithi190.stdout:4/378: truncate d7/d8/d14/d23/d28/d38/f62 4988995 0 2024-03-26T18:54:29.572 INFO:tasks.workunit.client.1.smithi190.stdout:0/468: getdents d0/dd/d20 0 2024-03-26T18:54:29.572 INFO:tasks.workunit.client.1.smithi190.stdout:0/469: chown d0/dd/d20/d1c/d4e/f4f 358 1 2024-03-26T18:54:29.572 INFO:tasks.workunit.client.1.smithi190.stdout:0/470: write d0/f5b [961732,35521] 0 2024-03-26T18:54:29.572 INFO:tasks.workunit.client.1.smithi190.stdout:0/471: write d0/dd/d20/d1c/d4e/f4f [666416,25694] 0 2024-03-26T18:54:29.573 INFO:tasks.workunit.client.1.smithi190.stdout:0/472: mknod d0/dd/d20/d1c/d46/d7c/ca0 0 2024-03-26T18:54:29.573 INFO:tasks.workunit.client.1.smithi190.stdout:0/473: write d0/dd/d24/d4b/d5e/f5f [459506,68109] 0 2024-03-26T18:54:29.573 INFO:tasks.workunit.client.1.smithi190.stdout:0/474: chown d0/dd/d20/d61/d7f/f9e 22520726 1 2024-03-26T18:54:29.573 INFO:tasks.workunit.client.1.smithi190.stdout:0/475: chown d0/dd/d24/f7a 838318007 1 2024-03-26T18:54:29.574 INFO:tasks.workunit.client.1.smithi190.stdout:0/476: mknod d0/dd/d20/d1c/d8b/d8d/d8f/ca1 0 2024-03-26T18:54:29.575 INFO:tasks.workunit.client.1.smithi190.stdout:0/477: truncate d0/f1f 1654082 0 2024-03-26T18:54:29.576 INFO:tasks.workunit.client.0.smithi102.stdout:5/293: dwrite d2/f10 [0,4194304] 0 2024-03-26T18:54:29.576 INFO:tasks.workunit.client.0.smithi102.stdout:5/294: creat d2/d25/d2a/f6c x:0 0 0 2024-03-26T18:54:29.576 INFO:tasks.workunit.client.0.smithi102.stdout:5/295: dread - d2/d2c/f5a zero size 2024-03-26T18:54:29.577 INFO:tasks.workunit.client.0.smithi102.stdout:5/296: chown d2/d25/d2a/d5f/l28 63 1 2024-03-26T18:54:29.579 INFO:tasks.workunit.client.1.smithi190.stdout:0/478: rename d0/dd/d20/d61/d7f to d0/dd/d24/da2 0 2024-03-26T18:54:29.579 INFO:tasks.workunit.client.1.smithi190.stdout:0/479: chown d0/dd/d24/f7a 2512 1 2024-03-26T18:54:29.580 INFO:tasks.workunit.client.1.smithi190.stdout:0/480: symlink d0/la3 0 2024-03-26T18:54:29.581 INFO:tasks.workunit.client.0.smithi102.stdout:5/297: mknod d2/d2c/d38/d43/c6d 0 2024-03-26T18:54:29.581 INFO:tasks.workunit.client.0.smithi102.stdout:5/298: readlink d2/d2c/d38/l45 0 2024-03-26T18:54:29.581 INFO:tasks.workunit.client.0.smithi102.stdout:5/299: chown d2/d14/d1a/l21 248203061 1 2024-03-26T18:54:29.581 INFO:tasks.workunit.client.0.smithi102.stdout:5/300: chown d2/d14/d1a 113942959 1 2024-03-26T18:54:29.582 INFO:tasks.workunit.client.0.smithi102.stdout:5/301: symlink d2/d25/d2a/d4f/l6e 0 2024-03-26T18:54:29.582 INFO:tasks.workunit.client.0.smithi102.stdout:5/302: read d2/f3c [294671,125603] 0 2024-03-26T18:54:29.584 INFO:tasks.workunit.client.1.smithi190.stdout:4/379: read d7/d8/d14/d4b/f5f [571915,20932] 0 2024-03-26T18:54:29.594 INFO:tasks.workunit.client.0.smithi102.stdout:5/303: fdatasync d2/d25/d2a/d5f/f22 0 2024-03-26T18:54:29.594 INFO:tasks.workunit.client.1.smithi190.stdout:6/358: dwrite d17/d28/d2c/f4c [0,4194304] 0 2024-03-26T18:54:29.595 INFO:tasks.workunit.client.1.smithi190.stdout:6/359: chown d17/d28/d2c/d63/c73 22760 1 2024-03-26T18:54:29.595 INFO:tasks.workunit.client.1.smithi190.stdout:2/402: dwrite d6/d9/d1a/d72/d7c/f7d [0,4194304] 0 2024-03-26T18:54:29.595 INFO:tasks.workunit.client.1.smithi190.stdout:6/360: creat d17/d28/d80/f81 x:0 0 0 2024-03-26T18:54:29.596 INFO:tasks.workunit.client.0.smithi102.stdout:5/304: mkdir d2/d2c/d6f 0 2024-03-26T18:54:29.598 INFO:tasks.workunit.client.1.smithi190.stdout:4/380: write d7/f4f [3792149,7331] 0 2024-03-26T18:54:29.601 INFO:tasks.workunit.client.1.smithi190.stdout:2/403: unlink d6/d9/d3a/d3d/d1c/f4a 0 2024-03-26T18:54:29.602 INFO:tasks.workunit.client.1.smithi190.stdout:2/404: readlink d6/d9/l12 0 2024-03-26T18:54:29.603 INFO:tasks.workunit.client.1.smithi190.stdout:6/361: creat d17/d28/d38/d79/f82 x:0 0 0 2024-03-26T18:54:29.603 INFO:tasks.workunit.client.1.smithi190.stdout:6/362: read - d17/d28/d2c/f47 zero size 2024-03-26T18:54:29.606 INFO:tasks.workunit.client.1.smithi190.stdout:4/381: rename d7/d1d/f27 to d7/d8/d14/d23/d28/d38/f7f 0 2024-03-26T18:54:29.613 INFO:tasks.workunit.client.0.smithi102.stdout:5/305: dread d2/d25/f2b [0,4194304] 0 2024-03-26T18:54:29.614 INFO:tasks.workunit.client.0.smithi102.stdout:5/306: creat d2/d14/f70 x:0 0 0 2024-03-26T18:54:29.616 INFO:tasks.workunit.client.0.smithi102.stdout:5/307: getdents d2 0 2024-03-26T18:54:29.616 INFO:tasks.workunit.client.1.smithi190.stdout:6/363: write f8 [1543328,94557] 0 2024-03-26T18:54:29.620 INFO:tasks.workunit.client.0.smithi102.stdout:5/308: rmdir d2/d14/d1a 39 2024-03-26T18:54:29.620 INFO:tasks.workunit.client.1.smithi190.stdout:6/364: mknod d17/d28/d46/d6a/c83 0 2024-03-26T18:54:29.620 INFO:tasks.workunit.client.1.smithi190.stdout:6/365: dread - d17/d28/d80/f81 zero size 2024-03-26T18:54:29.620 INFO:tasks.workunit.client.1.smithi190.stdout:6/366: fsync d17/d28/d38/d79/f82 0 2024-03-26T18:54:29.621 INFO:tasks.workunit.client.1.smithi190.stdout:0/481: dwrite d0/dd/d24/d4b/d5e/f5f [0,4194304] 0 2024-03-26T18:54:29.621 INFO:tasks.workunit.client.1.smithi190.stdout:0/482: chown d0/dd/fe 40661473 1 2024-03-26T18:54:29.621 INFO:tasks.workunit.client.0.smithi102.stdout:3/186: sync 2024-03-26T18:54:29.621 INFO:tasks.workunit.client.0.smithi102.stdout:8/37: sync 2024-03-26T18:54:29.621 INFO:tasks.workunit.client.0.smithi102.stdout:6/82: sync 2024-03-26T18:54:29.621 INFO:tasks.workunit.client.0.smithi102.stdout:0/72: sync 2024-03-26T18:54:29.621 INFO:tasks.workunit.client.0.smithi102.stdout:4/27: sync 2024-03-26T18:54:29.622 INFO:tasks.workunit.client.0.smithi102.stdout:2/54: sync 2024-03-26T18:54:29.622 INFO:tasks.workunit.client.0.smithi102.stdout:2/55: truncate d3/d6/fe 238961 0 2024-03-26T18:54:29.622 INFO:tasks.workunit.client.0.smithi102.stdout:7/2: sync 2024-03-26T18:54:29.622 INFO:tasks.workunit.client.0.smithi102.stdout:2/56: creat d3/d6/ff x:0 0 0 2024-03-26T18:54:29.622 INFO:tasks.workunit.client.0.smithi102.stdout:2/57: fdatasync d3/f4 0 2024-03-26T18:54:29.622 INFO:tasks.workunit.client.0.smithi102.stdout:1/9: sync 2024-03-26T18:54:29.623 INFO:tasks.workunit.client.0.smithi102.stdout:9/26: sync 2024-03-26T18:54:29.623 INFO:tasks.workunit.client.0.smithi102.stdout:3/187: dread df/d29/d31/d36/f41 [0,4194304] 0 2024-03-26T18:54:29.623 INFO:tasks.workunit.client.0.smithi102.stdout:3/188: write df/f39 [96023,100788] 0 2024-03-26T18:54:29.623 INFO:tasks.workunit.client.0.smithi102.stdout:5/309: fsync d2/d14/d1a/f50 0 2024-03-26T18:54:29.624 INFO:tasks.workunit.client.0.smithi102.stdout:0/73: link d6/fd d6/f1a 0 2024-03-26T18:54:29.625 INFO:tasks.workunit.client.0.smithi102.stdout:6/83: symlink d2/d7/l14 0 2024-03-26T18:54:29.631 INFO:tasks.workunit.client.1.smithi190.stdout:6/367: getdents d17/d28/d80 0 2024-03-26T18:54:29.635 INFO:tasks.workunit.client.1.smithi190.stdout:6/368: mknod d17/d28/d2c/d77/c84 0 2024-03-26T18:54:29.635 INFO:tasks.workunit.client.1.smithi190.stdout:6/369: write d17/d28/d2c/d50/f67 [823544,107392] 0 2024-03-26T18:54:29.635 INFO:tasks.workunit.client.1.smithi190.stdout:6/370: chown d17/l1f 15813573 1 2024-03-26T18:54:29.635 INFO:tasks.workunit.client.1.smithi190.stdout:6/371: chown d17/d28/d2c/d50/f7e 803750797 1 2024-03-26T18:54:29.635 INFO:tasks.workunit.client.0.smithi102.stdout:8/38: write d0/d3/f6 [92209,1053] 0 2024-03-26T18:54:29.636 INFO:tasks.workunit.client.0.smithi102.stdout:5/310: symlink d2/d14/d1a/l71 0 2024-03-26T18:54:29.636 INFO:tasks.workunit.client.0.smithi102.stdout:2/58: link d3/d6/ff d3/d6/f10 0 2024-03-26T18:54:29.637 INFO:tasks.workunit.client.0.smithi102.stdout:9/27: dread f4 [0,4194304] 0 2024-03-26T18:54:29.637 INFO:tasks.workunit.client.0.smithi102.stdout:9/28: rmdir - no directory 2024-03-26T18:54:29.638 INFO:tasks.workunit.client.0.smithi102.stdout:9/29: truncate f4 4876496 0 2024-03-26T18:54:29.638 INFO:tasks.workunit.client.1.smithi190.stdout:6/372: symlink d17/d28/d80/l85 0 2024-03-26T18:54:29.638 INFO:tasks.workunit.client.1.smithi190.stdout:6/373: fdatasync d17/d28/f58 0 2024-03-26T18:54:29.638 INFO:tasks.workunit.client.0.smithi102.stdout:3/189: rmdir df/d29/d31/d36 39 2024-03-26T18:54:29.641 INFO:tasks.workunit.client.1.smithi190.stdout:0/483: dread d0/dd/d24/f49 [0,4194304] 0 2024-03-26T18:54:29.641 INFO:tasks.workunit.client.1.smithi190.stdout:6/374: mknod d17/d28/d46/d6a/c86 0 2024-03-26T18:54:29.641 INFO:tasks.workunit.client.1.smithi190.stdout:6/375: dread - d17/d28/d2c/d63/f70 zero size 2024-03-26T18:54:29.641 INFO:tasks.workunit.client.1.smithi190.stdout:6/376: truncate d17/d28/d46/f5c 835304 0 2024-03-26T18:54:29.641 INFO:tasks.workunit.client.1.smithi190.stdout:6/377: read - d17/d28/d38/f45 zero size 2024-03-26T18:54:29.641 INFO:tasks.workunit.client.1.smithi190.stdout:6/378: chown d17/l1f 622277930 1 2024-03-26T18:54:29.642 INFO:tasks.workunit.client.0.smithi102.stdout:8/39: mkdir d0/d9 0 2024-03-26T18:54:29.642 INFO:tasks.workunit.client.0.smithi102.stdout:8/40: readlink d0/d3/l8 0 2024-03-26T18:54:29.642 INFO:tasks.workunit.client.0.smithi102.stdout:9/30: write f4 [899290,107274] 0 2024-03-26T18:54:29.642 INFO:tasks.workunit.client.1.smithi190.stdout:0/484: read d0/dd/d20/d61/f9c [362787,55078] 0 2024-03-26T18:54:29.643 INFO:tasks.workunit.client.1.smithi190.stdout:4/382: dwrite d7/d8/d1c/f7a [0,4194304] 0 2024-03-26T18:54:29.643 INFO:tasks.workunit.client.0.smithi102.stdout:2/59: mkdir d3/d11 0 2024-03-26T18:54:29.643 INFO:tasks.workunit.client.0.smithi102.stdout:2/60: truncate d3/fb 1899107 0 2024-03-26T18:54:29.646 INFO:tasks.workunit.client.0.smithi102.stdout:3/190: rmdir df/d29/d31/d36/d45 39 2024-03-26T18:54:29.650 INFO:tasks.workunit.client.0.smithi102.stdout:9/31: unlink f4 0 2024-03-26T18:54:29.650 INFO:tasks.workunit.client.0.smithi102.stdout:9/32: truncate - no filename 2024-03-26T18:54:29.652 INFO:tasks.workunit.client.0.smithi102.stdout:0/74: dread d6/fd [0,4194304] 0 2024-03-26T18:54:29.652 INFO:tasks.workunit.client.0.smithi102.stdout:0/75: stat d6/de/l12 0 2024-03-26T18:54:29.652 INFO:tasks.workunit.client.0.smithi102.stdout:2/61: link d3/d6/ff d3/f12 0 2024-03-26T18:54:29.654 INFO:tasks.workunit.client.1.smithi190.stdout:6/379: creat d17/f87 x:0 0 0 2024-03-26T18:54:29.654 INFO:tasks.workunit.client.1.smithi190.stdout:6/380: stat d17/d28/d2c/d77/c84 0 2024-03-26T18:54:29.654 INFO:tasks.workunit.client.1.smithi190.stdout:6/381: chown d17/l62 16346 1 2024-03-26T18:54:29.655 INFO:tasks.workunit.client.0.smithi102.stdout:3/191: creat df/d29/d31/d36/f48 x:0 0 0 2024-03-26T18:54:29.655 INFO:tasks.workunit.client.0.smithi102.stdout:3/192: chown df/f25 1 1 2024-03-26T18:54:29.656 INFO:tasks.workunit.client.1.smithi190.stdout:0/485: link d0/dd/d24/d4b/d5e/d81/f9b d0/dd/d20/fa4 0 2024-03-26T18:54:29.657 INFO:tasks.workunit.client.0.smithi102.stdout:9/33: link c3 c6 0 2024-03-26T18:54:29.657 INFO:tasks.workunit.client.0.smithi102.stdout:0/76: write d6/f11 [1866061,115043] 0 2024-03-26T18:54:29.657 INFO:tasks.workunit.client.0.smithi102.stdout:9/34: creat f7 x:0 0 0 2024-03-26T18:54:29.657 INFO:tasks.workunit.client.0.smithi102.stdout:0/77: chown d6/de/d13/c17 4163 1 2024-03-26T18:54:29.660 INFO:tasks.workunit.client.0.smithi102.stdout:3/193: rename df/d1c/f3e to df/d29/d31/d36/d3c/f49 0 2024-03-26T18:54:29.660 INFO:tasks.workunit.client.0.smithi102.stdout:3/194: dread - df/d29/d31/d36/f48 zero size 2024-03-26T18:54:29.663 INFO:tasks.workunit.client.0.smithi102.stdout:2/62: write d3/f7 [3516919,116592] 0 2024-03-26T18:54:29.663 INFO:tasks.workunit.client.0.smithi102.stdout:2/63: creat d3/f13 x:0 0 0 2024-03-26T18:54:29.663 INFO:tasks.workunit.client.0.smithi102.stdout:9/35: rename c6 to c8 0 2024-03-26T18:54:29.663 INFO:tasks.workunit.client.0.smithi102.stdout:9/36: write f7 [274765,12937] 0 2024-03-26T18:54:29.666 INFO:tasks.workunit.client.0.smithi102.stdout:3/195: rename l9 to df/d1c/l4a 0 2024-03-26T18:54:29.666 INFO:tasks.workunit.client.0.smithi102.stdout:3/196: read - df/f2e zero size 2024-03-26T18:54:29.668 INFO:tasks.workunit.client.0.smithi102.stdout:8/41: fsync d0/d3/f6 0 2024-03-26T18:54:29.668 INFO:tasks.workunit.client.0.smithi102.stdout:1/10: dwrite f1 [0,4194304] 0 2024-03-26T18:54:29.668 INFO:tasks.workunit.client.0.smithi102.stdout:1/11: fsync f1 0 2024-03-26T18:54:29.668 INFO:tasks.workunit.client.0.smithi102.stdout:1/12: creat f3 x:0 0 0 2024-03-26T18:54:29.668 INFO:tasks.workunit.client.0.smithi102.stdout:4/28: dwrite d1/f4 [0,4194304] 0 2024-03-26T18:54:29.668 INFO:tasks.workunit.client.0.smithi102.stdout:1/13: write f3 [966976,22564] 0 2024-03-26T18:54:29.668 INFO:tasks.workunit.client.0.smithi102.stdout:7/3: dwrite f0 [0,4194304] 0 2024-03-26T18:54:29.669 INFO:tasks.workunit.client.0.smithi102.stdout:5/311: dwrite d2/d25/d2a/d5f/f22 [0,4194304] 0 2024-03-26T18:54:29.669 INFO:tasks.workunit.client.1.smithi190.stdout:4/383: rmdir d7/d8/d1c/d20/d40 39 2024-03-26T18:54:29.669 INFO:tasks.workunit.client.1.smithi190.stdout:4/384: creat d7/d8/d1c/f80 x:0 0 0 2024-03-26T18:54:29.669 INFO:tasks.workunit.client.1.smithi190.stdout:4/385: creat d7/d1d/d5e/f81 x:0 0 0 2024-03-26T18:54:29.673 INFO:tasks.workunit.client.1.smithi190.stdout:6/382: link d17/f25 d17/d28/d80/f88 0 2024-03-26T18:54:29.678 INFO:tasks.workunit.client.1.smithi190.stdout:0/486: creat d0/dd/d24/da2/fa5 x:0 0 0 2024-03-26T18:54:29.680 INFO:tasks.workunit.client.0.smithi102.stdout:4/29: dread d1/f4 [0,4194304] 0 2024-03-26T18:54:29.684 INFO:tasks.workunit.client.1.smithi190.stdout:4/386: dread d7/d8/d1c/d20/f58 [0,4194304] 0 2024-03-26T18:54:29.684 INFO:tasks.workunit.client.0.smithi102.stdout:7/4: rename f0 to f1 0 2024-03-26T18:54:29.686 INFO:tasks.workunit.client.1.smithi190.stdout:0/487: link d0/dd/d24/c48 d0/dd/d24/d4b/d5e/d66/ca6 0 2024-03-26T18:54:29.686 INFO:tasks.workunit.client.0.smithi102.stdout:0/78: dwrite d6/de/f16 [0,4194304] 0 2024-03-26T18:54:29.691 INFO:tasks.workunit.client.1.smithi190.stdout:4/387: mknod d7/d1d/c82 0 2024-03-26T18:54:29.691 INFO:tasks.workunit.client.1.smithi190.stdout:4/388: dread - d7/d8/d1c/d20/f78 zero size 2024-03-26T18:54:29.691 INFO:tasks.workunit.client.1.smithi190.stdout:4/389: chown d7/d8/d14/d23/f55 115554 1 2024-03-26T18:54:29.692 INFO:tasks.workunit.client.0.smithi102.stdout:2/64: dread d3/f7 [0,4194304] 0 2024-03-26T18:54:29.692 INFO:tasks.workunit.client.0.smithi102.stdout:2/65: write d3/d6/ff [979148,37555] 0 2024-03-26T18:54:29.692 INFO:tasks.workunit.client.0.smithi102.stdout:2/66: chown d3/d6/ff 28 1 2024-03-26T18:54:29.694 INFO:tasks.workunit.client.1.smithi190.stdout:0/488: mkdir d0/dd/d20/d1c/d46/da7 0 2024-03-26T18:54:29.694 INFO:tasks.workunit.client.1.smithi190.stdout:0/489: dread - d0/dd/d24/d4b/d5e/f95 zero size 2024-03-26T18:54:29.695 INFO:tasks.workunit.client.0.smithi102.stdout:3/197: getdents df 0 2024-03-26T18:54:29.696 INFO:tasks.workunit.client.0.smithi102.stdout:3/198: write fd [1272559,17421] 0 2024-03-26T18:54:29.697 INFO:tasks.workunit.client.1.smithi190.stdout:4/390: rename d7/d8/d1c/f44 to d7/d8/d14/d5c/f83 0 2024-03-26T18:54:29.697 INFO:tasks.workunit.client.1.smithi190.stdout:4/391: chown d7/d1d/d41/d74 1 1 2024-03-26T18:54:29.698 INFO:tasks.workunit.client.1.smithi190.stdout:4/392: write d7/d1d/d42/d4a/f66 [966159,130697] 0 2024-03-26T18:54:29.698 INFO:tasks.workunit.client.0.smithi102.stdout:4/30: creat d1/f7 x:0 0 0 2024-03-26T18:54:29.700 INFO:tasks.workunit.client.1.smithi190.stdout:0/490: mknod d0/dd/d24/d4b/d5e/d66/ca8 0 2024-03-26T18:54:29.701 INFO:tasks.workunit.client.1.smithi190.stdout:0/491: dread d0/dd/d20/d1c/d4e/f7e [0,4194304] 0 2024-03-26T18:54:29.701 INFO:tasks.workunit.client.1.smithi190.stdout:0/492: creat d0/dd/d20/d1c/d46/d7c/fa9 x:0 0 0 2024-03-26T18:54:29.701 INFO:tasks.workunit.client.1.smithi190.stdout:0/493: stat d0/dd/d24/d4b/c52 0 2024-03-26T18:54:29.701 INFO:tasks.workunit.client.1.smithi190.stdout:0/494: dread - d0/dd/d20/d1c/d46/d7c/fa9 zero size 2024-03-26T18:54:29.703 INFO:tasks.workunit.client.0.smithi102.stdout:4/31: write d1/f4 [2900819,82737] 0 2024-03-26T18:54:29.703 INFO:tasks.workunit.client.0.smithi102.stdout:4/32: creat d1/f8 x:0 0 0 2024-03-26T18:54:29.703 INFO:tasks.workunit.client.0.smithi102.stdout:4/33: stat d1 0 2024-03-26T18:54:29.703 INFO:tasks.workunit.client.0.smithi102.stdout:4/34: creat d1/f9 x:0 0 0 2024-03-26T18:54:29.703 INFO:tasks.workunit.client.0.smithi102.stdout:4/35: creat d1/fa x:0 0 0 2024-03-26T18:54:29.703 INFO:tasks.workunit.client.0.smithi102.stdout:4/36: chown d1/f8 91112 1 2024-03-26T18:54:29.704 INFO:tasks.workunit.client.0.smithi102.stdout:7/5: dread f1 [0,4194304] 0 2024-03-26T18:54:29.705 INFO:tasks.workunit.client.0.smithi102.stdout:2/67: mknod d3/d11/c14 0 2024-03-26T18:54:29.705 INFO:tasks.workunit.client.0.smithi102.stdout:2/68: write d3/d6/fa [970147,69736] 0 2024-03-26T18:54:29.705 INFO:tasks.workunit.client.1.smithi190.stdout:3/346: sync 2024-03-26T18:54:29.705 INFO:tasks.workunit.client.1.smithi190.stdout:8/412: sync 2024-03-26T18:54:29.705 INFO:tasks.workunit.client.1.smithi190.stdout:7/388: sync 2024-03-26T18:54:29.706 INFO:tasks.workunit.client.1.smithi190.stdout:8/413: chown l1 62 1 2024-03-26T18:54:29.706 INFO:tasks.workunit.client.1.smithi190.stdout:6/383: dwrite d17/d28/d2c/d77/f69 [0,4194304] 0 2024-03-26T18:54:29.707 INFO:tasks.workunit.client.1.smithi190.stdout:0/495: write d0/dd/d24/f7a [1141401,113093] 0 2024-03-26T18:54:29.708 INFO:tasks.workunit.client.1.smithi190.stdout:0/496: creat d0/dd/d20/d1c/d46/d7c/faa x:0 0 0 2024-03-26T18:54:29.708 INFO:tasks.workunit.client.1.smithi190.stdout:0/497: readlink d0/dd/l1b 0 2024-03-26T18:54:29.708 INFO:tasks.workunit.client.1.smithi190.stdout:4/393: rename d7/d8/d1c/d20/d40/f54 to d7/d8/d14/d6e/f84 0 2024-03-26T18:54:29.708 INFO:tasks.workunit.client.1.smithi190.stdout:0/498: rename d0/dd/d20 to d0/dd/d20/d1c/d8b/dab 22 2024-03-26T18:54:29.709 INFO:tasks.workunit.client.1.smithi190.stdout:4/394: readlink d7/l43 0 2024-03-26T18:54:29.709 INFO:tasks.workunit.client.1.smithi190.stdout:0/499: creat d0/dd/d20/d61/fac x:0 0 0 2024-03-26T18:54:29.709 INFO:tasks.workunit.client.1.smithi190.stdout:0/500: write d0/dd/d24/d4b/d5e/d81/f36 [3260225,114584] 0 2024-03-26T18:54:29.709 INFO:tasks.workunit.client.0.smithi102.stdout:1/14: dwrite f1 [0,4194304] 0 2024-03-26T18:54:29.709 INFO:tasks.workunit.client.0.smithi102.stdout:9/37: dwrite f7 [0,4194304] 0 2024-03-26T18:54:29.710 INFO:tasks.workunit.client.0.smithi102.stdout:9/38: chown c3 17295 1 2024-03-26T18:54:29.710 INFO:tasks.workunit.client.0.smithi102.stdout:9/39: chown f7 842091384 1 2024-03-26T18:54:29.710 INFO:tasks.workunit.client.0.smithi102.stdout:8/42: dwrite d0/d3/f6 [0,4194304] 0 2024-03-26T18:54:29.710 INFO:tasks.workunit.client.0.smithi102.stdout:0/79: fsync d6/fd 0 2024-03-26T18:54:29.711 INFO:tasks.workunit.client.1.smithi190.stdout:3/347: mkdir d1/d3/df/d18/d35/d65 0 2024-03-26T18:54:29.711 INFO:tasks.workunit.client.1.smithi190.stdout:3/348: readlink d1/d3/df/d18/d35/d41/d51/l55 0 2024-03-26T18:54:29.711 INFO:tasks.workunit.client.1.smithi190.stdout:3/349: write d1/d3/d5a/f63 [1434149,30659] 0 2024-03-26T18:54:29.712 INFO:tasks.workunit.client.0.smithi102.stdout:3/199: symlink df/d29/d3a/d47/l4b 0 2024-03-26T18:54:29.713 INFO:tasks.workunit.client.0.smithi102.stdout:3/200: creat df/d1c/f4c x:0 0 0 2024-03-26T18:54:29.713 INFO:tasks.workunit.client.0.smithi102.stdout:3/201: write df/f2e [99901,110280] 0 2024-03-26T18:54:29.717 INFO:tasks.workunit.client.1.smithi190.stdout:7/389: truncate d3/d1b/f25 2538325 0 2024-03-26T18:54:29.717 INFO:tasks.workunit.client.1.smithi190.stdout:9/612: sync 2024-03-26T18:54:29.717 INFO:tasks.workunit.client.1.smithi190.stdout:9/613: write d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/db3/fc1 [639556,62491] 0 2024-03-26T18:54:29.717 INFO:tasks.workunit.client.1.smithi190.stdout:9/614: chown d12/d2d/d1d/d23/d2b/da0/fc8 0 1 2024-03-26T18:54:29.719 INFO:tasks.workunit.client.1.smithi190.stdout:8/414: link d9/d16/d73/c85 d9/d39/d67/c92 0 2024-03-26T18:54:29.719 INFO:tasks.workunit.client.1.smithi190.stdout:8/415: write d9/dd/f5a [36858,127922] 0 2024-03-26T18:54:29.726 INFO:tasks.workunit.client.0.smithi102.stdout:2/69: rmdir d3/d11 39 2024-03-26T18:54:29.727 INFO:tasks.workunit.client.0.smithi102.stdout:2/70: dread - d3/f9 zero size 2024-03-26T18:54:29.733 INFO:tasks.workunit.client.1.smithi190.stdout:6/384: creat d17/d28/d38/f89 x:0 0 0 2024-03-26T18:54:29.733 INFO:tasks.workunit.client.1.smithi190.stdout:6/385: rename d17/d28/d46 to d17/d28/d46/d8a 22 2024-03-26T18:54:29.733 INFO:tasks.workunit.client.0.smithi102.stdout:5/312: rename d2/d25/d2a/d5f to d2/d25/d5b/d72 0 2024-03-26T18:54:29.734 INFO:tasks.workunit.client.0.smithi102.stdout:0/80: dwrite d6/fb [4194304,4194304] 0 2024-03-26T18:54:29.735 INFO:tasks.workunit.client.0.smithi102.stdout:0/81: dread d6/f9 [0,4194304] 0 2024-03-26T18:54:29.735 INFO:tasks.workunit.client.0.smithi102.stdout:0/82: creat d6/de/f1b x:0 0 0 2024-03-26T18:54:29.736 INFO:tasks.workunit.client.0.smithi102.stdout:9/40: mknod c9 0 2024-03-26T18:54:29.742 INFO:tasks.workunit.client.1.smithi190.stdout:0/501: creat d0/dd/d24/d87/fad x:0 0 0 2024-03-26T18:54:29.742 INFO:tasks.workunit.client.1.smithi190.stdout:3/350: getdents d1/d3/df/d18 0 2024-03-26T18:54:29.745 INFO:tasks.workunit.client.1.smithi190.stdout:7/390: mknod d3/d1b/c96 0 2024-03-26T18:54:29.746 INFO:tasks.workunit.client.1.smithi190.stdout:7/391: chown d3/f7c 17729 1 2024-03-26T18:54:29.746 INFO:tasks.workunit.client.0.smithi102.stdout:4/37: dwrite d1/f8 [0,4194304] 0 2024-03-26T18:54:29.746 INFO:tasks.workunit.client.0.smithi102.stdout:7/6: rename f1 to f2 0 2024-03-26T18:54:29.749 INFO:tasks.workunit.client.0.smithi102.stdout:2/71: mkdir d3/d11/d15 0 2024-03-26T18:54:29.749 INFO:tasks.workunit.client.0.smithi102.stdout:2/72: write d3/f9 [767886,83816] 0 2024-03-26T18:54:29.749 INFO:tasks.workunit.client.1.smithi190.stdout:8/416: rmdir d9/dd/d23/d36/d37/d3a 39 2024-03-26T18:54:29.749 INFO:tasks.workunit.client.1.smithi190.stdout:8/417: truncate d9/dd/d15/f45 1326131 0 2024-03-26T18:54:29.749 INFO:tasks.workunit.client.1.smithi190.stdout:8/418: creat d9/dd/d23/d36/d37/d3a/d61/f93 x:0 0 0 2024-03-26T18:54:29.749 INFO:tasks.workunit.client.1.smithi190.stdout:8/419: readlink d9/dd/l47 0 2024-03-26T18:54:29.749 INFO:tasks.workunit.client.1.smithi190.stdout:8/420: truncate d9/dd/d15/d32/f53 118701 0 2024-03-26T18:54:29.750 INFO:tasks.workunit.client.1.smithi190.stdout:6/386: mkdir d17/d28/d8b 0 2024-03-26T18:54:29.750 INFO:tasks.workunit.client.0.smithi102.stdout:9/41: dread f7 [0,4194304] 0 2024-03-26T18:54:29.750 INFO:tasks.workunit.client.0.smithi102.stdout:1/15: unlink f1 0 2024-03-26T18:54:29.752 INFO:tasks.workunit.client.1.smithi190.stdout:0/502: mknod d0/dd/d24/da2/cae 0 2024-03-26T18:54:29.752 INFO:tasks.workunit.client.1.smithi190.stdout:1/300: sync 2024-03-26T18:54:29.752 INFO:tasks.workunit.client.1.smithi190.stdout:1/301: fsync d3/d9/db/f2c 0 2024-03-26T18:54:29.753 INFO:tasks.workunit.client.0.smithi102.stdout:5/313: symlink d2/d25/d2a/d5d/l73 0 2024-03-26T18:54:29.753 INFO:tasks.workunit.client.0.smithi102.stdout:5/314: truncate d2/f17 839802 0 2024-03-26T18:54:29.753 INFO:tasks.workunit.client.0.smithi102.stdout:9/42: write f7 [3103932,17525] 0 2024-03-26T18:54:29.754 INFO:tasks.workunit.client.1.smithi190.stdout:3/351: truncate d1/d3/f10 12529162 0 2024-03-26T18:54:29.755 INFO:tasks.workunit.client.1.smithi190.stdout:7/392: truncate d3/d6/d9/f24 1152601 0 2024-03-26T18:54:29.759 INFO:tasks.workunit.client.1.smithi190.stdout:6/387: symlink d17/d28/d2c/d77/l8c 0 2024-03-26T18:54:29.760 INFO:tasks.workunit.client.0.smithi102.stdout:3/202: dwrite f7 [4194304,4194304] 0 2024-03-26T18:54:29.760 INFO:tasks.workunit.client.0.smithi102.stdout:3/203: stat df/f26 0 2024-03-26T18:54:29.761 INFO:tasks.workunit.client.1.smithi190.stdout:0/503: creat d0/dd/d24/d4b/d5e/d6b/faf x:0 0 0 2024-03-26T18:54:29.762 INFO:tasks.workunit.client.0.smithi102.stdout:3/204: dread f2 [0,4194304] 0 2024-03-26T18:54:29.763 INFO:tasks.workunit.client.1.smithi190.stdout:1/302: symlink d3/d5f/l67 0 2024-03-26T18:54:29.763 INFO:tasks.workunit.client.1.smithi190.stdout:9/615: dwrite d12/f25 [0,4194304] 0 2024-03-26T18:54:29.763 INFO:tasks.workunit.client.1.smithi190.stdout:4/395: dwrite d7/d8/d1c/f7a [0,4194304] 0 2024-03-26T18:54:29.765 INFO:tasks.workunit.client.1.smithi190.stdout:3/352: mknod d1/d3/df/d18/d35/d41/d51/c66 0 2024-03-26T18:54:29.767 INFO:tasks.workunit.client.0.smithi102.stdout:1/16: mkdir d4 0 2024-03-26T18:54:29.767 INFO:tasks.workunit.client.1.smithi190.stdout:7/393: rename d3/d1b/c96 to d3/d6/d9/d1f/d41/d50/d53/d87/c97 0 2024-03-26T18:54:29.768 INFO:tasks.workunit.client.1.smithi190.stdout:7/394: chown d3/f43 389 1 2024-03-26T18:54:29.768 INFO:tasks.workunit.client.1.smithi190.stdout:2/405: sync 2024-03-26T18:54:29.768 INFO:tasks.workunit.client.1.smithi190.stdout:2/406: dread - d6/d21/f78 zero size 2024-03-26T18:54:29.769 INFO:tasks.workunit.client.1.smithi190.stdout:0/504: symlink d0/dd/d24/d4b/lb0 0 2024-03-26T18:54:29.769 INFO:tasks.workunit.client.1.smithi190.stdout:5/278: sync 2024-03-26T18:54:29.769 INFO:tasks.workunit.client.1.smithi190.stdout:5/279: readlink d2/le 0 2024-03-26T18:54:29.769 INFO:tasks.workunit.client.1.smithi190.stdout:5/280: fdatasync d2/f10 0 2024-03-26T18:54:29.770 INFO:tasks.workunit.client.0.smithi102.stdout:5/315: mknod d2/d25/d2a/d5d/c74 0 2024-03-26T18:54:29.776 INFO:tasks.workunit.client.1.smithi190.stdout:9/616: symlink d12/d2d/d1d/d23/d83/dc9/dd4/ld7 0 2024-03-26T18:54:29.776 INFO:tasks.workunit.client.1.smithi190.stdout:9/617: chown d12/d2d/d1d/d23/f33 32824 1 2024-03-26T18:54:29.778 INFO:tasks.workunit.client.1.smithi190.stdout:1/303: rename d3/da/dd/d2f/d33/l38 to d3/da/l68 0 2024-03-26T18:54:29.780 INFO:tasks.workunit.client.1.smithi190.stdout:7/395: rmdir d3/d6/d9/d1f/d41 39 2024-03-26T18:54:29.780 INFO:tasks.workunit.client.1.smithi190.stdout:7/396: read - d3/d6/d9/d14/f7e zero size 2024-03-26T18:54:29.780 INFO:tasks.workunit.client.1.smithi190.stdout:7/397: chown c1 2458 1 2024-03-26T18:54:29.780 INFO:tasks.workunit.client.1.smithi190.stdout:7/398: chown d3/d6/f56 15209 1 2024-03-26T18:54:29.781 INFO:tasks.workunit.client.1.smithi190.stdout:8/421: dwrite d9/d39/d79/f7d [0,4194304] 0 2024-03-26T18:54:29.781 INFO:tasks.workunit.client.1.smithi190.stdout:8/422: chown d9/dd/d15/d32/c52 5216 1 2024-03-26T18:54:29.781 INFO:tasks.workunit.client.1.smithi190.stdout:0/505: write d0/dd/d24/d4b/d5e/f5f [1489470,81104] 0 2024-03-26T18:54:29.782 INFO:tasks.workunit.client.1.smithi190.stdout:2/407: unlink d6/d9/d45/l5a 0 2024-03-26T18:54:29.785 INFO:tasks.workunit.client.1.smithi190.stdout:5/281: mkdir d2/d17/d62/d64 0 2024-03-26T18:54:29.789 INFO:tasks.workunit.client.0.smithi102.stdout:1/17: rmdir d4 0 2024-03-26T18:54:29.790 INFO:tasks.workunit.client.1.smithi190.stdout:9/618: dread d12/f25 [0,4194304] 0 2024-03-26T18:54:29.791 INFO:tasks.workunit.client.1.smithi190.stdout:7/399: mkdir d3/d1b/d98 0 2024-03-26T18:54:29.792 INFO:tasks.workunit.client.0.smithi102.stdout:5/316: truncate d2/f64 893221 0 2024-03-26T18:54:29.793 INFO:tasks.workunit.client.0.smithi102.stdout:5/317: read - d2/d14/f70 zero size 2024-03-26T18:54:29.793 INFO:tasks.workunit.client.0.smithi102.stdout:5/318: fdatasync d2/f10 0 2024-03-26T18:54:29.793 INFO:tasks.workunit.client.0.smithi102.stdout:5/319: write d2/f9 [182112,55824] 0 2024-03-26T18:54:29.794 INFO:tasks.workunit.client.0.smithi102.stdout:3/205: rmdir df/d1c 39 2024-03-26T18:54:29.797 INFO:tasks.workunit.client.1.smithi190.stdout:5/282: unlink d2/d17/d1b/l1f 0 2024-03-26T18:54:29.797 INFO:tasks.workunit.client.0.smithi102.stdout:3/206: getdents df/d29/d3a 0 2024-03-26T18:54:29.798 INFO:tasks.workunit.client.1.smithi190.stdout:1/304: rename d3/f2b to d3/da/dd/d2f/d34/f69 0 2024-03-26T18:54:29.799 INFO:tasks.workunit.client.1.smithi190.stdout:1/305: creat d3/da/dd/d2f/d34/d3e/f6a x:0 0 0 2024-03-26T18:54:29.800 INFO:tasks.workunit.client.1.smithi190.stdout:7/400: mkdir d3/d6/d9/d1f/d84/d99 0 2024-03-26T18:54:29.801 INFO:tasks.workunit.client.0.smithi102.stdout:3/207: mkdir df/d4d 0 2024-03-26T18:54:29.801 INFO:tasks.workunit.client.0.smithi102.stdout:3/208: rename df/d29 to df/d29/d31/d36/d4e 22 2024-03-26T18:54:29.801 INFO:tasks.workunit.client.0.smithi102.stdout:3/209: creat df/d29/d31/d36/f4f x:0 0 0 2024-03-26T18:54:29.802 INFO:tasks.workunit.client.1.smithi190.stdout:5/283: symlink d2/d17/d1b/d22/d24/l65 0 2024-03-26T18:54:29.804 INFO:tasks.workunit.client.1.smithi190.stdout:4/396: rename l4 to d7/d1d/d42/l85 0 2024-03-26T18:54:29.805 INFO:tasks.workunit.client.1.smithi190.stdout:9/619: dread d12/d2d/d1d/d23/d83/dc9/dd4/d3f/d55/f82 [0,4194304] 0 2024-03-26T18:54:29.806 INFO:tasks.workunit.client.0.smithi102.stdout:3/210: rename df/d1c/f4c to df/d29/d31/d36/f50 0 2024-03-26T18:54:29.808 INFO:tasks.workunit.client.0.smithi102.stdout:0/83: rmdir d6 39 2024-03-26T18:54:29.810 INFO:tasks.workunit.client.1.smithi190.stdout:7/401: mknod d3/d6/d9/d1f/d41/d50/d53/d87/c9a 0 2024-03-26T18:54:29.810 INFO:tasks.workunit.client.1.smithi190.stdout:1/306: write d3/d9/f20 [2774172,17667] 0 2024-03-26T18:54:29.811 INFO:tasks.workunit.client.1.smithi190.stdout:7/402: fsync d3/d6/d9/d1f/f3d 0 2024-03-26T18:54:29.811 INFO:tasks.workunit.client.1.smithi190.stdout:7/403: fsync d3/d6/d9/d14/d36/f48 0 2024-03-26T18:54:29.811 INFO:tasks.workunit.client.1.smithi190.stdout:7/404: dread - d3/d6/f91 zero size 2024-03-26T18:54:29.811 INFO:tasks.workunit.client.1.smithi190.stdout:1/307: dread - d3/d9/f3c zero size 2024-03-26T18:54:29.811 INFO:tasks.workunit.client.1.smithi190.stdout:7/405: fsync d3/d6/d9/d14/f18 0 2024-03-26T18:54:29.811 INFO:tasks.workunit.client.1.smithi190.stdout:1/308: readlink d3/d9/db/l26 0 2024-03-26T18:54:29.811 INFO:tasks.workunit.client.1.smithi190.stdout:1/309: creat d3/da/dd/d2f/d5b/f6b x:0 0 0 2024-03-26T18:54:29.811 INFO:tasks.workunit.client.1.smithi190.stdout:1/310: chown d3/d9/fe 1 1 2024-03-26T18:54:29.812 INFO:tasks.workunit.client.0.smithi102.stdout:0/84: rename d6/f14 to d6/de/d19/f1c 0 2024-03-26T18:54:29.812 INFO:tasks.workunit.client.0.smithi102.stdout:0/85: chown d6/f7 28353653 1 2024-03-26T18:54:29.812 INFO:tasks.workunit.client.1.smithi190.stdout:1/311: dread d3/da/d2e/f30 [0,4194304] 0 2024-03-26T18:54:29.813 INFO:tasks.workunit.client.0.smithi102.stdout:8/43: truncate d0/d3/f6 2006067 0 2024-03-26T18:54:29.814 INFO:tasks.workunit.client.1.smithi190.stdout:5/284: rmdir d2/d17 39 2024-03-26T18:54:29.814 INFO:tasks.workunit.client.1.smithi190.stdout:5/285: write d2/d17/d1b/d22/d24/d3b/f47 [802020,61327] 0 2024-03-26T18:54:29.814 INFO:tasks.workunit.client.1.smithi190.stdout:6/388: dwrite d17/d28/d2b/f3c [4194304,4194304] 0 2024-03-26T18:54:29.815 INFO:tasks.workunit.client.0.smithi102.stdout:2/73: dwrite d3/d6/ff [0,4194304] 0 2024-03-26T18:54:29.815 INFO:tasks.workunit.client.1.smithi190.stdout:3/353: dwrite d1/d3/df/d18/d1b/f21 [0,4194304] 0 2024-03-26T18:54:29.816 INFO:tasks.workunit.client.1.smithi190.stdout:3/354: chown d1/f26 5 1 2024-03-26T18:54:29.816 INFO:tasks.workunit.client.1.smithi190.stdout:3/355: readlink d1/de/d1c/l46 0 2024-03-26T18:54:29.816 INFO:tasks.workunit.client.1.smithi190.stdout:3/356: write d1/f42 [2936142,76258] 0 2024-03-26T18:54:29.816 INFO:tasks.workunit.client.0.smithi102.stdout:7/7: dwrite f2 [0,4194304] 0 2024-03-26T18:54:29.816 INFO:tasks.workunit.client.0.smithi102.stdout:9/43: dwrite f7 [0,4194304] 0 2024-03-26T18:54:29.816 INFO:tasks.workunit.client.0.smithi102.stdout:9/44: fsync f7 0 2024-03-26T18:54:29.817 INFO:tasks.workunit.client.1.smithi190.stdout:8/423: dwrite d9/dd/f2d [0,4194304] 0 2024-03-26T18:54:29.817 INFO:tasks.workunit.client.1.smithi190.stdout:0/506: rename d0/dd/d20/d1c/d8b/c1d to d0/dd/d24/d4b/d5e/cb1 0 2024-03-26T18:54:29.819 INFO:tasks.workunit.client.1.smithi190.stdout:9/620: write d12/d2d/d1d/d23/f5f [3160647,69513] 0 2024-03-26T18:54:29.819 INFO:tasks.workunit.client.1.smithi190.stdout:4/397: unlink d7/d8/f1e 0 2024-03-26T18:54:29.819 INFO:tasks.workunit.client.1.smithi190.stdout:4/398: chown d7/d8/d14/d6e/f84 0 1 2024-03-26T18:54:29.820 INFO:tasks.workunit.client.1.smithi190.stdout:7/406: truncate d3/d6/d9/d1f/f3d 1209363 0 2024-03-26T18:54:29.824 INFO:tasks.workunit.client.0.smithi102.stdout:2/74: read d3/f4 [531295,45471] 0 2024-03-26T18:54:29.824 INFO:tasks.workunit.client.0.smithi102.stdout:2/75: fdatasync d3/d6/f10 0 2024-03-26T18:54:29.825 INFO:tasks.workunit.client.1.smithi190.stdout:1/312: mknod d3/da/dd/c6c 0 2024-03-26T18:54:29.825 INFO:tasks.workunit.client.1.smithi190.stdout:1/313: write d3/da/dd/d2f/d34/f69 [516704,71609] 0 2024-03-26T18:54:29.825 INFO:tasks.workunit.client.1.smithi190.stdout:1/314: dread - d3/da/dd/d2f/f5e zero size 2024-03-26T18:54:29.827 INFO:tasks.workunit.client.1.smithi190.stdout:5/286: mknod d2/d17/d1b/d22/c66 0 2024-03-26T18:54:29.829 INFO:tasks.workunit.client.0.smithi102.stdout:2/76: rename d3/f9 to d3/d6/f16 0 2024-03-26T18:54:29.831 INFO:tasks.workunit.client.1.smithi190.stdout:6/389: creat d17/d28/d2c/d77/f8d x:0 0 0 2024-03-26T18:54:29.832 INFO:tasks.workunit.client.0.smithi102.stdout:7/8: dread f2 [0,4194304] 0 2024-03-26T18:54:29.832 INFO:tasks.workunit.client.0.smithi102.stdout:7/9: creat f3 x:0 0 0 2024-03-26T18:54:29.832 INFO:tasks.workunit.client.0.smithi102.stdout:3/211: dwrite df/d29/d31/d36/f50 [0,4194304] 0 2024-03-26T18:54:29.833 INFO:tasks.workunit.client.0.smithi102.stdout:9/45: dread f7 [0,4194304] 0 2024-03-26T18:54:29.834 INFO:tasks.workunit.client.0.smithi102.stdout:2/77: mknod d3/c17 0 2024-03-26T18:54:29.834 INFO:tasks.workunit.client.0.smithi102.stdout:2/78: read d3/f7 [2637351,51397] 0 2024-03-26T18:54:29.835 INFO:tasks.workunit.client.1.smithi190.stdout:8/424: dread d9/d48/f50 [4194304,4194304] 0 2024-03-26T18:54:29.836 INFO:tasks.workunit.client.1.smithi190.stdout:8/425: chown d9/dd/l28 8 1 2024-03-26T18:54:29.836 INFO:tasks.workunit.client.1.smithi190.stdout:8/426: stat d9/d16/d87 0 2024-03-26T18:54:29.836 INFO:tasks.workunit.client.1.smithi190.stdout:8/427: dread - d9/dd/f78 zero size 2024-03-26T18:54:29.836 INFO:tasks.workunit.client.0.smithi102.stdout:2/79: dread d3/f8 [0,4194304] 0 2024-03-26T18:54:29.837 INFO:tasks.workunit.client.0.smithi102.stdout:4/38: truncate d1/f4 3332936 0 2024-03-26T18:54:29.839 INFO:tasks.workunit.client.0.smithi102.stdout:1/18: getdents . 0 2024-03-26T18:54:29.839 INFO:tasks.workunit.client.0.smithi102.stdout:1/19: chown l0 27436097 1 2024-03-26T18:54:29.841 INFO:tasks.workunit.client.1.smithi190.stdout:3/357: dwrite d1/d3/d52/f62 [0,4194304] 0 2024-03-26T18:54:29.842 INFO:tasks.workunit.client.0.smithi102.stdout:7/10: rename f2 to f4 0 2024-03-26T18:54:29.845 INFO:tasks.workunit.client.1.smithi190.stdout:0/507: link d0/dd/d24/f2f d0/dd/d24/da2/d80/fb2 0 2024-03-26T18:54:29.847 INFO:tasks.workunit.client.0.smithi102.stdout:5/320: dread d2/d25/f2b [0,4194304] 0 2024-03-26T18:54:29.848 INFO:tasks.workunit.client.1.smithi190.stdout:2/408: rename d6/d9/d1a/c39 to d6/d9/d3a/d3d/d2e/c8e 0 2024-03-26T18:54:29.848 INFO:tasks.workunit.client.1.smithi190.stdout:2/409: creat d6/f8f x:0 0 0 2024-03-26T18:54:29.848 INFO:tasks.workunit.client.1.smithi190.stdout:2/410: chown d6/d9/d1a/d72/d7c 45780 1 2024-03-26T18:54:29.848 INFO:tasks.workunit.client.1.smithi190.stdout:2/411: truncate d6/d21/f78 500559 0 2024-03-26T18:54:29.849 INFO:tasks.workunit.client.0.smithi102.stdout:3/212: mkdir df/d29/d51 0 2024-03-26T18:54:29.849 INFO:tasks.workunit.client.0.smithi102.stdout:3/213: write df/f1b [8660783,7465] 0 2024-03-26T18:54:29.850 INFO:tasks.workunit.client.1.smithi190.stdout:9/621: mknod d12/d2d/d1d/d23/d83/dc9/dd4/d71/cd8 0 2024-03-26T18:54:29.851 INFO:tasks.workunit.client.0.smithi102.stdout:9/46: mknod ca 0 2024-03-26T18:54:29.853 INFO:tasks.workunit.client.0.smithi102.stdout:2/80: mknod d3/d11/c18 0 2024-03-26T18:54:29.853 INFO:tasks.workunit.client.0.smithi102.stdout:2/81: creat d3/f19 x:0 0 0 2024-03-26T18:54:29.853 INFO:tasks.workunit.client.0.smithi102.stdout:2/82: write d3/fb [1184105,71665] 0 2024-03-26T18:54:29.854 INFO:tasks.workunit.client.1.smithi190.stdout:7/407: mkdir d3/d6/d9/d1f/d41/d65/d90/d92/d9b 0 2024-03-26T18:54:29.856 INFO:tasks.workunit.client.0.smithi102.stdout:1/20: mknod c5 0 2024-03-26T18:54:29.859 INFO:tasks.workunit.client.0.smithi102.stdout:7/11: symlink l5 0 2024-03-26T18:54:29.861 INFO:tasks.workunit.client.0.smithi102.stdout:5/321: truncate d2/d14/f2f 1957857 0 2024-03-26T18:54:29.863 INFO:tasks.workunit.client.0.smithi102.stdout:3/214: creat df/d29/d3a/d47/f52 x:0 0 0 2024-03-26T18:54:29.865 INFO:tasks.workunit.client.0.smithi102.stdout:9/47: mknod cb 0 2024-03-26T18:54:29.865 INFO:tasks.workunit.client.0.smithi102.stdout:8/44: dread d0/d3/f6 [0,4194304] 0 2024-03-26T18:54:29.868 INFO:tasks.workunit.client.0.smithi102.stdout:0/86: truncate d6/fb 8341761 0 2024-03-26T18:54:29.869 INFO:tasks.workunit.client.0.smithi102.stdout:9/48: write f7 [602535,32671] 0 2024-03-26T18:54:29.869 INFO:tasks.workunit.client.0.smithi102.stdout:9/49: write f7 [4669539,67698] 0 2024-03-26T18:54:29.869 INFO:tasks.workunit.client.0.smithi102.stdout:9/50: chown l5 133518 1 2024-03-26T18:54:29.871 INFO:tasks.workunit.client.0.smithi102.stdout:9/51: write f7 [4576181,65757] 0 2024-03-26T18:54:29.871 INFO:tasks.workunit.client.0.smithi102.stdout:9/52: write f7 [283871,58715] 0 2024-03-26T18:54:29.871 INFO:tasks.workunit.client.0.smithi102.stdout:9/53: write f7 [4476864,55692] 0 2024-03-26T18:54:29.871 INFO:tasks.workunit.client.0.smithi102.stdout:2/83: mkdir d3/d6/d1a 0 2024-03-26T18:54:29.871 INFO:tasks.workunit.client.0.smithi102.stdout:2/84: write d3/d6/f16 [1548692,9186] 0 2024-03-26T18:54:29.872 INFO:tasks.workunit.client.0.smithi102.stdout:2/85: creat d3/d6/f1b x:0 0 0 2024-03-26T18:54:29.873 INFO:tasks.workunit.client.1.smithi190.stdout:3/358: mknod d1/d3/df/d18/d35/d65/c67 0 2024-03-26T18:54:29.876 INFO:tasks.workunit.client.0.smithi102.stdout:5/322: mkdir d2/d25/d75 0 2024-03-26T18:54:29.878 INFO:tasks.workunit.client.0.smithi102.stdout:8/45: symlink d0/la 0 2024-03-26T18:54:29.879 INFO:tasks.workunit.client.1.smithi190.stdout:7/408: dread d3/d1b/f66 [0,4194304] 0 2024-03-26T18:54:29.880 INFO:tasks.workunit.client.0.smithi102.stdout:0/87: getdents d6/de/d19 0 2024-03-26T18:54:29.884 INFO:tasks.workunit.client.0.smithi102.stdout:9/54: mknod cc 0 2024-03-26T18:54:29.884 INFO:tasks.workunit.client.0.smithi102.stdout:7/12: dwrite f4 [0,4194304] 0 2024-03-26T18:54:29.885 INFO:tasks.workunit.client.0.smithi102.stdout:7/13: chown f4 2011 1 2024-03-26T18:54:29.885 INFO:tasks.workunit.client.0.smithi102.stdout:7/14: creat f6 x:0 0 0 2024-03-26T18:54:29.886 INFO:tasks.workunit.client.0.smithi102.stdout:3/215: dwrite df/f39 [0,4194304] 0 2024-03-26T18:54:29.886 INFO:tasks.workunit.client.0.smithi102.stdout:3/216: write f2 [296005,117555] 0 2024-03-26T18:54:29.888 INFO:tasks.workunit.client.0.smithi102.stdout:9/55: fsync f7 0 2024-03-26T18:54:29.888 INFO:tasks.workunit.client.0.smithi102.stdout:1/21: dread f3 [0,4194304] 0 2024-03-26T18:54:29.891 INFO:tasks.workunit.client.0.smithi102.stdout:5/323: mkdir d2/d25/d5b/d76 0 2024-03-26T18:54:29.892 INFO:tasks.workunit.client.0.smithi102.stdout:9/56: dread f7 [0,4194304] 0 2024-03-26T18:54:29.892 INFO:tasks.workunit.client.0.smithi102.stdout:8/46: mknod d0/d9/cb 0 2024-03-26T18:54:29.894 INFO:tasks.workunit.client.0.smithi102.stdout:0/88: getdents d6 0 2024-03-26T18:54:29.904 INFO:tasks.workunit.client.0.smithi102.stdout:0/89: write d6/de/f16 [3958519,76801] 0 2024-03-26T18:54:29.905 INFO:tasks.workunit.client.0.smithi102.stdout:7/15: unlink f3 0 2024-03-26T18:54:29.907 INFO:tasks.workunit.client.1.smithi190.stdout:0/508: symlink d0/dd/lb3 0 2024-03-26T18:54:29.908 INFO:tasks.workunit.client.0.smithi102.stdout:4/39: rename d1/f4 to d1/fb 0 2024-03-26T18:54:29.909 INFO:tasks.workunit.client.1.smithi190.stdout:1/315: rename d3/l21 to d3/da/dd/d2f/d34/d3e/d52/l6d 0 2024-03-26T18:54:29.913 INFO:tasks.workunit.client.1.smithi190.stdout:2/412: symlink d6/d9/d3a/d3d/d2e/l90 0 2024-03-26T18:54:29.916 INFO:tasks.workunit.client.1.smithi190.stdout:4/399: getdents d7/d8/d14/d5c 0 2024-03-26T18:54:29.916 INFO:tasks.workunit.client.1.smithi190.stdout:0/509: dread d0/dd/d24/f7a [4194304,4194304] 0 2024-03-26T18:54:29.917 INFO:tasks.workunit.client.0.smithi102.stdout:9/57: dwrite f7 [0,4194304] 0 2024-03-26T18:54:29.917 INFO:tasks.workunit.client.0.smithi102.stdout:8/47: dwrite d0/d3/f6 [0,4194304] 0 2024-03-26T18:54:29.920 INFO:tasks.workunit.client.1.smithi190.stdout:6/390: getdents d17/d28/d2b 0 2024-03-26T18:54:29.922 INFO:tasks.workunit.client.1.smithi190.stdout:3/359: symlink d1/d3/df/l68 0 2024-03-26T18:54:29.922 INFO:tasks.workunit.client.1.smithi190.stdout:3/360: creat d1/de/f69 x:0 0 0 2024-03-26T18:54:29.922 INFO:tasks.workunit.client.0.smithi102.stdout:3/217: unlink df/l11 0 2024-03-26T18:54:29.924 INFO:tasks.workunit.client.1.smithi190.stdout:7/409: creat d3/d6/d9/d1f/d41/d65/d90/d92/f9c x:0 0 0 2024-03-26T18:54:29.928 INFO:tasks.workunit.client.1.smithi190.stdout:5/287: rename d2/d17/d1b/d1e/d3a/d3c/f43 to d2/d17/d1b/f67 0 2024-03-26T18:54:29.931 INFO:tasks.workunit.client.1.smithi190.stdout:2/413: creat d6/d9/d3a/d3d/d2e/d31/f91 x:0 0 0 2024-03-26T18:54:29.931 INFO:tasks.workunit.client.1.smithi190.stdout:2/414: chown d6/d9/d3a 16429 1 2024-03-26T18:54:29.931 INFO:tasks.workunit.client.1.smithi190.stdout:2/415: creat d6/d9/d3a/d3d/d55/f92 x:0 0 0 2024-03-26T18:54:29.932 INFO:tasks.workunit.client.1.smithi190.stdout:9/622: getdents d12/d2d/d1d/d23/d2b 0 2024-03-26T18:54:29.933 INFO:tasks.workunit.client.1.smithi190.stdout:9/623: creat d12/d2d/fd9 x:0 0 0 2024-03-26T18:54:29.933 INFO:tasks.workunit.client.1.smithi190.stdout:9/624: chown d12/d2d/d1d/d23/d83/d91/c9c 3667 1 2024-03-26T18:54:29.934 INFO:tasks.workunit.client.1.smithi190.stdout:4/400: stat d7/d1d/d41/d74/l79 0 2024-03-26T18:54:29.934 INFO:tasks.workunit.client.1.smithi190.stdout:4/401: write d7/d1d/d5e/f76 [785474,106843] 0 2024-03-26T18:54:29.937 INFO:tasks.workunit.client.1.smithi190.stdout:0/510: mkdir d0/dd/d24/db4 0 2024-03-26T18:54:29.939 INFO:tasks.workunit.client.1.smithi190.stdout:6/391: link d17/d28/d2c/f47 d17/d28/d38/d79/f8e 0 2024-03-26T18:54:29.939 INFO:tasks.workunit.client.1.smithi190.stdout:6/392: write d17/d28/d2c/d50/f7e [526063,25754] 0 2024-03-26T18:54:29.942 INFO:tasks.workunit.client.1.smithi190.stdout:7/410: creat d3/d6/d9/d1f/d41/f9d x:0 0 0 2024-03-26T18:54:29.942 INFO:tasks.workunit.client.1.smithi190.stdout:7/411: dread - d3/d6/d9/d1f/d41/d50/f60 zero size 2024-03-26T18:54:29.943 INFO:tasks.workunit.client.0.smithi102.stdout:8/48: dwrite d0/d3/f6 [4194304,4194304] 0 2024-03-26T18:54:29.944 INFO:tasks.workunit.client.1.smithi190.stdout:5/288: mkdir d2/d17/d1b/d22/d68 0 2024-03-26T18:54:29.948 INFO:tasks.workunit.client.1.smithi190.stdout:2/416: unlink d6/c18 0 2024-03-26T18:54:29.950 INFO:tasks.workunit.client.0.smithi102.stdout:5/324: creat d2/d14/d1a/f77 x:0 0 0 2024-03-26T18:54:29.950 INFO:tasks.workunit.client.0.smithi102.stdout:5/325: fdatasync d2/d25/d5b/d72/f66 0 2024-03-26T18:54:29.951 INFO:tasks.workunit.client.0.smithi102.stdout:0/90: creat d6/de/f1d x:0 0 0 2024-03-26T18:54:29.953 INFO:tasks.workunit.client.1.smithi190.stdout:4/402: creat d7/d1d/d41/f86 x:0 0 0 2024-03-26T18:54:29.953 INFO:tasks.workunit.client.1.smithi190.stdout:4/403: readlink d7/d1d/l59 0 2024-03-26T18:54:29.953 INFO:tasks.workunit.client.0.smithi102.stdout:7/16: mkdir d7 0 2024-03-26T18:54:29.955 INFO:tasks.workunit.client.1.smithi190.stdout:6/393: link d17/d28/d38/d79/f82 d17/d28/d2c/d50/f8f 0 2024-03-26T18:54:29.955 INFO:tasks.workunit.client.1.smithi190.stdout:6/394: read - d17/d28/d2c/d63/f70 zero size 2024-03-26T18:54:29.956 INFO:tasks.workunit.client.0.smithi102.stdout:4/40: symlink d1/lc 0 2024-03-26T18:54:29.958 INFO:tasks.workunit.client.1.smithi190.stdout:7/412: creat d3/d6/d9/d1f/d84/f9e x:0 0 0 2024-03-26T18:54:29.959 INFO:tasks.workunit.client.0.smithi102.stdout:9/58: mknod cd 0 2024-03-26T18:54:29.959 INFO:tasks.workunit.client.1.smithi190.stdout:9/625: dwrite d12/d2d/d1d/d23/d83/dc9/dd4/f6c [0,4194304] 0 2024-03-26T18:54:29.961 INFO:tasks.workunit.client.1.smithi190.stdout:5/289: symlink d2/d17/d1b/d22/l69 0 2024-03-26T18:54:29.962 INFO:tasks.workunit.client.1.smithi190.stdout:8/428: sync 2024-03-26T18:54:29.962 INFO:tasks.workunit.client.1.smithi190.stdout:8/429: creat d9/d39/d67/f94 x:0 0 0 2024-03-26T18:54:29.962 INFO:tasks.workunit.client.1.smithi190.stdout:8/430: creat d9/d39/d67/f95 x:0 0 0 2024-03-26T18:54:29.962 INFO:tasks.workunit.client.1.smithi190.stdout:8/431: fsync d9/dd/d23/f3e 0 2024-03-26T18:54:29.962 INFO:tasks.workunit.client.1.smithi190.stdout:8/432: read - d9/dd/f59 zero size 2024-03-26T18:54:29.962 INFO:tasks.workunit.client.1.smithi190.stdout:8/433: creat d9/dd/d15/d32/f96 x:0 0 0 2024-03-26T18:54:29.963 INFO:tasks.workunit.client.0.smithi102.stdout:4/41: dread d1/fb [0,4194304] 0 2024-03-26T18:54:29.963 INFO:tasks.workunit.client.0.smithi102.stdout:3/218: mkdir df/d1c/d34/d53 0 2024-03-26T18:54:29.963 INFO:tasks.workunit.client.1.smithi190.stdout:0/511: dwrite d0/dd/d24/f7a [4194304,4194304] 0 2024-03-26T18:54:29.964 INFO:tasks.workunit.client.1.smithi190.stdout:2/417: mkdir d6/d9/d45/d6b/d93 0 2024-03-26T18:54:29.964 INFO:tasks.workunit.client.1.smithi190.stdout:2/418: creat d6/d9/d3a/f94 x:0 0 0 2024-03-26T18:54:29.967 INFO:tasks.workunit.client.0.smithi102.stdout:5/326: dread d2/f10 [0,4194304] 0 2024-03-26T18:54:29.968 INFO:tasks.workunit.client.1.smithi190.stdout:4/404: rename d7/d1d/d5e/f6b to d7/d8/d14/d4b/f87 0 2024-03-26T18:54:29.969 INFO:tasks.workunit.client.0.smithi102.stdout:9/59: dread f7 [0,4194304] 0 2024-03-26T18:54:29.969 INFO:tasks.workunit.client.0.smithi102.stdout:9/60: write f7 [5506832,91661] 0 2024-03-26T18:54:29.969 INFO:tasks.workunit.client.0.smithi102.stdout:9/61: chown cb 123 1 2024-03-26T18:54:29.971 INFO:tasks.workunit.client.0.smithi102.stdout:1/22: link c5 c6 0 2024-03-26T18:54:29.974 INFO:tasks.workunit.client.0.smithi102.stdout:0/91: creat d6/de/d13/f1e x:0 0 0 2024-03-26T18:54:29.974 INFO:tasks.workunit.client.0.smithi102.stdout:0/92: chown d6/fb 10897 1 2024-03-26T18:54:29.977 INFO:tasks.workunit.client.0.smithi102.stdout:4/42: dread d1/f8 [0,4194304] 0 2024-03-26T18:54:29.977 INFO:tasks.workunit.client.1.smithi190.stdout:9/626: link d12/d2d/d1d/d23/d2b/f6e d12/d2d/d1d/da1/dba/dbc/fda 0 2024-03-26T18:54:29.978 INFO:tasks.workunit.client.1.smithi190.stdout:9/627: chown d12/d2d/l26 1041 1 2024-03-26T18:54:29.978 INFO:tasks.workunit.client.1.smithi190.stdout:9/628: stat d12/d2d/d1d/d23/d83 0 2024-03-26T18:54:29.978 INFO:tasks.workunit.client.1.smithi190.stdout:9/629: write d12/d2d/f5e [375388,109547] 0 2024-03-26T18:54:29.979 INFO:tasks.workunit.client.0.smithi102.stdout:7/17: link l5 d7/l8 0 2024-03-26T18:54:29.979 INFO:tasks.workunit.client.0.smithi102.stdout:7/18: stat d7 0 2024-03-26T18:54:29.984 INFO:tasks.workunit.client.0.smithi102.stdout:2/86: truncate d3/f7 61887 0 2024-03-26T18:54:29.985 INFO:tasks.workunit.client.0.smithi102.stdout:3/219: chown df/f26 63091367 1 2024-03-26T18:54:29.986 INFO:tasks.workunit.client.0.smithi102.stdout:2/87: dread d3/d6/fe [0,4194304] 0 2024-03-26T18:54:29.986 INFO:tasks.workunit.client.0.smithi102.stdout:2/88: chown d3 4 1 2024-03-26T18:54:29.986 INFO:tasks.workunit.client.0.smithi102.stdout:2/89: readlink d3/d6/ld 0 2024-03-26T18:54:29.986 INFO:tasks.workunit.client.1.smithi190.stdout:7/413: dwrite d3/d6/f1d [4194304,4194304] 0 2024-03-26T18:54:29.986 INFO:tasks.workunit.client.1.smithi190.stdout:5/290: dread d2/f4 [0,4194304] 0 2024-03-26T18:54:29.992 INFO:tasks.workunit.client.1.smithi190.stdout:2/419: dwrite d6/d9/d48/f7e [0,4194304] 0 2024-03-26T18:54:29.992 INFO:tasks.workunit.client.1.smithi190.stdout:8/434: truncate d9/dd/d15/f1c 475146 0 2024-03-26T18:54:29.992 INFO:tasks.workunit.client.1.smithi190.stdout:8/435: write d9/d39/d67/f94 [639303,126949] 0 2024-03-26T18:54:29.994 INFO:tasks.workunit.client.0.smithi102.stdout:0/93: dwrite d6/de/f1d [0,4194304] 0 2024-03-26T18:54:29.994 INFO:tasks.workunit.client.1.smithi190.stdout:0/512: mkdir d0/dd/d24/d87/d93/db5 0 2024-03-26T18:54:29.995 INFO:tasks.workunit.client.0.smithi102.stdout:5/327: link d2/d25/d5b/d72/f22 d2/f78 0 2024-03-26T18:54:30.000 INFO:tasks.workunit.client.0.smithi102.stdout:1/23: read - f2 zero size 2024-03-26T18:54:30.001 INFO:tasks.workunit.client.1.smithi190.stdout:9/630: symlink d12/ldb 0 2024-03-26T18:54:30.002 INFO:tasks.workunit.client.1.smithi190.stdout:9/631: readlink d12/d2d/l89 0 2024-03-26T18:54:30.003 INFO:tasks.workunit.client.0.smithi102.stdout:4/43: link d1/c5 d1/cd 0 2024-03-26T18:54:30.003 INFO:tasks.workunit.client.0.smithi102.stdout:4/44: dread - d1/fa zero size 2024-03-26T18:54:30.006 INFO:tasks.workunit.client.1.smithi190.stdout:5/291: creat d2/d17/d1b/d1e/d3a/d3c/f6a x:0 0 0 2024-03-26T18:54:30.006 INFO:tasks.workunit.client.1.smithi190.stdout:5/292: write d2/d17/d1b/d30/f3f [4370437,58902] 0 2024-03-26T18:54:30.007 INFO:tasks.workunit.client.1.smithi190.stdout:5/293: chown d2/d17/d1b/d1e/d3a/f52 8 1 2024-03-26T18:54:30.007 INFO:tasks.workunit.client.1.smithi190.stdout:5/294: chown d2/d17/d1b/d30/f51 33084 1 2024-03-26T18:54:30.007 INFO:tasks.workunit.client.1.smithi190.stdout:9/632: write d12/d2d/d1d/d54/f90 [1741185,103147] 0 2024-03-26T18:54:30.007 INFO:tasks.workunit.client.1.smithi190.stdout:9/633: write d12/f52 [171925,20460] 0 2024-03-26T18:54:30.007 INFO:tasks.workunit.client.1.smithi190.stdout:9/634: write d12/f16 [1029712,104909] 0 2024-03-26T18:54:30.007 INFO:tasks.workunit.client.1.smithi190.stdout:4/405: dread d7/d8/d1c/d20/f3d [4194304,4194304] 0 2024-03-26T18:54:30.008 INFO:tasks.workunit.client.0.smithi102.stdout:7/19: creat d7/f9 x:0 0 0 2024-03-26T18:54:30.008 INFO:tasks.workunit.client.0.smithi102.stdout:7/20: truncate f4 5143144 0 2024-03-26T18:54:30.012 INFO:tasks.workunit.client.0.smithi102.stdout:3/220: dread - df/d29/d31/d36/f4f zero size 2024-03-26T18:54:30.012 INFO:tasks.workunit.client.0.smithi102.stdout:3/221: write df/d29/d31/d36/d3c/f3f [117151,110916] 0 2024-03-26T18:54:30.017 INFO:tasks.workunit.client.0.smithi102.stdout:8/49: truncate d0/d3/f6 3782084 0 2024-03-26T18:54:30.019 INFO:tasks.workunit.client.1.smithi190.stdout:2/420: mknod d6/d9/d1a/d72/d8b/c95 0 2024-03-26T18:54:30.020 INFO:tasks.workunit.client.1.smithi190.stdout:4/406: dread d7/d8/f33 [0,4194304] 0 2024-03-26T18:54:30.022 INFO:tasks.workunit.client.1.smithi190.stdout:8/436: mknod d9/dd/d23/d36/d37/c97 0 2024-03-26T18:54:30.023 INFO:tasks.workunit.client.0.smithi102.stdout:0/94: truncate d6/fb 2240656 0 2024-03-26T18:54:30.024 INFO:tasks.workunit.client.1.smithi190.stdout:0/513: truncate d0/dd/fe 1527656 0 2024-03-26T18:54:30.025 INFO:tasks.workunit.client.0.smithi102.stdout:5/328: symlink d2/d25/d2a/l79 0 2024-03-26T18:54:30.026 INFO:tasks.workunit.client.1.smithi190.stdout:7/414: getdents d3/d1b 0 2024-03-26T18:54:30.027 INFO:tasks.workunit.client.0.smithi102.stdout:9/62: getdents . 0 2024-03-26T18:54:30.029 INFO:tasks.workunit.client.0.smithi102.stdout:1/24: creat f7 x:0 0 0 2024-03-26T18:54:30.030 INFO:tasks.workunit.client.0.smithi102.stdout:7/21: dwrite f4 [0,4194304] 0 2024-03-26T18:54:30.031 INFO:tasks.workunit.client.0.smithi102.stdout:7/22: write d7/f9 [532134,59494] 0 2024-03-26T18:54:30.032 INFO:tasks.workunit.client.1.smithi190.stdout:9/635: unlink d12/f25 0 2024-03-26T18:54:30.034 INFO:tasks.workunit.client.1.smithi190.stdout:4/407: write d7/d8/d14/d5c/f63 [2916882,126636] 0 2024-03-26T18:54:30.034 INFO:tasks.workunit.client.1.smithi190.stdout:4/408: fsync d7/d8/f2a 0 2024-03-26T18:54:30.034 INFO:tasks.workunit.client.0.smithi102.stdout:4/45: symlink d1/le 0 2024-03-26T18:54:30.035 INFO:tasks.workunit.client.0.smithi102.stdout:4/46: chown d1/l6 1933240 1 2024-03-26T18:54:30.035 INFO:tasks.workunit.client.0.smithi102.stdout:4/47: creat d1/ff x:0 0 0 2024-03-26T18:54:30.037 INFO:tasks.workunit.client.0.smithi102.stdout:6/84: sync 2024-03-26T18:54:30.037 INFO:tasks.workunit.client.0.smithi102.stdout:6/85: truncate d2/d7/f8 4892425 0 2024-03-26T18:54:30.038 INFO:tasks.workunit.client.0.smithi102.stdout:7/23: dread d7/f9 [0,4194304] 0 2024-03-26T18:54:30.038 INFO:tasks.workunit.client.0.smithi102.stdout:7/24: chown d7/f9 1 1 2024-03-26T18:54:30.039 INFO:tasks.workunit.client.0.smithi102.stdout:2/90: getdents d3 0 2024-03-26T18:54:30.045 INFO:tasks.workunit.client.0.smithi102.stdout:8/50: creat d0/d3/fc x:0 0 0 2024-03-26T18:54:30.045 INFO:tasks.workunit.client.0.smithi102.stdout:8/51: chown d0/d3/l8 2 1 2024-03-26T18:54:30.046 INFO:tasks.workunit.client.1.smithi190.stdout:2/421: mknod d6/d9/d1a/d72/d7c/c96 0 2024-03-26T18:54:30.046 INFO:tasks.workunit.client.1.smithi190.stdout:2/422: dread - d6/d9/d3a/f3b zero size 2024-03-26T18:54:30.047 INFO:tasks.workunit.client.0.smithi102.stdout:3/222: dread df/f20 [0,4194304] 0 2024-03-26T18:54:30.048 INFO:tasks.workunit.client.0.smithi102.stdout:0/95: mkdir d6/de/d13/d1f 0 2024-03-26T18:54:30.049 INFO:tasks.workunit.client.0.smithi102.stdout:3/223: write df/d29/f33 [855700,126984] 0 2024-03-26T18:54:30.049 INFO:tasks.workunit.client.0.smithi102.stdout:3/224: write f2 [1181249,27872] 0 2024-03-26T18:54:30.050 INFO:tasks.workunit.client.1.smithi190.stdout:8/437: creat d9/dd/d23/d36/d37/d3a/d43/d8f/f98 x:0 0 0 2024-03-26T18:54:30.050 INFO:tasks.workunit.client.1.smithi190.stdout:2/423: dread d6/d9/f38 [0,4194304] 0 2024-03-26T18:54:30.051 INFO:tasks.workunit.client.1.smithi190.stdout:2/424: creat d6/d9/d48/f97 x:0 0 0 2024-03-26T18:54:30.051 INFO:tasks.workunit.client.0.smithi102.stdout:5/329: unlink d2/l3f 0 2024-03-26T18:54:30.051 INFO:tasks.workunit.client.0.smithi102.stdout:5/330: dread - d2/f62 zero size 2024-03-26T18:54:30.053 INFO:tasks.workunit.client.1.smithi190.stdout:0/514: mknod d0/dd/cb6 0 2024-03-26T18:54:30.053 INFO:tasks.workunit.client.1.smithi190.stdout:0/515: fsync d0/dd/d20/d1c/d4e/f4f 0 2024-03-26T18:54:30.053 INFO:tasks.workunit.client.1.smithi190.stdout:0/516: fsync d0/dd/d20/d1c/d46/f72 0 2024-03-26T18:54:30.053 INFO:tasks.workunit.client.0.smithi102.stdout:1/25: creat f8 x:0 0 0 2024-03-26T18:54:30.055 INFO:tasks.workunit.client.1.smithi190.stdout:7/415: creat d3/d6/d9/d14/d59/f9f x:0 0 0 2024-03-26T18:54:30.055 INFO:tasks.workunit.client.1.smithi190.stdout:7/416: chown d3/d1b/f28 3861 1 2024-03-26T18:54:30.059 INFO:tasks.workunit.client.0.smithi102.stdout:5/331: dread d2/f54 [0,4194304] 0 2024-03-26T18:54:30.059 INFO:tasks.workunit.client.0.smithi102.stdout:5/332: dread - d2/d25/d2a/f6c zero size 2024-03-26T18:54:30.059 INFO:tasks.workunit.client.0.smithi102.stdout:2/91: dwrite d3/fb [0,4194304] 0 2024-03-26T18:54:30.059 INFO:tasks.workunit.client.1.smithi190.stdout:4/409: dwrite d7/d1d/d5e/f81 [0,4194304] 0 2024-03-26T18:54:30.060 INFO:tasks.workunit.client.1.smithi190.stdout:9/636: mknod d12/d2d/d1d/d23/d83/dc9/dd4/d3f/cdc 0 2024-03-26T18:54:30.066 INFO:tasks.workunit.client.0.smithi102.stdout:4/48: rename d1/fb to d1/f10 0 2024-03-26T18:54:30.066 INFO:tasks.workunit.client.0.smithi102.stdout:4/49: creat d1/f11 x:0 0 0 2024-03-26T18:54:30.067 INFO:tasks.workunit.client.1.smithi190.stdout:2/425: getdents d6/d9/d10/d5f 0 2024-03-26T18:54:30.068 INFO:tasks.workunit.client.0.smithi102.stdout:6/86: symlink d2/df/l15 0 2024-03-26T18:54:30.068 INFO:tasks.workunit.client.0.smithi102.stdout:6/87: fsync d2/d3/f5 0 2024-03-26T18:54:30.069 INFO:tasks.workunit.client.1.smithi190.stdout:0/517: mkdir d0/dd/d20/d1c/d8b/d8d/db7 0 2024-03-26T18:54:30.071 INFO:tasks.workunit.client.0.smithi102.stdout:2/92: dread d3/f8 [0,4194304] 0 2024-03-26T18:54:30.071 INFO:tasks.workunit.client.0.smithi102.stdout:2/93: fdatasync d3/f4 0 2024-03-26T18:54:30.071 INFO:tasks.workunit.client.0.smithi102.stdout:7/25: mknod d7/ca 0 2024-03-26T18:54:30.072 INFO:tasks.workunit.client.0.smithi102.stdout:4/50: dread d1/f10 [0,4194304] 0 2024-03-26T18:54:30.072 INFO:tasks.workunit.client.0.smithi102.stdout:4/51: readlink d1/l6 0 2024-03-26T18:54:30.074 INFO:tasks.workunit.client.1.smithi190.stdout:7/417: rename d3/d6/d9/d1f/f88 to d3/d1b/d98/fa0 0 2024-03-26T18:54:30.074 INFO:tasks.workunit.client.1.smithi190.stdout:7/418: write d3/d6/d9/d14/d36/f79 [73699,81068] 0 2024-03-26T18:54:30.074 INFO:tasks.workunit.client.1.smithi190.stdout:7/419: dread - d3/d6/d9/d14/d59/f61 zero size 2024-03-26T18:54:30.074 INFO:tasks.workunit.client.0.smithi102.stdout:8/52: rmdir d0 39 2024-03-26T18:54:30.076 INFO:tasks.workunit.client.0.smithi102.stdout:0/96: mkdir d6/de/d13/d1f/d20 0 2024-03-26T18:54:30.077 INFO:tasks.workunit.client.0.smithi102.stdout:1/26: dwrite f8 [0,4194304] 0 2024-03-26T18:54:30.077 INFO:tasks.workunit.client.0.smithi102.stdout:1/27: truncate f7 287561 0 2024-03-26T18:54:30.078 INFO:tasks.workunit.client.0.smithi102.stdout:5/333: creat d2/d2c/d6f/f7a x:0 0 0 2024-03-26T18:54:30.080 INFO:tasks.workunit.client.1.smithi190.stdout:4/410: symlink d7/d8/d14/d4b/l88 0 2024-03-26T18:54:30.081 INFO:tasks.workunit.client.0.smithi102.stdout:3/225: dwrite df/d29/d31/d36/d3c/f3f [0,4194304] 0 2024-03-26T18:54:30.083 INFO:tasks.workunit.client.0.smithi102.stdout:6/88: write d2/f11 [4181578,75070] 0 2024-03-26T18:54:30.084 INFO:tasks.workunit.client.0.smithi102.stdout:6/89: write d2/d7/f8 [1527001,106489] 0 2024-03-26T18:54:30.084 INFO:tasks.workunit.client.0.smithi102.stdout:2/94: creat d3/f1c x:0 0 0 2024-03-26T18:54:30.085 INFO:tasks.workunit.client.1.smithi190.stdout:9/637: creat d12/d2d/d1d/d23/d83/d91/fdd x:0 0 0 2024-03-26T18:54:30.085 INFO:tasks.workunit.client.1.smithi190.stdout:9/638: truncate d12/d2d/d1d/d23/d83/dc9/dd4/d3f/f81 331236 0 2024-03-26T18:54:30.086 INFO:tasks.workunit.client.1.smithi190.stdout:4/411: write d7/d8/f22 [479649,59875] 0 2024-03-26T18:54:30.087 INFO:tasks.workunit.client.0.smithi102.stdout:8/53: symlink d0/d3/ld 0 2024-03-26T18:54:30.087 INFO:tasks.workunit.client.0.smithi102.stdout:8/54: write d0/d3/fc [679945,52340] 0 2024-03-26T18:54:30.088 INFO:tasks.workunit.client.0.smithi102.stdout:0/97: rename c5 to d6/de/d13/d1f/d20/c21 0 2024-03-26T18:54:30.090 INFO:tasks.workunit.client.1.smithi190.stdout:8/438: getdents d9/dd/d23/d36/d37 0 2024-03-26T18:54:30.092 INFO:tasks.workunit.client.0.smithi102.stdout:1/28: creat f9 x:0 0 0 2024-03-26T18:54:30.092 INFO:tasks.workunit.client.0.smithi102.stdout:1/29: chown f9 65 1 2024-03-26T18:54:30.094 INFO:tasks.workunit.client.1.smithi190.stdout:9/639: mknod d12/d2d/cde 0 2024-03-26T18:54:30.094 INFO:tasks.workunit.client.1.smithi190.stdout:9/640: fdatasync d12/d2d/d1d/d23/d83/dc9/dd4/d71/f85 0 2024-03-26T18:54:30.095 INFO:tasks.workunit.client.1.smithi190.stdout:4/412: creat d7/d8/d14/d4b/d7d/f89 x:0 0 0 2024-03-26T18:54:30.096 INFO:tasks.workunit.client.1.smithi190.stdout:8/439: unlink d9/dd/d15/d32/f76 0 2024-03-26T18:54:30.096 INFO:tasks.workunit.client.1.smithi190.stdout:8/440: write d9/dd/d23/d36/f71 [222752,87686] 0 2024-03-26T18:54:30.096 INFO:tasks.workunit.client.1.smithi190.stdout:8/441: write d9/dd/f78 [973011,19512] 0 2024-03-26T18:54:30.096 INFO:tasks.workunit.client.1.smithi190.stdout:8/442: fsync d9/dd/d15/f2f 0 2024-03-26T18:54:30.097 INFO:tasks.workunit.client.0.smithi102.stdout:2/95: unlink c0 0 2024-03-26T18:54:30.098 INFO:tasks.workunit.client.0.smithi102.stdout:2/96: read d3/d6/fa [837323,34845] 0 2024-03-26T18:54:30.099 INFO:tasks.workunit.client.1.smithi190.stdout:2/426: rename d6/d9/l12 to d6/d9/d3a/d3d/d2e/l98 0 2024-03-26T18:54:30.100 INFO:tasks.workunit.client.0.smithi102.stdout:8/55: symlink d0/le 0 2024-03-26T18:54:30.101 INFO:tasks.workunit.client.1.smithi190.stdout:8/443: truncate d9/dd/f2d 1334298 0 2024-03-26T18:54:30.101 INFO:tasks.workunit.client.1.smithi190.stdout:8/444: creat d9/dd/f99 x:0 0 0 2024-03-26T18:54:30.102 INFO:tasks.workunit.client.0.smithi102.stdout:3/226: write df/f2d [3691777,102617] 0 2024-03-26T18:54:30.102 INFO:tasks.workunit.client.1.smithi190.stdout:2/427: symlink d6/d9/d3a/d3d/d55/l99 0 2024-03-26T18:54:30.104 INFO:tasks.workunit.client.0.smithi102.stdout:5/334: rename d2/d14/f2d to d2/d25/f7b 0 2024-03-26T18:54:30.104 INFO:tasks.workunit.client.0.smithi102.stdout:5/335: stat d2/l4b 0 2024-03-26T18:54:30.104 INFO:tasks.workunit.client.0.smithi102.stdout:5/336: readlink d2/l18 0 2024-03-26T18:54:30.104 INFO:tasks.workunit.client.1.smithi190.stdout:4/413: getdents d7/d8/d14/d6e/d6f 0 2024-03-26T18:54:30.104 INFO:tasks.workunit.client.1.smithi190.stdout:4/414: read - d7/d8/d1c/f80 zero size 2024-03-26T18:54:30.107 INFO:tasks.workunit.client.1.smithi190.stdout:0/518: dwrite d0/dd/d20/d1c/d46/f5d [0,4194304] 0 2024-03-26T18:54:30.108 INFO:tasks.workunit.client.0.smithi102.stdout:8/56: dread d0/d3/f6 [0,4194304] 0 2024-03-26T18:54:30.108 INFO:tasks.workunit.client.1.smithi190.stdout:8/445: symlink d9/d39/d67/l9a 0 2024-03-26T18:54:30.112 INFO:tasks.workunit.client.0.smithi102.stdout:2/97: rename d3/f12 to d3/d6/f1d 0 2024-03-26T18:54:30.115 INFO:tasks.workunit.client.1.smithi190.stdout:4/415: rename d7/d8/d14/d4b/f5f to d7/d8/d14/d23/d28/f8a 0 2024-03-26T18:54:30.115 INFO:tasks.workunit.client.0.smithi102.stdout:4/52: dwrite d1/fa [0,4194304] 0 2024-03-26T18:54:30.115 INFO:tasks.workunit.client.0.smithi102.stdout:4/53: truncate d1/f7 992599 0 2024-03-26T18:54:30.116 INFO:tasks.workunit.client.1.smithi190.stdout:7/420: dwrite d3/d6/d9/d14/d36/f6b [0,4194304] 0 2024-03-26T18:54:30.116 INFO:tasks.workunit.client.1.smithi190.stdout:7/421: fsync d3/d6/d9/d1f/f6c 0 2024-03-26T18:54:30.116 INFO:tasks.workunit.client.1.smithi190.stdout:9/641: dwrite d12/f14 [0,4194304] 0 2024-03-26T18:54:30.117 INFO:tasks.workunit.client.1.smithi190.stdout:0/519: mknod d0/dd/d20/d1c/d8b/d8d/cb8 0 2024-03-26T18:54:30.121 INFO:tasks.workunit.client.1.smithi190.stdout:8/446: unlink d9/d39/d67/f68 0 2024-03-26T18:54:30.121 INFO:tasks.workunit.client.1.smithi190.stdout:8/447: read d9/d48/f62 [2915696,5483] 0 2024-03-26T18:54:30.122 INFO:tasks.workunit.client.0.smithi102.stdout:3/227: creat df/d1c/d34/d53/f54 x:0 0 0 2024-03-26T18:54:30.124 INFO:tasks.workunit.client.1.smithi190.stdout:4/416: mknod d7/d8/d14/d4b/c8b 0 2024-03-26T18:54:30.124 INFO:tasks.workunit.client.0.smithi102.stdout:2/98: symlink d3/d11/l1e 0 2024-03-26T18:54:30.124 INFO:tasks.workunit.client.0.smithi102.stdout:7/26: rmdir d7 39 2024-03-26T18:54:30.129 INFO:tasks.workunit.client.1.smithi190.stdout:7/422: unlink d3/d6/c6a 0 2024-03-26T18:54:30.129 INFO:tasks.workunit.client.1.smithi190.stdout:7/423: creat d3/d6/d9/d1f/d41/d65/d90/d92/fa1 x:0 0 0 2024-03-26T18:54:30.129 INFO:tasks.workunit.client.1.smithi190.stdout:7/424: creat d3/d6/d9/d1f/fa2 x:0 0 0 2024-03-26T18:54:30.129 INFO:tasks.workunit.client.1.smithi190.stdout:7/425: write d3/d6/d9/d1f/d4c/f5c [4498257,119848] 0 2024-03-26T18:54:30.136 INFO:tasks.workunit.client.0.smithi102.stdout:5/337: symlink d2/d14/l7c 0 2024-03-26T18:54:30.136 INFO:tasks.workunit.client.0.smithi102.stdout:8/57: dwrite d0/d3/f6 [0,4194304] 0 2024-03-26T18:54:30.136 INFO:tasks.workunit.client.0.smithi102.stdout:3/228: dread f7 [4194304,4194304] 0 2024-03-26T18:54:30.137 INFO:tasks.workunit.client.1.smithi190.stdout:2/428: dwrite d6/d9/d3a/d3d/d2e/d86/f8a [0,4194304] 0 2024-03-26T18:54:30.137 INFO:tasks.workunit.client.1.smithi190.stdout:4/417: dread d7/d8/d14/d5c/f83 [0,4194304] 0 2024-03-26T18:54:30.137 INFO:tasks.workunit.client.1.smithi190.stdout:4/418: chown d7/d8/d1c/f7a 51030 1 2024-03-26T18:54:30.141 INFO:tasks.workunit.client.0.smithi102.stdout:1/30: rename l0 to la 0 2024-03-26T18:54:30.141 INFO:tasks.workunit.client.0.smithi102.stdout:1/31: chown f7 27 1 2024-03-26T18:54:30.141 INFO:tasks.workunit.client.0.smithi102.stdout:1/32: fdatasync f8 0 2024-03-26T18:54:30.142 INFO:tasks.workunit.client.1.smithi190.stdout:9/642: rename d12/d2d/l21 to d12/d2d/d1d/d23/d8e/ldf 0 2024-03-26T18:54:30.142 INFO:tasks.workunit.client.1.smithi190.stdout:9/643: chown d12/d2d/l29 1384 1 2024-03-26T18:54:30.143 INFO:tasks.workunit.client.1.smithi190.stdout:1/316: sync 2024-03-26T18:54:30.143 INFO:tasks.workunit.client.0.smithi102.stdout:2/99: creat d3/d6/d1a/f1f x:0 0 0 2024-03-26T18:54:30.143 INFO:tasks.workunit.client.0.smithi102.stdout:2/100: chown d3/d6/fa 156075701 1 2024-03-26T18:54:30.144 INFO:tasks.workunit.client.1.smithi190.stdout:0/520: symlink d0/dd/d20/d1c/d4e/lb9 0 2024-03-26T18:54:30.144 INFO:tasks.workunit.client.1.smithi190.stdout:0/521: chown d0/dd/d24/f49 10 1 2024-03-26T18:54:30.145 INFO:tasks.workunit.client.1.smithi190.stdout:0/522: write d0/dd/d20/f78 [582546,11373] 0 2024-03-26T18:54:30.145 INFO:tasks.workunit.client.1.smithi190.stdout:0/523: read d0/dd/d20/d61/f7b [7645356,31948] 0 2024-03-26T18:54:30.145 INFO:tasks.workunit.client.1.smithi190.stdout:0/524: chown d0/dd/d24/d87/d93/db5 14554 1 2024-03-26T18:54:30.145 INFO:tasks.workunit.client.1.smithi190.stdout:1/317: write d3/d9/f1c [489838,77089] 0 2024-03-26T18:54:30.145 INFO:tasks.workunit.client.0.smithi102.stdout:7/27: creat d7/fb x:0 0 0 2024-03-26T18:54:30.147 INFO:tasks.workunit.client.0.smithi102.stdout:4/54: rmdir d1 39 2024-03-26T18:54:30.147 INFO:tasks.workunit.client.0.smithi102.stdout:4/55: chown d1/lc 52340153 1 2024-03-26T18:54:30.150 INFO:tasks.workunit.client.0.smithi102.stdout:5/338: symlink d2/d2c/d6f/l7d 0 2024-03-26T18:54:30.152 INFO:tasks.workunit.client.0.smithi102.stdout:3/229: write df/f1a [2506462,114263] 0 2024-03-26T18:54:30.157 INFO:tasks.workunit.client.1.smithi190.stdout:7/426: creat d3/d6/d9/d1f/d84/d99/fa3 x:0 0 0 2024-03-26T18:54:30.158 INFO:tasks.workunit.client.0.smithi102.stdout:6/90: truncate d2/f11 470364 0 2024-03-26T18:54:30.165 INFO:tasks.workunit.client.0.smithi102.stdout:5/339: dread d2/f9 [0,4194304] 0 2024-03-26T18:54:30.165 INFO:tasks.workunit.client.0.smithi102.stdout:5/340: stat d2/f9 0 2024-03-26T18:54:30.166 INFO:tasks.workunit.client.0.smithi102.stdout:6/91: dread d2/d3/f5 [0,4194304] 0 2024-03-26T18:54:30.166 INFO:tasks.workunit.client.1.smithi190.stdout:4/419: symlink d7/d8/d1c/d20/l8c 0 2024-03-26T18:54:30.167 INFO:tasks.workunit.client.0.smithi102.stdout:1/33: creat fb x:0 0 0 2024-03-26T18:54:30.167 INFO:tasks.workunit.client.1.smithi190.stdout:0/525: dwrite d0/dd/d24/da2/fa5 [0,4194304] 0 2024-03-26T18:54:30.168 INFO:tasks.workunit.client.0.smithi102.stdout:2/101: dwrite d3/d6/f16 [0,4194304] 0 2024-03-26T18:54:30.171 INFO:tasks.workunit.client.0.smithi102.stdout:3/230: mkdir df/d29/d3a/d47/d55 0 2024-03-26T18:54:30.172 INFO:tasks.workunit.client.1.smithi190.stdout:9/644: mknod d12/ce0 0 2024-03-26T18:54:30.173 INFO:tasks.workunit.client.0.smithi102.stdout:5/341: symlink d2/d2c/d6f/l7e 0 2024-03-26T18:54:30.175 INFO:tasks.workunit.client.0.smithi102.stdout:6/92: mknod d2/d12/c16 0 2024-03-26T18:54:30.177 INFO:tasks.workunit.client.0.smithi102.stdout:1/34: creat fc x:0 0 0 2024-03-26T18:54:30.177 INFO:tasks.workunit.client.1.smithi190.stdout:7/427: getdents d3/d6/d9/d1f/d41/d65/d90 0 2024-03-26T18:54:30.179 INFO:tasks.workunit.client.1.smithi190.stdout:4/420: mknod d7/d1d/d41/c8d 0 2024-03-26T18:54:30.180 INFO:tasks.workunit.client.0.smithi102.stdout:5/342: dread d2/d14/f2f [0,4194304] 0 2024-03-26T18:54:30.183 INFO:tasks.workunit.client.0.smithi102.stdout:2/102: dwrite d3/f4 [0,4194304] 0 2024-03-26T18:54:30.183 INFO:tasks.workunit.client.0.smithi102.stdout:3/231: stat df/l18 0 2024-03-26T18:54:30.183 INFO:tasks.workunit.client.0.smithi102.stdout:3/232: dread - df/d29/d3a/d47/f52 zero size 2024-03-26T18:54:30.184 INFO:tasks.workunit.client.1.smithi190.stdout:2/429: rename d6/d9/d3a/d3d/d1c/d44/f4c to d6/d9/d3a/d3d/d2e/f9a 0 2024-03-26T18:54:30.185 INFO:tasks.workunit.client.0.smithi102.stdout:6/93: symlink d2/l17 0 2024-03-26T18:54:30.186 INFO:tasks.workunit.client.0.smithi102.stdout:7/28: getdents d7 0 2024-03-26T18:54:30.186 INFO:tasks.workunit.client.0.smithi102.stdout:7/29: readlink d7/l8 0 2024-03-26T18:54:30.186 INFO:tasks.workunit.client.0.smithi102.stdout:7/30: chown d7/fb 62763434 1 2024-03-26T18:54:30.187 INFO:tasks.workunit.client.1.smithi190.stdout:9/645: mkdir d12/d80/db2/de1 0 2024-03-26T18:54:30.188 INFO:tasks.workunit.client.1.smithi190.stdout:1/318: getdents d3/da/dd 0 2024-03-26T18:54:30.188 INFO:tasks.workunit.client.1.smithi190.stdout:1/319: fsync d3/da/d45/f49 0 2024-03-26T18:54:30.189 INFO:tasks.workunit.client.0.smithi102.stdout:2/103: mkdir d3/d20 0 2024-03-26T18:54:30.192 INFO:tasks.workunit.client.0.smithi102.stdout:5/343: write d2/f78 [2846401,57975] 0 2024-03-26T18:54:30.192 INFO:tasks.workunit.client.0.smithi102.stdout:3/233: mkdir df/d29/d51/d56 0 2024-03-26T18:54:30.193 INFO:tasks.workunit.client.1.smithi190.stdout:4/421: symlink d7/d8/d14/d23/l8e 0 2024-03-26T18:54:30.194 INFO:tasks.workunit.client.0.smithi102.stdout:6/94: symlink d2/df/l18 0 2024-03-26T18:54:30.197 INFO:tasks.workunit.client.1.smithi190.stdout:0/526: rename d0/dd/d20/d1c/f8c to d0/dd/d20/d1c/d8b/d8d/db7/fba 0 2024-03-26T18:54:30.198 INFO:tasks.workunit.client.0.smithi102.stdout:7/31: mknod d7/cc 0 2024-03-26T18:54:30.198 INFO:tasks.workunit.client.0.smithi102.stdout:7/32: fdatasync f6 0 2024-03-26T18:54:30.198 INFO:tasks.workunit.client.0.smithi102.stdout:7/33: stat f6 0 2024-03-26T18:54:30.198 INFO:tasks.workunit.client.0.smithi102.stdout:7/34: dread - f6 zero size 2024-03-26T18:54:30.200 INFO:tasks.workunit.client.1.smithi190.stdout:9/646: mknod d12/d2d/d1d/d23/d2b/ce2 0 2024-03-26T18:54:30.201 INFO:tasks.workunit.client.0.smithi102.stdout:6/95: dread d2/d3/f5 [0,4194304] 0 2024-03-26T18:54:30.201 INFO:tasks.workunit.client.0.smithi102.stdout:1/35: link c6 cd 0 2024-03-26T18:54:30.201 INFO:tasks.workunit.client.0.smithi102.stdout:1/36: truncate fc 767197 0 2024-03-26T18:54:30.201 INFO:tasks.workunit.client.0.smithi102.stdout:1/37: rmdir - no directory 2024-03-26T18:54:30.203 INFO:tasks.workunit.client.1.smithi190.stdout:1/320: unlink d3/d9/db/l39 0 2024-03-26T18:54:30.205 INFO:tasks.workunit.client.0.smithi102.stdout:2/104: symlink d3/d11/d15/l21 0 2024-03-26T18:54:30.205 INFO:tasks.workunit.client.0.smithi102.stdout:2/105: chown d3/f4 2086575232 1 2024-03-26T18:54:30.206 INFO:tasks.workunit.client.1.smithi190.stdout:0/527: dread d0/dd/d24/f7a [0,4194304] 0 2024-03-26T18:54:30.207 INFO:tasks.workunit.client.1.smithi190.stdout:0/528: chown d0/dd/d20/d1c/d46/d58 19809214 1 2024-03-26T18:54:30.208 INFO:tasks.workunit.client.0.smithi102.stdout:6/96: symlink d2/df/l19 0 2024-03-26T18:54:30.208 INFO:tasks.workunit.client.0.smithi102.stdout:6/97: readlink d2/df/l19 0 2024-03-26T18:54:30.210 INFO:tasks.workunit.client.1.smithi190.stdout:7/428: rmdir d3/d6/d9/d14 39 2024-03-26T18:54:30.211 INFO:tasks.workunit.client.1.smithi190.stdout:7/429: fsync d3/d6/d9/d14/d59/f61 0 2024-03-26T18:54:30.212 INFO:tasks.workunit.client.0.smithi102.stdout:2/106: unlink d3/d11/c14 0 2024-03-26T18:54:30.212 INFO:tasks.workunit.client.0.smithi102.stdout:2/107: dread - d3/f13 zero size 2024-03-26T18:54:30.212 INFO:tasks.workunit.client.0.smithi102.stdout:2/108: chown d3/f1c 9513 1 2024-03-26T18:54:30.213 INFO:tasks.workunit.client.0.smithi102.stdout:7/35: symlink d7/ld 0 2024-03-26T18:54:30.214 INFO:tasks.workunit.client.1.smithi190.stdout:1/321: dread d3/da/f37 [0,4194304] 0 2024-03-26T18:54:30.214 INFO:tasks.workunit.client.1.smithi190.stdout:2/430: rename d6/d9/d10/f4b to d6/f9b 0 2024-03-26T18:54:30.217 INFO:tasks.workunit.client.1.smithi190.stdout:9/647: mkdir d12/d2d/d1d/d54/d92/de3 0 2024-03-26T18:54:30.218 INFO:tasks.workunit.client.0.smithi102.stdout:5/344: dwrite d2/f62 [0,4194304] 0 2024-03-26T18:54:30.218 INFO:tasks.workunit.client.0.smithi102.stdout:5/345: fdatasync d2/d25/d5b/d72/f66 0 2024-03-26T18:54:30.218 INFO:tasks.workunit.client.0.smithi102.stdout:3/234: dwrite fd [0,4194304] 0 2024-03-26T18:54:30.221 INFO:tasks.workunit.client.0.smithi102.stdout:1/38: mkdir de 0 2024-03-26T18:54:30.225 INFO:tasks.workunit.client.1.smithi190.stdout:0/529: truncate d0/dd/d24/f2f 2391280 0 2024-03-26T18:54:30.226 INFO:tasks.workunit.client.0.smithi102.stdout:2/109: symlink d3/d6/l22 0 2024-03-26T18:54:30.226 INFO:tasks.workunit.client.0.smithi102.stdout:2/110: truncate d3/d6/d1a/f1f 311362 0 2024-03-26T18:54:30.228 INFO:tasks.workunit.client.1.smithi190.stdout:4/422: rename d7/d8/d1c/d20/d40/l53 to d7/d1d/d42/d4a/l8f 0 2024-03-26T18:54:30.229 INFO:tasks.workunit.client.0.smithi102.stdout:5/346: dread d2/f10 [0,4194304] 0 2024-03-26T18:54:30.229 INFO:tasks.workunit.client.0.smithi102.stdout:5/347: stat d2/d14/d1a/f50 0 2024-03-26T18:54:30.229 INFO:tasks.workunit.client.0.smithi102.stdout:5/348: chown d2/d14/f27 9890 1 2024-03-26T18:54:30.230 INFO:tasks.workunit.client.1.smithi190.stdout:2/431: write d6/d9/d3a/d3d/d2e/d31/f91 [985351,115624] 0 2024-03-26T18:54:30.231 INFO:tasks.workunit.client.0.smithi102.stdout:6/98: getdents d2/d3 0 2024-03-26T18:54:30.231 INFO:tasks.workunit.client.0.smithi102.stdout:6/99: fdatasync d2/f11 0 2024-03-26T18:54:30.231 INFO:tasks.workunit.client.0.smithi102.stdout:6/100: write d2/d7/f8 [2205948,45298] 0 2024-03-26T18:54:30.231 INFO:tasks.workunit.client.0.smithi102.stdout:2/111: dread d3/d6/fe [0,4194304] 0 2024-03-26T18:54:30.231 INFO:tasks.workunit.client.0.smithi102.stdout:5/349: dread d2/f54 [0,4194304] 0 2024-03-26T18:54:30.232 INFO:tasks.workunit.client.1.smithi190.stdout:8/448: truncate d9/dd/f2d 541430 0 2024-03-26T18:54:30.234 INFO:tasks.workunit.client.0.smithi102.stdout:7/36: dread d7/f9 [0,4194304] 0 2024-03-26T18:54:30.235 INFO:tasks.workunit.client.1.smithi190.stdout:0/530: symlink d0/dd/lbb 0 2024-03-26T18:54:30.235 INFO:tasks.workunit.client.1.smithi190.stdout:0/531: stat d0/dd/l97 0 2024-03-26T18:54:30.237 INFO:tasks.workunit.client.1.smithi190.stdout:7/430: dwrite d3/d1b/f25 [0,4194304] 0 2024-03-26T18:54:30.237 INFO:tasks.workunit.client.1.smithi190.stdout:7/431: write d3/d6/d9/f75 [680747,3076] 0 2024-03-26T18:54:30.239 INFO:tasks.workunit.client.1.smithi190.stdout:9/648: rename d12/d2d/d1d/d54/d92/cd6 to d12/d2d/d1d/ce4 0 2024-03-26T18:54:30.239 INFO:tasks.workunit.client.1.smithi190.stdout:9/649: dread - d12/d2d/d1d/d23/d83/d91/fdd zero size 2024-03-26T18:54:30.240 INFO:tasks.workunit.client.1.smithi190.stdout:9/650: read ff [427131,115601] 0 2024-03-26T18:54:30.240 INFO:tasks.workunit.client.1.smithi190.stdout:9/651: write d12/d2d/d1d/d23/d6a/f7e [651339,24258] 0 2024-03-26T18:54:30.240 INFO:tasks.workunit.client.1.smithi190.stdout:9/652: stat d12/d2d/d1d/d23/f33 0 2024-03-26T18:54:30.241 INFO:tasks.workunit.client.0.smithi102.stdout:3/235: rename df/f2d to df/d29/d3a/d47/f57 0 2024-03-26T18:54:30.242 INFO:tasks.workunit.client.1.smithi190.stdout:4/423: symlink d7/d1d/d41/d74/l90 0 2024-03-26T18:54:30.243 INFO:tasks.workunit.client.0.smithi102.stdout:2/112: link d3/d6/ff d3/d6/f23 0 2024-03-26T18:54:30.245 INFO:tasks.workunit.client.1.smithi190.stdout:1/322: dwrite d3/da/f47 [4194304,4194304] 0 2024-03-26T18:54:30.245 INFO:tasks.workunit.client.1.smithi190.stdout:1/323: truncate d3/d9/f15 1210568 0 2024-03-26T18:54:30.245 INFO:tasks.workunit.client.1.smithi190.stdout:1/324: read d3/d9/f16 [2372943,20964] 0 2024-03-26T18:54:30.245 INFO:tasks.workunit.client.1.smithi190.stdout:2/432: creat d6/d21/d61/d6c/f9c x:0 0 0 2024-03-26T18:54:30.246 INFO:tasks.workunit.client.0.smithi102.stdout:5/350: symlink d2/d25/d5b/d76/l7f 0 2024-03-26T18:54:30.248 INFO:tasks.workunit.client.1.smithi190.stdout:8/449: mknod d9/c9b 0 2024-03-26T18:54:30.248 INFO:tasks.workunit.client.1.smithi190.stdout:8/450: chown d9/d39/d67/l9a 2221 1 2024-03-26T18:54:30.250 INFO:tasks.workunit.client.0.smithi102.stdout:3/236: unlink f3 0 2024-03-26T18:54:30.252 INFO:tasks.workunit.client.1.smithi190.stdout:0/532: link d0/dd/d24/d4b/d5e/d81/l73 d0/dd/d20/d1c/d46/d58/lbc 0 2024-03-26T18:54:30.252 INFO:tasks.workunit.client.1.smithi190.stdout:0/533: readlink d0/dd/d24/d4b/d5e/l65 0 2024-03-26T18:54:30.254 INFO:tasks.workunit.client.1.smithi190.stdout:4/424: dread d7/d8/d14/f48 [0,4194304] 0 2024-03-26T18:54:30.255 INFO:tasks.workunit.client.1.smithi190.stdout:4/425: readlink d7/d1d/d41/d74/l90 0 2024-03-26T18:54:30.255 INFO:tasks.workunit.client.1.smithi190.stdout:4/426: creat d7/d1d/d41/f91 x:0 0 0 2024-03-26T18:54:30.255 INFO:tasks.workunit.client.1.smithi190.stdout:4/427: write d7/d8/d1c/f1f [115596,52651] 0 2024-03-26T18:54:30.257 INFO:tasks.workunit.client.0.smithi102.stdout:6/101: fdatasync d2/d7/f8 0 2024-03-26T18:54:30.258 INFO:tasks.workunit.client.0.smithi102.stdout:1/39: dwrite f7 [0,4194304] 0 2024-03-26T18:54:30.258 INFO:tasks.workunit.client.0.smithi102.stdout:1/40: truncate fb 126628 0 2024-03-26T18:54:30.259 INFO:tasks.workunit.client.0.smithi102.stdout:7/37: dwrite d7/f9 [0,4194304] 0 2024-03-26T18:54:30.265 INFO:tasks.workunit.client.0.smithi102.stdout:5/351: rename d2/d2c/d38/d43/c69 to d2/d25/d2a/d5d/c80 0 2024-03-26T18:54:30.268 INFO:tasks.workunit.client.0.smithi102.stdout:3/237: rmdir df/d29/d51/d56 0 2024-03-26T18:54:30.274 INFO:tasks.workunit.client.1.smithi190.stdout:3/361: sync 2024-03-26T18:54:30.274 INFO:tasks.workunit.client.1.smithi190.stdout:6/395: sync 2024-03-26T18:54:30.274 INFO:tasks.workunit.client.1.smithi190.stdout:5/295: sync 2024-03-26T18:54:30.279 INFO:tasks.workunit.client.0.smithi102.stdout:5/352: dread d2/d2c/d38/f5c [0,4194304] 0 2024-03-26T18:54:30.280 INFO:tasks.workunit.client.1.smithi190.stdout:2/433: link d6/d9/d45/d6b/c76 d6/d9/d45/c9d 0 2024-03-26T18:54:30.282 INFO:tasks.workunit.client.0.smithi102.stdout:1/41: creat de/ff x:0 0 0 2024-03-26T18:54:30.283 INFO:tasks.workunit.client.1.smithi190.stdout:8/451: mkdir d9/d39/d9c 0 2024-03-26T18:54:30.283 INFO:tasks.workunit.client.1.smithi190.stdout:8/452: fdatasync d9/d39/d67/f95 0 2024-03-26T18:54:30.284 INFO:tasks.workunit.client.0.smithi102.stdout:3/238: dwrite df/d29/d31/d36/f50 [0,4194304] 0 2024-03-26T18:54:30.284 INFO:tasks.workunit.client.0.smithi102.stdout:3/239: readlink df/l18 0 2024-03-26T18:54:30.284 INFO:tasks.workunit.client.0.smithi102.stdout:3/240: creat df/d29/d3a/d47/f58 x:0 0 0 2024-03-26T18:54:30.290 INFO:tasks.workunit.client.1.smithi190.stdout:4/428: rename d7/d1d/l68 to d7/d1d/l92 0 2024-03-26T18:54:30.291 INFO:tasks.workunit.client.1.smithi190.stdout:4/429: fsync d7/d1d/d42/f50 0 2024-03-26T18:54:30.298 INFO:tasks.workunit.client.0.smithi102.stdout:1/42: dwrite f9 [0,4194304] 0 2024-03-26T18:54:30.298 INFO:tasks.workunit.client.0.smithi102.stdout:1/43: creat de/f10 x:0 0 0 2024-03-26T18:54:30.302 INFO:tasks.workunit.client.0.smithi102.stdout:5/353: rename d2/d25/d5b/d72/f3e to d2/d25/f81 0 2024-03-26T18:54:30.302 INFO:tasks.workunit.client.0.smithi102.stdout:5/354: write d2/f9 [1477747,82802] 0 2024-03-26T18:54:30.302 INFO:tasks.workunit.client.0.smithi102.stdout:5/355: write d2/f9 [46027,9063] 0 2024-03-26T18:54:30.303 INFO:tasks.workunit.client.0.smithi102.stdout:5/356: write d2/d25/d5b/d72/f55 [560190,32681] 0 2024-03-26T18:54:30.308 INFO:tasks.workunit.client.0.smithi102.stdout:3/241: symlink df/d1c/d34/l59 0 2024-03-26T18:54:30.311 INFO:tasks.workunit.client.0.smithi102.stdout:7/38: mknod d7/ce 0 2024-03-26T18:54:30.319 INFO:tasks.workunit.client.0.smithi102.stdout:1/44: mknod de/c11 0 2024-03-26T18:54:30.321 INFO:tasks.workunit.client.0.smithi102.stdout:5/357: unlink d2/d14/f27 0 2024-03-26T18:54:30.322 INFO:tasks.workunit.client.0.smithi102.stdout:5/358: readlink d2/d2c/d38/l45 0 2024-03-26T18:54:30.327 INFO:tasks.workunit.client.0.smithi102.stdout:1/45: mknod de/c12 0 2024-03-26T18:54:30.329 INFO:tasks.workunit.client.0.smithi102.stdout:5/359: mkdir d2/d25/d2a/d5d/d82 0 2024-03-26T18:54:30.329 INFO:tasks.workunit.client.0.smithi102.stdout:1/46: dread f3 [0,4194304] 0 2024-03-26T18:54:30.329 INFO:tasks.workunit.client.0.smithi102.stdout:1/47: write f7 [4610505,82802] 0 2024-03-26T18:54:30.330 INFO:tasks.workunit.client.0.smithi102.stdout:1/48: stat fc 0 2024-03-26T18:54:30.330 INFO:tasks.workunit.client.0.smithi102.stdout:1/49: truncate fb 141953 0 2024-03-26T18:54:30.332 INFO:tasks.workunit.client.0.smithi102.stdout:5/360: symlink d2/d2c/l83 0 2024-03-26T18:54:30.332 INFO:tasks.workunit.client.0.smithi102.stdout:5/361: chown d2/f78 227771 1 2024-03-26T18:54:30.334 INFO:tasks.workunit.client.0.smithi102.stdout:3/242: dwrite df/d29/d31/d36/f4f [0,4194304] 0 2024-03-26T18:54:30.335 INFO:tasks.workunit.client.0.smithi102.stdout:7/39: dwrite f6 [0,4194304] 0 2024-03-26T18:54:30.335 INFO:tasks.workunit.client.0.smithi102.stdout:1/50: mknod de/c13 0 2024-03-26T18:54:30.336 INFO:tasks.workunit.client.0.smithi102.stdout:5/362: mknod d2/d25/d2a/d5d/c84 0 2024-03-26T18:54:30.351 INFO:tasks.workunit.client.0.smithi102.stdout:1/51: link fc de/f14 0 2024-03-26T18:54:30.353 INFO:tasks.workunit.client.0.smithi102.stdout:5/363: unlink d2/d25/d2a/d4f/l6e 0 2024-03-26T18:54:30.354 INFO:tasks.workunit.client.0.smithi102.stdout:0/98: sync 2024-03-26T18:54:30.355 INFO:tasks.workunit.client.0.smithi102.stdout:9/63: sync 2024-03-26T18:54:30.357 INFO:tasks.workunit.client.0.smithi102.stdout:3/243: dwrite f2 [0,4194304] 0 2024-03-26T18:54:30.358 INFO:tasks.workunit.client.0.smithi102.stdout:9/64: mknod ce 0 2024-03-26T18:54:30.363 INFO:tasks.workunit.client.0.smithi102.stdout:0/99: write d6/de/f1d [2226021,113565] 0 2024-03-26T18:54:30.364 INFO:tasks.workunit.client.0.smithi102.stdout:3/244: readlink df/l14 0 2024-03-26T18:54:30.365 INFO:tasks.workunit.client.0.smithi102.stdout:7/40: dwrite d7/fb [0,4194304] 0 2024-03-26T18:54:30.368 INFO:tasks.workunit.client.0.smithi102.stdout:9/65: mkdir df 0 2024-03-26T18:54:30.371 INFO:tasks.workunit.client.1.smithi190.stdout:9/653: mkdir d12/d2d/d1d/d23/de5 0 2024-03-26T18:54:30.373 INFO:tasks.workunit.client.1.smithi190.stdout:3/362: stat d1/de/d1c/c5c 0 2024-03-26T18:54:30.374 INFO:tasks.workunit.client.1.smithi190.stdout:3/363: truncate d1/f15 3954216 0 2024-03-26T18:54:30.374 INFO:tasks.workunit.client.1.smithi190.stdout:3/364: fdatasync d1/d3/df/d18/d1b/f21 0 2024-03-26T18:54:30.375 INFO:tasks.workunit.client.0.smithi102.stdout:9/66: write f7 [2981096,102004] 0 2024-03-26T18:54:30.375 INFO:tasks.workunit.client.0.smithi102.stdout:9/67: write f7 [1926634,104827] 0 2024-03-26T18:54:30.377 INFO:tasks.workunit.client.1.smithi190.stdout:6/396: mknod d17/d28/d74/c90 0 2024-03-26T18:54:30.377 INFO:tasks.workunit.client.1.smithi190.stdout:6/397: creat d17/d28/d2c/d77/d51/f91 x:0 0 0 2024-03-26T18:54:30.378 INFO:tasks.workunit.client.1.smithi190.stdout:5/296: creat d2/d17/d1b/d1e/d3a/f6b x:0 0 0 2024-03-26T18:54:30.378 INFO:tasks.workunit.client.1.smithi190.stdout:5/297: chown d2/d17/d1b/d22/f28 2146 1 2024-03-26T18:54:30.380 INFO:tasks.workunit.client.0.smithi102.stdout:9/68: link cd df/c10 0 2024-03-26T18:54:30.381 INFO:tasks.workunit.client.1.smithi190.stdout:1/325: symlink d3/da/dd/d2f/d34/l6e 0 2024-03-26T18:54:30.381 INFO:tasks.workunit.client.1.smithi190.stdout:1/326: dread - d3/d9/f3c zero size 2024-03-26T18:54:30.382 INFO:tasks.workunit.client.1.smithi190.stdout:1/327: fsync d3/da/dd/f12 0 2024-03-26T18:54:30.384 INFO:tasks.workunit.client.0.smithi102.stdout:9/69: symlink df/l11 0 2024-03-26T18:54:30.386 INFO:tasks.workunit.client.1.smithi190.stdout:0/534: getdents d0/dd/d20/d1c/d46/d7c 0 2024-03-26T18:54:30.386 INFO:tasks.workunit.client.0.smithi102.stdout:7/41: write f4 [852198,76756] 0 2024-03-26T18:54:30.388 INFO:tasks.workunit.client.1.smithi190.stdout:3/365: truncate d1/f42 1117241 0 2024-03-26T18:54:30.388 INFO:tasks.workunit.client.0.smithi102.stdout:9/70: mknod df/c12 0 2024-03-26T18:54:30.390 INFO:tasks.workunit.client.0.smithi102.stdout:0/100: dwrite d6/f11 [0,4194304] 0 2024-03-26T18:54:30.391 INFO:tasks.workunit.client.1.smithi190.stdout:7/432: rename d3/d6/d9/d1f/d4c/d71/f83 to d3/fa4 0 2024-03-26T18:54:30.391 INFO:tasks.workunit.client.0.smithi102.stdout:0/101: dread d6/f9 [0,4194304] 0 2024-03-26T18:54:30.392 INFO:tasks.workunit.client.0.smithi102.stdout:0/102: rename d6/de to d6/de/d13/d22 22 2024-03-26T18:54:30.393 INFO:tasks.workunit.client.0.smithi102.stdout:7/42: creat d7/ff x:0 0 0 2024-03-26T18:54:30.395 INFO:tasks.workunit.client.1.smithi190.stdout:5/298: link d2/d17/d1b/d30/l34 d2/d17/d1b/d1e/l6c 0 2024-03-26T18:54:30.395 INFO:tasks.workunit.client.1.smithi190.stdout:5/299: getdents d2/d17/d1b/d60/d63 0 2024-03-26T18:54:30.398 INFO:tasks.workunit.client.0.smithi102.stdout:9/71: creat df/f13 x:0 0 0 2024-03-26T18:54:30.401 INFO:tasks.workunit.client.1.smithi190.stdout:0/535: unlink d0/dd/d24/d4b/d5e/d66/l79 0 2024-03-26T18:54:30.402 INFO:tasks.workunit.client.0.smithi102.stdout:1/52: fsync fc 0 2024-03-26T18:54:30.402 INFO:tasks.workunit.client.0.smithi102.stdout:1/53: creat de/f15 x:0 0 0 2024-03-26T18:54:30.403 INFO:tasks.workunit.client.1.smithi190.stdout:3/366: dread d1/f5 [0,4194304] 0 2024-03-26T18:54:30.403 INFO:tasks.workunit.client.0.smithi102.stdout:0/103: dread d6/f7 [0,4194304] 0 2024-03-26T18:54:30.405 INFO:tasks.workunit.client.0.smithi102.stdout:9/72: mknod df/c14 0 2024-03-26T18:54:30.405 INFO:tasks.workunit.client.0.smithi102.stdout:9/73: read - df/f13 zero size 2024-03-26T18:54:30.406 INFO:tasks.workunit.client.1.smithi190.stdout:7/433: creat d3/d6/d9/d14/d59/fa5 x:0 0 0 2024-03-26T18:54:30.406 INFO:tasks.workunit.client.1.smithi190.stdout:7/434: write d3/d6/f4e [504954,127716] 0 2024-03-26T18:54:30.406 INFO:tasks.workunit.client.1.smithi190.stdout:7/435: chown d3/d6/d9/d14/f33 902922 1 2024-03-26T18:54:30.407 INFO:tasks.workunit.client.0.smithi102.stdout:7/43: mkdir d7/d10 0 2024-03-26T18:54:30.407 INFO:tasks.workunit.client.1.smithi190.stdout:0/536: symlink d0/dd/d24/d4b/d5e/lbd 0 2024-03-26T18:54:30.410 INFO:tasks.workunit.client.1.smithi190.stdout:3/367: mknod d1/d3/df/d18/d35/d41/d61/c6a 0 2024-03-26T18:54:30.411 INFO:tasks.workunit.client.0.smithi102.stdout:1/54: mkdir de/d16 0 2024-03-26T18:54:30.415 INFO:tasks.workunit.client.0.smithi102.stdout:0/104: mknod d6/c23 0 2024-03-26T18:54:30.417 INFO:tasks.workunit.client.1.smithi190.stdout:6/398: dwrite d17/f23 [0,4194304] 0 2024-03-26T18:54:30.417 INFO:tasks.workunit.client.1.smithi190.stdout:6/399: dread - d17/d28/d38/d79/f82 zero size 2024-03-26T18:54:30.417 INFO:tasks.workunit.client.1.smithi190.stdout:0/537: creat d0/dd/d20/fbe x:0 0 0 2024-03-26T18:54:30.417 INFO:tasks.workunit.client.1.smithi190.stdout:0/538: chown d0/dd/c6f 292557632 1 2024-03-26T18:54:30.419 INFO:tasks.workunit.client.0.smithi102.stdout:1/55: link f2 de/d16/f17 0 2024-03-26T18:54:30.419 INFO:tasks.workunit.client.0.smithi102.stdout:1/56: dread - f2 zero size 2024-03-26T18:54:30.420 INFO:tasks.workunit.client.0.smithi102.stdout:7/44: write f6 [3683124,13106] 0 2024-03-26T18:54:30.421 INFO:tasks.workunit.client.1.smithi190.stdout:3/368: creat d1/d3/df/d31/d5b/f6b x:0 0 0 2024-03-26T18:54:30.421 INFO:tasks.workunit.client.1.smithi190.stdout:3/369: readlink d1/de/d1c/l46 0 2024-03-26T18:54:30.422 INFO:tasks.workunit.client.1.smithi190.stdout:3/370: write d1/f3b [2449782,71324] 0 2024-03-26T18:54:30.423 INFO:tasks.workunit.client.0.smithi102.stdout:0/105: link d6/de/d13/c17 d6/de/d13/d1f/d20/c24 0 2024-03-26T18:54:30.426 INFO:tasks.workunit.client.0.smithi102.stdout:1/57: unlink c5 0 2024-03-26T18:54:30.426 INFO:tasks.workunit.client.0.smithi102.stdout:1/58: write de/f14 [147327,10931] 0 2024-03-26T18:54:30.427 INFO:tasks.workunit.client.1.smithi190.stdout:8/453: rename d9/dd/d15/c3b to d9/c9d 0 2024-03-26T18:54:30.429 INFO:tasks.workunit.client.1.smithi190.stdout:4/430: rename d7/d8/d14/d4b/f87 to d7/d8/d14/d6e/d6f/f93 0 2024-03-26T18:54:30.431 INFO:tasks.workunit.client.1.smithi190.stdout:8/454: mkdir d9/d39/d9e 0 2024-03-26T18:54:30.432 INFO:tasks.workunit.client.0.smithi102.stdout:0/106: mkdir d6/de/d13/d25 0 2024-03-26T18:54:30.432 INFO:tasks.workunit.client.0.smithi102.stdout:0/107: chown d6/de/l12 15914800 1 2024-03-26T18:54:30.435 INFO:tasks.workunit.client.0.smithi102.stdout:7/45: rename d7/ce to d7/c11 0 2024-03-26T18:54:30.436 INFO:tasks.workunit.client.1.smithi190.stdout:9/654: dwrite d12/d2d/d1d/d23/d83/dc9/dd4/d3f/f81 [0,4194304] 0 2024-03-26T18:54:30.436 INFO:tasks.workunit.client.1.smithi190.stdout:9/655: readlink d12/d2d/d1d/d23/d83/dc9/dd4/d3f/lc5 0 2024-03-26T18:54:30.437 INFO:tasks.workunit.client.1.smithi190.stdout:4/431: creat d7/d8/d14/f94 x:0 0 0 2024-03-26T18:54:30.440 INFO:tasks.workunit.client.1.smithi190.stdout:0/539: rename d0/dd/d20/d1c/l23 to d0/dd/d20/d1c/d46/d58/lbf 0 2024-03-26T18:54:30.440 INFO:tasks.workunit.client.1.smithi190.stdout:0/540: creat d0/dd/d24/da2/fc0 x:0 0 0 2024-03-26T18:54:30.445 INFO:tasks.workunit.client.0.smithi102.stdout:1/59: write f9 [1831698,80538] 0 2024-03-26T18:54:30.446 INFO:tasks.workunit.client.0.smithi102.stdout:9/74: dread f7 [4194304,4194304] 0 2024-03-26T18:54:30.447 INFO:tasks.workunit.client.1.smithi190.stdout:9/656: mknod d12/d2d/d1d/d23/d2b/ce6 0 2024-03-26T18:54:30.450 INFO:tasks.workunit.client.1.smithi190.stdout:9/657: truncate d12/d2d/d1d/f3a 2884260 0 2024-03-26T18:54:30.450 INFO:tasks.workunit.client.1.smithi190.stdout:9/658: creat d12/d2d/d1d/d23/d83/d91/fe7 x:0 0 0 2024-03-26T18:54:30.450 INFO:tasks.workunit.client.0.smithi102.stdout:1/60: link la de/l18 0 2024-03-26T18:54:30.451 INFO:tasks.workunit.client.0.smithi102.stdout:1/61: chown de/ff 13 1 2024-03-26T18:54:30.452 INFO:tasks.workunit.client.1.smithi190.stdout:9/659: link d12/d2d/d1d/d54/f90 d12/d2d/d1d/d23/d83/dc9/dd4/fe8 0 2024-03-26T18:54:30.453 INFO:tasks.workunit.client.0.smithi102.stdout:1/62: link f8 de/f19 0 2024-03-26T18:54:30.454 INFO:tasks.workunit.client.1.smithi190.stdout:9/660: getdents d12/d2d/d1d/d23/d8e 0 2024-03-26T18:54:30.454 INFO:tasks.workunit.client.1.smithi190.stdout:9/661: write d12/d2d/d1d/d23/d2b/f95 [816413,24454] 0 2024-03-26T18:54:30.454 INFO:tasks.workunit.client.1.smithi190.stdout:9/662: getdents d12/d2d/d1d/d54/d92/de3 0 2024-03-26T18:54:30.455 INFO:tasks.workunit.client.0.smithi102.stdout:1/63: mknod de/c1a 0 2024-03-26T18:54:30.455 INFO:tasks.workunit.client.0.smithi102.stdout:1/64: write f9 [1061131,109984] 0 2024-03-26T18:54:30.457 INFO:tasks.workunit.client.0.smithi102.stdout:1/65: symlink de/d16/l1b 0 2024-03-26T18:54:30.457 INFO:tasks.workunit.client.0.smithi102.stdout:1/66: chown f3 29113472 1 2024-03-26T18:54:30.457 INFO:tasks.workunit.client.0.smithi102.stdout:1/67: creat de/f1c x:0 0 0 2024-03-26T18:54:30.457 INFO:tasks.workunit.client.0.smithi102.stdout:1/68: write fc [1349427,64916] 0 2024-03-26T18:54:30.457 INFO:tasks.workunit.client.0.smithi102.stdout:1/69: chown de/f19 38 1 2024-03-26T18:54:30.458 INFO:tasks.workunit.client.1.smithi190.stdout:7/436: dwrite d3/d6/d9/d1f/d4c/f54 [0,4194304] 0 2024-03-26T18:54:30.458 INFO:tasks.workunit.client.1.smithi190.stdout:2/434: dwrite d6/d9/d3a/d3d/d2e/f51 [0,4194304] 0 2024-03-26T18:54:30.459 INFO:tasks.workunit.client.0.smithi102.stdout:1/70: mkdir de/d16/d1d 0 2024-03-26T18:54:30.460 INFO:tasks.workunit.client.1.smithi190.stdout:4/432: read d7/d8/d14/f5b [1081215,23538] 0 2024-03-26T18:54:30.460 INFO:tasks.workunit.client.1.smithi190.stdout:4/433: readlink d7/d8/d1c/l72 0 2024-03-26T18:54:30.460 INFO:tasks.workunit.client.1.smithi190.stdout:5/300: dwrite d2/f45 [0,4194304] 0 2024-03-26T18:54:30.460 INFO:tasks.workunit.client.1.smithi190.stdout:1/328: dwrite d3/d9/db/f2c [4194304,4194304] 0 2024-03-26T18:54:30.460 INFO:tasks.workunit.client.1.smithi190.stdout:1/329: creat d3/da/dd/d2f/d34/d3e/d52/f6f x:0 0 0 2024-03-26T18:54:30.461 INFO:tasks.workunit.client.1.smithi190.stdout:7/437: mknod d3/d6/d9/d1f/d41/d65/d90/ca6 0 2024-03-26T18:54:30.463 INFO:tasks.workunit.client.0.smithi102.stdout:1/71: creat de/d16/d1d/f1e x:0 0 0 2024-03-26T18:54:30.463 INFO:tasks.workunit.client.0.smithi102.stdout:1/72: fdatasync de/f10 0 2024-03-26T18:54:30.463 INFO:tasks.workunit.client.0.smithi102.stdout:1/73: dread - f2 zero size 2024-03-26T18:54:30.465 INFO:tasks.workunit.client.0.smithi102.stdout:1/74: mkdir de/d1f 0 2024-03-26T18:54:30.465 INFO:tasks.workunit.client.0.smithi102.stdout:1/75: write de/f10 [169516,101494] 0 2024-03-26T18:54:30.465 INFO:tasks.workunit.client.0.smithi102.stdout:1/76: truncate de/ff 633063 0 2024-03-26T18:54:30.466 INFO:tasks.workunit.client.0.smithi102.stdout:1/77: chown de/c13 1 1 2024-03-26T18:54:30.466 INFO:tasks.workunit.client.0.smithi102.stdout:1/78: chown de/f15 0 1 2024-03-26T18:54:30.467 INFO:tasks.workunit.client.0.smithi102.stdout:1/79: creat de/d16/f20 x:0 0 0 2024-03-26T18:54:30.467 INFO:tasks.workunit.client.0.smithi102.stdout:1/80: chown cd 11780837 1 2024-03-26T18:54:30.468 INFO:tasks.workunit.client.0.smithi102.stdout:1/81: write de/d16/f20 [408846,18413] 0 2024-03-26T18:54:30.469 INFO:tasks.workunit.client.1.smithi190.stdout:2/435: fdatasync d6/d9/d3a/d3d/d2e/f6e 0 2024-03-26T18:54:30.469 INFO:tasks.workunit.client.1.smithi190.stdout:2/436: chown d6/d9/d3a/d3d/d2e/f9a 246 1 2024-03-26T18:54:30.469 INFO:tasks.workunit.client.0.smithi102.stdout:1/82: mknod de/d1f/c21 0 2024-03-26T18:54:30.470 INFO:tasks.workunit.client.1.smithi190.stdout:4/434: mknod d7/c95 0 2024-03-26T18:54:30.473 INFO:tasks.workunit.client.1.smithi190.stdout:5/301: mknod d2/d17/d1b/c6d 0 2024-03-26T18:54:30.475 INFO:tasks.workunit.client.1.smithi190.stdout:1/330: link d3/d9/db/f32 d3/da/dd/d2f/d5b/f70 0 2024-03-26T18:54:30.475 INFO:tasks.workunit.client.1.smithi190.stdout:1/331: truncate d3/da/dd/d2f/d34/d3e/f56 302035 0 2024-03-26T18:54:30.475 INFO:tasks.workunit.client.1.smithi190.stdout:1/332: write d3/d9/f3c [559186,125727] 0 2024-03-26T18:54:30.475 INFO:tasks.workunit.client.1.smithi190.stdout:7/438: rmdir d3/d6/d9/d14/d59/d73 39 2024-03-26T18:54:30.475 INFO:tasks.workunit.client.1.smithi190.stdout:7/439: dread - d3/fa4 zero size 2024-03-26T18:54:30.475 INFO:tasks.workunit.client.1.smithi190.stdout:7/440: chown d3/d6/l95 53 1 2024-03-26T18:54:30.475 INFO:tasks.workunit.client.1.smithi190.stdout:7/441: fsync d3/d6/d9/d1f/d41/d50/d53/f8a 0 2024-03-26T18:54:30.476 INFO:tasks.workunit.client.1.smithi190.stdout:4/435: mkdir d7/d8/d14/d23/d28/d38/d96 0 2024-03-26T18:54:30.477 INFO:tasks.workunit.client.1.smithi190.stdout:5/302: rename d2/d17/d1b/d1e/d3a/f48 to d2/d17/d1b/d60/d63/f6e 0 2024-03-26T18:54:30.478 INFO:tasks.workunit.client.1.smithi190.stdout:1/333: creat d3/d5f/f71 x:0 0 0 2024-03-26T18:54:30.479 INFO:tasks.workunit.client.1.smithi190.stdout:4/436: creat d7/d8/d14/d5c/f97 x:0 0 0 2024-03-26T18:54:30.479 INFO:tasks.workunit.client.1.smithi190.stdout:4/437: fsync d7/d1d/f26 0 2024-03-26T18:54:30.479 INFO:tasks.workunit.client.1.smithi190.stdout:4/438: write d7/d8/d1c/d20/f58 [3223877,46080] 0 2024-03-26T18:54:30.479 INFO:tasks.workunit.client.1.smithi190.stdout:5/303: stat d2/c11 0 2024-03-26T18:54:30.479 INFO:tasks.workunit.client.1.smithi190.stdout:5/304: readlink d2/l2b 0 2024-03-26T18:54:30.480 INFO:tasks.workunit.client.1.smithi190.stdout:1/334: link d3/d9/db/f32 d3/da/dd/f72 0 2024-03-26T18:54:30.481 INFO:tasks.workunit.client.1.smithi190.stdout:4/439: unlink d7/d8/d1c/d20/d40/l69 0 2024-03-26T18:54:30.482 INFO:tasks.workunit.client.1.smithi190.stdout:5/305: symlink d2/d17/d1b/d1e/d3a/d3c/l6f 0 2024-03-26T18:54:30.482 INFO:tasks.workunit.client.1.smithi190.stdout:5/306: chown d2/d17/d1b/d22/d24/d3b/f46 1292 1 2024-03-26T18:54:30.482 INFO:tasks.workunit.client.1.smithi190.stdout:4/440: getdents d7/d1d/d41/d74 0 2024-03-26T18:54:30.483 INFO:tasks.workunit.client.1.smithi190.stdout:5/307: creat d2/d17/d62/d64/f70 x:0 0 0 2024-03-26T18:54:30.483 INFO:tasks.workunit.client.1.smithi190.stdout:4/441: mkdir d7/d1d/d5e/d98 0 2024-03-26T18:54:30.485 INFO:tasks.workunit.client.1.smithi190.stdout:5/308: unlink d2/d17/d1b/d1e/c39 0 2024-03-26T18:54:30.485 INFO:tasks.workunit.client.1.smithi190.stdout:5/309: write d2/d17/d1b/d22/d24/d3b/f46 [468238,24646] 0 2024-03-26T18:54:30.485 INFO:tasks.workunit.client.1.smithi190.stdout:4/442: mknod d7/d1d/d41/c99 0 2024-03-26T18:54:30.485 INFO:tasks.workunit.client.1.smithi190.stdout:4/443: fdatasync d7/d8/d14/d23/d28/d38/f7f 0 2024-03-26T18:54:30.486 INFO:tasks.workunit.client.1.smithi190.stdout:5/310: link d2/le d2/d17/d1b/d22/d68/l71 0 2024-03-26T18:54:30.486 INFO:tasks.workunit.client.1.smithi190.stdout:5/311: chown d2/d17/d1b/d1e/f5f 0 1 2024-03-26T18:54:30.486 INFO:tasks.workunit.client.1.smithi190.stdout:5/312: write d2/f14 [528562,122778] 0 2024-03-26T18:54:30.487 INFO:tasks.workunit.client.1.smithi190.stdout:4/444: unlink d7/d8/d1c/d20/f3d 0 2024-03-26T18:54:30.488 INFO:tasks.workunit.client.1.smithi190.stdout:5/313: creat d2/d17/d1b/f72 x:0 0 0 2024-03-26T18:54:30.488 INFO:tasks.workunit.client.1.smithi190.stdout:4/445: mkdir d7/d8/d14/d23/d28/d38/d96/d9a 0 2024-03-26T18:54:30.489 INFO:tasks.workunit.client.1.smithi190.stdout:4/446: truncate d7/d8/d14/f94 338701 0 2024-03-26T18:54:30.489 INFO:tasks.workunit.client.1.smithi190.stdout:4/447: fdatasync d7/d8/d14/f48 0 2024-03-26T18:54:30.489 INFO:tasks.workunit.client.1.smithi190.stdout:4/448: mknod d7/d8/d14/d23/c9b 0 2024-03-26T18:54:30.489 INFO:tasks.workunit.client.1.smithi190.stdout:4/449: readlink d7/d8/d14/d23/l8e 0 2024-03-26T18:54:30.490 INFO:tasks.workunit.client.1.smithi190.stdout:4/450: symlink d7/d1d/d42/d4a/l9c 0 2024-03-26T18:54:30.490 INFO:tasks.workunit.client.1.smithi190.stdout:1/335: dread d3/da/dd/d2f/d5b/f70 [0,4194304] 0 2024-03-26T18:54:30.497 INFO:tasks.workunit.client.1.smithi190.stdout:8/455: dwrite d9/d16/f51 [0,4194304] 0 2024-03-26T18:54:30.498 INFO:tasks.workunit.client.1.smithi190.stdout:6/400: dwrite d17/d28/d80/f88 [0,4194304] 0 2024-03-26T18:54:30.499 INFO:tasks.workunit.client.0.smithi102.stdout:1/83: read de/d16/f20 [222612,121260] 0 2024-03-26T18:54:30.499 INFO:tasks.workunit.client.0.smithi102.stdout:1/84: chown de/c13 223 1 2024-03-26T18:54:30.499 INFO:tasks.workunit.client.0.smithi102.stdout:1/85: creat de/f22 x:0 0 0 2024-03-26T18:54:30.499 INFO:tasks.workunit.client.1.smithi190.stdout:6/401: rmdir d17/d28/d2b 39 2024-03-26T18:54:30.500 INFO:tasks.workunit.client.1.smithi190.stdout:6/402: write d17/d28/f58 [1607865,39480] 0 2024-03-26T18:54:30.500 INFO:tasks.workunit.client.0.smithi102.stdout:1/86: write f7 [4278640,29071] 0 2024-03-26T18:54:30.500 INFO:tasks.workunit.client.0.smithi102.stdout:1/87: chown de/f1c 27954184 1 2024-03-26T18:54:30.500 INFO:tasks.workunit.client.0.smithi102.stdout:0/108: dwrite d6/de/d19/f1c [0,4194304] 0 2024-03-26T18:54:30.503 INFO:tasks.workunit.client.0.smithi102.stdout:0/109: rename d6/de/f16 to d6/de/d13/d1f/f26 0 2024-03-26T18:54:30.503 INFO:tasks.workunit.client.0.smithi102.stdout:0/110: write d6/de/d13/f1e [731559,49923] 0 2024-03-26T18:54:30.505 INFO:tasks.workunit.client.1.smithi190.stdout:3/371: dread d1/f3b [0,4194304] 0 2024-03-26T18:54:30.505 INFO:tasks.workunit.client.1.smithi190.stdout:3/372: dread - d1/d3/df/d18/d35/f60 zero size 2024-03-26T18:54:30.506 INFO:tasks.workunit.client.1.smithi190.stdout:5/314: dread d2/f12 [0,4194304] 0 2024-03-26T18:54:30.508 INFO:tasks.workunit.client.1.smithi190.stdout:2/437: dread d6/d9/d10/f34 [0,4194304] 0 2024-03-26T18:54:30.509 INFO:tasks.workunit.client.1.smithi190.stdout:7/442: write d3/d6/f5d [245253,86208] 0 2024-03-26T18:54:30.509 INFO:tasks.workunit.client.1.smithi190.stdout:7/443: dread - d3/f7c zero size 2024-03-26T18:54:30.509 INFO:tasks.workunit.client.1.smithi190.stdout:7/444: fdatasync d3/d6/f8c 0 2024-03-26T18:54:30.509 INFO:tasks.workunit.client.1.smithi190.stdout:7/445: dread - d3/d6/d9/d14/d59/f9f zero size 2024-03-26T18:54:30.510 INFO:tasks.workunit.client.0.smithi102.stdout:0/111: getdents d6/de/d13/d1f/d20 0 2024-03-26T18:54:30.510 INFO:tasks.workunit.client.1.smithi190.stdout:1/336: read d3/da/f24 [3568867,46192] 0 2024-03-26T18:54:30.513 INFO:tasks.workunit.client.1.smithi190.stdout:7/446: dread d3/d1b/f5e [0,4194304] 0 2024-03-26T18:54:30.517 INFO:tasks.workunit.client.1.smithi190.stdout:3/373: dread d1/de/d1c/d36/f3a [0,4194304] 0 2024-03-26T18:54:30.517 INFO:tasks.workunit.client.1.smithi190.stdout:3/374: chown d1/d47/c5e 13475 1 2024-03-26T18:54:30.517 INFO:tasks.workunit.client.1.smithi190.stdout:1/337: unlink d3/da/dd/d2f/d34/d3e/f6a 0 2024-03-26T18:54:30.517 INFO:tasks.workunit.client.0.smithi102.stdout:1/88: dread f7 [0,4194304] 0 2024-03-26T18:54:30.522 INFO:tasks.workunit.client.0.smithi102.stdout:7/46: getdents d7 0 2024-03-26T18:54:30.525 INFO:tasks.workunit.client.1.smithi190.stdout:7/447: dread d3/d6/d9/f40 [0,4194304] 0 2024-03-26T18:54:30.525 INFO:tasks.workunit.client.1.smithi190.stdout:7/448: readlink d3/d6/d9/d14/l4d 0 2024-03-26T18:54:30.525 INFO:tasks.workunit.client.1.smithi190.stdout:6/403: dwrite d17/d28/d2c/d77/f69 [4194304,4194304] 0 2024-03-26T18:54:30.525 INFO:tasks.workunit.client.1.smithi190.stdout:7/449: stat d3/d6/d9/d1f/d41/c68 0 2024-03-26T18:54:30.525 INFO:tasks.workunit.client.1.smithi190.stdout:7/450: truncate d3/d6/d9/d1f/d4c/f7f 456259 0 2024-03-26T18:54:30.525 INFO:tasks.workunit.client.1.smithi190.stdout:7/451: stat d3/d6/d9/d1f/d4c/d71 0 2024-03-26T18:54:30.525 INFO:tasks.workunit.client.1.smithi190.stdout:6/404: read d17/d28/d2c/f4c [1580132,115585] 0 2024-03-26T18:54:30.525 INFO:tasks.workunit.client.1.smithi190.stdout:6/405: write d17/d28/d2c/d77/f69 [9406827,115517] 0 2024-03-26T18:54:30.526 INFO:tasks.workunit.client.1.smithi190.stdout:3/375: symlink d1/d3/df/d18/d35/d65/l6c 0 2024-03-26T18:54:30.530 INFO:tasks.workunit.client.1.smithi190.stdout:1/338: mkdir d3/d73 0 2024-03-26T18:54:30.530 INFO:tasks.workunit.client.0.smithi102.stdout:1/89: chown de/f14 44393282 1 2024-03-26T18:54:30.531 INFO:tasks.workunit.client.0.smithi102.stdout:1/90: dread - de/f22 zero size 2024-03-26T18:54:30.531 INFO:tasks.workunit.client.0.smithi102.stdout:1/91: dread - de/f15 zero size 2024-03-26T18:54:30.531 INFO:tasks.workunit.client.0.smithi102.stdout:8/58: sync 2024-03-26T18:54:30.531 INFO:tasks.workunit.client.0.smithi102.stdout:7/47: mkdir d7/d12 0 2024-03-26T18:54:30.531 INFO:tasks.workunit.client.1.smithi190.stdout:7/452: symlink d3/d6/d9/d1f/d4c/d71/la7 0 2024-03-26T18:54:30.532 INFO:tasks.workunit.client.0.smithi102.stdout:8/59: dread d0/d3/fc [0,4194304] 0 2024-03-26T18:54:30.532 INFO:tasks.workunit.client.0.smithi102.stdout:1/92: mknod de/c23 0 2024-03-26T18:54:30.533 INFO:tasks.workunit.client.1.smithi190.stdout:3/376: rmdir d1/d3 39 2024-03-26T18:54:30.534 INFO:tasks.workunit.client.1.smithi190.stdout:3/377: creat d1/d3/d52/f6d x:0 0 0 2024-03-26T18:54:30.535 INFO:tasks.workunit.client.0.smithi102.stdout:8/60: unlink d0/d3/ld 0 2024-03-26T18:54:30.535 INFO:tasks.workunit.client.0.smithi102.stdout:8/61: chown d0/d3/fc 348977 1 2024-03-26T18:54:30.537 INFO:tasks.workunit.client.0.smithi102.stdout:7/48: dread f4 [0,4194304] 0 2024-03-26T18:54:30.538 INFO:tasks.workunit.client.1.smithi190.stdout:1/339: rename d3/da/dd/d2f/d34/d3e/d52/c61 to d3/d5f/c74 0 2024-03-26T18:54:30.539 INFO:tasks.workunit.client.1.smithi190.stdout:1/340: dread - d3/da/dd/f1b zero size 2024-03-26T18:54:30.539 INFO:tasks.workunit.client.1.smithi190.stdout:1/341: read - d3/da/d3a/f55 zero size 2024-03-26T18:54:30.542 INFO:tasks.workunit.client.1.smithi190.stdout:3/378: getdents d1/d3/df/d18/d35/d41/d51 0 2024-03-26T18:54:30.542 INFO:tasks.workunit.client.1.smithi190.stdout:3/379: write d1/de/f2d [873735,59834] 0 2024-03-26T18:54:30.542 INFO:tasks.workunit.client.1.smithi190.stdout:3/380: write d1/de/f29 [1111146,13055] 0 2024-03-26T18:54:30.543 INFO:tasks.workunit.client.1.smithi190.stdout:3/381: stat d1/d3/d52 0 2024-03-26T18:54:30.543 INFO:tasks.workunit.client.1.smithi190.stdout:3/382: readlink d1/de/d1c/d36/l4d 0 2024-03-26T18:54:30.543 INFO:tasks.workunit.client.1.smithi190.stdout:3/383: chown d1/d3/df/d31/f56 10235054 1 2024-03-26T18:54:30.543 INFO:tasks.workunit.client.1.smithi190.stdout:3/384: write d1/f2c [1734221,76132] 0 2024-03-26T18:54:30.543 INFO:tasks.workunit.client.1.smithi190.stdout:3/385: fdatasync d1/fb 0 2024-03-26T18:54:30.544 INFO:tasks.workunit.client.1.smithi190.stdout:1/342: symlink d3/l75 0 2024-03-26T18:54:30.545 INFO:tasks.workunit.client.1.smithi190.stdout:7/453: dread d3/d6/d9/d14/f33 [0,4194304] 0 2024-03-26T18:54:30.547 INFO:tasks.workunit.client.0.smithi102.stdout:7/49: symlink d7/d12/l13 0 2024-03-26T18:54:30.548 INFO:tasks.workunit.client.1.smithi190.stdout:6/406: dwrite d17/d28/d2c/d77/f66 [0,4194304] 0 2024-03-26T18:54:30.549 INFO:tasks.workunit.client.1.smithi190.stdout:1/343: mkdir d3/da/dd/d76 0 2024-03-26T18:54:30.549 INFO:tasks.workunit.client.1.smithi190.stdout:1/344: readlink d3/d9/l10 0 2024-03-26T18:54:30.550 INFO:tasks.workunit.client.1.smithi190.stdout:7/454: mknod d3/d6/d9/d1f/d41/d65/d90/d92/d9b/ca8 0 2024-03-26T18:54:30.553 INFO:tasks.workunit.client.0.smithi102.stdout:1/93: write f8 [1606661,9243] 0 2024-03-26T18:54:30.556 INFO:tasks.workunit.client.1.smithi190.stdout:1/345: mknod d3/da/dd/d2f/d5b/d5d/c77 0 2024-03-26T18:54:30.558 INFO:tasks.workunit.client.1.smithi190.stdout:1/346: creat d3/da/f78 x:0 0 0 2024-03-26T18:54:30.560 INFO:tasks.workunit.client.1.smithi190.stdout:6/407: write fc [309572,3711] 0 2024-03-26T18:54:30.561 INFO:tasks.workunit.client.1.smithi190.stdout:6/408: chown d17/d28/d2c/d50/f65 8646 1 2024-03-26T18:54:30.561 INFO:tasks.workunit.client.1.smithi190.stdout:1/347: unlink d3/da/dd/d2f/d34/f4c 0 2024-03-26T18:54:30.561 INFO:tasks.workunit.client.0.smithi102.stdout:8/62: dwrite d0/d3/fc [0,4194304] 0 2024-03-26T18:54:30.561 INFO:tasks.workunit.client.0.smithi102.stdout:8/63: fdatasync d0/d3/f6 0 2024-03-26T18:54:30.562 INFO:tasks.workunit.client.1.smithi190.stdout:1/348: write d3/da/dd/d2f/d34/f69 [179507,129210] 0 2024-03-26T18:54:30.562 INFO:tasks.workunit.client.1.smithi190.stdout:1/349: chown d3/d9/l10 78350 1 2024-03-26T18:54:30.564 INFO:tasks.workunit.client.1.smithi190.stdout:7/455: dread d3/d1b/d98/fa0 [0,4194304] 0 2024-03-26T18:54:30.567 INFO:tasks.workunit.client.1.smithi190.stdout:1/350: rename d3/d9/f13 to d3/da/dd/d2f/f79 0 2024-03-26T18:54:30.571 INFO:tasks.workunit.client.0.smithi102.stdout:7/50: dwrite d7/ff [0,4194304] 0 2024-03-26T18:54:30.574 INFO:tasks.workunit.client.0.smithi102.stdout:7/51: truncate d7/f9 1512534 0 2024-03-26T18:54:30.576 INFO:tasks.workunit.client.0.smithi102.stdout:7/52: creat d7/d10/f14 x:0 0 0 2024-03-26T18:54:30.584 INFO:tasks.workunit.client.1.smithi190.stdout:2/438: truncate d6/d9/d3a/d3d/d2e/d31/f91 983469 0 2024-03-26T18:54:30.585 INFO:tasks.workunit.client.1.smithi190.stdout:2/439: dread d6/d21/f78 [0,4194304] 0 2024-03-26T18:54:30.587 INFO:tasks.workunit.client.1.smithi190.stdout:2/440: mkdir d6/d9/d3a/d3d/d67/d9e 0 2024-03-26T18:54:30.587 INFO:tasks.workunit.client.1.smithi190.stdout:2/441: creat d6/d9/d3a/d3d/d55/f9f x:0 0 0 2024-03-26T18:54:30.589 INFO:tasks.workunit.client.1.smithi190.stdout:2/442: unlink d6/d9/d48/f63 0 2024-03-26T18:54:30.589 INFO:tasks.workunit.client.1.smithi190.stdout:2/443: chown d6/d9/d3a/d3d/d2e/d31/l88 371 1 2024-03-26T18:54:30.589 INFO:tasks.workunit.client.1.smithi190.stdout:7/456: dwrite d3/d6/d9/d14/d36/f6b [0,4194304] 0 2024-03-26T18:54:30.592 INFO:tasks.workunit.client.1.smithi190.stdout:1/351: dwrite d3/f29 [0,4194304] 0 2024-03-26T18:54:30.592 INFO:tasks.workunit.client.1.smithi190.stdout:1/352: truncate d3/da/dd/d2f/d34/d3e/f56 828574 0 2024-03-26T18:54:30.592 INFO:tasks.workunit.client.1.smithi190.stdout:1/353: chown d3/d9 1 1 2024-03-26T18:54:30.593 INFO:tasks.workunit.client.1.smithi190.stdout:2/444: mkdir d6/d9/d48/da0 0 2024-03-26T18:54:30.593 INFO:tasks.workunit.client.1.smithi190.stdout:7/457: write d3/f35 [2681094,73409] 0 2024-03-26T18:54:30.593 INFO:tasks.workunit.client.1.smithi190.stdout:7/458: read d3/d6/d9/d14/d36/f72 [247339,4397] 0 2024-03-26T18:54:30.594 INFO:tasks.workunit.client.0.smithi102.stdout:1/94: unlink cd 0 2024-03-26T18:54:30.594 INFO:tasks.workunit.client.0.smithi102.stdout:1/95: chown de/f15 99548223 1 2024-03-26T18:54:30.595 INFO:tasks.workunit.client.1.smithi190.stdout:2/445: creat d6/d9/d1a/d72/d8b/fa1 x:0 0 0 2024-03-26T18:54:30.595 INFO:tasks.workunit.client.1.smithi190.stdout:2/446: truncate d6/d9/d3a/d3d/d1c/f54 511157 0 2024-03-26T18:54:30.597 INFO:tasks.workunit.client.1.smithi190.stdout:7/459: mknod d3/d6/d9/d1f/d41/d50/d53/d8f/ca9 0 2024-03-26T18:54:30.597 INFO:tasks.workunit.client.1.smithi190.stdout:7/460: creat d3/d6/d9/d1f/d41/d50/d53/faa x:0 0 0 2024-03-26T18:54:30.597 INFO:tasks.workunit.client.1.smithi190.stdout:7/461: fdatasync d3/d6/d9/d14/d36/f70 0 2024-03-26T18:54:30.597 INFO:tasks.workunit.client.1.smithi190.stdout:1/354: dread d3/d9/f1c [0,4194304] 0 2024-03-26T18:54:30.597 INFO:tasks.workunit.client.1.smithi190.stdout:7/462: truncate d3/d6/d9/d1f/d41/d50/f69 101225 0 2024-03-26T18:54:30.598 INFO:tasks.workunit.client.1.smithi190.stdout:2/447: link d6/f7 d6/d9/d10/d5f/fa2 0 2024-03-26T18:54:30.598 INFO:tasks.workunit.client.1.smithi190.stdout:2/448: readlink d6/d9/d3a/d3d/d67/l6f 0 2024-03-26T18:54:30.598 INFO:tasks.workunit.client.1.smithi190.stdout:2/449: fsync d6/d9/d10/f70 0 2024-03-26T18:54:30.598 INFO:tasks.workunit.client.1.smithi190.stdout:2/450: fdatasync d6/d9/d10/f40 0 2024-03-26T18:54:30.599 INFO:tasks.workunit.client.1.smithi190.stdout:2/451: write d6/d9/d3a/d3d/d1c/f5e [4518895,74564] 0 2024-03-26T18:54:30.599 INFO:tasks.workunit.client.0.smithi102.stdout:7/53: getdents d7 0 2024-03-26T18:54:30.600 INFO:tasks.workunit.client.1.smithi190.stdout:1/355: mknod d3/da/d45/c7a 0 2024-03-26T18:54:30.600 INFO:tasks.workunit.client.1.smithi190.stdout:1/356: dread - d3/da/dd/d2f/d5b/f62 zero size 2024-03-26T18:54:30.600 INFO:tasks.workunit.client.1.smithi190.stdout:1/357: creat d3/da/dd/d2f/f7b x:0 0 0 2024-03-26T18:54:30.601 INFO:tasks.workunit.client.1.smithi190.stdout:7/463: rmdir d3/d6 39 2024-03-26T18:54:30.601 INFO:tasks.workunit.client.1.smithi190.stdout:7/464: write d3/d1b/d98/fa0 [85137,75469] 0 2024-03-26T18:54:30.603 INFO:tasks.workunit.client.1.smithi190.stdout:2/452: mkdir d6/d9/d3a/d3d/d67/d9e/da3 0 2024-03-26T18:54:30.603 INFO:tasks.workunit.client.1.smithi190.stdout:2/453: dread - d6/d9/d1a/d72/d8b/fa1 zero size 2024-03-26T18:54:30.605 INFO:tasks.workunit.client.1.smithi190.stdout:1/358: creat d3/da/dd/f7c x:0 0 0 2024-03-26T18:54:30.605 INFO:tasks.workunit.client.1.smithi190.stdout:1/359: chown d3/d9/db 3 1 2024-03-26T18:54:30.605 INFO:tasks.workunit.client.1.smithi190.stdout:1/360: write d3/d9/db/f23 [763294,16217] 0 2024-03-26T18:54:30.606 INFO:tasks.workunit.client.1.smithi190.stdout:7/465: mkdir d3/d6/d9/d1f/dab 0 2024-03-26T18:54:30.606 INFO:tasks.workunit.client.1.smithi190.stdout:7/466: stat d3/d6/d9/d1f/d4c/f5c 0 2024-03-26T18:54:30.607 INFO:tasks.workunit.client.1.smithi190.stdout:1/361: mknod d3/da/dd/d76/c7d 0 2024-03-26T18:54:30.608 INFO:tasks.workunit.client.1.smithi190.stdout:7/467: creat d3/d6/fac x:0 0 0 2024-03-26T18:54:30.608 INFO:tasks.workunit.client.1.smithi190.stdout:7/468: fsync d3/d6/d9/f24 0 2024-03-26T18:54:30.610 INFO:tasks.workunit.client.1.smithi190.stdout:7/469: symlink d3/lad 0 2024-03-26T18:54:30.614 INFO:tasks.workunit.client.0.smithi102.stdout:2/113: sync 2024-03-26T18:54:30.614 INFO:tasks.workunit.client.0.smithi102.stdout:6/102: sync 2024-03-26T18:54:30.614 INFO:tasks.workunit.client.0.smithi102.stdout:4/56: sync 2024-03-26T18:54:30.614 INFO:tasks.workunit.client.0.smithi102.stdout:6/103: write d2/f11 [942042,128766] 0 2024-03-26T18:54:30.614 INFO:tasks.workunit.client.0.smithi102.stdout:6/104: chown d2/l13 291 1 2024-03-26T18:54:30.614 INFO:tasks.workunit.client.0.smithi102.stdout:8/64: truncate d0/d3/fc 656249 0 2024-03-26T18:54:30.615 INFO:tasks.workunit.client.0.smithi102.stdout:4/57: getdents d1 0 2024-03-26T18:54:30.618 INFO:tasks.workunit.client.0.smithi102.stdout:6/105: rename d2/l17 to d2/d7/l1a 0 2024-03-26T18:54:30.618 INFO:tasks.workunit.client.0.smithi102.stdout:6/106: write d2/f11 [612033,65329] 0 2024-03-26T18:54:30.620 INFO:tasks.workunit.client.0.smithi102.stdout:4/58: unlink d1/f10 0 2024-03-26T18:54:30.622 INFO:tasks.workunit.client.0.smithi102.stdout:8/65: rename d0/d9/cb to d0/cf 0 2024-03-26T18:54:30.623 INFO:tasks.workunit.client.0.smithi102.stdout:4/59: write d1/f9 [1019656,55658] 0 2024-03-26T18:54:30.623 INFO:tasks.workunit.client.0.smithi102.stdout:2/114: write d3/d6/f10 [2386296,18201] 0 2024-03-26T18:54:30.623 INFO:tasks.workunit.client.0.smithi102.stdout:2/115: creat d3/d6/f24 x:0 0 0 2024-03-26T18:54:30.625 INFO:tasks.workunit.client.0.smithi102.stdout:6/107: dread d2/d3/f5 [0,4194304] 0 2024-03-26T18:54:30.625 INFO:tasks.workunit.client.0.smithi102.stdout:4/60: write d1/f11 [99703,47052] 0 2024-03-26T18:54:30.629 INFO:tasks.workunit.client.0.smithi102.stdout:2/116: mkdir d3/d6/d1a/d25 0 2024-03-26T18:54:30.629 INFO:tasks.workunit.client.0.smithi102.stdout:4/61: write d1/f7 [1525251,99987] 0 2024-03-26T18:54:30.631 INFO:tasks.workunit.client.0.smithi102.stdout:4/62: readlink d1/l6 0 2024-03-26T18:54:30.632 INFO:tasks.workunit.client.0.smithi102.stdout:4/63: chown d1/ff 0 1 2024-03-26T18:54:30.636 INFO:tasks.workunit.client.0.smithi102.stdout:4/64: truncate d1/fa 443624 0 2024-03-26T18:54:30.642 INFO:tasks.workunit.client.0.smithi102.stdout:4/65: mknod d1/c12 0 2024-03-26T18:54:30.643 INFO:tasks.workunit.client.1.smithi190.stdout:7/470: dwrite d3/d6/f5d [0,4194304] 0 2024-03-26T18:54:30.644 INFO:tasks.workunit.client.0.smithi102.stdout:4/66: rename d1/c12 to d1/c13 0 2024-03-26T18:54:30.647 INFO:tasks.workunit.client.1.smithi190.stdout:7/471: symlink d3/lae 0 2024-03-26T18:54:30.648 INFO:tasks.workunit.client.1.smithi190.stdout:2/454: dwrite d6/d9/d3a/d3d/d2e/f5c [0,4194304] 0 2024-03-26T18:54:30.648 INFO:tasks.workunit.client.0.smithi102.stdout:2/117: fdatasync d3/d6/f23 0 2024-03-26T18:54:30.649 INFO:tasks.workunit.client.0.smithi102.stdout:2/118: write d3/d6/f24 [189375,47420] 0 2024-03-26T18:54:30.649 INFO:tasks.workunit.client.0.smithi102.stdout:2/119: dread - d3/f19 zero size 2024-03-26T18:54:30.649 INFO:tasks.workunit.client.0.smithi102.stdout:2/120: readlink d3/d11/l1e 0 2024-03-26T18:54:30.649 INFO:tasks.workunit.client.1.smithi190.stdout:1/362: dwrite d3/da/dd/d2f/d34/f69 [0,4194304] 0 2024-03-26T18:54:30.651 INFO:tasks.workunit.client.0.smithi102.stdout:6/108: dwrite d2/d3/f5 [0,4194304] 0 2024-03-26T18:54:30.651 INFO:tasks.workunit.client.0.smithi102.stdout:2/121: mknod d3/d11/c26 0 2024-03-26T18:54:30.651 INFO:tasks.workunit.client.0.smithi102.stdout:2/122: stat d3/d6/f24 0 2024-03-26T18:54:30.655 INFO:tasks.workunit.client.1.smithi190.stdout:2/455: mknod d6/d9/d48/da0/ca4 0 2024-03-26T18:54:30.655 INFO:tasks.workunit.client.1.smithi190.stdout:2/456: creat d6/d9/d3a/d3d/d55/fa5 x:0 0 0 2024-03-26T18:54:30.656 INFO:tasks.workunit.client.0.smithi102.stdout:7/54: dread d7/f9 [0,4194304] 0 2024-03-26T18:54:30.659 INFO:tasks.workunit.client.1.smithi190.stdout:7/472: dread d3/d6/d9/d1f/d41/d50/f52 [0,4194304] 0 2024-03-26T18:54:30.663 INFO:tasks.workunit.client.0.smithi102.stdout:2/123: write d3/f8 [1709609,125411] 0 2024-03-26T18:54:30.665 INFO:tasks.workunit.client.0.smithi102.stdout:6/109: dread d2/d3/f5 [0,4194304] 0 2024-03-26T18:54:30.666 INFO:tasks.workunit.client.0.smithi102.stdout:7/55: rename d7/d12/l13 to d7/d12/l15 0 2024-03-26T18:54:30.666 INFO:tasks.workunit.client.0.smithi102.stdout:7/56: write f6 [4273997,8848] 0 2024-03-26T18:54:30.668 INFO:tasks.workunit.client.0.smithi102.stdout:8/66: rename d0/d3/fc to d0/d3/f10 0 2024-03-26T18:54:30.670 INFO:tasks.workunit.client.0.smithi102.stdout:6/110: read d2/d3/f5 [2746321,83577] 0 2024-03-26T18:54:30.674 INFO:tasks.workunit.client.0.smithi102.stdout:4/67: dwrite d1/f8 [0,4194304] 0 2024-03-26T18:54:30.675 INFO:tasks.workunit.client.0.smithi102.stdout:4/68: fdatasync d1/f8 0 2024-03-26T18:54:30.675 INFO:tasks.workunit.client.1.smithi190.stdout:1/363: dwrite d3/d9/f2a [0,4194304] 0 2024-03-26T18:54:30.675 INFO:tasks.workunit.client.1.smithi190.stdout:1/364: chown d3/d9/f57 15762 1 2024-03-26T18:54:30.675 INFO:tasks.workunit.client.1.smithi190.stdout:1/365: creat d3/da/dd/d2f/d34/d3e/d52/f7e x:0 0 0 2024-03-26T18:54:30.677 INFO:tasks.workunit.client.0.smithi102.stdout:2/124: mkdir d3/d11/d27 0 2024-03-26T18:54:30.679 INFO:tasks.workunit.client.0.smithi102.stdout:7/57: symlink d7/d12/l16 0 2024-03-26T18:54:30.679 INFO:tasks.workunit.client.0.smithi102.stdout:7/58: chown d7/d10 187092397 1 2024-03-26T18:54:30.681 INFO:tasks.workunit.client.0.smithi102.stdout:8/67: truncate d0/d3/f10 475152 0 2024-03-26T18:54:30.681 INFO:tasks.workunit.client.0.smithi102.stdout:8/68: chown d0/la 0 1 2024-03-26T18:54:30.683 INFO:tasks.workunit.client.0.smithi102.stdout:4/69: truncate d1/fa 213940 0 2024-03-26T18:54:30.683 INFO:tasks.workunit.client.0.smithi102.stdout:4/70: chown d1/f8 10 1 2024-03-26T18:54:30.684 INFO:tasks.workunit.client.1.smithi190.stdout:1/366: dread d3/d9/f16 [4194304,4194304] 0 2024-03-26T18:54:30.684 INFO:tasks.workunit.client.1.smithi190.stdout:1/367: dread - d3/da/dd/d2f/f5e zero size 2024-03-26T18:54:30.685 INFO:tasks.workunit.client.0.smithi102.stdout:2/125: creat d3/d11/d27/f28 x:0 0 0 2024-03-26T18:54:30.685 INFO:tasks.workunit.client.0.smithi102.stdout:2/126: write d3/f8 [641060,105429] 0 2024-03-26T18:54:30.687 INFO:tasks.workunit.client.0.smithi102.stdout:7/59: mkdir d7/d10/d17 0 2024-03-26T18:54:30.687 INFO:tasks.workunit.client.0.smithi102.stdout:7/60: write f4 [1219484,35520] 0 2024-03-26T18:54:30.689 INFO:tasks.workunit.client.0.smithi102.stdout:8/69: getdents d0/d9 0 2024-03-26T18:54:30.689 INFO:tasks.workunit.client.0.smithi102.stdout:8/70: chown d0/d3 273604 1 2024-03-26T18:54:30.689 INFO:tasks.workunit.client.0.smithi102.stdout:8/71: creat d0/d3/f11 x:0 0 0 2024-03-26T18:54:30.691 INFO:tasks.workunit.client.1.smithi190.stdout:2/457: link d6/d9/d48/da0/ca4 d6/d9/d1a/d72/ca6 0 2024-03-26T18:54:30.691 INFO:tasks.workunit.client.1.smithi190.stdout:2/458: dread - d6/d9/d3a/f94 zero size 2024-03-26T18:54:30.691 INFO:tasks.workunit.client.1.smithi190.stdout:2/459: stat d6/d9/d3a/d3d/d2e/d86/f8a 0 2024-03-26T18:54:30.691 INFO:tasks.workunit.client.1.smithi190.stdout:2/460: write d6/d9/d3a/d3d/d55/fa5 [751075,109617] 0 2024-03-26T18:54:30.692 INFO:tasks.workunit.client.0.smithi102.stdout:6/111: rmdir d2/d7 39 2024-03-26T18:54:30.692 INFO:tasks.workunit.client.1.smithi190.stdout:7/473: unlink d3/d6/f3e 0 2024-03-26T18:54:30.695 INFO:tasks.workunit.client.0.smithi102.stdout:7/61: symlink d7/d10/l18 0 2024-03-26T18:54:30.697 INFO:tasks.workunit.client.1.smithi190.stdout:1/368: unlink d3/da/dd/d2f/d34/d3e/d52/f6f 0 2024-03-26T18:54:30.697 INFO:tasks.workunit.client.1.smithi190.stdout:1/369: chown d3/d9/f15 1 1 2024-03-26T18:54:30.697 INFO:tasks.workunit.client.1.smithi190.stdout:1/370: dread - d3/da/d2e/f4a zero size 2024-03-26T18:54:30.699 INFO:tasks.workunit.client.1.smithi190.stdout:2/461: rename d6/d9/d45/d6b/f79 to d6/d9/d48/da0/fa7 0 2024-03-26T18:54:30.700 INFO:tasks.workunit.client.1.smithi190.stdout:2/462: creat d6/d9/d1a/d72/d8b/fa8 x:0 0 0 2024-03-26T18:54:30.700 INFO:tasks.workunit.client.1.smithi190.stdout:2/463: chown d6/d9/d3a/d3d/d55 1 1 2024-03-26T18:54:30.700 INFO:tasks.workunit.client.1.smithi190.stdout:2/464: creat d6/d9/d3a/d3d/d55/fa9 x:0 0 0 2024-03-26T18:54:30.700 INFO:tasks.workunit.client.1.smithi190.stdout:7/474: dread d3/f35 [0,4194304] 0 2024-03-26T18:54:30.700 INFO:tasks.workunit.client.0.smithi102.stdout:8/72: mkdir d0/d9/d12 0 2024-03-26T18:54:30.701 INFO:tasks.workunit.client.0.smithi102.stdout:6/112: unlink d2/f11 0 2024-03-26T18:54:30.703 INFO:tasks.workunit.client.0.smithi102.stdout:4/71: symlink d1/l14 0 2024-03-26T18:54:30.705 INFO:tasks.workunit.client.0.smithi102.stdout:2/127: dwrite d3/d6/fe [0,4194304] 0 2024-03-26T18:54:30.709 INFO:tasks.workunit.client.1.smithi190.stdout:2/465: unlink d6/d9/d3a/d3d/d55/f9f 0 2024-03-26T18:54:30.710 INFO:tasks.workunit.client.0.smithi102.stdout:2/128: dread d3/d6/f23 [0,4194304] 0 2024-03-26T18:54:30.713 INFO:tasks.workunit.client.0.smithi102.stdout:6/113: link d2/l13 d2/d7/l1b 0 2024-03-26T18:54:30.723 INFO:tasks.workunit.client.0.smithi102.stdout:8/73: mknod d0/d9/c13 0 2024-03-26T18:54:30.723 INFO:tasks.workunit.client.0.smithi102.stdout:8/74: creat d0/d3/f14 x:0 0 0 2024-03-26T18:54:30.723 INFO:tasks.workunit.client.0.smithi102.stdout:8/75: dread - d0/d3/f14 zero size 2024-03-26T18:54:30.723 INFO:tasks.workunit.client.0.smithi102.stdout:8/76: write d0/d3/f6 [5207576,77712] 0 2024-03-26T18:54:30.724 INFO:tasks.workunit.client.0.smithi102.stdout:4/72: symlink d1/l15 0 2024-03-26T18:54:30.724 INFO:tasks.workunit.client.1.smithi190.stdout:7/475: dwrite d3/d6/d9/d1f/d41/d50/d53/f58 [0,4194304] 0 2024-03-26T18:54:30.724 INFO:tasks.workunit.client.1.smithi190.stdout:7/476: readlink d3/d6/d9/d1f/l85 0 2024-03-26T18:54:30.728 INFO:tasks.workunit.client.0.smithi102.stdout:8/77: mkdir d0/d15 0 2024-03-26T18:54:30.728 INFO:tasks.workunit.client.0.smithi102.stdout:8/78: chown d0/c1 39 1 2024-03-26T18:54:30.733 INFO:tasks.workunit.client.0.smithi102.stdout:8/79: creat d0/d15/f16 x:0 0 0 2024-03-26T18:54:30.737 INFO:tasks.workunit.client.0.smithi102.stdout:6/114: getdents d2 0 2024-03-26T18:54:30.737 INFO:tasks.workunit.client.0.smithi102.stdout:6/115: fsync d2/d7/f8 0 2024-03-26T18:54:30.737 INFO:tasks.workunit.client.0.smithi102.stdout:6/116: write d2/d7/f8 [2670770,25203] 0 2024-03-26T18:54:30.738 INFO:tasks.workunit.client.0.smithi102.stdout:7/62: dwrite f6 [0,4194304] 0 2024-03-26T18:54:30.738 INFO:tasks.workunit.client.0.smithi102.stdout:7/63: chown d7/f9 2913458 1 2024-03-26T18:54:30.742 INFO:tasks.workunit.client.1.smithi190.stdout:2/466: dwrite d6/d9/d48/da0/fa7 [0,4194304] 0 2024-03-26T18:54:30.744 INFO:tasks.workunit.client.0.smithi102.stdout:5/364: sync 2024-03-26T18:54:30.744 INFO:tasks.workunit.client.0.smithi102.stdout:3/245: sync 2024-03-26T18:54:30.757 INFO:tasks.workunit.client.0.smithi102.stdout:8/80: rmdir d0/d3 39 2024-03-26T18:54:30.757 INFO:tasks.workunit.client.0.smithi102.stdout:8/81: chown d0/cf 40976 1 2024-03-26T18:54:30.762 INFO:tasks.workunit.client.1.smithi190.stdout:7/477: dwrite d3/d6/d9/d1f/d4c/f7f [0,4194304] 0 2024-03-26T18:54:30.762 INFO:tasks.workunit.client.1.smithi190.stdout:7/478: chown d3/d6/d9/d1f/d41/d50/d53/d8f/ca9 239947 1 2024-03-26T18:54:30.762 INFO:tasks.workunit.client.1.smithi190.stdout:7/479: dread - d3/d6/d9/d14/f7e zero size 2024-03-26T18:54:30.762 INFO:tasks.workunit.client.1.smithi190.stdout:7/480: chown d3/d6/d9 49101 1 2024-03-26T18:54:30.763 INFO:tasks.workunit.client.0.smithi102.stdout:2/129: dwrite d3/f4 [0,4194304] 0 2024-03-26T18:54:30.763 INFO:tasks.workunit.client.0.smithi102.stdout:2/130: truncate d3/f1c 841540 0 2024-03-26T18:54:30.763 INFO:tasks.workunit.client.0.smithi102.stdout:2/131: truncate d3/f7 423263 0 2024-03-26T18:54:30.764 INFO:tasks.workunit.client.0.smithi102.stdout:2/132: mkdir d3/d11/d15/d29 0 2024-03-26T18:54:30.777 INFO:tasks.workunit.client.0.smithi102.stdout:4/73: dwrite d1/fa [0,4194304] 0 2024-03-26T18:54:30.778 INFO:tasks.workunit.client.0.smithi102.stdout:8/82: truncate d0/d3/f6 3786005 0 2024-03-26T18:54:30.780 INFO:tasks.workunit.client.0.smithi102.stdout:4/74: link d1/c13 d1/c16 0 2024-03-26T18:54:30.781 INFO:tasks.workunit.client.0.smithi102.stdout:4/75: mkdir d1/d17 0 2024-03-26T18:54:30.782 INFO:tasks.workunit.client.0.smithi102.stdout:4/76: readlink d1/lc 0 2024-03-26T18:54:30.782 INFO:tasks.workunit.client.0.smithi102.stdout:4/77: mknod d1/d17/c18 0 2024-03-26T18:54:30.783 INFO:tasks.workunit.client.0.smithi102.stdout:4/78: symlink d1/l19 0 2024-03-26T18:54:30.785 INFO:tasks.workunit.client.0.smithi102.stdout:8/83: dread d0/d3/f10 [0,4194304] 0 2024-03-26T18:54:30.785 INFO:tasks.workunit.client.0.smithi102.stdout:8/84: chown d0 21 1 2024-03-26T18:54:30.785 INFO:tasks.workunit.client.0.smithi102.stdout:8/85: rmdir d0/d3 39 2024-03-26T18:54:30.788 INFO:tasks.workunit.client.0.smithi102.stdout:8/86: link d0/c5 d0/d9/c17 0 2024-03-26T18:54:30.790 INFO:tasks.workunit.client.0.smithi102.stdout:8/87: link d0/le d0/d9/d12/l18 0 2024-03-26T18:54:30.791 INFO:tasks.workunit.client.0.smithi102.stdout:8/88: mkdir d0/d9/d12/d19 0 2024-03-26T18:54:30.802 INFO:tasks.workunit.client.0.smithi102.stdout:7/64: dwrite f6 [0,4194304] 0 2024-03-26T18:54:30.802 INFO:tasks.workunit.client.0.smithi102.stdout:7/65: write f6 [5072412,75344] 0 2024-03-26T18:54:30.802 INFO:tasks.workunit.client.0.smithi102.stdout:7/66: write d7/ff [5086177,104896] 0 2024-03-26T18:54:30.802 INFO:tasks.workunit.client.0.smithi102.stdout:3/246: dwrite df/d29/f33 [0,4194304] 0 2024-03-26T18:54:30.802 INFO:tasks.workunit.client.0.smithi102.stdout:5/365: dwrite d2/d14/d1a/f5e [0,4194304] 0 2024-03-26T18:54:30.802 INFO:tasks.workunit.client.0.smithi102.stdout:6/117: dwrite d2/d7/f8 [4194304,4194304] 0 2024-03-26T18:54:30.803 INFO:tasks.workunit.client.1.smithi190.stdout:2/467: dwrite d6/f7 [4194304,4194304] 0 2024-03-26T18:54:30.803 INFO:tasks.workunit.client.1.smithi190.stdout:7/481: dwrite d3/d6/d9/d1f/d84/d99/fa3 [0,4194304] 0 2024-03-26T18:54:30.804 INFO:tasks.workunit.client.0.smithi102.stdout:2/133: dwrite d3/f7 [0,4194304] 0 2024-03-26T18:54:30.804 INFO:tasks.workunit.client.0.smithi102.stdout:2/134: write d3/f1c [1829708,18992] 0 2024-03-26T18:54:30.804 INFO:tasks.workunit.client.0.smithi102.stdout:4/79: dwrite d1/ff [0,4194304] 0 2024-03-26T18:54:30.804 INFO:tasks.workunit.client.0.smithi102.stdout:4/80: write d1/f8 [4370715,112855] 0 2024-03-26T18:54:30.805 INFO:tasks.workunit.client.0.smithi102.stdout:3/247: dread df/d29/d31/d36/d3c/f49 [0,4194304] 0 2024-03-26T18:54:30.807 INFO:tasks.workunit.client.0.smithi102.stdout:6/118: dread d2/d3/f5 [0,4194304] 0 2024-03-26T18:54:30.807 INFO:tasks.workunit.client.0.smithi102.stdout:6/119: chown d2/df/l19 1 1 2024-03-26T18:54:30.807 INFO:tasks.workunit.client.0.smithi102.stdout:6/120: chown d2/df/l19 0 1 2024-03-26T18:54:30.808 INFO:tasks.workunit.client.0.smithi102.stdout:7/67: symlink d7/d10/d17/l19 0 2024-03-26T18:54:30.813 INFO:tasks.workunit.client.1.smithi190.stdout:7/482: creat d3/d6/d9/d1f/d41/d65/d90/d92/d9b/faf x:0 0 0 2024-03-26T18:54:30.816 INFO:tasks.workunit.client.1.smithi190.stdout:2/468: getdents d6/d9/d45/d6b 0 2024-03-26T18:54:30.817 INFO:tasks.workunit.client.1.smithi190.stdout:2/469: readlink d6/d9/d3a/d3d/d2e/l84 0 2024-03-26T18:54:30.818 INFO:tasks.workunit.client.0.smithi102.stdout:8/89: dread d0/d3/f6 [0,4194304] 0 2024-03-26T18:54:30.824 INFO:tasks.workunit.client.0.smithi102.stdout:2/135: mkdir d3/d2a 0 2024-03-26T18:54:30.824 INFO:tasks.workunit.client.0.smithi102.stdout:2/136: creat d3/d6/f2b x:0 0 0 2024-03-26T18:54:30.829 INFO:tasks.workunit.client.1.smithi190.stdout:9/663: sync 2024-03-26T18:54:30.829 INFO:tasks.workunit.client.1.smithi190.stdout:5/315: sync 2024-03-26T18:54:30.829 INFO:tasks.workunit.client.1.smithi190.stdout:0/541: sync 2024-03-26T18:54:30.829 INFO:tasks.workunit.client.1.smithi190.stdout:6/409: sync 2024-03-26T18:54:30.829 INFO:tasks.workunit.client.1.smithi190.stdout:8/456: sync 2024-03-26T18:54:30.829 INFO:tasks.workunit.client.1.smithi190.stdout:6/410: write d17/d28/d46/f7f [573228,33817] 0 2024-03-26T18:54:30.829 INFO:tasks.workunit.client.1.smithi190.stdout:8/457: readlink d9/dd/l28 0 2024-03-26T18:54:30.830 INFO:tasks.workunit.client.0.smithi102.stdout:4/81: getdents d1/d17 0 2024-03-26T18:54:30.830 INFO:tasks.workunit.client.0.smithi102.stdout:4/82: readlink d1/l19 0 2024-03-26T18:54:30.831 INFO:tasks.workunit.client.1.smithi190.stdout:4/451: sync 2024-03-26T18:54:30.831 INFO:tasks.workunit.client.1.smithi190.stdout:4/452: chown d7/d8/d1c/d20/f3f 406631753 1 2024-03-26T18:54:30.831 INFO:tasks.workunit.client.1.smithi190.stdout:1/371: sync 2024-03-26T18:54:30.831 INFO:tasks.workunit.client.1.smithi190.stdout:3/386: sync 2024-03-26T18:54:30.831 INFO:tasks.workunit.client.0.smithi102.stdout:3/248: mknod df/d29/d3a/d47/c5a 0 2024-03-26T18:54:30.832 INFO:tasks.workunit.client.0.smithi102.stdout:6/121: mknod d2/c1c 0 2024-03-26T18:54:30.833 INFO:tasks.workunit.client.0.smithi102.stdout:0/112: sync 2024-03-26T18:54:30.834 INFO:tasks.workunit.client.0.smithi102.stdout:9/75: sync 2024-03-26T18:54:30.834 INFO:tasks.workunit.client.0.smithi102.stdout:9/76: truncate df/f13 808211 0 2024-03-26T18:54:30.835 INFO:tasks.workunit.client.0.smithi102.stdout:3/249: write df/f1a [1060844,64826] 0 2024-03-26T18:54:30.835 INFO:tasks.workunit.client.0.smithi102.stdout:3/250: creat df/d29/f5b x:0 0 0 2024-03-26T18:54:30.835 INFO:tasks.workunit.client.0.smithi102.stdout:3/251: stat df/d29/d3a 0 2024-03-26T18:54:30.835 INFO:tasks.workunit.client.1.smithi190.stdout:6/411: symlink d17/d28/d74/l92 0 2024-03-26T18:54:30.835 INFO:tasks.workunit.client.1.smithi190.stdout:6/412: readlink d17/d28/d2b/l3f 0 2024-03-26T18:54:30.835 INFO:tasks.workunit.client.1.smithi190.stdout:6/413: chown f8 213423 1 2024-03-26T18:54:30.837 INFO:tasks.workunit.client.0.smithi102.stdout:2/137: mknod d3/d20/c2c 0 2024-03-26T18:54:30.838 INFO:tasks.workunit.client.1.smithi190.stdout:8/458: creat d9/d16/d73/f9f x:0 0 0 2024-03-26T18:54:30.839 INFO:tasks.workunit.client.1.smithi190.stdout:4/453: symlink d7/d8/d14/d23/d28/d38/l9d 0 2024-03-26T18:54:30.839 INFO:tasks.workunit.client.1.smithi190.stdout:4/454: chown d7/d8/d1c/f39 2 1 2024-03-26T18:54:30.841 INFO:tasks.workunit.client.0.smithi102.stdout:0/113: rename d6/de/d13/f1e to d6/de/d13/d1f/f27 0 2024-03-26T18:54:30.843 INFO:tasks.workunit.client.1.smithi190.stdout:3/387: rename d1/d3/df/d31/d5b/d64 to d1/d3/d5a/d6e 0 2024-03-26T18:54:30.845 INFO:tasks.workunit.client.1.smithi190.stdout:0/542: dread d0/dd/d24/f2b [0,4194304] 0 2024-03-26T18:54:30.846 INFO:tasks.workunit.client.1.smithi190.stdout:0/543: creat d0/dd/d24/d87/fc1 x:0 0 0 2024-03-26T18:54:30.848 INFO:tasks.workunit.client.0.smithi102.stdout:4/83: dread d1/ff [0,4194304] 0 2024-03-26T18:54:30.849 INFO:tasks.workunit.client.1.smithi190.stdout:8/459: mknod d9/d39/d9c/ca0 0 2024-03-26T18:54:30.849 INFO:tasks.workunit.client.1.smithi190.stdout:4/455: mkdir d7/d8/d14/d23/d28/d9e 0 2024-03-26T18:54:30.849 INFO:tasks.workunit.client.1.smithi190.stdout:9/664: dread d12/d2d/f47 [0,4194304] 0 2024-03-26T18:54:30.850 INFO:tasks.workunit.client.1.smithi190.stdout:3/388: unlink d1/de/d1c/l46 0 2024-03-26T18:54:30.851 INFO:tasks.workunit.client.0.smithi102.stdout:4/84: mkdir d1/d17/d1a 0 2024-03-26T18:54:30.852 INFO:tasks.workunit.client.0.smithi102.stdout:3/252: read f7 [3792013,37687] 0 2024-03-26T18:54:30.853 INFO:tasks.workunit.client.1.smithi190.stdout:0/544: symlink d0/dd/d20/d1c/d46/da7/lc2 0 2024-03-26T18:54:30.853 INFO:tasks.workunit.client.1.smithi190.stdout:0/545: readlink d0/dd/d20/d1c/d8b/l31 0 2024-03-26T18:54:30.853 INFO:tasks.workunit.client.1.smithi190.stdout:0/546: stat d0/dd/d20/d1c/d4e/f9d 0 2024-03-26T18:54:30.853 INFO:tasks.workunit.client.0.smithi102.stdout:6/122: dread d2/d7/f8 [4194304,4194304] 0 2024-03-26T18:54:30.854 INFO:tasks.workunit.client.0.smithi102.stdout:4/85: rename d1/l19 to d1/d17/l1b 0 2024-03-26T18:54:30.855 INFO:tasks.workunit.client.0.smithi102.stdout:4/86: chown d1/l15 3 1 2024-03-26T18:54:30.855 INFO:tasks.workunit.client.0.smithi102.stdout:4/87: fdatasync d1/f11 0 2024-03-26T18:54:30.855 INFO:tasks.workunit.client.1.smithi190.stdout:8/460: stat d9/dd/l47 0 2024-03-26T18:54:30.855 INFO:tasks.workunit.client.0.smithi102.stdout:3/253: mknod df/d1c/c5c 0 2024-03-26T18:54:30.856 INFO:tasks.workunit.client.1.smithi190.stdout:4/456: read d7/d8/d14/f19 [1859694,71463] 0 2024-03-26T18:54:30.857 INFO:tasks.workunit.client.0.smithi102.stdout:6/123: creat d2/d7/f1d x:0 0 0 2024-03-26T18:54:30.858 INFO:tasks.workunit.client.1.smithi190.stdout:9/665: mknod d12/d2d/d1d/d23/d83/ce9 0 2024-03-26T18:54:30.858 INFO:tasks.workunit.client.1.smithi190.stdout:9/666: dread - d12/d2d/d1d/d23/f8f zero size 2024-03-26T18:54:30.859 INFO:tasks.workunit.client.0.smithi102.stdout:2/138: dread d3/d6/fe [0,4194304] 0 2024-03-26T18:54:30.859 INFO:tasks.workunit.client.0.smithi102.stdout:4/88: creat d1/d17/d1a/f1c x:0 0 0 2024-03-26T18:54:30.859 INFO:tasks.workunit.client.0.smithi102.stdout:4/89: chown d1/f8 10927 1 2024-03-26T18:54:30.859 INFO:tasks.workunit.client.0.smithi102.stdout:2/139: write d3/d6/f2b [491195,45258] 0 2024-03-26T18:54:30.859 INFO:tasks.workunit.client.0.smithi102.stdout:2/140: dread - d3/d11/d27/f28 zero size 2024-03-26T18:54:30.859 INFO:tasks.workunit.client.0.smithi102.stdout:2/141: chown d3/d11/c26 166 1 2024-03-26T18:54:30.860 INFO:tasks.workunit.client.1.smithi190.stdout:0/547: dread d0/dd/d20/d1c/f28 [0,4194304] 0 2024-03-26T18:54:30.861 INFO:tasks.workunit.client.1.smithi190.stdout:4/457: creat d7/d8/d1c/f9f x:0 0 0 2024-03-26T18:54:30.861 INFO:tasks.workunit.client.1.smithi190.stdout:9/667: truncate d12/d2d/d1d/d23/d2b/f68 286394 0 2024-03-26T18:54:30.861 INFO:tasks.workunit.client.1.smithi190.stdout:9/668: chown d12/d2d/d1d/d23/d6a/c79 2953148 1 2024-03-26T18:54:30.863 INFO:tasks.workunit.client.1.smithi190.stdout:0/548: creat d0/dd/d20/d1c/d8b/fc3 x:0 0 0 2024-03-26T18:54:30.863 INFO:tasks.workunit.client.1.smithi190.stdout:0/549: write d0/dd/d24/d4b/d5e/d81/f9b [657556,24993] 0 2024-03-26T18:54:30.864 INFO:tasks.workunit.client.1.smithi190.stdout:9/669: mknod d12/d80/cea 0 2024-03-26T18:54:30.867 INFO:tasks.workunit.client.1.smithi190.stdout:9/670: truncate d12/d2d/d1d/d23/d83/dc9/dd4/f73 1798974 0 2024-03-26T18:54:30.867 INFO:tasks.workunit.client.0.smithi102.stdout:6/124: write d2/d7/f8 [5039846,81493] 0 2024-03-26T18:54:30.867 INFO:tasks.workunit.client.0.smithi102.stdout:6/125: chown d2 1413 1 2024-03-26T18:54:30.869 INFO:tasks.workunit.client.1.smithi190.stdout:9/671: mkdir d12/d2d/d1d/d54/d92/deb 0 2024-03-26T18:54:30.869 INFO:tasks.workunit.client.1.smithi190.stdout:9/672: truncate d12/d2d/d1d/d23/d83/dc9/dd4/d3f/d55/f62 5159994 0 2024-03-26T18:54:30.869 INFO:tasks.workunit.client.1.smithi190.stdout:9/673: truncate d12/d2d/d1d/d23/d83/dc9/dd4/d3f/fbb 671084 0 2024-03-26T18:54:30.870 INFO:tasks.workunit.client.1.smithi190.stdout:9/674: getdents d12/d2d/d1d/da1 0 2024-03-26T18:54:30.872 INFO:tasks.workunit.client.0.smithi102.stdout:8/90: dwrite d0/d3/f14 [0,4194304] 0 2024-03-26T18:54:30.873 INFO:tasks.workunit.client.1.smithi190.stdout:9/675: symlink d12/d2d/d1d/d54/d92/lec 0 2024-03-26T18:54:30.873 INFO:tasks.workunit.client.0.smithi102.stdout:0/114: dread d6/fd [0,4194304] 0 2024-03-26T18:54:30.874 INFO:tasks.workunit.client.0.smithi102.stdout:8/91: symlink d0/d9/d12/d19/l1a 0 2024-03-26T18:54:30.876 INFO:tasks.workunit.client.1.smithi190.stdout:9/676: rename d12/d2d/d1d/d23/d83/dc9/dd4/d3f/cc2 to d12/d2d/d1d/d23/d6a/ced 0 2024-03-26T18:54:30.876 INFO:tasks.workunit.client.1.smithi190.stdout:4/458: write d7/d8/d14/f19 [7519406,82971] 0 2024-03-26T18:54:30.876 INFO:tasks.workunit.client.1.smithi190.stdout:8/461: write f6 [2966817,78719] 0 2024-03-26T18:54:30.876 INFO:tasks.workunit.client.1.smithi190.stdout:4/459: fdatasync d7/d8/d14/d23/d28/d38/f71 0 2024-03-26T18:54:30.876 INFO:tasks.workunit.client.1.smithi190.stdout:4/460: truncate d7/d8/d1c/d20/f47 969313 0 2024-03-26T18:54:30.877 INFO:tasks.workunit.client.1.smithi190.stdout:7/483: dwrite d3/f16 [0,4194304] 0 2024-03-26T18:54:30.877 INFO:tasks.workunit.client.0.smithi102.stdout:0/115: link d6/f1a d6/de/f28 0 2024-03-26T18:54:30.877 INFO:tasks.workunit.client.0.smithi102.stdout:0/116: fdatasync d6/de/d13/d1f/f27 0 2024-03-26T18:54:30.877 INFO:tasks.workunit.client.0.smithi102.stdout:1/96: sync 2024-03-26T18:54:30.878 INFO:tasks.workunit.client.0.smithi102.stdout:8/92: stat d0/cf 0 2024-03-26T18:54:30.878 INFO:tasks.workunit.client.1.smithi190.stdout:6/414: dwrite f9 [8388608,4194304] 0 2024-03-26T18:54:30.879 INFO:tasks.workunit.client.1.smithi190.stdout:6/415: write d17/d28/d2c/d77/f8d [815624,126721] 0 2024-03-26T18:54:30.879 INFO:tasks.workunit.client.0.smithi102.stdout:7/68: dwrite f4 [0,4194304] 0 2024-03-26T18:54:30.879 INFO:tasks.workunit.client.0.smithi102.stdout:5/366: dwrite d2/d2c/d38/d43/f4e [0,4194304] 0 2024-03-26T18:54:30.880 INFO:tasks.workunit.client.1.smithi190.stdout:1/372: dwrite d3/da/d2e/f4a [0,4194304] 0 2024-03-26T18:54:30.880 INFO:tasks.workunit.client.0.smithi102.stdout:0/117: unlink d6/de/d13/d1f/d20/c24 0 2024-03-26T18:54:30.880 INFO:tasks.workunit.client.0.smithi102.stdout:0/118: write d6/f18 [3198859,104382] 0 2024-03-26T18:54:30.883 INFO:tasks.workunit.client.0.smithi102.stdout:8/93: mkdir d0/d15/d1b 0 2024-03-26T18:54:30.884 INFO:tasks.workunit.client.1.smithi190.stdout:9/677: symlink d12/d6d/lee 0 2024-03-26T18:54:30.887 INFO:tasks.workunit.client.1.smithi190.stdout:8/462: truncate d9/d39/d79/f7e 3778046 0 2024-03-26T18:54:30.887 INFO:tasks.workunit.client.1.smithi190.stdout:8/463: dread - d9/d16/d73/f9f zero size 2024-03-26T18:54:30.888 INFO:tasks.workunit.client.0.smithi102.stdout:6/126: dread d2/d7/f8 [4194304,4194304] 0 2024-03-26T18:54:30.888 INFO:tasks.workunit.client.0.smithi102.stdout:6/127: fdatasync d2/d7/f8 0 2024-03-26T18:54:30.889 INFO:tasks.workunit.client.0.smithi102.stdout:6/128: readlink d2/d3/l6 0 2024-03-26T18:54:30.889 INFO:tasks.workunit.client.0.smithi102.stdout:7/69: creat d7/d10/d17/f1a x:0 0 0 2024-03-26T18:54:30.891 INFO:tasks.workunit.client.1.smithi190.stdout:7/484: truncate d3/f29 802211 0 2024-03-26T18:54:30.891 INFO:tasks.workunit.client.1.smithi190.stdout:7/485: creat d3/d6/d9/d1f/d41/d65/d90/d92/d9b/fb0 x:0 0 0 2024-03-26T18:54:30.893 INFO:tasks.workunit.client.0.smithi102.stdout:5/367: mkdir d2/d25/d2a/d85 0 2024-03-26T18:54:30.895 INFO:tasks.workunit.client.1.smithi190.stdout:6/416: symlink d17/d28/d38/d79/l93 0 2024-03-26T18:54:30.897 INFO:tasks.workunit.client.0.smithi102.stdout:1/97: rename c6 to de/d16/c24 0 2024-03-26T18:54:30.898 INFO:tasks.workunit.client.0.smithi102.stdout:0/119: dread d6/de/d19/f1c [0,4194304] 0 2024-03-26T18:54:30.898 INFO:tasks.workunit.client.0.smithi102.stdout:0/120: fsync d6/de/d13/d1f/f27 0 2024-03-26T18:54:30.899 INFO:tasks.workunit.client.0.smithi102.stdout:8/94: truncate d0/d3/f6 544588 0 2024-03-26T18:54:30.901 INFO:tasks.workunit.client.0.smithi102.stdout:6/129: symlink d2/d7/l1e 0 2024-03-26T18:54:30.903 INFO:tasks.workunit.client.1.smithi190.stdout:1/373: creat d3/f7f x:0 0 0 2024-03-26T18:54:30.906 INFO:tasks.workunit.client.0.smithi102.stdout:7/70: dread d7/f9 [0,4194304] 0 2024-03-26T18:54:30.906 INFO:tasks.workunit.client.0.smithi102.stdout:7/71: creat d7/d10/d17/f1b x:0 0 0 2024-03-26T18:54:30.908 INFO:tasks.workunit.client.0.smithi102.stdout:5/368: rename d2/d14/l7c to d2/d25/d2a/l86 0 2024-03-26T18:54:30.908 INFO:tasks.workunit.client.0.smithi102.stdout:5/369: read d2/d25/d5b/d72/f22 [472261,60673] 0 2024-03-26T18:54:30.910 INFO:tasks.workunit.client.0.smithi102.stdout:1/98: write fc [155020,76741] 0 2024-03-26T18:54:30.910 INFO:tasks.workunit.client.0.smithi102.stdout:1/99: chown de/l18 954118 1 2024-03-26T18:54:30.910 INFO:tasks.workunit.client.0.smithi102.stdout:1/100: write de/f19 [1304229,97236] 0 2024-03-26T18:54:30.915 INFO:tasks.workunit.client.0.smithi102.stdout:0/121: stat d6/de/d13/c17 0 2024-03-26T18:54:30.915 INFO:tasks.workunit.client.0.smithi102.stdout:4/90: dwrite d1/f9 [0,4194304] 0 2024-03-26T18:54:30.916 INFO:tasks.workunit.client.0.smithi102.stdout:7/72: rmdir d7 39 2024-03-26T18:54:30.918 INFO:tasks.workunit.client.1.smithi190.stdout:9/678: mknod d12/d2d/d1d/d54/d92/deb/cef 0 2024-03-26T18:54:30.918 INFO:tasks.workunit.client.1.smithi190.stdout:9/679: truncate d12/d2d/d1d/da1/dba/dbc/fda 884755 0 2024-03-26T18:54:30.918 INFO:tasks.workunit.client.0.smithi102.stdout:5/370: link d2/f6a d2/d2c/d38/f87 0 2024-03-26T18:54:30.919 INFO:tasks.workunit.client.1.smithi190.stdout:4/461: getdents d7/d1d/d42/d4a 0 2024-03-26T18:54:30.922 INFO:tasks.workunit.client.1.smithi190.stdout:9/680: dread d12/d2d/d1d/d23/d2b/fab [0,4194304] 0 2024-03-26T18:54:30.922 INFO:tasks.workunit.client.1.smithi190.stdout:9/681: write d12/d2d/d1d/fd3 [245166,60340] 0 2024-03-26T18:54:30.922 INFO:tasks.workunit.client.1.smithi190.stdout:9/682: chown d12/d2d/d1d/d23/d2b/f6e 246259145 1 2024-03-26T18:54:30.923 INFO:tasks.workunit.client.1.smithi190.stdout:9/683: truncate d12/d2d/d1d/d23/d2b/f6e 1034514 0 2024-03-26T18:54:30.923 INFO:tasks.workunit.client.1.smithi190.stdout:7/486: creat d3/d6/d9/d1f/d41/d50/d53/d8f/fb1 x:0 0 0 2024-03-26T18:54:30.924 INFO:tasks.workunit.client.0.smithi102.stdout:0/122: link d6/de/d13/d1f/f27 d6/de/d13/d1f/d20/f29 0 2024-03-26T18:54:30.924 INFO:tasks.workunit.client.0.smithi102.stdout:0/123: truncate d6/f11 4489573 0 2024-03-26T18:54:30.924 INFO:tasks.workunit.client.1.smithi190.stdout:6/417: mkdir d17/d28/d2c/d77/d94 0 2024-03-26T18:54:30.926 INFO:tasks.workunit.client.1.smithi190.stdout:1/374: symlink d3/da/dd/l80 0 2024-03-26T18:54:30.926 INFO:tasks.workunit.client.1.smithi190.stdout:1/375: read - d3/da/dd/d2f/d5b/f63 zero size 2024-03-26T18:54:30.928 INFO:tasks.workunit.client.1.smithi190.stdout:7/487: mknod d3/d6/d9/d1f/d41/d65/d90/d92/cb2 0 2024-03-26T18:54:30.929 INFO:tasks.workunit.client.0.smithi102.stdout:7/73: unlink d7/d12/l15 0 2024-03-26T18:54:30.929 INFO:tasks.workunit.client.0.smithi102.stdout:7/74: write d7/d10/d17/f1a [990705,89647] 0 2024-03-26T18:54:30.930 INFO:tasks.workunit.client.0.smithi102.stdout:3/254: dwrite df/d29/d31/d36/f41 [0,4194304] 0 2024-03-26T18:54:30.930 INFO:tasks.workunit.client.1.smithi190.stdout:6/418: truncate f2 975505 0 2024-03-26T18:54:30.931 INFO:tasks.workunit.client.0.smithi102.stdout:6/130: dwrite d2/d7/f8 [4194304,4194304] 0 2024-03-26T18:54:30.931 INFO:tasks.workunit.client.1.smithi190.stdout:8/464: dwrite d9/d16/f63 [0,4194304] 0 2024-03-26T18:54:30.931 INFO:tasks.workunit.client.1.smithi190.stdout:8/465: truncate d9/dd/d15/f1c 791375 0 2024-03-26T18:54:30.932 INFO:tasks.workunit.client.0.smithi102.stdout:2/142: dwrite d3/d6/f1d [0,4194304] 0 2024-03-26T18:54:30.932 INFO:tasks.workunit.client.0.smithi102.stdout:8/95: dwrite d0/d15/f16 [0,4194304] 0 2024-03-26T18:54:30.932 INFO:tasks.workunit.client.0.smithi102.stdout:4/91: mknod d1/c1d 0 2024-03-26T18:54:30.932 INFO:tasks.workunit.client.0.smithi102.stdout:1/101: fdatasync de/f19 0 2024-03-26T18:54:30.934 INFO:tasks.workunit.client.1.smithi190.stdout:1/376: symlink d3/da/dd/d2f/d34/d3e/d52/l81 0 2024-03-26T18:54:30.937 INFO:tasks.workunit.client.1.smithi190.stdout:4/462: rmdir d7/d1d/d5e 39 2024-03-26T18:54:30.940 INFO:tasks.workunit.client.1.smithi190.stdout:7/488: symlink d3/d1b/d98/lb3 0 2024-03-26T18:54:30.940 INFO:tasks.workunit.client.1.smithi190.stdout:7/489: stat d3/d6/d9/d14/d59/f9f 0 2024-03-26T18:54:30.941 INFO:tasks.workunit.client.1.smithi190.stdout:7/490: write d3/d6/f56 [1481393,95212] 0 2024-03-26T18:54:30.944 INFO:tasks.workunit.client.1.smithi190.stdout:6/419: mkdir d17/d28/d8b/d95 0 2024-03-26T18:54:30.944 INFO:tasks.workunit.client.1.smithi190.stdout:6/420: fsync d17/d28/d46/f7f 0 2024-03-26T18:54:30.946 INFO:tasks.workunit.client.1.smithi190.stdout:8/466: read d9/dd/d15/f2f [2345851,95475] 0 2024-03-26T18:54:30.946 INFO:tasks.workunit.client.1.smithi190.stdout:8/467: write d9/dd/d23/f27 [1979703,9129] 0 2024-03-26T18:54:30.948 INFO:tasks.workunit.client.1.smithi190.stdout:1/377: symlink d3/da/dd/d2f/d34/d3e/l82 0 2024-03-26T18:54:30.948 INFO:tasks.workunit.client.1.smithi190.stdout:1/378: fsync d3/da/d2e/d59/f65 0 2024-03-26T18:54:30.948 INFO:tasks.workunit.client.1.smithi190.stdout:1/379: dread - d3/da/dd/f7c zero size 2024-03-26T18:54:30.948 INFO:tasks.workunit.client.1.smithi190.stdout:1/380: readlink d3/d9/l17 0 2024-03-26T18:54:30.949 INFO:tasks.workunit.client.0.smithi102.stdout:6/131: link d2/d7/le d2/l1f 0 2024-03-26T18:54:30.952 INFO:tasks.workunit.client.0.smithi102.stdout:2/143: rename d3/d11/d15/d29 to d3/d2d 0 2024-03-26T18:54:30.952 INFO:tasks.workunit.client.0.smithi102.stdout:2/144: chown d3/fb 2885640 1 2024-03-26T18:54:30.953 INFO:tasks.workunit.client.1.smithi190.stdout:6/421: symlink d17/d28/d38/d79/l96 0 2024-03-26T18:54:30.955 INFO:tasks.workunit.client.1.smithi190.stdout:8/468: symlink d9/d39/la1 0 2024-03-26T18:54:30.957 INFO:tasks.workunit.client.0.smithi102.stdout:5/371: dwrite d2/f7 [0,4194304] 0 2024-03-26T18:54:30.958 INFO:tasks.workunit.client.0.smithi102.stdout:5/372: chown d2/d25/d2a/d4f/f52 217885247 1 2024-03-26T18:54:30.958 INFO:tasks.workunit.client.0.smithi102.stdout:8/96: dread d0/d15/f16 [0,4194304] 0 2024-03-26T18:54:30.960 INFO:tasks.workunit.client.1.smithi190.stdout:6/422: mkdir d17/d28/d74/d97 0 2024-03-26T18:54:30.963 INFO:tasks.workunit.client.0.smithi102.stdout:1/102: symlink de/d1f/l25 0 2024-03-26T18:54:30.963 INFO:tasks.workunit.client.0.smithi102.stdout:1/103: creat de/f26 x:0 0 0 2024-03-26T18:54:30.964 INFO:tasks.workunit.client.1.smithi190.stdout:1/381: rmdir d3/d5f 39 2024-03-26T18:54:30.970 INFO:tasks.workunit.client.1.smithi190.stdout:6/423: symlink d17/d28/d2c/d77/l98 0 2024-03-26T18:54:30.973 INFO:tasks.workunit.client.1.smithi190.stdout:1/382: mkdir d3/da/dd/d2f/d5b/d83 0 2024-03-26T18:54:30.973 INFO:tasks.workunit.client.1.smithi190.stdout:1/383: readlink d3/da/dd/d2f/d34/d3e/d52/l6d 0 2024-03-26T18:54:30.978 INFO:tasks.workunit.client.0.smithi102.stdout:3/255: rmdir df/d4d 0 2024-03-26T18:54:30.978 INFO:tasks.workunit.client.0.smithi102.stdout:3/256: readlink df/d1c/l4a 0 2024-03-26T18:54:30.980 INFO:tasks.workunit.client.0.smithi102.stdout:2/145: creat d3/d11/d15/f2e x:0 0 0 2024-03-26T18:54:30.983 INFO:tasks.workunit.client.0.smithi102.stdout:5/373: symlink d2/l88 0 2024-03-26T18:54:30.984 INFO:tasks.workunit.client.0.smithi102.stdout:5/374: creat d2/d2c/d38/f89 x:0 0 0 2024-03-26T18:54:30.984 INFO:tasks.workunit.client.0.smithi102.stdout:5/375: chown d2/d14/d1a/f77 0 1 2024-03-26T18:54:30.984 INFO:tasks.workunit.client.0.smithi102.stdout:5/376: dread - d2/d2c/d6f/f7a zero size 2024-03-26T18:54:30.985 INFO:tasks.workunit.client.0.smithi102.stdout:0/124: dwrite d6/f1a [0,4194304] 0 2024-03-26T18:54:30.985 INFO:tasks.workunit.client.0.smithi102.stdout:4/92: symlink d1/l1e 0 2024-03-26T18:54:30.987 INFO:tasks.workunit.client.0.smithi102.stdout:1/104: unlink de/d16/f17 0 2024-03-26T18:54:30.988 INFO:tasks.workunit.client.1.smithi190.stdout:1/384: write d3/da/dd/d2f/f79 [2454174,98532] 0 2024-03-26T18:54:30.988 INFO:tasks.workunit.client.0.smithi102.stdout:3/257: mknod df/d29/d31/d36/d45/c5d 0 2024-03-26T18:54:30.990 INFO:tasks.workunit.client.0.smithi102.stdout:2/146: rename d3/f7 to d3/d2a/f2f 0 2024-03-26T18:54:30.991 INFO:tasks.workunit.client.1.smithi190.stdout:1/385: symlink d3/d73/l84 0 2024-03-26T18:54:30.992 INFO:tasks.workunit.client.0.smithi102.stdout:5/377: creat d2/d2c/d38/d43/f8a x:0 0 0 2024-03-26T18:54:30.994 INFO:tasks.workunit.client.1.smithi190.stdout:1/386: mknod d3/da/dd/d2f/c85 0 2024-03-26T18:54:30.995 INFO:tasks.workunit.client.0.smithi102.stdout:8/97: mknod d0/d9/c1c 0 2024-03-26T18:54:30.995 INFO:tasks.workunit.client.0.smithi102.stdout:8/98: fdatasync d0/d15/f16 0 2024-03-26T18:54:30.995 INFO:tasks.workunit.client.0.smithi102.stdout:8/99: fdatasync d0/d15/f16 0 2024-03-26T18:54:30.996 INFO:tasks.workunit.client.0.smithi102.stdout:0/125: mknod d6/de/d13/d1f/c2a 0 2024-03-26T18:54:30.996 INFO:tasks.workunit.client.0.smithi102.stdout:0/126: fsync d6/f7 0 2024-03-26T18:54:30.997 INFO:tasks.workunit.client.1.smithi190.stdout:9/684: dwrite d12/d2d/d1d/d23/d83/d91/fe7 [0,4194304] 0 2024-03-26T18:54:30.997 INFO:tasks.workunit.client.0.smithi102.stdout:6/132: dwrite d2/d7/f1d [0,4194304] 0 2024-03-26T18:54:30.997 INFO:tasks.workunit.client.0.smithi102.stdout:6/133: chown d2/d3/l6 9 1 2024-03-26T18:54:30.998 INFO:tasks.workunit.client.0.smithi102.stdout:6/134: creat d2/d7/f20 x:0 0 0 2024-03-26T18:54:30.998 INFO:tasks.workunit.client.0.smithi102.stdout:7/75: dwrite f6 [4194304,4194304] 0 2024-03-26T18:54:30.998 INFO:tasks.workunit.client.0.smithi102.stdout:4/93: mkdir d1/d17/d1f 0 2024-03-26T18:54:30.999 INFO:tasks.workunit.client.1.smithi190.stdout:8/469: dwrite d9/dd/d15/f1c [0,4194304] 0 2024-03-26T18:54:30.999 INFO:tasks.workunit.client.1.smithi190.stdout:8/470: truncate d9/dd/d23/d36/d37/d3a/d61/f91 204011 0 2024-03-26T18:54:30.999 INFO:tasks.workunit.client.1.smithi190.stdout:8/471: dread - d9/dd/d23/d36/d37/f8d zero size 2024-03-26T18:54:30.999 INFO:tasks.workunit.client.1.smithi190.stdout:1/387: mknod d3/da/d45/c86 0 2024-03-26T18:54:31.000 INFO:tasks.workunit.client.1.smithi190.stdout:1/388: truncate d3/da/d3a/f55 385712 0 2024-03-26T18:54:31.000 INFO:tasks.workunit.client.1.smithi190.stdout:4/463: dwrite d7/d1d/d5e/f81 [4194304,4194304] 0 2024-03-26T18:54:31.000 INFO:tasks.workunit.client.1.smithi190.stdout:4/464: write d7/d8/d14/d4b/f77 [1333164,40307] 0 2024-03-26T18:54:31.000 INFO:tasks.workunit.client.1.smithi190.stdout:4/465: readlink d7/l43 0 2024-03-26T18:54:31.000 INFO:tasks.workunit.client.1.smithi190.stdout:7/491: dwrite d3/d6/d9/d1f/f3d [0,4194304] 0 2024-03-26T18:54:31.000 INFO:tasks.workunit.client.0.smithi102.stdout:1/105: truncate fb 716759 0 2024-03-26T18:54:31.002 INFO:tasks.workunit.client.0.smithi102.stdout:3/258: mknod df/d29/d31/c5e 0 2024-03-26T18:54:31.002 INFO:tasks.workunit.client.0.smithi102.stdout:3/259: creat df/d29/d3a/d47/f5f x:0 0 0 2024-03-26T18:54:31.002 INFO:tasks.workunit.client.0.smithi102.stdout:3/260: chown df/d29/f5b 58 1 2024-03-26T18:54:31.002 INFO:tasks.workunit.client.0.smithi102.stdout:3/261: fdatasync df/f26 0 2024-03-26T18:54:31.002 INFO:tasks.workunit.client.0.smithi102.stdout:3/262: write df/f1a [2803459,20433] 0 2024-03-26T18:54:31.005 INFO:tasks.workunit.client.0.smithi102.stdout:2/147: truncate d3/f4 3484214 0 2024-03-26T18:54:31.008 INFO:tasks.workunit.client.0.smithi102.stdout:5/378: rmdir d2/d25/d2a/d85 0 2024-03-26T18:54:31.009 INFO:tasks.workunit.client.1.smithi190.stdout:4/466: dread d7/d8/d1c/f7a [0,4194304] 0 2024-03-26T18:54:31.010 INFO:tasks.workunit.client.0.smithi102.stdout:8/100: mknod d0/d9/d12/c1d 0 2024-03-26T18:54:31.012 INFO:tasks.workunit.client.0.smithi102.stdout:0/127: mknod d6/de/c2b 0 2024-03-26T18:54:31.014 INFO:tasks.workunit.client.1.smithi190.stdout:9/685: write d12/d2d/d1d/d23/d83/dc9/dd4/f6c [2265629,129130] 0 2024-03-26T18:54:31.014 INFO:tasks.workunit.client.1.smithi190.stdout:9/686: truncate d12/d2d/d1d/f3b 4407684 0 2024-03-26T18:54:31.014 INFO:tasks.workunit.client.1.smithi190.stdout:9/687: fsync d12/d2d/f47 0 2024-03-26T18:54:31.014 INFO:tasks.workunit.client.1.smithi190.stdout:9/688: truncate d12/faf 505884 0 2024-03-26T18:54:31.014 INFO:tasks.workunit.client.1.smithi190.stdout:9/689: chown d12/d80 160211919 1 2024-03-26T18:54:31.014 INFO:tasks.workunit.client.1.smithi190.stdout:9/690: write d12/d2d/d1d/d23/d83/dc9/dd4/d71/fa6 [5227023,36470] 0 2024-03-26T18:54:31.017 INFO:tasks.workunit.client.0.smithi102.stdout:6/135: dread d2/d7/f1d [0,4194304] 0 2024-03-26T18:54:31.017 INFO:tasks.workunit.client.0.smithi102.stdout:7/76: dwrite f4 [0,4194304] 0 2024-03-26T18:54:31.017 INFO:tasks.workunit.client.0.smithi102.stdout:7/77: creat d7/d10/f1c x:0 0 0 2024-03-26T18:54:31.017 INFO:tasks.workunit.client.0.smithi102.stdout:7/78: write d7/d10/f14 [801027,21245] 0 2024-03-26T18:54:31.018 INFO:tasks.workunit.client.0.smithi102.stdout:4/94: write d1/f9 [1105032,44006] 0 2024-03-26T18:54:31.022 INFO:tasks.workunit.client.0.smithi102.stdout:1/106: fdatasync f9 0 2024-03-26T18:54:31.022 INFO:tasks.workunit.client.0.smithi102.stdout:1/107: chown de/d1f 593662056 1 2024-03-26T18:54:31.026 INFO:tasks.workunit.client.0.smithi102.stdout:3/263: link df/f26 df/d29/d31/d36/d45/f60 0 2024-03-26T18:54:31.026 INFO:tasks.workunit.client.0.smithi102.stdout:3/264: chown df/d1c/d34/l59 29298 1 2024-03-26T18:54:31.028 INFO:tasks.workunit.client.0.smithi102.stdout:8/101: mknod d0/c1e 0 2024-03-26T18:54:31.028 INFO:tasks.workunit.client.0.smithi102.stdout:8/102: readlink d0/d3/l8 0 2024-03-26T18:54:31.029 INFO:tasks.workunit.client.0.smithi102.stdout:0/128: mknod d6/c2c 0 2024-03-26T18:54:31.029 INFO:tasks.workunit.client.0.smithi102.stdout:0/129: write d6/de/d13/d1f/f26 [2751880,52910] 0 2024-03-26T18:54:31.030 INFO:tasks.workunit.client.0.smithi102.stdout:5/379: dwrite d2/d2c/f5a [0,4194304] 0 2024-03-26T18:54:31.030 INFO:tasks.workunit.client.0.smithi102.stdout:5/380: fdatasync d2/d25/d5b/d72/f55 0 2024-03-26T18:54:31.030 INFO:tasks.workunit.client.0.smithi102.stdout:5/381: getdents d2/d25/d2a/d5d/d82 0 2024-03-26T18:54:31.032 INFO:tasks.workunit.client.1.smithi190.stdout:7/492: symlink d3/d6/d9/d1f/d41/d50/d53/d87/lb4 0 2024-03-26T18:54:31.034 INFO:tasks.workunit.client.0.smithi102.stdout:7/79: rename d7/d12/l16 to d7/d12/l1d 0 2024-03-26T18:54:31.036 INFO:tasks.workunit.client.1.smithi190.stdout:4/467: unlink d7/d8/d14/d6e/f84 0 2024-03-26T18:54:31.037 INFO:tasks.workunit.client.0.smithi102.stdout:4/95: mknod d1/d17/d1f/c20 0 2024-03-26T18:54:31.038 INFO:tasks.workunit.client.1.smithi190.stdout:9/691: mknod d12/d80/db2/cf0 0 2024-03-26T18:54:31.038 INFO:tasks.workunit.client.1.smithi190.stdout:9/692: fsync d12/d2d/d1d/d54/f88 0 2024-03-26T18:54:31.039 INFO:tasks.workunit.client.1.smithi190.stdout:8/472: rename d9/d16 to d9/da2 0 2024-03-26T18:54:31.042 INFO:tasks.workunit.client.1.smithi190.stdout:7/493: mknod d3/d6/d9/cb5 0 2024-03-26T18:54:31.044 INFO:tasks.workunit.client.0.smithi102.stdout:8/103: unlink d0/d3/l8 0 2024-03-26T18:54:31.045 INFO:tasks.workunit.client.1.smithi190.stdout:8/473: dread d9/dd/f78 [0,4194304] 0 2024-03-26T18:54:31.045 INFO:tasks.workunit.client.1.smithi190.stdout:8/474: chown d9/dd/d23/d36/d37/d3a/d43/f80 7 1 2024-03-26T18:54:31.046 INFO:tasks.workunit.client.1.smithi190.stdout:4/468: mknod d7/d8/d14/ca0 0 2024-03-26T18:54:31.048 INFO:tasks.workunit.client.1.smithi190.stdout:9/693: truncate f6 2667311 0 2024-03-26T18:54:31.048 INFO:tasks.workunit.client.1.smithi190.stdout:9/694: creat d12/d2d/d1d/d23/d83/d91/ff1 x:0 0 0 2024-03-26T18:54:31.049 INFO:tasks.workunit.client.1.smithi190.stdout:8/475: dread d9/dd/d23/d36/f71 [0,4194304] 0 2024-03-26T18:54:31.049 INFO:tasks.workunit.client.1.smithi190.stdout:8/476: stat d9/dd/d15/d32/c52 0 2024-03-26T18:54:31.050 INFO:tasks.workunit.client.1.smithi190.stdout:8/477: creat d9/d39/d67/fa3 x:0 0 0 2024-03-26T18:54:31.050 INFO:tasks.workunit.client.1.smithi190.stdout:8/478: fsync d9/d39/d79/f90 0 2024-03-26T18:54:31.050 INFO:tasks.workunit.client.1.smithi190.stdout:8/479: chown d9/dd/d15/f1c 74805 1 2024-03-26T18:54:31.050 INFO:tasks.workunit.client.1.smithi190.stdout:8/480: fdatasync d9/dd/d15/f30 0 2024-03-26T18:54:31.051 INFO:tasks.workunit.client.1.smithi190.stdout:7/494: link d3/f19 d3/d6/d9/d14/d59/d73/d8b/fb6 0 2024-03-26T18:54:31.052 INFO:tasks.workunit.client.0.smithi102.stdout:5/382: mkdir d2/d25/d2a/d5d/d8b 0 2024-03-26T18:54:31.054 INFO:tasks.workunit.client.0.smithi102.stdout:7/80: symlink d7/d10/d17/l1e 0 2024-03-26T18:54:31.057 INFO:tasks.workunit.client.1.smithi190.stdout:8/481: symlink d9/dd/d23/d36/d37/d3a/d43/la4 0 2024-03-26T18:54:31.059 INFO:tasks.workunit.client.0.smithi102.stdout:5/383: creat d2/d25/d2a/d5d/f8c x:0 0 0 2024-03-26T18:54:31.059 INFO:tasks.workunit.client.1.smithi190.stdout:4/469: read d7/d8/d1c/f39 [4127920,60194] 0 2024-03-26T18:54:31.060 INFO:tasks.workunit.client.1.smithi190.stdout:8/482: dread d9/dd/f78 [0,4194304] 0 2024-03-26T18:54:31.060 INFO:tasks.workunit.client.0.smithi102.stdout:7/81: symlink d7/d10/d17/l1f 0 2024-03-26T18:54:31.060 INFO:tasks.workunit.client.0.smithi102.stdout:7/82: stat f4 0 2024-03-26T18:54:31.062 INFO:tasks.workunit.client.1.smithi190.stdout:7/495: rename d3/d6/d9/d14/d36/f6b to d3/d6/d9/d1f/d41/fb7 0 2024-03-26T18:54:31.062 INFO:tasks.workunit.client.1.smithi190.stdout:7/496: chown d3/f7c 402154712 1 2024-03-26T18:54:31.063 INFO:tasks.workunit.client.1.smithi190.stdout:4/470: rmdir d7/d1d/d41/d74 39 2024-03-26T18:54:31.065 INFO:tasks.workunit.client.0.smithi102.stdout:5/384: creat d2/d2c/f8d x:0 0 0 2024-03-26T18:54:31.066 INFO:tasks.workunit.client.1.smithi190.stdout:8/483: link d9/dd/d15/d32/c52 d9/dd/d23/d88/ca5 0 2024-03-26T18:54:31.067 INFO:tasks.workunit.client.0.smithi102.stdout:7/83: creat d7/f20 x:0 0 0 2024-03-26T18:54:31.069 INFO:tasks.workunit.client.1.smithi190.stdout:7/497: link d3/d6/d9/d1f/d4c/d71/la7 d3/d6/d9/d1f/d41/d50/d53/d87/lb8 0 2024-03-26T18:54:31.069 INFO:tasks.workunit.client.1.smithi190.stdout:7/498: fdatasync d3/f16 0 2024-03-26T18:54:31.069 INFO:tasks.workunit.client.1.smithi190.stdout:7/499: readlink d3/d6/d9/d14/d36/l42 0 2024-03-26T18:54:31.069 INFO:tasks.workunit.client.1.smithi190.stdout:7/500: fsync d3/d6/d9/d14/d59/f61 0 2024-03-26T18:54:31.069 INFO:tasks.workunit.client.0.smithi102.stdout:5/385: unlink d2/d25/d2a/f6c 0 2024-03-26T18:54:31.070 INFO:tasks.workunit.client.1.smithi190.stdout:4/471: mkdir d7/d8/d14/d4b/d7d/da1 0 2024-03-26T18:54:31.071 INFO:tasks.workunit.client.1.smithi190.stdout:4/472: fdatasync d7/d8/d14/d23/d28/d38/f62 0 2024-03-26T18:54:31.072 INFO:tasks.workunit.client.0.smithi102.stdout:7/84: unlink d7/d10/d17/f1a 0 2024-03-26T18:54:31.072 INFO:tasks.workunit.client.0.smithi102.stdout:7/85: fsync d7/d10/d17/f1b 0 2024-03-26T18:54:31.072 INFO:tasks.workunit.client.0.smithi102.stdout:7/86: creat d7/d10/d17/f21 x:0 0 0 2024-03-26T18:54:31.073 INFO:tasks.workunit.client.1.smithi190.stdout:7/501: creat d3/d6/d9/d1f/d41/d50/d53/d87/fb9 x:0 0 0 2024-03-26T18:54:31.075 INFO:tasks.workunit.client.0.smithi102.stdout:5/386: creat d2/d25/d2a/d5d/d8b/f8e x:0 0 0 2024-03-26T18:54:31.077 INFO:tasks.workunit.client.1.smithi190.stdout:7/502: mkdir d3/d6/d9/d1f/d84/d99/dba 0 2024-03-26T18:54:31.077 INFO:tasks.workunit.client.0.smithi102.stdout:6/136: fsync d2/d7/f20 0 2024-03-26T18:54:31.078 INFO:tasks.workunit.client.1.smithi190.stdout:7/503: rename d3/d6/l3a to d3/d6/d9/d14/d36/lbb 0 2024-03-26T18:54:31.078 INFO:tasks.workunit.client.1.smithi190.stdout:7/504: creat d3/d6/d9/d1f/d41/d65/d90/d92/fbc x:0 0 0 2024-03-26T18:54:31.079 INFO:tasks.workunit.client.0.smithi102.stdout:6/137: creat d2/f21 x:0 0 0 2024-03-26T18:54:31.079 INFO:tasks.workunit.client.0.smithi102.stdout:6/138: truncate d2/f21 108340 0 2024-03-26T18:54:31.081 INFO:tasks.workunit.client.1.smithi190.stdout:7/505: symlink d3/d6/d9/lbd 0 2024-03-26T18:54:31.081 INFO:tasks.workunit.client.1.smithi190.stdout:7/506: write d3/fa4 [379525,21800] 0 2024-03-26T18:54:31.083 INFO:tasks.workunit.client.1.smithi190.stdout:7/507: rename d3/d6/f56 to d3/d6/d9/d1f/dab/fbe 0 2024-03-26T18:54:31.083 INFO:tasks.workunit.client.1.smithi190.stdout:7/508: write d3/d6/d9/d14/d36/f3c [4648880,51052] 0 2024-03-26T18:54:31.083 INFO:tasks.workunit.client.0.smithi102.stdout:0/130: dwrite d6/de/d19/f1c [0,4194304] 0 2024-03-26T18:54:31.083 INFO:tasks.workunit.client.0.smithi102.stdout:1/108: dwrite fc [0,4194304] 0 2024-03-26T18:54:31.084 INFO:tasks.workunit.client.0.smithi102.stdout:3/265: rename df/d29/d3a/d47 to df/d61 0 2024-03-26T18:54:31.084 INFO:tasks.workunit.client.0.smithi102.stdout:3/266: fdatasync df/d29/d31/d36/d3c/f3f 0 2024-03-26T18:54:31.087 INFO:tasks.workunit.client.0.smithi102.stdout:4/96: dwrite d1/f11 [0,4194304] 0 2024-03-26T18:54:31.088 INFO:tasks.workunit.client.0.smithi102.stdout:8/104: dwrite d0/d3/f10 [0,4194304] 0 2024-03-26T18:54:31.088 INFO:tasks.workunit.client.0.smithi102.stdout:7/87: fsync d7/d10/f14 0 2024-03-26T18:54:31.088 INFO:tasks.workunit.client.0.smithi102.stdout:7/88: truncate d7/d10/d17/f21 55225 0 2024-03-26T18:54:31.089 INFO:tasks.workunit.client.1.smithi190.stdout:7/509: mkdir d3/d6/d9/d14/d59/dbf 0 2024-03-26T18:54:31.089 INFO:tasks.workunit.client.1.smithi190.stdout:4/473: dwrite d7/d1d/d5e/f81 [4194304,4194304] 0 2024-03-26T18:54:31.089 INFO:tasks.workunit.client.1.smithi190.stdout:4/474: creat d7/d8/d14/d4b/d7d/fa2 x:0 0 0 2024-03-26T18:54:31.090 INFO:tasks.workunit.client.0.smithi102.stdout:8/105: dread d0/d3/f6 [0,4194304] 0 2024-03-26T18:54:31.094 INFO:tasks.workunit.client.0.smithi102.stdout:2/148: dread d3/fb [0,4194304] 0 2024-03-26T18:54:31.095 INFO:tasks.workunit.client.1.smithi190.stdout:7/510: symlink d3/d6/d9/d1f/d41/d50/lc0 0 2024-03-26T18:54:31.095 INFO:tasks.workunit.client.1.smithi190.stdout:7/511: write d3/d6/d9/f40 [720,45347] 0 2024-03-26T18:54:31.099 INFO:tasks.workunit.client.0.smithi102.stdout:0/131: mkdir d6/d2d 0 2024-03-26T18:54:31.099 INFO:tasks.workunit.client.0.smithi102.stdout:0/132: chown d6/de/c2b 0 1 2024-03-26T18:54:31.103 INFO:tasks.workunit.client.1.smithi190.stdout:4/475: rename d7/d8/l2d to d7/d8/d14/d23/d28/d9e/la3 0 2024-03-26T18:54:31.103 INFO:tasks.workunit.client.1.smithi190.stdout:4/476: chown d7/d1d/d42/d4a/l8f 245834 1 2024-03-26T18:54:31.103 INFO:tasks.workunit.client.1.smithi190.stdout:4/477: rename d7/d1d/d42/d4a to d7/d1d/d42/d4a/da4 22 2024-03-26T18:54:31.110 INFO:tasks.workunit.client.1.smithi190.stdout:2/470: sync 2024-03-26T18:54:31.110 INFO:tasks.workunit.client.1.smithi190.stdout:5/316: sync 2024-03-26T18:54:31.110 INFO:tasks.workunit.client.1.smithi190.stdout:5/317: fsync d2/d17/d1b/d1e/d3a/f55 0 2024-03-26T18:54:31.110 INFO:tasks.workunit.client.1.smithi190.stdout:4/478: mknod d7/d8/d1c/d20/d40/ca5 0 2024-03-26T18:54:31.111 INFO:tasks.workunit.client.1.smithi190.stdout:4/479: stat d7/d8/d14/d4b/f75 0 2024-03-26T18:54:31.111 INFO:tasks.workunit.client.1.smithi190.stdout:2/471: mknod d6/d9/d3a/d3d/d55/caa 0 2024-03-26T18:54:31.113 INFO:tasks.workunit.client.1.smithi190.stdout:5/318: creat d2/d17/d1b/d30/f73 x:0 0 0 2024-03-26T18:54:31.113 INFO:tasks.workunit.client.1.smithi190.stdout:5/319: creat d2/d17/d62/d64/f74 x:0 0 0 2024-03-26T18:54:31.114 INFO:tasks.workunit.client.0.smithi102.stdout:1/109: symlink de/d16/d1d/l27 0 2024-03-26T18:54:31.114 INFO:tasks.workunit.client.0.smithi102.stdout:1/110: dread - de/f26 zero size 2024-03-26T18:54:31.114 INFO:tasks.workunit.client.0.smithi102.stdout:1/111: creat de/d16/d1d/f28 x:0 0 0 2024-03-26T18:54:31.115 INFO:tasks.workunit.client.1.smithi190.stdout:4/480: creat d7/d1d/d42/d4a/fa6 x:0 0 0 2024-03-26T18:54:31.115 INFO:tasks.workunit.client.1.smithi190.stdout:4/481: write d7/d1d/d5e/f76 [1094683,102710] 0 2024-03-26T18:54:31.115 INFO:tasks.workunit.client.1.smithi190.stdout:4/482: readlink d7/d8/d1c/l34 0 2024-03-26T18:54:31.115 INFO:tasks.workunit.client.1.smithi190.stdout:4/483: fsync d7/f18 0 2024-03-26T18:54:31.116 INFO:tasks.workunit.client.0.smithi102.stdout:5/387: rename d2/fa to d2/d25/d2a/d5d/d82/f8f 0 2024-03-26T18:54:31.119 INFO:tasks.workunit.client.1.smithi190.stdout:2/472: rename d6/d21/f78 to d6/d9/d3a/d3d/d1c/d8d/fab 0 2024-03-26T18:54:31.123 INFO:tasks.workunit.client.1.smithi190.stdout:5/320: creat d2/d17/d1b/d22/d68/f75 x:0 0 0 2024-03-26T18:54:31.125 INFO:tasks.workunit.client.0.smithi102.stdout:4/97: mkdir d1/d17/d1a/d21 0 2024-03-26T18:54:31.126 INFO:tasks.workunit.client.0.smithi102.stdout:7/89: link d7/ff d7/d12/f22 0 2024-03-26T18:54:31.127 INFO:tasks.workunit.client.1.smithi190.stdout:2/473: mkdir d6/d9/d1a/d72/d7c/dac 0 2024-03-26T18:54:31.128 INFO:tasks.workunit.client.0.smithi102.stdout:2/149: creat d3/d20/f30 x:0 0 0 2024-03-26T18:54:31.130 INFO:tasks.workunit.client.0.smithi102.stdout:0/133: symlink d6/de/l2e 0 2024-03-26T18:54:31.131 INFO:tasks.workunit.client.1.smithi190.stdout:2/474: link d6/d9/d10/f2d d6/d9/d10/d5f/fad 0 2024-03-26T18:54:31.132 INFO:tasks.workunit.client.1.smithi190.stdout:2/475: fsync d6/d9/d3a/d3d/d55/fa5 0 2024-03-26T18:54:31.132 INFO:tasks.workunit.client.0.smithi102.stdout:3/267: dwrite df/f20 [4194304,4194304] 0 2024-03-26T18:54:31.134 INFO:tasks.workunit.client.0.smithi102.stdout:6/139: dwrite d2/d7/f8 [4194304,4194304] 0 2024-03-26T18:54:31.134 INFO:tasks.workunit.client.0.smithi102.stdout:8/106: rename d0/d3/f10 to d0/d9/d12/f1f 0 2024-03-26T18:54:31.137 INFO:tasks.workunit.client.0.smithi102.stdout:0/134: unlink d6/de/c2b 0 2024-03-26T18:54:31.138 INFO:tasks.workunit.client.0.smithi102.stdout:0/135: write d6/de/f1b [868451,96366] 0 2024-03-26T18:54:31.138 INFO:tasks.workunit.client.0.smithi102.stdout:0/136: write d6/de/d13/d1f/d20/f29 [873446,31466] 0 2024-03-26T18:54:31.141 INFO:tasks.workunit.client.0.smithi102.stdout:1/112: dwrite de/f19 [4194304,4194304] 0 2024-03-26T18:54:31.143 INFO:tasks.workunit.client.0.smithi102.stdout:3/268: unlink df/d1c/d34/d53/f54 0 2024-03-26T18:54:31.147 INFO:tasks.workunit.client.0.smithi102.stdout:6/140: creat d2/d12/f22 x:0 0 0 2024-03-26T18:54:31.147 INFO:tasks.workunit.client.0.smithi102.stdout:6/141: dread - d2/d12/f22 zero size 2024-03-26T18:54:31.147 INFO:tasks.workunit.client.0.smithi102.stdout:6/142: write d2/d7/f20 [577010,92474] 0 2024-03-26T18:54:31.147 INFO:tasks.workunit.client.1.smithi190.stdout:5/321: write d2/d17/d1b/d22/f54 [2078782,79493] 0 2024-03-26T18:54:31.148 INFO:tasks.workunit.client.1.smithi190.stdout:5/322: chown d2/d17/d1b/d22/f2d 778 1 2024-03-26T18:54:31.148 INFO:tasks.workunit.client.1.smithi190.stdout:5/323: creat d2/d17/d1b/f76 x:0 0 0 2024-03-26T18:54:31.148 INFO:tasks.workunit.client.1.smithi190.stdout:5/324: fsync d2/d17/d1b/f72 0 2024-03-26T18:54:31.149 INFO:tasks.workunit.client.1.smithi190.stdout:2/476: mkdir d6/d9/d3a/d3d/d67/d9e/da3/dae 0 2024-03-26T18:54:31.150 INFO:tasks.workunit.client.0.smithi102.stdout:0/137: symlink d6/de/l2f 0 2024-03-26T18:54:31.155 INFO:tasks.workunit.client.0.smithi102.stdout:6/143: rename d2/d7/l14 to d2/df/l23 0 2024-03-26T18:54:31.159 INFO:tasks.workunit.client.1.smithi190.stdout:5/325: creat d2/d17/d1b/d22/f77 x:0 0 0 2024-03-26T18:54:31.159 INFO:tasks.workunit.client.0.smithi102.stdout:3/269: dread df/d29/d31/d36/f41 [0,4194304] 0 2024-03-26T18:54:31.161 INFO:tasks.workunit.client.1.smithi190.stdout:7/512: dwrite d3/d6/fd [0,4194304] 0 2024-03-26T18:54:31.162 INFO:tasks.workunit.client.1.smithi190.stdout:5/326: mknod d2/d17/d1b/d22/d24/d3b/c78 0 2024-03-26T18:54:31.163 INFO:tasks.workunit.client.0.smithi102.stdout:3/270: mknod df/d29/d3a/c62 0 2024-03-26T18:54:31.163 INFO:tasks.workunit.client.0.smithi102.stdout:3/271: chown df/d29 2171201 1 2024-03-26T18:54:31.164 INFO:tasks.workunit.client.1.smithi190.stdout:7/513: truncate d3/f16 1351573 0 2024-03-26T18:54:31.165 INFO:tasks.workunit.client.1.smithi190.stdout:2/477: write d6/d9/d3a/d3d/d2e/f5c [1738479,83412] 0 2024-03-26T18:54:31.165 INFO:tasks.workunit.client.1.smithi190.stdout:5/327: getdents d2/d17 0 2024-03-26T18:54:31.169 INFO:tasks.workunit.client.0.smithi102.stdout:0/138: dread d6/f18 [0,4194304] 0 2024-03-26T18:54:31.169 INFO:tasks.workunit.client.0.smithi102.stdout:0/139: write d6/de/d19/f1c [5108656,46216] 0 2024-03-26T18:54:31.169 INFO:tasks.workunit.client.0.smithi102.stdout:0/140: chown d6/de/d13/d1f/d20/c21 8 1 2024-03-26T18:54:31.169 INFO:tasks.workunit.client.1.smithi190.stdout:7/514: truncate d3/d6/d9/d14/f18 47565 0 2024-03-26T18:54:31.173 INFO:tasks.workunit.client.1.smithi190.stdout:2/478: mkdir d6/d9/d3a/d3d/d1c/d44/daf 0 2024-03-26T18:54:31.174 INFO:tasks.workunit.client.1.smithi190.stdout:2/479: write d6/d9/d1a/d72/d8b/fa8 [107854,26404] 0 2024-03-26T18:54:31.175 INFO:tasks.workunit.client.1.smithi190.stdout:5/328: symlink d2/d17/d1b/d1e/d3a/l79 0 2024-03-26T18:54:31.175 INFO:tasks.workunit.client.1.smithi190.stdout:5/329: fsync d2/d17/d1b/d60/d63/f6e 0 2024-03-26T18:54:31.176 INFO:tasks.workunit.client.0.smithi102.stdout:0/141: creat d6/de/d19/f30 x:0 0 0 2024-03-26T18:54:31.178 INFO:tasks.workunit.client.1.smithi190.stdout:7/515: link d3/d6/d9/f40 d3/d6/d9/d1f/d41/d65/d90/d92/d9b/fc1 0 2024-03-26T18:54:31.180 INFO:tasks.workunit.client.0.smithi102.stdout:3/272: rmdir df/d29/d51 0 2024-03-26T18:54:31.182 INFO:tasks.workunit.client.1.smithi190.stdout:7/516: creat d3/d6/d9/d14/fc2 x:0 0 0 2024-03-26T18:54:31.185 INFO:tasks.workunit.client.1.smithi190.stdout:7/517: rmdir d3/d6/d9/d14/d59/d73/d8b 39 2024-03-26T18:54:31.185 INFO:tasks.workunit.client.1.smithi190.stdout:7/518: write d3/d6/d9/d1f/d41/d50/f89 [321082,113962] 0 2024-03-26T18:54:31.185 INFO:tasks.workunit.client.1.smithi190.stdout:7/519: chown d3/d6/d9/d1f/d4c/f54 46 1 2024-03-26T18:54:31.190 INFO:tasks.workunit.client.1.smithi190.stdout:7/520: dread d3/fc [0,4194304] 0 2024-03-26T18:54:31.191 INFO:tasks.workunit.client.1.smithi190.stdout:7/521: fsync d3/d6/d9/d14/d36/f72 0 2024-03-26T18:54:31.192 INFO:tasks.workunit.client.1.smithi190.stdout:7/522: getdents d3/d1b/d57 0 2024-03-26T18:54:31.192 INFO:tasks.workunit.client.1.smithi190.stdout:7/523: creat d3/d6/d9/d14/d59/fc3 x:0 0 0 2024-03-26T18:54:31.192 INFO:tasks.workunit.client.1.smithi190.stdout:7/524: chown d3/d6/d9/d1f/dab/fbe 15 1 2024-03-26T18:54:31.192 INFO:tasks.workunit.client.1.smithi190.stdout:7/525: fsync d3/d6/d9/d1f/fa2 0 2024-03-26T18:54:31.192 INFO:tasks.workunit.client.0.smithi102.stdout:9/77: sync 2024-03-26T18:54:31.193 INFO:tasks.workunit.client.1.smithi190.stdout:7/526: mkdir d3/d6/d9/d1f/d41/d65/d90/d92/d9b/dc4 0 2024-03-26T18:54:31.194 INFO:tasks.workunit.client.1.smithi190.stdout:7/527: chown d3/d6/d9/d1f/d41/f44 466999484 1 2024-03-26T18:54:31.194 INFO:tasks.workunit.client.0.smithi102.stdout:9/78: symlink df/l15 0 2024-03-26T18:54:31.195 INFO:tasks.workunit.client.0.smithi102.stdout:9/79: dread df/f13 [0,4194304] 0 2024-03-26T18:54:31.196 INFO:tasks.workunit.client.1.smithi190.stdout:7/528: creat d3/d1b/d98/fc5 x:0 0 0 2024-03-26T18:54:31.196 INFO:tasks.workunit.client.1.smithi190.stdout:7/529: fsync d3/d6/fd 0 2024-03-26T18:54:31.196 INFO:tasks.workunit.client.1.smithi190.stdout:7/530: dread - d3/d6/d9/d1f/d41/d50/d53/f8a zero size 2024-03-26T18:54:31.198 INFO:tasks.workunit.client.0.smithi102.stdout:3/273: read df/f1b [4456306,101960] 0 2024-03-26T18:54:31.200 INFO:tasks.workunit.client.0.smithi102.stdout:9/80: dread f7 [0,4194304] 0 2024-03-26T18:54:31.200 INFO:tasks.workunit.client.0.smithi102.stdout:5/388: dwrite d2/d25/d2a/d5d/f68 [0,4194304] 0 2024-03-26T18:54:31.201 INFO:tasks.workunit.client.1.smithi190.stdout:4/484: dwrite d7/d8/f22 [0,4194304] 0 2024-03-26T18:54:31.201 INFO:tasks.workunit.client.0.smithi102.stdout:4/98: dwrite d1/f7 [0,4194304] 0 2024-03-26T18:54:31.201 INFO:tasks.workunit.client.0.smithi102.stdout:3/274: unlink df/f39 0 2024-03-26T18:54:31.210 INFO:tasks.workunit.client.0.smithi102.stdout:1/113: dwrite de/f14 [0,4194304] 0 2024-03-26T18:54:31.210 INFO:tasks.workunit.client.0.smithi102.stdout:8/107: dwrite d0/d3/f11 [0,4194304] 0 2024-03-26T18:54:31.211 INFO:tasks.workunit.client.0.smithi102.stdout:2/150: dwrite d3/d6/fe [0,4194304] 0 2024-03-26T18:54:31.211 INFO:tasks.workunit.client.0.smithi102.stdout:6/144: dwrite d2/d7/f8 [0,4194304] 0 2024-03-26T18:54:31.211 INFO:tasks.workunit.client.0.smithi102.stdout:9/81: dread f7 [4194304,4194304] 0 2024-03-26T18:54:31.211 INFO:tasks.workunit.client.0.smithi102.stdout:9/82: rename df to df/d16 22 2024-03-26T18:54:31.211 INFO:tasks.workunit.client.0.smithi102.stdout:9/83: creat df/f17 x:0 0 0 2024-03-26T18:54:31.211 INFO:tasks.workunit.client.0.smithi102.stdout:9/84: truncate df/f17 81404 0 2024-03-26T18:54:31.212 INFO:tasks.workunit.client.0.smithi102.stdout:4/99: mknod d1/d17/c22 0 2024-03-26T18:54:31.213 INFO:tasks.workunit.client.1.smithi190.stdout:2/480: dwrite d6/d9/d3a/f94 [0,4194304] 0 2024-03-26T18:54:31.213 INFO:tasks.workunit.client.1.smithi190.stdout:2/481: fsync d6/d9/d3a/d3d/d55/fa5 0 2024-03-26T18:54:31.217 INFO:tasks.workunit.client.0.smithi102.stdout:3/275: dread df/f20 [0,4194304] 0 2024-03-26T18:54:31.229 INFO:tasks.workunit.client.0.smithi102.stdout:3/276: dread df/d61/f57 [0,4194304] 0 2024-03-26T18:54:31.231 INFO:tasks.workunit.client.0.smithi102.stdout:5/389: dwrite d2/d14/d1a/f41 [0,4194304] 0 2024-03-26T18:54:31.232 INFO:tasks.workunit.client.0.smithi102.stdout:5/390: chown d2/l88 3 1 2024-03-26T18:54:31.232 INFO:tasks.workunit.client.0.smithi102.stdout:5/391: stat d2/d2c/d38/f87 0 2024-03-26T18:54:31.232 INFO:tasks.workunit.client.0.smithi102.stdout:8/108: dread d0/d3/f11 [0,4194304] 0 2024-03-26T18:54:31.234 INFO:tasks.workunit.client.0.smithi102.stdout:1/114: dread f9 [0,4194304] 0 2024-03-26T18:54:31.234 INFO:tasks.workunit.client.0.smithi102.stdout:1/115: dread - de/d16/d1d/f28 zero size 2024-03-26T18:54:31.234 INFO:tasks.workunit.client.0.smithi102.stdout:2/151: mkdir d3/d11/d15/d31 0 2024-03-26T18:54:31.237 INFO:tasks.workunit.client.0.smithi102.stdout:6/145: symlink d2/l24 0 2024-03-26T18:54:31.239 INFO:tasks.workunit.client.0.smithi102.stdout:4/100: mknod d1/d17/d1a/d21/c23 0 2024-03-26T18:54:31.241 INFO:tasks.workunit.client.0.smithi102.stdout:3/277: truncate f7 3153912 0 2024-03-26T18:54:31.245 INFO:tasks.workunit.client.1.smithi190.stdout:7/531: dwrite d3/d6/d9/d1f/d41/fb7 [4194304,4194304] 0 2024-03-26T18:54:31.246 INFO:tasks.workunit.client.0.smithi102.stdout:2/152: creat d3/d2d/f32 x:0 0 0 2024-03-26T18:54:31.249 INFO:tasks.workunit.client.0.smithi102.stdout:6/146: mknod d2/d7/c25 0 2024-03-26T18:54:31.252 INFO:tasks.workunit.client.0.smithi102.stdout:4/101: mknod d1/d17/d1a/c24 0 2024-03-26T18:54:31.255 INFO:tasks.workunit.client.1.smithi190.stdout:7/532: rename d3/d6/l26 to d3/d6/d9/d1f/lc6 0 2024-03-26T18:54:31.255 INFO:tasks.workunit.client.1.smithi190.stdout:7/533: creat d3/d6/d9/d14/fc7 x:0 0 0 2024-03-26T18:54:31.255 INFO:tasks.workunit.client.0.smithi102.stdout:8/109: dread d0/d9/d12/f1f [0,4194304] 0 2024-03-26T18:54:31.257 INFO:tasks.workunit.client.0.smithi102.stdout:1/116: getdents de 0 2024-03-26T18:54:31.257 INFO:tasks.workunit.client.0.smithi102.stdout:1/117: stat de 0 2024-03-26T18:54:31.257 INFO:tasks.workunit.client.0.smithi102.stdout:9/85: dwrite df/f17 [0,4194304] 0 2024-03-26T18:54:31.257 INFO:tasks.workunit.client.0.smithi102.stdout:9/86: write f7 [15158,6918] 0 2024-03-26T18:54:31.263 INFO:tasks.workunit.client.0.smithi102.stdout:4/102: rmdir d1/d17 39 2024-03-26T18:54:31.266 INFO:tasks.workunit.client.0.smithi102.stdout:5/392: dwrite d2/d14/f1c [0,4194304] 0 2024-03-26T18:54:31.266 INFO:tasks.workunit.client.1.smithi190.stdout:2/482: dwrite d6/d9/d3a/d3d/d1c/f5e [0,4194304] 0 2024-03-26T18:54:31.267 INFO:tasks.workunit.client.1.smithi190.stdout:4/485: dwrite d7/d8/f2a [0,4194304] 0 2024-03-26T18:54:31.267 INFO:tasks.workunit.client.0.smithi102.stdout:0/142: dwrite d6/de/d13/d1f/f27 [0,4194304] 0 2024-03-26T18:54:31.270 INFO:tasks.workunit.client.0.smithi102.stdout:0/143: dread d6/f9 [0,4194304] 0 2024-03-26T18:54:31.270 INFO:tasks.workunit.client.0.smithi102.stdout:3/278: dwrite df/d29/f40 [0,4194304] 0 2024-03-26T18:54:31.270 INFO:tasks.workunit.client.0.smithi102.stdout:3/279: fdatasync df/d29/d31/d36/f48 0 2024-03-26T18:54:31.277 INFO:tasks.workunit.client.0.smithi102.stdout:5/393: link d2/d25/d2a/d4f/c53 d2/d25/d2a/d4f/c90 0 2024-03-26T18:54:31.280 INFO:tasks.workunit.client.0.smithi102.stdout:0/144: mkdir d6/de/d13/d25/d31 0 2024-03-26T18:54:31.289 INFO:tasks.workunit.client.0.smithi102.stdout:3/280: truncate df/f25 5185616 0 2024-03-26T18:54:31.291 INFO:tasks.workunit.client.0.smithi102.stdout:3/281: symlink df/d1c/l63 0 2024-03-26T18:54:31.293 INFO:tasks.workunit.client.0.smithi102.stdout:3/282: rename df/d29/d31/d36/d3c/f49 to df/d29/f64 0 2024-03-26T18:54:31.295 INFO:tasks.workunit.client.0.smithi102.stdout:3/283: rename df/d29/d31/d36/f4f to df/d29/d31/d36/d45/f65 0 2024-03-26T18:54:31.297 INFO:tasks.workunit.client.0.smithi102.stdout:1/118: dwrite de/f10 [0,4194304] 0 2024-03-26T18:54:31.297 INFO:tasks.workunit.client.0.smithi102.stdout:1/119: creat de/d16/f29 x:0 0 0 2024-03-26T18:54:31.298 INFO:tasks.workunit.client.0.smithi102.stdout:1/120: dread - de/f1c zero size 2024-03-26T18:54:31.298 INFO:tasks.workunit.client.0.smithi102.stdout:8/110: dwrite d0/d9/d12/f1f [0,4194304] 0 2024-03-26T18:54:31.298 INFO:tasks.workunit.client.0.smithi102.stdout:3/284: creat df/d61/f66 x:0 0 0 2024-03-26T18:54:31.298 INFO:tasks.workunit.client.1.smithi190.stdout:7/534: dwrite d3/d6/f1d [4194304,4194304] 0 2024-03-26T18:54:31.298 INFO:tasks.workunit.client.1.smithi190.stdout:7/535: write d3/d1b/d98/fc5 [199174,28095] 0 2024-03-26T18:54:31.301 INFO:tasks.workunit.client.0.smithi102.stdout:1/121: fsync f3 0 2024-03-26T18:54:31.303 INFO:tasks.workunit.client.0.smithi102.stdout:8/111: write d0/d3/f11 [3227960,1453] 0 2024-03-26T18:54:31.303 INFO:tasks.workunit.client.1.smithi190.stdout:7/536: dread d3/d6/d9/d14/d59/d73/d8b/fb6 [0,4194304] 0 2024-03-26T18:54:31.303 INFO:tasks.workunit.client.1.smithi190.stdout:7/537: write d3/d6/d9/d1f/d41/d65/d90/d92/d9b/fb0 [755384,130697] 0 2024-03-26T18:54:31.304 INFO:tasks.workunit.client.1.smithi190.stdout:4/486: dwrite d7/d8/d14/d5c/f97 [0,4194304] 0 2024-03-26T18:54:31.304 INFO:tasks.workunit.client.0.smithi102.stdout:2/153: getdents d3/d11/d15 0 2024-03-26T18:54:31.304 INFO:tasks.workunit.client.0.smithi102.stdout:2/154: write d3/fb [3511309,17520] 0 2024-03-26T18:54:31.306 INFO:tasks.workunit.client.0.smithi102.stdout:2/155: symlink d3/l33 0 2024-03-26T18:54:31.307 INFO:tasks.workunit.client.1.smithi190.stdout:3/389: sync 2024-03-26T18:54:31.307 INFO:tasks.workunit.client.1.smithi190.stdout:0/550: sync 2024-03-26T18:54:31.307 INFO:tasks.workunit.client.1.smithi190.stdout:0/551: fsync d0/dd/d20/d1c/f6e 0 2024-03-26T18:54:31.307 INFO:tasks.workunit.client.1.smithi190.stdout:0/552: chown d0/la3 11291473 1 2024-03-26T18:54:31.307 INFO:tasks.workunit.client.1.smithi190.stdout:7/538: dread d3/fc [0,4194304] 0 2024-03-26T18:54:31.307 INFO:tasks.workunit.client.1.smithi190.stdout:7/539: readlink d3/d6/d9/d1f/lc6 0 2024-03-26T18:54:31.307 INFO:tasks.workunit.client.1.smithi190.stdout:7/540: truncate d3/d6/d9/f75 1024173 0 2024-03-26T18:54:31.307 INFO:tasks.workunit.client.1.smithi190.stdout:7/541: truncate d3/d1b/f3b 1511608 0 2024-03-26T18:54:31.308 INFO:tasks.workunit.client.0.smithi102.stdout:3/285: dread df/d61/f57 [0,4194304] 0 2024-03-26T18:54:31.308 INFO:tasks.workunit.client.0.smithi102.stdout:6/147: getdents d2 0 2024-03-26T18:54:31.308 INFO:tasks.workunit.client.0.smithi102.stdout:0/145: dwrite d6/de/f15 [0,4194304] 0 2024-03-26T18:54:31.308 INFO:tasks.workunit.client.0.smithi102.stdout:0/146: fdatasync d6/de/d19/f30 0 2024-03-26T18:54:31.308 INFO:tasks.workunit.client.0.smithi102.stdout:0/147: stat d6/de/d13/d25/d31 0 2024-03-26T18:54:31.309 INFO:tasks.workunit.client.0.smithi102.stdout:5/394: dwrite d2/f12 [0,4194304] 0 2024-03-26T18:54:31.309 INFO:tasks.workunit.client.0.smithi102.stdout:5/395: write d2/d14/f70 [399795,112561] 0 2024-03-26T18:54:31.309 INFO:tasks.workunit.client.0.smithi102.stdout:5/396: creat d2/d14/d1a/f91 x:0 0 0 2024-03-26T18:54:31.309 INFO:tasks.workunit.client.0.smithi102.stdout:5/397: creat d2/d2c/d38/d43/f92 x:0 0 0 2024-03-26T18:54:31.310 INFO:tasks.workunit.client.0.smithi102.stdout:4/103: getdents d1/d17/d1a/d21 0 2024-03-26T18:54:31.310 INFO:tasks.workunit.client.1.smithi190.stdout:7/542: dread d3/d6/d9/d1f/d41/d65/f7d [0,4194304] 0 2024-03-26T18:54:31.311 INFO:tasks.workunit.client.0.smithi102.stdout:2/156: creat d3/d2a/f34 x:0 0 0 2024-03-26T18:54:31.314 INFO:tasks.workunit.client.1.smithi190.stdout:3/390: unlink d1/d3/df/d18/d1b/l27 0 2024-03-26T18:54:31.317 INFO:tasks.workunit.client.1.smithi190.stdout:0/553: mkdir d0/dd/d24/d87/d93/dc4 0 2024-03-26T18:54:31.319 INFO:tasks.workunit.client.0.smithi102.stdout:6/148: write d2/d7/f1d [1609004,26109] 0 2024-03-26T18:54:31.319 INFO:tasks.workunit.client.0.smithi102.stdout:6/149: getdents d2 0 2024-03-26T18:54:31.321 INFO:tasks.workunit.client.1.smithi190.stdout:3/391: creat d1/d3/df/d18/f6f x:0 0 0 2024-03-26T18:54:31.322 INFO:tasks.workunit.client.1.smithi190.stdout:4/487: write d7/d1d/d5e/f81 [6717154,9732] 0 2024-03-26T18:54:31.322 INFO:tasks.workunit.client.1.smithi190.stdout:4/488: write d7/d8/d14/f57 [5605387,35189] 0 2024-03-26T18:54:31.322 INFO:tasks.workunit.client.1.smithi190.stdout:4/489: creat d7/d1d/fa7 x:0 0 0 2024-03-26T18:54:31.322 INFO:tasks.workunit.client.1.smithi190.stdout:4/490: write d7/d8/d14/d4b/f4c [5184125,16237] 0 2024-03-26T18:54:31.329 INFO:tasks.workunit.client.0.smithi102.stdout:9/87: truncate df/f17 2675597 0 2024-03-26T18:54:31.331 INFO:tasks.workunit.client.0.smithi102.stdout:2/157: rename d3/d6/d1a/f1f to d3/d11/d15/d31/f35 0 2024-03-26T18:54:31.332 INFO:tasks.workunit.client.0.smithi102.stdout:0/148: read d6/f11 [3022601,78865] 0 2024-03-26T18:54:31.333 INFO:tasks.workunit.client.1.smithi190.stdout:3/392: write d1/d3/f34 [176844,97431] 0 2024-03-26T18:54:31.333 INFO:tasks.workunit.client.1.smithi190.stdout:3/393: chown d1/d3/d5a 3562 1 2024-03-26T18:54:31.333 INFO:tasks.workunit.client.1.smithi190.stdout:3/394: dread - d1/d3/df/d31/f56 zero size 2024-03-26T18:54:31.333 INFO:tasks.workunit.client.1.smithi190.stdout:4/491: symlink d7/d8/d1c/d20/la8 0 2024-03-26T18:54:31.336 INFO:tasks.workunit.client.0.smithi102.stdout:6/150: symlink d2/d3/l26 0 2024-03-26T18:54:31.336 INFO:tasks.workunit.client.0.smithi102.stdout:6/151: readlink d2/l13 0 2024-03-26T18:54:31.338 INFO:tasks.workunit.client.1.smithi190.stdout:3/395: mkdir d1/d3/d52/d70 0 2024-03-26T18:54:31.339 INFO:tasks.workunit.client.0.smithi102.stdout:4/104: mkdir d1/d25 0 2024-03-26T18:54:31.340 INFO:tasks.workunit.client.1.smithi190.stdout:4/492: unlink d7/d8/d14/d23/d28/d9e/la3 0 2024-03-26T18:54:31.340 INFO:tasks.workunit.client.1.smithi190.stdout:4/493: stat d7/d8/d14/d4b/l88 0 2024-03-26T18:54:31.341 INFO:tasks.workunit.client.1.smithi190.stdout:4/494: chown d7/d1d/f26 46550510 1 2024-03-26T18:54:31.341 INFO:tasks.workunit.client.0.smithi102.stdout:9/88: mknod df/c18 0 2024-03-26T18:54:31.343 INFO:tasks.workunit.client.1.smithi190.stdout:3/396: mknod d1/de/d1c/c71 0 2024-03-26T18:54:31.344 INFO:tasks.workunit.client.0.smithi102.stdout:2/158: mknod d3/d11/d15/c36 0 2024-03-26T18:54:31.345 INFO:tasks.workunit.client.0.smithi102.stdout:8/112: dwrite d0/d15/f16 [0,4194304] 0 2024-03-26T18:54:31.346 INFO:tasks.workunit.client.1.smithi190.stdout:3/397: rename d1/f3b to d1/d3/df/d18/d35/d41/f72 0 2024-03-26T18:54:31.349 INFO:tasks.workunit.client.0.smithi102.stdout:8/113: dread d0/d3/f6 [0,4194304] 0 2024-03-26T18:54:31.350 INFO:tasks.workunit.client.1.smithi190.stdout:9/695: sync 2024-03-26T18:54:31.350 INFO:tasks.workunit.client.1.smithi190.stdout:6/424: sync 2024-03-26T18:54:31.350 INFO:tasks.workunit.client.1.smithi190.stdout:8/484: sync 2024-03-26T18:54:31.350 INFO:tasks.workunit.client.1.smithi190.stdout:1/389: sync 2024-03-26T18:54:31.350 INFO:tasks.workunit.client.1.smithi190.stdout:9/696: chown d12/d2d/d1d/d54/d92/cb1 181512 1 2024-03-26T18:54:31.350 INFO:tasks.workunit.client.1.smithi190.stdout:8/485: dread - d9/dd/d15/f4b zero size 2024-03-26T18:54:31.350 INFO:tasks.workunit.client.1.smithi190.stdout:8/486: write d9/dd/d23/d36/d37/d3a/d43/f80 [231950,108546] 0 2024-03-26T18:54:31.351 INFO:tasks.workunit.client.0.smithi102.stdout:0/149: rename d6/de/d13/d1f/f26 to d6/de/d13/d25/d31/f32 0 2024-03-26T18:54:31.351 INFO:tasks.workunit.client.0.smithi102.stdout:4/105: creat d1/f26 x:0 0 0 2024-03-26T18:54:31.351 INFO:tasks.workunit.client.0.smithi102.stdout:1/122: truncate f8 6189841 0 2024-03-26T18:54:31.351 INFO:tasks.workunit.client.0.smithi102.stdout:1/123: readlink de/d16/l1b 0 2024-03-26T18:54:31.351 INFO:tasks.workunit.client.1.smithi190.stdout:3/398: link d1/d3/df/d18/d35/f60 d1/de/f73 0 2024-03-26T18:54:31.353 INFO:tasks.workunit.client.0.smithi102.stdout:8/114: mkdir d0/d15/d1b/d20 0 2024-03-26T18:54:31.356 INFO:tasks.workunit.client.0.smithi102.stdout:0/150: mknod d6/de/c33 0 2024-03-26T18:54:31.356 INFO:tasks.workunit.client.0.smithi102.stdout:0/151: fsync d6/fd 0 2024-03-26T18:54:31.356 INFO:tasks.workunit.client.0.smithi102.stdout:0/152: fdatasync d6/f9 0 2024-03-26T18:54:31.356 INFO:tasks.workunit.client.0.smithi102.stdout:0/153: chown d6/de/d13/d1f 1409 1 2024-03-26T18:54:31.357 INFO:tasks.workunit.client.0.smithi102.stdout:0/154: fsync d6/de/f1d 0 2024-03-26T18:54:31.358 INFO:tasks.workunit.client.0.smithi102.stdout:1/124: mkdir de/d16/d1d/d2a 0 2024-03-26T18:54:31.358 INFO:tasks.workunit.client.0.smithi102.stdout:1/125: stat fb 0 2024-03-26T18:54:31.359 INFO:tasks.workunit.client.1.smithi190.stdout:7/543: dwrite d3/d6/d9/d14/d59/f61 [0,4194304] 0 2024-03-26T18:54:31.359 INFO:tasks.workunit.client.1.smithi190.stdout:7/544: truncate d3/d6/fac 694634 0 2024-03-26T18:54:31.359 INFO:tasks.workunit.client.1.smithi190.stdout:1/390: truncate f2 5801710 0 2024-03-26T18:54:31.360 INFO:tasks.workunit.client.1.smithi190.stdout:1/391: write d3/da/f78 [5660,21744] 0 2024-03-26T18:54:31.360 INFO:tasks.workunit.client.1.smithi190.stdout:1/392: write d3/f7f [433395,115358] 0 2024-03-26T18:54:31.360 INFO:tasks.workunit.client.1.smithi190.stdout:1/393: stat d3/da/l3f 0 2024-03-26T18:54:31.360 INFO:tasks.workunit.client.0.smithi102.stdout:1/126: dread f3 [0,4194304] 0 2024-03-26T18:54:31.361 INFO:tasks.workunit.client.1.smithi190.stdout:9/697: symlink d12/d2d/d1d/d23/d2b/da0/lf2 0 2024-03-26T18:54:31.361 INFO:tasks.workunit.client.1.smithi190.stdout:4/495: dread d7/d8/d14/f57 [0,4194304] 0 2024-03-26T18:54:31.361 INFO:tasks.workunit.client.1.smithi190.stdout:8/487: mknod d9/dd/ca6 0 2024-03-26T18:54:31.361 INFO:tasks.workunit.client.1.smithi190.stdout:8/488: fdatasync d9/dd/d15/d32/f35 0 2024-03-26T18:54:31.362 INFO:tasks.workunit.client.0.smithi102.stdout:4/106: write d1/f11 [3955132,26545] 0 2024-03-26T18:54:31.362 INFO:tasks.workunit.client.0.smithi102.stdout:4/107: write d1/d17/d1a/f1c [697097,80029] 0 2024-03-26T18:54:31.362 INFO:tasks.workunit.client.0.smithi102.stdout:1/127: link de/l18 de/d16/d1d/d2a/l2b 0 2024-03-26T18:54:31.363 INFO:tasks.workunit.client.0.smithi102.stdout:1/128: dread - de/d16/d1d/f1e zero size 2024-03-26T18:54:31.363 INFO:tasks.workunit.client.0.smithi102.stdout:1/129: chown de 0 1 2024-03-26T18:54:31.363 INFO:tasks.workunit.client.1.smithi190.stdout:0/554: dwrite d0/dd/d20/d1c/d8b/d8d/db7/fba [0,4194304] 0 2024-03-26T18:54:31.364 INFO:tasks.workunit.client.1.smithi190.stdout:3/399: truncate d1/f26 708234 0 2024-03-26T18:54:31.364 INFO:tasks.workunit.client.0.smithi102.stdout:6/152: dwrite d2/d3/f5 [0,4194304] 0 2024-03-26T18:54:31.364 INFO:tasks.workunit.client.0.smithi102.stdout:6/153: stat d2/d3 0 2024-03-26T18:54:31.364 INFO:tasks.workunit.client.0.smithi102.stdout:6/154: dread - d2/d12/f22 zero size 2024-03-26T18:54:31.364 INFO:tasks.workunit.client.0.smithi102.stdout:6/155: write d2/d7/f8 [8561490,128319] 0 2024-03-26T18:54:31.365 INFO:tasks.workunit.client.1.smithi190.stdout:6/425: read d17/d28/d2c/d77/f69 [6275276,45409] 0 2024-03-26T18:54:31.366 INFO:tasks.workunit.client.0.smithi102.stdout:5/398: dwrite d2/d14/d1a/f1b [0,4194304] 0 2024-03-26T18:54:31.366 INFO:tasks.workunit.client.0.smithi102.stdout:5/399: write d2/d14/d1a/f91 [368723,37636] 0 2024-03-26T18:54:31.366 INFO:tasks.workunit.client.0.smithi102.stdout:5/400: chown d2/f11 27 1 2024-03-26T18:54:31.367 INFO:tasks.workunit.client.0.smithi102.stdout:0/155: dread d6/de/d13/d1f/f27 [0,4194304] 0 2024-03-26T18:54:31.372 INFO:tasks.workunit.client.0.smithi102.stdout:1/130: rmdir de/d16/d1d/d2a 39 2024-03-26T18:54:31.372 INFO:tasks.workunit.client.0.smithi102.stdout:1/131: read f9 [2039433,109514] 0 2024-03-26T18:54:31.372 INFO:tasks.workunit.client.0.smithi102.stdout:1/132: read - de/d16/f29 zero size 2024-03-26T18:54:31.374 INFO:tasks.workunit.client.1.smithi190.stdout:7/545: symlink d3/d6/d9/d1f/d93/lc8 0 2024-03-26T18:54:31.375 INFO:tasks.workunit.client.1.smithi190.stdout:7/546: chown d3/d6/d9/d14/f2a 396 1 2024-03-26T18:54:31.375 INFO:tasks.workunit.client.0.smithi102.stdout:8/115: dread d0/d9/d12/f1f [0,4194304] 0 2024-03-26T18:54:31.379 INFO:tasks.workunit.client.1.smithi190.stdout:4/496: rename d7/d8/d14/d23/d28/d38/f7f to d7/d8/d14/d4b/d7d/da1/fa9 0 2024-03-26T18:54:31.383 INFO:tasks.workunit.client.0.smithi102.stdout:6/156: rmdir d2/d3 39 2024-03-26T18:54:31.383 INFO:tasks.workunit.client.0.smithi102.stdout:6/157: chown d2/d7/l1a 272596 1 2024-03-26T18:54:31.384 INFO:tasks.workunit.client.1.smithi190.stdout:8/489: mknod d9/d39/d67/ca7 0 2024-03-26T18:54:31.384 INFO:tasks.workunit.client.1.smithi190.stdout:8/490: truncate d9/dd/d15/d32/f66 335038 0 2024-03-26T18:54:31.386 INFO:tasks.workunit.client.0.smithi102.stdout:4/108: dwrite d1/ff [0,4194304] 0 2024-03-26T18:54:31.386 INFO:tasks.workunit.client.0.smithi102.stdout:4/109: readlink d1/lc 0 2024-03-26T18:54:31.387 INFO:tasks.workunit.client.0.smithi102.stdout:4/110: fdatasync d1/f9 0 2024-03-26T18:54:31.388 INFO:tasks.workunit.client.0.smithi102.stdout:1/133: mkdir de/d16/d1d/d2a/d2c 0 2024-03-26T18:54:31.395 INFO:tasks.workunit.client.1.smithi190.stdout:3/400: unlink d1/f42 0 2024-03-26T18:54:31.398 INFO:tasks.workunit.client.1.smithi190.stdout:6/426: mknod d17/d28/d2c/d77/d51/c99 0 2024-03-26T18:54:31.398 INFO:tasks.workunit.client.1.smithi190.stdout:6/427: truncate f2 1017534 0 2024-03-26T18:54:31.400 INFO:tasks.workunit.client.0.smithi102.stdout:1/134: dread f9 [0,4194304] 0 2024-03-26T18:54:31.402 INFO:tasks.workunit.client.0.smithi102.stdout:6/158: dread d2/d7/f8 [4194304,4194304] 0 2024-03-26T18:54:31.410 INFO:tasks.workunit.client.1.smithi190.stdout:9/698: dwrite d12/d2d/d1d/d54/f8a [0,4194304] 0 2024-03-26T18:54:31.416 INFO:tasks.workunit.client.0.smithi102.stdout:1/135: creat de/f2d x:0 0 0 2024-03-26T18:54:31.418 INFO:tasks.workunit.client.1.smithi190.stdout:7/547: unlink d3/d6/d9/d1f/d4c/d71/la7 0 2024-03-26T18:54:31.418 INFO:tasks.workunit.client.1.smithi190.stdout:7/548: readlink d3/d6/d9/d14/d36/l42 0 2024-03-26T18:54:31.418 INFO:tasks.workunit.client.1.smithi190.stdout:7/549: creat d3/d6/d9/d14/fc9 x:0 0 0 2024-03-26T18:54:31.418 INFO:tasks.workunit.client.1.smithi190.stdout:7/550: write d3/f7c [501931,69866] 0 2024-03-26T18:54:31.419 INFO:tasks.workunit.client.0.smithi102.stdout:6/159: creat d2/df/f27 x:0 0 0 2024-03-26T18:54:31.419 INFO:tasks.workunit.client.0.smithi102.stdout:6/160: write d2/d12/f22 [676949,66427] 0 2024-03-26T18:54:31.419 INFO:tasks.workunit.client.0.smithi102.stdout:6/161: creat d2/df/f28 x:0 0 0 2024-03-26T18:54:31.420 INFO:tasks.workunit.client.0.smithi102.stdout:4/111: mkdir d1/d27 0 2024-03-26T18:54:31.420 INFO:tasks.workunit.client.1.smithi190.stdout:1/394: write f2 [6663646,114390] 0 2024-03-26T18:54:31.421 INFO:tasks.workunit.client.1.smithi190.stdout:1/395: readlink d3/d9/l10 0 2024-03-26T18:54:31.422 INFO:tasks.workunit.client.1.smithi190.stdout:3/401: link d1/de/d4e/f54 d1/d3/df/f74 0 2024-03-26T18:54:31.422 INFO:tasks.workunit.client.1.smithi190.stdout:3/402: readlink d1/l1e 0 2024-03-26T18:54:31.425 INFO:tasks.workunit.client.1.smithi190.stdout:0/555: dwrite d0/dd/d24/da2/d80/fb2 [0,4194304] 0 2024-03-26T18:54:31.425 INFO:tasks.workunit.client.1.smithi190.stdout:6/428: mknod d17/d28/d74/d7b/c9a 0 2024-03-26T18:54:31.426 INFO:tasks.workunit.client.0.smithi102.stdout:0/156: dwrite d6/de/f1d [0,4194304] 0 2024-03-26T18:54:31.426 INFO:tasks.workunit.client.0.smithi102.stdout:8/116: dwrite d0/d3/f11 [0,4194304] 0 2024-03-26T18:54:31.426 INFO:tasks.workunit.client.0.smithi102.stdout:6/162: truncate d2/d7/f20 291940 0 2024-03-26T18:54:31.427 INFO:tasks.workunit.client.1.smithi190.stdout:0/556: dread d0/dd/d20/fa4 [0,4194304] 0 2024-03-26T18:54:31.427 INFO:tasks.workunit.client.1.smithi190.stdout:0/557: write d0/dd/d24/d4b/d5e/d66/f94 [796200,83377] 0 2024-03-26T18:54:31.427 INFO:tasks.workunit.client.1.smithi190.stdout:0/558: dread - d0/dd/d20/d61/fac zero size 2024-03-26T18:54:31.428 INFO:tasks.workunit.client.1.smithi190.stdout:9/699: dread d12/d2d/d1d/d23/f5f [0,4194304] 0 2024-03-26T18:54:31.428 INFO:tasks.workunit.client.1.smithi190.stdout:8/491: dwrite d9/f12 [8388608,4194304] 0 2024-03-26T18:54:31.428 INFO:tasks.workunit.client.1.smithi190.stdout:4/497: dwrite d7/d8/d14/d23/d28/d38/f62 [4194304,4194304] 0 2024-03-26T18:54:31.429 INFO:tasks.workunit.client.0.smithi102.stdout:0/157: truncate d6/de/d13/d25/d31/f32 719903 0 2024-03-26T18:54:31.434 INFO:tasks.workunit.client.0.smithi102.stdout:8/117: write d0/d9/d12/f1f [3869833,85220] 0 2024-03-26T18:54:31.434 INFO:tasks.workunit.client.0.smithi102.stdout:8/118: fdatasync d0/d3/f11 0 2024-03-26T18:54:31.434 INFO:tasks.workunit.client.0.smithi102.stdout:1/136: dread de/f19 [4194304,4194304] 0 2024-03-26T18:54:31.434 INFO:tasks.workunit.client.0.smithi102.stdout:1/137: write de/d16/d1d/f28 [27377,128504] 0 2024-03-26T18:54:31.435 INFO:tasks.workunit.client.0.smithi102.stdout:1/138: chown de/d16/d1d/l27 28499383 1 2024-03-26T18:54:31.435 INFO:tasks.workunit.client.1.smithi190.stdout:7/551: symlink d3/d1b/d57/lca 0 2024-03-26T18:54:31.435 INFO:tasks.workunit.client.1.smithi190.stdout:3/403: write d1/d3/f10 [11006573,113551] 0 2024-03-26T18:54:31.435 INFO:tasks.workunit.client.0.smithi102.stdout:6/163: unlink d2/d3/f5 0 2024-03-26T18:54:31.435 INFO:tasks.workunit.client.0.smithi102.stdout:6/164: dread - d2/df/f27 zero size 2024-03-26T18:54:31.437 INFO:tasks.workunit.client.0.smithi102.stdout:8/119: creat d0/d3/f21 x:0 0 0 2024-03-26T18:54:31.440 INFO:tasks.workunit.client.0.smithi102.stdout:1/139: mkdir de/d2e 0 2024-03-26T18:54:31.441 INFO:tasks.workunit.client.0.smithi102.stdout:8/120: read d0/d3/f6 [108255,76036] 0 2024-03-26T18:54:31.442 INFO:tasks.workunit.client.1.smithi190.stdout:6/429: stat d17/d28/c32 0 2024-03-26T18:54:31.444 INFO:tasks.workunit.client.0.smithi102.stdout:6/165: rename d2/df/f28 to d2/f29 0 2024-03-26T18:54:31.447 INFO:tasks.workunit.client.0.smithi102.stdout:4/112: dwrite d1/d17/d1a/f1c [0,4194304] 0 2024-03-26T18:54:31.448 INFO:tasks.workunit.client.1.smithi190.stdout:0/559: creat d0/dd/d24/d4b/fc5 x:0 0 0 2024-03-26T18:54:31.451 INFO:tasks.workunit.client.0.smithi102.stdout:4/113: unlink d1/f11 0 2024-03-26T18:54:31.457 INFO:tasks.workunit.client.1.smithi190.stdout:9/700: rename d12/d2d/d1d/d23/d83/dc9/dd4/c84 to d12/cf3 0 2024-03-26T18:54:31.457 INFO:tasks.workunit.client.1.smithi190.stdout:6/430: dread d17/d28/d2b/f60 [0,4194304] 0 2024-03-26T18:54:31.458 INFO:tasks.workunit.client.0.smithi102.stdout:4/114: truncate d1/f9 23957 0 2024-03-26T18:54:31.460 INFO:tasks.workunit.client.1.smithi190.stdout:8/492: mknod d9/dd/d23/ca8 0 2024-03-26T18:54:31.462 INFO:tasks.workunit.client.1.smithi190.stdout:4/498: creat d7/d8/faa x:0 0 0 2024-03-26T18:54:31.463 INFO:tasks.workunit.client.0.smithi102.stdout:4/115: link d1/lc d1/d25/l28 0 2024-03-26T18:54:31.473 INFO:tasks.workunit.client.1.smithi190.stdout:3/404: mkdir d1/d47/d75 0 2024-03-26T18:54:31.473 INFO:tasks.workunit.client.1.smithi190.stdout:3/405: read - d1/de/d4e/f54 zero size 2024-03-26T18:54:31.473 INFO:tasks.workunit.client.1.smithi190.stdout:3/406: creat d1/d3/df/d18/d35/f76 x:0 0 0 2024-03-26T18:54:31.474 INFO:tasks.workunit.client.1.smithi190.stdout:1/396: getdents d3/da 0 2024-03-26T18:54:31.476 INFO:tasks.workunit.client.1.smithi190.stdout:9/701: link d12/d2d/d1d/d23/d83/dc9/dd4/d3f/fc7 d12/d2d/d1d/d23/de5/ff4 0 2024-03-26T18:54:31.479 INFO:tasks.workunit.client.1.smithi190.stdout:6/431: symlink d17/d28/d74/l9b 0 2024-03-26T18:54:31.479 INFO:tasks.workunit.client.0.smithi102.stdout:8/121: dread d0/d9/d12/f1f [0,4194304] 0 2024-03-26T18:54:31.480 INFO:tasks.workunit.client.1.smithi190.stdout:6/432: chown d17/d28/d2c/d77/d51/f7c 57400 1 2024-03-26T18:54:31.480 INFO:tasks.workunit.client.1.smithi190.stdout:8/493: mkdir d9/da9 0 2024-03-26T18:54:31.480 INFO:tasks.workunit.client.1.smithi190.stdout:7/552: dwrite d3/d6/d9/d14/f2a [0,4194304] 0 2024-03-26T18:54:31.482 INFO:tasks.workunit.client.1.smithi190.stdout:4/499: getdents d7 0 2024-03-26T18:54:31.483 INFO:tasks.workunit.client.1.smithi190.stdout:4/500: chown d7/d8/d14/d6e/d6f 169156 1 2024-03-26T18:54:31.483 INFO:tasks.workunit.client.1.smithi190.stdout:4/501: truncate d7/f49 355336 0 2024-03-26T18:54:31.486 INFO:tasks.workunit.client.1.smithi190.stdout:3/407: dread d1/d3/df/d18/d1b/f21 [0,4194304] 0 2024-03-26T18:54:31.488 INFO:tasks.workunit.client.0.smithi102.stdout:1/140: dwrite de/f19 [0,4194304] 0 2024-03-26T18:54:31.488 INFO:tasks.workunit.client.0.smithi102.stdout:1/141: dread - de/f2d zero size 2024-03-26T18:54:31.488 INFO:tasks.workunit.client.1.smithi190.stdout:8/494: mknod d9/dd/d23/caa 0 2024-03-26T18:54:31.489 INFO:tasks.workunit.client.0.smithi102.stdout:0/158: dwrite d6/fb [0,4194304] 0 2024-03-26T18:54:31.490 INFO:tasks.workunit.client.0.smithi102.stdout:4/116: dwrite d1/fa [4194304,4194304] 0 2024-03-26T18:54:31.491 INFO:tasks.workunit.client.0.smithi102.stdout:6/166: dwrite d2/d7/f8 [4194304,4194304] 0 2024-03-26T18:54:31.492 INFO:tasks.workunit.client.0.smithi102.stdout:1/142: symlink de/l2f 0 2024-03-26T18:54:31.492 INFO:tasks.workunit.client.0.smithi102.stdout:1/143: fdatasync de/f10 0 2024-03-26T18:54:31.492 INFO:tasks.workunit.client.0.smithi102.stdout:1/144: write de/f1c [970715,110790] 0 2024-03-26T18:54:31.492 INFO:tasks.workunit.client.1.smithi190.stdout:0/560: dwrite d0/dd/d20/d1c/d46/d7c/faa [0,4194304] 0 2024-03-26T18:54:31.492 INFO:tasks.workunit.client.1.smithi190.stdout:0/561: write d0/dd/d24/da2/f9e [746399,39684] 0 2024-03-26T18:54:31.493 INFO:tasks.workunit.client.1.smithi190.stdout:0/562: creat d0/dd/d24/da2/fc6 x:0 0 0 2024-03-26T18:54:31.493 INFO:tasks.workunit.client.1.smithi190.stdout:0/563: truncate d0/dd/d24/d4b/d5e/d6b/faf 228563 0 2024-03-26T18:54:31.499 INFO:tasks.workunit.client.1.smithi190.stdout:4/502: mknod d7/d8/d1c/cab 0 2024-03-26T18:54:31.500 INFO:tasks.workunit.client.0.smithi102.stdout:8/122: write d0/d15/f16 [2001097,27059] 0 2024-03-26T18:54:31.501 INFO:tasks.workunit.client.1.smithi190.stdout:8/495: symlink d9/dd/d23/d36/d37/d3a/lab 0 2024-03-26T18:54:31.503 INFO:tasks.workunit.client.1.smithi190.stdout:1/397: dwrite d3/da/dd/d2f/d5b/f6b [0,4194304] 0 2024-03-26T18:54:31.503 INFO:tasks.workunit.client.1.smithi190.stdout:1/398: fsync d3/da/d2e/f4a 0 2024-03-26T18:54:31.504 INFO:tasks.workunit.client.1.smithi190.stdout:1/399: write d3/d9/db/f2c [3701296,58477] 0 2024-03-26T18:54:31.504 INFO:tasks.workunit.client.1.smithi190.stdout:4/503: dread d7/d1d/d5e/f76 [0,4194304] 0 2024-03-26T18:54:31.507 INFO:tasks.workunit.client.0.smithi102.stdout:0/159: rmdir d6/de/d13/d1f/d20 39 2024-03-26T18:54:31.511 INFO:tasks.workunit.client.1.smithi190.stdout:0/564: mknod d0/dd/d20/d1c/d4e/cc7 0 2024-03-26T18:54:31.514 INFO:tasks.workunit.client.1.smithi190.stdout:8/496: rename d9/dd/d23/d36/d37/d3a/c4a to d9/da2/d73/cac 0 2024-03-26T18:54:31.516 INFO:tasks.workunit.client.0.smithi102.stdout:6/167: creat d2/d3/f2a x:0 0 0 2024-03-26T18:54:31.516 INFO:tasks.workunit.client.0.smithi102.stdout:6/168: fdatasync d2/d7/f20 0 2024-03-26T18:54:31.518 INFO:tasks.workunit.client.1.smithi190.stdout:9/702: dwrite ff [0,4194304] 0 2024-03-26T18:54:31.518 INFO:tasks.workunit.client.1.smithi190.stdout:3/408: dwrite d1/d3/df/f19 [0,4194304] 0 2024-03-26T18:54:31.519 INFO:tasks.workunit.client.0.smithi102.stdout:1/145: unlink de/f10 0 2024-03-26T18:54:31.525 INFO:tasks.workunit.client.1.smithi190.stdout:5/330: sync 2024-03-26T18:54:31.525 INFO:tasks.workunit.client.1.smithi190.stdout:2/483: sync 2024-03-26T18:54:31.525 INFO:tasks.workunit.client.0.smithi102.stdout:6/169: dread d2/d7/f1d [0,4194304] 0 2024-03-26T18:54:31.526 INFO:tasks.workunit.client.0.smithi102.stdout:6/170: creat d2/df/f2b x:0 0 0 2024-03-26T18:54:31.526 INFO:tasks.workunit.client.0.smithi102.stdout:6/171: truncate d2/f29 33387 0 2024-03-26T18:54:31.526 INFO:tasks.workunit.client.0.smithi102.stdout:8/123: creat d0/d9/d12/f22 x:0 0 0 2024-03-26T18:54:31.527 INFO:tasks.workunit.client.0.smithi102.stdout:0/160: unlink d6/de/l12 0 2024-03-26T18:54:31.529 INFO:tasks.workunit.client.1.smithi190.stdout:4/504: link d7/d8/d1c/d20/l8c d7/d8/d14/d23/d28/d38/d96/d9a/lac 0 2024-03-26T18:54:31.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:31 smithi190 ceph-mon[35043]: pgmap v243: 65 pgs: 65 active+clean; 933 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 60 MiB/s rd, 91 MiB/s wr, 559 op/s 2024-03-26T18:54:31.530 INFO:tasks.workunit.client.0.smithi102.stdout:1/146: link de/ff de/d1f/f30 0 2024-03-26T18:54:31.532 INFO:tasks.workunit.client.1.smithi190.stdout:0/565: mkdir d0/dd/d20/d1c/d8b/d8d/dc8 0 2024-03-26T18:54:31.534 INFO:tasks.workunit.client.1.smithi190.stdout:9/703: mknod d12/d2d/d1d/d23/d83/dd5/cf5 0 2024-03-26T18:54:31.535 INFO:tasks.workunit.client.0.smithi102.stdout:6/172: mknod d2/d7/c2c 0 2024-03-26T18:54:31.535 INFO:tasks.workunit.client.0.smithi102.stdout:6/173: chown d2/l24 3 1 2024-03-26T18:54:31.535 INFO:tasks.workunit.client.0.smithi102.stdout:6/174: creat d2/d12/f2d x:0 0 0 2024-03-26T18:54:31.538 INFO:tasks.workunit.client.0.smithi102.stdout:8/124: rmdir d0/d15/d1b/d20 0 2024-03-26T18:54:31.538 INFO:tasks.workunit.client.0.smithi102.stdout:8/125: chown d0/d3/f11 33 1 2024-03-26T18:54:31.538 INFO:tasks.workunit.client.0.smithi102.stdout:8/126: write d0/d15/f16 [2039174,60427] 0 2024-03-26T18:54:31.538 INFO:tasks.workunit.client.0.smithi102.stdout:8/127: write d0/d15/f16 [1664833,114793] 0 2024-03-26T18:54:31.540 INFO:tasks.workunit.client.0.smithi102.stdout:4/117: dwrite d1/f9 [0,4194304] 0 2024-03-26T18:54:31.540 INFO:tasks.workunit.client.0.smithi102.stdout:0/161: dread d6/f11 [0,4194304] 0 2024-03-26T18:54:31.540 INFO:tasks.workunit.client.0.smithi102.stdout:0/162: write d6/de/d13/d25/d31/f32 [1620791,70349] 0 2024-03-26T18:54:31.543 INFO:tasks.workunit.client.1.smithi190.stdout:3/409: rename d1/de/l53 to d1/d3/d5a/d6e/l77 0 2024-03-26T18:54:31.545 INFO:tasks.workunit.client.0.smithi102.stdout:1/147: symlink de/d16/l31 0 2024-03-26T18:54:31.547 INFO:tasks.workunit.client.0.smithi102.stdout:8/128: dread d0/d3/f14 [0,4194304] 0 2024-03-26T18:54:31.548 INFO:tasks.workunit.client.1.smithi190.stdout:4/505: dwrite d7/f4f [0,4194304] 0 2024-03-26T18:54:31.550 INFO:tasks.workunit.client.1.smithi190.stdout:5/331: link d2/f6 d2/d17/d1b/d30/f7a 0 2024-03-26T18:54:31.553 INFO:tasks.workunit.client.0.smithi102.stdout:6/175: creat d2/f2e x:0 0 0 2024-03-26T18:54:31.553 INFO:tasks.workunit.client.0.smithi102.stdout:6/176: chown d2/df/l18 592 1 2024-03-26T18:54:31.553 INFO:tasks.workunit.client.0.smithi102.stdout:7/90: sync 2024-03-26T18:54:31.554 INFO:tasks.workunit.client.1.smithi190.stdout:1/400: rmdir d3/da/d2e/d59 39 2024-03-26T18:54:31.557 INFO:tasks.workunit.client.0.smithi102.stdout:4/118: dwrite d1/ff [0,4194304] 0 2024-03-26T18:54:31.557 INFO:tasks.workunit.client.0.smithi102.stdout:4/119: getdents d1/d27 0 2024-03-26T18:54:31.557 INFO:tasks.workunit.client.0.smithi102.stdout:4/120: fsync d1/fa 0 2024-03-26T18:54:31.558 INFO:tasks.workunit.client.0.smithi102.stdout:8/129: dread d0/d9/d12/f1f [0,4194304] 0 2024-03-26T18:54:31.558 INFO:tasks.workunit.client.0.smithi102.stdout:8/130: creat d0/d9/d12/f23 x:0 0 0 2024-03-26T18:54:31.558 INFO:tasks.workunit.client.0.smithi102.stdout:8/131: write d0/d9/d12/f22 [607267,68334] 0 2024-03-26T18:54:31.558 INFO:tasks.workunit.client.0.smithi102.stdout:8/132: write d0/d3/f6 [122297,56283] 0 2024-03-26T18:54:31.560 INFO:tasks.workunit.client.1.smithi190.stdout:0/566: symlink d0/dd/d24/d4b/d5e/d6b/lc9 0 2024-03-26T18:54:31.563 INFO:tasks.workunit.client.1.smithi190.stdout:9/704: mknod d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/db3/cf6 0 2024-03-26T18:54:31.563 INFO:tasks.workunit.client.1.smithi190.stdout:9/705: rename d12/d2d/d1d/d23/d83/dc9 to d12/d2d/d1d/d23/d83/dc9/dd4/d71/da7/df7 22 2024-03-26T18:54:31.563 INFO:tasks.workunit.client.1.smithi190.stdout:9/706: creat d12/d2d/d1d/d54/ff8 x:0 0 0 2024-03-26T18:54:31.563 INFO:tasks.workunit.client.1.smithi190.stdout:9/707: truncate d12/d2d/d1d/d23/d83/d91/fb7 515668 0 2024-03-26T18:54:31.563 INFO:tasks.workunit.client.1.smithi190.stdout:9/708: truncate d12/f14 4740516 0 2024-03-26T18:54:31.563 INFO:tasks.workunit.client.1.smithi190.stdout:5/332: read d2/d17/d1b/d22/f28 [3263298,11628] 0 2024-03-26T18:54:31.563 INFO:tasks.workunit.client.1.smithi190.stdout:5/333: truncate d2/d17/d1b/d22/d24/d3b/f47 1233850 0 2024-03-26T18:54:31.564 INFO:tasks.workunit.client.1.smithi190.stdout:5/334: truncate d2/d17/d1b/d22/f2d 979783 0 2024-03-26T18:54:31.568 INFO:tasks.workunit.client.1.smithi190.stdout:3/410: mknod d1/d3/df/c78 0 2024-03-26T18:54:31.568 INFO:tasks.workunit.client.0.smithi102.stdout:1/148: link de/f22 de/f32 0 2024-03-26T18:54:31.570 INFO:tasks.workunit.client.1.smithi190.stdout:9/709: dread d12/d2d/f9f [0,4194304] 0 2024-03-26T18:54:31.570 INFO:tasks.workunit.client.1.smithi190.stdout:9/710: chown d12/d2d/d1d/d23/d6a/c79 54052889 1 2024-03-26T18:54:31.570 INFO:tasks.workunit.client.1.smithi190.stdout:9/711: fsync d12/d2d/d1d/d23/d2b/f68 0 2024-03-26T18:54:31.570 INFO:tasks.workunit.client.1.smithi190.stdout:9/712: creat d12/d2d/d1d/d54/ff9 x:0 0 0 2024-03-26T18:54:31.571 INFO:tasks.workunit.client.0.smithi102.stdout:6/177: rename d2/d3/la to d2/d12/l2f 0 2024-03-26T18:54:31.571 INFO:tasks.workunit.client.0.smithi102.stdout:6/178: chown d2/d3 384144 1 2024-03-26T18:54:31.572 INFO:tasks.workunit.client.0.smithi102.stdout:7/91: symlink d7/d10/d17/l23 0 2024-03-26T18:54:31.574 INFO:tasks.workunit.client.1.smithi190.stdout:2/484: rmdir d6/d21/d61 39 2024-03-26T18:54:31.574 INFO:tasks.workunit.client.1.smithi190.stdout:1/401: dwrite d3/d5f/f71 [0,4194304] 0 2024-03-26T18:54:31.574 INFO:tasks.workunit.client.1.smithi190.stdout:4/506: dwrite d7/d8/f2a [4194304,4194304] 0 2024-03-26T18:54:31.574 INFO:tasks.workunit.client.1.smithi190.stdout:1/402: write d3/d9/f16 [1580100,83211] 0 2024-03-26T18:54:31.575 INFO:tasks.workunit.client.0.smithi102.stdout:4/121: mknod d1/d27/c29 0 2024-03-26T18:54:31.579 INFO:tasks.workunit.client.1.smithi190.stdout:3/411: write d1/de/d1c/f2b [1624643,15435] 0 2024-03-26T18:54:31.580 INFO:tasks.workunit.client.0.smithi102.stdout:8/133: creat d0/d9/f24 x:0 0 0 2024-03-26T18:54:31.580 INFO:tasks.workunit.client.0.smithi102.stdout:8/134: readlink d0/le 0 2024-03-26T18:54:31.580 INFO:tasks.workunit.client.0.smithi102.stdout:8/135: write d0/d9/d12/f23 [753446,91499] 0 2024-03-26T18:54:31.582 INFO:tasks.workunit.client.1.smithi190.stdout:5/335: rename d2/f5e to d2/d17/f7b 0 2024-03-26T18:54:31.582 INFO:tasks.workunit.client.1.smithi190.stdout:5/336: chown d2/d17/d1b/d22/l32 439651 1 2024-03-26T18:54:31.582 INFO:tasks.workunit.client.0.smithi102.stdout:0/163: rmdir d6/de/d19 39 2024-03-26T18:54:31.587 INFO:tasks.workunit.client.1.smithi190.stdout:9/713: link d12/d2d/d1d/d23/d83/dc9/dd4/d3f/lc5 d12/d2d/d1d/da1/dba/dd2/lfa 0 2024-03-26T18:54:31.587 INFO:tasks.workunit.client.1.smithi190.stdout:9/714: chown d12/d2d/d1d/d23/d83/dc9/dd4/d71/cd8 373 1 2024-03-26T18:54:31.594 INFO:tasks.workunit.client.0.smithi102.stdout:6/179: dwrite d2/d12/f22 [0,4194304] 0 2024-03-26T18:54:31.594 INFO:tasks.workunit.client.0.smithi102.stdout:6/180: readlink d2/d7/l1b 0 2024-03-26T18:54:31.594 INFO:tasks.workunit.client.0.smithi102.stdout:6/181: truncate d2/d7/f20 488520 0 2024-03-26T18:54:31.594 INFO:tasks.workunit.client.0.smithi102.stdout:1/149: unlink de/ff 0 2024-03-26T18:54:31.595 INFO:tasks.workunit.client.0.smithi102.stdout:1/150: readlink de/d16/l31 0 2024-03-26T18:54:31.595 INFO:tasks.workunit.client.1.smithi190.stdout:4/507: dread d7/d1d/d5e/f81 [0,4194304] 0 2024-03-26T18:54:31.598 INFO:tasks.workunit.client.1.smithi190.stdout:2/485: creat d6/d9/d45/d6b/fb0 x:0 0 0 2024-03-26T18:54:31.598 INFO:tasks.workunit.client.1.smithi190.stdout:2/486: write d6/d9/d3a/d3d/d2e/d31/f91 [1310012,47241] 0 2024-03-26T18:54:31.599 INFO:tasks.workunit.client.0.smithi102.stdout:4/122: mkdir d1/d17/d2a 0 2024-03-26T18:54:31.600 INFO:tasks.workunit.client.1.smithi190.stdout:1/403: chown d3/da/dd/d2f/d34/f35 992 1 2024-03-26T18:54:31.600 INFO:tasks.workunit.client.1.smithi190.stdout:1/404: fsync d3/d9/f42 0 2024-03-26T18:54:31.600 INFO:tasks.workunit.client.1.smithi190.stdout:1/405: dread - d3/da/dd/f7c zero size 2024-03-26T18:54:31.602 INFO:tasks.workunit.client.0.smithi102.stdout:8/136: rename d0/d9/d12/l18 to d0/d15/l25 0 2024-03-26T18:54:31.602 INFO:tasks.workunit.client.0.smithi102.stdout:8/137: truncate d0/d3/f21 961432 0 2024-03-26T18:54:31.602 INFO:tasks.workunit.client.1.smithi190.stdout:0/567: rmdir d0/dd/d24/d87 39 2024-03-26T18:54:31.602 INFO:tasks.workunit.client.1.smithi190.stdout:0/568: truncate d0/dd/d24/da2/f9e 1082749 0 2024-03-26T18:54:31.605 INFO:tasks.workunit.client.0.smithi102.stdout:0/164: rmdir d6/de/d19 39 2024-03-26T18:54:31.605 INFO:tasks.workunit.client.1.smithi190.stdout:3/412: dwrite d1/d3/f4b [0,4194304] 0 2024-03-26T18:54:31.605 INFO:tasks.workunit.client.1.smithi190.stdout:3/413: read - d1/d3/df/d31/d5b/f6b zero size 2024-03-26T18:54:31.605 INFO:tasks.workunit.client.1.smithi190.stdout:3/414: readlink d1/l1e 0 2024-03-26T18:54:31.605 INFO:tasks.workunit.client.1.smithi190.stdout:3/415: readlink d1/l23 0 2024-03-26T18:54:31.605 INFO:tasks.workunit.client.1.smithi190.stdout:3/416: chown d1/d3 3416 1 2024-03-26T18:54:31.607 INFO:tasks.workunit.client.1.smithi190.stdout:5/337: dwrite d2/d17/d1b/d1e/f5f [0,4194304] 0 2024-03-26T18:54:31.610 INFO:tasks.workunit.client.1.smithi190.stdout:4/508: symlink d7/d1d/d5e/d98/lad 0 2024-03-26T18:54:31.612 INFO:tasks.workunit.client.1.smithi190.stdout:1/406: creat d3/d5f/f87 x:0 0 0 2024-03-26T18:54:31.612 INFO:tasks.workunit.client.1.smithi190.stdout:1/407: chown d3/d9/f41 8171676 1 2024-03-26T18:54:31.612 INFO:tasks.workunit.client.1.smithi190.stdout:1/408: write d3/da/dd/d2f/d5b/f6b [4334580,24779] 0 2024-03-26T18:54:31.613 INFO:tasks.workunit.client.1.smithi190.stdout:3/417: rename d1/d3/df/d18/d1b/c32 to d1/d3/df/d18/d57/c79 0 2024-03-26T18:54:31.613 INFO:tasks.workunit.client.1.smithi190.stdout:3/418: write d1/de/f69 [538553,127205] 0 2024-03-26T18:54:31.618 INFO:tasks.workunit.client.0.smithi102.stdout:3/286: sync 2024-03-26T18:54:31.619 INFO:tasks.workunit.client.0.smithi102.stdout:1/151: creat de/d1f/f33 x:0 0 0 2024-03-26T18:54:31.620 INFO:tasks.workunit.client.0.smithi102.stdout:1/152: chown de/f22 8360131 1 2024-03-26T18:54:31.620 INFO:tasks.workunit.client.0.smithi102.stdout:1/153: creat de/d16/f34 x:0 0 0 2024-03-26T18:54:31.621 INFO:tasks.workunit.client.0.smithi102.stdout:7/92: getdents d7/d10 0 2024-03-26T18:54:31.624 INFO:tasks.workunit.client.0.smithi102.stdout:4/123: creat d1/d27/f2b x:0 0 0 2024-03-26T18:54:31.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:31 smithi102 ceph-mon[29002]: pgmap v243: 65 pgs: 65 active+clean; 933 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 60 MiB/s rd, 91 MiB/s wr, 559 op/s 2024-03-26T18:54:31.628 INFO:tasks.workunit.client.0.smithi102.stdout:8/138: getdents d0/d15/d1b 0 2024-03-26T18:54:31.629 INFO:tasks.workunit.client.0.smithi102.stdout:9/89: sync 2024-03-26T18:54:31.629 INFO:tasks.workunit.client.0.smithi102.stdout:5/401: sync 2024-03-26T18:54:31.629 INFO:tasks.workunit.client.0.smithi102.stdout:2/159: sync 2024-03-26T18:54:31.630 INFO:tasks.workunit.client.0.smithi102.stdout:3/287: dread df/f20 [0,4194304] 0 2024-03-26T18:54:31.631 INFO:tasks.workunit.client.0.smithi102.stdout:0/165: symlink d6/de/d13/d25/l34 0 2024-03-26T18:54:31.632 INFO:tasks.workunit.client.1.smithi190.stdout:2/487: dwrite d6/d9/d3a/d3d/d2e/f9a [0,4194304] 0 2024-03-26T18:54:31.632 INFO:tasks.workunit.client.1.smithi190.stdout:2/488: readlink d6/d9/d3a/d3d/d2e/l84 0 2024-03-26T18:54:31.632 INFO:tasks.workunit.client.1.smithi190.stdout:0/569: dwrite d0/dd/d24/da2/fc6 [0,4194304] 0 2024-03-26T18:54:31.632 INFO:tasks.workunit.client.1.smithi190.stdout:3/419: dwrite d1/d3/df/d18/d1b/f21 [0,4194304] 0 2024-03-26T18:54:31.632 INFO:tasks.workunit.client.1.smithi190.stdout:3/420: stat d1/d3/df/d18/d35/d41/l4c 0 2024-03-26T18:54:31.640 INFO:tasks.workunit.client.0.smithi102.stdout:1/154: rename de/d16/f20 to de/d16/f35 0 2024-03-26T18:54:31.647 INFO:tasks.workunit.client.0.smithi102.stdout:4/124: creat d1/d17/d1a/d21/f2c x:0 0 0 2024-03-26T18:54:31.647 INFO:tasks.workunit.client.0.smithi102.stdout:4/125: readlink d1/lc 0 2024-03-26T18:54:31.650 INFO:tasks.workunit.client.0.smithi102.stdout:8/139: unlink d0/d9/c1c 0 2024-03-26T18:54:31.652 INFO:tasks.workunit.client.0.smithi102.stdout:0/166: creat d6/d2d/f35 x:0 0 0 2024-03-26T18:54:31.652 INFO:tasks.workunit.client.0.smithi102.stdout:0/167: fdatasync d6/de/d19/f30 0 2024-03-26T18:54:31.653 INFO:tasks.workunit.client.1.smithi190.stdout:5/338: truncate d2/f6 136642 0 2024-03-26T18:54:31.653 INFO:tasks.workunit.client.1.smithi190.stdout:5/339: write d2/d17/d62/d64/f70 [336825,39020] 0 2024-03-26T18:54:31.653 INFO:tasks.workunit.client.1.smithi190.stdout:5/340: dread - d2/d17/d1b/d22/d68/f75 zero size 2024-03-26T18:54:31.653 INFO:tasks.workunit.client.1.smithi190.stdout:5/341: fdatasync d2/d17/d1b/d22/f2d 0 2024-03-26T18:54:31.654 INFO:tasks.workunit.client.1.smithi190.stdout:9/715: truncate ff 3363201 0 2024-03-26T18:54:31.654 INFO:tasks.workunit.client.1.smithi190.stdout:9/716: write d12/fc6 [160959,90545] 0 2024-03-26T18:54:31.654 INFO:tasks.workunit.client.0.smithi102.stdout:7/93: rmdir d7/d12 39 2024-03-26T18:54:31.655 INFO:tasks.workunit.client.1.smithi190.stdout:4/509: mknod d7/d1d/d42/d4a/cae 0 2024-03-26T18:54:31.656 INFO:tasks.workunit.client.0.smithi102.stdout:4/126: unlink d1/fa 0 2024-03-26T18:54:31.657 INFO:tasks.workunit.client.1.smithi190.stdout:1/409: mkdir d3/d73/d88 0 2024-03-26T18:54:31.657 INFO:tasks.workunit.client.1.smithi190.stdout:1/410: fdatasync d3/da/dd/d2f/d5b/f63 0 2024-03-26T18:54:31.658 INFO:tasks.workunit.client.0.smithi102.stdout:0/168: creat d6/f36 x:0 0 0 2024-03-26T18:54:31.660 INFO:tasks.workunit.client.0.smithi102.stdout:7/94: getdents d7/d12 0 2024-03-26T18:54:31.660 INFO:tasks.workunit.client.0.smithi102.stdout:7/95: chown d7/f20 12 1 2024-03-26T18:54:31.663 INFO:tasks.workunit.client.1.smithi190.stdout:2/489: mkdir d6/d9/d1a/d72/db1 0 2024-03-26T18:54:31.663 INFO:tasks.workunit.client.1.smithi190.stdout:5/342: creat d2/d17/d1b/d60/f7c x:0 0 0 2024-03-26T18:54:31.664 INFO:tasks.workunit.client.1.smithi190.stdout:5/343: fsync d2/d17/f21 0 2024-03-26T18:54:31.664 INFO:tasks.workunit.client.1.smithi190.stdout:5/344: readlink d2/d17/d1b/d1e/d3a/d3c/l50 0 2024-03-26T18:54:31.664 INFO:tasks.workunit.client.1.smithi190.stdout:9/717: creat d12/d2d/d1d/da1/dba/dbc/ffb x:0 0 0 2024-03-26T18:54:31.667 INFO:tasks.workunit.client.1.smithi190.stdout:4/510: mknod d7/d1d/d5e/d98/caf 0 2024-03-26T18:54:31.667 INFO:tasks.workunit.client.1.smithi190.stdout:0/570: dread d0/dd/d24/f7a [4194304,4194304] 0 2024-03-26T18:54:31.668 INFO:tasks.workunit.client.1.smithi190.stdout:0/571: creat d0/dd/d24/d4b/d5e/fca x:0 0 0 2024-03-26T18:54:31.670 INFO:tasks.workunit.client.1.smithi190.stdout:5/345: symlink d2/d17/d1b/d1e/d3a/d3c/l7d 0 2024-03-26T18:54:31.670 INFO:tasks.workunit.client.1.smithi190.stdout:0/572: rmdir d0/dd/d20/d1c 39 2024-03-26T18:54:31.671 INFO:tasks.workunit.client.1.smithi190.stdout:0/573: stat d0/dd/d24/d4b/d5e/d81/c62 0 2024-03-26T18:54:31.673 INFO:tasks.workunit.client.0.smithi102.stdout:7/96: write f6 [7917246,44977] 0 2024-03-26T18:54:31.673 INFO:tasks.workunit.client.0.smithi102.stdout:5/402: dwrite d2/d2c/d38/f49 [0,4194304] 0 2024-03-26T18:54:31.674 INFO:tasks.workunit.client.1.smithi190.stdout:0/574: mkdir d0/dd/d24/d4b/d5e/d66/dcb 0 2024-03-26T18:54:31.674 INFO:tasks.workunit.client.1.smithi190.stdout:0/575: write d0/dd/d20/d1c/d4e/f74 [357617,112592] 0 2024-03-26T18:54:31.674 INFO:tasks.workunit.client.1.smithi190.stdout:0/576: truncate d0/dd/d24/d4b/d5e/fca 696272 0 2024-03-26T18:54:31.678 INFO:tasks.workunit.client.0.smithi102.stdout:9/90: dwrite df/f13 [0,4194304] 0 2024-03-26T18:54:31.679 INFO:tasks.workunit.client.0.smithi102.stdout:7/97: creat d7/d10/f24 x:0 0 0 2024-03-26T18:54:31.680 INFO:tasks.workunit.client.1.smithi190.stdout:2/490: write d6/f36 [2217395,27283] 0 2024-03-26T18:54:31.686 INFO:tasks.workunit.client.1.smithi190.stdout:2/491: link d6/d9/d3a/d3d/d2e/l84 d6/d9/d3a/d3d/d2e/d31/lb2 0 2024-03-26T18:54:31.686 INFO:tasks.workunit.client.0.smithi102.stdout:5/403: getdents d2/d14 0 2024-03-26T18:54:31.687 INFO:tasks.workunit.client.0.smithi102.stdout:9/91: getdents df 0 2024-03-26T18:54:31.687 INFO:tasks.workunit.client.0.smithi102.stdout:9/92: chown df/f17 27938963 1 2024-03-26T18:54:31.689 INFO:tasks.workunit.client.1.smithi190.stdout:2/492: symlink d6/lb3 0 2024-03-26T18:54:31.689 INFO:tasks.workunit.client.0.smithi102.stdout:7/98: symlink d7/l25 0 2024-03-26T18:54:31.691 INFO:tasks.workunit.client.0.smithi102.stdout:0/169: read d6/de/f15 [4172417,126442] 0 2024-03-26T18:54:31.691 INFO:tasks.workunit.client.0.smithi102.stdout:0/170: chown d6/fb 144593856 1 2024-03-26T18:54:31.691 INFO:tasks.workunit.client.0.smithi102.stdout:5/404: mkdir d2/d14/d93 0 2024-03-26T18:54:31.692 INFO:tasks.workunit.client.0.smithi102.stdout:2/160: dwrite d3/d2a/f34 [0,4194304] 0 2024-03-26T18:54:31.692 INFO:tasks.workunit.client.0.smithi102.stdout:2/161: creat d3/d2a/f37 x:0 0 0 2024-03-26T18:54:31.692 INFO:tasks.workunit.client.0.smithi102.stdout:5/405: chown d2/d14/c1f 13 1 2024-03-26T18:54:31.700 INFO:tasks.workunit.client.0.smithi102.stdout:2/162: mkdir d3/d11/d15/d38 0 2024-03-26T18:54:31.705 INFO:tasks.workunit.client.0.smithi102.stdout:1/155: rename de/f1c to de/d1f/f36 0 2024-03-26T18:54:31.706 INFO:tasks.workunit.client.0.smithi102.stdout:5/406: dread d2/f3a [0,4194304] 0 2024-03-26T18:54:31.708 INFO:tasks.workunit.client.0.smithi102.stdout:7/99: dread d7/d12/f22 [0,4194304] 0 2024-03-26T18:54:31.708 INFO:tasks.workunit.client.0.smithi102.stdout:7/100: getdents d7/d12 0 2024-03-26T18:54:31.708 INFO:tasks.workunit.client.0.smithi102.stdout:7/101: chown d7/d10/d17/l23 0 1 2024-03-26T18:54:31.708 INFO:tasks.workunit.client.0.smithi102.stdout:7/102: fdatasync d7/d10/f24 0 2024-03-26T18:54:31.709 INFO:tasks.workunit.client.1.smithi190.stdout:1/411: dwrite d3/da/dd/d2f/f5e [0,4194304] 0 2024-03-26T18:54:31.710 INFO:tasks.workunit.client.1.smithi190.stdout:1/412: write d3/da/dd/d2f/d34/d3e/d52/f5a [97475,34453] 0 2024-03-26T18:54:31.710 INFO:tasks.workunit.client.0.smithi102.stdout:1/156: mknod de/d16/d1d/c37 0 2024-03-26T18:54:31.711 INFO:tasks.workunit.client.0.smithi102.stdout:2/163: dread d3/d2a/f2f [0,4194304] 0 2024-03-26T18:54:31.711 INFO:tasks.workunit.client.0.smithi102.stdout:3/288: dwrite df/f1b [8388608,4194304] 0 2024-03-26T18:54:31.712 INFO:tasks.workunit.client.1.smithi190.stdout:7/553: sync 2024-03-26T18:54:31.712 INFO:tasks.workunit.client.1.smithi190.stdout:6/433: sync 2024-03-26T18:54:31.712 INFO:tasks.workunit.client.1.smithi190.stdout:6/434: read - d17/d28/d2c/d63/f70 zero size 2024-03-26T18:54:31.712 INFO:tasks.workunit.client.1.smithi190.stdout:9/718: dwrite d12/d2d/d1d/d23/d83/dc9/dd4/d3f/d55/f62 [0,4194304] 0 2024-03-26T18:54:31.712 INFO:tasks.workunit.client.1.smithi190.stdout:6/435: creat d17/d28/d2c/d50/f9c x:0 0 0 2024-03-26T18:54:31.713 INFO:tasks.workunit.client.1.smithi190.stdout:9/719: stat d12/d2d/d1d/d23/f5f 0 2024-03-26T18:54:31.713 INFO:tasks.workunit.client.1.smithi190.stdout:8/497: sync 2024-03-26T18:54:31.713 INFO:tasks.workunit.client.1.smithi190.stdout:0/577: dwrite d0/dd/d24/d4b/fc5 [0,4194304] 0 2024-03-26T18:54:31.714 INFO:tasks.workunit.client.0.smithi102.stdout:5/407: chown d2/d14/d1a/f91 489 1 2024-03-26T18:54:31.714 INFO:tasks.workunit.client.0.smithi102.stdout:5/408: read - d2/d25/d2a/d4f/f52 zero size 2024-03-26T18:54:31.714 INFO:tasks.workunit.client.1.smithi190.stdout:9/720: read d12/f38 [319075,108777] 0 2024-03-26T18:54:31.714 INFO:tasks.workunit.client.1.smithi190.stdout:4/511: dwrite d7/f18 [0,4194304] 0 2024-03-26T18:54:31.715 INFO:tasks.workunit.client.0.smithi102.stdout:2/164: dread d3/d6/f2b [0,4194304] 0 2024-03-26T18:54:31.716 INFO:tasks.workunit.client.0.smithi102.stdout:7/103: unlink d7/l8 0 2024-03-26T18:54:31.719 INFO:tasks.workunit.client.0.smithi102.stdout:1/157: symlink de/d16/d1d/d2a/l38 0 2024-03-26T18:54:31.720 INFO:tasks.workunit.client.1.smithi190.stdout:7/554: creat d3/fcb x:0 0 0 2024-03-26T18:54:31.721 INFO:tasks.workunit.client.1.smithi190.stdout:2/493: dwrite d6/d9/d48/da0/fa7 [0,4194304] 0 2024-03-26T18:54:31.721 INFO:tasks.workunit.client.1.smithi190.stdout:2/494: chown d6/d9/d10/f8c 1743 1 2024-03-26T18:54:31.722 INFO:tasks.workunit.client.1.smithi190.stdout:2/495: fdatasync d6/d9/d1a/f26 0 2024-03-26T18:54:31.724 INFO:tasks.workunit.client.0.smithi102.stdout:2/165: rmdir d3/d11/d15/d38 0 2024-03-26T18:54:31.724 INFO:tasks.workunit.client.0.smithi102.stdout:5/409: dread d2/f62 [0,4194304] 0 2024-03-26T18:54:31.724 INFO:tasks.workunit.client.0.smithi102.stdout:5/410: write d2/d2c/d38/f87 [884394,104579] 0 2024-03-26T18:54:31.725 INFO:tasks.workunit.client.1.smithi190.stdout:6/436: symlink d17/d28/d80/l9d 0 2024-03-26T18:54:31.725 INFO:tasks.workunit.client.1.smithi190.stdout:6/437: chown d17/d28/d2c/d77/f8d 14192937 1 2024-03-26T18:54:31.725 INFO:tasks.workunit.client.1.smithi190.stdout:6/438: fdatasync d17/d28/d2c/f41 0 2024-03-26T18:54:31.726 INFO:tasks.workunit.client.0.smithi102.stdout:8/140: rename d0/d9/d12 to d0/d3/d26 0 2024-03-26T18:54:31.728 INFO:tasks.workunit.client.1.smithi190.stdout:8/498: mkdir d9/d39/d79/dad 0 2024-03-26T18:54:31.729 INFO:tasks.workunit.client.0.smithi102.stdout:9/93: dwrite df/f13 [0,4194304] 0 2024-03-26T18:54:31.730 INFO:tasks.workunit.client.0.smithi102.stdout:7/104: write d7/fb [600074,90572] 0 2024-03-26T18:54:31.730 INFO:tasks.workunit.client.0.smithi102.stdout:3/289: getdents df/d29/d31/d36 0 2024-03-26T18:54:31.730 INFO:tasks.workunit.client.0.smithi102.stdout:3/290: chown df/d1c/f2a 1783 1 2024-03-26T18:54:31.730 INFO:tasks.workunit.client.0.smithi102.stdout:3/291: read - df/d61/f66 zero size 2024-03-26T18:54:31.734 INFO:tasks.workunit.client.1.smithi190.stdout:0/578: mkdir d0/dd/d24/d87/d93/db5/dcc 0 2024-03-26T18:54:31.738 INFO:tasks.workunit.client.0.smithi102.stdout:2/166: write d3/d6/ff [2162864,4182] 0 2024-03-26T18:54:31.738 INFO:tasks.workunit.client.0.smithi102.stdout:2/167: write d3/d6/f24 [884163,125743] 0 2024-03-26T18:54:31.738 INFO:tasks.workunit.client.0.smithi102.stdout:4/127: rename d1/d27/f2b to d1/d17/d1f/f2d 0 2024-03-26T18:54:31.739 INFO:tasks.workunit.client.0.smithi102.stdout:0/171: rename d6/de to d6/de/d13/d37 22 2024-03-26T18:54:31.740 INFO:tasks.workunit.client.0.smithi102.stdout:8/141: mkdir d0/d3/d26/d19/d27 0 2024-03-26T18:54:31.740 INFO:tasks.workunit.client.0.smithi102.stdout:8/142: write d0/d3/f6 [1102822,19242] 0 2024-03-26T18:54:31.740 INFO:tasks.workunit.client.0.smithi102.stdout:8/143: truncate d0/d3/f21 1843725 0 2024-03-26T18:54:31.742 INFO:tasks.workunit.client.1.smithi190.stdout:9/721: truncate d12/d2d/d1d/d23/d83/dc9/dd4/d71/f85 2334911 0 2024-03-26T18:54:31.743 INFO:tasks.workunit.client.1.smithi190.stdout:6/439: dwrite fc [0,4194304] 0 2024-03-26T18:54:31.744 INFO:tasks.workunit.client.0.smithi102.stdout:8/144: dread d0/d3/d26/f22 [0,4194304] 0 2024-03-26T18:54:31.744 INFO:tasks.workunit.client.0.smithi102.stdout:7/105: truncate d7/ff 5094758 0 2024-03-26T18:54:31.744 INFO:tasks.workunit.client.0.smithi102.stdout:7/106: readlink d7/d10/d17/l1e 0 2024-03-26T18:54:31.745 INFO:tasks.workunit.client.1.smithi190.stdout:4/512: creat d7/d1d/d5e/d98/fb0 x:0 0 0 2024-03-26T18:54:31.746 INFO:tasks.workunit.client.0.smithi102.stdout:9/94: mknod df/c19 0 2024-03-26T18:54:31.747 INFO:tasks.workunit.client.1.smithi190.stdout:1/413: symlink d3/da/dd/d2f/d34/l89 0 2024-03-26T18:54:31.747 INFO:tasks.workunit.client.1.smithi190.stdout:1/414: fsync d3/da/d2e/f4a 0 2024-03-26T18:54:31.748 INFO:tasks.workunit.client.0.smithi102.stdout:3/292: creat df/f67 x:0 0 0 2024-03-26T18:54:31.749 INFO:tasks.workunit.client.1.smithi190.stdout:4/513: write d7/d8/d14/f57 [3247485,68334] 0 2024-03-26T18:54:31.749 INFO:tasks.workunit.client.1.smithi190.stdout:4/514: chown d7/d1d/d42/d4a/fa6 1118159 1 2024-03-26T18:54:31.749 INFO:tasks.workunit.client.1.smithi190.stdout:7/555: mkdir d3/d6/d9/d1f/d4c/d71/dcc 0 2024-03-26T18:54:31.751 INFO:tasks.workunit.client.0.smithi102.stdout:5/411: getdents d2/d2c/d38/d43 0 2024-03-26T18:54:31.752 INFO:tasks.workunit.client.0.smithi102.stdout:8/145: dread d0/d15/f16 [0,4194304] 0 2024-03-26T18:54:31.753 INFO:tasks.workunit.client.0.smithi102.stdout:8/146: creat d0/d3/f28 x:0 0 0 2024-03-26T18:54:31.755 INFO:tasks.workunit.client.1.smithi190.stdout:2/496: mkdir d6/d21/d61/d6c/db4 0 2024-03-26T18:54:31.756 INFO:tasks.workunit.client.0.smithi102.stdout:2/168: mkdir d3/d11/d15/d31/d39 0 2024-03-26T18:54:31.757 INFO:tasks.workunit.client.1.smithi190.stdout:8/499: getdents d9/dd/d23/d88 0 2024-03-26T18:54:31.758 INFO:tasks.workunit.client.0.smithi102.stdout:8/147: write d0/d3/f14 [1916885,59599] 0 2024-03-26T18:54:31.758 INFO:tasks.workunit.client.0.smithi102.stdout:8/148: dread - d0/d9/f24 zero size 2024-03-26T18:54:31.762 INFO:tasks.workunit.client.1.smithi190.stdout:0/579: mknod d0/dd/d20/d1c/d46/da7/ccd 0 2024-03-26T18:54:31.762 INFO:tasks.workunit.client.0.smithi102.stdout:4/128: mknod d1/d17/d2a/c2e 0 2024-03-26T18:54:31.763 INFO:tasks.workunit.client.0.smithi102.stdout:0/172: truncate d6/de/d19/f1c 1216171 0 2024-03-26T18:54:31.763 INFO:tasks.workunit.client.0.smithi102.stdout:0/173: creat d6/d2d/f38 x:0 0 0 2024-03-26T18:54:31.763 INFO:tasks.workunit.client.0.smithi102.stdout:0/174: write d6/de/d19/f30 [644748,114076] 0 2024-03-26T18:54:31.777 INFO:tasks.workunit.client.1.smithi190.stdout:9/722: rmdir d12/d2d/d1d/d23/d8e 39 2024-03-26T18:54:31.780 INFO:tasks.workunit.client.0.smithi102.stdout:1/158: dwrite de/d1f/f36 [0,4194304] 0 2024-03-26T18:54:31.780 INFO:tasks.workunit.client.1.smithi190.stdout:6/440: symlink d17/d28/d46/l9e 0 2024-03-26T18:54:31.781 INFO:tasks.workunit.client.0.smithi102.stdout:4/129: dwrite d1/d17/d1a/d21/f2c [0,4194304] 0 2024-03-26T18:54:31.784 INFO:tasks.workunit.client.1.smithi190.stdout:1/415: creat d3/da/dd/d2f/d33/f8a x:0 0 0 2024-03-26T18:54:31.786 INFO:tasks.workunit.client.1.smithi190.stdout:4/515: stat d7/d8/d1c/l25 0 2024-03-26T18:54:31.789 INFO:tasks.workunit.client.1.smithi190.stdout:7/556: unlink d3/d6/d9/d1f/d41/c80 0 2024-03-26T18:54:31.791 INFO:tasks.workunit.client.1.smithi190.stdout:2/497: getdents d6/d9/d3a/d3d 0 2024-03-26T18:54:31.791 INFO:tasks.workunit.client.1.smithi190.stdout:2/498: creat d6/d21/fb5 x:0 0 0 2024-03-26T18:54:31.792 INFO:tasks.workunit.client.0.smithi102.stdout:9/95: truncate f7 2614803 0 2024-03-26T18:54:31.793 INFO:tasks.workunit.client.1.smithi190.stdout:8/500: symlink d9/d39/d79/dad/lae 0 2024-03-26T18:54:31.794 INFO:tasks.workunit.client.1.smithi190.stdout:8/501: write d9/dd/d15/f4b [224199,80954] 0 2024-03-26T18:54:31.796 INFO:tasks.workunit.client.1.smithi190.stdout:0/580: creat d0/dd/d20/d1c/d8b/d8d/dc8/fce x:0 0 0 2024-03-26T18:54:31.801 INFO:tasks.workunit.client.0.smithi102.stdout:2/169: symlink d3/d2a/l3a 0 2024-03-26T18:54:31.801 INFO:tasks.workunit.client.0.smithi102.stdout:6/182: sync 2024-03-26T18:54:31.801 INFO:tasks.workunit.client.1.smithi190.stdout:7/557: readlink d3/d6/d9/d1f/d41/d50/d53/d87/lb8 0 2024-03-26T18:54:31.803 INFO:tasks.workunit.client.0.smithi102.stdout:8/149: creat d0/d3/d26/d19/d27/f29 x:0 0 0 2024-03-26T18:54:31.803 INFO:tasks.workunit.client.0.smithi102.stdout:8/150: dread - d0/d3/d26/d19/d27/f29 zero size 2024-03-26T18:54:31.804 INFO:tasks.workunit.client.0.smithi102.stdout:8/151: readlink d0/le 0 2024-03-26T18:54:31.805 INFO:tasks.workunit.client.1.smithi190.stdout:0/581: symlink d0/dd/d24/da2/lcf 0 2024-03-26T18:54:31.805 INFO:tasks.workunit.client.1.smithi190.stdout:0/582: write d0/dd/d20/d1c/f28 [49774,65954] 0 2024-03-26T18:54:31.806 INFO:tasks.workunit.client.0.smithi102.stdout:0/175: rename d6/de/d13/d1f/d20 to d6/de/d13/d1f/d39 0 2024-03-26T18:54:31.806 INFO:tasks.workunit.client.0.smithi102.stdout:0/176: stat d6/d2d/f38 0 2024-03-26T18:54:31.806 INFO:tasks.workunit.client.1.smithi190.stdout:1/416: getdents d3/d5f 0 2024-03-26T18:54:31.808 INFO:tasks.workunit.client.1.smithi190.stdout:7/558: rename d3/d6/d9/d1f/f6c to d3/d6/d9/fcd 0 2024-03-26T18:54:31.810 INFO:tasks.workunit.client.1.smithi190.stdout:2/499: getdents d6/d9/d1a 0 2024-03-26T18:54:31.811 INFO:tasks.workunit.client.0.smithi102.stdout:6/183: dread d2/d12/f22 [0,4194304] 0 2024-03-26T18:54:31.812 INFO:tasks.workunit.client.1.smithi190.stdout:0/583: symlink d0/dd/ld0 0 2024-03-26T18:54:31.813 INFO:tasks.workunit.client.1.smithi190.stdout:1/417: symlink d3/da/dd/d2f/d5b/d83/l8b 0 2024-03-26T18:54:31.814 INFO:tasks.workunit.client.1.smithi190.stdout:7/559: mknod d3/d6/d9/d1f/d4c/d71/cce 0 2024-03-26T18:54:31.814 INFO:tasks.workunit.client.1.smithi190.stdout:7/560: creat d3/d6/d9/d1f/d41/d50/d53/d8f/fcf x:0 0 0 2024-03-26T18:54:31.815 INFO:tasks.workunit.client.1.smithi190.stdout:0/584: link d0/dd/d24/da2/lcf d0/dd/d20/d1c/d46/ld1 0 2024-03-26T18:54:31.816 INFO:tasks.workunit.client.0.smithi102.stdout:7/107: symlink d7/l26 0 2024-03-26T18:54:31.816 INFO:tasks.workunit.client.1.smithi190.stdout:1/418: rmdir d3/da 39 2024-03-26T18:54:31.816 INFO:tasks.workunit.client.1.smithi190.stdout:1/419: fsync d3/d9/f20 0 2024-03-26T18:54:31.816 INFO:tasks.workunit.client.1.smithi190.stdout:1/420: chown d3/da/dd/d76/c7d 10234 1 2024-03-26T18:54:31.817 INFO:tasks.workunit.client.0.smithi102.stdout:3/293: rename df/d61/c5a to df/c68 0 2024-03-26T18:54:31.818 INFO:tasks.workunit.client.0.smithi102.stdout:5/412: truncate d2/d14/d1a/f50 3276192 0 2024-03-26T18:54:31.820 INFO:tasks.workunit.client.1.smithi190.stdout:7/561: rmdir d3/d6/d9/d1f/d84/d99/dba 0 2024-03-26T18:54:31.820 INFO:tasks.workunit.client.0.smithi102.stdout:2/170: mknod d3/d6/d1a/d25/c3b 0 2024-03-26T18:54:31.821 INFO:tasks.workunit.client.1.smithi190.stdout:0/585: creat d0/dd/d20/d1c/d8b/d8d/fd2 x:0 0 0 2024-03-26T18:54:31.821 INFO:tasks.workunit.client.1.smithi190.stdout:0/586: chown d0/dd/d20/d1c/d46/d58/lbc 5690 1 2024-03-26T18:54:31.822 INFO:tasks.workunit.client.0.smithi102.stdout:8/152: mknod d0/d3/d26/d19/d27/c2a 0 2024-03-26T18:54:31.822 INFO:tasks.workunit.client.1.smithi190.stdout:0/587: read d0/dd/d20/d1c/d4e/f90 [73318,119462] 0 2024-03-26T18:54:31.825 INFO:tasks.workunit.client.0.smithi102.stdout:0/177: mknod d6/de/d13/d25/c3a 0 2024-03-26T18:54:31.825 INFO:tasks.workunit.client.0.smithi102.stdout:0/178: chown d6/de/d13/d1f/c2a 42085 1 2024-03-26T18:54:31.825 INFO:tasks.workunit.client.1.smithi190.stdout:7/562: truncate d3/fc 3398395 0 2024-03-26T18:54:31.825 INFO:tasks.workunit.client.1.smithi190.stdout:7/563: read - d3/d6/f8c zero size 2024-03-26T18:54:31.825 INFO:tasks.workunit.client.1.smithi190.stdout:0/588: readlink d0/dd/d20/d1c/l86 0 2024-03-26T18:54:31.826 INFO:tasks.workunit.client.0.smithi102.stdout:1/159: rmdir de/d16/d1d 39 2024-03-26T18:54:31.826 INFO:tasks.workunit.client.0.smithi102.stdout:1/160: chown de/f14 59473281 1 2024-03-26T18:54:31.828 INFO:tasks.workunit.client.0.smithi102.stdout:4/130: symlink d1/l2f 0 2024-03-26T18:54:31.829 INFO:tasks.workunit.client.0.smithi102.stdout:7/108: unlink d7/ca 0 2024-03-26T18:54:31.837 INFO:tasks.workunit.client.1.smithi190.stdout:0/589: dread d0/dd/d24/f7d [0,4194304] 0 2024-03-26T18:54:31.856 INFO:tasks.workunit.client.0.smithi102.stdout:1/161: dread f3 [0,4194304] 0 2024-03-26T18:54:31.862 INFO:tasks.workunit.client.0.smithi102.stdout:9/96: write df/f13 [787421,53021] 0 2024-03-26T18:54:31.863 INFO:tasks.workunit.client.0.smithi102.stdout:3/294: creat df/d1c/f69 x:0 0 0 2024-03-26T18:54:31.864 INFO:tasks.workunit.client.0.smithi102.stdout:5/413: getdents d2/d25/d5b/d76 0 2024-03-26T18:54:31.865 INFO:tasks.workunit.client.0.smithi102.stdout:2/171: mkdir d3/d6/d1a/d25/d3c 0 2024-03-26T18:54:31.867 INFO:tasks.workunit.client.0.smithi102.stdout:0/179: rename d6/f18 to d6/de/d13/d1f/f3b 0 2024-03-26T18:54:31.870 INFO:tasks.workunit.client.0.smithi102.stdout:4/131: dread d1/f8 [0,4194304] 0 2024-03-26T18:54:31.870 INFO:tasks.workunit.client.0.smithi102.stdout:7/109: unlink d7/d12/l1d 0 2024-03-26T18:54:31.872 INFO:tasks.workunit.client.0.smithi102.stdout:9/97: mkdir df/d1a 0 2024-03-26T18:54:31.873 INFO:tasks.workunit.client.0.smithi102.stdout:1/162: rename de/d16/d1d/l27 to de/l39 0 2024-03-26T18:54:31.875 INFO:tasks.workunit.client.0.smithi102.stdout:9/98: rename df/c10 to df/c1b 0 2024-03-26T18:54:31.876 INFO:tasks.workunit.client.0.smithi102.stdout:9/99: write df/f13 [5126067,127083] 0 2024-03-26T18:54:31.876 INFO:tasks.workunit.client.0.smithi102.stdout:9/100: chown df/l11 10929084 1 2024-03-26T18:54:31.877 INFO:tasks.workunit.client.0.smithi102.stdout:1/163: creat de/d1f/f3a x:0 0 0 2024-03-26T18:54:31.877 INFO:tasks.workunit.client.0.smithi102.stdout:3/295: dread df/f26 [0,4194304] 0 2024-03-26T18:54:31.877 INFO:tasks.workunit.client.0.smithi102.stdout:1/164: truncate de/d1f/f3a 740461 0 2024-03-26T18:54:31.877 INFO:tasks.workunit.client.0.smithi102.stdout:3/296: write df/f67 [861001,36460] 0 2024-03-26T18:54:31.877 INFO:tasks.workunit.client.0.smithi102.stdout:1/165: creat de/d16/f3b x:0 0 0 2024-03-26T18:54:31.877 INFO:tasks.workunit.client.0.smithi102.stdout:3/297: readlink df/d1c/d34/l59 0 2024-03-26T18:54:31.878 INFO:tasks.workunit.client.0.smithi102.stdout:9/101: creat df/d1a/f1c x:0 0 0 2024-03-26T18:54:31.883 INFO:tasks.workunit.client.0.smithi102.stdout:1/166: unlink de/d1f/c21 0 2024-03-26T18:54:31.883 INFO:tasks.workunit.client.0.smithi102.stdout:1/167: dread - de/f15 zero size 2024-03-26T18:54:31.883 INFO:tasks.workunit.client.0.smithi102.stdout:1/168: rmdir de/d16/d1d 39 2024-03-26T18:54:31.885 INFO:tasks.workunit.client.0.smithi102.stdout:1/169: creat de/d16/d1d/d2a/d2c/f3c x:0 0 0 2024-03-26T18:54:31.886 INFO:tasks.workunit.client.0.smithi102.stdout:6/184: dread d2/d7/f1d [0,4194304] 0 2024-03-26T18:54:31.886 INFO:tasks.workunit.client.0.smithi102.stdout:6/185: truncate d2/f21 923249 0 2024-03-26T18:54:31.886 INFO:tasks.workunit.client.0.smithi102.stdout:7/110: write d7/f9 [78267,17233] 0 2024-03-26T18:54:31.887 INFO:tasks.workunit.client.0.smithi102.stdout:7/111: dread - d7/f20 zero size 2024-03-26T18:54:31.887 INFO:tasks.workunit.client.0.smithi102.stdout:7/112: chown d7/d12 337722895 1 2024-03-26T18:54:31.887 INFO:tasks.workunit.client.0.smithi102.stdout:7/113: write d7/f20 [623290,121461] 0 2024-03-26T18:54:31.889 INFO:tasks.workunit.client.0.smithi102.stdout:6/186: dread d2/d12/f22 [0,4194304] 0 2024-03-26T18:54:31.889 INFO:tasks.workunit.client.0.smithi102.stdout:6/187: dread - d2/f2e zero size 2024-03-26T18:54:31.890 INFO:tasks.workunit.client.0.smithi102.stdout:6/188: creat d2/d7/f30 x:0 0 0 2024-03-26T18:54:31.890 INFO:tasks.workunit.client.0.smithi102.stdout:3/298: write df/f17 [597490,30725] 0 2024-03-26T18:54:31.891 INFO:tasks.workunit.client.0.smithi102.stdout:7/114: dread d7/d10/d17/f21 [0,4194304] 0 2024-03-26T18:54:31.892 INFO:tasks.workunit.client.0.smithi102.stdout:3/299: creat df/d29/d31/d36/d3c/f6a x:0 0 0 2024-03-26T18:54:31.892 INFO:tasks.workunit.client.0.smithi102.stdout:3/300: write df/d29/d31/d36/d45/f60 [2943630,16257] 0 2024-03-26T18:54:31.893 INFO:tasks.workunit.client.0.smithi102.stdout:3/301: symlink df/d29/l6b 0 2024-03-26T18:54:31.894 INFO:tasks.workunit.client.0.smithi102.stdout:7/115: rmdir d7/d10/d17 39 2024-03-26T18:54:31.894 INFO:tasks.workunit.client.0.smithi102.stdout:7/116: readlink d7/d10/d17/l1e 0 2024-03-26T18:54:31.895 INFO:tasks.workunit.client.0.smithi102.stdout:7/117: mkdir d7/d10/d27 0 2024-03-26T18:54:31.902 INFO:tasks.workunit.client.1.smithi190.stdout:4/516: dwrite d7/d8/d14/d5c/f63 [0,4194304] 0 2024-03-26T18:54:31.902 INFO:tasks.workunit.client.1.smithi190.stdout:4/517: write d7/d8/d1c/f9f [781747,30505] 0 2024-03-26T18:54:31.904 INFO:tasks.workunit.client.0.smithi102.stdout:3/302: dread df/d61/f57 [0,4194304] 0 2024-03-26T18:54:31.904 INFO:tasks.workunit.client.0.smithi102.stdout:2/172: dread d3/d6/f24 [0,4194304] 0 2024-03-26T18:54:31.905 INFO:tasks.workunit.client.1.smithi190.stdout:0/590: dwrite d0/dd/d20/d61/fac [0,4194304] 0 2024-03-26T18:54:31.908 INFO:tasks.workunit.client.0.smithi102.stdout:2/173: getdents d3/d20 0 2024-03-26T18:54:31.908 INFO:tasks.workunit.client.0.smithi102.stdout:4/132: dwrite d1/d17/d1f/f2d [0,4194304] 0 2024-03-26T18:54:31.908 INFO:tasks.workunit.client.0.smithi102.stdout:1/170: write de/d1f/f36 [2081328,35189] 0 2024-03-26T18:54:31.908 INFO:tasks.workunit.client.0.smithi102.stdout:0/180: dwrite d6/d2d/f35 [0,4194304] 0 2024-03-26T18:54:31.908 INFO:tasks.workunit.client.1.smithi190.stdout:9/723: dwrite d12/d2d/f39 [0,4194304] 0 2024-03-26T18:54:31.909 INFO:tasks.workunit.client.1.smithi190.stdout:7/564: dwrite d3/d6/f8c [0,4194304] 0 2024-03-26T18:54:31.909 INFO:tasks.workunit.client.1.smithi190.stdout:9/724: chown d12/d2d/d1d/d23/d83/dc9/dd4/fe8 77214111 1 2024-03-26T18:54:31.909 INFO:tasks.workunit.client.1.smithi190.stdout:7/565: creat d3/d6/d9/d1f/fd0 x:0 0 0 2024-03-26T18:54:31.923 INFO:tasks.workunit.client.1.smithi190.stdout:4/518: dread d7/d8/f33 [0,4194304] 0 2024-03-26T18:54:31.923 INFO:tasks.workunit.client.1.smithi190.stdout:4/519: stat d7/d8/d14/d5c 0 2024-03-26T18:54:31.933 INFO:tasks.workunit.client.0.smithi102.stdout:4/133: dread d1/d17/d1a/f1c [0,4194304] 0 2024-03-26T18:54:31.936 INFO:tasks.workunit.client.0.smithi102.stdout:1/171: read f7 [1175757,102783] 0 2024-03-26T18:54:31.936 INFO:tasks.workunit.client.0.smithi102.stdout:1/172: fdatasync de/f26 0 2024-03-26T18:54:31.936 INFO:tasks.workunit.client.0.smithi102.stdout:1/173: fsync de/d16/f34 0 2024-03-26T18:54:31.936 INFO:tasks.workunit.client.0.smithi102.stdout:1/174: chown de/d1f 1309652 1 2024-03-26T18:54:31.942 INFO:tasks.workunit.client.1.smithi190.stdout:9/725: dwrite d12/d2d/f39 [4194304,4194304] 0 2024-03-26T18:54:31.943 INFO:tasks.workunit.client.0.smithi102.stdout:7/118: dwrite d7/d10/f24 [0,4194304] 0 2024-03-26T18:54:31.943 INFO:tasks.workunit.client.0.smithi102.stdout:7/119: getdents d7/d10/d27 0 2024-03-26T18:54:31.944 INFO:tasks.workunit.client.1.smithi190.stdout:4/520: symlink d7/d8/d14/d23/d28/d38/lb1 0 2024-03-26T18:54:31.944 INFO:tasks.workunit.client.1.smithi190.stdout:4/521: fsync d7/d1d/d5e/f76 0 2024-03-26T18:54:31.944 INFO:tasks.workunit.client.1.smithi190.stdout:4/522: chown d7/d1d/d42/d4a 8613 1 2024-03-26T18:54:31.944 INFO:tasks.workunit.client.0.smithi102.stdout:2/174: dwrite d3/d2d/f32 [0,4194304] 0 2024-03-26T18:54:31.944 INFO:tasks.workunit.client.0.smithi102.stdout:6/189: dwrite d2/d3/f2a [0,4194304] 0 2024-03-26T18:54:31.945 INFO:tasks.workunit.client.0.smithi102.stdout:6/190: chown d2/d12/f22 3156 1 2024-03-26T18:54:31.947 INFO:tasks.workunit.client.0.smithi102.stdout:1/175: mknod de/d2e/c3d 0 2024-03-26T18:54:31.949 INFO:tasks.workunit.client.1.smithi190.stdout:4/523: mkdir d7/d8/d14/d4b/db2 0 2024-03-26T18:54:31.949 INFO:tasks.workunit.client.1.smithi190.stdout:4/524: write d7/d8/f22 [4263997,5143] 0 2024-03-26T18:54:31.949 INFO:tasks.workunit.client.1.smithi190.stdout:4/525: readlink d7/d8/d1c/l34 0 2024-03-26T18:54:31.949 INFO:tasks.workunit.client.1.smithi190.stdout:4/526: readlink d7/d8/d1c/d20/l8c 0 2024-03-26T18:54:31.949 INFO:tasks.workunit.client.1.smithi190.stdout:4/527: creat d7/d1d/fb3 x:0 0 0 2024-03-26T18:54:31.949 INFO:tasks.workunit.client.1.smithi190.stdout:4/528: fdatasync d7/d8/d14/d4b/f75 0 2024-03-26T18:54:31.949 INFO:tasks.workunit.client.1.smithi190.stdout:4/529: readlink d7/d8/d1c/d20/l67 0 2024-03-26T18:54:31.950 INFO:tasks.workunit.client.0.smithi102.stdout:1/176: write f3 [521952,127772] 0 2024-03-26T18:54:31.952 INFO:tasks.workunit.client.0.smithi102.stdout:3/303: dwrite df/d29/d31/d36/f48 [0,4194304] 0 2024-03-26T18:54:31.952 INFO:tasks.workunit.client.0.smithi102.stdout:3/304: stat df/d1c/f69 0 2024-03-26T18:54:31.952 INFO:tasks.workunit.client.0.smithi102.stdout:0/181: dwrite d6/d2d/f38 [0,4194304] 0 2024-03-26T18:54:31.952 INFO:tasks.workunit.client.0.smithi102.stdout:0/182: write d6/de/d19/f30 [1621386,55495] 0 2024-03-26T18:54:31.953 INFO:tasks.workunit.client.0.smithi102.stdout:0/183: chown d6/de/d13 4410379 1 2024-03-26T18:54:31.953 INFO:tasks.workunit.client.0.smithi102.stdout:9/102: getdents df 0 2024-03-26T18:54:31.954 INFO:tasks.workunit.client.1.smithi190.stdout:9/726: dread d12/d2d/d1d/d23/d83/d91/fe7 [0,4194304] 0 2024-03-26T18:54:31.956 INFO:tasks.workunit.client.1.smithi190.stdout:9/727: dread d12/d2d/f3e [0,4194304] 0 2024-03-26T18:54:31.956 INFO:tasks.workunit.client.0.smithi102.stdout:6/191: dread d2/d7/f8 [4194304,4194304] 0 2024-03-26T18:54:31.957 INFO:tasks.workunit.client.0.smithi102.stdout:9/103: dread df/f13 [4194304,4194304] 0 2024-03-26T18:54:31.963 INFO:tasks.workunit.client.0.smithi102.stdout:2/175: symlink d3/d11/d15/d31/l3d 0 2024-03-26T18:54:31.965 INFO:tasks.workunit.client.1.smithi190.stdout:9/728: dread d12/d2d/d1d/d23/f59 [0,4194304] 0 2024-03-26T18:54:31.966 INFO:tasks.workunit.client.0.smithi102.stdout:1/177: link de/d16/f35 de/f3e 0 2024-03-26T18:54:31.966 INFO:tasks.workunit.client.0.smithi102.stdout:1/178: creat de/f3f x:0 0 0 2024-03-26T18:54:31.966 INFO:tasks.workunit.client.0.smithi102.stdout:1/179: creat de/d16/f40 x:0 0 0 2024-03-26T18:54:31.966 INFO:tasks.workunit.client.0.smithi102.stdout:1/180: write de/d16/d1d/f1e [233102,18422] 0 2024-03-26T18:54:31.967 INFO:tasks.workunit.client.0.smithi102.stdout:1/181: creat de/d16/f41 x:0 0 0 2024-03-26T18:54:31.970 INFO:tasks.workunit.client.1.smithi190.stdout:9/729: mkdir d12/d2d/dfc 0 2024-03-26T18:54:31.970 INFO:tasks.workunit.client.1.smithi190.stdout:9/730: creat d12/d2d/d1d/d23/d83/dc9/dd4/d3f/ffd x:0 0 0 2024-03-26T18:54:31.971 INFO:tasks.workunit.client.0.smithi102.stdout:3/305: readlink df/l2f 0 2024-03-26T18:54:31.971 INFO:tasks.workunit.client.0.smithi102.stdout:3/306: readlink df/d61/l4b 0 2024-03-26T18:54:31.972 INFO:tasks.workunit.client.0.smithi102.stdout:3/307: fdatasync df/d61/f57 0 2024-03-26T18:54:31.972 INFO:tasks.workunit.client.1.smithi190.stdout:9/731: symlink d12/d2d/dfc/lfe 0 2024-03-26T18:54:31.972 INFO:tasks.workunit.client.1.smithi190.stdout:3/421: sync 2024-03-26T18:54:31.972 INFO:tasks.workunit.client.1.smithi190.stdout:5/346: sync 2024-03-26T18:54:31.972 INFO:tasks.workunit.client.1.smithi190.stdout:5/347: truncate d2/d17/f44 838110 0 2024-03-26T18:54:31.973 INFO:tasks.workunit.client.1.smithi190.stdout:9/732: mkdir d12/d2d/d1d/d54/d92/de3/dff 0 2024-03-26T18:54:31.976 INFO:tasks.workunit.client.1.smithi190.stdout:5/348: creat d2/d17/d62/f7e x:0 0 0 2024-03-26T18:54:31.976 INFO:tasks.workunit.client.1.smithi190.stdout:5/349: stat d2/d17/d1b/d1e/d3a/d3c 0 2024-03-26T18:54:31.976 INFO:tasks.workunit.client.1.smithi190.stdout:5/350: write d2/d17/d1b/f76 [323671,75439] 0 2024-03-26T18:54:31.976 INFO:tasks.workunit.client.1.smithi190.stdout:5/351: fsync d2/d17/d1b/d30/f73 0 2024-03-26T18:54:31.977 INFO:tasks.workunit.client.1.smithi190.stdout:3/422: creat d1/d3/df/d18/d35/d41/d51/f7a x:0 0 0 2024-03-26T18:54:31.980 INFO:tasks.workunit.client.0.smithi102.stdout:6/192: mknod d2/c31 0 2024-03-26T18:54:31.982 INFO:tasks.workunit.client.0.smithi102.stdout:7/120: rename d7/l26 to d7/l28 0 2024-03-26T18:54:31.983 INFO:tasks.workunit.client.1.smithi190.stdout:3/423: creat d1/de/d1c/f7b x:0 0 0 2024-03-26T18:54:31.984 INFO:tasks.workunit.client.0.smithi102.stdout:9/104: stat c8 0 2024-03-26T18:54:31.985 INFO:tasks.workunit.client.1.smithi190.stdout:9/733: rmdir d12/d2d/d1d/da1/dba/dbc 39 2024-03-26T18:54:31.985 INFO:tasks.workunit.client.1.smithi190.stdout:9/734: read d12/d2d/f9f [56761,67733] 0 2024-03-26T18:54:31.985 INFO:tasks.workunit.client.1.smithi190.stdout:9/735: write d12/d2d/d1d/da1/fac [557977,12529] 0 2024-03-26T18:54:31.985 INFO:tasks.workunit.client.0.smithi102.stdout:2/176: creat d3/d6/d1a/d25/f3e x:0 0 0 2024-03-26T18:54:31.987 INFO:tasks.workunit.client.1.smithi190.stdout:5/352: rmdir d2/d17/d1b/d22/d24 39 2024-03-26T18:54:31.988 INFO:tasks.workunit.client.1.smithi190.stdout:5/353: truncate d2/f14 1095326 0 2024-03-26T18:54:31.988 INFO:tasks.workunit.client.1.smithi190.stdout:5/354: readlink d2/le 0 2024-03-26T18:54:31.988 INFO:tasks.workunit.client.1.smithi190.stdout:5/355: creat d2/d17/d1b/d22/f7f x:0 0 0 2024-03-26T18:54:31.988 INFO:tasks.workunit.client.1.smithi190.stdout:5/356: rename d2/d17/d1b/d22 to d2/d17/d1b/d22/d24/d80 22 2024-03-26T18:54:31.989 INFO:tasks.workunit.client.1.smithi190.stdout:3/424: unlink d1/de/d1c/f2b 0 2024-03-26T18:54:31.990 INFO:tasks.workunit.client.1.smithi190.stdout:9/736: mkdir d12/d2d/dfc/d100 0 2024-03-26T18:54:31.991 INFO:tasks.workunit.client.1.smithi190.stdout:9/737: write d12/d2d/d1d/d54/ff9 [97231,126845] 0 2024-03-26T18:54:31.991 INFO:tasks.workunit.client.1.smithi190.stdout:9/738: chown d12/d2d/d1d/d23/d83/dc9/dd4/d3f/ffd 1882676 1 2024-03-26T18:54:31.991 INFO:tasks.workunit.client.1.smithi190.stdout:9/739: fsync d12/d2d/d1d/d23/d83/dc9/dd4/f42 0 2024-03-26T18:54:31.992 INFO:tasks.workunit.client.0.smithi102.stdout:3/308: creat df/d29/d31/f6c x:0 0 0 2024-03-26T18:54:31.992 INFO:tasks.workunit.client.0.smithi102.stdout:3/309: read - df/d1c/f69 zero size 2024-03-26T18:54:31.992 INFO:tasks.workunit.client.0.smithi102.stdout:3/310: dread - df/d29/d31/f32 zero size 2024-03-26T18:54:31.993 INFO:tasks.workunit.client.0.smithi102.stdout:0/184: getdents d6/d2d 0 2024-03-26T18:54:31.994 INFO:tasks.workunit.client.1.smithi190.stdout:5/357: symlink d2/d17/l81 0 2024-03-26T18:54:31.994 INFO:tasks.workunit.client.1.smithi190.stdout:5/358: read - d2/d17/d1b/d22/d24/d3b/f56 zero size 2024-03-26T18:54:31.994 INFO:tasks.workunit.client.1.smithi190.stdout:5/359: creat d2/d17/d62/f82 x:0 0 0 2024-03-26T18:54:31.994 INFO:tasks.workunit.client.1.smithi190.stdout:5/360: chown d2/f10 503045078 1 2024-03-26T18:54:31.994 INFO:tasks.workunit.client.1.smithi190.stdout:5/361: stat d2/d17/d1b/d1e/d3a/f52 0 2024-03-26T18:54:31.994 INFO:tasks.workunit.client.1.smithi190.stdout:3/425: truncate d1/d3/df/d18/d35/d41/f72 1796503 0 2024-03-26T18:54:31.994 INFO:tasks.workunit.client.1.smithi190.stdout:3/426: readlink d1/d3/df/d18/d35/d65/l6c 0 2024-03-26T18:54:31.995 INFO:tasks.workunit.client.1.smithi190.stdout:5/362: dread d2/f6 [0,4194304] 0 2024-03-26T18:54:31.995 INFO:tasks.workunit.client.0.smithi102.stdout:6/193: symlink d2/df/l32 0 2024-03-26T18:54:31.996 INFO:tasks.workunit.client.1.smithi190.stdout:9/740: link d12/l4a d12/d2d/d1d/d23/d2b/da0/l101 0 2024-03-26T18:54:31.997 INFO:tasks.workunit.client.0.smithi102.stdout:9/105: mkdir df/d1a/d1d 0 2024-03-26T18:54:31.997 INFO:tasks.workunit.client.0.smithi102.stdout:9/106: fsync df/f17 0 2024-03-26T18:54:31.998 INFO:tasks.workunit.client.1.smithi190.stdout:3/427: symlink d1/d3/df/d18/d1b/l7c 0 2024-03-26T18:54:32.000 INFO:tasks.workunit.client.1.smithi190.stdout:9/741: rmdir d12/d2d/d1d/da1/dba 39 2024-03-26T18:54:32.001 INFO:tasks.workunit.client.1.smithi190.stdout:3/428: mkdir d1/d3/d52/d7d 0 2024-03-26T18:54:32.004 INFO:tasks.workunit.client.1.smithi190.stdout:9/742: symlink d12/d2d/d1d/d23/d83/dc9/dd4/d3f/l102 0 2024-03-26T18:54:32.005 INFO:tasks.workunit.client.0.smithi102.stdout:3/311: creat df/d29/d3a/f6d x:0 0 0 2024-03-26T18:54:32.005 INFO:tasks.workunit.client.0.smithi102.stdout:3/312: fsync df/f67 0 2024-03-26T18:54:32.006 INFO:tasks.workunit.client.0.smithi102.stdout:0/185: mknod d6/de/d13/d1f/c3c 0 2024-03-26T18:54:32.006 INFO:tasks.workunit.client.0.smithi102.stdout:0/186: chown d6/de/d19/f30 156806 1 2024-03-26T18:54:32.007 INFO:tasks.workunit.client.0.smithi102.stdout:0/187: write d6/f11 [906918,55009] 0 2024-03-26T18:54:32.008 INFO:tasks.workunit.client.1.smithi190.stdout:9/743: creat d12/f103 x:0 0 0 2024-03-26T18:54:32.008 INFO:tasks.workunit.client.1.smithi190.stdout:9/744: read - d12/d2d/d1d/d23/d83/dc9/dd4/d3f/fc7 zero size 2024-03-26T18:54:32.009 INFO:tasks.workunit.client.0.smithi102.stdout:6/194: unlink d2/d3/l6 0 2024-03-26T18:54:32.009 INFO:tasks.workunit.client.0.smithi102.stdout:6/195: chown d2/df/l15 765281 1 2024-03-26T18:54:32.011 INFO:tasks.workunit.client.1.smithi190.stdout:9/745: creat d12/d6d/f104 x:0 0 0 2024-03-26T18:54:32.015 INFO:tasks.workunit.client.0.smithi102.stdout:3/313: rename df/d29/d31/d36/d45/c46 to df/d61/d55/c6e 0 2024-03-26T18:54:32.015 INFO:tasks.workunit.client.0.smithi102.stdout:3/314: write df/d61/f66 [60308,65751] 0 2024-03-26T18:54:32.019 INFO:tasks.workunit.client.0.smithi102.stdout:3/315: truncate df/d61/f57 4148655 0 2024-03-26T18:54:32.020 INFO:tasks.workunit.client.0.smithi102.stdout:0/188: dread d6/d2d/f35 [0,4194304] 0 2024-03-26T18:54:32.021 INFO:tasks.workunit.client.1.smithi190.stdout:4/530: dwrite d7/f9 [0,4194304] 0 2024-03-26T18:54:32.022 INFO:tasks.workunit.client.0.smithi102.stdout:0/189: mkdir d6/d2d/d3d 0 2024-03-26T18:54:32.023 INFO:tasks.workunit.client.1.smithi190.stdout:5/363: dwrite d2/d17/d1b/d22/f7f [0,4194304] 0 2024-03-26T18:54:32.024 INFO:tasks.workunit.client.0.smithi102.stdout:9/107: dwrite df/f17 [0,4194304] 0 2024-03-26T18:54:32.024 INFO:tasks.workunit.client.0.smithi102.stdout:9/108: chown df/c18 1 1 2024-03-26T18:54:32.025 INFO:tasks.workunit.client.0.smithi102.stdout:6/196: write d2/d3/f2a [3678319,12134] 0 2024-03-26T18:54:32.025 INFO:tasks.workunit.client.0.smithi102.stdout:0/190: fdatasync d6/fd 0 2024-03-26T18:54:32.026 INFO:tasks.workunit.client.0.smithi102.stdout:3/316: write df/f67 [644535,130297] 0 2024-03-26T18:54:32.027 INFO:tasks.workunit.client.0.smithi102.stdout:9/109: mknod df/d1a/c1e 0 2024-03-26T18:54:32.030 INFO:tasks.workunit.client.1.smithi190.stdout:5/364: creat d2/d17/f83 x:0 0 0 2024-03-26T18:54:32.030 INFO:tasks.workunit.client.1.smithi190.stdout:5/365: write d2/d17/d62/d64/f74 [848519,84943] 0 2024-03-26T18:54:32.030 INFO:tasks.workunit.client.0.smithi102.stdout:0/191: dread d6/f7 [0,4194304] 0 2024-03-26T18:54:32.031 INFO:tasks.workunit.client.0.smithi102.stdout:7/121: rmdir d7 39 2024-03-26T18:54:32.031 INFO:tasks.workunit.client.0.smithi102.stdout:7/122: getdents d7/d10/d27 0 2024-03-26T18:54:32.032 INFO:tasks.workunit.client.1.smithi190.stdout:9/746: dwrite d12/f38 [0,4194304] 0 2024-03-26T18:54:32.038 INFO:tasks.workunit.client.0.smithi102.stdout:3/317: mkdir df/d29/d31/d36/d3c/d6f 0 2024-03-26T18:54:32.038 INFO:tasks.workunit.client.0.smithi102.stdout:3/318: stat df/f25 0 2024-03-26T18:54:32.039 INFO:tasks.workunit.client.0.smithi102.stdout:1/182: rmdir de/d16 39 2024-03-26T18:54:32.044 INFO:tasks.workunit.client.1.smithi190.stdout:9/747: rename d12/d2d/d1d/d23/d6a/c79 to d12/d2d/d1d/d54/d92/de3/dff/c105 0 2024-03-26T18:54:32.044 INFO:tasks.workunit.client.1.smithi190.stdout:9/748: truncate f8 8479377 0 2024-03-26T18:54:32.046 INFO:tasks.workunit.client.1.smithi190.stdout:9/749: mknod d12/d6d/c106 0 2024-03-26T18:54:32.049 INFO:tasks.workunit.client.1.smithi190.stdout:9/750: creat d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/db3/f107 x:0 0 0 2024-03-26T18:54:32.051 INFO:tasks.workunit.client.1.smithi190.stdout:9/751: getdents d12/d2d/d1d 0 2024-03-26T18:54:32.051 INFO:tasks.workunit.client.1.smithi190.stdout:9/752: truncate d12/d2d/d1d/fd3 538029 0 2024-03-26T18:54:32.054 INFO:tasks.workunit.client.1.smithi190.stdout:9/753: link d12/d2d/d1d/d54/d92/lec d12/d6d/l108 0 2024-03-26T18:54:32.055 INFO:tasks.workunit.client.0.smithi102.stdout:7/123: dread f4 [0,4194304] 0 2024-03-26T18:54:32.055 INFO:tasks.workunit.client.0.smithi102.stdout:7/124: fdatasync d7/ff 0 2024-03-26T18:54:32.055 INFO:tasks.workunit.client.0.smithi102.stdout:7/125: write f4 [6127672,24659] 0 2024-03-26T18:54:32.055 INFO:tasks.workunit.client.0.smithi102.stdout:0/192: dread d6/de/f15 [0,4194304] 0 2024-03-26T18:54:32.055 INFO:tasks.workunit.client.0.smithi102.stdout:0/193: chown d6/f36 313475 1 2024-03-26T18:54:32.056 INFO:tasks.workunit.client.1.smithi190.stdout:9/754: mknod d12/d2d/d1d/d23/d83/d91/c109 0 2024-03-26T18:54:32.056 INFO:tasks.workunit.client.0.smithi102.stdout:7/126: chown d7/c11 9483 1 2024-03-26T18:54:32.059 INFO:tasks.workunit.client.0.smithi102.stdout:0/194: mkdir d6/d3e 0 2024-03-26T18:54:32.061 INFO:tasks.workunit.client.1.smithi190.stdout:4/531: dwrite d7/d8/d14/f5b [0,4194304] 0 2024-03-26T18:54:32.062 INFO:tasks.workunit.client.1.smithi190.stdout:5/366: dwrite d2/d17/d1b/f36 [0,4194304] 0 2024-03-26T18:54:32.062 INFO:tasks.workunit.client.0.smithi102.stdout:1/183: read fc [2294425,108480] 0 2024-03-26T18:54:32.065 INFO:tasks.workunit.client.1.smithi190.stdout:4/532: truncate d7/d8/d14/d23/d28/d38/f71 626611 0 2024-03-26T18:54:32.073 INFO:tasks.workunit.client.0.smithi102.stdout:3/319: dread df/d29/d31/d36/d3c/f3f [0,4194304] 0 2024-03-26T18:54:32.073 INFO:tasks.workunit.client.0.smithi102.stdout:6/197: dwrite d2/d3/f2a [0,4194304] 0 2024-03-26T18:54:32.073 INFO:tasks.workunit.client.0.smithi102.stdout:3/320: readlink df/d29/l2b 0 2024-03-26T18:54:32.073 INFO:tasks.workunit.client.0.smithi102.stdout:6/198: fdatasync d2/d7/f1d 0 2024-03-26T18:54:32.073 INFO:tasks.workunit.client.0.smithi102.stdout:6/199: creat d2/f33 x:0 0 0 2024-03-26T18:54:32.073 INFO:tasks.workunit.client.0.smithi102.stdout:6/200: creat d2/f34 x:0 0 0 2024-03-26T18:54:32.074 INFO:tasks.workunit.client.1.smithi190.stdout:5/367: creat d2/d17/d1b/d22/d24/f84 x:0 0 0 2024-03-26T18:54:32.076 INFO:tasks.workunit.client.0.smithi102.stdout:3/321: mkdir df/d1c/d70 0 2024-03-26T18:54:32.079 INFO:tasks.workunit.client.1.smithi190.stdout:4/533: dread d7/d8/d14/d5c/f97 [0,4194304] 0 2024-03-26T18:54:32.079 INFO:tasks.workunit.client.1.smithi190.stdout:4/534: write d7/d8/d14/f48 [4694528,99266] 0 2024-03-26T18:54:32.079 INFO:tasks.workunit.client.1.smithi190.stdout:4/535: readlink d7/d8/d14/d23/d28/d38/lb1 0 2024-03-26T18:54:32.080 INFO:tasks.workunit.client.0.smithi102.stdout:6/201: mkdir d2/df/d35 0 2024-03-26T18:54:32.082 INFO:tasks.workunit.client.0.smithi102.stdout:3/322: read df/f2e [45197,46695] 0 2024-03-26T18:54:32.082 INFO:tasks.workunit.client.0.smithi102.stdout:3/323: fdatasync df/f20 0 2024-03-26T18:54:32.082 INFO:tasks.workunit.client.0.smithi102.stdout:3/324: fdatasync df/d1c/d34/f37 0 2024-03-26T18:54:32.082 INFO:tasks.workunit.client.0.smithi102.stdout:3/325: readlink df/d1c/d34/l59 0 2024-03-26T18:54:32.084 INFO:tasks.workunit.client.1.smithi190.stdout:4/536: mknod d7/cb4 0 2024-03-26T18:54:32.088 INFO:tasks.workunit.client.0.smithi102.stdout:3/326: link df/c68 df/c71 0 2024-03-26T18:54:32.088 INFO:tasks.workunit.client.0.smithi102.stdout:3/327: write df/d29/d31/f32 [764292,84195] 0 2024-03-26T18:54:32.091 INFO:tasks.workunit.client.0.smithi102.stdout:3/328: dread df/f2e [0,4194304] 0 2024-03-26T18:54:32.096 INFO:tasks.workunit.client.0.smithi102.stdout:3/329: dread df/d29/d31/d36/d3c/f3f [0,4194304] 0 2024-03-26T18:54:32.101 INFO:tasks.workunit.client.1.smithi190.stdout:4/537: dread d7/d8/d14/d4b/f77 [0,4194304] 0 2024-03-26T18:54:32.108 INFO:tasks.workunit.client.0.smithi102.stdout:0/195: dwrite d6/de/d13/d1f/f3b [4194304,4194304] 0 2024-03-26T18:54:32.108 INFO:tasks.workunit.client.0.smithi102.stdout:0/196: fdatasync d6/de/d13/d1f/f3b 0 2024-03-26T18:54:32.109 INFO:tasks.workunit.client.0.smithi102.stdout:0/197: getdents d6/d2d 0 2024-03-26T18:54:32.110 INFO:tasks.workunit.client.0.smithi102.stdout:7/127: dwrite d7/fb [0,4194304] 0 2024-03-26T18:54:32.111 INFO:tasks.workunit.client.1.smithi190.stdout:5/368: dwrite d2/d17/d62/d64/f74 [0,4194304] 0 2024-03-26T18:54:32.111 INFO:tasks.workunit.client.1.smithi190.stdout:5/369: stat d2/f4 0 2024-03-26T18:54:32.111 INFO:tasks.workunit.client.1.smithi190.stdout:9/755: dwrite d12/d2d/fd9 [0,4194304] 0 2024-03-26T18:54:32.111 INFO:tasks.workunit.client.0.smithi102.stdout:6/202: dwrite d2/d12/f2d [0,4194304] 0 2024-03-26T18:54:32.112 INFO:tasks.workunit.client.0.smithi102.stdout:1/184: dwrite de/d1f/f33 [0,4194304] 0 2024-03-26T18:54:32.112 INFO:tasks.workunit.client.0.smithi102.stdout:0/198: rename d6/de/f1b to d6/de/d13/f3f 0 2024-03-26T18:54:32.112 INFO:tasks.workunit.client.0.smithi102.stdout:0/199: write d6/f9 [1064867,52169] 0 2024-03-26T18:54:32.113 INFO:tasks.workunit.client.0.smithi102.stdout:3/330: write df/d29/d31/f32 [705025,97130] 0 2024-03-26T18:54:32.114 INFO:tasks.workunit.client.1.smithi190.stdout:5/370: mkdir d2/d17/d1b/d60/d63/d85 0 2024-03-26T18:54:32.115 INFO:tasks.workunit.client.1.smithi190.stdout:5/371: write d2/d17/d1b/d1e/d3a/f52 [856189,47026] 0 2024-03-26T18:54:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:7/128: write f4 [2716187,77369] 0 2024-03-26T18:54:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:7/129: chown d7/l25 449 1 2024-03-26T18:54:32.119 INFO:tasks.workunit.client.1.smithi190.stdout:9/756: creat d12/d2d/d1d/d23/d83/dc9/f10a x:0 0 0 2024-03-26T18:54:32.119 INFO:tasks.workunit.client.1.smithi190.stdout:9/757: dread - d12/d6d/f7d zero size 2024-03-26T18:54:32.120 INFO:tasks.workunit.client.1.smithi190.stdout:4/538: dwrite d7/d1d/d5e/f81 [0,4194304] 0 2024-03-26T18:54:32.121 INFO:tasks.workunit.client.1.smithi190.stdout:9/758: symlink d12/d2d/d1d/d54/d92/de3/l10b 0 2024-03-26T18:54:32.125 INFO:tasks.workunit.client.0.smithi102.stdout:6/203: truncate d2/f21 346584 0 2024-03-26T18:54:32.129 INFO:tasks.workunit.client.1.smithi190.stdout:4/539: unlink d7/d8/d14/d4b/d7d/fa2 0 2024-03-26T18:54:32.130 INFO:tasks.workunit.client.1.smithi190.stdout:4/540: readlink d7/d1d/d41/d74/l79 0 2024-03-26T18:54:32.132 INFO:tasks.workunit.client.0.smithi102.stdout:1/185: symlink de/d16/d1d/d2a/l42 0 2024-03-26T18:54:32.132 INFO:tasks.workunit.client.0.smithi102.stdout:1/186: dread - de/d16/f3b zero size 2024-03-26T18:54:32.132 INFO:tasks.workunit.client.0.smithi102.stdout:1/187: chown de/f15 311029931 1 2024-03-26T18:54:32.132 INFO:tasks.workunit.client.0.smithi102.stdout:0/200: creat d6/d2d/d3d/f40 x:0 0 0 2024-03-26T18:54:32.133 INFO:tasks.workunit.client.0.smithi102.stdout:0/201: read d6/f9 [592600,127568] 0 2024-03-26T18:54:32.134 INFO:tasks.workunit.client.0.smithi102.stdout:3/331: link df/d61/l4b df/d61/l72 0 2024-03-26T18:54:32.135 INFO:tasks.workunit.client.0.smithi102.stdout:7/130: truncate f6 5946702 0 2024-03-26T18:54:32.135 INFO:tasks.workunit.client.0.smithi102.stdout:7/131: stat d7/d12 0 2024-03-26T18:54:32.136 INFO:tasks.workunit.client.0.smithi102.stdout:6/204: truncate d2/d7/f20 25455 0 2024-03-26T18:54:32.136 INFO:tasks.workunit.client.0.smithi102.stdout:6/205: creat d2/d7/f36 x:0 0 0 2024-03-26T18:54:32.137 INFO:tasks.workunit.client.0.smithi102.stdout:1/188: mkdir de/d16/d1d/d2a/d2c/d43 0 2024-03-26T18:54:32.138 INFO:tasks.workunit.client.0.smithi102.stdout:0/202: link d6/de/c33 d6/de/d13/d25/c41 0 2024-03-26T18:54:32.142 INFO:tasks.workunit.client.0.smithi102.stdout:7/132: creat d7/f29 x:0 0 0 2024-03-26T18:54:32.144 INFO:tasks.workunit.client.1.smithi190.stdout:4/541: write d7/f4f [8154000,97402] 0 2024-03-26T18:54:32.144 INFO:tasks.workunit.client.1.smithi190.stdout:4/542: chown d7/d8/d14/d6e 914639197 1 2024-03-26T18:54:32.144 INFO:tasks.workunit.client.1.smithi190.stdout:5/372: dwrite d2/d17/d1b/d22/f3d [4194304,4194304] 0 2024-03-26T18:54:32.144 INFO:tasks.workunit.client.0.smithi102.stdout:6/206: mknod d2/d7/c37 0 2024-03-26T18:54:32.145 INFO:tasks.workunit.client.0.smithi102.stdout:6/207: creat d2/d3/f38 x:0 0 0 2024-03-26T18:54:32.145 INFO:tasks.workunit.client.1.smithi190.stdout:5/373: dread d2/f4 [0,4194304] 0 2024-03-26T18:54:32.146 INFO:tasks.workunit.client.0.smithi102.stdout:8/153: sync 2024-03-26T18:54:32.146 INFO:tasks.workunit.client.0.smithi102.stdout:0/203: dread d6/de/d13/d1f/f27 [0,4194304] 0 2024-03-26T18:54:32.147 INFO:tasks.workunit.client.0.smithi102.stdout:7/133: unlink d7/d10/f24 0 2024-03-26T18:54:32.148 INFO:tasks.workunit.client.1.smithi190.stdout:9/759: dwrite d12/d2d/d1d/da1/dba/fd0 [0,4194304] 0 2024-03-26T18:54:32.151 INFO:tasks.workunit.client.1.smithi190.stdout:5/374: mknod d2/d17/d1b/d60/d63/c86 0 2024-03-26T18:54:32.152 INFO:tasks.workunit.client.0.smithi102.stdout:3/332: dread df/d29/f40 [0,4194304] 0 2024-03-26T18:54:32.153 INFO:tasks.workunit.client.0.smithi102.stdout:1/189: unlink de/d1f/f30 0 2024-03-26T18:54:32.153 INFO:tasks.workunit.client.0.smithi102.stdout:1/190: chown de/d16/f40 48 1 2024-03-26T18:54:32.154 INFO:tasks.workunit.client.0.smithi102.stdout:1/191: fdatasync de/d16/f29 0 2024-03-26T18:54:32.154 INFO:tasks.workunit.client.0.smithi102.stdout:1/192: write fb [710117,98417] 0 2024-03-26T18:54:32.154 INFO:tasks.workunit.client.0.smithi102.stdout:1/193: write de/d16/f29 [297382,13863] 0 2024-03-26T18:54:32.154 INFO:tasks.workunit.client.0.smithi102.stdout:1/194: chown de/d1f/f3a 20377084 1 2024-03-26T18:54:32.154 INFO:tasks.workunit.client.0.smithi102.stdout:1/195: truncate de/d16/f29 523491 0 2024-03-26T18:54:32.155 INFO:tasks.workunit.client.1.smithi190.stdout:4/543: getdents d7/d8/d14/d6e 0 2024-03-26T18:54:32.155 INFO:tasks.workunit.client.1.smithi190.stdout:4/544: truncate d7/d8/d14/d5c/f83 4618906 0 2024-03-26T18:54:32.155 INFO:tasks.workunit.client.1.smithi190.stdout:4/545: creat d7/d1d/fb5 x:0 0 0 2024-03-26T18:54:32.155 INFO:tasks.workunit.client.1.smithi190.stdout:4/546: stat d7/d8/faa 0 2024-03-26T18:54:32.156 INFO:tasks.workunit.client.0.smithi102.stdout:8/154: creat d0/d9/f2b x:0 0 0 2024-03-26T18:54:32.158 INFO:tasks.workunit.client.0.smithi102.stdout:0/204: unlink d6/de/d13/d25/d31/f32 0 2024-03-26T18:54:32.162 INFO:tasks.workunit.client.0.smithi102.stdout:7/134: dwrite d7/d10/d17/f1b [0,4194304] 0 2024-03-26T18:54:32.166 INFO:tasks.workunit.client.0.smithi102.stdout:0/205: read d6/de/d13/d1f/f27 [3725536,75473] 0 2024-03-26T18:54:32.166 INFO:tasks.workunit.client.0.smithi102.stdout:0/206: dread - d6/d2d/d3d/f40 zero size 2024-03-26T18:54:32.169 INFO:tasks.workunit.client.0.smithi102.stdout:4/134: sync 2024-03-26T18:54:32.169 INFO:tasks.workunit.client.0.smithi102.stdout:5/414: sync 2024-03-26T18:54:32.169 INFO:tasks.workunit.client.0.smithi102.stdout:2/177: sync 2024-03-26T18:54:32.170 INFO:tasks.workunit.client.0.smithi102.stdout:1/196: fsync f2 0 2024-03-26T18:54:32.173 INFO:tasks.workunit.client.0.smithi102.stdout:8/155: mkdir d0/d3/d2c 0 2024-03-26T18:54:32.178 INFO:tasks.workunit.client.0.smithi102.stdout:8/156: dread d0/d3/d26/f1f [0,4194304] 0 2024-03-26T18:54:32.178 INFO:tasks.workunit.client.0.smithi102.stdout:8/157: stat d0/d3 0 2024-03-26T18:54:32.179 INFO:tasks.workunit.client.1.smithi190.stdout:9/760: symlink d12/d2d/d1d/d54/d92/de3/dff/l10c 0 2024-03-26T18:54:32.179 INFO:tasks.workunit.client.1.smithi190.stdout:9/761: write d12/d2d/d1d/f77 [1140277,48790] 0 2024-03-26T18:54:32.180 INFO:tasks.workunit.client.0.smithi102.stdout:3/333: link df/c15 df/d29/c73 0 2024-03-26T18:54:32.182 INFO:tasks.workunit.client.1.smithi190.stdout:5/375: mknod d2/d17/d1b/d22/d24/c87 0 2024-03-26T18:54:32.182 INFO:tasks.workunit.client.1.smithi190.stdout:5/376: fdatasync d2/d17/d62/f7e 0 2024-03-26T18:54:32.182 INFO:tasks.workunit.client.0.smithi102.stdout:7/135: dwrite d7/f20 [0,4194304] 0 2024-03-26T18:54:32.182 INFO:tasks.workunit.client.0.smithi102.stdout:7/136: chown d7/fb 67031 1 2024-03-26T18:54:32.182 INFO:tasks.workunit.client.0.smithi102.stdout:7/137: read - d7/f29 zero size 2024-03-26T18:54:32.185 INFO:tasks.workunit.client.0.smithi102.stdout:4/135: unlink d1/ff 0 2024-03-26T18:54:32.187 INFO:tasks.workunit.client.0.smithi102.stdout:2/178: dwrite d3/f8 [0,4194304] 0 2024-03-26T18:54:32.187 INFO:tasks.workunit.client.1.smithi190.stdout:4/547: mknod d7/d8/d14/d23/d28/d9e/cb6 0 2024-03-26T18:54:32.187 INFO:tasks.workunit.client.1.smithi190.stdout:4/548: write d7/d1d/fb5 [167601,109307] 0 2024-03-26T18:54:32.189 INFO:tasks.workunit.client.0.smithi102.stdout:5/415: creat d2/d25/d5b/f94 x:0 0 0 2024-03-26T18:54:32.189 INFO:tasks.workunit.client.0.smithi102.stdout:5/416: fdatasync d2/f78 0 2024-03-26T18:54:32.191 INFO:tasks.workunit.client.1.smithi190.stdout:5/377: write d2/d17/d62/d64/f74 [1913892,92840] 0 2024-03-26T18:54:32.191 INFO:tasks.workunit.client.0.smithi102.stdout:1/197: rename de/d16/d1d/d2a/l38 to de/l44 0 2024-03-26T18:54:32.192 INFO:tasks.workunit.client.1.smithi190.stdout:6/441: sync 2024-03-26T18:54:32.192 INFO:tasks.workunit.client.0.smithi102.stdout:8/158: link d0/c1 d0/d3/d2c/c2d 0 2024-03-26T18:54:32.194 INFO:tasks.workunit.client.0.smithi102.stdout:3/334: creat df/d1c/d34/f74 x:0 0 0 2024-03-26T18:54:32.197 INFO:tasks.workunit.client.1.smithi190.stdout:9/762: dwrite d12/d2d/d1d/d23/d83/dc9/dd4/fe8 [0,4194304] 0 2024-03-26T18:54:32.202 INFO:tasks.workunit.client.1.smithi190.stdout:8/502: sync 2024-03-26T18:54:32.204 INFO:tasks.workunit.client.1.smithi190.stdout:4/549: creat d7/d1d/d42/fb7 x:0 0 0 2024-03-26T18:54:32.204 INFO:tasks.workunit.client.0.smithi102.stdout:0/207: rmdir d6/d2d/d3d 39 2024-03-26T18:54:32.207 INFO:tasks.workunit.client.1.smithi190.stdout:5/378: mkdir d2/d17/d1b/d22/d88 0 2024-03-26T18:54:32.208 INFO:tasks.workunit.client.1.smithi190.stdout:9/763: mkdir d12/d2d/d1d/d23/d10d 0 2024-03-26T18:54:32.210 INFO:tasks.workunit.client.0.smithi102.stdout:2/179: link d3/d11/d15/l21 d3/d11/l3f 0 2024-03-26T18:54:32.210 INFO:tasks.workunit.client.0.smithi102.stdout:2/180: chown d3/d6/d1a/d25/f3e 3 1 2024-03-26T18:54:32.211 INFO:tasks.workunit.client.1.smithi190.stdout:8/503: mkdir d9/dd/d23/d36/d37/d3a/d61/daf 0 2024-03-26T18:54:32.211 INFO:tasks.workunit.client.1.smithi190.stdout:8/504: chown l3 13434439 1 2024-03-26T18:54:32.212 INFO:tasks.workunit.client.1.smithi190.stdout:8/505: getdents d9/da9 0 2024-03-26T18:54:32.213 INFO:tasks.workunit.client.0.smithi102.stdout:1/198: mknod de/d2e/c45 0 2024-03-26T18:54:32.215 INFO:tasks.workunit.client.0.smithi102.stdout:8/159: getdents d0/d3/d26 0 2024-03-26T18:54:32.216 INFO:tasks.workunit.client.0.smithi102.stdout:7/138: mkdir d7/d2a 0 2024-03-26T18:54:32.216 INFO:tasks.workunit.client.0.smithi102.stdout:7/139: readlink d7/d10/l18 0 2024-03-26T18:54:32.217 INFO:tasks.workunit.client.0.smithi102.stdout:7/140: chown d7/d12 851 1 2024-03-26T18:54:32.217 INFO:tasks.workunit.client.0.smithi102.stdout:7/141: truncate d7/d10/d17/f1b 4254761 0 2024-03-26T18:54:32.218 INFO:tasks.workunit.client.0.smithi102.stdout:0/208: symlink d6/de/d13/d25/l42 0 2024-03-26T18:54:32.218 INFO:tasks.workunit.client.0.smithi102.stdout:0/209: write d6/fd [3309217,63633] 0 2024-03-26T18:54:32.218 INFO:tasks.workunit.client.1.smithi190.stdout:5/379: write d2/f3 [1874809,51013] 0 2024-03-26T18:54:32.218 INFO:tasks.workunit.client.0.smithi102.stdout:4/136: mknod d1/c30 0 2024-03-26T18:54:32.220 INFO:tasks.workunit.client.0.smithi102.stdout:5/417: getdents d2/d25/d2a 0 2024-03-26T18:54:32.221 INFO:tasks.workunit.client.0.smithi102.stdout:5/418: chown d2/d14 1265357 1 2024-03-26T18:54:32.222 INFO:tasks.workunit.client.0.smithi102.stdout:1/199: creat de/d16/d1d/d2a/d2c/d43/f46 x:0 0 0 2024-03-26T18:54:32.222 INFO:tasks.workunit.client.1.smithi190.stdout:9/764: dread d12/d2d/f22 [0,4194304] 0 2024-03-26T18:54:32.223 INFO:tasks.workunit.client.0.smithi102.stdout:8/160: link d0/la d0/d3/d26/l2e 0 2024-03-26T18:54:32.225 INFO:tasks.workunit.client.0.smithi102.stdout:7/142: mkdir d7/d12/d2b 0 2024-03-26T18:54:32.227 INFO:tasks.workunit.client.1.smithi190.stdout:9/765: rename d12/d2d/d1d/d23/d6a to d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/d10e 0 2024-03-26T18:54:32.228 INFO:tasks.workunit.client.1.smithi190.stdout:9/766: creat d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/db3/f10f x:0 0 0 2024-03-26T18:54:32.228 INFO:tasks.workunit.client.1.smithi190.stdout:9/767: readlink d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/db3/lcb 0 2024-03-26T18:54:32.229 INFO:tasks.workunit.client.0.smithi102.stdout:0/210: rename d6/f9 to d6/de/d13/f43 0 2024-03-26T18:54:32.229 INFO:tasks.workunit.client.0.smithi102.stdout:0/211: readlink d6/de/l2e 0 2024-03-26T18:54:32.231 INFO:tasks.workunit.client.0.smithi102.stdout:4/137: stat d1/cd 0 2024-03-26T18:54:32.233 INFO:tasks.workunit.client.0.smithi102.stdout:3/335: dwrite df/f17 [0,4194304] 0 2024-03-26T18:54:32.233 INFO:tasks.workunit.client.0.smithi102.stdout:3/336: read df/d29/d31/f32 [704178,71861] 0 2024-03-26T18:54:32.234 INFO:tasks.workunit.client.1.smithi190.stdout:9/768: rename d12/d2d/f5e to d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/d10e/f110 0 2024-03-26T18:54:32.234 INFO:tasks.workunit.client.0.smithi102.stdout:2/181: rmdir d3/d2a 39 2024-03-26T18:54:32.235 INFO:tasks.workunit.client.0.smithi102.stdout:5/419: dread d2/d2c/d38/f5c [0,4194304] 0 2024-03-26T18:54:32.236 INFO:tasks.workunit.client.0.smithi102.stdout:1/200: symlink de/d16/d1d/d2a/d2c/d43/l47 0 2024-03-26T18:54:32.237 INFO:tasks.workunit.client.0.smithi102.stdout:8/161: fsync d0/d3/d26/d19/d27/f29 0 2024-03-26T18:54:32.238 INFO:tasks.workunit.client.0.smithi102.stdout:0/212: truncate d6/d2d/f35 2778614 0 2024-03-26T18:54:32.239 INFO:tasks.workunit.client.1.smithi190.stdout:1/421: sync 2024-03-26T18:54:32.239 INFO:tasks.workunit.client.1.smithi190.stdout:0/591: sync 2024-03-26T18:54:32.239 INFO:tasks.workunit.client.1.smithi190.stdout:7/566: sync 2024-03-26T18:54:32.239 INFO:tasks.workunit.client.1.smithi190.stdout:2/500: sync 2024-03-26T18:54:32.239 INFO:tasks.workunit.client.1.smithi190.stdout:3/429: sync 2024-03-26T18:54:32.240 INFO:tasks.workunit.client.0.smithi102.stdout:4/138: creat d1/f31 x:0 0 0 2024-03-26T18:54:32.240 INFO:tasks.workunit.client.0.smithi102.stdout:4/139: chown d1/d17/d1f/c20 17589 1 2024-03-26T18:54:32.244 INFO:tasks.workunit.client.1.smithi190.stdout:5/380: dwrite d2/d17/f83 [0,4194304] 0 2024-03-26T18:54:32.245 INFO:tasks.workunit.client.0.smithi102.stdout:3/337: read df/d29/d31/d36/f50 [1663329,24147] 0 2024-03-26T18:54:32.246 INFO:tasks.workunit.client.0.smithi102.stdout:3/338: truncate df/d29/d31/d36/d3c/f6a 209831 0 2024-03-26T18:54:32.247 INFO:tasks.workunit.client.1.smithi190.stdout:6/442: dwrite d17/d28/d2c/d77/f8d [0,4194304] 0 2024-03-26T18:54:32.247 INFO:tasks.workunit.client.1.smithi190.stdout:6/443: write d17/d28/d2c/f53 [1002322,74168] 0 2024-03-26T18:54:32.247 INFO:tasks.workunit.client.1.smithi190.stdout:9/769: write d12/d2d/d1d/d23/d83/dc9/dd4/d3f/f81 [2531392,4541] 0 2024-03-26T18:54:32.247 INFO:tasks.workunit.client.1.smithi190.stdout:9/770: write d12/d2d/d1d/fd3 [743033,100188] 0 2024-03-26T18:54:32.247 INFO:tasks.workunit.client.1.smithi190.stdout:9/771: readlink d12/d6d/lee 0 2024-03-26T18:54:32.247 INFO:tasks.workunit.client.1.smithi190.stdout:9/772: chown d12/d2d/d1d/d23/d83/d91/c109 3288109 1 2024-03-26T18:54:32.248 INFO:tasks.workunit.client.1.smithi190.stdout:0/592: rename d0/dd/d20/d1c/f28 to d0/dd/d24/d4b/fd3 0 2024-03-26T18:54:32.248 INFO:tasks.workunit.client.1.smithi190.stdout:0/593: truncate d0/dd/d20/d1c/d8b/d8d/dc8/fce 163597 0 2024-03-26T18:54:32.248 INFO:tasks.workunit.client.1.smithi190.stdout:5/381: dread d2/d17/d1b/d22/d24/d3b/f46 [0,4194304] 0 2024-03-26T18:54:32.248 INFO:tasks.workunit.client.1.smithi190.stdout:4/550: dwrite d7/d8/d1c/f9f [0,4194304] 0 2024-03-26T18:54:32.248 INFO:tasks.workunit.client.1.smithi190.stdout:5/382: write d2/d17/d1b/d30/f3f [2663848,16157] 0 2024-03-26T18:54:32.248 INFO:tasks.workunit.client.1.smithi190.stdout:5/383: readlink d2/d17/d1b/d22/d68/l71 0 2024-03-26T18:54:32.248 INFO:tasks.workunit.client.0.smithi102.stdout:1/201: write f8 [2150954,103635] 0 2024-03-26T18:54:32.249 INFO:tasks.workunit.client.0.smithi102.stdout:1/202: write de/f3f [743127,81344] 0 2024-03-26T18:54:32.249 INFO:tasks.workunit.client.0.smithi102.stdout:7/143: write d7/ff [1480139,83265] 0 2024-03-26T18:54:32.249 INFO:tasks.workunit.client.0.smithi102.stdout:7/144: chown d7/d10/d17/f1b 28402318 1 2024-03-26T18:54:32.250 INFO:tasks.workunit.client.0.smithi102.stdout:7/145: read d7/d10/d17/f21 [6765,90366] 0 2024-03-26T18:54:32.250 INFO:tasks.workunit.client.0.smithi102.stdout:7/146: stat d7/d10/d17/l23 0 2024-03-26T18:54:32.250 INFO:tasks.workunit.client.1.smithi190.stdout:2/501: symlink d6/d9/d3a/d3d/lb6 0 2024-03-26T18:54:32.252 INFO:tasks.workunit.client.1.smithi190.stdout:3/430: link d1/de/c1a d1/d3/df/d18/d57/c7e 0 2024-03-26T18:54:32.254 INFO:tasks.workunit.client.0.smithi102.stdout:2/182: unlink d3/l33 0 2024-03-26T18:54:32.254 INFO:tasks.workunit.client.0.smithi102.stdout:0/213: dread d6/d2d/f38 [0,4194304] 0 2024-03-26T18:54:32.257 INFO:tasks.workunit.client.0.smithi102.stdout:5/420: mknod d2/d25/d2a/c95 0 2024-03-26T18:54:32.258 INFO:tasks.workunit.client.0.smithi102.stdout:5/421: fsync d2/d2c/d38/d43/f8a 0 2024-03-26T18:54:32.263 INFO:tasks.workunit.client.1.smithi190.stdout:9/773: dread d12/d2d/d1d/d23/d83/dc9/dd4/f42 [0,4194304] 0 2024-03-26T18:54:32.263 INFO:tasks.workunit.client.1.smithi190.stdout:9/774: chown d12/d2d/d1d/d23/de5 3 1 2024-03-26T18:54:32.263 INFO:tasks.workunit.client.1.smithi190.stdout:1/422: rmdir d3/da/d3a 39 2024-03-26T18:54:32.263 INFO:tasks.workunit.client.1.smithi190.stdout:1/423: stat d3/d9 0 2024-03-26T18:54:32.265 INFO:tasks.workunit.client.0.smithi102.stdout:1/203: creat de/d1f/f48 x:0 0 0 2024-03-26T18:54:32.268 INFO:tasks.workunit.client.1.smithi190.stdout:2/502: dread d6/d9/d10/f13 [4194304,4194304] 0 2024-03-26T18:54:32.273 INFO:tasks.workunit.client.1.smithi190.stdout:6/444: symlink d17/d28/d8b/l9f 0 2024-03-26T18:54:32.274 INFO:tasks.workunit.client.0.smithi102.stdout:7/147: rename d7/d10/d17/l1f to d7/d10/d17/l2c 0 2024-03-26T18:54:32.278 INFO:tasks.workunit.client.0.smithi102.stdout:0/214: creat d6/d2d/d3d/f44 x:0 0 0 2024-03-26T18:54:32.278 INFO:tasks.workunit.client.1.smithi190.stdout:0/594: mknod d0/dd/d24/d4b/d5e/cd4 0 2024-03-26T18:54:32.279 INFO:tasks.workunit.client.0.smithi102.stdout:5/422: truncate d2/d14/f70 752511 0 2024-03-26T18:54:32.279 INFO:tasks.workunit.client.0.smithi102.stdout:5/423: fdatasync d2/f11 0 2024-03-26T18:54:32.279 INFO:tasks.workunit.client.0.smithi102.stdout:5/424: creat d2/d25/d2a/d4f/f96 x:0 0 0 2024-03-26T18:54:32.281 INFO:tasks.workunit.client.0.smithi102.stdout:8/162: truncate d0/d3/d26/f23 267051 0 2024-03-26T18:54:32.283 INFO:tasks.workunit.client.0.smithi102.stdout:1/204: creat de/d2e/f49 x:0 0 0 2024-03-26T18:54:32.283 INFO:tasks.workunit.client.0.smithi102.stdout:1/205: creat de/d2e/f4a x:0 0 0 2024-03-26T18:54:32.284 INFO:tasks.workunit.client.1.smithi190.stdout:7/567: dwrite d3/d6/d9/d1f/dab/fbe [0,4194304] 0 2024-03-26T18:54:32.285 INFO:tasks.workunit.client.1.smithi190.stdout:7/568: dread - d3/d6/d9/d1f/d41/d50/f60 zero size 2024-03-26T18:54:32.285 INFO:tasks.workunit.client.1.smithi190.stdout:7/569: chown d3/d6/d9/d1f/d41/d50/f89 0 1 2024-03-26T18:54:32.285 INFO:tasks.workunit.client.1.smithi190.stdout:7/570: readlink d3/d6/d9/d1f/d41/d50/d53/l7b 0 2024-03-26T18:54:32.285 INFO:tasks.workunit.client.1.smithi190.stdout:7/571: stat d3/d1b/d57/c77 0 2024-03-26T18:54:32.285 INFO:tasks.workunit.client.1.smithi190.stdout:7/572: chown d3/d6/d9/d1f/l76 9459 1 2024-03-26T18:54:32.285 INFO:tasks.workunit.client.1.smithi190.stdout:7/573: write d3/d6/d9/d14/f7e [810099,53118] 0 2024-03-26T18:54:32.285 INFO:tasks.workunit.client.1.smithi190.stdout:7/574: fsync d3/d6/d9/f75 0 2024-03-26T18:54:32.287 INFO:tasks.workunit.client.0.smithi102.stdout:5/425: creat d2/d25/d2a/d5d/d82/f97 x:0 0 0 2024-03-26T18:54:32.288 INFO:tasks.workunit.client.0.smithi102.stdout:8/163: mkdir d0/d3/d2c/d2f 0 2024-03-26T18:54:32.295 INFO:tasks.workunit.client.1.smithi190.stdout:3/431: symlink d1/d47/l7f 0 2024-03-26T18:54:32.295 INFO:tasks.workunit.client.1.smithi190.stdout:3/432: stat d1/de/f2d 0 2024-03-26T18:54:32.296 INFO:tasks.workunit.client.0.smithi102.stdout:4/140: dwrite d1/f9 [0,4194304] 0 2024-03-26T18:54:32.296 INFO:tasks.workunit.client.0.smithi102.stdout:0/215: unlink d6/de/d13/f43 0 2024-03-26T18:54:32.297 INFO:tasks.workunit.client.1.smithi190.stdout:9/775: truncate d12/d2d/d1d/d23/d83/dc9/dd4/f6c 3273754 0 2024-03-26T18:54:32.298 INFO:tasks.workunit.client.1.smithi190.stdout:1/424: link d3/da/dd/d2f/d5b/f63 d3/da/dd/f8c 0 2024-03-26T18:54:32.301 INFO:tasks.workunit.client.1.smithi190.stdout:6/445: creat d17/d28/d2c/d63/fa0 x:0 0 0 2024-03-26T18:54:32.301 INFO:tasks.workunit.client.0.smithi102.stdout:4/141: mknod d1/d17/d1f/c32 0 2024-03-26T18:54:32.304 INFO:tasks.workunit.client.1.smithi190.stdout:0/595: mkdir d0/dd5 0 2024-03-26T18:54:32.304 INFO:tasks.workunit.client.0.smithi102.stdout:0/216: mkdir d6/de/d13/d25/d31/d45 0 2024-03-26T18:54:32.305 INFO:tasks.workunit.client.0.smithi102.stdout:4/142: rmdir d1/d17/d1f 39 2024-03-26T18:54:32.306 INFO:tasks.workunit.client.1.smithi190.stdout:3/433: write d1/d3/df/d18/d1b/f21 [3995995,88323] 0 2024-03-26T18:54:32.307 INFO:tasks.workunit.client.1.smithi190.stdout:7/575: mkdir d3/d1b/d57/dd1 0 2024-03-26T18:54:32.307 INFO:tasks.workunit.client.1.smithi190.stdout:7/576: fsync d3/d6/d9/d1f/d84/d99/fa3 0 2024-03-26T18:54:32.308 INFO:tasks.workunit.client.0.smithi102.stdout:0/217: creat d6/de/d19/f46 x:0 0 0 2024-03-26T18:54:32.309 INFO:tasks.workunit.client.0.smithi102.stdout:0/218: write d6/de/d19/f30 [35582,113294] 0 2024-03-26T18:54:32.310 INFO:tasks.workunit.client.0.smithi102.stdout:0/219: chown d6/de/d13/d1f/c3c 5 1 2024-03-26T18:54:32.310 INFO:tasks.workunit.client.0.smithi102.stdout:0/220: readlink d6/de/l2e 0 2024-03-26T18:54:32.310 INFO:tasks.workunit.client.1.smithi190.stdout:1/425: dread d3/d9/f2a [0,4194304] 0 2024-03-26T18:54:32.310 INFO:tasks.workunit.client.0.smithi102.stdout:7/148: dwrite f4 [4194304,4194304] 0 2024-03-26T18:54:32.311 INFO:tasks.workunit.client.0.smithi102.stdout:0/221: creat d6/d2d/f47 x:0 0 0 2024-03-26T18:54:32.311 INFO:tasks.workunit.client.0.smithi102.stdout:0/222: read d6/de/d13/d1f/f27 [3703507,57633] 0 2024-03-26T18:54:32.312 INFO:tasks.workunit.client.1.smithi190.stdout:2/503: dwrite d6/d9/d3a/d3d/d2e/f51 [0,4194304] 0 2024-03-26T18:54:32.314 INFO:tasks.workunit.client.0.smithi102.stdout:3/339: dwrite df/f1a [0,4194304] 0 2024-03-26T18:54:32.314 INFO:tasks.workunit.client.0.smithi102.stdout:3/340: dread - df/d1c/d34/f74 zero size 2024-03-26T18:54:32.314 INFO:tasks.workunit.client.0.smithi102.stdout:2/183: dwrite d3/d6/f10 [0,4194304] 0 2024-03-26T18:54:32.314 INFO:tasks.workunit.client.0.smithi102.stdout:2/184: chown d3/d11/d27/f28 13 1 2024-03-26T18:54:32.317 INFO:tasks.workunit.client.1.smithi190.stdout:6/446: write d17/d28/d2b/f3c [1144785,31740] 0 2024-03-26T18:54:32.320 INFO:tasks.workunit.client.1.smithi190.stdout:3/434: rename d1/d3/c7 to d1/d47/c80 0 2024-03-26T18:54:32.322 INFO:tasks.workunit.client.1.smithi190.stdout:1/426: dread d3/d9/f20 [0,4194304] 0 2024-03-26T18:54:32.322 INFO:tasks.workunit.client.1.smithi190.stdout:1/427: read - d3/da/dd/d2f/d34/d3e/d52/f7e zero size 2024-03-26T18:54:32.322 INFO:tasks.workunit.client.1.smithi190.stdout:1/428: write d3/da/dd/d2f/d34/f69 [5099370,84349] 0 2024-03-26T18:54:32.322 INFO:tasks.workunit.client.1.smithi190.stdout:1/429: stat d3/da/f78 0 2024-03-26T18:54:32.322 INFO:tasks.workunit.client.1.smithi190.stdout:1/430: write d3/f53 [421026,38733] 0 2024-03-26T18:54:32.323 INFO:tasks.workunit.client.0.smithi102.stdout:9/110: sync 2024-03-26T18:54:32.323 INFO:tasks.workunit.client.0.smithi102.stdout:9/111: getdents df/d1a/d1d 0 2024-03-26T18:54:32.323 INFO:tasks.workunit.client.0.smithi102.stdout:9/112: write df/d1a/f1c [83793,65050] 0 2024-03-26T18:54:32.323 INFO:tasks.workunit.client.0.smithi102.stdout:7/149: mknod d7/d12/c2d 0 2024-03-26T18:54:32.324 INFO:tasks.workunit.client.1.smithi190.stdout:9/776: rmdir d12/d2d/d1d/d23/d83/dc9/dd4/d3f 39 2024-03-26T18:54:32.324 INFO:tasks.workunit.client.0.smithi102.stdout:4/143: write d1/d17/d1f/f2d [729218,113323] 0 2024-03-26T18:54:32.324 INFO:tasks.workunit.client.0.smithi102.stdout:4/144: chown d1/c13 25944107 1 2024-03-26T18:54:32.329 INFO:tasks.workunit.client.1.smithi190.stdout:0/596: symlink d0/dd/d20/d1c/ld6 0 2024-03-26T18:54:32.333 INFO:tasks.workunit.client.0.smithi102.stdout:3/341: mkdir df/d29/d31/d36/d3c/d6f/d75 0 2024-03-26T18:54:32.336 INFO:tasks.workunit.client.0.smithi102.stdout:9/113: write df/f17 [1488428,79149] 0 2024-03-26T18:54:32.339 INFO:tasks.workunit.client.1.smithi190.stdout:3/435: mkdir d1/de/d81 0 2024-03-26T18:54:32.341 INFO:tasks.workunit.client.0.smithi102.stdout:2/185: creat d3/d2a/f40 x:0 0 0 2024-03-26T18:54:32.341 INFO:tasks.workunit.client.0.smithi102.stdout:2/186: rename d3/d11 to d3/d11/d15/d41 22 2024-03-26T18:54:32.343 INFO:tasks.workunit.client.1.smithi190.stdout:9/777: creat d12/d2d/d1d/da1/dba/dd2/f111 x:0 0 0 2024-03-26T18:54:32.345 INFO:tasks.workunit.client.1.smithi190.stdout:0/597: truncate d0/f5b 336505 0 2024-03-26T18:54:32.348 INFO:tasks.workunit.client.1.smithi190.stdout:6/447: symlink d17/la1 0 2024-03-26T18:54:32.351 INFO:tasks.workunit.client.1.smithi190.stdout:0/598: creat d0/dd/d24/db4/fd7 x:0 0 0 2024-03-26T18:54:32.351 INFO:tasks.workunit.client.1.smithi190.stdout:0/599: chown d0/dd/d20/d1c/d46/da7 8 1 2024-03-26T18:54:32.351 INFO:tasks.workunit.client.0.smithi102.stdout:3/342: mknod df/d29/d31/d36/c76 0 2024-03-26T18:54:32.353 INFO:tasks.workunit.client.0.smithi102.stdout:3/343: dread df/f25 [4194304,4194304] 0 2024-03-26T18:54:32.353 INFO:tasks.workunit.client.0.smithi102.stdout:0/223: rmdir d6/d2d/d3d 39 2024-03-26T18:54:32.354 INFO:tasks.workunit.client.0.smithi102.stdout:9/114: symlink df/d1a/d1d/l1f 0 2024-03-26T18:54:32.354 INFO:tasks.workunit.client.0.smithi102.stdout:9/115: creat df/d1a/f20 x:0 0 0 2024-03-26T18:54:32.354 INFO:tasks.workunit.client.0.smithi102.stdout:9/116: fsync f7 0 2024-03-26T18:54:32.357 INFO:tasks.workunit.client.0.smithi102.stdout:3/344: mkdir df/d29/d31/d77 0 2024-03-26T18:54:32.358 INFO:tasks.workunit.client.1.smithi190.stdout:3/436: write d1/d3/f13 [3371812,62994] 0 2024-03-26T18:54:32.359 INFO:tasks.workunit.client.1.smithi190.stdout:7/577: dwrite d3/d6/d9/d14/f2a [0,4194304] 0 2024-03-26T18:54:32.359 INFO:tasks.workunit.client.1.smithi190.stdout:7/578: stat d3/d1b/c31 0 2024-03-26T18:54:32.360 INFO:tasks.workunit.client.1.smithi190.stdout:7/579: chown d3/f7c 1 1 2024-03-26T18:54:32.363 INFO:tasks.workunit.client.0.smithi102.stdout:8/164: unlink d0/d3/d26/f23 0 2024-03-26T18:54:32.363 INFO:tasks.workunit.client.1.smithi190.stdout:6/448: write d17/f21 [4185352,76752] 0 2024-03-26T18:54:32.364 INFO:tasks.workunit.client.1.smithi190.stdout:3/437: link d1/d3/d5a/d6e/l77 d1/d3/d5a/d6e/l82 0 2024-03-26T18:54:32.365 INFO:tasks.workunit.client.0.smithi102.stdout:3/345: symlink df/d29/d3a/l78 0 2024-03-26T18:54:32.365 INFO:tasks.workunit.client.0.smithi102.stdout:3/346: chown df/d1c/d34/f74 1279420 1 2024-03-26T18:54:32.368 INFO:tasks.workunit.client.1.smithi190.stdout:7/580: mkdir d3/d1b/dd2 0 2024-03-26T18:54:32.368 INFO:tasks.workunit.client.1.smithi190.stdout:7/581: write d3/d6/d9/d1f/d41/d65/d90/d92/d9b/fc1 [3560526,121204] 0 2024-03-26T18:54:32.368 INFO:tasks.workunit.client.0.smithi102.stdout:8/165: symlink d0/d3/d26/d19/d27/l30 0 2024-03-26T18:54:32.370 INFO:tasks.workunit.client.0.smithi102.stdout:3/347: dread df/d61/f66 [0,4194304] 0 2024-03-26T18:54:32.370 INFO:tasks.workunit.client.1.smithi190.stdout:2/504: dwrite d6/d9/d3a/d3d/d2e/d86/f8a [4194304,4194304] 0 2024-03-26T18:54:32.371 INFO:tasks.workunit.client.1.smithi190.stdout:2/505: fsync d6/d9/d3a/d3d/d1c/d44/f49 0 2024-03-26T18:54:32.371 INFO:tasks.workunit.client.1.smithi190.stdout:2/506: stat d6/d9/d3a/d3d/f57 0 2024-03-26T18:54:32.373 INFO:tasks.workunit.client.0.smithi102.stdout:8/166: mkdir d0/d3/d31 0 2024-03-26T18:54:32.373 INFO:tasks.workunit.client.0.smithi102.stdout:8/167: read d0/d3/f6 [192774,3350] 0 2024-03-26T18:54:32.373 INFO:tasks.workunit.client.0.smithi102.stdout:9/117: fsync df/d1a/f1c 0 2024-03-26T18:54:32.373 INFO:tasks.workunit.client.0.smithi102.stdout:2/187: dwrite d3/d6/ff [0,4194304] 0 2024-03-26T18:54:32.373 INFO:tasks.workunit.client.0.smithi102.stdout:4/145: dwrite d1/d17/d1a/d21/f2c [4194304,4194304] 0 2024-03-26T18:54:32.374 INFO:tasks.workunit.client.1.smithi190.stdout:9/778: dwrite d12/d2d/d1d/d54/f8a [0,4194304] 0 2024-03-26T18:54:32.374 INFO:tasks.workunit.client.1.smithi190.stdout:1/431: dwrite d3/da/dd/d2f/f5e [0,4194304] 0 2024-03-26T18:54:32.375 INFO:tasks.workunit.client.1.smithi190.stdout:7/582: mkdir d3/d1b/dd3 0 2024-03-26T18:54:32.375 INFO:tasks.workunit.client.1.smithi190.stdout:7/583: creat d3/d6/d9/d1f/d4c/fd4 x:0 0 0 2024-03-26T18:54:32.377 INFO:tasks.workunit.client.0.smithi102.stdout:3/348: link df/f17 df/d29/d31/f79 0 2024-03-26T18:54:32.380 INFO:tasks.workunit.client.0.smithi102.stdout:8/168: truncate d0/d3/d26/f1f 4237638 0 2024-03-26T18:54:32.380 INFO:tasks.workunit.client.0.smithi102.stdout:8/169: creat d0/d3/f32 x:0 0 0 2024-03-26T18:54:32.384 INFO:tasks.workunit.client.0.smithi102.stdout:0/224: rmdir d6/de/d13/d25/d31 39 2024-03-26T18:54:32.384 INFO:tasks.workunit.client.0.smithi102.stdout:0/225: write d6/de/d13/d1f/d39/f29 [32644,94504] 0 2024-03-26T18:54:32.384 INFO:tasks.workunit.client.0.smithi102.stdout:0/226: readlink d6/de/l2f 0 2024-03-26T18:54:32.384 INFO:tasks.workunit.client.0.smithi102.stdout:9/118: mknod df/c21 0 2024-03-26T18:54:32.387 INFO:tasks.workunit.client.1.smithi190.stdout:9/779: mkdir d12/d80/db2/d112 0 2024-03-26T18:54:32.389 INFO:tasks.workunit.client.0.smithi102.stdout:2/188: symlink d3/d2d/l42 0 2024-03-26T18:54:32.391 INFO:tasks.workunit.client.1.smithi190.stdout:1/432: unlink d3/da/dd/l80 0 2024-03-26T18:54:32.391 INFO:tasks.workunit.client.1.smithi190.stdout:1/433: fdatasync d3/d9/f20 0 2024-03-26T18:54:32.392 INFO:tasks.workunit.client.1.smithi190.stdout:1/434: dread d3/da/dd/d2f/d34/d3e/f56 [0,4194304] 0 2024-03-26T18:54:32.393 INFO:tasks.workunit.client.0.smithi102.stdout:7/150: truncate f4 5185843 0 2024-03-26T18:54:32.393 INFO:tasks.workunit.client.0.smithi102.stdout:7/151: chown d7/d10/d17/l23 0 1 2024-03-26T18:54:32.393 INFO:tasks.workunit.client.1.smithi190.stdout:6/449: dwrite d17/d28/d2c/d77/f69 [0,4194304] 0 2024-03-26T18:54:32.394 INFO:tasks.workunit.client.0.smithi102.stdout:9/119: dread f7 [0,4194304] 0 2024-03-26T18:54:32.394 INFO:tasks.workunit.client.0.smithi102.stdout:9/120: write df/d1a/f1c [126034,125938] 0 2024-03-26T18:54:32.396 INFO:tasks.workunit.client.1.smithi190.stdout:3/438: dwrite d1/de/d1c/d36/f3a [0,4194304] 0 2024-03-26T18:54:32.398 INFO:tasks.workunit.client.0.smithi102.stdout:3/349: symlink df/d29/d3a/l7a 0 2024-03-26T18:54:32.402 INFO:tasks.workunit.client.0.smithi102.stdout:8/170: mkdir d0/d3/d2c/d33 0 2024-03-26T18:54:32.402 INFO:tasks.workunit.client.0.smithi102.stdout:8/171: chown d0/d3/f6 764 1 2024-03-26T18:54:32.402 INFO:tasks.workunit.client.1.smithi190.stdout:8/506: sync 2024-03-26T18:54:32.402 INFO:tasks.workunit.client.1.smithi190.stdout:8/507: write d9/d39/d79/f90 [850003,32082] 0 2024-03-26T18:54:32.405 INFO:tasks.workunit.client.0.smithi102.stdout:2/189: truncate d3/f4 936311 0 2024-03-26T18:54:32.408 INFO:tasks.workunit.client.1.smithi190.stdout:7/584: creat d3/d6/d9/fd5 x:0 0 0 2024-03-26T18:54:32.408 INFO:tasks.workunit.client.1.smithi190.stdout:7/585: write d3/d6/d9/d1f/d4c/f54 [4501707,47045] 0 2024-03-26T18:54:32.408 INFO:tasks.workunit.client.0.smithi102.stdout:7/152: getdents d7/d10/d17 0 2024-03-26T18:54:32.410 INFO:tasks.workunit.client.0.smithi102.stdout:9/121: truncate df/f13 1136333 0 2024-03-26T18:54:32.411 INFO:tasks.workunit.client.1.smithi190.stdout:7/586: dread d3/d6/d9/d1f/d41/d65/f7d [0,4194304] 0 2024-03-26T18:54:32.411 INFO:tasks.workunit.client.1.smithi190.stdout:7/587: creat d3/d6/d9/d1f/d84/d99/fd6 x:0 0 0 2024-03-26T18:54:32.411 INFO:tasks.workunit.client.0.smithi102.stdout:4/146: symlink d1/l33 0 2024-03-26T18:54:32.413 INFO:tasks.workunit.client.1.smithi190.stdout:1/435: mknod d3/d9/c8d 0 2024-03-26T18:54:32.413 INFO:tasks.workunit.client.1.smithi190.stdout:1/436: stat d3/da/dd/d2f/d5b/f62 0 2024-03-26T18:54:32.413 INFO:tasks.workunit.client.1.smithi190.stdout:1/437: chown d3/da/dd/d2f/d34/d3e/l82 238 1 2024-03-26T18:54:32.413 INFO:tasks.workunit.client.1.smithi190.stdout:1/438: fdatasync d3/d9/f16 0 2024-03-26T18:54:32.413 INFO:tasks.workunit.client.1.smithi190.stdout:1/439: dread - d3/da/d2e/d59/f65 zero size 2024-03-26T18:54:32.413 INFO:tasks.workunit.client.1.smithi190.stdout:1/440: chown d3/da/dd/d2f/d33/c44 5949 1 2024-03-26T18:54:32.413 INFO:tasks.workunit.client.1.smithi190.stdout:1/441: truncate d3/d9/f57 597529 0 2024-03-26T18:54:32.413 INFO:tasks.workunit.client.1.smithi190.stdout:1/442: creat d3/da/dd/d2f/d34/d3e/f8e x:0 0 0 2024-03-26T18:54:32.414 INFO:tasks.workunit.client.1.smithi190.stdout:1/443: chown d3/da/d2e 168 1 2024-03-26T18:54:32.415 INFO:tasks.workunit.client.0.smithi102.stdout:8/172: creat d0/d3/d2c/f34 x:0 0 0 2024-03-26T18:54:32.419 INFO:tasks.workunit.client.1.smithi190.stdout:8/508: creat d9/da2/d73/fb0 x:0 0 0 2024-03-26T18:54:32.419 INFO:tasks.workunit.client.0.smithi102.stdout:6/208: sync 2024-03-26T18:54:32.420 INFO:tasks.workunit.client.0.smithi102.stdout:0/227: rename d6/d2d/f47 to d6/de/d13/d25/d31/d45/f48 0 2024-03-26T18:54:32.423 INFO:tasks.workunit.client.0.smithi102.stdout:0/228: dread d6/f7 [0,4194304] 0 2024-03-26T18:54:32.426 INFO:tasks.workunit.client.0.smithi102.stdout:9/122: link df/c21 df/c22 0 2024-03-26T18:54:32.428 INFO:tasks.workunit.client.1.smithi190.stdout:2/507: rename d6/d9/d3a/d3d/d1c/d44 to d6/d9/d10/db7 0 2024-03-26T18:54:32.429 INFO:tasks.workunit.client.0.smithi102.stdout:3/350: rename df/f20 to df/f7b 0 2024-03-26T18:54:32.432 INFO:tasks.workunit.client.0.smithi102.stdout:6/209: mknod d2/df/c39 0 2024-03-26T18:54:32.433 INFO:tasks.workunit.client.0.smithi102.stdout:4/147: dread d1/d17/d1a/f1c [0,4194304] 0 2024-03-26T18:54:32.434 INFO:tasks.workunit.client.1.smithi190.stdout:1/444: mkdir d3/d8f 0 2024-03-26T18:54:32.434 INFO:tasks.workunit.client.1.smithi190.stdout:1/445: chown d3/d5f/l67 264 1 2024-03-26T18:54:32.434 INFO:tasks.workunit.client.1.smithi190.stdout:1/446: creat d3/da/dd/d2f/d5b/f90 x:0 0 0 2024-03-26T18:54:32.436 INFO:tasks.workunit.client.0.smithi102.stdout:0/229: write d6/fb [7785337,74615] 0 2024-03-26T18:54:32.436 INFO:tasks.workunit.client.0.smithi102.stdout:7/153: rename d7/d10/d17/l2c to d7/d12/l2e 0 2024-03-26T18:54:32.437 INFO:tasks.workunit.client.1.smithi190.stdout:8/509: rmdir d9/d39/d79/dad 39 2024-03-26T18:54:32.438 INFO:tasks.workunit.client.0.smithi102.stdout:2/190: rmdir d3/d2a 39 2024-03-26T18:54:32.439 INFO:tasks.workunit.client.1.smithi190.stdout:2/508: mknod d6/d9/d1a/cb8 0 2024-03-26T18:54:32.440 INFO:tasks.workunit.client.0.smithi102.stdout:9/123: mkdir df/d1a/d23 0 2024-03-26T18:54:32.440 INFO:tasks.workunit.client.0.smithi102.stdout:9/124: creat df/d1a/f24 x:0 0 0 2024-03-26T18:54:32.443 INFO:tasks.workunit.client.0.smithi102.stdout:6/210: link d2/d7/f1d d2/f3a 0 2024-03-26T18:54:32.443 INFO:tasks.workunit.client.0.smithi102.stdout:6/211: read - d2/d7/f30 zero size 2024-03-26T18:54:32.445 INFO:tasks.workunit.client.1.smithi190.stdout:8/510: mkdir d9/dd/d23/db1 0 2024-03-26T18:54:32.446 INFO:tasks.workunit.client.0.smithi102.stdout:0/230: dread d6/de/f1d [0,4194304] 0 2024-03-26T18:54:32.447 INFO:tasks.workunit.client.0.smithi102.stdout:4/148: rename d1/d17/d1f/c32 to d1/d17/d1f/c34 0 2024-03-26T18:54:32.447 INFO:tasks.workunit.client.0.smithi102.stdout:4/149: write d1/f8 [4435159,106775] 0 2024-03-26T18:54:32.448 INFO:tasks.workunit.client.1.smithi190.stdout:1/447: dread d3/da/f24 [0,4194304] 0 2024-03-26T18:54:32.449 INFO:tasks.workunit.client.0.smithi102.stdout:3/351: dread df/d29/f33 [0,4194304] 0 2024-03-26T18:54:32.452 INFO:tasks.workunit.client.0.smithi102.stdout:7/154: unlink d7/l28 0 2024-03-26T18:54:32.452 INFO:tasks.workunit.client.0.smithi102.stdout:7/155: truncate d7/d10/f1c 990980 0 2024-03-26T18:54:32.453 INFO:tasks.workunit.client.0.smithi102.stdout:2/191: symlink d3/d11/d15/d31/l43 0 2024-03-26T18:54:32.454 INFO:tasks.workunit.client.0.smithi102.stdout:2/192: chown d3/d11/l1e 317 1 2024-03-26T18:54:32.455 INFO:tasks.workunit.client.1.smithi190.stdout:1/448: truncate d3/da/dd/d2f/f79 3652432 0 2024-03-26T18:54:32.459 INFO:tasks.workunit.client.1.smithi190.stdout:1/449: symlink d3/da/l91 0 2024-03-26T18:54:32.459 INFO:tasks.workunit.client.1.smithi190.stdout:1/450: chown d3/d9/l10 4 1 2024-03-26T18:54:32.460 INFO:tasks.workunit.client.1.smithi190.stdout:8/511: write d9/dd/d23/d36/d37/f5f [4014565,60535] 0 2024-03-26T18:54:32.462 INFO:tasks.workunit.client.0.smithi102.stdout:4/150: dread d1/f7 [0,4194304] 0 2024-03-26T18:54:32.462 INFO:tasks.workunit.client.0.smithi102.stdout:4/151: read - d1/f31 zero size 2024-03-26T18:54:32.462 INFO:tasks.workunit.client.0.smithi102.stdout:4/152: chown d1/d17/d1f/c34 44278106 1 2024-03-26T18:54:32.465 INFO:tasks.workunit.client.0.smithi102.stdout:4/153: unlink d1/l6 0 2024-03-26T18:54:32.465 INFO:tasks.workunit.client.0.smithi102.stdout:4/154: stat d1/d27/c29 0 2024-03-26T18:54:32.466 INFO:tasks.workunit.client.1.smithi190.stdout:8/512: dread d9/da2/f2b [0,4194304] 0 2024-03-26T18:54:32.466 INFO:tasks.workunit.client.1.smithi190.stdout:8/513: write d9/da2/d73/f9f [521246,107887] 0 2024-03-26T18:54:32.466 INFO:tasks.workunit.client.1.smithi190.stdout:8/514: truncate d9/da2/d73/fb0 378070 0 2024-03-26T18:54:32.466 INFO:tasks.workunit.client.1.smithi190.stdout:6/450: dwrite d17/d28/d2b/f3c [4194304,4194304] 0 2024-03-26T18:54:32.467 INFO:tasks.workunit.client.1.smithi190.stdout:6/451: write d17/d28/d2c/d77/d51/f91 [963257,74162] 0 2024-03-26T18:54:32.467 INFO:tasks.workunit.client.1.smithi190.stdout:6/452: truncate d17/d28/d2c/d77/f66 4578021 0 2024-03-26T18:54:32.467 INFO:tasks.workunit.client.1.smithi190.stdout:8/515: mknod d9/d39/cb2 0 2024-03-26T18:54:32.467 INFO:tasks.workunit.client.1.smithi190.stdout:8/516: fdatasync d9/dd/d23/d36/d37/d3a/d43/d8f/f98 0 2024-03-26T18:54:32.467 INFO:tasks.workunit.client.1.smithi190.stdout:8/517: chown d9/dd/f78 1639 1 2024-03-26T18:54:32.467 INFO:tasks.workunit.client.1.smithi190.stdout:8/518: chown d9/f14 105514087 1 2024-03-26T18:54:32.468 INFO:tasks.workunit.client.0.smithi102.stdout:0/231: dread d6/de/f28 [0,4194304] 0 2024-03-26T18:54:32.468 INFO:tasks.workunit.client.1.smithi190.stdout:3/439: dwrite d1/de/f73 [0,4194304] 0 2024-03-26T18:54:32.468 INFO:tasks.workunit.client.1.smithi190.stdout:3/440: creat d1/de/d1c/f83 x:0 0 0 2024-03-26T18:54:32.469 INFO:tasks.workunit.client.1.smithi190.stdout:3/441: read d1/d3/df/d18/f30 [2859902,54593] 0 2024-03-26T18:54:32.469 INFO:tasks.workunit.client.1.smithi190.stdout:6/453: creat d17/d28/d74/fa2 x:0 0 0 2024-03-26T18:54:32.470 INFO:tasks.workunit.client.0.smithi102.stdout:0/232: rename d6/de/d13/d1f/c3c to d6/de/d13/d1f/c49 0 2024-03-26T18:54:32.471 INFO:tasks.workunit.client.1.smithi190.stdout:8/519: mkdir d9/dd/d23/d36/d37/d3a/d61/daf/db3 0 2024-03-26T18:54:32.474 INFO:tasks.workunit.client.0.smithi102.stdout:0/233: mkdir d6/de/d13/d4a 0 2024-03-26T18:54:32.475 INFO:tasks.workunit.client.1.smithi190.stdout:7/588: dwrite d3/fe [4194304,4194304] 0 2024-03-26T18:54:32.475 INFO:tasks.workunit.client.1.smithi190.stdout:7/589: chown d3/d6/d9/d1f/d41/d50/c74 1594 1 2024-03-26T18:54:32.475 INFO:tasks.workunit.client.1.smithi190.stdout:7/590: write d3/d6/d9/d1f/d41/d65/f7d [687242,121659] 0 2024-03-26T18:54:32.475 INFO:tasks.workunit.client.1.smithi190.stdout:0/600: dwrite d0/f5b [0,4194304] 0 2024-03-26T18:54:32.475 INFO:tasks.workunit.client.1.smithi190.stdout:0/601: write d0/dd/d20/d1c/d4e/f9d [1002364,54415] 0 2024-03-26T18:54:32.475 INFO:tasks.workunit.client.1.smithi190.stdout:0/602: readlink d0/dd/d24/d4b/d5e/d81/l5a 0 2024-03-26T18:54:32.476 INFO:tasks.workunit.client.0.smithi102.stdout:8/173: dwrite d0/d3/f21 [0,4194304] 0 2024-03-26T18:54:32.476 INFO:tasks.workunit.client.0.smithi102.stdout:8/174: fdatasync d0/d3/f14 0 2024-03-26T18:54:32.477 INFO:tasks.workunit.client.0.smithi102.stdout:8/175: readlink d0/d3/l7 0 2024-03-26T18:54:32.477 INFO:tasks.workunit.client.0.smithi102.stdout:4/155: dread d1/f7 [0,4194304] 0 2024-03-26T18:54:32.477 INFO:tasks.workunit.client.0.smithi102.stdout:4/156: fdatasync d1/f31 0 2024-03-26T18:54:32.477 INFO:tasks.workunit.client.1.smithi190.stdout:2/509: dwrite d6/f3c [0,4194304] 0 2024-03-26T18:54:32.480 INFO:tasks.workunit.client.1.smithi190.stdout:3/442: symlink d1/d3/d52/d7d/l84 0 2024-03-26T18:54:32.480 INFO:tasks.workunit.client.1.smithi190.stdout:3/443: truncate d1/de/d1c/d36/f40 966945 0 2024-03-26T18:54:32.480 INFO:tasks.workunit.client.0.smithi102.stdout:6/212: dwrite d2/d3/f38 [0,4194304] 0 2024-03-26T18:54:32.481 INFO:tasks.workunit.client.0.smithi102.stdout:6/213: getdents d2/df/d35 0 2024-03-26T18:54:32.482 INFO:tasks.workunit.client.1.smithi190.stdout:8/520: truncate d9/d39/d79/f7e 838412 0 2024-03-26T18:54:32.483 INFO:tasks.workunit.client.1.smithi190.stdout:6/454: dread fc [0,4194304] 0 2024-03-26T18:54:32.486 INFO:tasks.workunit.client.0.smithi102.stdout:2/193: fsync d3/f4 0 2024-03-26T18:54:32.487 INFO:tasks.workunit.client.0.smithi102.stdout:8/176: truncate d0/d3/d26/f22 1270589 0 2024-03-26T18:54:32.490 INFO:tasks.workunit.client.0.smithi102.stdout:8/177: dread d0/d3/d26/f22 [0,4194304] 0 2024-03-26T18:54:32.492 INFO:tasks.workunit.client.1.smithi190.stdout:1/451: dwrite d3/da/f24 [4194304,4194304] 0 2024-03-26T18:54:32.494 INFO:tasks.workunit.client.0.smithi102.stdout:2/194: dread d3/f8 [0,4194304] 0 2024-03-26T18:54:32.495 INFO:tasks.workunit.client.1.smithi190.stdout:0/603: mkdir d0/dd/d24/d87/d93/dc4/dd8 0 2024-03-26T18:54:32.496 INFO:tasks.workunit.client.0.smithi102.stdout:6/214: mknod d2/d12/c3b 0 2024-03-26T18:54:32.497 INFO:tasks.workunit.client.0.smithi102.stdout:6/215: read d2/f21 [75042,115151] 0 2024-03-26T18:54:32.501 INFO:tasks.workunit.client.1.smithi190.stdout:2/510: unlink d6/d9/d10/f70 0 2024-03-26T18:54:32.501 INFO:tasks.workunit.client.1.smithi190.stdout:2/511: readlink d6/d9/d3a/d3d/d1c/l53 0 2024-03-26T18:54:32.501 INFO:tasks.workunit.client.1.smithi190.stdout:2/512: write d6/d21/fb5 [567130,14895] 0 2024-03-26T18:54:32.503 INFO:tasks.workunit.client.1.smithi190.stdout:3/444: rmdir d1/d47/d75 0 2024-03-26T18:54:32.510 INFO:tasks.workunit.client.1.smithi190.stdout:8/521: creat d9/dd/d23/d36/d37/fb4 x:0 0 0 2024-03-26T18:54:32.514 INFO:tasks.workunit.client.1.smithi190.stdout:4/551: sync 2024-03-26T18:54:32.514 INFO:tasks.workunit.client.1.smithi190.stdout:5/384: sync 2024-03-26T18:54:32.514 INFO:tasks.workunit.client.0.smithi102.stdout:9/125: rename df/f13 to df/d1a/d23/f25 0 2024-03-26T18:54:32.514 INFO:tasks.workunit.client.0.smithi102.stdout:2/195: read d3/d6/fe [891799,47023] 0 2024-03-26T18:54:32.514 INFO:tasks.workunit.client.0.smithi102.stdout:2/196: creat d3/d6/f44 x:0 0 0 2024-03-26T18:54:32.515 INFO:tasks.workunit.client.1.smithi190.stdout:2/513: write d6/d9/d3a/d3d/d1c/f5e [2097004,126424] 0 2024-03-26T18:54:32.515 INFO:tasks.workunit.client.1.smithi190.stdout:6/455: symlink d17/d28/d38/d79/la3 0 2024-03-26T18:54:32.515 INFO:tasks.workunit.client.1.smithi190.stdout:6/456: fsync d17/f25 0 2024-03-26T18:54:32.515 INFO:tasks.workunit.client.1.smithi190.stdout:6/457: fsync d17/f19 0 2024-03-26T18:54:32.515 INFO:tasks.workunit.client.1.smithi190.stdout:6/458: fsync f2 0 2024-03-26T18:54:32.519 INFO:tasks.workunit.client.0.smithi102.stdout:7/156: dwrite f4 [0,4194304] 0 2024-03-26T18:54:32.519 INFO:tasks.workunit.client.0.smithi102.stdout:7/157: chown d7/d12/c2d 85118 1 2024-03-26T18:54:32.519 INFO:tasks.workunit.client.0.smithi102.stdout:0/234: dwrite d6/d2d/d3d/f44 [0,4194304] 0 2024-03-26T18:54:32.519 INFO:tasks.workunit.client.0.smithi102.stdout:0/235: creat d6/de/f4b x:0 0 0 2024-03-26T18:54:32.526 INFO:tasks.workunit.client.0.smithi102.stdout:8/178: rename d0/d3/f14 to d0/d15/d1b/f35 0 2024-03-26T18:54:32.529 INFO:tasks.workunit.client.0.smithi102.stdout:9/126: creat df/d1a/d1d/f26 x:0 0 0 2024-03-26T18:54:32.529 INFO:tasks.workunit.client.1.smithi190.stdout:4/552: dread d7/f18 [0,4194304] 0 2024-03-26T18:54:32.529 INFO:tasks.workunit.client.1.smithi190.stdout:4/553: truncate d7/d1d/fb3 199514 0 2024-03-26T18:54:32.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:32 smithi190 ceph-mon[35043]: pgmap v244: 65 pgs: 65 active+clean; 1.2 GiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 112 MiB/s rd, 142 MiB/s wr, 616 op/s 2024-03-26T18:54:32.533 INFO:tasks.workunit.client.0.smithi102.stdout:7/158: dread d7/f20 [0,4194304] 0 2024-03-26T18:54:32.533 INFO:tasks.workunit.client.0.smithi102.stdout:7/159: fdatasync d7/f9 0 2024-03-26T18:54:32.533 INFO:tasks.workunit.client.0.smithi102.stdout:7/160: readlink d7/d10/l18 0 2024-03-26T18:54:32.533 INFO:tasks.workunit.client.0.smithi102.stdout:7/161: chown d7/fb 14828 1 2024-03-26T18:54:32.533 INFO:tasks.workunit.client.0.smithi102.stdout:7/162: write d7/f20 [4807005,9522] 0 2024-03-26T18:54:32.533 INFO:tasks.workunit.client.0.smithi102.stdout:2/197: symlink d3/d11/d15/d31/l45 0 2024-03-26T18:54:32.534 INFO:tasks.workunit.client.1.smithi190.stdout:0/604: dwrite d0/dd/d20/f78 [0,4194304] 0 2024-03-26T18:54:32.535 INFO:tasks.workunit.client.0.smithi102.stdout:4/157: dwrite d1/d17/d1f/f2d [4194304,4194304] 0 2024-03-26T18:54:32.535 INFO:tasks.workunit.client.1.smithi190.stdout:3/445: dwrite d1/d3/df/d18/d1b/f21 [0,4194304] 0 2024-03-26T18:54:32.535 INFO:tasks.workunit.client.1.smithi190.stdout:3/446: fdatasync d1/d3/df/d18/f30 0 2024-03-26T18:54:32.538 INFO:tasks.workunit.client.1.smithi190.stdout:4/554: dread d7/f18 [0,4194304] 0 2024-03-26T18:54:32.540 INFO:tasks.workunit.client.0.smithi102.stdout:6/216: getdents d2/df 0 2024-03-26T18:54:32.542 INFO:tasks.workunit.client.0.smithi102.stdout:8/179: mknod d0/d3/c36 0 2024-03-26T18:54:32.542 INFO:tasks.workunit.client.0.smithi102.stdout:8/180: chown d0/c1 11623 1 2024-03-26T18:54:32.546 INFO:tasks.workunit.client.0.smithi102.stdout:9/127: creat df/f27 x:0 0 0 2024-03-26T18:54:32.546 INFO:tasks.workunit.client.0.smithi102.stdout:9/128: write df/f17 [2302988,102011] 0 2024-03-26T18:54:32.549 INFO:tasks.workunit.client.0.smithi102.stdout:2/198: mkdir d3/d11/d15/d46 0 2024-03-26T18:54:32.552 INFO:tasks.workunit.client.1.smithi190.stdout:7/591: rmdir d3/d6 39 2024-03-26T18:54:32.552 INFO:tasks.workunit.client.1.smithi190.stdout:7/592: write d3/d6/d9/d14/fc2 [485179,123595] 0 2024-03-26T18:54:32.553 INFO:tasks.workunit.client.0.smithi102.stdout:4/158: unlink d1/d27/c29 0 2024-03-26T18:54:32.554 INFO:tasks.workunit.client.0.smithi102.stdout:4/159: write d1/f31 [657876,119443] 0 2024-03-26T18:54:32.554 INFO:tasks.workunit.client.0.smithi102.stdout:4/160: chown d1/lc 11262 1 2024-03-26T18:54:32.554 INFO:tasks.workunit.client.0.smithi102.stdout:4/161: chown d1 8 1 2024-03-26T18:54:32.554 INFO:tasks.workunit.client.0.smithi102.stdout:4/162: chown d1 896 1 2024-03-26T18:54:32.554 INFO:tasks.workunit.client.0.smithi102.stdout:4/163: write d1/f8 [4336126,40219] 0 2024-03-26T18:54:32.554 INFO:tasks.workunit.client.0.smithi102.stdout:4/164: truncate d1/f8 5178820 0 2024-03-26T18:54:32.555 INFO:tasks.workunit.client.0.smithi102.stdout:4/165: fdatasync d1/f9 0 2024-03-26T18:54:32.555 INFO:tasks.workunit.client.0.smithi102.stdout:8/181: unlink d0/la 0 2024-03-26T18:54:32.556 INFO:tasks.workunit.client.0.smithi102.stdout:8/182: chown d0/c5 18 1 2024-03-26T18:54:32.560 INFO:tasks.workunit.client.1.smithi190.stdout:8/522: mkdir d9/dd/d23/d36/d37/d3a/d61/daf/db3/db5 0 2024-03-26T18:54:32.560 INFO:tasks.workunit.client.1.smithi190.stdout:8/523: getdents d9/da2/d87 0 2024-03-26T18:54:32.560 INFO:tasks.workunit.client.1.smithi190.stdout:5/385: mkdir d2/d17/d1b/d60/d63/d85/d89 0 2024-03-26T18:54:32.561 INFO:tasks.workunit.client.1.smithi190.stdout:5/386: dread - d2/d17/f7b zero size 2024-03-26T18:54:32.562 INFO:tasks.workunit.client.0.smithi102.stdout:0/236: dwrite d6/f11 [0,4194304] 0 2024-03-26T18:54:32.562 INFO:tasks.workunit.client.1.smithi190.stdout:0/605: dwrite d0/dd/d20/d61/f7b [8388608,4194304] 0 2024-03-26T18:54:32.563 INFO:tasks.workunit.client.0.smithi102.stdout:2/199: rename d3/d6/f10 to d3/d11/d15/d46/f47 0 2024-03-26T18:54:32.564 INFO:tasks.workunit.client.0.smithi102.stdout:6/217: dwrite d2/d12/f2d [0,4194304] 0 2024-03-26T18:54:32.564 INFO:tasks.workunit.client.0.smithi102.stdout:6/218: truncate d2/d7/f1d 5017511 0 2024-03-26T18:54:32.565 INFO:tasks.workunit.client.1.smithi190.stdout:2/514: mknod d6/cb9 0 2024-03-26T18:54:32.568 INFO:tasks.workunit.client.1.smithi190.stdout:6/459: creat d17/d28/d46/d6a/fa4 x:0 0 0 2024-03-26T18:54:32.571 INFO:tasks.workunit.client.0.smithi102.stdout:4/166: symlink d1/d17/l35 0 2024-03-26T18:54:32.571 INFO:tasks.workunit.client.0.smithi102.stdout:4/167: fdatasync d1/f7 0 2024-03-26T18:54:32.571 INFO:tasks.workunit.client.0.smithi102.stdout:4/168: write d1/f26 [697956,6654] 0 2024-03-26T18:54:32.572 INFO:tasks.workunit.client.0.smithi102.stdout:4/169: chown d1/f9 1163396937 1 2024-03-26T18:54:32.575 INFO:tasks.workunit.client.0.smithi102.stdout:8/183: dwrite d0/d3/d2c/f34 [0,4194304] 0 2024-03-26T18:54:32.594 INFO:tasks.workunit.client.0.smithi102.stdout:9/129: fsync df/f27 0 2024-03-26T18:54:32.595 INFO:tasks.workunit.client.1.smithi190.stdout:3/447: creat d1/d3/d52/d70/f85 x:0 0 0 2024-03-26T18:54:32.596 INFO:tasks.workunit.client.0.smithi102.stdout:9/130: truncate df/d1a/d23/f25 63866 0 2024-03-26T18:54:32.596 INFO:tasks.workunit.client.0.smithi102.stdout:9/131: creat df/f28 x:0 0 0 2024-03-26T18:54:32.597 INFO:tasks.workunit.client.1.smithi190.stdout:7/593: symlink d3/d6/d9/d1f/d41/d50/d53/d8f/ld7 0 2024-03-26T18:54:32.600 INFO:tasks.workunit.client.1.smithi190.stdout:1/452: rename d3/da/dd/d2f/f79 to d3/d73/d88/f92 0 2024-03-26T18:54:32.601 INFO:tasks.workunit.client.0.smithi102.stdout:9/132: mkdir df/d1a/d23/d29 0 2024-03-26T18:54:32.603 INFO:tasks.workunit.client.1.smithi190.stdout:8/524: unlink d9/dd/d15/f2f 0 2024-03-26T18:54:32.603 INFO:tasks.workunit.client.1.smithi190.stdout:8/525: truncate d9/dd/d15/f7a 938061 0 2024-03-26T18:54:32.603 INFO:tasks.workunit.client.1.smithi190.stdout:8/526: creat d9/dd/d23/d36/d37/d3a/d61/fb6 x:0 0 0 2024-03-26T18:54:32.603 INFO:tasks.workunit.client.1.smithi190.stdout:8/527: write d9/dd/d15/d32/f96 [677249,38652] 0 2024-03-26T18:54:32.603 INFO:tasks.workunit.client.1.smithi190.stdout:8/528: write d9/dd/d15/f2c [340143,37684] 0 2024-03-26T18:54:32.605 INFO:tasks.workunit.client.0.smithi102.stdout:9/133: mknod df/d1a/c2a 0 2024-03-26T18:54:32.605 INFO:tasks.workunit.client.0.smithi102.stdout:9/134: readlink l5 0 2024-03-26T18:54:32.607 INFO:tasks.workunit.client.0.smithi102.stdout:6/219: dwrite d2/d7/f36 [0,4194304] 0 2024-03-26T18:54:32.608 INFO:tasks.workunit.client.1.smithi190.stdout:5/387: mknod d2/d17/d1b/d1e/d3a/d3c/c8a 0 2024-03-26T18:54:32.608 INFO:tasks.workunit.client.1.smithi190.stdout:5/388: fsync d2/f16 0 2024-03-26T18:54:32.608 INFO:tasks.workunit.client.1.smithi190.stdout:5/389: write d2/d17/d1b/d30/f73 [553403,116569] 0 2024-03-26T18:54:32.608 INFO:tasks.workunit.client.1.smithi190.stdout:5/390: truncate d2/d17/d1b/f72 476823 0 2024-03-26T18:54:32.609 INFO:tasks.workunit.client.0.smithi102.stdout:2/200: dwrite d3/d11/d15/d46/f47 [0,4194304] 0 2024-03-26T18:54:32.610 INFO:tasks.workunit.client.1.smithi190.stdout:2/515: creat d6/d9/d1a/fba x:0 0 0 2024-03-26T18:54:32.611 INFO:tasks.workunit.client.0.smithi102.stdout:9/135: rmdir df/d1a/d23 39 2024-03-26T18:54:32.614 INFO:tasks.workunit.client.0.smithi102.stdout:8/184: dwrite d0/d3/f28 [0,4194304] 0 2024-03-26T18:54:32.616 INFO:tasks.workunit.client.1.smithi190.stdout:3/448: mkdir d1/d3/df/d18/d35/d41/d51/d86 0 2024-03-26T18:54:32.616 INFO:tasks.workunit.client.0.smithi102.stdout:6/220: mkdir d2/d7/d3c 0 2024-03-26T18:54:32.616 INFO:tasks.workunit.client.0.smithi102.stdout:6/221: dread - d2/f33 zero size 2024-03-26T18:54:32.617 INFO:tasks.workunit.client.1.smithi190.stdout:7/594: symlink d3/d6/d9/d1f/d41/d65/d90/ld8 0 2024-03-26T18:54:32.617 INFO:tasks.workunit.client.1.smithi190.stdout:7/595: dread - d3/d6/f91 zero size 2024-03-26T18:54:32.619 INFO:tasks.workunit.client.1.smithi190.stdout:4/555: rename d7/d8/d1c/l72 to d7/d1d/d42/d4a/lb8 0 2024-03-26T18:54:32.621 INFO:tasks.workunit.client.1.smithi190.stdout:1/453: dwrite d3/da/dd/d2f/d33/f8a [0,4194304] 0 2024-03-26T18:54:32.621 INFO:tasks.workunit.client.1.smithi190.stdout:1/454: creat d3/da/dd/d2f/d34/d3e/d52/f93 x:0 0 0 2024-03-26T18:54:32.623 INFO:tasks.workunit.client.1.smithi190.stdout:0/606: rename d0/dd/d24/d87/fc1 to d0/dd/d20/d1c/d8b/fd9 0 2024-03-26T18:54:32.624 INFO:tasks.workunit.client.0.smithi102.stdout:4/170: dwrite d1/d17/d1a/d21/f2c [8388608,4194304] 0 2024-03-26T18:54:32.624 INFO:tasks.workunit.client.1.smithi190.stdout:5/391: dread d2/d17/f83 [0,4194304] 0 2024-03-26T18:54:32.625 INFO:tasks.workunit.client.1.smithi190.stdout:5/392: chown d2/d17/d1b/d60/f7c 102765252 1 2024-03-26T18:54:32.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:32 smithi102 ceph-mon[29002]: pgmap v244: 65 pgs: 65 active+clean; 1.2 GiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 112 MiB/s rd, 142 MiB/s wr, 616 op/s 2024-03-26T18:54:32.628 INFO:tasks.workunit.client.1.smithi190.stdout:1/455: creat d3/da/f94 x:0 0 0 2024-03-26T18:54:32.628 INFO:tasks.workunit.client.1.smithi190.stdout:7/596: dread d3/d6/d9/d1f/d4c/f7f [0,4194304] 0 2024-03-26T18:54:32.628 INFO:tasks.workunit.client.1.smithi190.stdout:7/597: chown d3/f29 44273495 1 2024-03-26T18:54:32.629 INFO:tasks.workunit.client.1.smithi190.stdout:8/529: rename d9/dd/d23/d36/d37/f5f to d9/dd/d23/d36/d37/d3a/fb7 0 2024-03-26T18:54:32.630 INFO:tasks.workunit.client.0.smithi102.stdout:2/201: creat d3/d11/d15/f48 x:0 0 0 2024-03-26T18:54:32.631 INFO:tasks.workunit.client.1.smithi190.stdout:5/393: creat d2/d17/d1b/d22/d88/f8b x:0 0 0 2024-03-26T18:54:32.631 INFO:tasks.workunit.client.1.smithi190.stdout:3/449: dwrite d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:32.632 INFO:tasks.workunit.client.0.smithi102.stdout:9/136: symlink df/d1a/l2b 0 2024-03-26T18:54:32.635 INFO:tasks.workunit.client.1.smithi190.stdout:8/530: mkdir d9/dd/d23/d36/d37/db8 0 2024-03-26T18:54:32.638 INFO:tasks.workunit.client.1.smithi190.stdout:2/516: rename d6/d21/d61/c71 to d6/cbb 0 2024-03-26T18:54:32.638 INFO:tasks.workunit.client.1.smithi190.stdout:2/517: chown d6/l7b 12000380 1 2024-03-26T18:54:32.639 INFO:tasks.workunit.client.0.smithi102.stdout:8/185: mknod d0/d3/d31/c37 0 2024-03-26T18:54:32.639 INFO:tasks.workunit.client.0.smithi102.stdout:8/186: chown d0/d15/d1b 1752762930 1 2024-03-26T18:54:32.639 INFO:tasks.workunit.client.0.smithi102.stdout:8/187: chown d0/d3/d26 107 1 2024-03-26T18:54:32.640 INFO:tasks.workunit.client.1.smithi190.stdout:5/394: truncate d2/d17/d1b/d1e/f5f 3488453 0 2024-03-26T18:54:32.642 INFO:tasks.workunit.client.0.smithi102.stdout:6/222: mkdir d2/d3d 0 2024-03-26T18:54:32.644 INFO:tasks.workunit.client.0.smithi102.stdout:4/171: mknod d1/d17/d1a/c36 0 2024-03-26T18:54:32.646 INFO:tasks.workunit.client.1.smithi190.stdout:0/607: dwrite d0/f5b [0,4194304] 0 2024-03-26T18:54:32.647 INFO:tasks.workunit.client.0.smithi102.stdout:2/202: mknod d3/d11/d15/c49 0 2024-03-26T18:54:32.649 INFO:tasks.workunit.client.0.smithi102.stdout:9/137: creat df/d1a/d23/f2c x:0 0 0 2024-03-26T18:54:32.651 INFO:tasks.workunit.client.0.smithi102.stdout:8/188: rename d0/d3/d2c/c2d to d0/d3/d2c/d2f/c38 0 2024-03-26T18:54:32.654 INFO:tasks.workunit.client.1.smithi190.stdout:9/780: sync 2024-03-26T18:54:32.654 INFO:tasks.workunit.client.0.smithi102.stdout:4/172: unlink d1/l33 0 2024-03-26T18:54:32.654 INFO:tasks.workunit.client.0.smithi102.stdout:6/223: write d2/d3/f38 [1851881,125898] 0 2024-03-26T18:54:32.655 INFO:tasks.workunit.client.0.smithi102.stdout:6/224: stat d2/df/f27 0 2024-03-26T18:54:32.655 INFO:tasks.workunit.client.0.smithi102.stdout:6/225: chown d2/d7/l1a 3054 1 2024-03-26T18:54:32.656 INFO:tasks.workunit.client.1.smithi190.stdout:8/531: mkdir d9/dd/d23/d36/d37/db9 0 2024-03-26T18:54:32.656 INFO:tasks.workunit.client.1.smithi190.stdout:3/450: dwrite d1/d3/df/d31/d5b/f6b [0,4194304] 0 2024-03-26T18:54:32.656 INFO:tasks.workunit.client.1.smithi190.stdout:3/451: readlink d1/de/d1c/d36/l4d 0 2024-03-26T18:54:32.658 INFO:tasks.workunit.client.0.smithi102.stdout:2/203: symlink d3/d11/d15/l4a 0 2024-03-26T18:54:32.661 INFO:tasks.workunit.client.1.smithi190.stdout:4/556: rename d7/d8/d14/d4b/f77 to d7/fb9 0 2024-03-26T18:54:32.663 INFO:tasks.workunit.client.1.smithi190.stdout:2/518: dwrite d6/d9/d3a/f94 [4194304,4194304] 0 2024-03-26T18:54:32.663 INFO:tasks.workunit.client.1.smithi190.stdout:2/519: dread - d6/d9/d10/db7/f49 zero size 2024-03-26T18:54:32.664 INFO:tasks.workunit.client.0.smithi102.stdout:9/138: link df/f27 df/d1a/f2d 0 2024-03-26T18:54:32.664 INFO:tasks.workunit.client.1.smithi190.stdout:5/395: dread d2/f15 [0,4194304] 0 2024-03-26T18:54:32.664 INFO:tasks.workunit.client.1.smithi190.stdout:8/532: dread d9/d48/f58 [0,4194304] 0 2024-03-26T18:54:32.666 INFO:tasks.workunit.client.1.smithi190.stdout:0/608: creat d0/dd/d20/d1c/d46/da7/fda x:0 0 0 2024-03-26T18:54:32.666 INFO:tasks.workunit.client.1.smithi190.stdout:0/609: dread - d0/dd/d20/d1c/d8b/fc3 zero size 2024-03-26T18:54:32.669 INFO:tasks.workunit.client.1.smithi190.stdout:9/781: unlink d12/d2d/f97 0 2024-03-26T18:54:32.672 INFO:tasks.workunit.client.0.smithi102.stdout:8/189: chown d0/d3/d26/c1d 691055 1 2024-03-26T18:54:32.675 INFO:tasks.workunit.client.0.smithi102.stdout:4/173: mknod d1/d25/c37 0 2024-03-26T18:54:32.675 INFO:tasks.workunit.client.0.smithi102.stdout:4/174: truncate d1/f31 1708312 0 2024-03-26T18:54:32.675 INFO:tasks.workunit.client.0.smithi102.stdout:6/226: dwrite d2/f29 [0,4194304] 0 2024-03-26T18:54:32.679 INFO:tasks.workunit.client.0.smithi102.stdout:2/204: mkdir d3/d6/d1a/d4b 0 2024-03-26T18:54:32.682 INFO:tasks.workunit.client.1.smithi190.stdout:7/598: rename d3/d6/d9/fd5 to d3/d6/d9/d1f/d41/d65/d90/d92/d9b/fd9 0 2024-03-26T18:54:32.684 INFO:tasks.workunit.client.0.smithi102.stdout:9/139: mknod df/d1a/d23/c2e 0 2024-03-26T18:54:32.688 INFO:tasks.workunit.client.0.smithi102.stdout:8/190: mknod d0/d3/d31/c39 0 2024-03-26T18:54:32.688 INFO:tasks.workunit.client.0.smithi102.stdout:1/206: sync 2024-03-26T18:54:32.690 INFO:tasks.workunit.client.1.smithi190.stdout:1/456: rmdir d3/da/dd/d2f/d34/d3e 39 2024-03-26T18:54:32.691 INFO:tasks.workunit.client.0.smithi102.stdout:6/227: dread d2/f29 [0,4194304] 0 2024-03-26T18:54:32.692 INFO:tasks.workunit.client.0.smithi102.stdout:4/175: mknod d1/d17/c38 0 2024-03-26T18:54:32.698 INFO:tasks.workunit.client.0.smithi102.stdout:4/176: read d1/f7 [2304373,84355] 0 2024-03-26T18:54:32.698 INFO:tasks.workunit.client.0.smithi102.stdout:4/177: chown d1/d17/d1a/d21/c23 116877 1 2024-03-26T18:54:32.698 INFO:tasks.workunit.client.0.smithi102.stdout:4/178: fsync d1/d17/d1a/f1c 0 2024-03-26T18:54:32.700 INFO:tasks.workunit.client.0.smithi102.stdout:8/191: creat d0/d3/d31/f3a x:0 0 0 2024-03-26T18:54:32.701 INFO:tasks.workunit.client.0.smithi102.stdout:5/426: sync 2024-03-26T18:54:32.702 INFO:tasks.workunit.client.0.smithi102.stdout:1/207: mkdir de/d16/d1d/d2a/d2c/d4b 0 2024-03-26T18:54:32.702 INFO:tasks.workunit.client.0.smithi102.stdout:1/208: write de/f22 [780774,85551] 0 2024-03-26T18:54:32.704 INFO:tasks.workunit.client.1.smithi190.stdout:5/396: dwrite d2/d17/d1b/d22/d24/f61 [0,4194304] 0 2024-03-26T18:54:32.704 INFO:tasks.workunit.client.1.smithi190.stdout:5/397: truncate d2/d17/d1b/d22/d68/f75 295135 0 2024-03-26T18:54:32.704 INFO:tasks.workunit.client.1.smithi190.stdout:5/398: chown d2/d17/d1b/d22/d24/d3b 27137118 1 2024-03-26T18:54:32.704 INFO:tasks.workunit.client.1.smithi190.stdout:5/399: creat d2/d17/d1b/d22/d88/f8c x:0 0 0 2024-03-26T18:54:32.704 INFO:tasks.workunit.client.1.smithi190.stdout:3/452: dwrite d1/d3/df/f74 [0,4194304] 0 2024-03-26T18:54:32.704 INFO:tasks.workunit.client.1.smithi190.stdout:8/533: dwrite d9/d39/f6f [0,4194304] 0 2024-03-26T18:54:32.704 INFO:tasks.workunit.client.1.smithi190.stdout:8/534: write d9/d39/d67/f95 [903186,56345] 0 2024-03-26T18:54:32.704 INFO:tasks.workunit.client.1.smithi190.stdout:8/535: write d9/dd/d15/f30 [1379873,71366] 0 2024-03-26T18:54:32.705 INFO:tasks.workunit.client.0.smithi102.stdout:6/228: dread d2/d12/f2d [0,4194304] 0 2024-03-26T18:54:32.705 INFO:tasks.workunit.client.0.smithi102.stdout:2/205: dwrite d3/d6/f24 [0,4194304] 0 2024-03-26T18:54:32.705 INFO:tasks.workunit.client.0.smithi102.stdout:4/179: rmdir d1/d17 39 2024-03-26T18:54:32.707 INFO:tasks.workunit.client.0.smithi102.stdout:9/140: dwrite df/f17 [0,4194304] 0 2024-03-26T18:54:32.707 INFO:tasks.workunit.client.0.smithi102.stdout:9/141: write df/d1a/f20 [932669,100179] 0 2024-03-26T18:54:32.708 INFO:tasks.workunit.client.0.smithi102.stdout:9/142: chown df/c1b 434 1 2024-03-26T18:54:32.708 INFO:tasks.workunit.client.0.smithi102.stdout:9/143: chown c3 0 1 2024-03-26T18:54:32.708 INFO:tasks.workunit.client.0.smithi102.stdout:8/192: dread - d0/d3/d26/d19/d27/f29 zero size 2024-03-26T18:54:32.711 INFO:tasks.workunit.client.0.smithi102.stdout:4/180: dread d1/f26 [0,4194304] 0 2024-03-26T18:54:32.711 INFO:tasks.workunit.client.1.smithi190.stdout:2/520: mkdir d6/dbc 0 2024-03-26T18:54:32.711 INFO:tasks.workunit.client.1.smithi190.stdout:2/521: read - d6/d9/d3a/d3d/d55/fa9 zero size 2024-03-26T18:54:32.714 INFO:tasks.workunit.client.0.smithi102.stdout:5/427: symlink d2/d25/d2a/l98 0 2024-03-26T18:54:32.716 INFO:tasks.workunit.client.1.smithi190.stdout:0/610: link d0/dd/d20/fa4 d0/dd/d24/d4b/d5e/fdb 0 2024-03-26T18:54:32.719 INFO:tasks.workunit.client.1.smithi190.stdout:7/599: rename d3/d6/d9/d14/f2a to d3/d6/d9/d1f/d41/d50/d53/d8f/fda 0 2024-03-26T18:54:32.720 INFO:tasks.workunit.client.1.smithi190.stdout:4/557: fsync d7/d8/d14/f5b 0 2024-03-26T18:54:32.722 INFO:tasks.workunit.client.1.smithi190.stdout:1/457: truncate d3/d5f/f71 2535864 0 2024-03-26T18:54:32.722 INFO:tasks.workunit.client.1.smithi190.stdout:8/536: dread d9/dd/d15/d32/f4f [0,4194304] 0 2024-03-26T18:54:32.723 INFO:tasks.workunit.client.0.smithi102.stdout:2/206: creat d3/d20/f4c x:0 0 0 2024-03-26T18:54:32.725 INFO:tasks.workunit.client.1.smithi190.stdout:7/600: dread d3/f7c [0,4194304] 0 2024-03-26T18:54:32.725 INFO:tasks.workunit.client.1.smithi190.stdout:7/601: stat d3/d6/d9/d1f/d4c/d71 0 2024-03-26T18:54:32.726 INFO:tasks.workunit.client.0.smithi102.stdout:1/209: dwrite de/d2e/f4a [0,4194304] 0 2024-03-26T18:54:32.726 INFO:tasks.workunit.client.0.smithi102.stdout:6/229: dwrite d2/f21 [0,4194304] 0 2024-03-26T18:54:32.726 INFO:tasks.workunit.client.0.smithi102.stdout:6/230: chown d2/d12 133728984 1 2024-03-26T18:54:32.727 INFO:tasks.workunit.client.0.smithi102.stdout:6/231: stat d2/d7/f36 0 2024-03-26T18:54:32.732 INFO:tasks.workunit.client.1.smithi190.stdout:5/400: mkdir d2/d17/d1b/d22/d88/d8d 0 2024-03-26T18:54:32.735 INFO:tasks.workunit.client.0.smithi102.stdout:8/193: creat d0/d3/d26/f3b x:0 0 0 2024-03-26T18:54:32.736 INFO:tasks.workunit.client.0.smithi102.stdout:4/181: mkdir d1/d25/d39 0 2024-03-26T18:54:32.738 INFO:tasks.workunit.client.0.smithi102.stdout:4/182: dread d1/f26 [0,4194304] 0 2024-03-26T18:54:32.738 INFO:tasks.workunit.client.0.smithi102.stdout:5/428: rmdir d2/d14/d93 0 2024-03-26T18:54:32.738 INFO:tasks.workunit.client.0.smithi102.stdout:4/183: read d1/f31 [1249123,56352] 0 2024-03-26T18:54:32.739 INFO:tasks.workunit.client.0.smithi102.stdout:4/184: read d1/f31 [1558990,17253] 0 2024-03-26T18:54:32.739 INFO:tasks.workunit.client.1.smithi190.stdout:2/522: chown d6/d9/d3a/d3d/d2e/l90 161746 1 2024-03-26T18:54:32.739 INFO:tasks.workunit.client.1.smithi190.stdout:2/523: stat d6/d9/d1a/d72/d7c/dac 0 2024-03-26T18:54:32.740 INFO:tasks.workunit.client.0.smithi102.stdout:2/207: symlink d3/d6/d1a/d25/l4d 0 2024-03-26T18:54:32.741 INFO:tasks.workunit.client.1.smithi190.stdout:0/611: creat d0/dd/d24/d87/d93/db5/dcc/fdc x:0 0 0 2024-03-26T18:54:32.743 INFO:tasks.workunit.client.1.smithi190.stdout:1/458: rename d3/da/dd/d2f/d34/d3e to d3/da/dd/d95 0 2024-03-26T18:54:32.744 INFO:tasks.workunit.client.1.smithi190.stdout:5/401: dread d2/d17/d1b/f36 [0,4194304] 0 2024-03-26T18:54:32.745 INFO:tasks.workunit.client.0.smithi102.stdout:1/210: creat de/d16/d1d/d2a/f4c x:0 0 0 2024-03-26T18:54:32.745 INFO:tasks.workunit.client.0.smithi102.stdout:1/211: chown fb 8525 1 2024-03-26T18:54:32.745 INFO:tasks.workunit.client.0.smithi102.stdout:1/212: chown de/l39 1 1 2024-03-26T18:54:32.745 INFO:tasks.workunit.client.0.smithi102.stdout:1/213: write de/d2e/f49 [223825,10311] 0 2024-03-26T18:54:32.746 INFO:tasks.workunit.client.1.smithi190.stdout:6/460: sync 2024-03-26T18:54:32.746 INFO:tasks.workunit.client.1.smithi190.stdout:6/461: truncate d17/f87 983516 0 2024-03-26T18:54:32.746 INFO:tasks.workunit.client.1.smithi190.stdout:7/602: symlink d3/d6/d9/d1f/d84/ldb 0 2024-03-26T18:54:32.748 INFO:tasks.workunit.client.1.smithi190.stdout:5/402: dread d2/d17/d1b/f76 [0,4194304] 0 2024-03-26T18:54:32.748 INFO:tasks.workunit.client.1.smithi190.stdout:5/403: readlink d2/d17/l1c 0 2024-03-26T18:54:32.748 INFO:tasks.workunit.client.1.smithi190.stdout:5/404: read - d2/d17/d62/f82 zero size 2024-03-26T18:54:32.752 INFO:tasks.workunit.client.0.smithi102.stdout:6/232: creat d2/d7/d3c/f3e x:0 0 0 2024-03-26T18:54:32.754 INFO:tasks.workunit.client.0.smithi102.stdout:9/144: dwrite df/d1a/f20 [0,4194304] 0 2024-03-26T18:54:32.755 INFO:tasks.workunit.client.0.smithi102.stdout:8/194: mkdir d0/d3c 0 2024-03-26T18:54:32.758 INFO:tasks.workunit.client.0.smithi102.stdout:5/429: symlink d2/d25/d5b/d72/l99 0 2024-03-26T18:54:32.759 INFO:tasks.workunit.client.1.smithi190.stdout:0/612: read d0/dd/d24/da2/fc6 [3387077,39630] 0 2024-03-26T18:54:32.761 INFO:tasks.workunit.client.1.smithi190.stdout:1/459: link d3/da/d45/c86 d3/da/dd/c96 0 2024-03-26T18:54:32.762 INFO:tasks.workunit.client.0.smithi102.stdout:3/352: sync 2024-03-26T18:54:32.763 INFO:tasks.workunit.client.0.smithi102.stdout:4/185: chown d1/c5 4516030 1 2024-03-26T18:54:32.763 INFO:tasks.workunit.client.1.smithi190.stdout:6/462: unlink d17/d28/l33 0 2024-03-26T18:54:32.764 INFO:tasks.workunit.client.0.smithi102.stdout:2/208: rmdir d3/d6/d1a/d4b 0 2024-03-26T18:54:32.767 INFO:tasks.workunit.client.1.smithi190.stdout:5/405: mkdir d2/d17/d1b/d8e 0 2024-03-26T18:54:32.767 INFO:tasks.workunit.client.0.smithi102.stdout:1/214: creat de/d16/d1d/f4d x:0 0 0 2024-03-26T18:54:32.767 INFO:tasks.workunit.client.0.smithi102.stdout:1/215: fdatasync de/d16/f29 0 2024-03-26T18:54:32.767 INFO:tasks.workunit.client.0.smithi102.stdout:1/216: chown de/d1f 1260 1 2024-03-26T18:54:32.768 INFO:tasks.workunit.client.1.smithi190.stdout:0/613: creat d0/dd/d24/d4b/d5e/d81/fdd x:0 0 0 2024-03-26T18:54:32.776 INFO:tasks.workunit.client.1.smithi190.stdout:0/614: mknod d0/dd/d24/da2/d80/cde 0 2024-03-26T18:54:32.776 INFO:tasks.workunit.client.1.smithi190.stdout:0/615: stat d0/dd/l44 0 2024-03-26T18:54:32.776 INFO:tasks.workunit.client.1.smithi190.stdout:2/524: dwrite d6/f36 [0,4194304] 0 2024-03-26T18:54:32.776 INFO:tasks.workunit.client.1.smithi190.stdout:2/525: chown d6/d9/d10/f34 10 1 2024-03-26T18:54:32.776 INFO:tasks.workunit.client.1.smithi190.stdout:2/526: fdatasync d6/d9/d48/f7e 0 2024-03-26T18:54:32.776 INFO:tasks.workunit.client.0.smithi102.stdout:3/353: dread f7 [0,4194304] 0 2024-03-26T18:54:32.776 INFO:tasks.workunit.client.0.smithi102.stdout:3/354: fdatasync df/d29/d31/d36/d45/f65 0 2024-03-26T18:54:32.776 INFO:tasks.workunit.client.0.smithi102.stdout:3/355: chown df/d29/d31/d36/d3c/d6f/d75 93596 1 2024-03-26T18:54:32.776 INFO:tasks.workunit.client.0.smithi102.stdout:3/356: chown df/d29/d31/f32 303 1 2024-03-26T18:54:32.777 INFO:tasks.workunit.client.1.smithi190.stdout:5/406: symlink d2/l8f 0 2024-03-26T18:54:32.778 INFO:tasks.workunit.client.1.smithi190.stdout:1/460: rmdir d3/da/dd/d95 39 2024-03-26T18:54:32.782 INFO:tasks.workunit.client.0.smithi102.stdout:8/195: creat d0/d3/d2c/d33/f3d x:0 0 0 2024-03-26T18:54:32.786 INFO:tasks.workunit.client.0.smithi102.stdout:4/186: link d1/d17/l35 d1/d17/d2a/l3a 0 2024-03-26T18:54:32.789 INFO:tasks.workunit.client.0.smithi102.stdout:2/209: creat d3/d11/d15/d31/d39/f4e x:0 0 0 2024-03-26T18:54:32.790 INFO:tasks.workunit.client.1.smithi190.stdout:2/527: write d6/d9/d1a/d72/d7c/f7d [2974306,116212] 0 2024-03-26T18:54:32.790 INFO:tasks.workunit.client.1.smithi190.stdout:2/528: rename d6/d9/d1a to d6/d9/d1a/d72/d7c/dbd 22 2024-03-26T18:54:32.791 INFO:tasks.workunit.client.0.smithi102.stdout:1/217: unlink de/d1f/f36 0 2024-03-26T18:54:32.794 INFO:tasks.workunit.client.0.smithi102.stdout:4/187: dread d1/f31 [0,4194304] 0 2024-03-26T18:54:32.797 INFO:tasks.workunit.client.0.smithi102.stdout:3/357: creat df/d61/d55/f7c x:0 0 0 2024-03-26T18:54:32.798 INFO:tasks.workunit.client.0.smithi102.stdout:8/196: rename d0/d3/d26/f3b to d0/d3/d26/d19/d27/f3e 0 2024-03-26T18:54:32.798 INFO:tasks.workunit.client.0.smithi102.stdout:8/197: chown d0/d3/d26/d19/d27/f3e 0 1 2024-03-26T18:54:32.799 INFO:tasks.workunit.client.0.smithi102.stdout:3/358: dread df/f67 [0,4194304] 0 2024-03-26T18:54:32.799 INFO:tasks.workunit.client.0.smithi102.stdout:3/359: write df/d29/f64 [564002,46666] 0 2024-03-26T18:54:32.799 INFO:tasks.workunit.client.0.smithi102.stdout:3/360: chown fd 17512795 1 2024-03-26T18:54:32.799 INFO:tasks.workunit.client.0.smithi102.stdout:2/210: unlink d3/d11/d15/d31/l3d 0 2024-03-26T18:54:32.800 INFO:tasks.workunit.client.1.smithi190.stdout:7/603: dwrite d3/d6/d9/f75 [0,4194304] 0 2024-03-26T18:54:32.801 INFO:tasks.workunit.client.1.smithi190.stdout:2/529: mknod d6/d9/d3a/d3d/d2e/cbe 0 2024-03-26T18:54:32.801 INFO:tasks.workunit.client.1.smithi190.stdout:2/530: fdatasync d6/f7 0 2024-03-26T18:54:32.801 INFO:tasks.workunit.client.1.smithi190.stdout:2/531: readlink d6/d9/d3a/d3d/d2e/l98 0 2024-03-26T18:54:32.802 INFO:tasks.workunit.client.1.smithi190.stdout:2/532: dread - d6/d9/d10/f8c zero size 2024-03-26T18:54:32.802 INFO:tasks.workunit.client.0.smithi102.stdout:4/188: rename d1/d17/c18 to d1/d17/d1f/c3b 0 2024-03-26T18:54:32.802 INFO:tasks.workunit.client.0.smithi102.stdout:4/189: creat d1/d27/f3c x:0 0 0 2024-03-26T18:54:32.803 INFO:tasks.workunit.client.0.smithi102.stdout:8/198: truncate d0/d3/f21 1379713 0 2024-03-26T18:54:32.804 INFO:tasks.workunit.client.1.smithi190.stdout:2/533: getdents d6/d9/d3a/d3d/d1c/d8d 0 2024-03-26T18:54:32.806 INFO:tasks.workunit.client.0.smithi102.stdout:4/190: rename d1/d17/d1f/c34 to d1/d17/d1a/d21/c3d 0 2024-03-26T18:54:32.807 INFO:tasks.workunit.client.0.smithi102.stdout:8/199: link d0/d3/l7 d0/d3/d2c/d33/l3f 0 2024-03-26T18:54:32.807 INFO:tasks.workunit.client.0.smithi102.stdout:8/200: write d0/d3/f6 [2146921,120233] 0 2024-03-26T18:54:32.808 INFO:tasks.workunit.client.1.smithi190.stdout:2/534: dread d6/d9/d3a/d3d/d55/fa5 [0,4194304] 0 2024-03-26T18:54:32.810 INFO:tasks.workunit.client.1.smithi190.stdout:2/535: creat d6/d9/d3a/d3d/fbf x:0 0 0 2024-03-26T18:54:32.811 INFO:tasks.workunit.client.0.smithi102.stdout:5/430: dwrite d2/d2c/d38/f5c [0,4194304] 0 2024-03-26T18:54:32.811 INFO:tasks.workunit.client.0.smithi102.stdout:5/431: stat d2/d2c/d38/d43/c56 0 2024-03-26T18:54:32.812 INFO:tasks.workunit.client.0.smithi102.stdout:5/432: creat d2/d25/d5b/d76/f9a x:0 0 0 2024-03-26T18:54:32.813 INFO:tasks.workunit.client.0.smithi102.stdout:3/361: dread df/f17 [0,4194304] 0 2024-03-26T18:54:32.813 INFO:tasks.workunit.client.0.smithi102.stdout:5/433: stat d2/d14/d1a/f91 0 2024-03-26T18:54:32.814 INFO:tasks.workunit.client.0.smithi102.stdout:5/434: mknod d2/d25/d75/c9b 0 2024-03-26T18:54:32.814 INFO:tasks.workunit.client.0.smithi102.stdout:5/435: creat d2/d25/d2a/d4f/f9c x:0 0 0 2024-03-26T18:54:32.814 INFO:tasks.workunit.client.0.smithi102.stdout:5/436: rename d2/d2c/d38 to d2/d2c/d38/d9d 22 2024-03-26T18:54:32.814 INFO:tasks.workunit.client.1.smithi190.stdout:3/453: dwrite d1/d3/df/f74 [4194304,4194304] 0 2024-03-26T18:54:32.814 INFO:tasks.workunit.client.1.smithi190.stdout:5/407: dwrite d2/d17/d1b/d22/d24/d3b/f46 [0,4194304] 0 2024-03-26T18:54:32.815 INFO:tasks.workunit.client.1.smithi190.stdout:1/461: dwrite d3/d9/fe [0,4194304] 0 2024-03-26T18:54:32.817 INFO:tasks.workunit.client.1.smithi190.stdout:6/463: dwrite d17/d28/d2c/d50/f67 [0,4194304] 0 2024-03-26T18:54:32.817 INFO:tasks.workunit.client.1.smithi190.stdout:7/604: dread d3/d6/d9/d1f/d84/d99/fa3 [0,4194304] 0 2024-03-26T18:54:32.817 INFO:tasks.workunit.client.1.smithi190.stdout:7/605: fdatasync d3/d6/fac 0 2024-03-26T18:54:32.819 INFO:tasks.workunit.client.1.smithi190.stdout:3/454: link d1/d3/df/d18/d1b/l37 d1/d3/d52/d70/l87 0 2024-03-26T18:54:32.821 INFO:tasks.workunit.client.1.smithi190.stdout:6/464: dread d17/d28/d46/f56 [0,4194304] 0 2024-03-26T18:54:32.822 INFO:tasks.workunit.client.1.smithi190.stdout:6/465: chown d17/f19 1749889 1 2024-03-26T18:54:32.822 INFO:tasks.workunit.client.1.smithi190.stdout:6/466: truncate d17/d28/d2c/d50/f9c 951086 0 2024-03-26T18:54:32.826 INFO:tasks.workunit.client.0.smithi102.stdout:3/362: read df/d29/d31/d36/f48 [3811372,109615] 0 2024-03-26T18:54:32.829 INFO:tasks.workunit.client.0.smithi102.stdout:5/437: rename d2/d25/d2a/d5d to d2/d2c/d38/d43/d9e 0 2024-03-26T18:54:32.830 INFO:tasks.workunit.client.1.smithi190.stdout:0/616: getdents d0/dd/d24/d4b/d5e/d81 0 2024-03-26T18:54:32.830 INFO:tasks.workunit.client.1.smithi190.stdout:0/617: chown d0/dd/d24/d4b/d5e/d6b/faf 74 1 2024-03-26T18:54:32.831 INFO:tasks.workunit.client.1.smithi190.stdout:0/618: dread - d0/dd/d24/d4b/d5e/f82 zero size 2024-03-26T18:54:32.831 INFO:tasks.workunit.client.1.smithi190.stdout:0/619: creat d0/dd/d20/fdf x:0 0 0 2024-03-26T18:54:32.831 INFO:tasks.workunit.client.1.smithi190.stdout:0/620: fdatasync d0/dd/d20/d1c/d4e/f90 0 2024-03-26T18:54:32.831 INFO:tasks.workunit.client.0.smithi102.stdout:3/363: creat df/d29/d31/d36/d3c/d6f/f7d x:0 0 0 2024-03-26T18:54:32.832 INFO:tasks.workunit.client.0.smithi102.stdout:3/364: chown df/d1c/d34/d53 414518 1 2024-03-26T18:54:32.835 INFO:tasks.workunit.client.0.smithi102.stdout:5/438: creat d2/d25/d2a/f9f x:0 0 0 2024-03-26T18:54:32.836 INFO:tasks.workunit.client.0.smithi102.stdout:5/439: stat d2/d25/f7b 0 2024-03-26T18:54:32.836 INFO:tasks.workunit.client.0.smithi102.stdout:7/163: sync 2024-03-26T18:54:32.836 INFO:tasks.workunit.client.0.smithi102.stdout:0/237: sync 2024-03-26T18:54:32.836 INFO:tasks.workunit.client.0.smithi102.stdout:7/164: readlink d7/d10/l18 0 2024-03-26T18:54:32.839 INFO:tasks.workunit.client.0.smithi102.stdout:5/440: truncate d2/d2c/d38/d43/d9e/d82/f8f 3539553 0 2024-03-26T18:54:32.839 INFO:tasks.workunit.client.0.smithi102.stdout:5/441: read - d2/d2c/d38/f89 zero size 2024-03-26T18:54:32.839 INFO:tasks.workunit.client.0.smithi102.stdout:1/218: getdents de/d1f 0 2024-03-26T18:54:32.841 INFO:tasks.workunit.client.0.smithi102.stdout:1/219: read de/f32 [125081,126042] 0 2024-03-26T18:54:32.842 INFO:tasks.workunit.client.0.smithi102.stdout:7/165: creat d7/d12/d2b/f2f x:0 0 0 2024-03-26T18:54:32.844 INFO:tasks.workunit.client.1.smithi190.stdout:7/606: rename d3/d6/d9/d1f/d4c/d71/cce to d3/d6/d9/d1f/d41/d65/d90/cdc 0 2024-03-26T18:54:32.845 INFO:tasks.workunit.client.0.smithi102.stdout:3/365: getdents df/d29/d31 0 2024-03-26T18:54:32.846 INFO:tasks.workunit.client.0.smithi102.stdout:0/238: read d6/f1a [3336388,118116] 0 2024-03-26T18:54:32.846 INFO:tasks.workunit.client.0.smithi102.stdout:0/239: creat d6/de/f4c x:0 0 0 2024-03-26T18:54:32.847 INFO:tasks.workunit.client.1.smithi190.stdout:2/536: dwrite d6/d9/d10/d5f/fa2 [8388608,4194304] 0 2024-03-26T18:54:32.848 INFO:tasks.workunit.client.0.smithi102.stdout:8/201: dwrite d0/d3/d26/d19/d27/f3e [0,4194304] 0 2024-03-26T18:54:32.848 INFO:tasks.workunit.client.0.smithi102.stdout:8/202: truncate d0/d3/d2c/d33/f3d 939505 0 2024-03-26T18:54:32.848 INFO:tasks.workunit.client.0.smithi102.stdout:8/203: creat d0/d3/d31/f40 x:0 0 0 2024-03-26T18:54:32.848 INFO:tasks.workunit.client.0.smithi102.stdout:8/204: chown d0/d15/d1b 53892 1 2024-03-26T18:54:32.849 INFO:tasks.workunit.client.1.smithi190.stdout:3/455: mknod d1/c88 0 2024-03-26T18:54:32.849 INFO:tasks.workunit.client.0.smithi102.stdout:1/220: symlink de/d2e/l4e 0 2024-03-26T18:54:32.851 INFO:tasks.workunit.client.0.smithi102.stdout:5/442: read d2/f12 [41138,96056] 0 2024-03-26T18:54:32.852 INFO:tasks.workunit.client.0.smithi102.stdout:7/166: creat d7/d2a/f30 x:0 0 0 2024-03-26T18:54:32.852 INFO:tasks.workunit.client.0.smithi102.stdout:7/167: write d7/f9 [965404,103067] 0 2024-03-26T18:54:32.854 INFO:tasks.workunit.client.0.smithi102.stdout:3/366: symlink df/d1c/d34/l7e 0 2024-03-26T18:54:32.854 INFO:tasks.workunit.client.0.smithi102.stdout:3/367: chown df/d1c 2 1 2024-03-26T18:54:32.859 INFO:tasks.workunit.client.0.smithi102.stdout:0/240: symlink d6/de/d13/d4a/l4d 0 2024-03-26T18:54:32.859 INFO:tasks.workunit.client.0.smithi102.stdout:0/241: truncate d6/de/f4b 262470 0 2024-03-26T18:54:32.863 INFO:tasks.workunit.client.0.smithi102.stdout:3/368: write df/f25 [3432635,5528] 0 2024-03-26T18:54:32.865 INFO:tasks.workunit.client.0.smithi102.stdout:8/205: mknod d0/d15/d1b/c41 0 2024-03-26T18:54:32.865 INFO:tasks.workunit.client.0.smithi102.stdout:8/206: creat d0/d3/d2c/f42 x:0 0 0 2024-03-26T18:54:32.865 INFO:tasks.workunit.client.0.smithi102.stdout:8/207: write d0/d3/f32 [506705,17934] 0 2024-03-26T18:54:32.867 INFO:tasks.workunit.client.0.smithi102.stdout:1/221: truncate f9 3099417 0 2024-03-26T18:54:32.867 INFO:tasks.workunit.client.0.smithi102.stdout:1/222: readlink de/d16/l31 0 2024-03-26T18:54:32.867 INFO:tasks.workunit.client.0.smithi102.stdout:1/223: fdatasync de/d16/d1d/d2a/d2c/d43/f46 0 2024-03-26T18:54:32.867 INFO:tasks.workunit.client.1.smithi190.stdout:7/607: dwrite d3/d1b/d57/f5f [0,4194304] 0 2024-03-26T18:54:32.871 INFO:tasks.workunit.client.0.smithi102.stdout:7/168: truncate d7/d12/f22 3307671 0 2024-03-26T18:54:32.873 INFO:tasks.workunit.client.0.smithi102.stdout:0/242: creat d6/de/d13/d25/d31/f4e x:0 0 0 2024-03-26T18:54:32.875 INFO:tasks.workunit.client.0.smithi102.stdout:5/443: dread d2/d14/f2f [0,4194304] 0 2024-03-26T18:54:32.877 INFO:tasks.workunit.client.0.smithi102.stdout:3/369: truncate df/f1a 2027495 0 2024-03-26T18:54:32.877 INFO:tasks.workunit.client.0.smithi102.stdout:3/370: write df/d29/d3a/f6d [967751,66124] 0 2024-03-26T18:54:32.878 INFO:tasks.workunit.client.1.smithi190.stdout:5/408: rmdir d2/d17 39 2024-03-26T18:54:32.878 INFO:tasks.workunit.client.1.smithi190.stdout:4/558: sync 2024-03-26T18:54:32.878 INFO:tasks.workunit.client.1.smithi190.stdout:9/782: sync 2024-03-26T18:54:32.878 INFO:tasks.workunit.client.1.smithi190.stdout:9/783: fdatasync d12/d2d/d1d/d23/de5/ff4 0 2024-03-26T18:54:32.882 INFO:tasks.workunit.client.0.smithi102.stdout:8/208: creat d0/d3c/f43 x:0 0 0 2024-03-26T18:54:32.882 INFO:tasks.workunit.client.0.smithi102.stdout:8/209: dread - d0/d3/d2c/f42 zero size 2024-03-26T18:54:32.882 INFO:tasks.workunit.client.0.smithi102.stdout:8/210: creat d0/d3/d2c/d33/f44 x:0 0 0 2024-03-26T18:54:32.882 INFO:tasks.workunit.client.0.smithi102.stdout:8/211: creat d0/d3/d26/d19/d27/f45 x:0 0 0 2024-03-26T18:54:32.882 INFO:tasks.workunit.client.0.smithi102.stdout:8/212: dread - d0/d3/d31/f40 zero size 2024-03-26T18:54:32.882 INFO:tasks.workunit.client.0.smithi102.stdout:8/213: chown d0/d3/d2c/f34 495 1 2024-03-26T18:54:32.883 INFO:tasks.workunit.client.1.smithi190.stdout:0/621: symlink d0/dd5/le0 0 2024-03-26T18:54:32.884 INFO:tasks.workunit.client.0.smithi102.stdout:1/224: link de/f3f de/d16/d1d/d2a/d2c/d43/f4f 0 2024-03-26T18:54:32.885 INFO:tasks.workunit.client.0.smithi102.stdout:5/444: symlink d2/d2c/la0 0 2024-03-26T18:54:32.887 INFO:tasks.workunit.client.1.smithi190.stdout:2/537: mkdir d6/d9/d45/d6b/d93/dc0 0 2024-03-26T18:54:32.889 INFO:tasks.workunit.client.1.smithi190.stdout:3/456: mkdir d1/d3/df/d18/d35/d89 0 2024-03-26T18:54:32.891 INFO:tasks.workunit.client.1.smithi190.stdout:6/467: getdents d17/d28/d46 0 2024-03-26T18:54:32.892 INFO:tasks.workunit.client.1.smithi190.stdout:9/784: write ff [374305,6281] 0 2024-03-26T18:54:32.892 INFO:tasks.workunit.client.1.smithi190.stdout:7/608: dwrite d3/d6/d9/d1f/d41/d65/d90/d92/d9b/fb0 [0,4194304] 0 2024-03-26T18:54:32.892 INFO:tasks.workunit.client.1.smithi190.stdout:7/609: readlink d3/d6/d9/d1f/d41/d50/d53/d87/lb8 0 2024-03-26T18:54:32.892 INFO:tasks.workunit.client.1.smithi190.stdout:7/610: write d3/d6/d9/d1f/d41/d50/d53/f8a [614192,103049] 0 2024-03-26T18:54:32.893 INFO:tasks.workunit.client.1.smithi190.stdout:0/622: write d0/dd/d24/d4b/f89 [4038416,87058] 0 2024-03-26T18:54:32.894 INFO:tasks.workunit.client.0.smithi102.stdout:3/371: creat df/d29/d31/d36/f7f x:0 0 0 2024-03-26T18:54:32.894 INFO:tasks.workunit.client.0.smithi102.stdout:3/372: write f7 [3358887,106416] 0 2024-03-26T18:54:32.894 INFO:tasks.workunit.client.0.smithi102.stdout:3/373: write df/d29/d31/d36/d3c/d6f/f7d [28710,58296] 0 2024-03-26T18:54:32.894 INFO:tasks.workunit.client.0.smithi102.stdout:3/374: chown df/d29/d3a/l7a 603 1 2024-03-26T18:54:32.894 INFO:tasks.workunit.client.0.smithi102.stdout:3/375: creat df/d1c/d34/d53/f80 x:0 0 0 2024-03-26T18:54:32.894 INFO:tasks.workunit.client.0.smithi102.stdout:3/376: creat df/d1c/f81 x:0 0 0 2024-03-26T18:54:32.894 INFO:tasks.workunit.client.0.smithi102.stdout:3/377: truncate df/d1c/f69 576415 0 2024-03-26T18:54:32.894 INFO:tasks.workunit.client.0.smithi102.stdout:3/378: chown df/d29/d31/d36/d3c/d6f/f7d 127357268 1 2024-03-26T18:54:32.896 INFO:tasks.workunit.client.0.smithi102.stdout:5/445: read d2/f62 [975772,112349] 0 2024-03-26T18:54:32.898 INFO:tasks.workunit.client.0.smithi102.stdout:8/214: mkdir d0/d3/d46 0 2024-03-26T18:54:32.900 INFO:tasks.workunit.client.0.smithi102.stdout:1/225: mknod de/d16/c50 0 2024-03-26T18:54:32.902 INFO:tasks.workunit.client.0.smithi102.stdout:7/169: dread d7/d12/f22 [0,4194304] 0 2024-03-26T18:54:32.902 INFO:tasks.workunit.client.1.smithi190.stdout:5/409: rename d2/d17/l19 to d2/d17/d1b/d1e/d3a/d3c/l90 0 2024-03-26T18:54:32.905 INFO:tasks.workunit.client.1.smithi190.stdout:2/538: mknod d6/d9/d10/db7/daf/cc1 0 2024-03-26T18:54:32.905 INFO:tasks.workunit.client.1.smithi190.stdout:2/539: fsync d6/d9/d3a/d3d/d1c/f54 0 2024-03-26T18:54:32.905 INFO:tasks.workunit.client.1.smithi190.stdout:2/540: dread - d6/d9/d3a/d3d/d2e/f6e zero size 2024-03-26T18:54:32.905 INFO:tasks.workunit.client.1.smithi190.stdout:2/541: chown d6/d9/f4e 1404866 1 2024-03-26T18:54:32.907 INFO:tasks.workunit.client.1.smithi190.stdout:3/457: mkdir d1/de/d8a 0 2024-03-26T18:54:32.909 INFO:tasks.workunit.client.0.smithi102.stdout:3/379: rename df/d29/d31/c5e to df/d29/d3a/c82 0 2024-03-26T18:54:32.910 INFO:tasks.workunit.client.1.smithi190.stdout:9/785: creat d12/d2d/d1d/d23/d83/dc9/dd4/d3f/f113 x:0 0 0 2024-03-26T18:54:32.910 INFO:tasks.workunit.client.1.smithi190.stdout:9/786: fdatasync d12/d2d/d1d/d23/d83/d91/fdd 0 2024-03-26T18:54:32.911 INFO:tasks.workunit.client.0.smithi102.stdout:5/446: truncate d2/d25/f26 1962218 0 2024-03-26T18:54:32.914 INFO:tasks.workunit.client.0.smithi102.stdout:1/226: creat de/d16/f51 x:0 0 0 2024-03-26T18:54:32.915 INFO:tasks.workunit.client.1.smithi190.stdout:8/537: sync 2024-03-26T18:54:32.915 INFO:tasks.workunit.client.1.smithi190.stdout:8/538: creat d9/d39/d67/fba x:0 0 0 2024-03-26T18:54:32.915 INFO:tasks.workunit.client.1.smithi190.stdout:8/539: chown d9/dd/d23/d36/d37/d3a/d43/f80 54256 1 2024-03-26T18:54:32.915 INFO:tasks.workunit.client.1.smithi190.stdout:8/540: truncate d9/dd/d23/d36/f71 428702 0 2024-03-26T18:54:32.916 INFO:tasks.workunit.client.1.smithi190.stdout:7/611: symlink d3/d6/d9/d1f/d84/ldd 0 2024-03-26T18:54:32.916 INFO:tasks.workunit.client.1.smithi190.stdout:7/612: chown d3/d6/d9/d1f/f22 22 1 2024-03-26T18:54:32.917 INFO:tasks.workunit.client.1.smithi190.stdout:0/623: mkdir d0/dd/d24/d87/d93/de1 0 2024-03-26T18:54:32.920 INFO:tasks.workunit.client.1.smithi190.stdout:5/410: creat d2/d17/d62/d64/f91 x:0 0 0 2024-03-26T18:54:32.920 INFO:tasks.workunit.client.1.smithi190.stdout:5/411: truncate d2/d17/d1b/f67 198734 0 2024-03-26T18:54:32.920 INFO:tasks.workunit.client.1.smithi190.stdout:5/412: dread - d2/d17/d1b/d22/d88/f8c zero size 2024-03-26T18:54:32.920 INFO:tasks.workunit.client.1.smithi190.stdout:5/413: chown d2/d17/d1b/d22/d24/l65 6596 1 2024-03-26T18:54:32.923 INFO:tasks.workunit.client.1.smithi190.stdout:2/542: rename d6/d9/d3a/d3d/d55/fa9 to d6/d21/d61/fc2 0 2024-03-26T18:54:32.926 INFO:tasks.workunit.client.0.smithi102.stdout:5/447: creat d2/d14/fa1 x:0 0 0 2024-03-26T18:54:32.927 INFO:tasks.workunit.client.1.smithi190.stdout:9/787: dwrite d12/f4c [0,4194304] 0 2024-03-26T18:54:32.929 INFO:tasks.workunit.client.0.smithi102.stdout:8/215: mknod d0/d3/d26/c47 0 2024-03-26T18:54:32.933 INFO:tasks.workunit.client.0.smithi102.stdout:1/227: link de/d16/d1d/f28 de/f52 0 2024-03-26T18:54:32.934 INFO:tasks.workunit.client.0.smithi102.stdout:7/170: mkdir d7/d31 0 2024-03-26T18:54:32.936 INFO:tasks.workunit.client.1.smithi190.stdout:8/541: dwrite f6 [4194304,4194304] 0 2024-03-26T18:54:32.936 INFO:tasks.workunit.client.1.smithi190.stdout:8/542: fdatasync d9/dd/d15/f7a 0 2024-03-26T18:54:32.936 INFO:tasks.workunit.client.1.smithi190.stdout:7/613: mkdir d3/d6/d9/d1f/d41/d50/dde 0 2024-03-26T18:54:32.936 INFO:tasks.workunit.client.1.smithi190.stdout:7/614: creat d3/fdf x:0 0 0 2024-03-26T18:54:32.938 INFO:tasks.workunit.client.0.smithi102.stdout:8/216: symlink d0/d3/d2c/d33/l48 0 2024-03-26T18:54:32.938 INFO:tasks.workunit.client.1.smithi190.stdout:0/624: rmdir d0/dd/d24/d87/d93/db5 39 2024-03-26T18:54:32.940 INFO:tasks.workunit.client.0.smithi102.stdout:1/228: creat de/d1f/f53 x:0 0 0 2024-03-26T18:54:32.940 INFO:tasks.workunit.client.0.smithi102.stdout:1/229: chown de/d16/l1b 14738683 1 2024-03-26T18:54:32.940 INFO:tasks.workunit.client.0.smithi102.stdout:1/230: stat de/f52 0 2024-03-26T18:54:32.940 INFO:tasks.workunit.client.0.smithi102.stdout:1/231: dread - f2 zero size 2024-03-26T18:54:32.941 INFO:tasks.workunit.client.0.smithi102.stdout:1/232: truncate de/d16/d1d/d2a/f4c 723726 0 2024-03-26T18:54:32.941 INFO:tasks.workunit.client.1.smithi190.stdout:9/788: dwrite d12/d2d/d1d/d23/d2b/da0/fc8 [0,4194304] 0 2024-03-26T18:54:32.941 INFO:tasks.workunit.client.1.smithi190.stdout:5/414: symlink d2/d17/d1b/d8e/l92 0 2024-03-26T18:54:32.942 INFO:tasks.workunit.client.1.smithi190.stdout:5/415: chown d2/d17/c2c 1709271348 1 2024-03-26T18:54:32.942 INFO:tasks.workunit.client.0.smithi102.stdout:8/217: chown d0/d3/d26/l2e 4 1 2024-03-26T18:54:32.942 INFO:tasks.workunit.client.0.smithi102.stdout:8/218: creat d0/d3/d2c/d33/f49 x:0 0 0 2024-03-26T18:54:32.944 INFO:tasks.workunit.client.1.smithi190.stdout:2/543: mknod d6/d9/d3a/d3d/d1c/cc3 0 2024-03-26T18:54:32.945 INFO:tasks.workunit.client.0.smithi102.stdout:1/233: rmdir de/d16/d1d/d2a/d2c/d4b 0 2024-03-26T18:54:32.945 INFO:tasks.workunit.client.0.smithi102.stdout:1/234: creat de/d16/f54 x:0 0 0 2024-03-26T18:54:32.953 INFO:tasks.workunit.client.0.smithi102.stdout:8/219: write d0/d3/f11 [2481649,97321] 0 2024-03-26T18:54:32.953 INFO:tasks.workunit.client.0.smithi102.stdout:8/220: truncate d0/d3/d31/f3a 855735 0 2024-03-26T18:54:32.953 INFO:tasks.workunit.client.0.smithi102.stdout:8/221: fsync d0/d3/d26/d19/d27/f45 0 2024-03-26T18:54:32.953 INFO:tasks.workunit.client.0.smithi102.stdout:8/222: dread - d0/d9/f24 zero size 2024-03-26T18:54:32.954 INFO:tasks.workunit.client.0.smithi102.stdout:7/171: dwrite d7/ff [0,4194304] 0 2024-03-26T18:54:32.954 INFO:tasks.workunit.client.0.smithi102.stdout:7/172: readlink d7/ld 0 2024-03-26T18:54:32.954 INFO:tasks.workunit.client.0.smithi102.stdout:1/235: unlink de/d16/f3b 0 2024-03-26T18:54:32.954 INFO:tasks.workunit.client.0.smithi102.stdout:1/236: creat de/d2e/f55 x:0 0 0 2024-03-26T18:54:32.955 INFO:tasks.workunit.client.1.smithi190.stdout:8/543: symlink d9/da2/lbb 0 2024-03-26T18:54:32.958 INFO:tasks.workunit.client.0.smithi102.stdout:8/223: dread d0/d15/f16 [0,4194304] 0 2024-03-26T18:54:32.960 INFO:tasks.workunit.client.1.smithi190.stdout:9/789: write d12/d2d/fd9 [4184779,46207] 0 2024-03-26T18:54:32.960 INFO:tasks.workunit.client.1.smithi190.stdout:5/416: dread d2/d17/d1b/d22/f7f [0,4194304] 0 2024-03-26T18:54:32.960 INFO:tasks.workunit.client.0.smithi102.stdout:8/224: read d0/d3/d31/f3a [813321,77430] 0 2024-03-26T18:54:32.960 INFO:tasks.workunit.client.1.smithi190.stdout:7/615: link d3/d6/d9/d14/fc2 d3/d6/d9/d1f/d41/d50/dde/fe0 0 2024-03-26T18:54:32.962 INFO:tasks.workunit.client.0.smithi102.stdout:7/173: symlink d7/d12/l32 0 2024-03-26T18:54:32.962 INFO:tasks.workunit.client.0.smithi102.stdout:7/174: readlink d7/ld 0 2024-03-26T18:54:32.963 INFO:tasks.workunit.client.1.smithi190.stdout:0/625: symlink d0/dd/d20/d1c/d4e/le2 0 2024-03-26T18:54:32.963 INFO:tasks.workunit.client.1.smithi190.stdout:0/626: creat d0/dd/d24/d87/d93/db5/dcc/fe3 x:0 0 0 2024-03-26T18:54:32.966 INFO:tasks.workunit.client.1.smithi190.stdout:5/417: dread d2/d17/d1b/d1e/d3a/f4d [0,4194304] 0 2024-03-26T18:54:32.969 INFO:tasks.workunit.client.0.smithi102.stdout:8/225: rmdir d0/d3/d46 0 2024-03-26T18:54:32.969 INFO:tasks.workunit.client.0.smithi102.stdout:8/226: creat d0/d3/d31/f4a x:0 0 0 2024-03-26T18:54:32.970 INFO:tasks.workunit.client.0.smithi102.stdout:1/237: getdents de/d16/d1d/d2a/d2c 0 2024-03-26T18:54:32.970 INFO:tasks.workunit.client.0.smithi102.stdout:1/238: write fc [3628069,83489] 0 2024-03-26T18:54:32.970 INFO:tasks.workunit.client.0.smithi102.stdout:1/239: truncate de/d1f/f3a 1197066 0 2024-03-26T18:54:32.978 INFO:tasks.workunit.client.1.smithi190.stdout:2/544: creat d6/d21/d61/d6c/fc4 x:0 0 0 2024-03-26T18:54:32.978 INFO:tasks.workunit.client.1.smithi190.stdout:8/544: mknod d9/dd/d15/d32/cbc 0 2024-03-26T18:54:32.980 INFO:tasks.workunit.client.1.smithi190.stdout:9/790: symlink d12/d2d/d1d/d23/d83/dd5/l114 0 2024-03-26T18:54:32.982 INFO:tasks.workunit.client.1.smithi190.stdout:7/616: link d3/d6/d9/d1f/d41/l47 d3/d6/d9/d1f/d41/d65/d90/d92/le1 0 2024-03-26T18:54:32.982 INFO:tasks.workunit.client.1.smithi190.stdout:7/617: getdents d3/d1b/dd3 0 2024-03-26T18:54:32.982 INFO:tasks.workunit.client.1.smithi190.stdout:7/618: readlink d3/d6/d9/d1f/d84/ldd 0 2024-03-26T18:54:32.982 INFO:tasks.workunit.client.1.smithi190.stdout:7/619: write d3/f2b [2833444,118123] 0 2024-03-26T18:54:32.982 INFO:tasks.workunit.client.1.smithi190.stdout:7/620: fdatasync d3/d6/d9/d1f/d4c/f5c 0 2024-03-26T18:54:32.983 INFO:tasks.workunit.client.0.smithi102.stdout:7/175: dwrite d7/f29 [0,4194304] 0 2024-03-26T18:54:32.984 INFO:tasks.workunit.client.1.smithi190.stdout:0/627: mkdir d0/dd/d20/d1c/d46/d58/de4 0 2024-03-26T18:54:32.986 INFO:tasks.workunit.client.1.smithi190.stdout:5/418: creat d2/f93 x:0 0 0 2024-03-26T18:54:32.989 INFO:tasks.workunit.client.1.smithi190.stdout:8/545: truncate d9/f1a 3026057 0 2024-03-26T18:54:32.992 INFO:tasks.workunit.client.1.smithi190.stdout:9/791: mknod d12/d80/c115 0 2024-03-26T18:54:32.994 INFO:tasks.workunit.client.1.smithi190.stdout:5/419: rename d2/d17/d1b/d1e/d3a/f40 to d2/d17/d1b/d1e/d3a/d3c/f94 0 2024-03-26T18:54:32.994 INFO:tasks.workunit.client.1.smithi190.stdout:5/420: truncate d2/d17/d1b/d22/f77 339397 0 2024-03-26T18:54:32.995 INFO:tasks.workunit.client.1.smithi190.stdout:2/545: getdents d6/d9/d3a/d3d/d1c 0 2024-03-26T18:54:32.995 INFO:tasks.workunit.client.1.smithi190.stdout:0/628: write d0/dd/d24/f2f [202388,76012] 0 2024-03-26T18:54:32.996 INFO:tasks.workunit.client.1.smithi190.stdout:8/546: getdents d9/dd/d23 0 2024-03-26T18:54:32.996 INFO:tasks.workunit.client.1.smithi190.stdout:8/547: creat d9/dd/d23/d36/d37/d3a/d61/fbd x:0 0 0 2024-03-26T18:54:32.996 INFO:tasks.workunit.client.1.smithi190.stdout:8/548: readlink l5 0 2024-03-26T18:54:32.996 INFO:tasks.workunit.client.1.smithi190.stdout:8/549: readlink d9/dd/d23/d36/d37/d3a/l44 0 2024-03-26T18:54:32.996 INFO:tasks.workunit.client.1.smithi190.stdout:8/550: chown d9/dd/c75 52330081 1 2024-03-26T18:54:32.997 INFO:tasks.workunit.client.1.smithi190.stdout:9/792: rename d12/d2d/d1d/d23/d2b/f68 to d12/d2d/d1d/d23/d2b/da0/f116 0 2024-03-26T18:54:32.998 INFO:tasks.workunit.client.1.smithi190.stdout:5/421: creat d2/d17/d1b/d22/d88/d8d/f95 x:0 0 0 2024-03-26T18:54:32.998 INFO:tasks.workunit.client.1.smithi190.stdout:5/422: write d2/d17/d1b/d22/f2d [789133,107846] 0 2024-03-26T18:54:32.999 INFO:tasks.workunit.client.1.smithi190.stdout:1/462: sync 2024-03-26T18:54:32.999 INFO:tasks.workunit.client.1.smithi190.stdout:2/546: mknod d6/d9/d1a/d72/d7c/dac/cc5 0 2024-03-26T18:54:32.999 INFO:tasks.workunit.client.1.smithi190.stdout:2/547: chown d6/d9/d3a/d3d/d2e/l98 512183530 1 2024-03-26T18:54:33.000 INFO:tasks.workunit.client.1.smithi190.stdout:8/551: creat d9/d48/d49/fbe x:0 0 0 2024-03-26T18:54:33.001 INFO:tasks.workunit.client.1.smithi190.stdout:8/552: fdatasync d9/d39/f6f 0 2024-03-26T18:54:33.001 INFO:tasks.workunit.client.1.smithi190.stdout:8/553: readlink d9/d48/l6c 0 2024-03-26T18:54:33.001 INFO:tasks.workunit.client.1.smithi190.stdout:8/554: write d9/d48/f62 [1899338,5743] 0 2024-03-26T18:54:33.001 INFO:tasks.workunit.client.1.smithi190.stdout:0/629: rename d0/dd/d24/d4b/d5e/d81/fdd to d0/dd/fe5 0 2024-03-26T18:54:33.002 INFO:tasks.workunit.client.1.smithi190.stdout:9/793: creat d12/d2d/d1d/d54/d92/de3/f117 x:0 0 0 2024-03-26T18:54:33.003 INFO:tasks.workunit.client.1.smithi190.stdout:8/555: dread d9/dd/d15/f60 [0,4194304] 0 2024-03-26T18:54:33.003 INFO:tasks.workunit.client.1.smithi190.stdout:5/423: mkdir d2/d17/d1b/d22/d96 0 2024-03-26T18:54:33.004 INFO:tasks.workunit.client.1.smithi190.stdout:1/463: mknod d3/da/dd/c97 0 2024-03-26T18:54:33.004 INFO:tasks.workunit.client.1.smithi190.stdout:2/548: rmdir d6/d9/d45/d6b/d93 39 2024-03-26T18:54:33.007 INFO:tasks.workunit.client.1.smithi190.stdout:0/630: mkdir d0/dd/d20/d1c/d8b/d8d/db7/de6 0 2024-03-26T18:54:33.007 INFO:tasks.workunit.client.1.smithi190.stdout:0/631: dread - d0/dd/d20/fbe zero size 2024-03-26T18:54:33.008 INFO:tasks.workunit.client.1.smithi190.stdout:9/794: link d12/f4c d12/d2d/d1d/f118 0 2024-03-26T18:54:33.008 INFO:tasks.workunit.client.1.smithi190.stdout:9/795: write d12/d2d/f99 [832174,3912] 0 2024-03-26T18:54:33.008 INFO:tasks.workunit.client.1.smithi190.stdout:9/796: write d12/d2d/d1d/d54/d92/de3/f117 [350869,25105] 0 2024-03-26T18:54:33.009 INFO:tasks.workunit.client.1.smithi190.stdout:9/797: stat d12/d80/cea 0 2024-03-26T18:54:33.009 INFO:tasks.workunit.client.1.smithi190.stdout:9/798: chown d12/d2d/d1d/d23/l65 151837 1 2024-03-26T18:54:33.009 INFO:tasks.workunit.client.0.smithi102.stdout:8/227: dwrite d0/d3/d31/f40 [0,4194304] 0 2024-03-26T18:54:33.010 INFO:tasks.workunit.client.1.smithi190.stdout:5/424: rename d2/d17/d1b/d30/f51 to d2/d17/d1b/d1e/f97 0 2024-03-26T18:54:33.010 INFO:tasks.workunit.client.1.smithi190.stdout:5/425: write d2/d17/d1b/d1e/f97 [83223,69764] 0 2024-03-26T18:54:33.010 INFO:tasks.workunit.client.1.smithi190.stdout:5/426: read - d2/d17/f4f zero size 2024-03-26T18:54:33.010 INFO:tasks.workunit.client.1.smithi190.stdout:5/427: readlink d2/d17/d1b/l37 0 2024-03-26T18:54:33.010 INFO:tasks.workunit.client.0.smithi102.stdout:7/176: dwrite d7/f20 [4194304,4194304] 0 2024-03-26T18:54:33.011 INFO:tasks.workunit.client.0.smithi102.stdout:1/240: dwrite de/d16/d1d/f1e [0,4194304] 0 2024-03-26T18:54:33.011 INFO:tasks.workunit.client.0.smithi102.stdout:1/241: write de/f3e [706916,128636] 0 2024-03-26T18:54:33.011 INFO:tasks.workunit.client.0.smithi102.stdout:1/242: dread - de/d16/d1d/d2a/d2c/f3c zero size 2024-03-26T18:54:33.011 INFO:tasks.workunit.client.1.smithi190.stdout:0/632: mkdir d0/dd/d24/d4b/d5e/d66/dcb/de7 0 2024-03-26T18:54:33.013 INFO:tasks.workunit.client.1.smithi190.stdout:9/799: mkdir d12/d2d/dfc/d119 0 2024-03-26T18:54:33.013 INFO:tasks.workunit.client.1.smithi190.stdout:9/800: dread - d12/d2d/d1d/d23/d83/dc9/f10a zero size 2024-03-26T18:54:33.013 INFO:tasks.workunit.client.1.smithi190.stdout:8/556: write f6 [4764664,119666] 0 2024-03-26T18:54:33.014 INFO:tasks.workunit.client.0.smithi102.stdout:8/228: unlink d0/d3/d2c/d33/f3d 0 2024-03-26T18:54:33.014 INFO:tasks.workunit.client.0.smithi102.stdout:8/229: creat d0/d15/f4b x:0 0 0 2024-03-26T18:54:33.021 INFO:tasks.workunit.client.1.smithi190.stdout:1/464: mknod d3/da/dd/d2f/d34/c98 0 2024-03-26T18:54:33.022 INFO:tasks.workunit.client.0.smithi102.stdout:7/177: write d7/f29 [153371,37104] 0 2024-03-26T18:54:33.022 INFO:tasks.workunit.client.0.smithi102.stdout:8/230: write d0/d3/d31/f40 [2656368,115384] 0 2024-03-26T18:54:33.030 INFO:tasks.workunit.client.0.smithi102.stdout:7/178: dread d7/f29 [0,4194304] 0 2024-03-26T18:54:33.032 INFO:tasks.workunit.client.0.smithi102.stdout:8/231: unlink d0/d3/d26/d19/d27/f3e 0 2024-03-26T18:54:33.036 INFO:tasks.workunit.client.1.smithi190.stdout:0/633: mkdir d0/dd/d24/d4b/d5e/d6b/de8 0 2024-03-26T18:54:33.036 INFO:tasks.workunit.client.1.smithi190.stdout:9/801: rmdir d12/d2d/d1d/da1 39 2024-03-26T18:54:33.036 INFO:tasks.workunit.client.1.smithi190.stdout:9/802: chown d12/d80 9 1 2024-03-26T18:54:33.038 INFO:tasks.workunit.client.1.smithi190.stdout:5/428: rmdir d2/d17/d62/d64 39 2024-03-26T18:54:33.038 INFO:tasks.workunit.client.1.smithi190.stdout:5/429: readlink d2/l2b 0 2024-03-26T18:54:33.040 INFO:tasks.workunit.client.1.smithi190.stdout:1/465: mkdir d3/da/d3a/d99 0 2024-03-26T18:54:33.042 INFO:tasks.workunit.client.1.smithi190.stdout:0/634: creat d0/dd/d20/d1c/d8b/d8d/db7/de6/fe9 x:0 0 0 2024-03-26T18:54:33.045 INFO:tasks.workunit.client.1.smithi190.stdout:8/557: dwrite d9/f33 [0,4194304] 0 2024-03-26T18:54:33.047 INFO:tasks.workunit.client.0.smithi102.stdout:1/243: rmdir de 39 2024-03-26T18:54:33.049 INFO:tasks.workunit.client.0.smithi102.stdout:7/179: write d7/d12/f22 [287272,80906] 0 2024-03-26T18:54:33.049 INFO:tasks.workunit.client.0.smithi102.stdout:7/180: chown d7/d12/c2d 3 1 2024-03-26T18:54:33.053 INFO:tasks.workunit.client.1.smithi190.stdout:8/558: symlink d9/dd/d15/lbf 0 2024-03-26T18:54:33.053 INFO:tasks.workunit.client.1.smithi190.stdout:9/803: dread d12/f14 [0,4194304] 0 2024-03-26T18:54:33.053 INFO:tasks.workunit.client.1.smithi190.stdout:8/559: creat d9/d48/d49/fc0 x:0 0 0 2024-03-26T18:54:33.053 INFO:tasks.workunit.client.1.smithi190.stdout:8/560: chown d9/d39/c56 233885948 1 2024-03-26T18:54:33.053 INFO:tasks.workunit.client.1.smithi190.stdout:8/561: chown d9/da2/c2a 14 1 2024-03-26T18:54:33.053 INFO:tasks.workunit.client.1.smithi190.stdout:8/562: chown d9/dd/d23 115 1 2024-03-26T18:54:33.054 INFO:tasks.workunit.client.0.smithi102.stdout:7/181: creat d7/d10/d27/f33 x:0 0 0 2024-03-26T18:54:33.061 INFO:tasks.workunit.client.1.smithi190.stdout:8/563: mkdir d9/d39/d67/dc1 0 2024-03-26T18:54:33.061 INFO:tasks.workunit.client.1.smithi190.stdout:8/564: getdents d9/da2/d87 0 2024-03-26T18:54:33.063 INFO:tasks.workunit.client.1.smithi190.stdout:9/804: rmdir d12/d2d/d1d/d23/d2b/da0 39 2024-03-26T18:54:33.063 INFO:tasks.workunit.client.1.smithi190.stdout:9/805: dread - d12/d2d/d1d/d23/d83/d91/fae zero size 2024-03-26T18:54:33.065 INFO:tasks.workunit.client.1.smithi190.stdout:8/565: mkdir d9/dd/d23/d36/d37/d3a/d43/dc2 0 2024-03-26T18:54:33.066 INFO:tasks.workunit.client.1.smithi190.stdout:8/566: write d9/da2/d73/fb0 [1017450,28949] 0 2024-03-26T18:54:33.066 INFO:tasks.workunit.client.1.smithi190.stdout:5/430: dwrite d2/d17/d1b/d22/f2d [0,4194304] 0 2024-03-26T18:54:33.071 INFO:tasks.workunit.client.1.smithi190.stdout:8/567: mknod d9/d48/cc3 0 2024-03-26T18:54:33.071 INFO:tasks.workunit.client.1.smithi190.stdout:8/568: fdatasync d9/dd/d15/f4b 0 2024-03-26T18:54:33.073 INFO:tasks.workunit.client.1.smithi190.stdout:5/431: rename d2/d17/d1b/d1e to d2/d17/d98 0 2024-03-26T18:54:33.074 INFO:tasks.workunit.client.1.smithi190.stdout:1/466: dwrite d3/f29 [4194304,4194304] 0 2024-03-26T18:54:33.074 INFO:tasks.workunit.client.1.smithi190.stdout:5/432: read d2/d17/d1b/f76 [319499,92802] 0 2024-03-26T18:54:33.075 INFO:tasks.workunit.client.1.smithi190.stdout:0/635: dwrite d0/dd/d24/f7d [0,4194304] 0 2024-03-26T18:54:33.075 INFO:tasks.workunit.client.1.smithi190.stdout:0/636: chown d0/dd 895332 1 2024-03-26T18:54:33.076 INFO:tasks.workunit.client.0.smithi102.stdout:7/182: rmdir d7/d10/d27 39 2024-03-26T18:54:33.076 INFO:tasks.workunit.client.0.smithi102.stdout:7/183: write d7/f29 [3814937,18311] 0 2024-03-26T18:54:33.078 INFO:tasks.workunit.client.0.smithi102.stdout:7/184: truncate d7/d10/f14 595225 0 2024-03-26T18:54:33.081 INFO:tasks.workunit.client.1.smithi190.stdout:9/806: rmdir d12/d2d 39 2024-03-26T18:54:33.082 INFO:tasks.workunit.client.0.smithi102.stdout:8/232: dwrite d0/d3/d26/f22 [0,4194304] 0 2024-03-26T18:54:33.083 INFO:tasks.workunit.client.0.smithi102.stdout:7/185: truncate f6 2211548 0 2024-03-26T18:54:33.083 INFO:tasks.workunit.client.0.smithi102.stdout:7/186: dread - d7/d2a/f30 zero size 2024-03-26T18:54:33.084 INFO:tasks.workunit.client.1.smithi190.stdout:1/467: write d3/da/f47 [2280366,15043] 0 2024-03-26T18:54:33.084 INFO:tasks.workunit.client.1.smithi190.stdout:1/468: truncate d3/da/f94 562342 0 2024-03-26T18:54:33.084 INFO:tasks.workunit.client.1.smithi190.stdout:1/469: chown d3/da/dd/d95/d52/f7e 17232436 1 2024-03-26T18:54:33.086 INFO:tasks.workunit.client.1.smithi190.stdout:0/637: write d0/dd/d20/f30 [983463,37021] 0 2024-03-26T18:54:33.089 INFO:tasks.workunit.client.1.smithi190.stdout:4/559: sync 2024-03-26T18:54:33.089 INFO:tasks.workunit.client.1.smithi190.stdout:5/433: getdents d2/d17/d1b/d22/d88 0 2024-03-26T18:54:33.089 INFO:tasks.workunit.client.1.smithi190.stdout:5/434: readlink d2/d17/d98/d3a/d3c/l6f 0 2024-03-26T18:54:33.095 INFO:tasks.workunit.client.1.smithi190.stdout:5/435: read d2/d17/d1b/f67 [52394,127804] 0 2024-03-26T18:54:33.098 INFO:tasks.workunit.client.1.smithi190.stdout:9/807: mknod d12/d2d/d1d/d23/d83/d91/c11a 0 2024-03-26T18:54:33.098 INFO:tasks.workunit.client.1.smithi190.stdout:9/808: truncate d12/d2d/d1d/d23/d83/d91/fdd 194781 0 2024-03-26T18:54:33.101 INFO:tasks.workunit.client.1.smithi190.stdout:5/436: mkdir d2/d17/d1b/d60/d63/d85/d99 0 2024-03-26T18:54:33.104 INFO:tasks.workunit.client.1.smithi190.stdout:8/569: rename d9/dd/d15/d32 to d9/da2/dc4 0 2024-03-26T18:54:33.104 INFO:tasks.workunit.client.1.smithi190.stdout:8/570: write d9/da2/dc4/f96 [881656,75812] 0 2024-03-26T18:54:33.104 INFO:tasks.workunit.client.1.smithi190.stdout:8/571: dread - d9/dd/d23/d36/d37/fb4 zero size 2024-03-26T18:54:33.105 INFO:tasks.workunit.client.0.smithi102.stdout:7/187: mkdir d7/d10/d27/d34 0 2024-03-26T18:54:33.105 INFO:tasks.workunit.client.1.smithi190.stdout:8/572: read d9/dd/d23/d36/d37/d3a/d43/f80 [208873,59625] 0 2024-03-26T18:54:33.105 INFO:tasks.workunit.client.1.smithi190.stdout:8/573: fdatasync d9/dd/d23/d36/d37/d3a/d43/f80 0 2024-03-26T18:54:33.106 INFO:tasks.workunit.client.0.smithi102.stdout:8/233: getdents d0/d3 0 2024-03-26T18:54:33.106 INFO:tasks.workunit.client.0.smithi102.stdout:7/188: dread d7/d10/f14 [0,4194304] 0 2024-03-26T18:54:33.107 INFO:tasks.workunit.client.0.smithi102.stdout:7/189: dread d7/d10/f1c [0,4194304] 0 2024-03-26T18:54:33.107 INFO:tasks.workunit.client.1.smithi190.stdout:9/809: rename d12/d2d/d1d/d23/f59 to d12/d80/db2/f11b 0 2024-03-26T18:54:33.110 INFO:tasks.workunit.client.0.smithi102.stdout:7/190: creat d7/d10/d17/f35 x:0 0 0 2024-03-26T18:54:33.110 INFO:tasks.workunit.client.0.smithi102.stdout:7/191: fdatasync d7/f29 0 2024-03-26T18:54:33.113 INFO:tasks.workunit.client.1.smithi190.stdout:9/810: creat d12/d2d/d1d/d23/d83/dc9/dd4/f11c x:0 0 0 2024-03-26T18:54:33.113 INFO:tasks.workunit.client.1.smithi190.stdout:9/811: chown d12/d2d/d1d/d23/d2b/da0/lf2 35 1 2024-03-26T18:54:33.114 INFO:tasks.workunit.client.0.smithi102.stdout:7/192: mknod d7/d10/d17/c36 0 2024-03-26T18:54:33.116 INFO:tasks.workunit.client.1.smithi190.stdout:9/812: symlink d12/d2d/d1d/da1/dba/dbc/l11d 0 2024-03-26T18:54:33.117 INFO:tasks.workunit.client.1.smithi190.stdout:5/437: dread d2/d17/d1b/d22/f2d [0,4194304] 0 2024-03-26T18:54:33.117 INFO:tasks.workunit.client.1.smithi190.stdout:5/438: fdatasync d2/f14 0 2024-03-26T18:54:33.119 INFO:tasks.workunit.client.1.smithi190.stdout:5/439: rename d2/d17/d1b/d22/f3d to d2/d17/d1b/d22/f9a 0 2024-03-26T18:54:33.119 INFO:tasks.workunit.client.1.smithi190.stdout:5/440: chown d2/d17/d98/f5f 0 1 2024-03-26T18:54:33.119 INFO:tasks.workunit.client.1.smithi190.stdout:5/441: truncate d2/d17/d1b/d22/f7f 5059888 0 2024-03-26T18:54:33.119 INFO:tasks.workunit.client.1.smithi190.stdout:1/470: dwrite d3/da/f37 [0,4194304] 0 2024-03-26T18:54:33.130 INFO:tasks.workunit.client.1.smithi190.stdout:0/638: dwrite d0/dd/d24/f49 [0,4194304] 0 2024-03-26T18:54:33.130 INFO:tasks.workunit.client.1.smithi190.stdout:9/813: dread d12/d2d/d1d/d23/d83/dc9/dd4/d3f/d55/f82 [0,4194304] 0 2024-03-26T18:54:33.131 INFO:tasks.workunit.client.0.smithi102.stdout:7/193: write f4 [3313484,81933] 0 2024-03-26T18:54:33.131 INFO:tasks.workunit.client.0.smithi102.stdout:7/194: creat d7/d10/d17/f37 x:0 0 0 2024-03-26T18:54:33.134 INFO:tasks.workunit.client.0.smithi102.stdout:7/195: rename d7/d10/d17/f21 to d7/d12/f38 0 2024-03-26T18:54:33.135 INFO:tasks.workunit.client.1.smithi190.stdout:9/814: getdents d12/d2d/d1d/d23/d83/dc9/dd4/d3f 0 2024-03-26T18:54:33.136 INFO:tasks.workunit.client.0.smithi102.stdout:7/196: write d7/d10/f14 [450579,36376] 0 2024-03-26T18:54:33.136 INFO:tasks.workunit.client.0.smithi102.stdout:7/197: stat d7/d10/d17/f1b 0 2024-03-26T18:54:33.137 INFO:tasks.workunit.client.1.smithi190.stdout:9/815: creat d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/d10e/f11e x:0 0 0 2024-03-26T18:54:33.140 INFO:tasks.workunit.client.1.smithi190.stdout:8/574: dwrite d9/dd/f59 [0,4194304] 0 2024-03-26T18:54:33.140 INFO:tasks.workunit.client.1.smithi190.stdout:8/575: chown d9/da2/dc4/f66 42 1 2024-03-26T18:54:33.140 INFO:tasks.workunit.client.1.smithi190.stdout:8/576: chown d9/d39/d79/f90 999747410 1 2024-03-26T18:54:33.140 INFO:tasks.workunit.client.1.smithi190.stdout:5/442: dwrite d2/f16 [4194304,4194304] 0 2024-03-26T18:54:33.141 INFO:tasks.workunit.client.1.smithi190.stdout:8/577: read d9/d39/d79/f90 [245410,50198] 0 2024-03-26T18:54:33.144 INFO:tasks.workunit.client.1.smithi190.stdout:5/443: symlink d2/d17/d1b/d60/d63/d85/l9b 0 2024-03-26T18:54:33.144 INFO:tasks.workunit.client.1.smithi190.stdout:5/444: dread - d2/d17/d98/d3a/f55 zero size 2024-03-26T18:54:33.146 INFO:tasks.workunit.client.1.smithi190.stdout:5/445: creat d2/d17/d1b/d60/d63/d85/d99/f9c x:0 0 0 2024-03-26T18:54:33.149 INFO:tasks.workunit.client.1.smithi190.stdout:5/446: mknod d2/d17/d1b/d22/d88/c9d 0 2024-03-26T18:54:33.158 INFO:tasks.workunit.client.0.smithi102.stdout:8/234: dwrite d0/d3/f6 [0,4194304] 0 2024-03-26T18:54:33.168 INFO:tasks.workunit.client.0.smithi102.stdout:8/235: write d0/d3/f6 [789655,41200] 0 2024-03-26T18:54:33.174 INFO:tasks.workunit.client.1.smithi190.stdout:3/458: sync 2024-03-26T18:54:33.175 INFO:tasks.workunit.client.1.smithi190.stdout:6/468: sync 2024-03-26T18:54:33.175 INFO:tasks.workunit.client.1.smithi190.stdout:6/469: chown f9 117388626 1 2024-03-26T18:54:33.175 INFO:tasks.workunit.client.1.smithi190.stdout:6/470: write d17/d28/d2c/d50/f9c [1690036,107136] 0 2024-03-26T18:54:33.176 INFO:tasks.workunit.client.1.smithi190.stdout:3/459: link d1/d3/d5a/f63 d1/de/d4e/f8b 0 2024-03-26T18:54:33.176 INFO:tasks.workunit.client.1.smithi190.stdout:1/471: dwrite d3/da/dd/d2f/d34/f35 [0,4194304] 0 2024-03-26T18:54:33.176 INFO:tasks.workunit.client.1.smithi190.stdout:0/639: dwrite d0/dd/d20/d1c/d46/f72 [0,4194304] 0 2024-03-26T18:54:33.178 INFO:tasks.workunit.client.1.smithi190.stdout:6/471: truncate d17/d28/d2c/d77/f66 90700 0 2024-03-26T18:54:33.178 INFO:tasks.workunit.client.1.smithi190.stdout:5/447: dwrite d2/d17/d1b/d22/d24/d3b/f56 [0,4194304] 0 2024-03-26T18:54:33.178 INFO:tasks.workunit.client.1.smithi190.stdout:6/472: write d17/d28/d2c/d77/d51/f54 [1398345,101740] 0 2024-03-26T18:54:33.178 INFO:tasks.workunit.client.1.smithi190.stdout:6/473: write d17/d28/d2c/d50/f8f [1037342,16530] 0 2024-03-26T18:54:33.178 INFO:tasks.workunit.client.1.smithi190.stdout:3/460: mkdir d1/d3/d52/d7d/d8c 0 2024-03-26T18:54:33.179 INFO:tasks.workunit.client.0.smithi102.stdout:7/198: dwrite d7/f9 [0,4194304] 0 2024-03-26T18:54:33.180 INFO:tasks.workunit.client.1.smithi190.stdout:6/474: read d17/d28/d2c/d50/f8f [629955,96498] 0 2024-03-26T18:54:33.180 INFO:tasks.workunit.client.1.smithi190.stdout:6/475: stat d17/d28/f58 0 2024-03-26T18:54:33.180 INFO:tasks.workunit.client.1.smithi190.stdout:8/578: dwrite d9/dd/d15/f30 [0,4194304] 0 2024-03-26T18:54:33.180 INFO:tasks.workunit.client.1.smithi190.stdout:8/579: chown d9/d48/c5e 362750715 1 2024-03-26T18:54:33.180 INFO:tasks.workunit.client.1.smithi190.stdout:9/816: dwrite d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/d10e/f7e [0,4194304] 0 2024-03-26T18:54:33.180 INFO:tasks.workunit.client.1.smithi190.stdout:1/472: symlink d3/d5f/l9a 0 2024-03-26T18:54:33.182 INFO:tasks.workunit.client.1.smithi190.stdout:0/640: dread d0/dd/d20/d1c/d4e/f74 [0,4194304] 0 2024-03-26T18:54:33.190 INFO:tasks.workunit.client.1.smithi190.stdout:5/448: link d2/d17/l4a d2/d17/d1b/d22/d96/l9e 0 2024-03-26T18:54:33.192 INFO:tasks.workunit.client.1.smithi190.stdout:3/461: creat d1/d3/d52/d7d/d8c/f8d x:0 0 0 2024-03-26T18:54:33.196 INFO:tasks.workunit.client.1.smithi190.stdout:8/580: creat d9/d39/d9e/fc5 x:0 0 0 2024-03-26T18:54:33.202 INFO:tasks.workunit.client.1.smithi190.stdout:0/641: getdents d0/dd/d24/d4b/d5e/d81 0 2024-03-26T18:54:33.202 INFO:tasks.workunit.client.1.smithi190.stdout:5/449: symlink d2/d17/l9f 0 2024-03-26T18:54:33.202 INFO:tasks.workunit.client.1.smithi190.stdout:3/462: mknod d1/de/d8a/c8e 0 2024-03-26T18:54:33.202 INFO:tasks.workunit.client.1.smithi190.stdout:3/463: creat d1/de/d4e/f8f x:0 0 0 2024-03-26T18:54:33.202 INFO:tasks.workunit.client.1.smithi190.stdout:3/464: chown d1/d3/df/d18/d57/c7e 1475 1 2024-03-26T18:54:33.202 INFO:tasks.workunit.client.1.smithi190.stdout:8/581: mknod d9/d39/d79/dad/cc6 0 2024-03-26T18:54:33.203 INFO:tasks.workunit.client.1.smithi190.stdout:8/582: read d9/da2/dc4/f53 [19609,57301] 0 2024-03-26T18:54:33.204 INFO:tasks.workunit.client.1.smithi190.stdout:9/817: link d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/db3/lcb d12/d2d/d1d/d23/d83/dc9/l11f 0 2024-03-26T18:54:33.207 INFO:tasks.workunit.client.0.smithi102.stdout:4/191: sync 2024-03-26T18:54:33.207 INFO:tasks.workunit.client.0.smithi102.stdout:4/192: fsync d1/f8 0 2024-03-26T18:54:33.208 INFO:tasks.workunit.client.0.smithi102.stdout:4/193: fdatasync d1/d17/d1a/f1c 0 2024-03-26T18:54:33.208 INFO:tasks.workunit.client.0.smithi102.stdout:6/233: sync 2024-03-26T18:54:33.208 INFO:tasks.workunit.client.0.smithi102.stdout:9/145: sync 2024-03-26T18:54:33.208 INFO:tasks.workunit.client.0.smithi102.stdout:6/234: truncate d2/d7/f30 171737 0 2024-03-26T18:54:33.208 INFO:tasks.workunit.client.0.smithi102.stdout:6/235: write d2/f3a [3940968,81014] 0 2024-03-26T18:54:33.208 INFO:tasks.workunit.client.0.smithi102.stdout:6/236: creat d2/d12/f3f x:0 0 0 2024-03-26T18:54:33.208 INFO:tasks.workunit.client.1.smithi190.stdout:1/473: symlink d3/da/dd/d2f/d34/l9b 0 2024-03-26T18:54:33.208 INFO:tasks.workunit.client.1.smithi190.stdout:1/474: dread - d3/da/d3a/f50 zero size 2024-03-26T18:54:33.209 INFO:tasks.workunit.client.0.smithi102.stdout:2/211: sync 2024-03-26T18:54:33.210 INFO:tasks.workunit.client.1.smithi190.stdout:6/476: dwrite d17/d28/d2b/f6e [0,4194304] 0 2024-03-26T18:54:33.211 INFO:tasks.workunit.client.1.smithi190.stdout:5/450: symlink d2/d17/d98/d3a/d3c/la0 0 2024-03-26T18:54:33.212 INFO:tasks.workunit.client.0.smithi102.stdout:7/199: dwrite d7/d10/f1c [0,4194304] 0 2024-03-26T18:54:33.215 INFO:tasks.workunit.client.0.smithi102.stdout:9/146: rename df/d1a/d23/c2e to df/d1a/d23/c2f 0 2024-03-26T18:54:33.215 INFO:tasks.workunit.client.0.smithi102.stdout:9/147: creat df/f30 x:0 0 0 2024-03-26T18:54:33.215 INFO:tasks.workunit.client.0.smithi102.stdout:9/148: chown df/d1a/d23/d29 114528357 1 2024-03-26T18:54:33.224 INFO:tasks.workunit.client.0.smithi102.stdout:9/149: creat df/d1a/d23/d29/f31 x:0 0 0 2024-03-26T18:54:33.224 INFO:tasks.workunit.client.1.smithi190.stdout:6/477: creat d17/d28/d74/d7b/fa5 x:0 0 0 2024-03-26T18:54:33.226 INFO:tasks.workunit.client.1.smithi190.stdout:5/451: symlink d2/d17/la1 0 2024-03-26T18:54:33.228 INFO:tasks.workunit.client.0.smithi102.stdout:8/236: dwrite d0/d15/d1b/f35 [4194304,4194304] 0 2024-03-26T18:54:33.228 INFO:tasks.workunit.client.0.smithi102.stdout:4/194: dread d1/d17/d1f/f2d [4194304,4194304] 0 2024-03-26T18:54:33.228 INFO:tasks.workunit.client.0.smithi102.stdout:8/237: dread - d0/d3/d2c/f42 zero size 2024-03-26T18:54:33.229 INFO:tasks.workunit.client.1.smithi190.stdout:5/452: creat d2/d17/d98/d3a/d3c/fa2 x:0 0 0 2024-03-26T18:54:33.230 INFO:tasks.workunit.client.0.smithi102.stdout:4/195: mkdir d1/d25/d39/d3e 0 2024-03-26T18:54:33.234 INFO:tasks.workunit.client.1.smithi190.stdout:5/453: creat d2/d17/d1b/d60/d63/fa3 x:0 0 0 2024-03-26T18:54:33.235 INFO:tasks.workunit.client.0.smithi102.stdout:4/196: mknod d1/d17/c3f 0 2024-03-26T18:54:33.236 INFO:tasks.workunit.client.0.smithi102.stdout:8/238: rmdir d0/d15 39 2024-03-26T18:54:33.236 INFO:tasks.workunit.client.0.smithi102.stdout:7/200: dread d7/d12/f22 [0,4194304] 0 2024-03-26T18:54:33.237 INFO:tasks.workunit.client.0.smithi102.stdout:4/197: symlink d1/d17/d2a/l40 0 2024-03-26T18:54:33.239 INFO:tasks.workunit.client.0.smithi102.stdout:7/201: mknod d7/d10/c39 0 2024-03-26T18:54:33.239 INFO:tasks.workunit.client.0.smithi102.stdout:7/202: chown d7/d12/c2d 156 1 2024-03-26T18:54:33.239 INFO:tasks.workunit.client.1.smithi190.stdout:5/454: dread d2/d17/d98/d3a/f52 [0,4194304] 0 2024-03-26T18:54:33.239 INFO:tasks.workunit.client.1.smithi190.stdout:0/642: dwrite d0/dd/d24/da2/d80/fb2 [0,4194304] 0 2024-03-26T18:54:33.239 INFO:tasks.workunit.client.1.smithi190.stdout:5/455: write d2/f6 [996643,34458] 0 2024-03-26T18:54:33.240 INFO:tasks.workunit.client.0.smithi102.stdout:9/150: read df/d1a/f20 [2967448,123006] 0 2024-03-26T18:54:33.240 INFO:tasks.workunit.client.0.smithi102.stdout:9/151: creat df/d1a/d1d/f32 x:0 0 0 2024-03-26T18:54:33.240 INFO:tasks.workunit.client.0.smithi102.stdout:9/152: truncate df/f28 84923 0 2024-03-26T18:54:33.240 INFO:tasks.workunit.client.0.smithi102.stdout:9/153: truncate df/f28 711159 0 2024-03-26T18:54:33.241 INFO:tasks.workunit.client.0.smithi102.stdout:9/154: dread df/f28 [0,4194304] 0 2024-03-26T18:54:33.242 INFO:tasks.workunit.client.0.smithi102.stdout:4/198: rename d1/le to d1/d17/d1a/d21/l41 0 2024-03-26T18:54:33.244 INFO:tasks.workunit.client.0.smithi102.stdout:7/203: symlink d7/l3a 0 2024-03-26T18:54:33.246 INFO:tasks.workunit.client.0.smithi102.stdout:4/199: mknod d1/d17/d2a/c42 0 2024-03-26T18:54:33.248 INFO:tasks.workunit.client.0.smithi102.stdout:7/204: link d7/ld d7/d10/d27/l3b 0 2024-03-26T18:54:33.248 INFO:tasks.workunit.client.1.smithi190.stdout:1/475: dwrite d3/da/dd/f12 [4194304,4194304] 0 2024-03-26T18:54:33.249 INFO:tasks.workunit.client.1.smithi190.stdout:9/818: dwrite d12/d2d/d1d/d23/f33 [0,4194304] 0 2024-03-26T18:54:33.250 INFO:tasks.workunit.client.0.smithi102.stdout:6/237: dwrite d2/f2e [0,4194304] 0 2024-03-26T18:54:33.250 INFO:tasks.workunit.client.0.smithi102.stdout:7/205: rename d7/f29 to d7/d12/f3c 0 2024-03-26T18:54:33.253 INFO:tasks.workunit.client.1.smithi190.stdout:9/819: creat d12/d2d/d1d/da1/dba/f120 x:0 0 0 2024-03-26T18:54:33.257 INFO:tasks.workunit.client.1.smithi190.stdout:9/820: write d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/db3/fca [600481,83451] 0 2024-03-26T18:54:33.258 INFO:tasks.workunit.client.1.smithi190.stdout:9/821: write d12/d2d/f3e [564127,78588] 0 2024-03-26T18:54:33.261 INFO:tasks.workunit.client.1.smithi190.stdout:9/822: creat d12/d2d/d1d/d23/d10d/f121 x:0 0 0 2024-03-26T18:54:33.264 INFO:tasks.workunit.client.1.smithi190.stdout:1/476: write d3/da/f24 [4799542,45003] 0 2024-03-26T18:54:33.264 INFO:tasks.workunit.client.0.smithi102.stdout:7/206: symlink d7/l3d 0 2024-03-26T18:54:33.266 INFO:tasks.workunit.client.1.smithi190.stdout:6/478: dwrite d17/d28/d80/f81 [0,4194304] 0 2024-03-26T18:54:33.266 INFO:tasks.workunit.client.1.smithi190.stdout:1/477: unlink d3/d9/db/f2c 0 2024-03-26T18:54:33.267 INFO:tasks.workunit.client.1.smithi190.stdout:1/478: read d3/d9/f3c [302819,113100] 0 2024-03-26T18:54:33.267 INFO:tasks.workunit.client.1.smithi190.stdout:1/479: chown d3/da/dd/c96 105394 1 2024-03-26T18:54:33.267 INFO:tasks.workunit.client.1.smithi190.stdout:1/480: write d3/d9/f20 [3605800,68505] 0 2024-03-26T18:54:33.267 INFO:tasks.workunit.client.1.smithi190.stdout:1/481: dread - d3/da/dd/d95/d52/f93 zero size 2024-03-26T18:54:33.268 INFO:tasks.workunit.client.1.smithi190.stdout:8/583: dwrite d9/dd/d23/f27 [0,4194304] 0 2024-03-26T18:54:33.270 INFO:tasks.workunit.client.1.smithi190.stdout:6/479: rmdir d17/d28/d2c/d77/d94 0 2024-03-26T18:54:33.272 INFO:tasks.workunit.client.1.smithi190.stdout:1/482: mkdir d3/da/dd/d2f/d33/d9c 0 2024-03-26T18:54:33.272 INFO:tasks.workunit.client.1.smithi190.stdout:1/483: write d3/d73/d88/f92 [609843,20967] 0 2024-03-26T18:54:33.274 INFO:tasks.workunit.client.1.smithi190.stdout:8/584: rename d9/l26 to d9/d48/d49/lc7 0 2024-03-26T18:54:33.276 INFO:tasks.workunit.client.1.smithi190.stdout:1/484: mknod d3/d9/db/c9d 0 2024-03-26T18:54:33.277 INFO:tasks.workunit.client.1.smithi190.stdout:1/485: symlink d3/da/dd/d2f/l9e 0 2024-03-26T18:54:33.277 INFO:tasks.workunit.client.1.smithi190.stdout:1/486: chown d3/d9 912220083 1 2024-03-26T18:54:33.277 INFO:tasks.workunit.client.1.smithi190.stdout:1/487: write d3/d9/f2a [2875487,52346] 0 2024-03-26T18:54:33.279 INFO:tasks.workunit.client.1.smithi190.stdout:1/488: mkdir d3/d9/d9f 0 2024-03-26T18:54:33.279 INFO:tasks.workunit.client.1.smithi190.stdout:8/585: dread d9/dd/d23/d36/d37/d3a/d61/f91 [0,4194304] 0 2024-03-26T18:54:33.279 INFO:tasks.workunit.client.1.smithi190.stdout:8/586: readlink l5 0 2024-03-26T18:54:33.280 INFO:tasks.workunit.client.1.smithi190.stdout:1/489: mknod d3/d9/d9f/ca0 0 2024-03-26T18:54:33.280 INFO:tasks.workunit.client.1.smithi190.stdout:1/490: truncate d3/da/f94 786705 0 2024-03-26T18:54:33.288 INFO:tasks.workunit.client.0.smithi102.stdout:8/239: dwrite d0/d3/d31/f40 [4194304,4194304] 0 2024-03-26T18:54:33.288 INFO:tasks.workunit.client.0.smithi102.stdout:4/200: dwrite d1/f8 [4194304,4194304] 0 2024-03-26T18:54:33.289 INFO:tasks.workunit.client.1.smithi190.stdout:2/549: sync 2024-03-26T18:54:33.289 INFO:tasks.workunit.client.1.smithi190.stdout:7/621: sync 2024-03-26T18:54:33.291 INFO:tasks.workunit.client.1.smithi190.stdout:5/456: dwrite d2/d17/d62/d64/f74 [4194304,4194304] 0 2024-03-26T18:54:33.291 INFO:tasks.workunit.client.1.smithi190.stdout:5/457: dread - d2/d17/d1b/d22/d88/f8b zero size 2024-03-26T18:54:33.292 INFO:tasks.workunit.client.0.smithi102.stdout:9/155: dwrite df/d1a/d23/d29/f31 [0,4194304] 0 2024-03-26T18:54:33.292 INFO:tasks.workunit.client.0.smithi102.stdout:8/240: creat d0/d15/d1b/f4c x:0 0 0 2024-03-26T18:54:33.292 INFO:tasks.workunit.client.0.smithi102.stdout:8/241: fdatasync d0/d3/d2c/f42 0 2024-03-26T18:54:33.293 INFO:tasks.workunit.client.0.smithi102.stdout:4/201: rmdir d1/d17 39 2024-03-26T18:54:33.296 INFO:tasks.workunit.client.1.smithi190.stdout:9/823: dwrite d12/d2d/d1d/d23/f33 [0,4194304] 0 2024-03-26T18:54:33.300 INFO:tasks.workunit.client.0.smithi102.stdout:6/238: dwrite d2/d12/f3f [0,4194304] 0 2024-03-26T18:54:33.300 INFO:tasks.workunit.client.0.smithi102.stdout:9/156: mkdir df/d1a/d33 0 2024-03-26T18:54:33.300 INFO:tasks.workunit.client.0.smithi102.stdout:9/157: creat df/d1a/d23/d29/f34 x:0 0 0 2024-03-26T18:54:33.309 INFO:tasks.workunit.client.1.smithi190.stdout:9/824: write d12/d2d/f31 [1313119,13651] 0 2024-03-26T18:54:33.309 INFO:tasks.workunit.client.1.smithi190.stdout:9/825: fdatasync d12/fb0 0 2024-03-26T18:54:33.309 INFO:tasks.workunit.client.0.smithi102.stdout:6/239: symlink d2/df/l40 0 2024-03-26T18:54:33.309 INFO:tasks.workunit.client.0.smithi102.stdout:6/240: fdatasync d2/d12/f2d 0 2024-03-26T18:54:33.311 INFO:tasks.workunit.client.1.smithi190.stdout:6/480: dwrite d17/d28/d2b/f3c [0,4194304] 0 2024-03-26T18:54:33.311 INFO:tasks.workunit.client.1.smithi190.stdout:6/481: chown d17/d28/d2c/d77/d51/c6d 822 1 2024-03-26T18:54:33.311 INFO:tasks.workunit.client.1.smithi190.stdout:8/587: dwrite d9/dd/d15/f2c [0,4194304] 0 2024-03-26T18:54:33.312 INFO:tasks.workunit.client.1.smithi190.stdout:6/482: chown d17/d28/d2c/d77/d51/f54 5766 1 2024-03-26T18:54:33.312 INFO:tasks.workunit.client.1.smithi190.stdout:1/491: getdents d3/d9 0 2024-03-26T18:54:33.313 INFO:tasks.workunit.client.1.smithi190.stdout:7/622: mknod d3/d6/d9/d1f/d4c/d71/ce2 0 2024-03-26T18:54:33.314 INFO:tasks.workunit.client.1.smithi190.stdout:8/588: dread d9/dd/d15/f60 [0,4194304] 0 2024-03-26T18:54:33.314 INFO:tasks.workunit.client.0.smithi102.stdout:9/158: symlink df/d1a/l35 0 2024-03-26T18:54:33.314 INFO:tasks.workunit.client.0.smithi102.stdout:9/159: dread - df/d1a/d23/f2c zero size 2024-03-26T18:54:33.316 INFO:tasks.workunit.client.1.smithi190.stdout:5/458: creat d2/d17/d1b/d22/d96/fa4 x:0 0 0 2024-03-26T18:54:33.317 INFO:tasks.workunit.client.0.smithi102.stdout:6/241: truncate d2/d12/f22 272123 0 2024-03-26T18:54:33.318 INFO:tasks.workunit.client.0.smithi102.stdout:6/242: dread d2/d7/f30 [0,4194304] 0 2024-03-26T18:54:33.318 INFO:tasks.workunit.client.0.smithi102.stdout:6/243: dread - d2/df/f2b zero size 2024-03-26T18:54:33.318 INFO:tasks.workunit.client.1.smithi190.stdout:9/826: mkdir d12/d2d/d1d/d23/de5/d122 0 2024-03-26T18:54:33.319 INFO:tasks.workunit.client.1.smithi190.stdout:9/827: dread - d12/d2d/d1d/d23/d83/dc9/dd4/d71/f8b zero size 2024-03-26T18:54:33.320 INFO:tasks.workunit.client.0.smithi102.stdout:9/160: symlink df/d1a/d23/d29/l36 0 2024-03-26T18:54:33.324 INFO:tasks.workunit.client.0.smithi102.stdout:9/161: rename df/d1a/l35 to df/d1a/d1d/l37 0 2024-03-26T18:54:33.324 INFO:tasks.workunit.client.1.smithi190.stdout:2/550: dwrite d6/d9/f4e [0,4194304] 0 2024-03-26T18:54:33.324 INFO:tasks.workunit.client.1.smithi190.stdout:2/551: read d6/d9/d3a/d3d/d1c/f5e [2010240,5793] 0 2024-03-26T18:54:33.325 INFO:tasks.workunit.client.1.smithi190.stdout:1/492: mknod d3/da/d3a/ca1 0 2024-03-26T18:54:33.327 INFO:tasks.workunit.client.1.smithi190.stdout:6/483: dread d17/d28/d2b/f6e [0,4194304] 0 2024-03-26T18:54:33.330 INFO:tasks.workunit.client.1.smithi190.stdout:8/589: creat d9/fc8 x:0 0 0 2024-03-26T18:54:33.330 INFO:tasks.workunit.client.1.smithi190.stdout:8/590: read - d9/dd/f99 zero size 2024-03-26T18:54:33.331 INFO:tasks.workunit.client.0.smithi102.stdout:6/244: dread d2/d7/f1d [0,4194304] 0 2024-03-26T18:54:33.331 INFO:tasks.workunit.client.0.smithi102.stdout:6/245: creat d2/d7/d3c/f41 x:0 0 0 2024-03-26T18:54:33.332 INFO:tasks.workunit.client.0.smithi102.stdout:0/243: sync 2024-03-26T18:54:33.332 INFO:tasks.workunit.client.1.smithi190.stdout:5/459: mkdir d2/d17/d1b/d22/d24/d3b/da5 0 2024-03-26T18:54:33.333 INFO:tasks.workunit.client.1.smithi190.stdout:7/623: dread d3/d6/d9/d1f/d41/d50/f52 [0,4194304] 0 2024-03-26T18:54:33.333 INFO:tasks.workunit.client.1.smithi190.stdout:9/828: symlink d12/d2d/d1d/d23/d83/l123 0 2024-03-26T18:54:33.336 INFO:tasks.workunit.client.0.smithi102.stdout:6/246: link d2/d7/c25 d2/c42 0 2024-03-26T18:54:33.339 INFO:tasks.workunit.client.0.smithi102.stdout:6/247: mkdir d2/df/d35/d43 0 2024-03-26T18:54:33.341 INFO:tasks.workunit.client.0.smithi102.stdout:8/242: truncate d0/d3/d31/f40 2585154 0 2024-03-26T18:54:33.343 INFO:tasks.workunit.client.1.smithi190.stdout:2/552: dwrite d6/d9/d10/d5f/fa2 [4194304,4194304] 0 2024-03-26T18:54:33.343 INFO:tasks.workunit.client.1.smithi190.stdout:2/553: creat d6/d9/d3a/d3d/d55/fc6 x:0 0 0 2024-03-26T18:54:33.343 INFO:tasks.workunit.client.1.smithi190.stdout:2/554: write f5 [2391989,91684] 0 2024-03-26T18:54:33.344 INFO:tasks.workunit.client.0.smithi102.stdout:6/248: rename d2/c1c to d2/c44 0 2024-03-26T18:54:33.344 INFO:tasks.workunit.client.0.smithi102.stdout:6/249: creat d2/d12/f45 x:0 0 0 2024-03-26T18:54:33.347 INFO:tasks.workunit.client.0.smithi102.stdout:6/250: dread d2/d7/f20 [0,4194304] 0 2024-03-26T18:54:33.348 INFO:tasks.workunit.client.0.smithi102.stdout:0/244: write d6/d2d/f38 [2093687,29683] 0 2024-03-26T18:54:33.351 INFO:tasks.workunit.client.0.smithi102.stdout:9/162: dwrite df/d1a/f20 [0,4194304] 0 2024-03-26T18:54:33.352 INFO:tasks.workunit.client.1.smithi190.stdout:1/493: mknod d3/d9/d9f/ca2 0 2024-03-26T18:54:33.354 INFO:tasks.workunit.client.0.smithi102.stdout:0/245: mkdir d6/de/d13/d25/d4f 0 2024-03-26T18:54:33.354 INFO:tasks.workunit.client.0.smithi102.stdout:0/246: fsync d6/f11 0 2024-03-26T18:54:33.354 INFO:tasks.workunit.client.0.smithi102.stdout:0/247: creat d6/de/d13/d25/d31/f50 x:0 0 0 2024-03-26T18:54:33.354 INFO:tasks.workunit.client.0.smithi102.stdout:0/248: chown d6/d2d/d3d/f40 83 1 2024-03-26T18:54:33.354 INFO:tasks.workunit.client.0.smithi102.stdout:0/249: fdatasync d6/fd 0 2024-03-26T18:54:33.354 INFO:tasks.workunit.client.0.smithi102.stdout:0/250: readlink d6/de/d13/d25/l34 0 2024-03-26T18:54:33.354 INFO:tasks.workunit.client.0.smithi102.stdout:0/251: write d6/de/d13/d1f/f3b [8486054,83771] 0 2024-03-26T18:54:33.354 INFO:tasks.workunit.client.0.smithi102.stdout:0/252: stat d6/d2d/f38 0 2024-03-26T18:54:33.355 INFO:tasks.workunit.client.1.smithi190.stdout:6/484: link d17/d28/d46/l9e d17/d28/d2c/la6 0 2024-03-26T18:54:33.355 INFO:tasks.workunit.client.0.smithi102.stdout:9/163: stat df/c22 0 2024-03-26T18:54:33.358 INFO:tasks.workunit.client.1.smithi190.stdout:8/591: rename d9/dd/d23/caa to d9/da2/cc9 0 2024-03-26T18:54:33.360 INFO:tasks.workunit.client.1.smithi190.stdout:5/460: mknod d2/d17/d1b/d22/d68/ca6 0 2024-03-26T18:54:33.361 INFO:tasks.workunit.client.1.smithi190.stdout:6/485: write d17/d28/d2b/f6e [412105,112732] 0 2024-03-26T18:54:33.361 INFO:tasks.workunit.client.1.smithi190.stdout:6/486: read - d17/d28/d2c/d50/f65 zero size 2024-03-26T18:54:33.361 INFO:tasks.workunit.client.1.smithi190.stdout:6/487: write d17/d28/d38/f89 [668632,46712] 0 2024-03-26T18:54:33.361 INFO:tasks.workunit.client.0.smithi102.stdout:0/253: link d6/de/d19/f30 d6/f51 0 2024-03-26T18:54:33.363 INFO:tasks.workunit.client.1.smithi190.stdout:9/829: link d12/d2d/d1d/d23/d2b/c7a d12/d2d/d1d/d23/d8e/c124 0 2024-03-26T18:54:33.363 INFO:tasks.workunit.client.1.smithi190.stdout:9/830: readlink d12/l30 0 2024-03-26T18:54:33.363 INFO:tasks.workunit.client.0.smithi102.stdout:9/164: symlink df/d1a/d1d/l38 0 2024-03-26T18:54:33.365 INFO:tasks.workunit.client.1.smithi190.stdout:7/624: dwrite d3/d6/f30 [0,4194304] 0 2024-03-26T18:54:33.366 INFO:tasks.workunit.client.0.smithi102.stdout:0/254: dread d6/f51 [0,4194304] 0 2024-03-26T18:54:33.367 INFO:tasks.workunit.client.0.smithi102.stdout:0/255: truncate d6/d2d/f38 4450981 0 2024-03-26T18:54:33.367 INFO:tasks.workunit.client.0.smithi102.stdout:9/165: rename df/d1a/d23/c2f to df/d1a/d33/c39 0 2024-03-26T18:54:33.367 INFO:tasks.workunit.client.0.smithi102.stdout:9/166: creat df/d1a/d1d/f3a x:0 0 0 2024-03-26T18:54:33.368 INFO:tasks.workunit.client.1.smithi190.stdout:1/494: mkdir d3/d9/da3 0 2024-03-26T18:54:33.369 INFO:tasks.workunit.client.0.smithi102.stdout:8/243: dwrite d0/d3/d26/d19/d27/f29 [0,4194304] 0 2024-03-26T18:54:33.375 INFO:tasks.workunit.client.0.smithi102.stdout:6/251: dwrite d2/d12/f45 [0,4194304] 0 2024-03-26T18:54:33.376 INFO:tasks.workunit.client.1.smithi190.stdout:8/592: truncate d9/da2/dc4/f35 2150521 0 2024-03-26T18:54:33.377 INFO:tasks.workunit.client.1.smithi190.stdout:5/461: mkdir d2/d17/d1b/d22/d24/d3b/da7 0 2024-03-26T18:54:33.379 INFO:tasks.workunit.client.1.smithi190.stdout:0/643: sync 2024-03-26T18:54:33.379 INFO:tasks.workunit.client.1.smithi190.stdout:4/560: sync 2024-03-26T18:54:33.379 INFO:tasks.workunit.client.1.smithi190.stdout:4/561: chown d7/d8/d14/d6e/d6f 642 1 2024-03-26T18:54:33.379 INFO:tasks.workunit.client.1.smithi190.stdout:3/465: sync 2024-03-26T18:54:33.380 INFO:tasks.workunit.client.1.smithi190.stdout:6/488: link d17/d28/c64 d17/d28/d2c/d77/ca7 0 2024-03-26T18:54:33.380 INFO:tasks.workunit.client.1.smithi190.stdout:6/489: dread - d17/d28/d2c/f47 zero size 2024-03-26T18:54:33.383 INFO:tasks.workunit.client.0.smithi102.stdout:8/244: dread d0/d15/d1b/f35 [0,4194304] 0 2024-03-26T18:54:33.383 INFO:tasks.workunit.client.1.smithi190.stdout:7/625: write d3/d6/d9/d1f/d41/d50/d53/d87/fb9 [272135,51580] 0 2024-03-26T18:54:33.383 INFO:tasks.workunit.client.1.smithi190.stdout:7/626: chown d3/d6/d9/d1f/dab 27540 1 2024-03-26T18:54:33.383 INFO:tasks.workunit.client.1.smithi190.stdout:7/627: write d3/d6/d9/d1f/d41/f44 [1526185,125176] 0 2024-03-26T18:54:33.384 INFO:tasks.workunit.client.0.smithi102.stdout:8/245: chown d0/d3 7590696 1 2024-03-26T18:54:33.385 INFO:tasks.workunit.client.1.smithi190.stdout:9/831: dread d12/f28 [0,4194304] 0 2024-03-26T18:54:33.386 INFO:tasks.workunit.client.1.smithi190.stdout:9/832: stat d12/d2d/d1d/d23/f33 0 2024-03-26T18:54:33.386 INFO:tasks.workunit.client.1.smithi190.stdout:9/833: write d12/d2d/d1d/da1/dba/dd2/f111 [863819,41393] 0 2024-03-26T18:54:33.386 INFO:tasks.workunit.client.1.smithi190.stdout:9/834: creat d12/d2d/d1d/d23/d10d/f125 x:0 0 0 2024-03-26T18:54:33.386 INFO:tasks.workunit.client.1.smithi190.stdout:9/835: creat d12/d2d/d1d/da1/dba/f126 x:0 0 0 2024-03-26T18:54:33.387 INFO:tasks.workunit.client.0.smithi102.stdout:8/246: creat d0/f4d x:0 0 0 2024-03-26T18:54:33.389 INFO:tasks.workunit.client.1.smithi190.stdout:3/466: write d1/d3/df/d31/d5b/f6b [126702,44862] 0 2024-03-26T18:54:33.389 INFO:tasks.workunit.client.1.smithi190.stdout:3/467: creat d1/d3/d52/f90 x:0 0 0 2024-03-26T18:54:33.395 INFO:tasks.workunit.client.0.smithi102.stdout:8/247: mkdir d0/d3/d2c/d4e 0 2024-03-26T18:54:33.397 INFO:tasks.workunit.client.1.smithi190.stdout:8/593: mknod d9/da2/d73/cca 0 2024-03-26T18:54:33.399 INFO:tasks.workunit.client.1.smithi190.stdout:0/644: link d0/dd/d24/f2f d0/dd/d24/d87/d93/de1/fea 0 2024-03-26T18:54:33.399 INFO:tasks.workunit.client.1.smithi190.stdout:0/645: chown d0/dd/d24/d4b/d5e/d6b/l84 4559902 1 2024-03-26T18:54:33.400 INFO:tasks.workunit.client.1.smithi190.stdout:6/490: dread f1 [0,4194304] 0 2024-03-26T18:54:33.401 INFO:tasks.workunit.client.0.smithi102.stdout:8/248: rename d0/d15/d1b/f35 to d0/d3/d31/f4f 0 2024-03-26T18:54:33.405 INFO:tasks.workunit.client.1.smithi190.stdout:4/562: mknod d7/d1d/d42/cba 0 2024-03-26T18:54:33.406 INFO:tasks.workunit.client.1.smithi190.stdout:3/468: dread d1/fb [0,4194304] 0 2024-03-26T18:54:33.406 INFO:tasks.workunit.client.1.smithi190.stdout:8/594: dread d9/d39/d79/f7e [0,4194304] 0 2024-03-26T18:54:33.406 INFO:tasks.workunit.client.0.smithi102.stdout:8/249: unlink d0/d3/d26/l2e 0 2024-03-26T18:54:33.406 INFO:tasks.workunit.client.0.smithi102.stdout:8/250: dread - d0/d9/f2b zero size 2024-03-26T18:54:33.407 INFO:tasks.workunit.client.1.smithi190.stdout:7/628: truncate d3/fe 4248655 0 2024-03-26T18:54:33.412 INFO:tasks.workunit.client.0.smithi102.stdout:8/251: symlink d0/d3/d26/d19/l50 0 2024-03-26T18:54:33.412 INFO:tasks.workunit.client.0.smithi102.stdout:8/252: chown d0/d3/d2c/d33/f49 235 1 2024-03-26T18:54:33.412 INFO:tasks.workunit.client.0.smithi102.stdout:8/253: stat d0/d9 0 2024-03-26T18:54:33.413 INFO:tasks.workunit.client.0.smithi102.stdout:0/256: dwrite d6/de/f1d [0,4194304] 0 2024-03-26T18:54:33.415 INFO:tasks.workunit.client.1.smithi190.stdout:9/836: creat d12/d2d/dfc/f127 x:0 0 0 2024-03-26T18:54:33.416 INFO:tasks.workunit.client.0.smithi102.stdout:8/254: rename d0/d3/d2c/d33/f44 to d0/d3/d26/d19/f51 0 2024-03-26T18:54:33.419 INFO:tasks.workunit.client.0.smithi102.stdout:9/167: dwrite df/d1a/f1c [0,4194304] 0 2024-03-26T18:54:33.419 INFO:tasks.workunit.client.0.smithi102.stdout:9/168: creat df/f3b x:0 0 0 2024-03-26T18:54:33.419 INFO:tasks.workunit.client.0.smithi102.stdout:9/169: readlink df/d1a/d1d/l37 0 2024-03-26T18:54:33.419 INFO:tasks.workunit.client.0.smithi102.stdout:9/170: chown cd 58057720 1 2024-03-26T18:54:33.420 INFO:tasks.workunit.client.0.smithi102.stdout:0/257: creat d6/de/d13/d1f/d39/f52 x:0 0 0 2024-03-26T18:54:33.421 INFO:tasks.workunit.client.1.smithi190.stdout:2/555: rename d6/d9/d1a to d6/d9/d3a/d3d/d1c/d8d/dc7 0 2024-03-26T18:54:33.423 INFO:tasks.workunit.client.0.smithi102.stdout:8/255: link d0/d3/d26/f22 d0/d3/d2c/d4e/f52 0 2024-03-26T18:54:33.423 INFO:tasks.workunit.client.0.smithi102.stdout:8/256: dread - d0/d3c/f43 zero size 2024-03-26T18:54:33.424 INFO:tasks.workunit.client.1.smithi190.stdout:6/491: mkdir d17/d28/d80/da8 0 2024-03-26T18:54:33.426 INFO:tasks.workunit.client.1.smithi190.stdout:3/469: rmdir d1/de/d8a 39 2024-03-26T18:54:33.428 INFO:tasks.workunit.client.1.smithi190.stdout:8/595: mknod d9/dd/d15/ccb 0 2024-03-26T18:54:33.429 INFO:tasks.workunit.client.0.smithi102.stdout:9/171: symlink df/d1a/l3c 0 2024-03-26T18:54:33.430 INFO:tasks.workunit.client.1.smithi190.stdout:2/556: dread d6/d9/d3a/d3d/d1c/d8d/dc7/f26 [0,4194304] 0 2024-03-26T18:54:33.432 INFO:tasks.workunit.client.1.smithi190.stdout:7/629: mknod d3/d6/d9/d1f/d4c/d71/ce3 0 2024-03-26T18:54:33.433 INFO:tasks.workunit.client.0.smithi102.stdout:0/258: symlink d6/l53 0 2024-03-26T18:54:33.433 INFO:tasks.workunit.client.0.smithi102.stdout:0/259: fdatasync d6/de/d13/d25/d31/f4e 0 2024-03-26T18:54:33.434 INFO:tasks.workunit.client.1.smithi190.stdout:9/837: rmdir d12/d2d/d1d/d23/d83 39 2024-03-26T18:54:33.434 INFO:tasks.workunit.client.1.smithi190.stdout:9/838: chown d12/d2d/d1d/d23/d83/dc9/dd4/d71/da7 1115664 1 2024-03-26T18:54:33.436 INFO:tasks.workunit.client.0.smithi102.stdout:8/257: read d0/d3/f28 [2938776,124351] 0 2024-03-26T18:54:33.436 INFO:tasks.workunit.client.0.smithi102.stdout:6/252: dwrite d2/f21 [0,4194304] 0 2024-03-26T18:54:33.437 INFO:tasks.workunit.client.0.smithi102.stdout:6/253: creat d2/d7/f46 x:0 0 0 2024-03-26T18:54:33.437 INFO:tasks.workunit.client.0.smithi102.stdout:6/254: creat d2/f47 x:0 0 0 2024-03-26T18:54:33.438 INFO:tasks.workunit.client.1.smithi190.stdout:0/646: dwrite d0/dd/d24/d4b/fd3 [0,4194304] 0 2024-03-26T18:54:33.438 INFO:tasks.workunit.client.1.smithi190.stdout:4/563: dwrite d7/d8/d14/d6e/d6f/f7e [0,4194304] 0 2024-03-26T18:54:33.438 INFO:tasks.workunit.client.1.smithi190.stdout:1/495: dwrite d3/da/f24 [0,4194304] 0 2024-03-26T18:54:33.438 INFO:tasks.workunit.client.1.smithi190.stdout:1/496: stat d3/da/dd/d2f/d5b/d5d/c77 0 2024-03-26T18:54:33.438 INFO:tasks.workunit.client.1.smithi190.stdout:1/497: chown d3/da/d3a 343210 1 2024-03-26T18:54:33.438 INFO:tasks.workunit.client.1.smithi190.stdout:5/462: dwrite d2/d17/d1b/d22/d96/fa4 [0,4194304] 0 2024-03-26T18:54:33.438 INFO:tasks.workunit.client.1.smithi190.stdout:5/463: write d2/d17/f21 [3475562,62110] 0 2024-03-26T18:54:33.438 INFO:tasks.workunit.client.1.smithi190.stdout:5/464: fsync d2/d17/d1b/d60/d63/fa3 0 2024-03-26T18:54:33.438 INFO:tasks.workunit.client.1.smithi190.stdout:6/492: creat d17/d28/d80/fa9 x:0 0 0 2024-03-26T18:54:33.439 INFO:tasks.workunit.client.0.smithi102.stdout:9/172: symlink df/d1a/d1d/l3d 0 2024-03-26T18:54:33.439 INFO:tasks.workunit.client.0.smithi102.stdout:9/173: chown c3 32010 1 2024-03-26T18:54:33.441 INFO:tasks.workunit.client.1.smithi190.stdout:8/596: rename d9/da2/dc4/c3d to d9/d39/d9e/ccc 0 2024-03-26T18:54:33.445 INFO:tasks.workunit.client.0.smithi102.stdout:8/258: creat d0/d3/d2c/d2f/f53 x:0 0 0 2024-03-26T18:54:33.446 INFO:tasks.workunit.client.0.smithi102.stdout:8/259: write d0/d3/d2c/f42 [611592,128509] 0 2024-03-26T18:54:33.446 INFO:tasks.workunit.client.0.smithi102.stdout:8/260: chown d0/c5 7763315 1 2024-03-26T18:54:33.447 INFO:tasks.workunit.client.1.smithi190.stdout:9/839: dread d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/db3/fc1 [0,4194304] 0 2024-03-26T18:54:33.449 INFO:tasks.workunit.client.1.smithi190.stdout:7/630: write d3/d6/d9/d14/d59/f61 [3409131,48654] 0 2024-03-26T18:54:33.449 INFO:tasks.workunit.client.1.smithi190.stdout:7/631: write d3/d6/f91 [871150,98126] 0 2024-03-26T18:54:33.449 INFO:tasks.workunit.client.1.smithi190.stdout:7/632: chown d3/d6/d9/f75 27305 1 2024-03-26T18:54:33.451 INFO:tasks.workunit.client.0.smithi102.stdout:9/174: write df/d1a/f20 [3273504,116127] 0 2024-03-26T18:54:33.455 INFO:tasks.workunit.client.1.smithi190.stdout:6/493: dread d17/d28/d2c/f41 [4194304,4194304] 0 2024-03-26T18:54:33.460 INFO:tasks.workunit.client.1.smithi190.stdout:0/647: mknod d0/dd/d24/da2/d80/ceb 0 2024-03-26T18:54:33.460 INFO:tasks.workunit.client.1.smithi190.stdout:0/648: chown d0/dd/d20/d1c/d8b/d8d/dc8/fce 60 1 2024-03-26T18:54:33.462 INFO:tasks.workunit.client.1.smithi190.stdout:4/564: write d7/d8/d14/d5c/f83 [1645710,119492] 0 2024-03-26T18:54:33.462 INFO:tasks.workunit.client.1.smithi190.stdout:3/470: dwrite d1/de/d4e/f8f [0,4194304] 0 2024-03-26T18:54:33.465 INFO:tasks.workunit.client.1.smithi190.stdout:1/498: mknod d3/da/dd/d95/ca4 0 2024-03-26T18:54:33.465 INFO:tasks.workunit.client.1.smithi190.stdout:1/499: fsync d3/f29 0 2024-03-26T18:54:33.465 INFO:tasks.workunit.client.1.smithi190.stdout:1/500: stat d3/d73/l84 0 2024-03-26T18:54:33.465 INFO:tasks.workunit.client.1.smithi190.stdout:1/501: fdatasync d3/da/dd/f1b 0 2024-03-26T18:54:33.465 INFO:tasks.workunit.client.1.smithi190.stdout:1/502: truncate d3/d9/db/f23 1560342 0 2024-03-26T18:54:33.466 INFO:tasks.workunit.client.1.smithi190.stdout:2/557: dwrite d6/d9/d10/d5f/f7f [0,4194304] 0 2024-03-26T18:54:33.467 INFO:tasks.workunit.client.1.smithi190.stdout:5/465: unlink d2/d17/d98/d3a/d3c/l90 0 2024-03-26T18:54:33.468 INFO:tasks.workunit.client.1.smithi190.stdout:7/633: mknod d3/d6/d9/d1f/d4c/d71/dcc/ce4 0 2024-03-26T18:54:33.470 INFO:tasks.workunit.client.1.smithi190.stdout:0/649: write d0/dd/d20/d1c/d46/f72 [3326539,45245] 0 2024-03-26T18:54:33.473 INFO:tasks.workunit.client.1.smithi190.stdout:6/494: rename d17/d28/d46/d6a/c7d to d17/d28/d46/caa 0 2024-03-26T18:54:33.474 INFO:tasks.workunit.client.0.smithi102.stdout:0/260: rmdir d6 39 2024-03-26T18:54:33.475 INFO:tasks.workunit.client.0.smithi102.stdout:0/261: write d6/de/f4b [564580,53204] 0 2024-03-26T18:54:33.476 INFO:tasks.workunit.client.1.smithi190.stdout:4/565: symlink d7/d1d/d5e/d98/lbb 0 2024-03-26T18:54:33.476 INFO:tasks.workunit.client.1.smithi190.stdout:4/566: fsync d7/d1d/f26 0 2024-03-26T18:54:33.476 INFO:tasks.workunit.client.1.smithi190.stdout:2/558: write d6/d9/f38 [2769197,35949] 0 2024-03-26T18:54:33.479 INFO:tasks.workunit.client.1.smithi190.stdout:8/597: dwrite d9/d48/d49/fbe [0,4194304] 0 2024-03-26T18:54:33.479 INFO:tasks.workunit.client.1.smithi190.stdout:8/598: write d9/d39/d67/f95 [1185858,110276] 0 2024-03-26T18:54:33.479 INFO:tasks.workunit.client.1.smithi190.stdout:9/840: dwrite d12/d2d/d1d/f3b [0,4194304] 0 2024-03-26T18:54:33.479 INFO:tasks.workunit.client.1.smithi190.stdout:9/841: write d12/d2d/d1d/d23/d83/dc9/dd4/d3f/fbb [1094715,37960] 0 2024-03-26T18:54:33.480 INFO:tasks.workunit.client.1.smithi190.stdout:1/503: mkdir d3/da/d3a/da5 0 2024-03-26T18:54:33.484 INFO:tasks.workunit.client.1.smithi190.stdout:5/466: rename d2/d17/l1c to d2/d17/la8 0 2024-03-26T18:54:33.487 INFO:tasks.workunit.client.1.smithi190.stdout:7/634: symlink d3/d6/d9/le5 0 2024-03-26T18:54:33.488 INFO:tasks.workunit.client.1.smithi190.stdout:0/650: mknod d0/dd/d20/d61/cec 0 2024-03-26T18:54:33.488 INFO:tasks.workunit.client.1.smithi190.stdout:9/842: dread d12/d2d/d1d/d23/d83/dc9/dd4/d71/fa6 [0,4194304] 0 2024-03-26T18:54:33.488 INFO:tasks.workunit.client.1.smithi190.stdout:9/843: write d12/d2d/d1d/fd3 [1778005,50568] 0 2024-03-26T18:54:33.489 INFO:tasks.workunit.client.0.smithi102.stdout:9/175: mkdir df/d1a/d23/d3e 0 2024-03-26T18:54:33.491 INFO:tasks.workunit.client.0.smithi102.stdout:6/255: truncate d2/f21 2701519 0 2024-03-26T18:54:33.491 INFO:tasks.workunit.client.0.smithi102.stdout:6/256: creat d2/d7/d3c/f48 x:0 0 0 2024-03-26T18:54:33.491 INFO:tasks.workunit.client.0.smithi102.stdout:6/257: fdatasync d2/d7/f8 0 2024-03-26T18:54:33.492 INFO:tasks.workunit.client.1.smithi190.stdout:4/567: mknod d7/d8/d1c/d20/cbc 0 2024-03-26T18:54:33.492 INFO:tasks.workunit.client.1.smithi190.stdout:4/568: chown d7/d1d/fb5 1 1 2024-03-26T18:54:33.492 INFO:tasks.workunit.client.1.smithi190.stdout:4/569: readlink d7/d8/d14/d5c/l7b 0 2024-03-26T18:54:33.494 INFO:tasks.workunit.client.1.smithi190.stdout:5/467: dread d2/f15 [0,4194304] 0 2024-03-26T18:54:33.495 INFO:tasks.workunit.client.0.smithi102.stdout:0/262: unlink d6/de/d13/c17 0 2024-03-26T18:54:33.496 INFO:tasks.workunit.client.1.smithi190.stdout:2/559: symlink d6/d9/d3a/d3d/d1c/d8d/dc7/d72/lc8 0 2024-03-26T18:54:33.497 INFO:tasks.workunit.client.1.smithi190.stdout:9/844: dread d12/d2d/f22 [0,4194304] 0 2024-03-26T18:54:33.497 INFO:tasks.workunit.client.1.smithi190.stdout:9/845: creat d12/d6d/f128 x:0 0 0 2024-03-26T18:54:33.506 INFO:tasks.workunit.client.0.smithi102.stdout:8/261: getdents d0/d3/d2c 0 2024-03-26T18:54:33.506 INFO:tasks.workunit.client.0.smithi102.stdout:8/262: write d0/d3/d26/f22 [4857121,32039] 0 2024-03-26T18:54:33.506 INFO:tasks.workunit.client.1.smithi190.stdout:8/599: link d9/dd/d15/lbf d9/dd/d23/d36/d37/d3a/d43/dc2/lcd 0 2024-03-26T18:54:33.506 INFO:tasks.workunit.client.1.smithi190.stdout:5/468: write d2/d17/d98/f5f [2831354,73028] 0 2024-03-26T18:54:33.508 INFO:tasks.workunit.client.1.smithi190.stdout:1/504: creat d3/da/dd/d76/fa6 x:0 0 0 2024-03-26T18:54:33.511 INFO:tasks.workunit.client.1.smithi190.stdout:7/635: creat d3/d6/d9/d1f/d4c/fe6 x:0 0 0 2024-03-26T18:54:33.511 INFO:tasks.workunit.client.0.smithi102.stdout:8/263: getdents d0/d15 0 2024-03-26T18:54:33.511 INFO:tasks.workunit.client.0.smithi102.stdout:8/264: chown d0/d3/d26/d19/d27/l30 14310 1 2024-03-26T18:54:33.511 INFO:tasks.workunit.client.0.smithi102.stdout:8/265: readlink d0/d3/d26/d19/l1a 0 2024-03-26T18:54:33.512 INFO:tasks.workunit.client.0.smithi102.stdout:9/176: rmdir df/d1a/d1d 39 2024-03-26T18:54:33.512 INFO:tasks.workunit.client.0.smithi102.stdout:9/177: truncate df/d1a/d23/f2c 371910 0 2024-03-26T18:54:33.512 INFO:tasks.workunit.client.0.smithi102.stdout:9/178: creat df/d1a/f3f x:0 0 0 2024-03-26T18:54:33.513 INFO:tasks.workunit.client.1.smithi190.stdout:0/651: dwrite d0/dd/d24/d4b/d5e/fdb [0,4194304] 0 2024-03-26T18:54:33.515 INFO:tasks.workunit.client.1.smithi190.stdout:3/471: rename d1/d3/d52/d7d to d1/d3/df/d18/d57/d91 0 2024-03-26T18:54:33.518 INFO:tasks.workunit.client.0.smithi102.stdout:8/266: link d0/d3/d31/f4f d0/d3/d2c/d33/f54 0 2024-03-26T18:54:33.518 INFO:tasks.workunit.client.0.smithi102.stdout:8/267: write d0/d9/f2b [25085,109526] 0 2024-03-26T18:54:33.521 INFO:tasks.workunit.client.0.smithi102.stdout:6/258: read d2/f21 [2282821,106086] 0 2024-03-26T18:54:33.521 INFO:tasks.workunit.client.0.smithi102.stdout:6/259: readlink d2/d7/le 0 2024-03-26T18:54:33.524 INFO:tasks.workunit.client.0.smithi102.stdout:1/244: sync 2024-03-26T18:54:33.524 INFO:tasks.workunit.client.0.smithi102.stdout:3/380: sync 2024-03-26T18:54:33.524 INFO:tasks.workunit.client.0.smithi102.stdout:3/381: dread - df/d1c/d34/f74 zero size 2024-03-26T18:54:33.526 INFO:tasks.workunit.client.0.smithi102.stdout:5/448: sync 2024-03-26T18:54:33.527 INFO:tasks.workunit.client.0.smithi102.stdout:9/179: rename df/d1a/f20 to df/f40 0 2024-03-26T18:54:33.528 INFO:tasks.workunit.client.0.smithi102.stdout:8/268: symlink d0/l55 0 2024-03-26T18:54:33.529 INFO:tasks.workunit.client.0.smithi102.stdout:8/269: write d0/d3/d2c/f42 [457353,119151] 0 2024-03-26T18:54:33.530 INFO:tasks.workunit.client.1.smithi190.stdout:4/570: mknod d7/d8/d14/cbd 0 2024-03-26T18:54:33.530 INFO:tasks.workunit.client.1.smithi190.stdout:4/571: creat d7/d1d/d42/fbe x:0 0 0 2024-03-26T18:54:33.530 INFO:tasks.workunit.client.1.smithi190.stdout:4/572: write d7/d1d/fb5 [909629,7905] 0 2024-03-26T18:54:33.530 INFO:tasks.workunit.client.1.smithi190.stdout:1/505: mkdir d3/d8f/da7 0 2024-03-26T18:54:33.532 INFO:tasks.workunit.client.0.smithi102.stdout:6/260: link d2/d7/l1a d2/d12/l49 0 2024-03-26T18:54:33.532 INFO:tasks.workunit.client.0.smithi102.stdout:6/261: fdatasync d2/f47 0 2024-03-26T18:54:33.532 INFO:tasks.workunit.client.0.smithi102.stdout:6/262: stat d2/df/c39 0 2024-03-26T18:54:33.536 INFO:tasks.workunit.client.0.smithi102.stdout:3/382: mknod df/d61/c83 0 2024-03-26T18:54:33.541 INFO:tasks.workunit.client.0.smithi102.stdout:5/449: rename d2/d2c/d38/f87 to d2/d2c/d38/d43/d9e/d82/fa2 0 2024-03-26T18:54:33.541 INFO:tasks.workunit.client.0.smithi102.stdout:5/450: write d2/d14/d1a/f77 [547244,33226] 0 2024-03-26T18:54:33.542 INFO:tasks.workunit.client.0.smithi102.stdout:5/451: write d2/d2c/f4c [111798,56842] 0 2024-03-26T18:54:33.544 INFO:tasks.workunit.client.0.smithi102.stdout:9/180: symlink df/d1a/d23/d29/l41 0 2024-03-26T18:54:33.544 INFO:tasks.workunit.client.0.smithi102.stdout:9/181: dread - df/d1a/f24 zero size 2024-03-26T18:54:33.544 INFO:tasks.workunit.client.0.smithi102.stdout:9/182: creat df/f42 x:0 0 0 2024-03-26T18:54:33.544 INFO:tasks.workunit.client.0.smithi102.stdout:9/183: fdatasync df/d1a/d23/f25 0 2024-03-26T18:54:33.544 INFO:tasks.workunit.client.0.smithi102.stdout:9/184: chown df/f28 2584 1 2024-03-26T18:54:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:6/263: symlink d2/d7/l4a 0 2024-03-26T18:54:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:6/264: read - d2/f33 zero size 2024-03-26T18:54:33.550 INFO:tasks.workunit.client.0.smithi102.stdout:6/265: stat d2/df/l40 0 2024-03-26T18:54:33.551 INFO:tasks.workunit.client.1.smithi190.stdout:2/560: dwrite d6/d9/d3a/f83 [0,4194304] 0 2024-03-26T18:54:33.551 INFO:tasks.workunit.client.1.smithi190.stdout:5/469: dwrite d2/d17/d1b/d22/f2d [0,4194304] 0 2024-03-26T18:54:33.551 INFO:tasks.workunit.client.1.smithi190.stdout:2/561: chown d6/d9/d3a/l82 2722 1 2024-03-26T18:54:33.552 INFO:tasks.workunit.client.1.smithi190.stdout:8/600: dwrite d9/dd/d23/d36/d37/d3a/d61/fb6 [0,4194304] 0 2024-03-26T18:54:33.552 INFO:tasks.workunit.client.1.smithi190.stdout:0/652: mknod d0/dd/d20/d1c/d8b/d8d/db7/ced 0 2024-03-26T18:54:33.552 INFO:tasks.workunit.client.1.smithi190.stdout:0/653: write d0/dd/d24/d4b/d5e/d6b/faf [1069775,97172] 0 2024-03-26T18:54:33.553 INFO:tasks.workunit.client.1.smithi190.stdout:1/506: dwrite d3/da/dd/d2f/f7b [0,4194304] 0 2024-03-26T18:54:33.554 INFO:tasks.workunit.client.0.smithi102.stdout:3/383: mknod df/d29/d31/d36/d3c/d6f/d75/c84 0 2024-03-26T18:54:33.554 INFO:tasks.workunit.client.0.smithi102.stdout:3/384: creat df/d29/d3a/f85 x:0 0 0 2024-03-26T18:54:33.554 INFO:tasks.workunit.client.0.smithi102.stdout:3/385: truncate df/d29/d3a/f85 317881 0 2024-03-26T18:54:33.554 INFO:tasks.workunit.client.0.smithi102.stdout:3/386: chown df/d61/l72 57710383 1 2024-03-26T18:54:33.559 INFO:tasks.workunit.client.0.smithi102.stdout:9/185: mkdir df/d1a/d33/d43 0 2024-03-26T18:54:33.559 INFO:tasks.workunit.client.0.smithi102.stdout:9/186: fdatasync df/f40 0 2024-03-26T18:54:33.563 INFO:tasks.workunit.client.0.smithi102.stdout:8/270: link d0/d3/d26/d19/d27/c2a d0/d9/c56 0 2024-03-26T18:54:33.563 INFO:tasks.workunit.client.0.smithi102.stdout:8/271: chown d0/d3 51569 1 2024-03-26T18:54:33.564 INFO:tasks.workunit.client.0.smithi102.stdout:5/452: dwrite d2/d2c/d38/f5c [0,4194304] 0 2024-03-26T18:54:33.564 INFO:tasks.workunit.client.0.smithi102.stdout:5/453: dread - d2/d25/d2a/d4f/f9c zero size 2024-03-26T18:54:33.564 INFO:tasks.workunit.client.0.smithi102.stdout:5/454: creat d2/d25/d5b/d76/fa3 x:0 0 0 2024-03-26T18:54:33.564 INFO:tasks.workunit.client.0.smithi102.stdout:5/455: fdatasync d2/d25/d2a/d4f/f9c 0 2024-03-26T18:54:33.564 INFO:tasks.workunit.client.1.smithi190.stdout:6/495: rename d17/d28/d2c/d77/d51/c99 to d17/d28/d46/cab 0 2024-03-26T18:54:33.566 INFO:tasks.workunit.client.0.smithi102.stdout:6/266: link d2/d7/f46 d2/d3/f4b 0 2024-03-26T18:54:33.567 INFO:tasks.workunit.client.1.smithi190.stdout:5/470: write d2/d17/d1b/d22/f2d [734278,10513] 0 2024-03-26T18:54:33.575 INFO:tasks.workunit.client.0.smithi102.stdout:9/187: rename df/l11 to df/d1a/l44 0 2024-03-26T18:54:33.575 INFO:tasks.workunit.client.0.smithi102.stdout:6/267: rename d2 to d2/df/d35/d43/d4c 22 2024-03-26T18:54:33.575 INFO:tasks.workunit.client.0.smithi102.stdout:9/188: chown df 13 1 2024-03-26T18:54:33.575 INFO:tasks.workunit.client.1.smithi190.stdout:6/496: dread d17/d28/d2c/d50/f67 [0,4194304] 0 2024-03-26T18:54:33.575 INFO:tasks.workunit.client.1.smithi190.stdout:8/601: dwrite d9/dd/f99 [0,4194304] 0 2024-03-26T18:54:33.576 INFO:tasks.workunit.client.0.smithi102.stdout:5/456: dread d2/f11 [0,4194304] 0 2024-03-26T18:54:33.576 INFO:tasks.workunit.client.0.smithi102.stdout:5/457: dread - d2/d25/d2a/f9f zero size 2024-03-26T18:54:33.576 INFO:tasks.workunit.client.0.smithi102.stdout:5/458: creat d2/d25/d5b/fa4 x:0 0 0 2024-03-26T18:54:33.576 INFO:tasks.workunit.client.0.smithi102.stdout:5/459: write d2/d25/d5b/d76/fa3 [323131,82339] 0 2024-03-26T18:54:33.576 INFO:tasks.workunit.client.0.smithi102.stdout:8/272: mkdir d0/d15/d57 0 2024-03-26T18:54:33.581 INFO:tasks.workunit.client.1.smithi190.stdout:2/562: mknod d6/d9/d3a/cc9 0 2024-03-26T18:54:33.590 INFO:tasks.workunit.client.1.smithi190.stdout:0/654: mkdir d0/dd5/dee 0 2024-03-26T18:54:33.590 INFO:tasks.workunit.client.1.smithi190.stdout:6/497: dwrite d17/f23 [0,4194304] 0 2024-03-26T18:54:33.590 INFO:tasks.workunit.client.1.smithi190.stdout:6/498: creat d17/d28/d2c/d77/fac x:0 0 0 2024-03-26T18:54:33.590 INFO:tasks.workunit.client.1.smithi190.stdout:6/499: chown d17/d28/d80/l85 1610812 1 2024-03-26T18:54:33.591 INFO:tasks.workunit.client.1.smithi190.stdout:1/507: rmdir d3/da/d2e/d59 39 2024-03-26T18:54:33.591 INFO:tasks.workunit.client.1.smithi190.stdout:1/508: readlink d3/l36 0 2024-03-26T18:54:33.594 INFO:tasks.workunit.client.1.smithi190.stdout:7/636: rename d3/d6/d9/d14/d59/fa5 to d3/d6/d9/d1f/d4c/fe7 0 2024-03-26T18:54:33.595 INFO:tasks.workunit.client.1.smithi190.stdout:5/471: mknod d2/d17/d1b/d22/d24/d3b/da5/ca9 0 2024-03-26T18:54:33.595 INFO:tasks.workunit.client.1.smithi190.stdout:5/472: chown d2/d17/d98/l6c 26 1 2024-03-26T18:54:33.601 INFO:tasks.workunit.client.1.smithi190.stdout:8/602: symlink d9/d39/lce 0 2024-03-26T18:54:33.601 INFO:tasks.workunit.client.1.smithi190.stdout:8/603: chown d9/d39/d9e/fc5 492436921 1 2024-03-26T18:54:33.601 INFO:tasks.workunit.client.1.smithi190.stdout:8/604: write d9/dd/d23/d36/d37/d3a/d61/f91 [94403,1687] 0 2024-03-26T18:54:33.601 INFO:tasks.workunit.client.0.smithi102.stdout:3/387: rmdir df 39 2024-03-26T18:54:33.602 INFO:tasks.workunit.client.1.smithi190.stdout:2/563: rmdir d6/d9/d3a/d3d/d1c/d8d/dc7/d72 39 2024-03-26T18:54:33.604 INFO:tasks.workunit.client.0.smithi102.stdout:9/189: mkdir df/d1a/d23/d29/d45 0 2024-03-26T18:54:33.606 INFO:tasks.workunit.client.0.smithi102.stdout:6/268: mknod d2/d3d/c4d 0 2024-03-26T18:54:33.606 INFO:tasks.workunit.client.0.smithi102.stdout:6/269: readlink d2/d12/l49 0 2024-03-26T18:54:33.607 INFO:tasks.workunit.client.1.smithi190.stdout:6/500: dwrite d17/d28/d2c/d63/f70 [0,4194304] 0 2024-03-26T18:54:33.612 INFO:tasks.workunit.client.1.smithi190.stdout:4/573: rename d7/d8/d14/f57 to d7/d8/d14/d4b/db2/fbf 0 2024-03-26T18:54:33.615 INFO:tasks.workunit.client.1.smithi190.stdout:7/637: link d3/d6/d9/d1f/d4c/f7f d3/d6/d9/d14/fe8 0 2024-03-26T18:54:33.617 INFO:tasks.workunit.client.1.smithi190.stdout:5/473: creat d2/d17/d98/d3a/faa x:0 0 0 2024-03-26T18:54:33.617 INFO:tasks.workunit.client.1.smithi190.stdout:5/474: write d2/f93 [622989,56407] 0 2024-03-26T18:54:33.618 INFO:tasks.workunit.client.0.smithi102.stdout:3/388: rmdir df/d1c 39 2024-03-26T18:54:33.620 INFO:tasks.workunit.client.1.smithi190.stdout:2/564: symlink d6/d9/d10/db7/lca 0 2024-03-26T18:54:33.623 INFO:tasks.workunit.client.0.smithi102.stdout:9/190: unlink df/f28 0 2024-03-26T18:54:33.623 INFO:tasks.workunit.client.0.smithi102.stdout:9/191: chown df/c19 97648725 1 2024-03-26T18:54:33.625 INFO:tasks.workunit.client.0.smithi102.stdout:6/270: mknod d2/df/d35/c4e 0 2024-03-26T18:54:33.626 INFO:tasks.workunit.client.1.smithi190.stdout:7/638: write d3/fc [1024791,36539] 0 2024-03-26T18:54:33.628 INFO:tasks.workunit.client.0.smithi102.stdout:5/460: symlink d2/d14/la5 0 2024-03-26T18:54:33.629 INFO:tasks.workunit.client.0.smithi102.stdout:8/273: getdents d0/d3/d31 0 2024-03-26T18:54:33.630 INFO:tasks.workunit.client.1.smithi190.stdout:4/574: symlink d7/d8/d14/d5c/lc0 0 2024-03-26T18:54:33.632 INFO:tasks.workunit.client.0.smithi102.stdout:6/271: link d2/df/l18 d2/df/d35/l4f 0 2024-03-26T18:54:33.632 INFO:tasks.workunit.client.0.smithi102.stdout:3/389: write df/d29/f33 [3216002,5347] 0 2024-03-26T18:54:33.634 INFO:tasks.workunit.client.1.smithi190.stdout:2/565: read d6/f7a [252831,120218] 0 2024-03-26T18:54:33.634 INFO:tasks.workunit.client.1.smithi190.stdout:2/566: dread - d6/d9/d3a/d3d/d55/fc6 zero size 2024-03-26T18:54:33.634 INFO:tasks.workunit.client.1.smithi190.stdout:2/567: fsync d6/d9/d3a/f83 0 2024-03-26T18:54:33.634 INFO:tasks.workunit.client.1.smithi190.stdout:2/568: chown d6/d21/d61/fc2 26 1 2024-03-26T18:54:33.634 INFO:tasks.workunit.client.1.smithi190.stdout:2/569: write d6/d9/f38 [1637777,100046] 0 2024-03-26T18:54:33.636 INFO:tasks.workunit.client.0.smithi102.stdout:8/274: rename d0/d15/d57 to d0/d3/d2c/d58 0 2024-03-26T18:54:33.636 INFO:tasks.workunit.client.0.smithi102.stdout:8/275: fdatasync d0/d3/d2c/f42 0 2024-03-26T18:54:33.638 INFO:tasks.workunit.client.1.smithi190.stdout:7/639: rename d3/d6/d9/d1f/d41/d50/d53 to d3/d6/d9/d1f/d41/d65/d90/d92/d9b/de9 0 2024-03-26T18:54:33.640 INFO:tasks.workunit.client.1.smithi190.stdout:4/575: unlink d7/d1d/fb3 0 2024-03-26T18:54:33.640 INFO:tasks.workunit.client.1.smithi190.stdout:4/576: truncate d7/d1d/f3c 4707517 0 2024-03-26T18:54:33.641 INFO:tasks.workunit.client.1.smithi190.stdout:2/570: creat d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d7c/fcb x:0 0 0 2024-03-26T18:54:33.641 INFO:tasks.workunit.client.1.smithi190.stdout:2/571: dread - d6/d9/d3a/d3d/d55/fc6 zero size 2024-03-26T18:54:33.641 INFO:tasks.workunit.client.1.smithi190.stdout:2/572: creat d6/d21/d61/d6c/fcc x:0 0 0 2024-03-26T18:54:33.641 INFO:tasks.workunit.client.0.smithi102.stdout:8/276: creat d0/d3/d31/f59 x:0 0 0 2024-03-26T18:54:33.642 INFO:tasks.workunit.client.1.smithi190.stdout:7/640: creat d3/d1b/d57/dd1/fea x:0 0 0 2024-03-26T18:54:33.642 INFO:tasks.workunit.client.0.smithi102.stdout:3/390: mknod df/c86 0 2024-03-26T18:54:33.643 INFO:tasks.workunit.client.1.smithi190.stdout:4/577: symlink d7/d1d/lc1 0 2024-03-26T18:54:33.645 INFO:tasks.workunit.client.1.smithi190.stdout:6/501: dwrite d17/d28/d80/fa9 [0,4194304] 0 2024-03-26T18:54:33.645 INFO:tasks.workunit.client.1.smithi190.stdout:8/605: dwrite d9/dd/d15/f60 [0,4194304] 0 2024-03-26T18:54:33.645 INFO:tasks.workunit.client.1.smithi190.stdout:8/606: dread - d9/dd/d23/d36/d37/d3a/d61/fbd zero size 2024-03-26T18:54:33.645 INFO:tasks.workunit.client.1.smithi190.stdout:8/607: chown d9/d39/cb2 27730 1 2024-03-26T18:54:33.645 INFO:tasks.workunit.client.1.smithi190.stdout:8/608: write d9/fc8 [361378,46084] 0 2024-03-26T18:54:33.645 INFO:tasks.workunit.client.1.smithi190.stdout:5/475: dwrite d2/d17/d1b/d22/f77 [0,4194304] 0 2024-03-26T18:54:33.646 INFO:tasks.workunit.client.0.smithi102.stdout:9/192: dwrite f7 [0,4194304] 0 2024-03-26T18:54:33.646 INFO:tasks.workunit.client.0.smithi102.stdout:8/277: rename d0/d3/d26/c1d to d0/d15/d1b/c5a 0 2024-03-26T18:54:33.646 INFO:tasks.workunit.client.0.smithi102.stdout:8/278: dread - d0/d3/d31/f59 zero size 2024-03-26T18:54:33.646 INFO:tasks.workunit.client.0.smithi102.stdout:8/279: creat d0/d3/d26/d19/d27/f5b x:0 0 0 2024-03-26T18:54:33.646 INFO:tasks.workunit.client.0.smithi102.stdout:3/391: rmdir df/d1c/d34/d53 39 2024-03-26T18:54:33.648 INFO:tasks.workunit.client.1.smithi190.stdout:2/573: mknod d6/d9/d45/d6b/d93/dc0/ccd 0 2024-03-26T18:54:33.651 INFO:tasks.workunit.client.1.smithi190.stdout:7/641: rename d3/d6/d9/d1f/d84 to d3/d6/d9/d1f/d41/d65/d90/d92/d9b/deb 0 2024-03-26T18:54:33.651 INFO:tasks.workunit.client.1.smithi190.stdout:7/642: stat d3/f2b 0 2024-03-26T18:54:33.651 INFO:tasks.workunit.client.1.smithi190.stdout:7/643: stat d3/d6/d9/d1f/dab 0 2024-03-26T18:54:33.653 INFO:tasks.workunit.client.1.smithi190.stdout:4/578: unlink d7/d8/d14/d23/d28/c5a 0 2024-03-26T18:54:33.654 INFO:tasks.workunit.client.0.smithi102.stdout:9/193: getdents df/d1a/d33 0 2024-03-26T18:54:33.658 INFO:tasks.workunit.client.0.smithi102.stdout:5/461: getdents d2/d14 0 2024-03-26T18:54:33.659 INFO:tasks.workunit.client.1.smithi190.stdout:5/476: dwrite d2/f16 [0,4194304] 0 2024-03-26T18:54:33.661 INFO:tasks.workunit.client.0.smithi102.stdout:3/392: creat df/d1c/d70/f87 x:0 0 0 2024-03-26T18:54:33.661 INFO:tasks.workunit.client.0.smithi102.stdout:3/393: truncate df/d29/d31/d36/f41 4833984 0 2024-03-26T18:54:33.666 INFO:tasks.workunit.client.1.smithi190.stdout:6/502: truncate d17/d28/d2c/d50/f8f 544333 0 2024-03-26T18:54:33.669 INFO:tasks.workunit.client.1.smithi190.stdout:8/609: symlink d9/da2/d87/lcf 0 2024-03-26T18:54:33.671 INFO:tasks.workunit.client.1.smithi190.stdout:4/579: write d7/d8/f2a [3268961,59231] 0 2024-03-26T18:54:33.671 INFO:tasks.workunit.client.0.smithi102.stdout:9/194: mkdir df/d46 0 2024-03-26T18:54:33.673 INFO:tasks.workunit.client.0.smithi102.stdout:8/280: mkdir d0/d3/d26/d5c 0 2024-03-26T18:54:33.673 INFO:tasks.workunit.client.0.smithi102.stdout:8/281: stat d0/d3/d31/f3a 0 2024-03-26T18:54:33.679 INFO:tasks.workunit.client.0.smithi102.stdout:5/462: dwrite d2/d25/d5b/d72/f57 [0,4194304] 0 2024-03-26T18:54:33.679 INFO:tasks.workunit.client.0.smithi102.stdout:9/195: dread df/f40 [0,4194304] 0 2024-03-26T18:54:33.685 INFO:tasks.workunit.client.0.smithi102.stdout:9/196: mknod df/d1a/d33/d43/c47 0 2024-03-26T18:54:33.685 INFO:tasks.workunit.client.0.smithi102.stdout:9/197: dread - df/d1a/d23/d29/f34 zero size 2024-03-26T18:54:33.686 INFO:tasks.workunit.client.0.smithi102.stdout:5/463: symlink d2/d2c/d38/d43/la6 0 2024-03-26T18:54:33.686 INFO:tasks.workunit.client.0.smithi102.stdout:5/464: read - d2/d25/d2a/d4f/f96 zero size 2024-03-26T18:54:33.687 INFO:tasks.workunit.client.0.smithi102.stdout:8/282: rmdir d0/d15/d1b 39 2024-03-26T18:54:33.687 INFO:tasks.workunit.client.0.smithi102.stdout:8/283: stat d0/d9/f24 0 2024-03-26T18:54:33.689 INFO:tasks.workunit.client.0.smithi102.stdout:9/198: link df/f3b df/d1a/d23/f48 0 2024-03-26T18:54:33.689 INFO:tasks.workunit.client.0.smithi102.stdout:2/212: sync 2024-03-26T18:54:33.689 INFO:tasks.workunit.client.0.smithi102.stdout:2/213: write d3/d11/d15/f48 [170132,118929] 0 2024-03-26T18:54:33.690 INFO:tasks.workunit.client.1.smithi190.stdout:2/574: link d6/d9/d3a/d3d/d55/caa d6/d9/d10/d5f/cce 0 2024-03-26T18:54:33.691 INFO:tasks.workunit.client.0.smithi102.stdout:3/394: dwrite df/d29/d31/d36/f41 [0,4194304] 0 2024-03-26T18:54:33.691 INFO:tasks.workunit.client.1.smithi190.stdout:6/503: dwrite d17/d28/d2c/d77/f6f [0,4194304] 0 2024-03-26T18:54:33.691 INFO:tasks.workunit.client.0.smithi102.stdout:5/465: mknod d2/d25/d5b/d76/ca7 0 2024-03-26T18:54:33.693 INFO:tasks.workunit.client.1.smithi190.stdout:7/644: mknod d3/d1b/d57/cec 0 2024-03-26T18:54:33.693 INFO:tasks.workunit.client.1.smithi190.stdout:7/645: write d3/d6/d9/d1f/d4c/fe6 [941304,106527] 0 2024-03-26T18:54:33.694 INFO:tasks.workunit.client.0.smithi102.stdout:8/284: mkdir d0/d15/d5d 0 2024-03-26T18:54:33.695 INFO:tasks.workunit.client.1.smithi190.stdout:5/477: getdents d2/d17/d62 0 2024-03-26T18:54:33.696 INFO:tasks.workunit.client.1.smithi190.stdout:5/478: truncate d2/d17/d62/f82 257510 0 2024-03-26T18:54:33.696 INFO:tasks.workunit.client.1.smithi190.stdout:5/479: write d2/d17/d1b/d22/f38 [104075,66883] 0 2024-03-26T18:54:33.696 INFO:tasks.workunit.client.1.smithi190.stdout:5/480: read d2/d17/d1b/d22/f2d [2605339,124653] 0 2024-03-26T18:54:33.697 INFO:tasks.workunit.client.0.smithi102.stdout:9/199: symlink df/d46/l49 0 2024-03-26T18:54:33.699 INFO:tasks.workunit.client.0.smithi102.stdout:2/214: unlink d3/d6/d1a/d25/f3e 0 2024-03-26T18:54:33.701 INFO:tasks.workunit.client.0.smithi102.stdout:3/395: creat df/d29/d31/d36/d3c/d6f/d75/f88 x:0 0 0 2024-03-26T18:54:33.702 INFO:tasks.workunit.client.0.smithi102.stdout:5/466: rename d2/d2c/l63 to d2/la8 0 2024-03-26T18:54:33.703 INFO:tasks.workunit.client.0.smithi102.stdout:5/467: stat d2/f3a 0 2024-03-26T18:54:33.704 INFO:tasks.workunit.client.1.smithi190.stdout:8/610: mknod d9/dd/d23/d36/d37/d3a/d61/daf/db3/cd0 0 2024-03-26T18:54:33.704 INFO:tasks.workunit.client.1.smithi190.stdout:8/611: write d9/dd/d23/d36/f71 [629329,48370] 0 2024-03-26T18:54:33.704 INFO:tasks.workunit.client.0.smithi102.stdout:9/200: creat df/d1a/d33/d43/f4a x:0 0 0 2024-03-26T18:54:33.705 INFO:tasks.workunit.client.0.smithi102.stdout:9/201: fsync f7 0 2024-03-26T18:54:33.705 INFO:tasks.workunit.client.0.smithi102.stdout:9/202: write df/d1a/d23/f48 [452103,49725] 0 2024-03-26T18:54:33.705 INFO:tasks.workunit.client.0.smithi102.stdout:5/468: dread d2/d14/d1a/f5e [0,4194304] 0 2024-03-26T18:54:33.706 INFO:tasks.workunit.client.0.smithi102.stdout:5/469: write d2/f12 [725410,59050] 0 2024-03-26T18:54:33.707 INFO:tasks.workunit.client.0.smithi102.stdout:2/215: rmdir d3/d6/d1a/d25/d3c 0 2024-03-26T18:54:33.707 INFO:tasks.workunit.client.1.smithi190.stdout:4/580: mkdir d7/d8/d14/dc2 0 2024-03-26T18:54:33.709 INFO:tasks.workunit.client.0.smithi102.stdout:3/396: mknod df/d1c/d34/c89 0 2024-03-26T18:54:33.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/575: mkdir d6/d9/d10/db7/dcf 0 2024-03-26T18:54:33.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/576: readlink d6/lb3 0 2024-03-26T18:54:33.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/577: write d6/d9/d3a/d3d/d1c/d8d/fab [380190,13815] 0 2024-03-26T18:54:33.711 INFO:tasks.workunit.client.0.smithi102.stdout:2/216: dread d3/f1c [0,4194304] 0 2024-03-26T18:54:33.711 INFO:tasks.workunit.client.0.smithi102.stdout:2/217: write d3/fb [1515638,39620] 0 2024-03-26T18:54:33.711 INFO:tasks.workunit.client.0.smithi102.stdout:9/203: symlink df/d1a/d1d/l4b 0 2024-03-26T18:54:33.711 INFO:tasks.workunit.client.0.smithi102.stdout:9/204: dread - df/d1a/f24 zero size 2024-03-26T18:54:33.711 INFO:tasks.workunit.client.0.smithi102.stdout:9/205: readlink df/l15 0 2024-03-26T18:54:33.714 INFO:tasks.workunit.client.0.smithi102.stdout:8/285: dwrite d0/d3/d2c/d33/f49 [0,4194304] 0 2024-03-26T18:54:33.714 INFO:tasks.workunit.client.0.smithi102.stdout:8/286: chown d0/d15 20 1 2024-03-26T18:54:33.714 INFO:tasks.workunit.client.0.smithi102.stdout:8/287: chown d0/d3/l7 238 1 2024-03-26T18:54:33.715 INFO:tasks.workunit.client.0.smithi102.stdout:5/470: mkdir d2/d25/d2a/d4f/da9 0 2024-03-26T18:54:33.717 INFO:tasks.workunit.client.1.smithi190.stdout:6/504: mknod d17/d28/d2c/d77/cad 0 2024-03-26T18:54:33.717 INFO:tasks.workunit.client.1.smithi190.stdout:6/505: chown d17/d28/d38/f89 4258 1 2024-03-26T18:54:33.717 INFO:tasks.workunit.client.1.smithi190.stdout:6/506: chown d17/d28/d2c/d50/f8f 17 1 2024-03-26T18:54:33.717 INFO:tasks.workunit.client.1.smithi190.stdout:6/507: creat d17/d28/d74/fae x:0 0 0 2024-03-26T18:54:33.719 INFO:tasks.workunit.client.0.smithi102.stdout:3/397: getdents df/d29/d3a 0 2024-03-26T18:54:33.721 INFO:tasks.workunit.client.1.smithi190.stdout:7/646: creat d3/d6/d9/d1f/d41/d50/dde/fed x:0 0 0 2024-03-26T18:54:33.721 INFO:tasks.workunit.client.0.smithi102.stdout:2/218: dread d3/d11/d15/d46/f47 [0,4194304] 0 2024-03-26T18:54:33.726 INFO:tasks.workunit.client.1.smithi190.stdout:5/481: creat d2/d17/d1b/d22/d24/fab x:0 0 0 2024-03-26T18:54:33.728 INFO:tasks.workunit.client.1.smithi190.stdout:5/482: read d2/d17/d98/d3a/d3c/f94 [240688,73562] 0 2024-03-26T18:54:33.728 INFO:tasks.workunit.client.1.smithi190.stdout:8/612: symlink d9/da2/d87/ld1 0 2024-03-26T18:54:33.729 INFO:tasks.workunit.client.0.smithi102.stdout:3/398: mkdir df/d29/d31/d77/d8a 0 2024-03-26T18:54:33.731 INFO:tasks.workunit.client.1.smithi190.stdout:4/581: truncate d7/f49 486004 0 2024-03-26T18:54:33.739 INFO:tasks.workunit.client.1.smithi190.stdout:2/578: rename d6/d9/d10/db7/daf to d6/d21/dd0 0 2024-03-26T18:54:33.740 INFO:tasks.workunit.client.1.smithi190.stdout:2/579: chown d6/cb9 1296098819 1 2024-03-26T18:54:33.742 INFO:tasks.workunit.client.1.smithi190.stdout:6/508: mknod d17/d28/d8b/d95/caf 0 2024-03-26T18:54:33.742 INFO:tasks.workunit.client.1.smithi190.stdout:6/509: truncate d17/d28/d2c/d50/f65 427763 0 2024-03-26T18:54:33.742 INFO:tasks.workunit.client.1.smithi190.stdout:6/510: creat d17/d28/d74/d7b/fb0 x:0 0 0 2024-03-26T18:54:33.742 INFO:tasks.workunit.client.1.smithi190.stdout:6/511: creat d17/d28/d38/fb1 x:0 0 0 2024-03-26T18:54:33.743 INFO:tasks.workunit.client.0.smithi102.stdout:2/219: rmdir d3/d6 39 2024-03-26T18:54:33.743 INFO:tasks.workunit.client.1.smithi190.stdout:7/647: mknod d3/d6/d9/d14/d36/cee 0 2024-03-26T18:54:33.744 INFO:tasks.workunit.client.1.smithi190.stdout:7/648: fdatasync d3/f5a 0 2024-03-26T18:54:33.744 INFO:tasks.workunit.client.1.smithi190.stdout:7/649: chown d3/d6/f1d 84096995 1 2024-03-26T18:54:33.744 INFO:tasks.workunit.client.0.smithi102.stdout:2/220: creat d3/d2a/f4f x:0 0 0 2024-03-26T18:54:33.745 INFO:tasks.workunit.client.0.smithi102.stdout:2/221: chown d3/d6/d1a/d25 3092 1 2024-03-26T18:54:33.745 INFO:tasks.workunit.client.1.smithi190.stdout:5/483: mknod d2/d17/d1b/d22/d24/d3b/da7/cac 0 2024-03-26T18:54:33.746 INFO:tasks.workunit.client.1.smithi190.stdout:4/582: symlink d7/d8/d14/d23/d28/d9e/lc3 0 2024-03-26T18:54:33.746 INFO:tasks.workunit.client.0.smithi102.stdout:2/222: creat d3/d11/f50 x:0 0 0 2024-03-26T18:54:33.749 INFO:tasks.workunit.client.0.smithi102.stdout:8/288: dwrite d0/d3/f21 [0,4194304] 0 2024-03-26T18:54:33.749 INFO:tasks.workunit.client.1.smithi190.stdout:7/650: link d3/d6/la d3/lef 0 2024-03-26T18:54:33.750 INFO:tasks.workunit.client.0.smithi102.stdout:5/471: dwrite d2/d2c/d38/d43/d9e/d82/f97 [0,4194304] 0 2024-03-26T18:54:33.750 INFO:tasks.workunit.client.0.smithi102.stdout:9/206: dwrite df/d1a/d23/f48 [0,4194304] 0 2024-03-26T18:54:33.750 INFO:tasks.workunit.client.0.smithi102.stdout:9/207: creat df/d1a/d23/d29/f4c x:0 0 0 2024-03-26T18:54:33.752 INFO:tasks.workunit.client.0.smithi102.stdout:8/289: unlink d0/d3/d2c/d33/l48 0 2024-03-26T18:54:33.752 INFO:tasks.workunit.client.0.smithi102.stdout:8/290: stat d0/d3/d31/c39 0 2024-03-26T18:54:33.754 INFO:tasks.workunit.client.0.smithi102.stdout:2/223: read d3/f8 [1318823,54175] 0 2024-03-26T18:54:33.754 INFO:tasks.workunit.client.0.smithi102.stdout:2/224: write d3/d11/f50 [654937,25494] 0 2024-03-26T18:54:33.754 INFO:tasks.workunit.client.0.smithi102.stdout:2/225: stat d3/d11/f50 0 2024-03-26T18:54:33.755 INFO:tasks.workunit.client.1.smithi190.stdout:2/580: dread d6/d9/d10/f34 [0,4194304] 0 2024-03-26T18:54:33.755 INFO:tasks.workunit.client.0.smithi102.stdout:5/472: mknod d2/d25/d2a/d4f/caa 0 2024-03-26T18:54:33.758 INFO:tasks.workunit.client.1.smithi190.stdout:2/581: mkdir d6/d9/d3a/d3d/d1c/d8d/dc7/dd1 0 2024-03-26T18:54:33.761 INFO:tasks.workunit.client.1.smithi190.stdout:6/512: write d17/d28/d2c/f35 [2699760,76491] 0 2024-03-26T18:54:33.761 INFO:tasks.workunit.client.0.smithi102.stdout:8/291: mknod d0/d3/d2c/d58/c5e 0 2024-03-26T18:54:33.761 INFO:tasks.workunit.client.0.smithi102.stdout:8/292: readlink d0/d3/l7 0 2024-03-26T18:54:33.762 INFO:tasks.workunit.client.0.smithi102.stdout:8/293: getdents d0/d3/d26/d5c 0 2024-03-26T18:54:33.762 INFO:tasks.workunit.client.0.smithi102.stdout:8/294: write d0/d3/d2c/d33/f54 [3708423,48444] 0 2024-03-26T18:54:33.762 INFO:tasks.workunit.client.0.smithi102.stdout:8/295: creat d0/d3/d31/f5f x:0 0 0 2024-03-26T18:54:33.762 INFO:tasks.workunit.client.0.smithi102.stdout:8/296: getdents d0/d15/d5d 0 2024-03-26T18:54:33.762 INFO:tasks.workunit.client.0.smithi102.stdout:8/297: dread d0/d3/d31/f3a [0,4194304] 0 2024-03-26T18:54:33.762 INFO:tasks.workunit.client.1.smithi190.stdout:6/513: dread d17/d28/d2c/d50/f65 [0,4194304] 0 2024-03-26T18:54:33.763 INFO:tasks.workunit.client.1.smithi190.stdout:6/514: write d17/d28/d2c/d77/fac [323422,17192] 0 2024-03-26T18:54:33.763 INFO:tasks.workunit.client.0.smithi102.stdout:2/226: rename d3/d11/d15/l21 to d3/l51 0 2024-03-26T18:54:33.764 INFO:tasks.workunit.client.1.smithi190.stdout:4/583: fsync d7/f49 0 2024-03-26T18:54:33.772 INFO:tasks.workunit.client.0.smithi102.stdout:3/399: dwrite df/d1c/f2a [0,4194304] 0 2024-03-26T18:54:33.772 INFO:tasks.workunit.client.0.smithi102.stdout:3/400: readlink df/l2f 0 2024-03-26T18:54:33.773 INFO:tasks.workunit.client.1.smithi190.stdout:2/582: write d6/d9/d3a/d3d/d2e/f9a [4030962,5644] 0 2024-03-26T18:54:33.773 INFO:tasks.workunit.client.1.smithi190.stdout:6/515: rename d17/d28/d2c/d77/f69 to d17/d28/d80/da8/fb2 0 2024-03-26T18:54:33.773 INFO:tasks.workunit.client.1.smithi190.stdout:2/583: readlink d6/d9/d3a/d3d/d1c/l53 0 2024-03-26T18:54:33.774 INFO:tasks.workunit.client.1.smithi190.stdout:2/584: write d6/d9/f41 [92935,55090] 0 2024-03-26T18:54:33.775 INFO:tasks.workunit.client.0.smithi102.stdout:5/473: rename d2/f78 to d2/d2c/d38/d43/d9e/d8b/fab 0 2024-03-26T18:54:33.775 INFO:tasks.workunit.client.0.smithi102.stdout:2/227: dread d3/d6/f24 [0,4194304] 0 2024-03-26T18:54:33.777 INFO:tasks.workunit.client.1.smithi190.stdout:4/584: creat d7/d8/d14/d4b/fc4 x:0 0 0 2024-03-26T18:54:33.781 INFO:tasks.workunit.client.1.smithi190.stdout:8/613: dwrite d9/da2/dc4/f4f [0,4194304] 0 2024-03-26T18:54:33.781 INFO:tasks.workunit.client.1.smithi190.stdout:8/614: readlink d9/dd/d23/d36/d37/d3a/d43/dc2/lcd 0 2024-03-26T18:54:33.781 INFO:tasks.workunit.client.1.smithi190.stdout:5/484: dwrite d2/d17/d1b/d22/d24/f84 [0,4194304] 0 2024-03-26T18:54:33.782 INFO:tasks.workunit.client.0.smithi102.stdout:2/228: mknod d3/d6/d1a/d25/c52 0 2024-03-26T18:54:33.782 INFO:tasks.workunit.client.1.smithi190.stdout:5/485: dread d2/d17/d1b/d22/d68/f75 [0,4194304] 0 2024-03-26T18:54:33.782 INFO:tasks.workunit.client.0.smithi102.stdout:8/298: dwrite d0/d3/d2c/f34 [0,4194304] 0 2024-03-26T18:54:33.785 INFO:tasks.workunit.client.0.smithi102.stdout:3/401: mkdir df/d8b 0 2024-03-26T18:54:33.786 INFO:tasks.workunit.client.0.smithi102.stdout:5/474: write d2/d14/d1a/f41 [2930930,33938] 0 2024-03-26T18:54:33.786 INFO:tasks.workunit.client.1.smithi190.stdout:6/516: mknod d17/d28/d8b/cb3 0 2024-03-26T18:54:33.787 INFO:tasks.workunit.client.1.smithi190.stdout:6/517: creat d17/d28/d74/fb4 x:0 0 0 2024-03-26T18:54:33.787 INFO:tasks.workunit.client.1.smithi190.stdout:6/518: getdents d17/d28/d74/d97 0 2024-03-26T18:54:33.788 INFO:tasks.workunit.client.1.smithi190.stdout:2/585: mknod d6/d9/d3a/d3d/d67/d9e/cd2 0 2024-03-26T18:54:33.791 INFO:tasks.workunit.client.1.smithi190.stdout:4/585: unlink d7/d1d/d41/c8d 0 2024-03-26T18:54:33.792 INFO:tasks.workunit.client.1.smithi190.stdout:3/472: sync 2024-03-26T18:54:33.792 INFO:tasks.workunit.client.1.smithi190.stdout:9/846: sync 2024-03-26T18:54:33.792 INFO:tasks.workunit.client.1.smithi190.stdout:3/473: dread - d1/d3/d52/f90 zero size 2024-03-26T18:54:33.792 INFO:tasks.workunit.client.1.smithi190.stdout:9/847: chown d12/d2d/d1d/d54/l9d 1 1 2024-03-26T18:54:33.793 INFO:tasks.workunit.client.0.smithi102.stdout:2/229: mknod d3/d6/c53 0 2024-03-26T18:54:33.793 INFO:tasks.workunit.client.0.smithi102.stdout:2/230: chown d3/d11/d15/d31/d39 16 1 2024-03-26T18:54:33.797 INFO:tasks.workunit.client.1.smithi190.stdout:1/509: sync 2024-03-26T18:54:33.797 INFO:tasks.workunit.client.1.smithi190.stdout:1/510: write d3/da/dd/d95/d52/f93 [624487,16545] 0 2024-03-26T18:54:33.797 INFO:tasks.workunit.client.1.smithi190.stdout:0/655: sync 2024-03-26T18:54:33.797 INFO:tasks.workunit.client.1.smithi190.stdout:8/615: mkdir d9/dd/d23/d36/d37/d3a/d43/d8f/dd2 0 2024-03-26T18:54:33.805 INFO:tasks.workunit.client.0.smithi102.stdout:3/402: rename df/f17 to df/d29/f8c 0 2024-03-26T18:54:33.805 INFO:tasks.workunit.client.0.smithi102.stdout:2/231: rename d3/d6/d1a to d3/d6/d1a/d25/d54 22 2024-03-26T18:54:33.805 INFO:tasks.workunit.client.0.smithi102.stdout:2/232: write d3/fb [621942,105009] 0 2024-03-26T18:54:33.806 INFO:tasks.workunit.client.0.smithi102.stdout:2/233: stat d3/d20/f30 0 2024-03-26T18:54:33.806 INFO:tasks.workunit.client.0.smithi102.stdout:2/234: creat d3/d11/f55 x:0 0 0 2024-03-26T18:54:33.806 INFO:tasks.workunit.client.0.smithi102.stdout:2/235: truncate d3/d11/d27/f28 224245 0 2024-03-26T18:54:33.806 INFO:tasks.workunit.client.0.smithi102.stdout:2/236: chown d3/d2a/l3a 146449 1 2024-03-26T18:54:33.806 INFO:tasks.workunit.client.0.smithi102.stdout:2/237: chown d3/d6/d1a/d25 2 1 2024-03-26T18:54:33.806 INFO:tasks.workunit.client.1.smithi190.stdout:3/474: dread d1/d3/df/f74 [4194304,4194304] 0 2024-03-26T18:54:33.807 INFO:tasks.workunit.client.1.smithi190.stdout:5/486: dwrite d2/d17/d1b/f58 [0,4194304] 0 2024-03-26T18:54:33.807 INFO:tasks.workunit.client.0.smithi102.stdout:5/475: unlink d2/f9 0 2024-03-26T18:54:33.807 INFO:tasks.workunit.client.0.smithi102.stdout:5/476: readlink d2/d25/d2a/l86 0 2024-03-26T18:54:33.808 INFO:tasks.workunit.client.0.smithi102.stdout:9/208: dwrite df/f3b [4194304,4194304] 0 2024-03-26T18:54:33.808 INFO:tasks.workunit.client.0.smithi102.stdout:3/403: mknod df/d29/d31/d36/d3c/d6f/c8d 0 2024-03-26T18:54:33.810 INFO:tasks.workunit.client.1.smithi190.stdout:6/519: rename d17/d28/d2c/d77/cad to d17/d28/d80/cb5 0 2024-03-26T18:54:33.814 INFO:tasks.workunit.client.0.smithi102.stdout:5/477: link d2/f64 d2/d25/d5b/d76/fac 0 2024-03-26T18:54:33.816 INFO:tasks.workunit.client.1.smithi190.stdout:2/586: mknod d6/d9/d3a/d3d/d67/d9e/da3/dae/cd3 0 2024-03-26T18:54:33.819 INFO:tasks.workunit.client.1.smithi190.stdout:4/586: symlink d7/d8/d1c/lc5 0 2024-03-26T18:54:33.819 INFO:tasks.workunit.client.1.smithi190.stdout:4/587: creat d7/d1d/fc6 x:0 0 0 2024-03-26T18:54:33.820 INFO:tasks.workunit.client.0.smithi102.stdout:9/209: symlink df/d1a/d1d/l4d 0 2024-03-26T18:54:33.820 INFO:tasks.workunit.client.0.smithi102.stdout:9/210: truncate df/d1a/d1d/f32 948396 0 2024-03-26T18:54:33.820 INFO:tasks.workunit.client.0.smithi102.stdout:9/211: chown df/d1a/d1d/l1f 26485 1 2024-03-26T18:54:33.820 INFO:tasks.workunit.client.0.smithi102.stdout:9/212: readlink df/d1a/d1d/l38 0 2024-03-26T18:54:33.821 INFO:tasks.workunit.client.1.smithi190.stdout:9/848: mkdir d12/d2d/d1d/d23/d83/dc9/dd4/d71/d129 0 2024-03-26T18:54:33.822 INFO:tasks.workunit.client.0.smithi102.stdout:3/404: creat df/d1c/d34/f8e x:0 0 0 2024-03-26T18:54:33.822 INFO:tasks.workunit.client.0.smithi102.stdout:3/405: write df/d61/f5f [835453,113064] 0 2024-03-26T18:54:33.823 INFO:tasks.workunit.client.1.smithi190.stdout:1/511: link d3/d9/c40 d3/da/ca8 0 2024-03-26T18:54:33.824 INFO:tasks.workunit.client.0.smithi102.stdout:5/478: mknod d2/cad 0 2024-03-26T18:54:33.825 INFO:tasks.workunit.client.1.smithi190.stdout:8/616: mkdir d9/dd/d15/dd3 0 2024-03-26T18:54:33.825 INFO:tasks.workunit.client.1.smithi190.stdout:8/617: stat d9/dd/d23/d36/d37/d3a/d43/l64 0 2024-03-26T18:54:33.826 INFO:tasks.workunit.client.0.smithi102.stdout:9/213: mkdir df/d1a/d23/d4e 0 2024-03-26T18:54:33.826 INFO:tasks.workunit.client.0.smithi102.stdout:2/238: dwrite d3/d11/d15/d31/f35 [0,4194304] 0 2024-03-26T18:54:33.827 INFO:tasks.workunit.client.1.smithi190.stdout:3/475: dwrite d1/de/d1c/f7b [0,4194304] 0 2024-03-26T18:54:33.827 INFO:tasks.workunit.client.1.smithi190.stdout:8/618: dread d9/d39/d79/f90 [0,4194304] 0 2024-03-26T18:54:33.828 INFO:tasks.workunit.client.1.smithi190.stdout:8/619: chown d9/dd/d23/d36/d37/d3a/d43/l64 6932172 1 2024-03-26T18:54:33.828 INFO:tasks.workunit.client.0.smithi102.stdout:3/406: symlink df/d1c/d34/d53/l8f 0 2024-03-26T18:54:33.832 INFO:tasks.workunit.client.0.smithi102.stdout:5/479: mkdir d2/d14/dae 0 2024-03-26T18:54:33.836 INFO:tasks.workunit.client.0.smithi102.stdout:9/214: creat df/d1a/d23/d4e/f4f x:0 0 0 2024-03-26T18:54:33.836 INFO:tasks.workunit.client.0.smithi102.stdout:9/215: chown df/d1a/d23/f2c 3445285 1 2024-03-26T18:54:33.849 INFO:tasks.workunit.client.1.smithi190.stdout:0/656: rename d0/dd/c32 to d0/dd/d20/d1c/d8b/d8d/d8f/cef 0 2024-03-26T18:54:33.849 INFO:tasks.workunit.client.1.smithi190.stdout:4/588: rename d7 to d7/d1d/d42/dc7 22 2024-03-26T18:54:33.850 INFO:tasks.workunit.client.1.smithi190.stdout:6/520: symlink d17/d28/d2c/d50/lb6 0 2024-03-26T18:54:33.851 INFO:tasks.workunit.client.1.smithi190.stdout:6/521: write d17/d28/d74/d7b/fa5 [903167,110688] 0 2024-03-26T18:54:33.852 INFO:tasks.workunit.client.1.smithi190.stdout:2/587: link d6/f7a d6/d9/d10/db7/dcf/fd4 0 2024-03-26T18:54:33.855 INFO:tasks.workunit.client.1.smithi190.stdout:9/849: getdents d12/d2d/d1d/d23/d83/dc9 0 2024-03-26T18:54:33.855 INFO:tasks.workunit.client.1.smithi190.stdout:9/850: write d12/d2d/d1d/d23/d83/dc9/dd4/f73 [1999533,39745] 0 2024-03-26T18:54:33.857 INFO:tasks.workunit.client.1.smithi190.stdout:1/512: mknod d3/da/d2e/ca9 0 2024-03-26T18:54:33.857 INFO:tasks.workunit.client.1.smithi190.stdout:1/513: fsync d3/da/dd/d2f/d33/f8a 0 2024-03-26T18:54:33.858 INFO:tasks.workunit.client.0.smithi102.stdout:3/407: chown df/l21 11036447 1 2024-03-26T18:54:33.860 INFO:tasks.workunit.client.0.smithi102.stdout:5/480: mkdir d2/d2c/d38/d43/d9e/daf 0 2024-03-26T18:54:33.863 INFO:tasks.workunit.client.1.smithi190.stdout:8/620: dwrite d9/dd/d23/d36/d37/d3a/d61/f93 [0,4194304] 0 2024-03-26T18:54:33.863 INFO:tasks.workunit.client.1.smithi190.stdout:8/621: write d9/dd/d15/f7a [1067173,63143] 0 2024-03-26T18:54:33.863 INFO:tasks.workunit.client.1.smithi190.stdout:8/622: write d9/d39/d9e/fc5 [471177,57886] 0 2024-03-26T18:54:33.863 INFO:tasks.workunit.client.1.smithi190.stdout:8/623: read f6 [5145760,101125] 0 2024-03-26T18:54:33.863 INFO:tasks.workunit.client.0.smithi102.stdout:2/239: dwrite d3/f8 [0,4194304] 0 2024-03-26T18:54:33.864 INFO:tasks.workunit.client.0.smithi102.stdout:9/216: symlink df/d1a/l50 0 2024-03-26T18:54:33.864 INFO:tasks.workunit.client.0.smithi102.stdout:9/217: truncate df/f42 113748 0 2024-03-26T18:54:33.865 INFO:tasks.workunit.client.0.smithi102.stdout:3/408: creat df/d61/d55/f90 x:0 0 0 2024-03-26T18:54:33.865 INFO:tasks.workunit.client.0.smithi102.stdout:3/409: chown df/d29/d31/f79 4234828 1 2024-03-26T18:54:33.865 INFO:tasks.workunit.client.0.smithi102.stdout:3/410: dread - df/d29/f5b zero size 2024-03-26T18:54:33.865 INFO:tasks.workunit.client.1.smithi190.stdout:3/476: dwrite d1/d3/df/d18/d35/d41/d51/f7a [0,4194304] 0 2024-03-26T18:54:33.866 INFO:tasks.workunit.client.1.smithi190.stdout:3/477: dread - d1/d3/df/d18/d57/d91/d8c/f8d zero size 2024-03-26T18:54:33.866 INFO:tasks.workunit.client.1.smithi190.stdout:4/589: dread d7/f4f [4194304,4194304] 0 2024-03-26T18:54:33.866 INFO:tasks.workunit.client.1.smithi190.stdout:4/590: write d7/f4f [6306307,7136] 0 2024-03-26T18:54:33.866 INFO:tasks.workunit.client.0.smithi102.stdout:5/481: mkdir d2/d25/db0 0 2024-03-26T18:54:33.868 INFO:tasks.workunit.client.1.smithi190.stdout:1/514: truncate d3/da/d45/f48 3232084 0 2024-03-26T18:54:33.869 INFO:tasks.workunit.client.0.smithi102.stdout:2/240: truncate d3/d11/d15/f48 122014 0 2024-03-26T18:54:33.869 INFO:tasks.workunit.client.0.smithi102.stdout:2/241: chown d3/d11/d15/d46/f47 161 1 2024-03-26T18:54:33.871 INFO:tasks.workunit.client.0.smithi102.stdout:9/218: rename df/c21 to df/d1a/d33/d43/c51 0 2024-03-26T18:54:33.871 INFO:tasks.workunit.client.0.smithi102.stdout:9/219: creat df/d1a/f52 x:0 0 0 2024-03-26T18:54:33.871 INFO:tasks.workunit.client.0.smithi102.stdout:9/220: truncate df/d1a/d1d/f32 955776 0 2024-03-26T18:54:33.871 INFO:tasks.workunit.client.0.smithi102.stdout:9/221: creat df/d1a/d23/d4e/f53 x:0 0 0 2024-03-26T18:54:33.872 INFO:tasks.workunit.client.1.smithi190.stdout:6/522: rmdir d17/d28/d2c/d77 39 2024-03-26T18:54:33.874 INFO:tasks.workunit.client.0.smithi102.stdout:3/411: rename df/d1c/l23 to df/d29/d31/d36/d45/l91 0 2024-03-26T18:54:33.875 INFO:tasks.workunit.client.1.smithi190.stdout:3/478: dread d1/d3/d52/f62 [0,4194304] 0 2024-03-26T18:54:33.875 INFO:tasks.workunit.client.0.smithi102.stdout:5/482: write d2/f10 [2068244,45987] 0 2024-03-26T18:54:33.883 INFO:tasks.workunit.client.1.smithi190.stdout:6/523: dread d17/f24 [0,4194304] 0 2024-03-26T18:54:33.883 INFO:tasks.workunit.client.1.smithi190.stdout:6/524: creat d17/d28/d2c/fb7 x:0 0 0 2024-03-26T18:54:33.884 INFO:tasks.workunit.client.0.smithi102.stdout:9/222: mknod df/d1a/d1d/c54 0 2024-03-26T18:54:33.884 INFO:tasks.workunit.client.0.smithi102.stdout:9/223: chown f7 31434 1 2024-03-26T18:54:33.886 INFO:tasks.workunit.client.0.smithi102.stdout:7/207: sync 2024-03-26T18:54:33.887 INFO:tasks.workunit.client.0.smithi102.stdout:4/202: sync 2024-03-26T18:54:33.887 INFO:tasks.workunit.client.0.smithi102.stdout:0/263: sync 2024-03-26T18:54:33.887 INFO:tasks.workunit.client.0.smithi102.stdout:4/203: rename d1/d25/d39/d3e to d1/d25/d39/d3e/d43 22 2024-03-26T18:54:33.887 INFO:tasks.workunit.client.0.smithi102.stdout:7/208: truncate d7/d10/d17/f35 195747 0 2024-03-26T18:54:33.887 INFO:tasks.workunit.client.0.smithi102.stdout:7/209: truncate d7/d12/f22 5097025 0 2024-03-26T18:54:33.887 INFO:tasks.workunit.client.0.smithi102.stdout:7/210: write d7/d10/f14 [469583,9154] 0 2024-03-26T18:54:33.887 INFO:tasks.workunit.client.1.smithi190.stdout:2/588: symlink d6/d9/d3a/d3d/d2e/ld5 0 2024-03-26T18:54:33.888 INFO:tasks.workunit.client.1.smithi190.stdout:2/589: truncate d6/d21/d61/d6c/fc4 407103 0 2024-03-26T18:54:33.888 INFO:tasks.workunit.client.0.smithi102.stdout:3/412: mknod df/d1c/c92 0 2024-03-26T18:54:33.890 INFO:tasks.workunit.client.0.smithi102.stdout:5/483: symlink d2/d2c/d38/d43/d9e/d8b/lb1 0 2024-03-26T18:54:33.890 INFO:tasks.workunit.client.0.smithi102.stdout:5/484: creat d2/d25/d2a/d4f/fb2 x:0 0 0 2024-03-26T18:54:33.892 INFO:tasks.workunit.client.0.smithi102.stdout:9/224: rename df/d1a/d1d/l3d to df/d1a/d1d/l55 0 2024-03-26T18:54:33.893 INFO:tasks.workunit.client.1.smithi190.stdout:4/591: mkdir d7/dc8 0 2024-03-26T18:54:33.893 INFO:tasks.workunit.client.1.smithi190.stdout:4/592: write d7/d1d/d42/fbe [830218,128364] 0 2024-03-26T18:54:33.893 INFO:tasks.workunit.client.1.smithi190.stdout:4/593: write d7/d1d/d42/d4a/fa6 [357603,71906] 0 2024-03-26T18:54:33.893 INFO:tasks.workunit.client.1.smithi190.stdout:4/594: creat d7/d1d/d5e/d98/fc9 x:0 0 0 2024-03-26T18:54:33.894 INFO:tasks.workunit.client.0.smithi102.stdout:0/264: mkdir d6/d3e/d54 0 2024-03-26T18:54:33.896 INFO:tasks.workunit.client.0.smithi102.stdout:4/204: symlink d1/d25/d39/l44 0 2024-03-26T18:54:33.898 INFO:tasks.workunit.client.0.smithi102.stdout:0/265: dread d6/de/f4b [0,4194304] 0 2024-03-26T18:54:33.898 INFO:tasks.workunit.client.1.smithi190.stdout:1/515: creat d3/d73/faa x:0 0 0 2024-03-26T18:54:33.899 INFO:tasks.workunit.client.0.smithi102.stdout:3/413: mkdir df/d29/d3a/d93 0 2024-03-26T18:54:33.899 INFO:tasks.workunit.client.0.smithi102.stdout:3/414: creat df/d29/d31/d36/d3c/d6f/d75/f94 x:0 0 0 2024-03-26T18:54:33.899 INFO:tasks.workunit.client.0.smithi102.stdout:3/415: creat df/d61/d55/f95 x:0 0 0 2024-03-26T18:54:33.899 INFO:tasks.workunit.client.0.smithi102.stdout:2/242: dwrite d3/d2a/f4f [0,4194304] 0 2024-03-26T18:54:33.900 INFO:tasks.workunit.client.1.smithi190.stdout:3/479: dwrite d1/de/f29 [0,4194304] 0 2024-03-26T18:54:33.903 INFO:tasks.workunit.client.0.smithi102.stdout:5/485: mkdir d2/d2c/d38/db3 0 2024-03-26T18:54:33.903 INFO:tasks.workunit.client.0.smithi102.stdout:5/486: chown d2/f7 1020 1 2024-03-26T18:54:33.906 INFO:tasks.workunit.client.1.smithi190.stdout:2/590: unlink d6/d9/d3a/d3d/d1c/f5e 0 2024-03-26T18:54:33.909 INFO:tasks.workunit.client.0.smithi102.stdout:4/205: rename d1/d17/c3f to d1/d17/d2a/c45 0 2024-03-26T18:54:33.909 INFO:tasks.workunit.client.0.smithi102.stdout:4/206: chown d1/d25 944 1 2024-03-26T18:54:33.909 INFO:tasks.workunit.client.0.smithi102.stdout:4/207: write d1/f26 [702375,130852] 0 2024-03-26T18:54:33.910 INFO:tasks.workunit.client.1.smithi190.stdout:4/595: mknod d7/d8/d14/d6e/d6f/cca 0 2024-03-26T18:54:33.910 INFO:tasks.workunit.client.0.smithi102.stdout:0/266: mkdir d6/de/d13/d1f/d55 0 2024-03-26T18:54:33.918 INFO:tasks.workunit.client.0.smithi102.stdout:2/243: symlink d3/d11/d27/l56 0 2024-03-26T18:54:33.918 INFO:tasks.workunit.client.0.smithi102.stdout:2/244: creat d3/d11/d27/f57 x:0 0 0 2024-03-26T18:54:33.918 INFO:tasks.workunit.client.0.smithi102.stdout:2/245: truncate d3/f19 384300 0 2024-03-26T18:54:33.918 INFO:tasks.workunit.client.0.smithi102.stdout:2/246: write d3/d6/f2b [937102,33095] 0 2024-03-26T18:54:33.919 INFO:tasks.workunit.client.1.smithi190.stdout:2/591: dread d6/d9/d3a/d3d/d2e/f51 [0,4194304] 0 2024-03-26T18:54:33.919 INFO:tasks.workunit.client.0.smithi102.stdout:5/487: mkdir d2/d25/d2a/db4 0 2024-03-26T18:54:33.921 INFO:tasks.workunit.client.1.smithi190.stdout:3/480: truncate d1/de/d1c/d36/f40 93611 0 2024-03-26T18:54:33.921 INFO:tasks.workunit.client.1.smithi190.stdout:3/481: read - d1/d3/df/d31/f56 zero size 2024-03-26T18:54:33.921 INFO:tasks.workunit.client.1.smithi190.stdout:3/482: chown d1/l3e 168064 1 2024-03-26T18:54:33.921 INFO:tasks.workunit.client.1.smithi190.stdout:3/483: readlink d1/d3/d5a/d6e/l77 0 2024-03-26T18:54:33.921 INFO:tasks.workunit.client.1.smithi190.stdout:3/484: stat d1/f2c 0 2024-03-26T18:54:33.921 INFO:tasks.workunit.client.1.smithi190.stdout:3/485: fsync d1/d3/df/d18/f6f 0 2024-03-26T18:54:33.921 INFO:tasks.workunit.client.1.smithi190.stdout:3/486: readlink d1/d3/d52/d70/l87 0 2024-03-26T18:54:33.922 INFO:tasks.workunit.client.0.smithi102.stdout:9/225: rmdir df/d1a/d33/d43 39 2024-03-26T18:54:33.922 INFO:tasks.workunit.client.1.smithi190.stdout:6/525: getdents d17/d28/d38/d79 0 2024-03-26T18:54:33.924 INFO:tasks.workunit.client.0.smithi102.stdout:4/208: truncate d1/d17/d1f/f2d 3435916 0 2024-03-26T18:54:33.924 INFO:tasks.workunit.client.0.smithi102.stdout:4/209: fsync d1/d27/f3c 0 2024-03-26T18:54:33.924 INFO:tasks.workunit.client.0.smithi102.stdout:4/210: chown d1/cd 252 1 2024-03-26T18:54:33.924 INFO:tasks.workunit.client.0.smithi102.stdout:4/211: chown d1/f9 785274 1 2024-03-26T18:54:33.924 INFO:tasks.workunit.client.1.smithi190.stdout:4/596: symlink d7/d8/d1c/d20/lcb 0 2024-03-26T18:54:33.925 INFO:tasks.workunit.client.1.smithi190.stdout:4/597: fdatasync d7/d8/d14/f48 0 2024-03-26T18:54:33.925 INFO:tasks.workunit.client.1.smithi190.stdout:4/598: creat d7/d1d/d42/d4a/fcc x:0 0 0 2024-03-26T18:54:33.925 INFO:tasks.workunit.client.0.smithi102.stdout:0/267: creat d6/d2d/d3d/f56 x:0 0 0 2024-03-26T18:54:33.926 INFO:tasks.workunit.client.0.smithi102.stdout:0/268: dread - d6/de/d19/f46 zero size 2024-03-26T18:54:33.928 INFO:tasks.workunit.client.1.smithi190.stdout:2/592: mkdir d6/d9/d3a/d3d/d67/d9e/dd6 0 2024-03-26T18:54:33.931 INFO:tasks.workunit.client.0.smithi102.stdout:5/488: mknod d2/d25/db0/cb5 0 2024-03-26T18:54:33.931 INFO:tasks.workunit.client.0.smithi102.stdout:5/489: fdatasync d2/d14/d1a/f5e 0 2024-03-26T18:54:33.931 INFO:tasks.workunit.client.0.smithi102.stdout:5/490: write d2/d14/d1a/f91 [886215,120093] 0 2024-03-26T18:54:33.932 INFO:tasks.workunit.client.0.smithi102.stdout:3/416: dwrite df/d29/f33 [0,4194304] 0 2024-03-26T18:54:33.932 INFO:tasks.workunit.client.0.smithi102.stdout:3/417: creat df/d1c/d70/f96 x:0 0 0 2024-03-26T18:54:33.933 INFO:tasks.workunit.client.0.smithi102.stdout:2/247: dread d3/d2a/f34 [0,4194304] 0 2024-03-26T18:54:33.933 INFO:tasks.workunit.client.0.smithi102.stdout:2/248: chown d3/d11/c18 130519 1 2024-03-26T18:54:33.934 INFO:tasks.workunit.client.1.smithi190.stdout:4/599: mknod d7/d8/d14/d4b/db2/ccd 0 2024-03-26T18:54:33.935 INFO:tasks.workunit.client.0.smithi102.stdout:5/491: rmdir d2/d25/d75 39 2024-03-26T18:54:33.937 INFO:tasks.workunit.client.1.smithi190.stdout:4/600: creat d7/d1d/d5e/fce x:0 0 0 2024-03-26T18:54:33.937 INFO:tasks.workunit.client.1.smithi190.stdout:4/601: write d7/d8/d14/f19 [3603956,49426] 0 2024-03-26T18:54:33.938 INFO:tasks.workunit.client.1.smithi190.stdout:6/526: read d17/d28/f2d [1305003,73130] 0 2024-03-26T18:54:33.938 INFO:tasks.workunit.client.0.smithi102.stdout:3/418: rename df/d29/f5b to df/d29/d31/d77/d8a/f97 0 2024-03-26T18:54:33.939 INFO:tasks.workunit.client.0.smithi102.stdout:2/249: mkdir d3/d11/d15/d31/d58 0 2024-03-26T18:54:33.941 INFO:tasks.workunit.client.0.smithi102.stdout:5/492: mknod d2/d2c/d6f/cb6 0 2024-03-26T18:54:33.941 INFO:tasks.workunit.client.0.smithi102.stdout:5/493: write d2/d14/d1a/f61 [907477,69209] 0 2024-03-26T18:54:33.941 INFO:tasks.workunit.client.0.smithi102.stdout:4/212: dread d1/f7 [0,4194304] 0 2024-03-26T18:54:33.942 INFO:tasks.workunit.client.1.smithi190.stdout:6/527: getdents d17/d28/d8b 0 2024-03-26T18:54:33.944 INFO:tasks.workunit.client.0.smithi102.stdout:5/494: symlink d2/d2c/d38/lb7 0 2024-03-26T18:54:33.945 INFO:tasks.workunit.client.0.smithi102.stdout:5/495: write d2/d25/d2a/d4f/f9c [643154,89116] 0 2024-03-26T18:54:33.948 INFO:tasks.workunit.client.0.smithi102.stdout:4/213: mknod d1/d27/c46 0 2024-03-26T18:54:33.949 INFO:tasks.workunit.client.0.smithi102.stdout:3/419: dread df/d61/f57 [0,4194304] 0 2024-03-26T18:54:33.949 INFO:tasks.workunit.client.0.smithi102.stdout:5/496: mkdir d2/d2c/d6f/db8 0 2024-03-26T18:54:33.951 INFO:tasks.workunit.client.0.smithi102.stdout:4/214: mkdir d1/d25/d39/d47 0 2024-03-26T18:54:33.953 INFO:tasks.workunit.client.0.smithi102.stdout:4/215: creat d1/d17/d1f/f48 x:0 0 0 2024-03-26T18:54:33.953 INFO:tasks.workunit.client.0.smithi102.stdout:4/216: read d1/f7 [1345213,31524] 0 2024-03-26T18:54:33.957 INFO:tasks.workunit.client.1.smithi190.stdout:3/487: dwrite d1/de/d4e/f8f [4194304,4194304] 0 2024-03-26T18:54:33.961 INFO:tasks.workunit.client.1.smithi190.stdout:2/593: dwrite d6/d9/d10/f2d [0,4194304] 0 2024-03-26T18:54:33.971 INFO:tasks.workunit.client.1.smithi190.stdout:3/488: dread d1/d3/df/f2a [0,4194304] 0 2024-03-26T18:54:33.973 INFO:tasks.workunit.client.1.smithi190.stdout:3/489: mknod d1/d3/df/d18/d35/d41/d61/c92 0 2024-03-26T18:54:33.973 INFO:tasks.workunit.client.1.smithi190.stdout:3/490: creat d1/d3/df/d18/d35/d41/d51/f93 x:0 0 0 2024-03-26T18:54:33.976 INFO:tasks.workunit.client.1.smithi190.stdout:3/491: truncate d1/f11 3377212 0 2024-03-26T18:54:33.976 INFO:tasks.workunit.client.1.smithi190.stdout:3/492: fsync d1/de/f29 0 2024-03-26T18:54:33.976 INFO:tasks.workunit.client.1.smithi190.stdout:3/493: dread - d1/de/d1c/f83 zero size 2024-03-26T18:54:33.977 INFO:tasks.workunit.client.0.smithi102.stdout:2/250: write d3/d6/f16 [2812024,113259] 0 2024-03-26T18:54:33.979 INFO:tasks.workunit.client.1.smithi190.stdout:3/494: truncate d1/f26 1665846 0 2024-03-26T18:54:33.979 INFO:tasks.workunit.client.1.smithi190.stdout:2/594: write d6/d9/d3a/f94 [5663383,74159] 0 2024-03-26T18:54:33.980 INFO:tasks.workunit.client.0.smithi102.stdout:0/269: dwrite d6/fb [0,4194304] 0 2024-03-26T18:54:33.982 INFO:tasks.workunit.client.1.smithi190.stdout:3/495: getdents d1/d3/d5a 0 2024-03-26T18:54:33.983 INFO:tasks.workunit.client.0.smithi102.stdout:7/211: dwrite d7/d10/d17/f35 [0,4194304] 0 2024-03-26T18:54:33.983 INFO:tasks.workunit.client.0.smithi102.stdout:7/212: dread - d7/d2a/f30 zero size 2024-03-26T18:54:33.984 INFO:tasks.workunit.client.1.smithi190.stdout:3/496: truncate d1/d3/f13 3317207 0 2024-03-26T18:54:33.986 INFO:tasks.workunit.client.1.smithi190.stdout:1/516: dwrite d3/da/dd/f7c [0,4194304] 0 2024-03-26T18:54:33.987 INFO:tasks.workunit.client.1.smithi190.stdout:1/517: creat d3/fab x:0 0 0 2024-03-26T18:54:33.992 INFO:tasks.workunit.client.0.smithi102.stdout:9/226: dwrite df/d1a/f52 [0,4194304] 0 2024-03-26T18:54:33.992 INFO:tasks.workunit.client.1.smithi190.stdout:4/602: write d7/d1d/d42/fbe [577133,67975] 0 2024-03-26T18:54:33.994 INFO:tasks.workunit.client.1.smithi190.stdout:1/518: rename d3/da/f37 to d3/da/dd/d2f/d33/fac 0 2024-03-26T18:54:33.994 INFO:tasks.workunit.client.1.smithi190.stdout:1/519: truncate d3/da/dd/d95/f46 1503864 0 2024-03-26T18:54:33.994 INFO:tasks.workunit.client.1.smithi190.stdout:1/520: truncate d3/d9/f3c 784244 0 2024-03-26T18:54:33.994 INFO:tasks.workunit.client.1.smithi190.stdout:1/521: chown d3/da/dd/d2f/d34/f69 215 1 2024-03-26T18:54:33.995 INFO:tasks.workunit.client.1.smithi190.stdout:6/528: dwrite d17/f87 [0,4194304] 0 2024-03-26T18:54:33.995 INFO:tasks.workunit.client.1.smithi190.stdout:6/529: chown d17/d28/d2b/c72 99083817 1 2024-03-26T18:54:33.995 INFO:tasks.workunit.client.0.smithi102.stdout:3/420: dwrite df/d29/d31/d36/d3c/d6f/f7d [0,4194304] 0 2024-03-26T18:54:33.998 INFO:tasks.workunit.client.1.smithi190.stdout:4/603: creat d7/d8/d14/d23/d28/d38/fcf x:0 0 0 2024-03-26T18:54:34.006 INFO:tasks.workunit.client.1.smithi190.stdout:6/530: rename d17/d28/d2c/d77/d51 to d17/d28/d2c/db8 0 2024-03-26T18:54:34.008 INFO:tasks.workunit.client.1.smithi190.stdout:6/531: getdents d17/d28/d46 0 2024-03-26T18:54:34.012 INFO:tasks.workunit.client.0.smithi102.stdout:9/227: creat df/d1a/d23/d29/d45/f56 x:0 0 0 2024-03-26T18:54:34.014 INFO:tasks.workunit.client.1.smithi190.stdout:2/595: dwrite f5 [0,4194304] 0 2024-03-26T18:54:34.018 INFO:tasks.workunit.client.1.smithi190.stdout:3/497: dwrite d1/d3/df/d18/d1b/f21 [4194304,4194304] 0 2024-03-26T18:54:34.018 INFO:tasks.workunit.client.1.smithi190.stdout:3/498: write d1/de/d1c/f83 [130316,77154] 0 2024-03-26T18:54:34.018 INFO:tasks.workunit.client.1.smithi190.stdout:1/522: dwrite d3/d9/fe [0,4194304] 0 2024-03-26T18:54:34.018 INFO:tasks.workunit.client.0.smithi102.stdout:3/421: symlink df/d29/d31/d36/d3c/d6f/l98 0 2024-03-26T18:54:34.018 INFO:tasks.workunit.client.0.smithi102.stdout:3/422: chown df/d29/f2c 3 1 2024-03-26T18:54:34.018 INFO:tasks.workunit.client.0.smithi102.stdout:7/213: mknod d7/c3e 0 2024-03-26T18:54:34.019 INFO:tasks.workunit.client.1.smithi190.stdout:2/596: symlink d6/d9/d3a/d3d/d1c/d8d/ld7 0 2024-03-26T18:54:34.021 INFO:tasks.workunit.client.0.smithi102.stdout:9/228: mkdir df/d1a/d57 0 2024-03-26T18:54:34.024 INFO:tasks.workunit.client.0.smithi102.stdout:7/214: symlink d7/d12/l3f 0 2024-03-26T18:54:34.027 INFO:tasks.workunit.client.0.smithi102.stdout:9/229: mknod df/d1a/d23/d29/c58 0 2024-03-26T18:54:34.029 INFO:tasks.workunit.client.0.smithi102.stdout:5/497: rename d2/d2c/d38 to d2/d25/d75/db9 0 2024-03-26T18:54:34.032 INFO:tasks.workunit.client.0.smithi102.stdout:9/230: mknod df/d1a/c59 0 2024-03-26T18:54:34.035 INFO:tasks.workunit.client.0.smithi102.stdout:4/217: rename d1/d17/d1a/c24 to d1/d25/d39/c49 0 2024-03-26T18:54:34.035 INFO:tasks.workunit.client.0.smithi102.stdout:4/218: chown d1/f31 8572460 1 2024-03-26T18:54:34.035 INFO:tasks.workunit.client.0.smithi102.stdout:4/219: truncate d1/d27/f3c 265045 0 2024-03-26T18:54:34.037 INFO:tasks.workunit.client.1.smithi190.stdout:6/532: dwrite d17/d28/d38/d79/f82 [0,4194304] 0 2024-03-26T18:54:34.037 INFO:tasks.workunit.client.0.smithi102.stdout:7/215: getdents d7/d12/d2b 0 2024-03-26T18:54:34.037 INFO:tasks.workunit.client.0.smithi102.stdout:7/216: fsync f4 0 2024-03-26T18:54:34.040 INFO:tasks.workunit.client.0.smithi102.stdout:0/270: rename d6/de/d19 to d6/de/d13/d1f/d55/d57 0 2024-03-26T18:54:34.041 INFO:tasks.workunit.client.1.smithi190.stdout:3/499: creat d1/d3/df/d18/d57/f94 x:0 0 0 2024-03-26T18:54:34.044 INFO:tasks.workunit.client.0.smithi102.stdout:5/498: dread d2/d25/f2b [0,4194304] 0 2024-03-26T18:54:34.044 INFO:tasks.workunit.client.0.smithi102.stdout:5/499: fsync d2/f6a 0 2024-03-26T18:54:34.044 INFO:tasks.workunit.client.0.smithi102.stdout:5/500: dread - d2/d2c/f8d zero size 2024-03-26T18:54:34.044 INFO:tasks.workunit.client.0.smithi102.stdout:5/501: fdatasync d2/d25/d2a/d4f/fb2 0 2024-03-26T18:54:34.044 INFO:tasks.workunit.client.0.smithi102.stdout:5/502: chown d2/d2c/f8d 622 1 2024-03-26T18:54:34.044 INFO:tasks.workunit.client.0.smithi102.stdout:5/503: truncate d2/d25/d75/db9/d43/f92 842525 0 2024-03-26T18:54:34.044 INFO:tasks.workunit.client.0.smithi102.stdout:0/271: creat d6/d3e/d54/f58 x:0 0 0 2024-03-26T18:54:34.046 INFO:tasks.workunit.client.1.smithi190.stdout:1/523: link d3/da/dd/d95/f8e d3/d73/d88/fad 0 2024-03-26T18:54:34.046 INFO:tasks.workunit.client.1.smithi190.stdout:1/524: getdents d3/d8f/da7 0 2024-03-26T18:54:34.046 INFO:tasks.workunit.client.1.smithi190.stdout:1/525: readlink d3/d9/db/l26 0 2024-03-26T18:54:34.046 INFO:tasks.workunit.client.1.smithi190.stdout:1/526: write d3/d9/f1c [330225,48939] 0 2024-03-26T18:54:34.046 INFO:tasks.workunit.client.1.smithi190.stdout:1/527: write d3/d9/f5c [514829,113016] 0 2024-03-26T18:54:34.047 INFO:tasks.workunit.client.0.smithi102.stdout:7/217: dread d7/f20 [0,4194304] 0 2024-03-26T18:54:34.047 INFO:tasks.workunit.client.0.smithi102.stdout:7/218: chown d7/c3e 114 1 2024-03-26T18:54:34.047 INFO:tasks.workunit.client.0.smithi102.stdout:7/219: rename d7/d10 to d7/d10/d40 22 2024-03-26T18:54:34.047 INFO:tasks.workunit.client.0.smithi102.stdout:5/504: unlink d2/cad 0 2024-03-26T18:54:34.047 INFO:tasks.workunit.client.0.smithi102.stdout:7/220: write d7/d2a/f30 [986629,105361] 0 2024-03-26T18:54:34.047 INFO:tasks.workunit.client.0.smithi102.stdout:3/423: dwrite df/d29/f64 [0,4194304] 0 2024-03-26T18:54:34.047 INFO:tasks.workunit.client.0.smithi102.stdout:3/424: stat df/d1c 0 2024-03-26T18:54:34.048 INFO:tasks.workunit.client.0.smithi102.stdout:0/272: creat d6/de/d13/d1f/f59 x:0 0 0 2024-03-26T18:54:34.048 INFO:tasks.workunit.client.1.smithi190.stdout:2/597: dwrite d6/d9/f38 [0,4194304] 0 2024-03-26T18:54:34.048 INFO:tasks.workunit.client.1.smithi190.stdout:4/604: dwrite d7/d1d/d42/fbe [0,4194304] 0 2024-03-26T18:54:34.049 INFO:tasks.workunit.client.1.smithi190.stdout:4/605: creat d7/d8/d14/d4b/d7d/fd0 x:0 0 0 2024-03-26T18:54:34.055 INFO:tasks.workunit.client.1.smithi190.stdout:6/533: rename d17/d28/d74/fa2 to d17/d28/d74/d97/fb9 0 2024-03-26T18:54:34.055 INFO:tasks.workunit.client.0.smithi102.stdout:0/273: dread d6/f1a [0,4194304] 0 2024-03-26T18:54:34.057 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:33 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:54:34.057 INFO:tasks.workunit.client.1.smithi190.stdout:3/500: creat d1/de/d8a/f95 x:0 0 0 2024-03-26T18:54:34.058 INFO:tasks.workunit.client.1.smithi190.stdout:3/501: creat d1/d3/df/d18/d35/d41/d51/f96 x:0 0 0 2024-03-26T18:54:34.060 INFO:tasks.workunit.client.0.smithi102.stdout:3/425: mknod df/d29/d31/d36/c99 0 2024-03-26T18:54:34.061 INFO:tasks.workunit.client.0.smithi102.stdout:5/505: dread d2/d25/d5b/d72/f22 [0,4194304] 0 2024-03-26T18:54:34.062 INFO:tasks.workunit.client.0.smithi102.stdout:0/274: rmdir d6 39 2024-03-26T18:54:34.065 INFO:tasks.workunit.client.0.smithi102.stdout:3/426: rename df/d29/d3a/c62 to df/d29/d3a/d93/c9a 0 2024-03-26T18:54:34.065 INFO:tasks.workunit.client.0.smithi102.stdout:7/221: dread d7/fb [0,4194304] 0 2024-03-26T18:54:34.065 INFO:tasks.workunit.client.0.smithi102.stdout:7/222: fdatasync d7/ff 0 2024-03-26T18:54:34.065 INFO:tasks.workunit.client.1.smithi190.stdout:1/528: dwrite d3/da/dd/d2f/d5b/f63 [0,4194304] 0 2024-03-26T18:54:34.066 INFO:tasks.workunit.client.1.smithi190.stdout:2/598: mkdir d6/d21/dd8 0 2024-03-26T18:54:34.067 INFO:tasks.workunit.client.0.smithi102.stdout:5/506: dread d2/f11 [0,4194304] 0 2024-03-26T18:54:34.069 INFO:tasks.workunit.client.0.smithi102.stdout:0/275: rename d6/d2d/f35 to d6/de/d13/f5a 0 2024-03-26T18:54:34.069 INFO:tasks.workunit.client.0.smithi102.stdout:5/507: dread d2/d2c/f4c [0,4194304] 0 2024-03-26T18:54:34.069 INFO:tasks.workunit.client.0.smithi102.stdout:5/508: write d2/d25/d75/db9/d43/d9e/d82/fa2 [1304137,122127] 0 2024-03-26T18:54:34.069 INFO:tasks.workunit.client.0.smithi102.stdout:5/509: write d2/d2c/f8d [577961,130280] 0 2024-03-26T18:54:34.071 INFO:tasks.workunit.client.1.smithi190.stdout:4/606: symlink d7/d8/d14/d23/d28/d9e/ld1 0 2024-03-26T18:54:34.071 INFO:tasks.workunit.client.1.smithi190.stdout:4/607: rename d7 to d7/d8/d14/d23/d28/d38/d96/d9a/dd2 22 2024-03-26T18:54:34.072 INFO:tasks.workunit.client.0.smithi102.stdout:3/427: mkdir df/d1c/d34/d9b 0 2024-03-26T18:54:34.072 INFO:tasks.workunit.client.0.smithi102.stdout:3/428: truncate df/d1c/d34/f37 58316 0 2024-03-26T18:54:34.072 INFO:tasks.workunit.client.0.smithi102.stdout:3/429: fsync df/d29/d31/d36/f50 0 2024-03-26T18:54:34.072 INFO:tasks.workunit.client.0.smithi102.stdout:3/430: write df/d29/d31/d36/d3c/d6f/d75/f88 [573549,20658] 0 2024-03-26T18:54:34.073 INFO:tasks.workunit.client.0.smithi102.stdout:7/223: rmdir d7/d12/d2b 39 2024-03-26T18:54:34.075 INFO:tasks.workunit.client.0.smithi102.stdout:0/276: link d6/d2d/d3d/f44 d6/de/d13/d4a/f5b 0 2024-03-26T18:54:34.075 INFO:tasks.workunit.client.0.smithi102.stdout:0/277: write d6/f36 [402791,3823] 0 2024-03-26T18:54:34.077 INFO:tasks.workunit.client.0.smithi102.stdout:7/224: creat d7/f41 x:0 0 0 2024-03-26T18:54:34.077 INFO:tasks.workunit.client.0.smithi102.stdout:7/225: stat d7/f41 0 2024-03-26T18:54:34.077 INFO:tasks.workunit.client.0.smithi102.stdout:7/226: getdents d7/d12/d2b 0 2024-03-26T18:54:34.079 INFO:tasks.workunit.client.0.smithi102.stdout:0/278: link d6/f51 d6/de/d13/d1f/f5c 0 2024-03-26T18:54:34.079 INFO:tasks.workunit.client.0.smithi102.stdout:0/279: chown d6/de/f28 66158 1 2024-03-26T18:54:34.081 INFO:tasks.workunit.client.0.smithi102.stdout:7/227: symlink d7/d12/d2b/l42 0 2024-03-26T18:54:34.084 INFO:tasks.workunit.client.1.smithi190.stdout:6/534: creat d17/d28/d46/fba x:0 0 0 2024-03-26T18:54:34.086 INFO:tasks.workunit.client.0.smithi102.stdout:1/245: sync 2024-03-26T18:54:34.086 INFO:tasks.workunit.client.0.smithi102.stdout:8/299: sync 2024-03-26T18:54:34.086 INFO:tasks.workunit.client.0.smithi102.stdout:6/272: sync 2024-03-26T18:54:34.086 INFO:tasks.workunit.client.1.smithi190.stdout:7/651: sync 2024-03-26T18:54:34.087 INFO:tasks.workunit.client.0.smithi102.stdout:3/431: write df/d29/d3a/f6d [765775,27940] 0 2024-03-26T18:54:34.089 INFO:tasks.workunit.client.0.smithi102.stdout:7/228: creat d7/d12/f43 x:0 0 0 2024-03-26T18:54:34.091 INFO:tasks.workunit.client.1.smithi190.stdout:3/502: unlink d1/d3/df/d18/d35/f76 0 2024-03-26T18:54:34.092 INFO:tasks.workunit.client.0.smithi102.stdout:6/273: symlink d2/d7/d3c/l50 0 2024-03-26T18:54:34.093 INFO:tasks.workunit.client.0.smithi102.stdout:6/274: creat d2/d7/f51 x:0 0 0 2024-03-26T18:54:34.096 INFO:tasks.workunit.client.1.smithi190.stdout:5/487: sync 2024-03-26T18:54:34.097 INFO:tasks.workunit.client.1.smithi190.stdout:5/488: write d2/d17/f4f [924091,87572] 0 2024-03-26T18:54:34.097 INFO:tasks.workunit.client.0.smithi102.stdout:7/229: unlink d7/l3d 0 2024-03-26T18:54:34.098 INFO:tasks.workunit.client.1.smithi190.stdout:4/608: mkdir d7/d8/d14/d23/d28/d38/dd3 0 2024-03-26T18:54:34.098 INFO:tasks.workunit.client.1.smithi190.stdout:4/609: dread - d7/d8/d14/d4b/fc4 zero size 2024-03-26T18:54:34.099 INFO:tasks.workunit.client.1.smithi190.stdout:2/599: dwrite d6/d9/d3a/d3d/fbf [0,4194304] 0 2024-03-26T18:54:34.099 INFO:tasks.workunit.client.1.smithi190.stdout:2/600: creat d6/d9/d10/fd9 x:0 0 0 2024-03-26T18:54:34.099 INFO:tasks.workunit.client.1.smithi190.stdout:2/601: readlink d6/d9/d3a/d3d/l77 0 2024-03-26T18:54:34.099 INFO:tasks.workunit.client.1.smithi190.stdout:2/602: creat d6/d9/d3a/fda x:0 0 0 2024-03-26T18:54:34.100 INFO:tasks.workunit.client.1.smithi190.stdout:6/535: link d17/d28/d74/fae d17/d28/d2c/d77/fbb 0 2024-03-26T18:54:34.103 INFO:tasks.workunit.client.1.smithi190.stdout:7/652: creat d3/d1b/dd3/ff0 x:0 0 0 2024-03-26T18:54:34.103 INFO:tasks.workunit.client.1.smithi190.stdout:7/653: fsync d3/d6/d9/d14/fc9 0 2024-03-26T18:54:34.103 INFO:tasks.workunit.client.1.smithi190.stdout:1/529: dwrite d3/d9/f5c [0,4194304] 0 2024-03-26T18:54:34.103 INFO:tasks.workunit.client.1.smithi190.stdout:1/530: dread - d3/da/d3a/f50 zero size 2024-03-26T18:54:34.105 INFO:tasks.workunit.client.0.smithi102.stdout:3/432: rename f2 to df/f9c 0 2024-03-26T18:54:34.106 INFO:tasks.workunit.client.1.smithi190.stdout:3/503: getdents d1/d3/df/d18/d57/d91/d8c 0 2024-03-26T18:54:34.109 INFO:tasks.workunit.client.1.smithi190.stdout:4/610: symlink d7/d1d/d41/ld4 0 2024-03-26T18:54:34.111 INFO:tasks.workunit.client.0.smithi102.stdout:6/275: link d2/d7/f51 d2/f52 0 2024-03-26T18:54:34.111 INFO:tasks.workunit.client.0.smithi102.stdout:6/276: fdatasync d2/df/f27 0 2024-03-26T18:54:34.111 INFO:tasks.workunit.client.0.smithi102.stdout:6/277: chown d2/df/l32 105 1 2024-03-26T18:54:34.112 INFO:tasks.workunit.client.1.smithi190.stdout:5/489: dread d2/d17/d1b/d22/f77 [0,4194304] 0 2024-03-26T18:54:34.112 INFO:tasks.workunit.client.1.smithi190.stdout:5/490: write d2/f3 [9016104,76696] 0 2024-03-26T18:54:34.113 INFO:tasks.workunit.client.0.smithi102.stdout:7/230: mkdir d7/d44 0 2024-03-26T18:54:34.118 INFO:tasks.workunit.client.0.smithi102.stdout:6/278: unlink d2/d12/f22 0 2024-03-26T18:54:34.122 INFO:tasks.workunit.client.0.smithi102.stdout:6/279: mkdir d2/d3d/d53 0 2024-03-26T18:54:34.122 INFO:tasks.workunit.client.0.smithi102.stdout:6/280: creat d2/d7/f54 x:0 0 0 2024-03-26T18:54:34.123 INFO:tasks.workunit.client.0.smithi102.stdout:0/280: dwrite d6/de/d13/f5a [0,4194304] 0 2024-03-26T18:54:34.123 INFO:tasks.workunit.client.0.smithi102.stdout:0/281: chown d6/de/f1d 1410655428 1 2024-03-26T18:54:34.125 INFO:tasks.workunit.client.1.smithi190.stdout:2/603: mkdir d6/d9/d45/d6b/ddb 0 2024-03-26T18:54:34.125 INFO:tasks.workunit.client.1.smithi190.stdout:0/657: sync 2024-03-26T18:54:34.125 INFO:tasks.workunit.client.1.smithi190.stdout:0/658: fdatasync d0/dd/d24/d4b/f89 0 2024-03-26T18:54:34.127 INFO:tasks.workunit.client.1.smithi190.stdout:1/531: unlink d3/da/d2e/c28 0 2024-03-26T18:54:34.129 INFO:tasks.workunit.client.1.smithi190.stdout:3/504: read d1/de/d4e/f8f [6567308,44502] 0 2024-03-26T18:54:34.129 INFO:tasks.workunit.client.1.smithi190.stdout:5/491: creat d2/d17/d62/fad x:0 0 0 2024-03-26T18:54:34.129 INFO:tasks.workunit.client.1.smithi190.stdout:5/492: write d2/d17/d62/f7e [449125,30295] 0 2024-03-26T18:54:34.129 INFO:tasks.workunit.client.1.smithi190.stdout:5/493: chown d2/d17/d1b/d22/f2d 109848 1 2024-03-26T18:54:34.130 INFO:tasks.workunit.client.0.smithi102.stdout:8/300: dwrite d0/d3/d31/f3a [0,4194304] 0 2024-03-26T18:54:34.130 INFO:tasks.workunit.client.0.smithi102.stdout:8/301: creat d0/d3/d2c/d2f/f60 x:0 0 0 2024-03-26T18:54:34.131 INFO:tasks.workunit.client.0.smithi102.stdout:0/282: truncate d6/de/d13/d1f/f5c 372036 0 2024-03-26T18:54:34.132 INFO:tasks.workunit.client.1.smithi190.stdout:2/604: symlink d6/d9/d45/d6b/d93/ldc 0 2024-03-26T18:54:34.134 INFO:tasks.workunit.client.0.smithi102.stdout:3/433: dwrite df/d29/d31/d36/f50 [0,4194304] 0 2024-03-26T18:54:34.134 INFO:tasks.workunit.client.0.smithi102.stdout:3/434: chown df/d29/d31/d36/d45/f65 218 1 2024-03-26T18:54:34.135 INFO:tasks.workunit.client.1.smithi190.stdout:0/659: truncate d0/dd/d20/d1c/d8b/d8d/db7/fba 1850898 0 2024-03-26T18:54:34.135 INFO:tasks.workunit.client.1.smithi190.stdout:0/660: write d0/dd/d20/d1c/d46/da7/fda [499195,23855] 0 2024-03-26T18:54:34.137 INFO:tasks.workunit.client.1.smithi190.stdout:1/532: mkdir d3/da/d2e/d59/dae 0 2024-03-26T18:54:34.137 INFO:tasks.workunit.client.1.smithi190.stdout:1/533: write d3/da/dd/d95/f56 [1513261,52466] 0 2024-03-26T18:54:34.139 INFO:tasks.workunit.client.1.smithi190.stdout:8/624: sync 2024-03-26T18:54:34.139 INFO:tasks.workunit.client.1.smithi190.stdout:9/851: sync 2024-03-26T18:54:34.139 INFO:tasks.workunit.client.1.smithi190.stdout:9/852: truncate d12/d2d/d1d/d23/d2b/f95 1826131 0 2024-03-26T18:54:34.140 INFO:tasks.workunit.client.0.smithi102.stdout:5/510: dwrite d2/d25/d75/db9/d43/f4e [0,4194304] 0 2024-03-26T18:54:34.141 INFO:tasks.workunit.client.1.smithi190.stdout:6/536: dwrite d17/d28/d80/fa9 [0,4194304] 0 2024-03-26T18:54:34.141 INFO:tasks.workunit.client.1.smithi190.stdout:3/505: mknod d1/c97 0 2024-03-26T18:54:34.142 INFO:tasks.workunit.client.0.smithi102.stdout:0/283: mknod d6/de/d13/d1f/d55/c5d 0 2024-03-26T18:54:34.142 INFO:tasks.workunit.client.0.smithi102.stdout:0/284: write d6/de/d13/f3f [1049223,52415] 0 2024-03-26T18:54:34.143 INFO:tasks.workunit.client.0.smithi102.stdout:1/246: dwrite f2 [0,4194304] 0 2024-03-26T18:54:34.144 INFO:tasks.workunit.client.1.smithi190.stdout:2/605: symlink d6/dbc/ldd 0 2024-03-26T18:54:34.147 INFO:tasks.workunit.client.1.smithi190.stdout:0/661: unlink d0/f99 0 2024-03-26T18:54:34.149 INFO:tasks.workunit.client.0.smithi102.stdout:3/435: unlink c6 0 2024-03-26T18:54:34.151 INFO:tasks.workunit.client.1.smithi190.stdout:1/534: mkdir d3/da/daf 0 2024-03-26T18:54:34.151 INFO:tasks.workunit.client.1.smithi190.stdout:1/535: truncate d3/d9/db/f32 4859374 0 2024-03-26T18:54:34.152 INFO:tasks.workunit.client.1.smithi190.stdout:1/536: write d3/d9/f57 [758515,98322] 0 2024-03-26T18:54:34.152 INFO:tasks.workunit.client.0.smithi102.stdout:5/511: rename d2/d25/d2a/d4f/caa to d2/d25/d75/db9/db3/cba 0 2024-03-26T18:54:34.154 INFO:tasks.workunit.client.1.smithi190.stdout:8/625: truncate d9/dd/f59 3162830 0 2024-03-26T18:54:34.154 INFO:tasks.workunit.client.1.smithi190.stdout:3/506: write d1/d3/df/d18/d1b/f21 [3855796,48087] 0 2024-03-26T18:54:34.154 INFO:tasks.workunit.client.0.smithi102.stdout:0/285: truncate d6/d2d/f38 1681804 0 2024-03-26T18:54:34.156 INFO:tasks.workunit.client.1.smithi190.stdout:6/537: creat d17/d28/d74/d97/fbc x:0 0 0 2024-03-26T18:54:34.156 INFO:tasks.workunit.client.1.smithi190.stdout:6/538: stat d17/d28/d2c/f35 0 2024-03-26T18:54:34.156 INFO:tasks.workunit.client.1.smithi190.stdout:6/539: write d17/d28/d2c/d77/fbb [942858,109030] 0 2024-03-26T18:54:34.161 INFO:tasks.workunit.client.1.smithi190.stdout:8/626: dread d9/dd/d15/f45 [0,4194304] 0 2024-03-26T18:54:34.163 INFO:tasks.workunit.client.0.smithi102.stdout:0/286: write d6/f36 [48038,109916] 0 2024-03-26T18:54:34.163 INFO:tasks.workunit.client.1.smithi190.stdout:1/537: mkdir d3/d9/db/db0 0 2024-03-26T18:54:34.166 INFO:tasks.workunit.client.0.smithi102.stdout:3/436: write df/d29/d31/d36/d45/f65 [2341374,9862] 0 2024-03-26T18:54:34.166 INFO:tasks.workunit.client.1.smithi190.stdout:2/606: rmdir d6/d21 39 2024-03-26T18:54:34.167 INFO:tasks.workunit.client.0.smithi102.stdout:7/231: dwrite d7/d12/f22 [0,4194304] 0 2024-03-26T18:54:34.167 INFO:tasks.workunit.client.0.smithi102.stdout:3/437: dread df/d29/d3a/f85 [0,4194304] 0 2024-03-26T18:54:34.167 INFO:tasks.workunit.client.1.smithi190.stdout:6/540: dread d17/d28/d74/d7b/fa5 [0,4194304] 0 2024-03-26T18:54:34.167 INFO:tasks.workunit.client.1.smithi190.stdout:6/541: write d17/d28/f2d [743085,24274] 0 2024-03-26T18:54:34.171 INFO:tasks.workunit.client.0.smithi102.stdout:0/287: creat d6/de/d13/d1f/d55/d57/f5e x:0 0 0 2024-03-26T18:54:34.171 INFO:tasks.workunit.client.0.smithi102.stdout:0/288: chown d6/fb 36 1 2024-03-26T18:54:34.172 INFO:tasks.workunit.client.1.smithi190.stdout:9/853: dwrite d12/d2d/d1d/d54/ff8 [0,4194304] 0 2024-03-26T18:54:34.173 INFO:tasks.workunit.client.1.smithi190.stdout:0/662: rename d0/dd/d24 to d0/dd/d20/d1c/d46/da7/df0 0 2024-03-26T18:54:34.175 INFO:tasks.workunit.client.0.smithi102.stdout:7/232: unlink d7/d12/d2b/l42 0 2024-03-26T18:54:34.175 INFO:tasks.workunit.client.1.smithi190.stdout:5/494: dwrite d2/f14 [0,4194304] 0 2024-03-26T18:54:34.178 INFO:tasks.workunit.client.1.smithi190.stdout:8/627: symlink d9/dd/d23/d36/d37/d3a/d43/d8f/ld4 0 2024-03-26T18:54:34.178 INFO:tasks.workunit.client.0.smithi102.stdout:3/438: symlink df/d29/d3a/l9d 0 2024-03-26T18:54:34.179 INFO:tasks.workunit.client.1.smithi190.stdout:4/611: dwrite d7/d8/faa [0,4194304] 0 2024-03-26T18:54:34.183 INFO:tasks.workunit.client.0.smithi102.stdout:5/512: dwrite d2/d14/d1a/f77 [0,4194304] 0 2024-03-26T18:54:34.185 INFO:tasks.workunit.client.1.smithi190.stdout:6/542: symlink d17/d28/d80/da8/lbd 0 2024-03-26T18:54:34.185 INFO:tasks.workunit.client.1.smithi190.stdout:6/543: chown f1 666988 1 2024-03-26T18:54:34.185 INFO:tasks.workunit.client.1.smithi190.stdout:6/544: write d17/d28/d80/da8/fb2 [9461929,91445] 0 2024-03-26T18:54:34.185 INFO:tasks.workunit.client.1.smithi190.stdout:6/545: chown d17/d28/d2c/d77/f66 48245 1 2024-03-26T18:54:34.185 INFO:tasks.workunit.client.1.smithi190.stdout:7/654: fsync d3/d1b/dd3/ff0 0 2024-03-26T18:54:34.185 INFO:tasks.workunit.client.1.smithi190.stdout:7/655: readlink d3/d6/d9/d1f/d41/d65/d90/d92/d9b/deb/ldd 0 2024-03-26T18:54:34.187 INFO:tasks.workunit.client.0.smithi102.stdout:0/289: mknod d6/d3e/d54/c5f 0 2024-03-26T18:54:34.187 INFO:tasks.workunit.client.0.smithi102.stdout:0/290: chown d6/d3e/d54/c5f 60651997 1 2024-03-26T18:54:34.189 INFO:tasks.workunit.client.0.smithi102.stdout:3/439: mkdir df/d1c/d70/d9e 0 2024-03-26T18:54:34.190 INFO:tasks.workunit.client.0.smithi102.stdout:3/440: fdatasync fd 0 2024-03-26T18:54:34.190 INFO:tasks.workunit.client.1.smithi190.stdout:4/612: write d7/d1d/d42/fbe [3495824,39411] 0 2024-03-26T18:54:34.192 INFO:tasks.workunit.client.0.smithi102.stdout:7/233: dread d7/d12/f3c [0,4194304] 0 2024-03-26T18:54:34.194 INFO:tasks.workunit.client.1.smithi190.stdout:2/607: dwrite d6/d9/d3a/d3d/fbf [0,4194304] 0 2024-03-26T18:54:34.197 INFO:tasks.workunit.client.1.smithi190.stdout:0/663: dwrite d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/fea [0,4194304] 0 2024-03-26T18:54:34.198 INFO:tasks.workunit.client.1.smithi190.stdout:9/854: unlink d12/d2d/d1d/d23/d83/d91/fb9 0 2024-03-26T18:54:34.204 INFO:tasks.workunit.client.1.smithi190.stdout:8/628: unlink d9/dd/le 0 2024-03-26T18:54:34.208 INFO:tasks.workunit.client.0.smithi102.stdout:5/513: mkdir d2/d25/d2a/dbb 0 2024-03-26T18:54:34.209 INFO:tasks.workunit.client.0.smithi102.stdout:5/514: stat d2/l18 0 2024-03-26T18:54:34.209 INFO:tasks.workunit.client.1.smithi190.stdout:0/664: write d0/dd/d20/d1c/d46/f5d [3911827,94555] 0 2024-03-26T18:54:34.211 INFO:tasks.workunit.client.0.smithi102.stdout:0/291: link d6/de/d13/d4a/l4d d6/de/d13/d1f/d55/d57/l60 0 2024-03-26T18:54:34.211 INFO:tasks.workunit.client.0.smithi102.stdout:0/292: stat d6/de/d13/d25/d31/f4e 0 2024-03-26T18:54:34.214 INFO:tasks.workunit.client.0.smithi102.stdout:3/441: rename df/d61/f66 to df/d29/d31/d77/f9f 0 2024-03-26T18:54:34.215 INFO:tasks.workunit.client.1.smithi190.stdout:4/613: dwrite d7/d8/d14/d5c/f83 [0,4194304] 0 2024-03-26T18:54:34.217 INFO:tasks.workunit.client.0.smithi102.stdout:7/234: creat d7/d31/f45 x:0 0 0 2024-03-26T18:54:34.217 INFO:tasks.workunit.client.0.smithi102.stdout:7/235: dread - d7/d12/f43 zero size 2024-03-26T18:54:34.218 INFO:tasks.workunit.client.1.smithi190.stdout:1/538: rmdir d3/da/d2e 39 2024-03-26T18:54:34.222 INFO:tasks.workunit.client.0.smithi102.stdout:5/515: dread d2/d25/d5b/d76/fac [0,4194304] 0 2024-03-26T18:54:34.222 INFO:tasks.workunit.client.0.smithi102.stdout:0/293: link d6/fd d6/d3e/d54/f61 0 2024-03-26T18:54:34.223 INFO:tasks.workunit.client.0.smithi102.stdout:0/294: chown d6/c2c 14 1 2024-03-26T18:54:34.223 INFO:tasks.workunit.client.0.smithi102.stdout:0/295: readlink d6/de/d13/d4a/l4d 0 2024-03-26T18:54:34.223 INFO:tasks.workunit.client.0.smithi102.stdout:0/296: dread - d6/de/d13/d1f/f59 zero size 2024-03-26T18:54:34.223 INFO:tasks.workunit.client.0.smithi102.stdout:0/297: write d6/de/d13/d1f/d55/d57/f30 [834213,20414] 0 2024-03-26T18:54:34.223 INFO:tasks.workunit.client.0.smithi102.stdout:0/298: dread - d6/de/f4c zero size 2024-03-26T18:54:34.223 INFO:tasks.workunit.client.0.smithi102.stdout:0/299: readlink d6/de/d13/d25/l34 0 2024-03-26T18:54:34.225 INFO:tasks.workunit.client.1.smithi190.stdout:6/546: unlink d17/d28/d2c/db8/f7c 0 2024-03-26T18:54:34.225 INFO:tasks.workunit.client.1.smithi190.stdout:6/547: creat d17/d28/d46/fbe x:0 0 0 2024-03-26T18:54:34.225 INFO:tasks.workunit.client.1.smithi190.stdout:6/548: chown d17/d28/d80/cb5 3 1 2024-03-26T18:54:34.226 INFO:tasks.workunit.client.0.smithi102.stdout:3/442: symlink df/d29/d3a/la0 0 2024-03-26T18:54:34.227 INFO:tasks.workunit.client.1.smithi190.stdout:7/656: mknod d3/d6/d9/d1f/d41/d65/d90/d92/d9b/de9/cf1 0 2024-03-26T18:54:34.228 INFO:tasks.workunit.client.1.smithi190.stdout:7/657: truncate d3/d6/d9/d1f/fa2 893418 0 2024-03-26T18:54:34.228 INFO:tasks.workunit.client.1.smithi190.stdout:7/658: truncate d3/d1b/f5e 1705438 0 2024-03-26T18:54:34.228 INFO:tasks.workunit.client.1.smithi190.stdout:7/659: creat d3/d6/d9/d1f/d41/d50/dde/ff2 x:0 0 0 2024-03-26T18:54:34.233 INFO:tasks.workunit.client.0.smithi102.stdout:5/516: symlink d2/d25/d75/db9/db3/lbc 0 2024-03-26T18:54:34.233 INFO:tasks.workunit.client.0.smithi102.stdout:5/517: chown d2/d25/f81 187818726 1 2024-03-26T18:54:34.233 INFO:tasks.workunit.client.0.smithi102.stdout:5/518: dread - d2/d25/d2a/f9f zero size 2024-03-26T18:54:34.234 INFO:tasks.workunit.client.0.smithi102.stdout:0/300: mknod d6/de/d13/d1f/d55/d57/c62 0 2024-03-26T18:54:34.235 INFO:tasks.workunit.client.0.smithi102.stdout:0/301: chown d6/c2c 15524828 1 2024-03-26T18:54:34.236 INFO:tasks.workunit.client.1.smithi190.stdout:6/549: read d17/d28/d2c/d77/f6f [2639607,12864] 0 2024-03-26T18:54:34.236 INFO:tasks.workunit.client.1.smithi190.stdout:6/550: write d17/d28/d2c/d63/fa0 [79178,18331] 0 2024-03-26T18:54:34.236 INFO:tasks.workunit.client.0.smithi102.stdout:3/443: mknod df/d29/ca1 0 2024-03-26T18:54:34.240 INFO:tasks.workunit.client.0.smithi102.stdout:5/519: truncate d2/f10 1732909 0 2024-03-26T18:54:34.246 INFO:tasks.workunit.client.0.smithi102.stdout:3/444: write df/d29/f8c [817176,125359] 0 2024-03-26T18:54:34.247 INFO:tasks.workunit.client.1.smithi190.stdout:2/608: unlink d6/d9/d10/f40 0 2024-03-26T18:54:34.248 INFO:tasks.workunit.client.1.smithi190.stdout:2/609: truncate d6/d9/d10/f87 1069140 0 2024-03-26T18:54:34.249 INFO:tasks.workunit.client.0.smithi102.stdout:5/520: dread d2/d25/d5b/d72/f42 [0,4194304] 0 2024-03-26T18:54:34.254 INFO:tasks.workunit.client.0.smithi102.stdout:1/247: dwrite f7 [0,4194304] 0 2024-03-26T18:54:34.254 INFO:tasks.workunit.client.1.smithi190.stdout:9/855: rename d12/d2d/d1d/d23/d83/l123 to d12/l12a 0 2024-03-26T18:54:34.255 INFO:tasks.workunit.client.0.smithi102.stdout:5/521: creat d2/d25/db0/fbd x:0 0 0 2024-03-26T18:54:34.257 INFO:tasks.workunit.client.1.smithi190.stdout:5/495: getdents d2/d17/d98/d3a 0 2024-03-26T18:54:34.257 INFO:tasks.workunit.client.1.smithi190.stdout:5/496: write d2/d17/d1b/f72 [751415,98346] 0 2024-03-26T18:54:34.259 INFO:tasks.workunit.client.1.smithi190.stdout:8/629: mknod d9/dd/cd5 0 2024-03-26T18:54:34.259 INFO:tasks.workunit.client.1.smithi190.stdout:8/630: fsync d9/fa 0 2024-03-26T18:54:34.259 INFO:tasks.workunit.client.1.smithi190.stdout:8/631: creat d9/dd/d23/d36/d37/d3a/d61/fd6 x:0 0 0 2024-03-26T18:54:34.260 INFO:tasks.workunit.client.0.smithi102.stdout:5/522: symlink d2/d25/d2a/db4/lbe 0 2024-03-26T18:54:34.260 INFO:tasks.workunit.client.1.smithi190.stdout:0/665: rmdir d0/dd/d20/d1c/d46/d58 39 2024-03-26T18:54:34.261 INFO:tasks.workunit.client.1.smithi190.stdout:0/666: write d0/dd/d20/d1c/d46/da7/df0/da2/f9e [634553,8510] 0 2024-03-26T18:54:34.262 INFO:tasks.workunit.client.0.smithi102.stdout:1/248: mkdir de/d16/d1d/d2a/d2c/d56 0 2024-03-26T18:54:34.264 INFO:tasks.workunit.client.1.smithi190.stdout:4/614: unlink d7/d1d/d41/ld4 0 2024-03-26T18:54:34.264 INFO:tasks.workunit.client.1.smithi190.stdout:4/615: fsync d7/d1d/d42/d4a/fcc 0 2024-03-26T18:54:34.265 INFO:tasks.workunit.client.0.smithi102.stdout:7/236: truncate d7/d12/f22 155566 0 2024-03-26T18:54:34.265 INFO:tasks.workunit.client.0.smithi102.stdout:7/237: chown d7/d2a 226674279 1 2024-03-26T18:54:34.265 INFO:tasks.workunit.client.0.smithi102.stdout:7/238: chown d7/d31 236018028 1 2024-03-26T18:54:34.268 INFO:tasks.workunit.client.1.smithi190.stdout:1/539: symlink d3/d8f/lb1 0 2024-03-26T18:54:34.270 INFO:tasks.workunit.client.0.smithi102.stdout:3/445: dwrite df/d1c/d70/f87 [0,4194304] 0 2024-03-26T18:54:34.271 INFO:tasks.workunit.client.0.smithi102.stdout:0/302: dwrite d6/de/d13/d1f/d55/d57/f1c [0,4194304] 0 2024-03-26T18:54:34.271 INFO:tasks.workunit.client.0.smithi102.stdout:0/303: rename d6/de/d13/d25 to d6/de/d13/d25/d31/d63 22 2024-03-26T18:54:34.272 INFO:tasks.workunit.client.0.smithi102.stdout:7/239: dread d7/fb [0,4194304] 0 2024-03-26T18:54:34.272 INFO:tasks.workunit.client.0.smithi102.stdout:7/240: chown d7/d12/f38 5414 1 2024-03-26T18:54:34.272 INFO:tasks.workunit.client.0.smithi102.stdout:7/241: chown d7/d10/f1c 122415687 1 2024-03-26T18:54:34.272 INFO:tasks.workunit.client.0.smithi102.stdout:7/242: readlink d7/d10/d17/l1e 0 2024-03-26T18:54:34.272 INFO:tasks.workunit.client.0.smithi102.stdout:7/243: write f4 [822386,68332] 0 2024-03-26T18:54:34.272 INFO:tasks.workunit.client.0.smithi102.stdout:7/244: chown d7/d2a 1 1 2024-03-26T18:54:34.272 INFO:tasks.workunit.client.0.smithi102.stdout:1/249: creat de/d1f/f57 x:0 0 0 2024-03-26T18:54:34.272 INFO:tasks.workunit.client.0.smithi102.stdout:1/250: truncate de/d1f/f57 868738 0 2024-03-26T18:54:34.272 INFO:tasks.workunit.client.0.smithi102.stdout:1/251: write de/f22 [649008,39872] 0 2024-03-26T18:54:34.272 INFO:tasks.workunit.client.0.smithi102.stdout:1/252: chown de/d2e/f49 6837 1 2024-03-26T18:54:34.273 INFO:tasks.workunit.client.0.smithi102.stdout:1/253: dread de/d1f/f3a [0,4194304] 0 2024-03-26T18:54:34.278 INFO:tasks.workunit.client.1.smithi190.stdout:5/497: write d2/d17/d1b/d22/f9a [5184780,66191] 0 2024-03-26T18:54:34.278 INFO:tasks.workunit.client.1.smithi190.stdout:5/498: readlink d2/d17/l9f 0 2024-03-26T18:54:34.279 INFO:tasks.workunit.client.1.smithi190.stdout:7/660: mknod d3/d6/d9/d1f/d4c/cf3 0 2024-03-26T18:54:34.281 INFO:tasks.workunit.client.1.smithi190.stdout:6/551: mknod d17/d28/d8b/cbf 0 2024-03-26T18:54:34.281 INFO:tasks.workunit.client.1.smithi190.stdout:6/552: fdatasync d17/d28/d74/fae 0 2024-03-26T18:54:34.281 INFO:tasks.workunit.client.1.smithi190.stdout:6/553: creat d17/d28/d80/fc0 x:0 0 0 2024-03-26T18:54:34.281 INFO:tasks.workunit.client.1.smithi190.stdout:5/499: dread d2/d17/d62/f82 [0,4194304] 0 2024-03-26T18:54:34.296 INFO:tasks.workunit.client.1.smithi190.stdout:0/667: dwrite d0/dd/d20/d61/f7b [0,4194304] 0 2024-03-26T18:54:34.296 INFO:tasks.workunit.client.1.smithi190.stdout:6/554: dread d17/d28/d2c/f4c [0,4194304] 0 2024-03-26T18:54:34.300 INFO:tasks.workunit.client.0.smithi102.stdout:3/446: symlink df/d29/d31/d36/d3c/la2 0 2024-03-26T18:54:34.302 INFO:tasks.workunit.client.1.smithi190.stdout:2/610: rename d6/d9/d3a/d3d/l77 to d6/d9/d3a/d3d/d1c/d8d/dc7/d72/lde 0 2024-03-26T18:54:34.302 INFO:tasks.workunit.client.1.smithi190.stdout:2/611: fsync d6/f4d 0 2024-03-26T18:54:34.308 INFO:tasks.workunit.client.0.smithi102.stdout:1/254: dwrite de/f22 [0,4194304] 0 2024-03-26T18:54:34.308 INFO:tasks.workunit.client.1.smithi190.stdout:5/500: dwrite d2/f14 [4194304,4194304] 0 2024-03-26T18:54:34.308 INFO:tasks.workunit.client.1.smithi190.stdout:5/501: stat d2/d17/d1b/d22/d24/d3b/da5 0 2024-03-26T18:54:34.308 INFO:tasks.workunit.client.1.smithi190.stdout:5/502: creat d2/d17/d1b/d22/d88/d8d/fae x:0 0 0 2024-03-26T18:54:34.310 INFO:tasks.workunit.client.0.smithi102.stdout:0/304: rename d6/de/d13/d1f/f5c to d6/de/d13/d1f/f64 0 2024-03-26T18:54:34.310 INFO:tasks.workunit.client.0.smithi102.stdout:0/305: write d6/de/d13/d25/d31/f4e [113120,69871] 0 2024-03-26T18:54:34.313 INFO:tasks.workunit.client.0.smithi102.stdout:7/245: mknod d7/d12/c46 0 2024-03-26T18:54:34.315 INFO:tasks.workunit.client.1.smithi190.stdout:8/632: symlink d9/dd/d23/d88/ld7 0 2024-03-26T18:54:34.319 INFO:tasks.workunit.client.1.smithi190.stdout:4/616: mkdir d7/dc8/dd5 0 2024-03-26T18:54:34.320 INFO:tasks.workunit.client.1.smithi190.stdout:4/617: chown d7/d8/d14/d6e/d6f/f93 10168 1 2024-03-26T18:54:34.320 INFO:tasks.workunit.client.0.smithi102.stdout:3/447: dwrite df/d1c/d70/f87 [0,4194304] 0 2024-03-26T18:54:34.323 INFO:tasks.workunit.client.1.smithi190.stdout:1/540: mkdir d3/da/db2 0 2024-03-26T18:54:34.325 INFO:tasks.workunit.client.1.smithi190.stdout:7/661: rmdir d3/d6 39 2024-03-26T18:54:34.330 INFO:tasks.workunit.client.0.smithi102.stdout:1/255: creat de/d16/d1d/d2a/f58 x:0 0 0 2024-03-26T18:54:34.331 INFO:tasks.workunit.client.0.smithi102.stdout:5/523: rmdir d2 39 2024-03-26T18:54:34.331 INFO:tasks.workunit.client.0.smithi102.stdout:5/524: write d2/d25/d5b/f94 [416576,78601] 0 2024-03-26T18:54:34.333 INFO:tasks.workunit.client.1.smithi190.stdout:1/541: write d3/d5f/f71 [667380,33055] 0 2024-03-26T18:54:34.334 INFO:tasks.workunit.client.1.smithi190.stdout:1/542: dread - d3/da/dd/d95/d52/f7e zero size 2024-03-26T18:54:34.334 INFO:tasks.workunit.client.1.smithi190.stdout:0/668: creat d0/dd5/dee/ff1 x:0 0 0 2024-03-26T18:54:34.334 INFO:tasks.workunit.client.1.smithi190.stdout:2/612: dwrite d6/d9/d3a/d3d/d55/f92 [0,4194304] 0 2024-03-26T18:54:34.337 INFO:tasks.workunit.client.1.smithi190.stdout:2/613: dread d6/d9/d10/f34 [0,4194304] 0 2024-03-26T18:54:34.338 INFO:tasks.workunit.client.1.smithi190.stdout:9/856: link d12/d2d/d1d/d23/d83/dc9/dd4/d3f/fc7 d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/db3/f12b 0 2024-03-26T18:54:34.338 INFO:tasks.workunit.client.0.smithi102.stdout:3/448: dwrite df/d29/d31/d36/d3c/f6a [0,4194304] 0 2024-03-26T18:54:34.340 INFO:tasks.workunit.client.0.smithi102.stdout:1/256: unlink de/l18 0 2024-03-26T18:54:34.342 INFO:tasks.workunit.client.0.smithi102.stdout:7/246: read d7/ff [120869,17326] 0 2024-03-26T18:54:34.342 INFO:tasks.workunit.client.0.smithi102.stdout:7/247: dread - d7/d12/d2b/f2f zero size 2024-03-26T18:54:34.342 INFO:tasks.workunit.client.1.smithi190.stdout:6/555: rename d17/l20 to d17/d28/d46/lc1 0 2024-03-26T18:54:34.343 INFO:tasks.workunit.client.0.smithi102.stdout:5/525: mknod d2/d2c/d6f/cbf 0 2024-03-26T18:54:34.343 INFO:tasks.workunit.client.0.smithi102.stdout:5/526: fsync d2/d25/d75/db9/d43/f8a 0 2024-03-26T18:54:34.343 INFO:tasks.workunit.client.0.smithi102.stdout:5/527: fdatasync d2/d25/d75/db9/d43/f8a 0 2024-03-26T18:54:34.344 INFO:tasks.workunit.client.0.smithi102.stdout:2/251: sync 2024-03-26T18:54:34.347 INFO:tasks.workunit.client.0.smithi102.stdout:5/528: dread d2/f12 [0,4194304] 0 2024-03-26T18:54:34.348 INFO:tasks.workunit.client.1.smithi190.stdout:8/633: mknod d9/dd/d23/d36/d37/d3a/cd8 0 2024-03-26T18:54:34.348 INFO:tasks.workunit.client.1.smithi190.stdout:8/634: write d9/da2/f63 [4313248,114379] 0 2024-03-26T18:54:34.348 INFO:tasks.workunit.client.1.smithi190.stdout:8/635: truncate d9/d39/d67/fa3 683579 0 2024-03-26T18:54:34.348 INFO:tasks.workunit.client.0.smithi102.stdout:3/449: dread df/d61/f57 [0,4194304] 0 2024-03-26T18:54:34.350 INFO:tasks.workunit.client.1.smithi190.stdout:8/636: dread d9/dd/d23/d36/f71 [0,4194304] 0 2024-03-26T18:54:34.352 INFO:tasks.workunit.client.1.smithi190.stdout:8/637: dread d9/d39/d67/fa3 [0,4194304] 0 2024-03-26T18:54:34.354 INFO:tasks.workunit.client.1.smithi190.stdout:4/618: creat d7/fd6 x:0 0 0 2024-03-26T18:54:34.355 INFO:tasks.workunit.client.0.smithi102.stdout:1/257: mknod de/d16/d1d/d2a/d2c/d56/c59 0 2024-03-26T18:54:34.355 INFO:tasks.workunit.client.0.smithi102.stdout:5/529: dread d2/d25/f7b [0,4194304] 0 2024-03-26T18:54:34.355 INFO:tasks.workunit.client.0.smithi102.stdout:5/530: write d2/d25/f81 [539436,94123] 0 2024-03-26T18:54:34.358 INFO:tasks.workunit.client.1.smithi190.stdout:7/662: creat d3/d6/d9/d1f/d41/d65/d90/d92/d9b/ff4 x:0 0 0 2024-03-26T18:54:34.358 INFO:tasks.workunit.client.1.smithi190.stdout:7/663: truncate d3/d1b/f5e 1862213 0 2024-03-26T18:54:34.358 INFO:tasks.workunit.client.1.smithi190.stdout:7/664: dread - d3/d6/d9/d1f/d41/d65/d90/d92/fa1 zero size 2024-03-26T18:54:34.360 INFO:tasks.workunit.client.0.smithi102.stdout:7/248: creat d7/d12/d2b/f47 x:0 0 0 2024-03-26T18:54:34.362 INFO:tasks.workunit.client.1.smithi190.stdout:0/669: mknod d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d66/cf2 0 2024-03-26T18:54:34.362 INFO:tasks.workunit.client.1.smithi190.stdout:0/670: chown d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1 191 1 2024-03-26T18:54:34.367 INFO:tasks.workunit.client.1.smithi190.stdout:3/507: sync 2024-03-26T18:54:34.367 INFO:tasks.workunit.client.1.smithi190.stdout:3/508: readlink d1/d3/d5a/d6e/l82 0 2024-03-26T18:54:34.367 INFO:tasks.workunit.client.0.smithi102.stdout:2/252: dwrite d3/f4 [0,4194304] 0 2024-03-26T18:54:34.368 INFO:tasks.workunit.client.1.smithi190.stdout:2/614: rmdir d6/d9/d3a/d3d/d67/d9e/da3/dae 39 2024-03-26T18:54:34.369 INFO:tasks.workunit.client.0.smithi102.stdout:3/450: dwrite df/d29/d31/d36/d3c/d6f/f7d [0,4194304] 0 2024-03-26T18:54:34.371 INFO:tasks.workunit.client.0.smithi102.stdout:1/258: creat de/d16/d1d/d2a/d2c/d43/f5a x:0 0 0 2024-03-26T18:54:34.371 INFO:tasks.workunit.client.0.smithi102.stdout:1/259: chown de/d16/c50 402792878 1 2024-03-26T18:54:34.371 INFO:tasks.workunit.client.0.smithi102.stdout:7/249: write f6 [1634549,114989] 0 2024-03-26T18:54:34.372 INFO:tasks.workunit.client.1.smithi190.stdout:9/857: mkdir d12/d2d/d1d/d23/de5/d122/d12c 0 2024-03-26T18:54:34.373 INFO:tasks.workunit.client.0.smithi102.stdout:0/306: unlink d6/de/d13/d1f/f64 0 2024-03-26T18:54:34.375 INFO:tasks.workunit.client.0.smithi102.stdout:2/253: symlink d3/d2d/l59 0 2024-03-26T18:54:34.375 INFO:tasks.workunit.client.0.smithi102.stdout:2/254: fsync d3/d2a/f4f 0 2024-03-26T18:54:34.375 INFO:tasks.workunit.client.0.smithi102.stdout:2/255: truncate d3/d2a/f37 677455 0 2024-03-26T18:54:34.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:33 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:54:34.378 INFO:tasks.workunit.client.1.smithi190.stdout:5/503: rename d2/d17/d62/d64/f91 to d2/d17/d1b/d22/d24/d3b/faf 0 2024-03-26T18:54:34.380 INFO:tasks.workunit.client.0.smithi102.stdout:5/531: dwrite d2/d25/d75/db9/f5c [0,4194304] 0 2024-03-26T18:54:34.382 INFO:tasks.workunit.client.0.smithi102.stdout:5/532: read d2/d14/f70 [298491,14215] 0 2024-03-26T18:54:34.382 INFO:tasks.workunit.client.1.smithi190.stdout:6/556: link d17/d28/d80/f88 d17/d28/d2c/d50/fc2 0 2024-03-26T18:54:34.384 INFO:tasks.workunit.client.0.smithi102.stdout:3/451: mknod df/d1c/d34/ca3 0 2024-03-26T18:54:34.384 INFO:tasks.workunit.client.0.smithi102.stdout:3/452: write df/d1c/d34/d53/f80 [675452,118268] 0 2024-03-26T18:54:34.384 INFO:tasks.workunit.client.0.smithi102.stdout:7/250: read d7/d10/f1c [461417,124714] 0 2024-03-26T18:54:34.386 INFO:tasks.workunit.client.0.smithi102.stdout:1/260: symlink de/d16/d1d/d2a/d2c/l5b 0 2024-03-26T18:54:34.386 INFO:tasks.workunit.client.0.smithi102.stdout:1/261: dread - de/d16/f54 zero size 2024-03-26T18:54:34.386 INFO:tasks.workunit.client.0.smithi102.stdout:1/262: fdatasync de/d16/f34 0 2024-03-26T18:54:34.389 INFO:tasks.workunit.client.1.smithi190.stdout:8/638: truncate d9/da2/dc4/f66 297960 0 2024-03-26T18:54:34.391 INFO:tasks.workunit.client.1.smithi190.stdout:4/619: mknod d7/d8/cd7 0 2024-03-26T18:54:34.393 INFO:tasks.workunit.client.1.smithi190.stdout:7/665: creat d3/d1b/d57/ff5 x:0 0 0 2024-03-26T18:54:34.393 INFO:tasks.workunit.client.1.smithi190.stdout:6/557: dread fc [0,4194304] 0 2024-03-26T18:54:34.393 INFO:tasks.workunit.client.1.smithi190.stdout:7/666: write d3/d6/d9/fcd [313134,123894] 0 2024-03-26T18:54:34.396 INFO:tasks.workunit.client.0.smithi102.stdout:5/533: write d2/d14/f1c [3029628,113717] 0 2024-03-26T18:54:34.396 INFO:tasks.workunit.client.0.smithi102.stdout:5/534: fdatasync d2/d25/d2a/f59 0 2024-03-26T18:54:34.396 INFO:tasks.workunit.client.0.smithi102.stdout:5/535: read - d2/d25/d75/db9/d43/d9e/f8c zero size 2024-03-26T18:54:34.396 INFO:tasks.workunit.client.0.smithi102.stdout:5/536: fdatasync d2/d25/d75/db9/d43/d9e/d82/f97 0 2024-03-26T18:54:34.397 INFO:tasks.workunit.client.1.smithi190.stdout:0/671: link d0/dd/lbb d0/dd/d20/d1c/d46/d58/lf3 0 2024-03-26T18:54:34.401 INFO:tasks.workunit.client.0.smithi102.stdout:7/251: dread d7/f9 [0,4194304] 0 2024-03-26T18:54:34.401 INFO:tasks.workunit.client.0.smithi102.stdout:7/252: truncate d7/d10/f14 940099 0 2024-03-26T18:54:34.402 INFO:tasks.workunit.client.0.smithi102.stdout:0/307: dwrite d6/d3e/d54/f58 [0,4194304] 0 2024-03-26T18:54:34.402 INFO:tasks.workunit.client.0.smithi102.stdout:0/308: dread - d6/de/f4c zero size 2024-03-26T18:54:34.403 INFO:tasks.workunit.client.1.smithi190.stdout:2/615: unlink d6/d9/d48/f97 0 2024-03-26T18:54:34.404 INFO:tasks.workunit.client.0.smithi102.stdout:2/256: dwrite d3/d11/d15/d31/f35 [0,4194304] 0 2024-03-26T18:54:34.405 INFO:tasks.workunit.client.1.smithi190.stdout:9/858: mknod d12/d80/db2/d112/c12d 0 2024-03-26T18:54:34.407 INFO:tasks.workunit.client.1.smithi190.stdout:1/543: rename d3/da/l91 to d3/d9/d9f/lb3 0 2024-03-26T18:54:34.408 INFO:tasks.workunit.client.1.smithi190.stdout:5/504: creat d2/fb0 x:0 0 0 2024-03-26T18:54:34.408 INFO:tasks.workunit.client.1.smithi190.stdout:5/505: stat d2/d17/d98/f2e 0 2024-03-26T18:54:34.408 INFO:tasks.workunit.client.1.smithi190.stdout:5/506: fdatasync d2/d17/d62/d64/f70 0 2024-03-26T18:54:34.409 INFO:tasks.workunit.client.0.smithi102.stdout:3/453: creat df/d1c/d34/d53/fa4 x:0 0 0 2024-03-26T18:54:34.410 INFO:tasks.workunit.client.0.smithi102.stdout:1/263: truncate de/d1f/f33 3862148 0 2024-03-26T18:54:34.410 INFO:tasks.workunit.client.0.smithi102.stdout:1/264: dread - de/d2e/f55 zero size 2024-03-26T18:54:34.411 INFO:tasks.workunit.client.0.smithi102.stdout:1/265: chown de/d16/d1d/d2a/d2c/d56/c59 5132 1 2024-03-26T18:54:34.411 INFO:tasks.workunit.client.1.smithi190.stdout:8/639: creat d9/dd/d23/d36/d37/db8/fd9 x:0 0 0 2024-03-26T18:54:34.414 INFO:tasks.workunit.client.1.smithi190.stdout:4/620: unlink d7/d1d/d42/d4a/fcc 0 2024-03-26T18:54:34.415 INFO:tasks.workunit.client.0.smithi102.stdout:7/253: unlink f4 0 2024-03-26T18:54:34.416 INFO:tasks.workunit.client.1.smithi190.stdout:1/544: dread d3/d5f/f71 [0,4194304] 0 2024-03-26T18:54:34.416 INFO:tasks.workunit.client.1.smithi190.stdout:1/545: write d3/d5f/f71 [629290,77407] 0 2024-03-26T18:54:34.416 INFO:tasks.workunit.client.1.smithi190.stdout:1/546: creat d3/d9/db/fb4 x:0 0 0 2024-03-26T18:54:34.417 INFO:tasks.workunit.client.1.smithi190.stdout:7/667: mkdir d3/d6/d9/d1f/d4c/d71/df6 0 2024-03-26T18:54:34.419 INFO:tasks.workunit.client.1.smithi190.stdout:0/672: truncate d0/dd/d20/d1c/d46/da7/df0/d4b/f89 890907 0 2024-03-26T18:54:34.420 INFO:tasks.workunit.client.0.smithi102.stdout:0/309: creat d6/de/d13/d25/d31/d45/f65 x:0 0 0 2024-03-26T18:54:34.421 INFO:tasks.workunit.client.0.smithi102.stdout:9/231: sync 2024-03-26T18:54:34.421 INFO:tasks.workunit.client.0.smithi102.stdout:9/232: getdents df/d1a/d33 0 2024-03-26T18:54:34.423 INFO:tasks.workunit.client.1.smithi190.stdout:3/509: getdents d1/de/d4e 0 2024-03-26T18:54:34.423 INFO:tasks.workunit.client.0.smithi102.stdout:3/454: dread df/d29/f33 [0,4194304] 0 2024-03-26T18:54:34.426 INFO:tasks.workunit.client.1.smithi190.stdout:2/616: symlink d6/d9/d3a/d3d/d67/ldf 0 2024-03-26T18:54:34.426 INFO:tasks.workunit.client.1.smithi190.stdout:2/617: truncate d6/d9/f41 1116789 0 2024-03-26T18:54:34.429 INFO:tasks.workunit.client.1.smithi190.stdout:2/618: chown d6/d9/d3a/d3d/d1c/f54 812983 1 2024-03-26T18:54:34.429 INFO:tasks.workunit.client.1.smithi190.stdout:2/619: creat d6/d9/d45/fe0 x:0 0 0 2024-03-26T18:54:34.430 INFO:tasks.workunit.client.1.smithi190.stdout:9/859: creat d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/f12e x:0 0 0 2024-03-26T18:54:34.430 INFO:tasks.workunit.client.1.smithi190.stdout:9/860: write d12/d2d/d1d/d23/d2b/da0/f116 [31651,115990] 0 2024-03-26T18:54:34.430 INFO:tasks.workunit.client.1.smithi190.stdout:9/861: chown d12/d2d/d1d/d54/d92/de3 77 1 2024-03-26T18:54:34.430 INFO:tasks.workunit.client.1.smithi190.stdout:9/862: creat d12/d2d/d1d/da1/dba/f12f x:0 0 0 2024-03-26T18:54:34.431 INFO:tasks.workunit.client.1.smithi190.stdout:5/507: creat d2/d17/d1b/d30/fb1 x:0 0 0 2024-03-26T18:54:34.431 INFO:tasks.workunit.client.1.smithi190.stdout:5/508: chown d2/l18 80165714 1 2024-03-26T18:54:34.432 INFO:tasks.workunit.client.0.smithi102.stdout:1/266: symlink de/d1f/l5c 0 2024-03-26T18:54:34.432 INFO:tasks.workunit.client.0.smithi102.stdout:1/267: creat de/d16/d1d/d2a/f5d x:0 0 0 2024-03-26T18:54:34.433 INFO:tasks.workunit.client.1.smithi190.stdout:9/863: dread d12/d2d/d1d/d23/d2b/f95 [0,4194304] 0 2024-03-26T18:54:34.436 INFO:tasks.workunit.client.1.smithi190.stdout:4/621: unlink d7/d8/d14/d23/d28/d38/l9d 0 2024-03-26T18:54:34.436 INFO:tasks.workunit.client.1.smithi190.stdout:4/622: read - d7/d1d/d42/fb7 zero size 2024-03-26T18:54:34.436 INFO:tasks.workunit.client.0.smithi102.stdout:7/254: creat d7/d10/d27/d34/f48 x:0 0 0 2024-03-26T18:54:34.438 INFO:tasks.workunit.client.1.smithi190.stdout:8/640: dwrite d9/dd/d23/d36/d37/d3a/d61/fd6 [0,4194304] 0 2024-03-26T18:54:34.438 INFO:tasks.workunit.client.1.smithi190.stdout:8/641: chown d9 1355919 1 2024-03-26T18:54:34.438 INFO:tasks.workunit.client.1.smithi190.stdout:8/642: write d9/dd/d23/d36/d37/f8d [893464,28272] 0 2024-03-26T18:54:34.441 INFO:tasks.workunit.client.1.smithi190.stdout:6/558: mknod d17/cc3 0 2024-03-26T18:54:34.441 INFO:tasks.workunit.client.1.smithi190.stdout:6/559: fdatasync d17/d28/d2c/d63/f70 0 2024-03-26T18:54:34.441 INFO:tasks.workunit.client.1.smithi190.stdout:6/560: readlink d17/d28/d8b/l9f 0 2024-03-26T18:54:34.441 INFO:tasks.workunit.client.0.smithi102.stdout:9/233: rename df/d1a/d23/d29/l36 to df/d1a/d23/d4e/l5a 0 2024-03-26T18:54:34.442 INFO:tasks.workunit.client.0.smithi102.stdout:7/255: read d7/d2a/f30 [768421,113708] 0 2024-03-26T18:54:34.442 INFO:tasks.workunit.client.0.smithi102.stdout:9/234: dread - df/d1a/d23/d29/f34 zero size 2024-03-26T18:54:34.442 INFO:tasks.workunit.client.0.smithi102.stdout:9/235: write df/d1a/d23/d29/f31 [4720206,117932] 0 2024-03-26T18:54:34.447 INFO:tasks.workunit.client.1.smithi190.stdout:1/547: dwrite d3/f53 [0,4194304] 0 2024-03-26T18:54:34.451 INFO:tasks.workunit.client.0.smithi102.stdout:3/455: mkdir df/d1c/d34/da5 0 2024-03-26T18:54:34.453 INFO:tasks.workunit.client.1.smithi190.stdout:7/668: rename d3/d6/d9/d1f/d41/d65/d90/d92/d9b/deb/ldd to d3/d6/d9/d1f/d41/d65/d90/d92/d9b/deb/lf7 0 2024-03-26T18:54:34.453 INFO:tasks.workunit.client.0.smithi102.stdout:2/257: rmdir d3/d20 39 2024-03-26T18:54:34.454 INFO:tasks.workunit.client.1.smithi190.stdout:0/673: rmdir d0/dd 39 2024-03-26T18:54:34.456 INFO:tasks.workunit.client.0.smithi102.stdout:4/220: sync 2024-03-26T18:54:34.456 INFO:tasks.workunit.client.0.smithi102.stdout:8/302: sync 2024-03-26T18:54:34.456 INFO:tasks.workunit.client.0.smithi102.stdout:8/303: chown d0/d3/d31 2 1 2024-03-26T18:54:34.456 INFO:tasks.workunit.client.0.smithi102.stdout:4/221: write d1/f31 [942294,66076] 0 2024-03-26T18:54:34.456 INFO:tasks.workunit.client.0.smithi102.stdout:6/281: sync 2024-03-26T18:54:34.456 INFO:tasks.workunit.client.0.smithi102.stdout:6/282: chown d2/d7/d3c/f3e 12919 1 2024-03-26T18:54:34.459 INFO:tasks.workunit.client.0.smithi102.stdout:1/268: rename de/d2e/f4a to de/d16/f5e 0 2024-03-26T18:54:34.462 INFO:tasks.workunit.client.1.smithi190.stdout:2/620: rmdir d6/d9/d3a/d3d/d67/d9e/dd6 0 2024-03-26T18:54:34.463 INFO:tasks.workunit.client.0.smithi102.stdout:0/310: dread d6/d2d/d3d/f44 [0,4194304] 0 2024-03-26T18:54:34.463 INFO:tasks.workunit.client.1.smithi190.stdout:6/561: dread d17/d28/d2c/f4c [0,4194304] 0 2024-03-26T18:54:34.465 INFO:tasks.workunit.client.0.smithi102.stdout:9/236: truncate df/f17 4114550 0 2024-03-26T18:54:34.468 INFO:tasks.workunit.client.0.smithi102.stdout:3/456: unlink df/d61/f57 0 2024-03-26T18:54:34.471 INFO:tasks.workunit.client.1.smithi190.stdout:5/509: unlink d2/d17/d1b/d30/f7a 0 2024-03-26T18:54:34.471 INFO:tasks.workunit.client.1.smithi190.stdout:5/510: write d2/d17/d1b/f67 [49264,17835] 0 2024-03-26T18:54:34.471 INFO:tasks.workunit.client.1.smithi190.stdout:5/511: chown d2/f45 1 1 2024-03-26T18:54:34.471 INFO:tasks.workunit.client.1.smithi190.stdout:8/643: dwrite d9/dd/d23/d36/d37/d3a/d43/f80 [0,4194304] 0 2024-03-26T18:54:34.471 INFO:tasks.workunit.client.1.smithi190.stdout:8/644: chown d9/dd/d23/d36/d37/d3a/l44 300 1 2024-03-26T18:54:34.472 INFO:tasks.workunit.client.0.smithi102.stdout:7/256: dwrite d7/d12/f43 [0,4194304] 0 2024-03-26T18:54:34.472 INFO:tasks.workunit.client.0.smithi102.stdout:7/257: fdatasync d7/d2a/f30 0 2024-03-26T18:54:34.474 INFO:tasks.workunit.client.1.smithi190.stdout:9/864: dwrite d12/d2d/f1b [0,4194304] 0 2024-03-26T18:54:34.476 INFO:tasks.workunit.client.0.smithi102.stdout:8/304: unlink d0/d3/d2c/d33/f54 0 2024-03-26T18:54:34.478 INFO:tasks.workunit.client.1.smithi190.stdout:4/623: creat d7/d1d/d41/d74/fd8 x:0 0 0 2024-03-26T18:54:34.479 INFO:tasks.workunit.client.0.smithi102.stdout:6/283: symlink d2/d3d/l55 0 2024-03-26T18:54:34.482 INFO:tasks.workunit.client.0.smithi102.stdout:4/222: mknod d1/d17/d1a/d21/c4a 0 2024-03-26T18:54:34.485 INFO:tasks.workunit.client.0.smithi102.stdout:2/258: rename d3/d11/d15/d31/f35 to d3/d11/d15/d31/d39/f5a 0 2024-03-26T18:54:34.487 INFO:tasks.workunit.client.0.smithi102.stdout:1/269: mkdir de/d16/d1d/d2a/d2c/d56/d5f 0 2024-03-26T18:54:34.489 INFO:tasks.workunit.client.0.smithi102.stdout:0/311: mkdir d6/de/d13/d25/d4f/d66 0 2024-03-26T18:54:34.489 INFO:tasks.workunit.client.0.smithi102.stdout:0/312: chown d6/de/d13/d25 69454 1 2024-03-26T18:54:34.489 INFO:tasks.workunit.client.1.smithi190.stdout:4/624: dread d7/d8/d14/d5c/f63 [0,4194304] 0 2024-03-26T18:54:34.491 INFO:tasks.workunit.client.0.smithi102.stdout:9/237: mknod df/d1a/d57/c5b 0 2024-03-26T18:54:34.491 INFO:tasks.workunit.client.0.smithi102.stdout:9/238: read - df/f27 zero size 2024-03-26T18:54:34.495 INFO:tasks.workunit.client.1.smithi190.stdout:1/548: mkdir d3/da/dd/d2f/d5b/db5 0 2024-03-26T18:54:34.495 INFO:tasks.workunit.client.1.smithi190.stdout:1/549: truncate d3/da/d3a/f50 428669 0 2024-03-26T18:54:34.495 INFO:tasks.workunit.client.1.smithi190.stdout:1/550: write d3/fab [750815,108400] 0 2024-03-26T18:54:34.499 INFO:tasks.workunit.client.1.smithi190.stdout:0/674: creat d0/dd/d20/d1c/d46/da7/df0/db4/ff4 x:0 0 0 2024-03-26T18:54:34.501 INFO:tasks.workunit.client.0.smithi102.stdout:8/305: creat d0/d15/d5d/f61 x:0 0 0 2024-03-26T18:54:34.503 INFO:tasks.workunit.client.0.smithi102.stdout:3/457: dwrite df/f9c [0,4194304] 0 2024-03-26T18:54:34.503 INFO:tasks.workunit.client.1.smithi190.stdout:2/621: unlink d6/d9/d48/f65 0 2024-03-26T18:54:34.503 INFO:tasks.workunit.client.1.smithi190.stdout:2/622: chown d6/c30 74 1 2024-03-26T18:54:34.504 INFO:tasks.workunit.client.0.smithi102.stdout:6/284: mknod d2/d3d/d53/c56 0 2024-03-26T18:54:34.506 INFO:tasks.workunit.client.1.smithi190.stdout:6/562: link d17/d28/d46/f7f d17/d28/d2b/fc4 0 2024-03-26T18:54:34.507 INFO:tasks.workunit.client.0.smithi102.stdout:8/306: dread d0/d9/f2b [0,4194304] 0 2024-03-26T18:54:34.507 INFO:tasks.workunit.client.0.smithi102.stdout:8/307: truncate d0/d15/d5d/f61 395979 0 2024-03-26T18:54:34.508 INFO:tasks.workunit.client.0.smithi102.stdout:2/259: mknod d3/d2d/c5b 0 2024-03-26T18:54:34.509 INFO:tasks.workunit.client.0.smithi102.stdout:1/270: chown la 10 1 2024-03-26T18:54:34.511 INFO:tasks.workunit.client.0.smithi102.stdout:7/258: dwrite d7/d10/d27/f33 [0,4194304] 0 2024-03-26T18:54:34.513 INFO:tasks.workunit.client.0.smithi102.stdout:9/239: mknod df/d46/c5c 0 2024-03-26T18:54:34.513 INFO:tasks.workunit.client.0.smithi102.stdout:9/240: write df/f27 [366736,53532] 0 2024-03-26T18:54:34.513 INFO:tasks.workunit.client.0.smithi102.stdout:9/241: readlink df/d1a/d1d/l1f 0 2024-03-26T18:54:34.513 INFO:tasks.workunit.client.0.smithi102.stdout:9/242: creat df/d1a/d23/f5d x:0 0 0 2024-03-26T18:54:34.513 INFO:tasks.workunit.client.0.smithi102.stdout:9/243: creat df/d1a/f5e x:0 0 0 2024-03-26T18:54:34.513 INFO:tasks.workunit.client.0.smithi102.stdout:9/244: creat df/d1a/d23/d29/d45/f5f x:0 0 0 2024-03-26T18:54:34.516 INFO:tasks.workunit.client.0.smithi102.stdout:3/458: dread df/d29/d31/d36/d3c/d6f/f7d [0,4194304] 0 2024-03-26T18:54:34.516 INFO:tasks.workunit.client.0.smithi102.stdout:6/285: creat d2/df/d35/d43/f57 x:0 0 0 2024-03-26T18:54:34.521 INFO:tasks.workunit.client.0.smithi102.stdout:8/308: rename d0/d3/d26/d19/f51 to d0/d3/d26/d5c/f62 0 2024-03-26T18:54:34.524 INFO:tasks.workunit.client.0.smithi102.stdout:3/459: read df/d29/f64 [882014,91495] 0 2024-03-26T18:54:34.524 INFO:tasks.workunit.client.0.smithi102.stdout:3/460: creat df/d1c/d70/fa6 x:0 0 0 2024-03-26T18:54:34.525 INFO:tasks.workunit.client.0.smithi102.stdout:3/461: fsync df/d61/f58 0 2024-03-26T18:54:34.525 INFO:tasks.workunit.client.0.smithi102.stdout:4/223: dwrite d1/f9 [0,4194304] 0 2024-03-26T18:54:34.525 INFO:tasks.workunit.client.0.smithi102.stdout:4/224: stat d1/l1e 0 2024-03-26T18:54:34.526 INFO:tasks.workunit.client.0.smithi102.stdout:4/225: creat d1/d17/d1a/f4b x:0 0 0 2024-03-26T18:54:34.526 INFO:tasks.workunit.client.0.smithi102.stdout:2/260: symlink d3/d11/d15/d46/l5c 0 2024-03-26T18:54:34.526 INFO:tasks.workunit.client.0.smithi102.stdout:0/313: dwrite d6/de/d13/d25/d31/f4e [0,4194304] 0 2024-03-26T18:54:34.526 INFO:tasks.workunit.client.1.smithi190.stdout:5/512: write d2/f4 [1039855,75100] 0 2024-03-26T18:54:34.528 INFO:tasks.workunit.client.0.smithi102.stdout:1/271: creat de/d16/d1d/d2a/d2c/f60 x:0 0 0 2024-03-26T18:54:34.528 INFO:tasks.workunit.client.0.smithi102.stdout:1/272: chown de/f22 58440687 1 2024-03-26T18:54:34.533 INFO:tasks.workunit.client.1.smithi190.stdout:8/645: creat d9/dd/d23/d36/d37/d3a/d43/d8f/dd2/fda x:0 0 0 2024-03-26T18:54:34.533 INFO:tasks.workunit.client.1.smithi190.stdout:8/646: chown d9/dd/d23/d36/d37/d3a/d43/la4 35 1 2024-03-26T18:54:34.533 INFO:tasks.workunit.client.1.smithi190.stdout:8/647: chown d9/dd/d23/d36/d37/d3a/d43/l64 704340 1 2024-03-26T18:54:34.533 INFO:tasks.workunit.client.1.smithi190.stdout:8/648: fsync d9/dd/d23/d36/d37/d3a/d61/fbd 0 2024-03-26T18:54:34.535 INFO:tasks.workunit.client.0.smithi102.stdout:6/286: dwrite d2/d3/f38 [4194304,4194304] 0 2024-03-26T18:54:34.539 INFO:tasks.workunit.client.1.smithi190.stdout:9/865: mkdir d12/d2d/d1d/d23/d130 0 2024-03-26T18:54:34.539 INFO:tasks.workunit.client.1.smithi190.stdout:9/866: truncate d12/d2d/d1d/d23/d83/dc9/dd4/d71/f8b 562706 0 2024-03-26T18:54:34.539 INFO:tasks.workunit.client.1.smithi190.stdout:9/867: creat d12/d2d/d1d/d23/d83/dc9/dd4/f131 x:0 0 0 2024-03-26T18:54:34.542 INFO:tasks.workunit.client.0.smithi102.stdout:3/462: dwrite df/d29/d31/d36/f48 [0,4194304] 0 2024-03-26T18:54:34.544 INFO:tasks.workunit.client.1.smithi190.stdout:4/625: creat d7/dc8/fd9 x:0 0 0 2024-03-26T18:54:34.544 INFO:tasks.workunit.client.1.smithi190.stdout:4/626: fdatasync d7/d1d/d41/f91 0 2024-03-26T18:54:34.544 INFO:tasks.workunit.client.1.smithi190.stdout:4/627: write d7/d8/d1c/d20/f58 [4839024,109121] 0 2024-03-26T18:54:34.545 INFO:tasks.workunit.client.1.smithi190.stdout:0/675: mkdir d0/dd5/dee/df5 0 2024-03-26T18:54:34.546 INFO:tasks.workunit.client.0.smithi102.stdout:8/309: link d0/d3/d26/d19/d27/l30 d0/l63 0 2024-03-26T18:54:34.548 INFO:tasks.workunit.client.1.smithi190.stdout:2/623: creat d6/dbc/fe1 x:0 0 0 2024-03-26T18:54:34.548 INFO:tasks.workunit.client.1.smithi190.stdout:2/624: write d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d7c/f7d [1187519,78733] 0 2024-03-26T18:54:34.549 INFO:tasks.workunit.client.1.smithi190.stdout:6/563: mknod d17/d28/d38/cc5 0 2024-03-26T18:54:34.555 INFO:tasks.workunit.client.0.smithi102.stdout:4/226: mkdir d1/d25/d39/d3e/d4c 0 2024-03-26T18:54:34.558 INFO:tasks.workunit.client.0.smithi102.stdout:2/261: mknod d3/d11/d15/d31/d58/c5d 0 2024-03-26T18:54:34.559 INFO:tasks.workunit.client.1.smithi190.stdout:5/513: unlink d2/d17/d62/d64/f74 0 2024-03-26T18:54:34.562 INFO:tasks.workunit.client.1.smithi190.stdout:8/649: symlink d9/d39/d67/dc1/ldb 0 2024-03-26T18:54:34.562 INFO:tasks.workunit.client.1.smithi190.stdout:6/564: dread d17/d28/d2c/f35 [0,4194304] 0 2024-03-26T18:54:34.564 INFO:tasks.workunit.client.0.smithi102.stdout:0/314: symlink d6/de/d13/d1f/l67 0 2024-03-26T18:54:34.564 INFO:tasks.workunit.client.0.smithi102.stdout:0/315: dread - d6/d2d/d3d/f40 zero size 2024-03-26T18:54:34.565 INFO:tasks.workunit.client.1.smithi190.stdout:8/650: dread d9/d48/f58 [0,4194304] 0 2024-03-26T18:54:34.567 INFO:tasks.workunit.client.1.smithi190.stdout:9/868: mknod d12/d2d/d1d/d23/de5/c132 0 2024-03-26T18:54:34.567 INFO:tasks.workunit.client.1.smithi190.stdout:9/869: truncate d12/d6d/f7d 425795 0 2024-03-26T18:54:34.568 INFO:tasks.workunit.client.0.smithi102.stdout:1/273: mkdir de/d16/d1d/d61 0 2024-03-26T18:54:34.570 INFO:tasks.workunit.client.1.smithi190.stdout:4/628: link d7/d1d/d41/d74/fd8 d7/d8/d14/d5c/fda 0 2024-03-26T18:54:34.570 INFO:tasks.workunit.client.1.smithi190.stdout:4/629: write d7/d1d/d42/fbe [2543448,127547] 0 2024-03-26T18:54:34.579 INFO:tasks.workunit.client.0.smithi102.stdout:7/259: rmdir d7/d12 39 2024-03-26T18:54:34.579 INFO:tasks.workunit.client.0.smithi102.stdout:7/260: chown d7/d12/c46 26696437 1 2024-03-26T18:54:34.584 INFO:tasks.workunit.client.1.smithi190.stdout:0/676: mknod d0/cf6 0 2024-03-26T18:54:34.584 INFO:tasks.workunit.client.1.smithi190.stdout:8/651: dread d9/dd/d23/d36/d37/d3a/d43/f80 [0,4194304] 0 2024-03-26T18:54:34.585 INFO:tasks.workunit.client.0.smithi102.stdout:6/287: mknod d2/df/d35/c58 0 2024-03-26T18:54:34.587 INFO:tasks.workunit.client.1.smithi190.stdout:9/870: write d12/f38 [2028278,9014] 0 2024-03-26T18:54:34.588 INFO:tasks.workunit.client.0.smithi102.stdout:9/245: truncate df/f17 1205254 0 2024-03-26T18:54:34.589 INFO:tasks.workunit.client.0.smithi102.stdout:1/274: write de/f32 [813183,128105] 0 2024-03-26T18:54:34.590 INFO:tasks.workunit.client.1.smithi190.stdout:2/625: creat d6/d9/fe2 x:0 0 0 2024-03-26T18:54:34.590 INFO:tasks.workunit.client.0.smithi102.stdout:7/261: dread d7/d12/f3c [0,4194304] 0 2024-03-26T18:54:34.593 INFO:tasks.workunit.client.0.smithi102.stdout:3/463: rename df/d1c/d70/d9e to df/d29/d31/d36/d3c/da7 0 2024-03-26T18:54:34.595 INFO:tasks.workunit.client.0.smithi102.stdout:2/262: mknod d3/d6/d1a/d25/c5e 0 2024-03-26T18:54:34.597 INFO:tasks.workunit.client.1.smithi190.stdout:5/514: unlink d2/d17/d98/f97 0 2024-03-26T18:54:34.598 INFO:tasks.workunit.client.1.smithi190.stdout:5/515: dread d2/d17/d62/f82 [0,4194304] 0 2024-03-26T18:54:34.598 INFO:tasks.workunit.client.0.smithi102.stdout:0/316: creat d6/de/d13/d4a/f68 x:0 0 0 2024-03-26T18:54:34.598 INFO:tasks.workunit.client.0.smithi102.stdout:0/317: chown d6/de/d13/d1f/f3b 81877 1 2024-03-26T18:54:34.598 INFO:tasks.workunit.client.0.smithi102.stdout:0/318: truncate d6/de/d13/d1f/f3b 9177599 0 2024-03-26T18:54:34.599 INFO:tasks.workunit.client.0.smithi102.stdout:4/227: dwrite d1/f31 [0,4194304] 0 2024-03-26T18:54:34.600 INFO:tasks.workunit.client.1.smithi190.stdout:4/630: dwrite d7/d8/d14/d4b/d7d/f89 [0,4194304] 0 2024-03-26T18:54:34.601 INFO:tasks.workunit.client.0.smithi102.stdout:6/288: link d2/d7/l1a d2/d7/l59 0 2024-03-26T18:54:34.601 INFO:tasks.workunit.client.0.smithi102.stdout:0/319: dread d6/de/d13/f3f [0,4194304] 0 2024-03-26T18:54:34.601 INFO:tasks.workunit.client.0.smithi102.stdout:6/289: write d2/d7/d3c/f41 [425617,1351] 0 2024-03-26T18:54:34.601 INFO:tasks.workunit.client.0.smithi102.stdout:0/320: dread - d6/de/d13/d1f/d55/d57/f46 zero size 2024-03-26T18:54:34.601 INFO:tasks.workunit.client.0.smithi102.stdout:0/321: chown d6/l10 7 1 2024-03-26T18:54:34.601 INFO:tasks.workunit.client.0.smithi102.stdout:0/322: write d6/de/d13/d1f/f27 [2005339,49327] 0 2024-03-26T18:54:34.602 INFO:tasks.workunit.client.1.smithi190.stdout:0/677: rename d0/dd/d20/d1c/d8b/l31 to d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d66/lf7 0 2024-03-26T18:54:34.603 INFO:tasks.workunit.client.1.smithi190.stdout:0/678: chown d0/l3 22445 1 2024-03-26T18:54:34.603 INFO:tasks.workunit.client.1.smithi190.stdout:0/679: dread - d0/dd/d20/d1c/d8b/fd9 zero size 2024-03-26T18:54:34.603 INFO:tasks.workunit.client.1.smithi190.stdout:6/565: dwrite d17/d28/d2c/d77/f8d [0,4194304] 0 2024-03-26T18:54:34.604 INFO:tasks.workunit.client.0.smithi102.stdout:9/246: mkdir df/d1a/d23/d60 0 2024-03-26T18:54:34.604 INFO:tasks.workunit.client.0.smithi102.stdout:9/247: fsync df/d1a/d23/d4e/f4f 0 2024-03-26T18:54:34.607 INFO:tasks.workunit.client.0.smithi102.stdout:1/275: mkdir de/d2e/d62 0 2024-03-26T18:54:34.607 INFO:tasks.workunit.client.0.smithi102.stdout:1/276: fsync de/d16/d1d/d2a/d2c/f60 0 2024-03-26T18:54:34.607 INFO:tasks.workunit.client.0.smithi102.stdout:1/277: dread - de/d2e/f55 zero size 2024-03-26T18:54:34.608 INFO:tasks.workunit.client.1.smithi190.stdout:8/652: truncate d9/da2/f1e 2206581 0 2024-03-26T18:54:34.608 INFO:tasks.workunit.client.1.smithi190.stdout:8/653: write d9/d39/f6d [41539,69617] 0 2024-03-26T18:54:34.609 INFO:tasks.workunit.client.1.smithi190.stdout:2/626: write d6/d9/d10/d5f/f7f [2144679,43848] 0 2024-03-26T18:54:34.609 INFO:tasks.workunit.client.1.smithi190.stdout:2/627: chown d6/d9/d3a/f94 66 1 2024-03-26T18:54:34.611 INFO:tasks.workunit.client.0.smithi102.stdout:7/262: unlink d7/l3a 0 2024-03-26T18:54:34.614 INFO:tasks.workunit.client.1.smithi190.stdout:9/871: creat d12/d2d/d1d/d23/d8e/d9b/f133 x:0 0 0 2024-03-26T18:54:34.616 INFO:tasks.workunit.client.1.smithi190.stdout:0/680: dread d0/dd/d20/d1c/d46/da7/df0/da2/fa5 [0,4194304] 0 2024-03-26T18:54:34.616 INFO:tasks.workunit.client.1.smithi190.stdout:0/681: fsync d0/dd/d20/d1c/d46/da7/df0/db4/fd7 0 2024-03-26T18:54:34.616 INFO:tasks.workunit.client.1.smithi190.stdout:0/682: truncate d0/dd/d20/d1c/d46/da7/df0/db4/fd7 625514 0 2024-03-26T18:54:34.618 INFO:tasks.workunit.client.1.smithi190.stdout:5/516: mknod d2/d17/d1b/d60/d63/cb2 0 2024-03-26T18:54:34.618 INFO:tasks.workunit.client.1.smithi190.stdout:5/517: write d2/d17/d62/fad [758905,87625] 0 2024-03-26T18:54:34.618 INFO:tasks.workunit.client.0.smithi102.stdout:3/464: unlink df/d29/d3a/l9d 0 2024-03-26T18:54:34.620 INFO:tasks.workunit.client.0.smithi102.stdout:8/310: symlink d0/d3/d26/l64 0 2024-03-26T18:54:34.623 INFO:tasks.workunit.client.0.smithi102.stdout:2/263: mkdir d3/d2a/d5f 0 2024-03-26T18:54:34.625 INFO:tasks.workunit.client.1.smithi190.stdout:8/654: write d9/da2/f51 [345410,124384] 0 2024-03-26T18:54:34.629 INFO:tasks.workunit.client.0.smithi102.stdout:4/228: dwrite d1/d27/f3c [0,4194304] 0 2024-03-26T18:54:34.630 INFO:tasks.workunit.client.1.smithi190.stdout:4/631: mknod d7/d8/d14/d6e/d6f/cdb 0 2024-03-26T18:54:34.634 INFO:tasks.workunit.client.1.smithi190.stdout:3/510: sync 2024-03-26T18:54:34.634 INFO:tasks.workunit.client.1.smithi190.stdout:3/511: chown d1/d3/df/d18/d35 1118 1 2024-03-26T18:54:34.634 INFO:tasks.workunit.client.1.smithi190.stdout:3/512: readlink d1/l1e 0 2024-03-26T18:54:34.637 INFO:tasks.workunit.client.1.smithi190.stdout:2/628: unlink d6/d9/d3a/d3d/d1c/cc3 0 2024-03-26T18:54:34.637 INFO:tasks.workunit.client.1.smithi190.stdout:2/629: fsync d6/d9/d3a/d3d/d1c/d8d/dc7/f26 0 2024-03-26T18:54:34.638 INFO:tasks.workunit.client.1.smithi190.stdout:1/551: sync 2024-03-26T18:54:34.638 INFO:tasks.workunit.client.1.smithi190.stdout:7/669: sync 2024-03-26T18:54:34.640 INFO:tasks.workunit.client.1.smithi190.stdout:9/872: symlink d12/d2d/d1d/d23/d83/dc9/dd4/d71/l134 0 2024-03-26T18:54:34.640 INFO:tasks.workunit.client.1.smithi190.stdout:9/873: chown d12/d2d/d1d/d23/d83/dc9/dd4/d3f/f113 167 1 2024-03-26T18:54:34.643 INFO:tasks.workunit.client.1.smithi190.stdout:0/683: rename d0/dd/d20/d1c/d8b/fc3 to d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/ff8 0 2024-03-26T18:54:34.643 INFO:tasks.workunit.client.1.smithi190.stdout:0/684: creat d0/dd/d20/d1c/d46/da7/ff9 x:0 0 0 2024-03-26T18:54:34.643 INFO:tasks.workunit.client.1.smithi190.stdout:0/685: dread - d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/ff8 zero size 2024-03-26T18:54:34.643 INFO:tasks.workunit.client.1.smithi190.stdout:0/686: write d0/dd/d20/d1c/d4e/f9d [1630967,8422] 0 2024-03-26T18:54:34.645 INFO:tasks.workunit.client.1.smithi190.stdout:5/518: creat d2/d17/d1b/d60/d63/d85/d89/fb3 x:0 0 0 2024-03-26T18:54:34.647 INFO:tasks.workunit.client.0.smithi102.stdout:6/290: mkdir d2/d5a 0 2024-03-26T18:54:34.647 INFO:tasks.workunit.client.0.smithi102.stdout:6/291: write d2/d7/f51 [1029951,33212] 0 2024-03-26T18:54:34.647 INFO:tasks.workunit.client.0.smithi102.stdout:6/292: dread - d2/f34 zero size 2024-03-26T18:54:34.647 INFO:tasks.workunit.client.0.smithi102.stdout:6/293: fsync d2/d3/f2a 0 2024-03-26T18:54:34.649 INFO:tasks.workunit.client.1.smithi190.stdout:6/566: getdents d17/d28/d74/d97 0 2024-03-26T18:54:34.649 INFO:tasks.workunit.client.0.smithi102.stdout:0/323: mkdir d6/d3e/d69 0 2024-03-26T18:54:34.650 INFO:tasks.workunit.client.1.smithi190.stdout:3/513: symlink d1/de/l98 0 2024-03-26T18:54:34.650 INFO:tasks.workunit.client.1.smithi190.stdout:4/632: dread d7/d8/d14/d23/d28/f8a [0,4194304] 0 2024-03-26T18:54:34.652 INFO:tasks.workunit.client.1.smithi190.stdout:6/567: dread d17/d28/d46/f7f [0,4194304] 0 2024-03-26T18:54:34.653 INFO:tasks.workunit.client.1.smithi190.stdout:2/630: link d6/d9/d3a/d3d/d2e/d31/f91 d6/d9/d45/d6b/d93/dc0/fe3 0 2024-03-26T18:54:34.653 INFO:tasks.workunit.client.1.smithi190.stdout:2/631: chown d6/d9/d3a/d3d/d1c/d8d/dc7/l2a 6364168 1 2024-03-26T18:54:34.653 INFO:tasks.workunit.client.0.smithi102.stdout:1/278: dwrite de/d1f/f53 [0,4194304] 0 2024-03-26T18:54:34.653 INFO:tasks.workunit.client.0.smithi102.stdout:1/279: write de/f22 [690403,56498] 0 2024-03-26T18:54:34.653 INFO:tasks.workunit.client.0.smithi102.stdout:1/280: write de/d16/d1d/d2a/f5d [14134,51788] 0 2024-03-26T18:54:34.655 INFO:tasks.workunit.client.1.smithi190.stdout:7/670: mknod d3/d1b/cf8 0 2024-03-26T18:54:34.655 INFO:tasks.workunit.client.1.smithi190.stdout:7/671: chown c1 12120 1 2024-03-26T18:54:34.660 INFO:tasks.workunit.client.0.smithi102.stdout:6/294: creat d2/d3d/d53/f5b x:0 0 0 2024-03-26T18:54:34.661 INFO:tasks.workunit.client.1.smithi190.stdout:5/519: symlink d2/d17/d1b/d22/d88/d8d/lb4 0 2024-03-26T18:54:34.661 INFO:tasks.workunit.client.1.smithi190.stdout:5/520: write d2/d17/d1b/d60/d63/d85/d99/f9c [957853,50376] 0 2024-03-26T18:54:34.663 INFO:tasks.workunit.client.0.smithi102.stdout:0/324: creat d6/d3e/f6a x:0 0 0 2024-03-26T18:54:34.665 INFO:tasks.workunit.client.0.smithi102.stdout:4/229: write d1/f8 [6334236,88861] 0 2024-03-26T18:54:34.665 INFO:tasks.workunit.client.0.smithi102.stdout:4/230: creat d1/d27/f4d x:0 0 0 2024-03-26T18:54:34.665 INFO:tasks.workunit.client.0.smithi102.stdout:7/263: dwrite d7/d12/d2b/f47 [0,4194304] 0 2024-03-26T18:54:34.666 INFO:tasks.workunit.client.0.smithi102.stdout:8/311: dwrite d0/d3/d2c/d2f/f53 [0,4194304] 0 2024-03-26T18:54:34.667 INFO:tasks.workunit.client.1.smithi190.stdout:4/633: unlink d7/d8/d1c/d20/l8c 0 2024-03-26T18:54:34.669 INFO:tasks.workunit.client.1.smithi190.stdout:6/568: creat d17/d28/d8b/d95/fc6 x:0 0 0 2024-03-26T18:54:34.671 INFO:tasks.workunit.client.1.smithi190.stdout:9/874: rename d12/d2d/d1d/d23/d83/dc9/dd4/d3f/d55/c5a to d12/d2d/d1d/da1/dba/dd2/c135 0 2024-03-26T18:54:34.671 INFO:tasks.workunit.client.1.smithi190.stdout:9/875: creat d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/f136 x:0 0 0 2024-03-26T18:54:34.674 INFO:tasks.workunit.client.0.smithi102.stdout:3/465: symlink df/la8 0 2024-03-26T18:54:34.674 INFO:tasks.workunit.client.0.smithi102.stdout:3/466: truncate df/f25 5783931 0 2024-03-26T18:54:34.674 INFO:tasks.workunit.client.0.smithi102.stdout:2/264: dwrite d3/d20/f30 [0,4194304] 0 2024-03-26T18:54:34.676 INFO:tasks.workunit.client.0.smithi102.stdout:6/295: mkdir d2/d3d/d53/d5c 0 2024-03-26T18:54:34.676 INFO:tasks.workunit.client.1.smithi190.stdout:5/521: unlink d2/d17/d1b/d22/d88/d8d/fae 0 2024-03-26T18:54:34.676 INFO:tasks.workunit.client.1.smithi190.stdout:5/522: write d2/d17/d62/f82 [863726,53299] 0 2024-03-26T18:54:34.677 INFO:tasks.workunit.client.0.smithi102.stdout:3/467: dread df/d29/f64 [0,4194304] 0 2024-03-26T18:54:34.677 INFO:tasks.workunit.client.0.smithi102.stdout:3/468: fdatasync df/f1b 0 2024-03-26T18:54:34.678 INFO:tasks.workunit.client.0.smithi102.stdout:0/325: mknod d6/de/d13/d25/d4f/c6b 0 2024-03-26T18:54:34.678 INFO:tasks.workunit.client.0.smithi102.stdout:0/326: write d6/d3e/d54/f61 [2950371,7553] 0 2024-03-26T18:54:34.678 INFO:tasks.workunit.client.0.smithi102.stdout:7/264: dread d7/d12/f3c [0,4194304] 0 2024-03-26T18:54:34.679 INFO:tasks.workunit.client.1.smithi190.stdout:8/655: dwrite d9/d39/d67/fba [0,4194304] 0 2024-03-26T18:54:34.682 INFO:tasks.workunit.client.0.smithi102.stdout:0/327: dread d6/de/f4b [0,4194304] 0 2024-03-26T18:54:34.684 INFO:tasks.workunit.client.1.smithi190.stdout:5/523: getdents d2/d17/d1b/d60/d63/d85 0 2024-03-26T18:54:34.684 INFO:tasks.workunit.client.1.smithi190.stdout:3/514: write d1/de/f73 [760614,100217] 0 2024-03-26T18:54:34.687 INFO:tasks.workunit.client.1.smithi190.stdout:4/634: getdents d7/d8 0 2024-03-26T18:54:34.687 INFO:tasks.workunit.client.1.smithi190.stdout:4/635: chown d7/d8/d1c/d20/la8 12 1 2024-03-26T18:54:34.687 INFO:tasks.workunit.client.1.smithi190.stdout:4/636: read - d7/d1d/d5e/d98/fb0 zero size 2024-03-26T18:54:34.687 INFO:tasks.workunit.client.0.smithi102.stdout:6/296: creat d2/d3d/d53/d5c/f5d x:0 0 0 2024-03-26T18:54:34.690 INFO:tasks.workunit.client.1.smithi190.stdout:9/876: dread d12/d2d/d1d/d23/d83/dc9/dd4/d3f/d55/f82 [0,4194304] 0 2024-03-26T18:54:34.690 INFO:tasks.workunit.client.1.smithi190.stdout:9/877: fsync d12/d2d/d1d/da1/dba/dbc/fda 0 2024-03-26T18:54:34.690 INFO:tasks.workunit.client.1.smithi190.stdout:4/637: link d7/d1d/d42/fbe d7/d8/d14/d23/fdc 0 2024-03-26T18:54:34.690 INFO:tasks.workunit.client.1.smithi190.stdout:4/638: dread - d7/d1d/fa7 zero size 2024-03-26T18:54:34.691 INFO:tasks.workunit.client.0.smithi102.stdout:2/265: write d3/d6/f1d [778619,52147] 0 2024-03-26T18:54:34.693 INFO:tasks.workunit.client.1.smithi190.stdout:9/878: dread d12/d2d/f99 [0,4194304] 0 2024-03-26T18:54:34.696 INFO:tasks.workunit.client.1.smithi190.stdout:8/656: read d9/da2/dc4/f5c [485018,52790] 0 2024-03-26T18:54:34.700 INFO:tasks.workunit.client.0.smithi102.stdout:8/312: mkdir d0/d9/d65 0 2024-03-26T18:54:34.701 INFO:tasks.workunit.client.0.smithi102.stdout:8/313: dread - d0/d3/d26/d19/d27/f5b zero size 2024-03-26T18:54:34.706 INFO:tasks.workunit.client.0.smithi102.stdout:6/297: dread d2/d7/d3c/f41 [0,4194304] 0 2024-03-26T18:54:34.707 INFO:tasks.workunit.client.0.smithi102.stdout:7/265: dread d7/f9 [0,4194304] 0 2024-03-26T18:54:34.708 INFO:tasks.workunit.client.0.smithi102.stdout:2/266: truncate d3/d6/f24 2650078 0 2024-03-26T18:54:34.709 INFO:tasks.workunit.client.0.smithi102.stdout:2/267: fsync d3/f1c 0 2024-03-26T18:54:34.709 INFO:tasks.workunit.client.0.smithi102.stdout:2/268: fsync d3/d11/f50 0 2024-03-26T18:54:34.711 INFO:tasks.workunit.client.0.smithi102.stdout:3/469: getdents df/d29/d3a 0 2024-03-26T18:54:34.711 INFO:tasks.workunit.client.0.smithi102.stdout:3/470: dread - df/d29/d31/d36/f7f zero size 2024-03-26T18:54:34.712 INFO:tasks.workunit.client.1.smithi190.stdout:4/639: mknod d7/d8/d14/d23/d28/d38/d96/cdd 0 2024-03-26T18:54:34.712 INFO:tasks.workunit.client.1.smithi190.stdout:4/640: chown d7/d1d/l3e 62 1 2024-03-26T18:54:34.713 INFO:tasks.workunit.client.1.smithi190.stdout:8/657: symlink d9/dd/d23/d36/d37/db8/ldc 0 2024-03-26T18:54:34.713 INFO:tasks.workunit.client.1.smithi190.stdout:8/658: chown d9/dd/d15/f45 19179576 1 2024-03-26T18:54:34.714 INFO:tasks.workunit.client.1.smithi190.stdout:3/515: dread d1/d3/df/d18/d1b/f21 [4194304,4194304] 0 2024-03-26T18:54:34.714 INFO:tasks.workunit.client.0.smithi102.stdout:8/314: dread d0/d3/d26/d19/d27/f29 [0,4194304] 0 2024-03-26T18:54:34.714 INFO:tasks.workunit.client.0.smithi102.stdout:8/315: chown d0/d3/f11 10468157 1 2024-03-26T18:54:34.714 INFO:tasks.workunit.client.0.smithi102.stdout:3/471: rmdir df/d29/d31/d36/d3c/d6f 39 2024-03-26T18:54:34.715 INFO:tasks.workunit.client.0.smithi102.stdout:1/281: dwrite de/d16/d1d/f4d [0,4194304] 0 2024-03-26T18:54:34.717 INFO:tasks.workunit.client.0.smithi102.stdout:8/316: mkdir d0/d3c/d66 0 2024-03-26T18:54:34.717 INFO:tasks.workunit.client.0.smithi102.stdout:8/317: write d0/d3/d26/d19/d27/f45 [431419,44231] 0 2024-03-26T18:54:34.717 INFO:tasks.workunit.client.0.smithi102.stdout:8/318: chown d0/d3/d2c/f34 25438557 1 2024-03-26T18:54:34.718 INFO:tasks.workunit.client.0.smithi102.stdout:3/472: mknod df/d29/d3a/ca9 0 2024-03-26T18:54:34.720 INFO:tasks.workunit.client.1.smithi190.stdout:8/659: link d9/da2/dc4/cbc d9/dd/d15/cdd 0 2024-03-26T18:54:34.720 INFO:tasks.workunit.client.1.smithi190.stdout:8/660: readlink d9/d39/d67/dc1/ldb 0 2024-03-26T18:54:34.721 INFO:tasks.workunit.client.0.smithi102.stdout:8/319: dread d0/d3/d26/d19/d27/f29 [0,4194304] 0 2024-03-26T18:54:34.721 INFO:tasks.workunit.client.0.smithi102.stdout:8/320: read - d0/d3/d2c/d2f/f60 zero size 2024-03-26T18:54:34.725 INFO:tasks.workunit.client.1.smithi190.stdout:3/516: symlink d1/d3/df/d18/d35/d41/d61/l99 0 2024-03-26T18:54:34.725 INFO:tasks.workunit.client.1.smithi190.stdout:3/517: dread - d1/d3/d52/f90 zero size 2024-03-26T18:54:34.726 INFO:tasks.workunit.client.0.smithi102.stdout:5/537: sync 2024-03-26T18:54:34.726 INFO:tasks.workunit.client.0.smithi102.stdout:5/538: truncate d2/d25/d5b/d72/f22 4685717 0 2024-03-26T18:54:34.727 INFO:tasks.workunit.client.0.smithi102.stdout:1/282: link de/c1a de/d1f/c63 0 2024-03-26T18:54:34.729 INFO:tasks.workunit.client.1.smithi190.stdout:8/661: creat d9/dd/d15/dd3/fde x:0 0 0 2024-03-26T18:54:34.729 INFO:tasks.workunit.client.1.smithi190.stdout:8/662: write d9/dd/d23/d36/d37/d3a/d61/f91 [640268,101041] 0 2024-03-26T18:54:34.729 INFO:tasks.workunit.client.1.smithi190.stdout:8/663: dread - d9/dd/d23/d36/d37/fb4 zero size 2024-03-26T18:54:34.729 INFO:tasks.workunit.client.1.smithi190.stdout:8/664: write d9/d39/d79/f90 [398064,1201] 0 2024-03-26T18:54:34.729 INFO:tasks.workunit.client.1.smithi190.stdout:3/518: link d1/de/d8a/c8e d1/d3/df/d18/d35/d89/c9a 0 2024-03-26T18:54:34.729 INFO:tasks.workunit.client.1.smithi190.stdout:3/519: write d1/d3/df/d18/d35/f60 [514808,103215] 0 2024-03-26T18:54:34.732 INFO:tasks.workunit.client.0.smithi102.stdout:3/473: rename df/d29/d31/d36/f48 to df/d1c/d70/faa 0 2024-03-26T18:54:34.735 INFO:tasks.workunit.client.1.smithi190.stdout:4/641: dread d7/fb9 [0,4194304] 0 2024-03-26T18:54:34.736 INFO:tasks.workunit.client.1.smithi190.stdout:4/642: chown d7/d1d/c82 4 1 2024-03-26T18:54:34.737 INFO:tasks.workunit.client.1.smithi190.stdout:6/569: dwrite d17/d28/d46/f5c [0,4194304] 0 2024-03-26T18:54:34.737 INFO:tasks.workunit.client.1.smithi190.stdout:6/570: write d17/d28/d38/fb1 [1006561,93965] 0 2024-03-26T18:54:34.737 INFO:tasks.workunit.client.1.smithi190.stdout:6/571: chown d17/d28/d46/d6a/fa4 1818 1 2024-03-26T18:54:34.737 INFO:tasks.workunit.client.1.smithi190.stdout:6/572: write d17/d28/d74/d97/fb9 [573564,19909] 0 2024-03-26T18:54:34.738 INFO:tasks.workunit.client.1.smithi190.stdout:8/665: symlink d9/d39/d79/ldf 0 2024-03-26T18:54:34.739 INFO:tasks.workunit.client.1.smithi190.stdout:8/666: readlink d9/d48/l8a 0 2024-03-26T18:54:34.739 INFO:tasks.workunit.client.1.smithi190.stdout:8/667: write d9/dd/d23/d36/d37/d3a/fb7 [3272249,66461] 0 2024-03-26T18:54:34.740 INFO:tasks.workunit.client.0.smithi102.stdout:7/266: dread d7/d10/f14 [0,4194304] 0 2024-03-26T18:54:34.740 INFO:tasks.workunit.client.0.smithi102.stdout:7/267: write f6 [2340861,103451] 0 2024-03-26T18:54:34.740 INFO:tasks.workunit.client.0.smithi102.stdout:7/268: creat d7/d10/d27/d34/f49 x:0 0 0 2024-03-26T18:54:34.740 INFO:tasks.workunit.client.0.smithi102.stdout:7/269: write d7/d12/f22 [813688,122663] 0 2024-03-26T18:54:34.740 INFO:tasks.workunit.client.0.smithi102.stdout:7/270: creat d7/d2a/f4a x:0 0 0 2024-03-26T18:54:34.741 INFO:tasks.workunit.client.0.smithi102.stdout:1/283: rename de/f52 to de/d2e/d62/f64 0 2024-03-26T18:54:34.742 INFO:tasks.workunit.client.0.smithi102.stdout:1/284: creat de/d16/d1d/d2a/f65 x:0 0 0 2024-03-26T18:54:34.742 INFO:tasks.workunit.client.0.smithi102.stdout:1/285: chown de/f14 0 1 2024-03-26T18:54:34.742 INFO:tasks.workunit.client.0.smithi102.stdout:1/286: fsync de/d16/f35 0 2024-03-26T18:54:34.742 INFO:tasks.workunit.client.0.smithi102.stdout:1/287: creat de/d16/d1d/d2a/d2c/f66 x:0 0 0 2024-03-26T18:54:34.742 INFO:tasks.workunit.client.1.smithi190.stdout:0/687: dwrite d0/dd/d20/d1c/d46/da7/df0/d87/d93/db5/dcc/fdc [0,4194304] 0 2024-03-26T18:54:34.749 INFO:tasks.workunit.client.0.smithi102.stdout:4/231: dwrite d1/f26 [0,4194304] 0 2024-03-26T18:54:34.749 INFO:tasks.workunit.client.0.smithi102.stdout:4/232: chown d1/d27/f4d 346 1 2024-03-26T18:54:34.750 INFO:tasks.workunit.client.0.smithi102.stdout:8/321: truncate d0/d3/d31/f4f 4610294 0 2024-03-26T18:54:34.750 INFO:tasks.workunit.client.1.smithi190.stdout:6/573: dread d17/d28/d38/f89 [0,4194304] 0 2024-03-26T18:54:34.751 INFO:tasks.workunit.client.1.smithi190.stdout:6/574: write d17/d28/d2c/d50/f65 [530351,82257] 0 2024-03-26T18:54:34.757 INFO:tasks.workunit.client.1.smithi190.stdout:8/668: symlink d9/d48/d49/le0 0 2024-03-26T18:54:34.759 INFO:tasks.workunit.client.0.smithi102.stdout:7/271: mkdir d7/d12/d4b 0 2024-03-26T18:54:34.763 INFO:tasks.workunit.client.0.smithi102.stdout:1/288: mkdir de/d16/d1d/d2a/d67 0 2024-03-26T18:54:34.763 INFO:tasks.workunit.client.0.smithi102.stdout:1/289: dread - de/d16/d1d/d2a/d2c/d43/f5a zero size 2024-03-26T18:54:34.766 INFO:tasks.workunit.client.0.smithi102.stdout:8/322: creat d0/d15/f67 x:0 0 0 2024-03-26T18:54:34.766 INFO:tasks.workunit.client.0.smithi102.stdout:8/323: fsync d0/d3/d31/f4a 0 2024-03-26T18:54:34.766 INFO:tasks.workunit.client.0.smithi102.stdout:7/272: mknod d7/d10/d27/d34/c4c 0 2024-03-26T18:54:34.768 INFO:tasks.workunit.client.1.smithi190.stdout:5/524: dwrite d2/f10 [0,4194304] 0 2024-03-26T18:54:34.768 INFO:tasks.workunit.client.1.smithi190.stdout:7/672: dwrite d3/d6/d9/d1f/d41/d65/d90/d92/d9b/de9/f8a [0,4194304] 0 2024-03-26T18:54:34.769 INFO:tasks.workunit.client.0.smithi102.stdout:1/290: link de/d16/d1d/c37 de/d2e/c68 0 2024-03-26T18:54:34.769 INFO:tasks.workunit.client.1.smithi190.stdout:2/632: dwrite f5 [0,4194304] 0 2024-03-26T18:54:34.769 INFO:tasks.workunit.client.1.smithi190.stdout:2/633: chown d6/d9/d3a/d3d/d2e/f5c 378667 1 2024-03-26T18:54:34.770 INFO:tasks.workunit.client.0.smithi102.stdout:8/324: symlink d0/d3/d2c/l68 0 2024-03-26T18:54:34.773 INFO:tasks.workunit.client.0.smithi102.stdout:7/273: mkdir d7/d12/d4d 0 2024-03-26T18:54:34.774 INFO:tasks.workunit.client.1.smithi190.stdout:0/688: truncate d0/dd/d20/d1c/d4e/f9d 497043 0 2024-03-26T18:54:34.776 INFO:tasks.workunit.client.0.smithi102.stdout:1/291: rmdir de/d16/d1d/d2a/d2c/d56 39 2024-03-26T18:54:34.776 INFO:tasks.workunit.client.1.smithi190.stdout:9/879: dwrite d12/d2d/d1d/d23/d83/dc9/dd4/f131 [0,4194304] 0 2024-03-26T18:54:34.779 INFO:tasks.workunit.client.1.smithi190.stdout:7/673: mknod d3/cf9 0 2024-03-26T18:54:34.779 INFO:tasks.workunit.client.0.smithi102.stdout:8/325: symlink d0/d3/l69 0 2024-03-26T18:54:34.781 INFO:tasks.workunit.client.1.smithi190.stdout:0/689: creat d0/dd/ffa x:0 0 0 2024-03-26T18:54:34.787 INFO:tasks.workunit.client.0.smithi102.stdout:1/292: mknod de/d1f/c69 0 2024-03-26T18:54:34.787 INFO:tasks.workunit.client.0.smithi102.stdout:1/293: write de/f19 [495270,68448] 0 2024-03-26T18:54:34.788 INFO:tasks.workunit.client.0.smithi102.stdout:1/294: chown de/f32 110276 1 2024-03-26T18:54:34.788 INFO:tasks.workunit.client.1.smithi190.stdout:5/525: write d2/f15 [399799,61840] 0 2024-03-26T18:54:34.789 INFO:tasks.workunit.client.0.smithi102.stdout:0/328: dwrite d6/de/d13/d1f/d55/d57/f1c [0,4194304] 0 2024-03-26T18:54:34.791 INFO:tasks.workunit.client.0.smithi102.stdout:6/298: dwrite d2/d7/f46 [0,4194304] 0 2024-03-26T18:54:34.791 INFO:tasks.workunit.client.0.smithi102.stdout:7/274: mkdir d7/d4e 0 2024-03-26T18:54:34.798 INFO:tasks.workunit.client.0.smithi102.stdout:8/326: dread d0/d3/f11 [0,4194304] 0 2024-03-26T18:54:34.798 INFO:tasks.workunit.client.0.smithi102.stdout:2/269: dwrite d3/d20/f4c [0,4194304] 0 2024-03-26T18:54:34.799 INFO:tasks.workunit.client.0.smithi102.stdout:2/270: chown d3/d6/ff 15 1 2024-03-26T18:54:34.800 INFO:tasks.workunit.client.0.smithi102.stdout:7/275: creat d7/d12/d4d/f4f x:0 0 0 2024-03-26T18:54:34.801 INFO:tasks.workunit.client.0.smithi102.stdout:7/276: dread d7/d12/f38 [0,4194304] 0 2024-03-26T18:54:34.804 INFO:tasks.workunit.client.0.smithi102.stdout:6/299: dread d2/f52 [0,4194304] 0 2024-03-26T18:54:34.804 INFO:tasks.workunit.client.0.smithi102.stdout:6/300: readlink d2/d7/l4a 0 2024-03-26T18:54:34.808 INFO:tasks.workunit.client.0.smithi102.stdout:7/277: mkdir d7/d12/d2b/d50 0 2024-03-26T18:54:34.809 INFO:tasks.workunit.client.0.smithi102.stdout:3/474: dwrite df/f25 [0,4194304] 0 2024-03-26T18:54:34.809 INFO:tasks.workunit.client.0.smithi102.stdout:3/475: write df/d29/f8c [3249697,52092] 0 2024-03-26T18:54:34.810 INFO:tasks.workunit.client.1.smithi190.stdout:3/520: dwrite d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:34.811 INFO:tasks.workunit.client.0.smithi102.stdout:0/329: dread d6/de/d13/d1f/f27 [0,4194304] 0 2024-03-26T18:54:34.812 INFO:tasks.workunit.client.1.smithi190.stdout:4/643: dwrite d7/d1d/d41/d74/fd8 [0,4194304] 0 2024-03-26T18:54:34.812 INFO:tasks.workunit.client.1.smithi190.stdout:7/674: link d3/d6/d9/d1f/d41/d65/d90/d92/d9b/fb0 d3/d6/d9/d14/d59/d73/d8b/ffa 0 2024-03-26T18:54:34.813 INFO:tasks.workunit.client.0.smithi102.stdout:8/327: getdents d0/d3/d2c/d33 0 2024-03-26T18:54:34.813 INFO:tasks.workunit.client.0.smithi102.stdout:8/328: getdents d0/d3c/d66 0 2024-03-26T18:54:34.813 INFO:tasks.workunit.client.0.smithi102.stdout:4/233: dwrite d1/d17/d1a/d21/f2c [8388608,4194304] 0 2024-03-26T18:54:34.815 INFO:tasks.workunit.client.1.smithi190.stdout:0/690: rename d0/dd/d20/d1c/d46/da7/df0/d4b/c71 to d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/cfb 0 2024-03-26T18:54:34.816 INFO:tasks.workunit.client.1.smithi190.stdout:6/575: dread d17/d28/d2c/d50/f65 [0,4194304] 0 2024-03-26T18:54:34.817 INFO:tasks.workunit.client.0.smithi102.stdout:6/301: link d2/f29 d2/d3d/d53/d5c/f5e 0 2024-03-26T18:54:34.820 INFO:tasks.workunit.client.0.smithi102.stdout:5/539: dwrite d2/d2c/f8d [0,4194304] 0 2024-03-26T18:54:34.822 INFO:tasks.workunit.client.0.smithi102.stdout:7/278: mkdir d7/d12/d51 0 2024-03-26T18:54:34.822 INFO:tasks.workunit.client.0.smithi102.stdout:7/279: write d7/f41 [374514,23132] 0 2024-03-26T18:54:34.829 INFO:tasks.workunit.client.1.smithi190.stdout:8/669: dwrite d9/d39/d67/fa3 [0,4194304] 0 2024-03-26T18:54:34.830 INFO:tasks.workunit.client.0.smithi102.stdout:3/476: mknod df/d29/d3a/d93/cab 0 2024-03-26T18:54:34.830 INFO:tasks.workunit.client.0.smithi102.stdout:3/477: dread - df/d61/d55/f90 zero size 2024-03-26T18:54:34.830 INFO:tasks.workunit.client.0.smithi102.stdout:3/478: truncate df/f1a 2238240 0 2024-03-26T18:54:34.830 INFO:tasks.workunit.client.0.smithi102.stdout:3/479: fdatasync df/f26 0 2024-03-26T18:54:34.830 INFO:tasks.workunit.client.1.smithi190.stdout:3/521: read d1/de/d1c/d36/f5d [1471727,125400] 0 2024-03-26T18:54:34.830 INFO:tasks.workunit.client.1.smithi190.stdout:3/522: creat d1/d3/df/d18/d1b/f9b x:0 0 0 2024-03-26T18:54:34.833 INFO:tasks.workunit.client.0.smithi102.stdout:4/234: creat d1/d25/d39/f4e x:0 0 0 2024-03-26T18:54:34.833 INFO:tasks.workunit.client.0.smithi102.stdout:4/235: write d1/d17/d1f/f48 [647883,58501] 0 2024-03-26T18:54:34.836 INFO:tasks.workunit.client.0.smithi102.stdout:6/302: truncate d2/d3/f38 7115316 0 2024-03-26T18:54:34.837 INFO:tasks.workunit.client.0.smithi102.stdout:6/303: dread - d2/f47 zero size 2024-03-26T18:54:34.837 INFO:tasks.workunit.client.0.smithi102.stdout:6/304: dread - d2/df/d35/d43/f57 zero size 2024-03-26T18:54:34.837 INFO:tasks.workunit.client.0.smithi102.stdout:6/305: chown d2/d7/d3c/f41 8674547 1 2024-03-26T18:54:34.838 INFO:tasks.workunit.client.1.smithi190.stdout:9/880: dwrite d12/d2d/d1d/f3b [0,4194304] 0 2024-03-26T18:54:34.838 INFO:tasks.workunit.client.1.smithi190.stdout:9/881: write d12/d2d/d1d/d23/d83/dc9/dd4/d3f/f113 [345756,128256] 0 2024-03-26T18:54:34.842 INFO:tasks.workunit.client.1.smithi190.stdout:7/675: unlink d3/d1b/d57/c77 0 2024-03-26T18:54:34.842 INFO:tasks.workunit.client.1.smithi190.stdout:4/644: dread d7/d8/d14/d6e/d6f/f7e [0,4194304] 0 2024-03-26T18:54:34.842 INFO:tasks.workunit.client.1.smithi190.stdout:2/634: dwrite d6/d9/d3a/d3d/d1c/d8d/dc7/f43 [0,4194304] 0 2024-03-26T18:54:34.842 INFO:tasks.workunit.client.1.smithi190.stdout:2/635: readlink d6/d9/d3a/d3d/d1c/l25 0 2024-03-26T18:54:34.843 INFO:tasks.workunit.client.1.smithi190.stdout:6/576: dread d17/d28/d2c/d63/f70 [0,4194304] 0 2024-03-26T18:54:34.843 INFO:tasks.workunit.client.1.smithi190.stdout:6/577: chown d17/d28/d2c/f41 13229631 1 2024-03-26T18:54:34.843 INFO:tasks.workunit.client.1.smithi190.stdout:6/578: chown d17/d28/d2b/fc4 1014 1 2024-03-26T18:54:34.844 INFO:tasks.workunit.client.0.smithi102.stdout:2/271: dwrite d3/f1c [0,4194304] 0 2024-03-26T18:54:34.845 INFO:tasks.workunit.client.0.smithi102.stdout:1/295: dwrite fc [0,4194304] 0 2024-03-26T18:54:34.845 INFO:tasks.workunit.client.0.smithi102.stdout:1/296: dread - de/d1f/f48 zero size 2024-03-26T18:54:34.845 INFO:tasks.workunit.client.1.smithi190.stdout:5/526: rename d2/d17/d98/d3a/d3c/l7d to d2/d17/d1b/d60/d63/lb5 0 2024-03-26T18:54:34.846 INFO:tasks.workunit.client.1.smithi190.stdout:5/527: write d2/d17/d1b/d22/d24/d3b/faf [815534,86270] 0 2024-03-26T18:54:34.850 INFO:tasks.workunit.client.0.smithi102.stdout:4/236: write d1/f26 [2854608,13614] 0 2024-03-26T18:54:34.851 INFO:tasks.workunit.client.0.smithi102.stdout:8/329: dwrite d0/d3/d26/d19/d27/f29 [0,4194304] 0 2024-03-26T18:54:34.852 INFO:tasks.workunit.client.1.smithi190.stdout:8/670: dread d9/d39/d79/f90 [0,4194304] 0 2024-03-26T18:54:34.852 INFO:tasks.workunit.client.1.smithi190.stdout:8/671: write d9/dd/d15/dd3/fde [1002544,114341] 0 2024-03-26T18:54:34.852 INFO:tasks.workunit.client.1.smithi190.stdout:0/691: mkdir d0/dd5/dee/dfc 0 2024-03-26T18:54:34.862 INFO:tasks.workunit.client.1.smithi190.stdout:2/636: dread d6/d9/d48/da0/fa7 [0,4194304] 0 2024-03-26T18:54:34.862 INFO:tasks.workunit.client.1.smithi190.stdout:2/637: write d6/d9/d3a/d3d/f22 [708294,1084] 0 2024-03-26T18:54:34.863 INFO:tasks.workunit.client.1.smithi190.stdout:6/579: dread d17/d28/d2c/d50/f7e [0,4194304] 0 2024-03-26T18:54:34.863 INFO:tasks.workunit.client.1.smithi190.stdout:9/882: symlink d12/d2d/d1d/d23/d83/dc9/dd4/l137 0 2024-03-26T18:54:34.867 INFO:tasks.workunit.client.0.smithi102.stdout:0/330: dwrite d6/de/d13/d1f/f27 [0,4194304] 0 2024-03-26T18:54:34.867 INFO:tasks.workunit.client.1.smithi190.stdout:7/676: mkdir d3/d6/d9/d1f/d93/dfb 0 2024-03-26T18:54:34.870 INFO:tasks.workunit.client.0.smithi102.stdout:5/540: rename d2/d25/d75/db9/d43/c6d to d2/d25/d5b/d72/cc0 0 2024-03-26T18:54:34.875 INFO:tasks.workunit.client.1.smithi190.stdout:6/580: write d17/d28/d2c/d77/f8d [25337,36058] 0 2024-03-26T18:54:34.875 INFO:tasks.workunit.client.1.smithi190.stdout:6/581: dread - d17/d28/d8b/d95/fc6 zero size 2024-03-26T18:54:34.877 INFO:tasks.workunit.client.1.smithi190.stdout:8/672: fsync d9/dd/d23/d36/d37/d3a/d61/f91 0 2024-03-26T18:54:34.879 INFO:tasks.workunit.client.1.smithi190.stdout:3/523: dwrite d1/d3/df/d18/d1b/f21 [4194304,4194304] 0 2024-03-26T18:54:34.879 INFO:tasks.workunit.client.0.smithi102.stdout:2/272: truncate d3/f19 157302 0 2024-03-26T18:54:34.881 INFO:tasks.workunit.client.1.smithi190.stdout:0/692: rmdir d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b 39 2024-03-26T18:54:34.881 INFO:tasks.workunit.client.1.smithi190.stdout:0/693: stat d0/dd/d20/d1c/d46/da7/df0/da2/fc6 0 2024-03-26T18:54:34.883 INFO:tasks.workunit.client.1.smithi190.stdout:7/677: rename d3/d6/d9/d1f/d41/d65/d90/d92/f9c to d3/d6/d9/d1f/d41/d65/d90/d92/ffc 0 2024-03-26T18:54:34.887 INFO:tasks.workunit.client.0.smithi102.stdout:0/331: symlink d6/de/d13/d1f/d39/l6c 0 2024-03-26T18:54:34.888 INFO:tasks.workunit.client.1.smithi190.stdout:6/582: getdents d17/d28/d80/da8 0 2024-03-26T18:54:34.890 INFO:tasks.workunit.client.0.smithi102.stdout:2/273: link d3/d2a/f37 d3/d11/d15/d31/f60 0 2024-03-26T18:54:34.890 INFO:tasks.workunit.client.0.smithi102.stdout:2/274: chown d3/d11/d15/f48 148284405 1 2024-03-26T18:54:34.892 INFO:tasks.workunit.client.0.smithi102.stdout:8/330: write d0/d3/d31/f4f [4543931,34360] 0 2024-03-26T18:54:34.893 INFO:tasks.workunit.client.1.smithi190.stdout:8/673: creat d9/da2/d87/fe1 x:0 0 0 2024-03-26T18:54:34.898 INFO:tasks.workunit.client.1.smithi190.stdout:0/694: write d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/fdb [1990539,44835] 0 2024-03-26T18:54:34.899 INFO:tasks.workunit.client.0.smithi102.stdout:8/331: rename d0/d3/d26/d19/l1a to d0/d15/l6a 0 2024-03-26T18:54:34.903 INFO:tasks.workunit.client.1.smithi190.stdout:5/528: rename d2/d17/d1b/d22 to d2/d17/d1b/db6 0 2024-03-26T18:54:34.906 INFO:tasks.workunit.client.1.smithi190.stdout:3/524: mkdir d1/d3/df/d18/d35/d41/d51/d9c 0 2024-03-26T18:54:34.906 INFO:tasks.workunit.client.1.smithi190.stdout:7/678: mkdir d3/d6/d9/d1f/d41/d50/dde/dfd 0 2024-03-26T18:54:34.906 INFO:tasks.workunit.client.1.smithi190.stdout:7/679: fdatasync d3/d6/f5d 0 2024-03-26T18:54:34.906 INFO:tasks.workunit.client.1.smithi190.stdout:3/525: read - d1/d3/df/d18/d35/d41/d51/f96 zero size 2024-03-26T18:54:34.906 INFO:tasks.workunit.client.1.smithi190.stdout:7/680: stat d3/d6/d9/d1f/d41/d65/f7d 0 2024-03-26T18:54:34.907 INFO:tasks.workunit.client.1.smithi190.stdout:6/583: truncate d17/d28/d46/f5c 2659499 0 2024-03-26T18:54:34.908 INFO:tasks.workunit.client.0.smithi102.stdout:3/480: dwrite df/d29/f33 [0,4194304] 0 2024-03-26T18:54:34.908 INFO:tasks.workunit.client.0.smithi102.stdout:3/481: write df/f9c [5121009,66697] 0 2024-03-26T18:54:34.909 INFO:tasks.workunit.client.0.smithi102.stdout:0/332: dread d6/f1a [0,4194304] 0 2024-03-26T18:54:34.909 INFO:tasks.workunit.client.1.smithi190.stdout:8/674: creat d9/dd/d23/d36/d37/d3a/d61/daf/db3/fe2 x:0 0 0 2024-03-26T18:54:34.909 INFO:tasks.workunit.client.1.smithi190.stdout:2/638: dwrite d6/d9/d3a/d3d/d1c/d8d/dc7/f64 [4194304,4194304] 0 2024-03-26T18:54:34.909 INFO:tasks.workunit.client.1.smithi190.stdout:2/639: chown d6/d9/d3a/d3d/d1c/d8d/dc7/d72/lc8 83998 1 2024-03-26T18:54:34.911 INFO:tasks.workunit.client.1.smithi190.stdout:3/526: link d1/d3/df/d18/c3f d1/c9d 0 2024-03-26T18:54:34.911 INFO:tasks.workunit.client.1.smithi190.stdout:3/527: chown d1/c88 35469 1 2024-03-26T18:54:34.912 INFO:tasks.workunit.client.0.smithi102.stdout:8/332: dread d0/d3/d2c/f34 [0,4194304] 0 2024-03-26T18:54:34.912 INFO:tasks.workunit.client.0.smithi102.stdout:8/333: creat d0/d15/f6b x:0 0 0 2024-03-26T18:54:34.912 INFO:tasks.workunit.client.0.smithi102.stdout:8/334: chown d0/d3/d2c/d33/f49 110685633 1 2024-03-26T18:54:34.912 INFO:tasks.workunit.client.0.smithi102.stdout:6/306: dwrite d2/f47 [0,4194304] 0 2024-03-26T18:54:34.913 INFO:tasks.workunit.client.0.smithi102.stdout:8/335: write d0/d15/f6b [889519,99876] 0 2024-03-26T18:54:34.913 INFO:tasks.workunit.client.0.smithi102.stdout:8/336: fsync d0/d3/d2c/d33/f49 0 2024-03-26T18:54:34.915 INFO:tasks.workunit.client.0.smithi102.stdout:0/333: write d6/de/d13/d4a/f5b [3374622,83588] 0 2024-03-26T18:54:34.917 INFO:tasks.workunit.client.0.smithi102.stdout:3/482: mknod df/d8b/cac 0 2024-03-26T18:54:34.917 INFO:tasks.workunit.client.1.smithi190.stdout:7/681: rename d3/d6/d9/d1f/d41/d50/dde/ff2 to d3/d6/d9/d1f/dab/ffe 0 2024-03-26T18:54:34.917 INFO:tasks.workunit.client.1.smithi190.stdout:7/682: fsync d3/d6/f1d 0 2024-03-26T18:54:34.920 INFO:tasks.workunit.client.0.smithi102.stdout:1/297: dwrite de/f22 [0,4194304] 0 2024-03-26T18:54:34.920 INFO:tasks.workunit.client.0.smithi102.stdout:1/298: fdatasync de/d16/d1d/d2a/d2c/d43/f46 0 2024-03-26T18:54:34.920 INFO:tasks.workunit.client.1.smithi190.stdout:6/584: creat d17/d28/d80/da8/fc7 x:0 0 0 2024-03-26T18:54:34.923 INFO:tasks.workunit.client.0.smithi102.stdout:2/275: dwrite d3/d2a/f40 [0,4194304] 0 2024-03-26T18:54:34.923 INFO:tasks.workunit.client.1.smithi190.stdout:8/675: mkdir d9/dd/d23/d36/d37/d3a/d61/daf/db3/db5/de3 0 2024-03-26T18:54:34.925 INFO:tasks.workunit.client.0.smithi102.stdout:5/541: dwrite d2/d14/d1a/f1b [0,4194304] 0 2024-03-26T18:54:34.927 INFO:tasks.workunit.client.0.smithi102.stdout:8/337: creat d0/d3c/d66/f6c x:0 0 0 2024-03-26T18:54:34.927 INFO:tasks.workunit.client.0.smithi102.stdout:8/338: dread - d0/d3/d26/d5c/f62 zero size 2024-03-26T18:54:34.928 INFO:tasks.workunit.client.1.smithi190.stdout:1/552: sync 2024-03-26T18:54:34.928 INFO:tasks.workunit.client.1.smithi190.stdout:1/553: creat d3/da/dd/fb6 x:0 0 0 2024-03-26T18:54:34.928 INFO:tasks.workunit.client.1.smithi190.stdout:1/554: stat d3/da/dd/d95 0 2024-03-26T18:54:34.929 INFO:tasks.workunit.client.1.smithi190.stdout:3/528: link d1/de/d1c/c5c d1/d3/df/d18/d35/d41/d51/d9c/c9e 0 2024-03-26T18:54:34.930 INFO:tasks.workunit.client.0.smithi102.stdout:6/307: dread d2/d3/f4b [0,4194304] 0 2024-03-26T18:54:34.933 INFO:tasks.workunit.client.0.smithi102.stdout:0/334: unlink d6/f36 0 2024-03-26T18:54:34.935 INFO:tasks.workunit.client.1.smithi190.stdout:0/695: dwrite d0/dd/d20/d1c/d46/f91 [0,4194304] 0 2024-03-26T18:54:34.937 INFO:tasks.workunit.client.0.smithi102.stdout:3/483: rename df/d1c/d70/f96 to df/d29/d31/d36/d3c/da7/fad 0 2024-03-26T18:54:34.938 INFO:tasks.workunit.client.1.smithi190.stdout:5/529: dwrite d2/d17/d1b/db6/f9a [0,4194304] 0 2024-03-26T18:54:34.942 INFO:tasks.workunit.client.0.smithi102.stdout:1/299: mkdir de/d16/d1d/d61/d6a 0 2024-03-26T18:54:34.948 INFO:tasks.workunit.client.1.smithi190.stdout:1/555: rename d3/da/dd/f1b to d3/da/dd/d2f/d33/d9c/fb7 0 2024-03-26T18:54:34.950 INFO:tasks.workunit.client.1.smithi190.stdout:1/556: dread d3/da/d3a/f50 [0,4194304] 0 2024-03-26T18:54:34.951 INFO:tasks.workunit.client.1.smithi190.stdout:0/696: write d0/dd/d20/d1c/d46/f91 [1214471,71403] 0 2024-03-26T18:54:34.951 INFO:tasks.workunit.client.0.smithi102.stdout:8/339: dread d0/d3/d26/d19/d27/f45 [0,4194304] 0 2024-03-26T18:54:34.952 INFO:tasks.workunit.client.1.smithi190.stdout:3/529: mkdir d1/de/d9f 0 2024-03-26T18:54:34.952 INFO:tasks.workunit.client.1.smithi190.stdout:3/530: fdatasync d1/d3/df/d18/d35/d41/d51/f7a 0 2024-03-26T18:54:34.954 INFO:tasks.workunit.client.0.smithi102.stdout:2/276: symlink d3/d11/d15/d31/l61 0 2024-03-26T18:54:34.958 INFO:tasks.workunit.client.0.smithi102.stdout:5/542: symlink d2/d14/d1a/lc1 0 2024-03-26T18:54:34.958 INFO:tasks.workunit.client.0.smithi102.stdout:5/543: readlink d2/d2c/l83 0 2024-03-26T18:54:34.963 INFO:tasks.workunit.client.0.smithi102.stdout:9/248: sync 2024-03-26T18:54:34.966 INFO:tasks.workunit.client.1.smithi190.stdout:1/557: dread d3/da/f24 [0,4194304] 0 2024-03-26T18:54:34.967 INFO:tasks.workunit.client.1.smithi190.stdout:0/697: dread d0/dd/d20/d1c/d46/da7/df0/da2/d80/fb2 [0,4194304] 0 2024-03-26T18:54:34.969 INFO:tasks.workunit.client.1.smithi190.stdout:1/558: creat d3/da/dd/d95/fb8 x:0 0 0 2024-03-26T18:54:34.969 INFO:tasks.workunit.client.1.smithi190.stdout:1/559: rename d3 to d3/da/d2e/d59/db9 22 2024-03-26T18:54:34.969 INFO:tasks.workunit.client.1.smithi190.stdout:1/560: read - d3/da/dd/d76/fa6 zero size 2024-03-26T18:54:34.971 INFO:tasks.workunit.client.1.smithi190.stdout:0/698: mkdir d0/dd5/dee/df5/dfd 0 2024-03-26T18:54:34.971 INFO:tasks.workunit.client.1.smithi190.stdout:6/585: dwrite d17/d28/d46/f56 [0,4194304] 0 2024-03-26T18:54:34.972 INFO:tasks.workunit.client.0.smithi102.stdout:0/335: symlink d6/d2d/l6d 0 2024-03-26T18:54:34.972 INFO:tasks.workunit.client.0.smithi102.stdout:0/336: readlink d6/de/l2e 0 2024-03-26T18:54:34.973 INFO:tasks.workunit.client.1.smithi190.stdout:1/561: rename d3/d9/f41 to d3/da/dd/d95/d52/fba 0 2024-03-26T18:54:34.974 INFO:tasks.workunit.client.0.smithi102.stdout:6/308: rename d2/c44 to d2/d3/c5f 0 2024-03-26T18:54:34.976 INFO:tasks.workunit.client.1.smithi190.stdout:0/699: mkdir d0/dd/d20/d1c/d8b/d8d/d8f/dfe 0 2024-03-26T18:54:34.977 INFO:tasks.workunit.client.0.smithi102.stdout:3/484: mkdir df/d29/d31/d77/d8a/dae 0 2024-03-26T18:54:34.977 INFO:tasks.workunit.client.0.smithi102.stdout:3/485: stat df/l21 0 2024-03-26T18:54:34.980 INFO:tasks.workunit.client.0.smithi102.stdout:1/300: link de/d16/d1d/f1e de/d2e/d62/f6b 0 2024-03-26T18:54:34.982 INFO:tasks.workunit.client.0.smithi102.stdout:2/277: creat d3/d20/f62 x:0 0 0 2024-03-26T18:54:34.982 INFO:tasks.workunit.client.0.smithi102.stdout:2/278: write d3/d6/fe [3823437,97265] 0 2024-03-26T18:54:34.982 INFO:tasks.workunit.client.0.smithi102.stdout:2/279: creat d3/d11/d15/d31/d39/f63 x:0 0 0 2024-03-26T18:54:34.983 INFO:tasks.workunit.client.1.smithi190.stdout:4/645: write d7/d1d/f3c [2794088,15032] 0 2024-03-26T18:54:34.983 INFO:tasks.workunit.client.1.smithi190.stdout:4/646: write d7/d8/d14/d23/d28/d38/fcf [533513,59578] 0 2024-03-26T18:54:34.984 INFO:tasks.workunit.client.0.smithi102.stdout:5/544: creat d2/d25/d2a/d4f/fc2 x:0 0 0 2024-03-26T18:54:34.985 INFO:tasks.workunit.client.1.smithi190.stdout:8/676: dwrite d9/dd/f2d [0,4194304] 0 2024-03-26T18:54:34.986 INFO:tasks.workunit.client.0.smithi102.stdout:0/337: dread d6/fb [0,4194304] 0 2024-03-26T18:54:34.986 INFO:tasks.workunit.client.1.smithi190.stdout:5/530: dwrite d2/d17/d1b/f58 [0,4194304] 0 2024-03-26T18:54:34.987 INFO:tasks.workunit.client.1.smithi190.stdout:7/683: dwrite d3/d6/d9/d1f/d4c/f54 [0,4194304] 0 2024-03-26T18:54:34.987 INFO:tasks.workunit.client.0.smithi102.stdout:4/237: dwrite d1/d17/d1f/f48 [0,4194304] 0 2024-03-26T18:54:34.988 INFO:tasks.workunit.client.0.smithi102.stdout:4/238: stat d1/d17/c38 0 2024-03-26T18:54:34.989 INFO:tasks.workunit.client.1.smithi190.stdout:1/562: creat d3/d5f/fbb x:0 0 0 2024-03-26T18:54:34.989 INFO:tasks.workunit.client.1.smithi190.stdout:7/684: dread d3/d6/d9/d1f/fa2 [0,4194304] 0 2024-03-26T18:54:34.990 INFO:tasks.workunit.client.0.smithi102.stdout:3/486: mkdir df/d29/d31/d36/d45/daf 0 2024-03-26T18:54:34.990 INFO:tasks.workunit.client.0.smithi102.stdout:5/545: dread d2/d25/d2a/f2e [0,4194304] 0 2024-03-26T18:54:34.990 INFO:tasks.workunit.client.0.smithi102.stdout:8/340: dwrite d0/d3/d31/f40 [0,4194304] 0 2024-03-26T18:54:34.990 INFO:tasks.workunit.client.0.smithi102.stdout:8/341: write d0/d3/f6 [2370301,103075] 0 2024-03-26T18:54:34.991 INFO:tasks.workunit.client.1.smithi190.stdout:0/700: truncate d0/dd/d20/d1c/d46/da7/df0/f2f 3039835 0 2024-03-26T18:54:34.994 INFO:tasks.workunit.client.0.smithi102.stdout:1/301: rename de/l2f to de/d16/d1d/d61/l6c 0 2024-03-26T18:54:34.996 INFO:tasks.workunit.client.1.smithi190.stdout:6/586: dwrite f9 [8388608,4194304] 0 2024-03-26T18:54:34.997 INFO:tasks.workunit.client.1.smithi190.stdout:4/647: mknod d7/d1d/d42/cde 0 2024-03-26T18:54:34.997 INFO:tasks.workunit.client.1.smithi190.stdout:4/648: truncate d7/d1d/fb5 1330292 0 2024-03-26T18:54:35.003 INFO:tasks.workunit.client.0.smithi102.stdout:0/338: dread d6/de/d13/d4a/f5b [0,4194304] 0 2024-03-26T18:54:35.003 INFO:tasks.workunit.client.0.smithi102.stdout:0/339: fdatasync d6/de/d13/d25/d31/d45/f48 0 2024-03-26T18:54:35.003 INFO:tasks.workunit.client.0.smithi102.stdout:0/340: dread - d6/de/d13/d25/d31/d45/f65 zero size 2024-03-26T18:54:35.004 INFO:tasks.workunit.client.1.smithi190.stdout:5/531: write d2/d17/d1b/db6/d24/f61 [1297806,104183] 0 2024-03-26T18:54:35.004 INFO:tasks.workunit.client.1.smithi190.stdout:8/677: read d9/dd/d23/f27 [3084207,130896] 0 2024-03-26T18:54:35.004 INFO:tasks.workunit.client.1.smithi190.stdout:8/678: fdatasync d9/da2/dc4/f5c 0 2024-03-26T18:54:35.008 INFO:tasks.workunit.client.1.smithi190.stdout:5/532: dread d2/d17/f4f [0,4194304] 0 2024-03-26T18:54:35.008 INFO:tasks.workunit.client.1.smithi190.stdout:5/533: stat d2/d17/d1b/db6/f77 0 2024-03-26T18:54:35.008 INFO:tasks.workunit.client.1.smithi190.stdout:1/563: rename d3/l36 to d3/da/d3a/lbc 0 2024-03-26T18:54:35.009 INFO:tasks.workunit.client.0.smithi102.stdout:4/239: truncate d1/d17/d1a/f1c 2172925 0 2024-03-26T18:54:35.011 INFO:tasks.workunit.client.1.smithi190.stdout:7/685: mknod d3/d6/d9/d1f/d41/d50/dde/dfd/cff 0 2024-03-26T18:54:35.011 INFO:tasks.workunit.client.0.smithi102.stdout:3/487: creat df/d61/fb0 x:0 0 0 2024-03-26T18:54:35.011 INFO:tasks.workunit.client.0.smithi102.stdout:3/488: fdatasync df/d29/f2c 0 2024-03-26T18:54:35.012 INFO:tasks.workunit.client.1.smithi190.stdout:0/701: mknod d0/dd/d20/d1c/d46/d58/de4/cff 0 2024-03-26T18:54:35.013 INFO:tasks.workunit.client.0.smithi102.stdout:3/489: read df/d1c/d34/d53/f80 [64474,47174] 0 2024-03-26T18:54:35.013 INFO:tasks.workunit.client.0.smithi102.stdout:3/490: write df/d29/d31/d36/f7f [38458,49266] 0 2024-03-26T18:54:35.014 INFO:tasks.workunit.client.0.smithi102.stdout:2/280: rename d3/d11/d27/f28 to d3/d11/d15/d31/d39/f64 0 2024-03-26T18:54:35.020 INFO:tasks.workunit.client.1.smithi190.stdout:6/587: link c13 d17/d28/d80/da8/cc8 0 2024-03-26T18:54:35.024 INFO:tasks.workunit.client.0.smithi102.stdout:1/302: dwrite de/d1f/f48 [0,4194304] 0 2024-03-26T18:54:35.025 INFO:tasks.workunit.client.0.smithi102.stdout:5/546: dwrite d2/d25/d5b/d76/fa3 [0,4194304] 0 2024-03-26T18:54:35.025 INFO:tasks.workunit.client.0.smithi102.stdout:8/342: dwrite d0/d3/f28 [0,4194304] 0 2024-03-26T18:54:35.025 INFO:tasks.workunit.client.0.smithi102.stdout:8/343: truncate d0/d3/d31/f5f 1024282 0 2024-03-26T18:54:35.025 INFO:tasks.workunit.client.0.smithi102.stdout:8/344: write d0/d3/d31/f4f [2839835,14375] 0 2024-03-26T18:54:35.026 INFO:tasks.workunit.client.0.smithi102.stdout:8/345: write d0/d15/f4b [707821,84204] 0 2024-03-26T18:54:35.026 INFO:tasks.workunit.client.0.smithi102.stdout:8/346: fsync d0/f4d 0 2024-03-26T18:54:35.026 INFO:tasks.workunit.client.0.smithi102.stdout:3/491: creat df/fb1 x:0 0 0 2024-03-26T18:54:35.026 INFO:tasks.workunit.client.0.smithi102.stdout:4/240: dread d1/d17/d1f/f48 [0,4194304] 0 2024-03-26T18:54:35.027 INFO:tasks.workunit.client.0.smithi102.stdout:2/281: mkdir d3/d65 0 2024-03-26T18:54:35.028 INFO:tasks.workunit.client.0.smithi102.stdout:0/341: getdents d6/de/d13 0 2024-03-26T18:54:35.037 INFO:tasks.workunit.client.1.smithi190.stdout:8/679: creat d9/dd/d23/d88/fe4 x:0 0 0 2024-03-26T18:54:35.038 INFO:tasks.workunit.client.0.smithi102.stdout:4/241: dread d1/d17/d1f/f2d [0,4194304] 0 2024-03-26T18:54:35.041 INFO:tasks.workunit.client.0.smithi102.stdout:5/547: mknod d2/d14/dae/cc3 0 2024-03-26T18:54:35.047 INFO:tasks.workunit.client.1.smithi190.stdout:5/534: unlink d2/d17/f83 0 2024-03-26T18:54:35.048 INFO:tasks.workunit.client.1.smithi190.stdout:5/535: chown d2/d17/d1b/f76 216304254 1 2024-03-26T18:54:35.048 INFO:tasks.workunit.client.0.smithi102.stdout:8/347: rmdir d0/d3/d2c/d4e 39 2024-03-26T18:54:35.048 INFO:tasks.workunit.client.0.smithi102.stdout:8/348: fdatasync d0/d3/d26/f22 0 2024-03-26T18:54:35.049 INFO:tasks.workunit.client.1.smithi190.stdout:7/686: mknod d3/d6/d9/d1f/c100 0 2024-03-26T18:54:35.049 INFO:tasks.workunit.client.1.smithi190.stdout:7/687: chown d3/d6/d9/d1f/d41/d50/dde/fe0 3705378 1 2024-03-26T18:54:35.049 INFO:tasks.workunit.client.1.smithi190.stdout:7/688: write d3/d1b/d98/fc5 [365592,72847] 0 2024-03-26T18:54:35.050 INFO:tasks.workunit.client.0.smithi102.stdout:3/492: mknod df/d29/d31/d77/d8a/dae/cb2 0 2024-03-26T18:54:35.051 INFO:tasks.workunit.client.1.smithi190.stdout:8/680: write d9/da2/dc4/f35 [2058242,69780] 0 2024-03-26T18:54:35.051 INFO:tasks.workunit.client.1.smithi190.stdout:8/681: fsync d9/da2/dc4/f66 0 2024-03-26T18:54:35.052 INFO:tasks.workunit.client.1.smithi190.stdout:4/649: rmdir d7/d1d/d41/d74 39 2024-03-26T18:54:35.053 INFO:tasks.workunit.client.0.smithi102.stdout:0/342: creat d6/de/d13/d25/d4f/d66/f6e x:0 0 0 2024-03-26T18:54:35.054 INFO:tasks.workunit.client.1.smithi190.stdout:1/564: dwrite d3/d9/db/f32 [0,4194304] 0 2024-03-26T18:54:35.056 INFO:tasks.workunit.client.1.smithi190.stdout:5/536: creat d2/d17/d1b/fb7 x:0 0 0 2024-03-26T18:54:35.057 INFO:tasks.workunit.client.0.smithi102.stdout:8/349: dread d0/d3/d31/f40 [0,4194304] 0 2024-03-26T18:54:35.058 INFO:tasks.workunit.client.1.smithi190.stdout:7/689: creat d3/d6/d9/d14/d59/d73/d8b/f101 x:0 0 0 2024-03-26T18:54:35.059 INFO:tasks.workunit.client.0.smithi102.stdout:0/343: dread d6/f11 [0,4194304] 0 2024-03-26T18:54:35.059 INFO:tasks.workunit.client.0.smithi102.stdout:0/344: chown d6/de/d13/d1f/d55/d57/c62 13537 1 2024-03-26T18:54:35.061 INFO:tasks.workunit.client.1.smithi190.stdout:5/537: dread d2/f4 [0,4194304] 0 2024-03-26T18:54:35.064 INFO:tasks.workunit.client.0.smithi102.stdout:1/303: dwrite de/d2e/f49 [0,4194304] 0 2024-03-26T18:54:35.064 INFO:tasks.workunit.client.0.smithi102.stdout:1/304: write de/d1f/f57 [1846009,64243] 0 2024-03-26T18:54:35.064 INFO:tasks.workunit.client.0.smithi102.stdout:1/305: creat de/d1f/f6d x:0 0 0 2024-03-26T18:54:35.064 INFO:tasks.workunit.client.0.smithi102.stdout:1/306: chown de/d16/d1d/d2a/f5d 0 1 2024-03-26T18:54:35.065 INFO:tasks.workunit.client.1.smithi190.stdout:8/682: rename d9/dd/l29 to d9/le5 0 2024-03-26T18:54:35.066 INFO:tasks.workunit.client.0.smithi102.stdout:4/242: dwrite d1/d25/d39/f4e [0,4194304] 0 2024-03-26T18:54:35.067 INFO:tasks.workunit.client.0.smithi102.stdout:2/282: dwrite d3/d20/f62 [0,4194304] 0 2024-03-26T18:54:35.067 INFO:tasks.workunit.client.0.smithi102.stdout:5/548: creat d2/d25/d5b/fc4 x:0 0 0 2024-03-26T18:54:35.067 INFO:tasks.workunit.client.1.smithi190.stdout:0/702: dwrite d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/f3f [0,4194304] 0 2024-03-26T18:54:35.067 INFO:tasks.workunit.client.1.smithi190.stdout:0/703: creat d0/dd/d20/d1c/d4e/f100 x:0 0 0 2024-03-26T18:54:35.069 INFO:tasks.workunit.client.0.smithi102.stdout:3/493: mkdir df/d8b/db3 0 2024-03-26T18:54:35.072 INFO:tasks.workunit.client.1.smithi190.stdout:6/588: rmdir d17/d28/d46 39 2024-03-26T18:54:35.073 INFO:tasks.workunit.client.0.smithi102.stdout:0/345: symlink d6/de/d13/d25/d4f/l6f 0 2024-03-26T18:54:35.075 INFO:tasks.workunit.client.0.smithi102.stdout:8/350: write d0/d3/d2c/d4e/f52 [272836,49655] 0 2024-03-26T18:54:35.076 INFO:tasks.workunit.client.1.smithi190.stdout:4/650: creat d7/d8/d14/d23/d28/d38/d96/fdf x:0 0 0 2024-03-26T18:54:35.079 INFO:tasks.workunit.client.1.smithi190.stdout:8/683: truncate d9/dd/d15/f30 2884025 0 2024-03-26T18:54:35.079 INFO:tasks.workunit.client.1.smithi190.stdout:8/684: fsync d9/dd/d23/d36/f71 0 2024-03-26T18:54:35.083 INFO:tasks.workunit.client.0.smithi102.stdout:0/346: dread d6/de/d13/d1f/f3b [4194304,4194304] 0 2024-03-26T18:54:35.083 INFO:tasks.workunit.client.0.smithi102.stdout:0/347: truncate d6/de/f4c 982198 0 2024-03-26T18:54:35.085 INFO:tasks.workunit.client.1.smithi190.stdout:4/651: dread d7/d1d/d42/fbe [0,4194304] 0 2024-03-26T18:54:35.085 INFO:tasks.workunit.client.1.smithi190.stdout:1/565: rename d3/d9/db/c9d to d3/da/dd/d2f/cbd 0 2024-03-26T18:54:35.085 INFO:tasks.workunit.client.0.smithi102.stdout:1/307: mkdir de/d16/d1d/d2a/d2c/d56/d6e 0 2024-03-26T18:54:35.093 INFO:tasks.workunit.client.0.smithi102.stdout:4/243: truncate d1/f8 7934901 0 2024-03-26T18:54:35.095 INFO:tasks.workunit.client.0.smithi102.stdout:2/283: rmdir d3/d6/d1a 39 2024-03-26T18:54:35.095 INFO:tasks.workunit.client.0.smithi102.stdout:2/284: truncate d3/d11/d27/f57 421764 0 2024-03-26T18:54:35.096 INFO:tasks.workunit.client.1.smithi190.stdout:8/685: dread d9/da2/f63 [0,4194304] 0 2024-03-26T18:54:35.096 INFO:tasks.workunit.client.1.smithi190.stdout:8/686: stat d9/d39/d67/l9a 0 2024-03-26T18:54:35.100 INFO:tasks.workunit.client.0.smithi102.stdout:3/494: symlink df/d29/d31/d36/d3c/d6f/lb4 0 2024-03-26T18:54:35.102 INFO:tasks.workunit.client.1.smithi190.stdout:4/652: truncate d7/d8/d14/d4b/f75 3878958 0 2024-03-26T18:54:35.105 INFO:tasks.workunit.client.1.smithi190.stdout:4/653: write d7/d8/d1c/d20/f58 [4129725,128934] 0 2024-03-26T18:54:35.108 INFO:tasks.workunit.client.0.smithi102.stdout:0/348: mknod d6/d2d/c70 0 2024-03-26T18:54:35.108 INFO:tasks.workunit.client.0.smithi102.stdout:4/244: dread d1/f31 [0,4194304] 0 2024-03-26T18:54:35.111 INFO:tasks.workunit.client.0.smithi102.stdout:1/308: mknod de/d16/d1d/d2a/c6f 0 2024-03-26T18:54:35.111 INFO:tasks.workunit.client.0.smithi102.stdout:1/309: chown de/d2e 261687776 1 2024-03-26T18:54:35.111 INFO:tasks.workunit.client.0.smithi102.stdout:1/310: truncate de/d16/f5e 4962768 0 2024-03-26T18:54:35.111 INFO:tasks.workunit.client.0.smithi102.stdout:1/311: read - de/d16/d1d/d2a/d2c/f66 zero size 2024-03-26T18:54:35.112 INFO:tasks.workunit.client.0.smithi102.stdout:3/495: creat df/d29/d31/fb5 x:0 0 0 2024-03-26T18:54:35.114 INFO:tasks.workunit.client.0.smithi102.stdout:8/351: rename d0/le to d0/d3/d26/l6d 0 2024-03-26T18:54:35.115 INFO:tasks.workunit.client.0.smithi102.stdout:1/312: dread de/d16/d1d/d2a/d2c/d43/f4f [0,4194304] 0 2024-03-26T18:54:35.116 INFO:tasks.workunit.client.0.smithi102.stdout:0/349: mkdir d6/de/d13/d1f/d71 0 2024-03-26T18:54:35.116 INFO:tasks.workunit.client.0.smithi102.stdout:0/350: dread - d6/d2d/d3d/f40 zero size 2024-03-26T18:54:35.118 INFO:tasks.workunit.client.1.smithi190.stdout:7/690: dwrite d3/d6/d9/d1f/d41/d50/f52 [0,4194304] 0 2024-03-26T18:54:35.118 INFO:tasks.workunit.client.0.smithi102.stdout:1/313: dread de/d16/d1d/f28 [0,4194304] 0 2024-03-26T18:54:35.119 INFO:tasks.workunit.client.0.smithi102.stdout:1/314: chown f7 0 1 2024-03-26T18:54:35.119 INFO:tasks.workunit.client.0.smithi102.stdout:1/315: fdatasync de/d2e/d62/f6b 0 2024-03-26T18:54:35.119 INFO:tasks.workunit.client.0.smithi102.stdout:0/351: dread d6/de/d13/f3f [0,4194304] 0 2024-03-26T18:54:35.119 INFO:tasks.workunit.client.0.smithi102.stdout:0/352: write d6/de/d13/f3f [2065731,6039] 0 2024-03-26T18:54:35.119 INFO:tasks.workunit.client.0.smithi102.stdout:8/352: link d0/l55 d0/d3c/l6e 0 2024-03-26T18:54:35.119 INFO:tasks.workunit.client.0.smithi102.stdout:8/353: fsync d0/d3c/d66/f6c 0 2024-03-26T18:54:35.120 INFO:tasks.workunit.client.0.smithi102.stdout:8/354: stat d0/d3c 0 2024-03-26T18:54:35.120 INFO:tasks.workunit.client.0.smithi102.stdout:5/549: dwrite d2/d25/d2a/d4f/f52 [0,4194304] 0 2024-03-26T18:54:35.121 INFO:tasks.workunit.client.0.smithi102.stdout:5/550: dread - d2/d25/d75/db9/d43/f8a zero size 2024-03-26T18:54:35.121 INFO:tasks.workunit.client.0.smithi102.stdout:4/245: getdents d1/d17/d1a 0 2024-03-26T18:54:35.121 INFO:tasks.workunit.client.0.smithi102.stdout:4/246: write d1/d17/d1a/f4b [544557,62470] 0 2024-03-26T18:54:35.122 INFO:tasks.workunit.client.1.smithi190.stdout:0/704: dwrite d0/dd/d20/d1c/d46/d7c/fa9 [0,4194304] 0 2024-03-26T18:54:35.122 INFO:tasks.workunit.client.1.smithi190.stdout:0/705: chown d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/f36 8680 1 2024-03-26T18:54:35.123 INFO:tasks.workunit.client.1.smithi190.stdout:1/566: dwrite d3/da/dd/d2f/d5b/f70 [4194304,4194304] 0 2024-03-26T18:54:35.123 INFO:tasks.workunit.client.1.smithi190.stdout:0/706: read d0/dd/d20/d1c/f6c [219409,68438] 0 2024-03-26T18:54:35.124 INFO:tasks.workunit.client.0.smithi102.stdout:2/285: dwrite d3/d20/f62 [0,4194304] 0 2024-03-26T18:54:35.124 INFO:tasks.workunit.client.0.smithi102.stdout:2/286: readlink d3/l51 0 2024-03-26T18:54:35.124 INFO:tasks.workunit.client.0.smithi102.stdout:1/316: symlink de/d16/d1d/d61/l70 0 2024-03-26T18:54:35.124 INFO:tasks.workunit.client.0.smithi102.stdout:1/317: chown de/c1a 8787972 1 2024-03-26T18:54:35.124 INFO:tasks.workunit.client.0.smithi102.stdout:3/496: dread df/d29/f33 [0,4194304] 0 2024-03-26T18:54:35.125 INFO:tasks.workunit.client.1.smithi190.stdout:8/687: dwrite d9/da2/d87/fe1 [0,4194304] 0 2024-03-26T18:54:35.125 INFO:tasks.workunit.client.1.smithi190.stdout:8/688: chown d9/dd/d23/d36/d37/d3a/c7c 4160533 1 2024-03-26T18:54:35.126 INFO:tasks.workunit.client.0.smithi102.stdout:0/353: stat d6/de/d13/d1f/d55/d57/f30 0 2024-03-26T18:54:35.126 INFO:tasks.workunit.client.0.smithi102.stdout:0/354: dread - d6/de/d13/d25/d31/d45/f48 zero size 2024-03-26T18:54:35.127 INFO:tasks.workunit.client.0.smithi102.stdout:8/355: dread d0/d3/d2c/d4e/f52 [4194304,4194304] 0 2024-03-26T18:54:35.127 INFO:tasks.workunit.client.0.smithi102.stdout:8/356: stat d0/d3/d31/c37 0 2024-03-26T18:54:35.127 INFO:tasks.workunit.client.0.smithi102.stdout:8/357: read - d0/d3c/f43 zero size 2024-03-26T18:54:35.130 INFO:tasks.workunit.client.1.smithi190.stdout:7/691: write d3/d6/d9/d1f/f3d [3528918,18580] 0 2024-03-26T18:54:35.130 INFO:tasks.workunit.client.1.smithi190.stdout:6/589: rmdir d17/d28 39 2024-03-26T18:54:35.130 INFO:tasks.workunit.client.1.smithi190.stdout:6/590: dread - d17/d28/d74/d7b/fb0 zero size 2024-03-26T18:54:35.135 INFO:tasks.workunit.client.1.smithi190.stdout:4/654: link d7/d8/d14/d23/d28/d9e/cb6 d7/d8/d14/d23/d28/d38/dd3/ce0 0 2024-03-26T18:54:35.135 INFO:tasks.workunit.client.1.smithi190.stdout:4/655: chown d7/d8/d1c/f1f 1106 1 2024-03-26T18:54:35.136 INFO:tasks.workunit.client.0.smithi102.stdout:1/318: dread f2 [0,4194304] 0 2024-03-26T18:54:35.136 INFO:tasks.workunit.client.0.smithi102.stdout:1/319: fsync de/d2e/f55 0 2024-03-26T18:54:35.136 INFO:tasks.workunit.client.0.smithi102.stdout:5/551: unlink d2/d25/d75/db9/d43/d9e/d8b/f8e 0 2024-03-26T18:54:35.137 INFO:tasks.workunit.client.1.smithi190.stdout:4/656: read d7/d8/d14/d23/d28/d38/f71 [478585,119380] 0 2024-03-26T18:54:35.140 INFO:tasks.workunit.client.0.smithi102.stdout:4/247: creat d1/d17/d1a/f4f x:0 0 0 2024-03-26T18:54:35.140 INFO:tasks.workunit.client.0.smithi102.stdout:4/248: write d1/f26 [4788341,67808] 0 2024-03-26T18:54:35.140 INFO:tasks.workunit.client.0.smithi102.stdout:4/249: readlink d1/d17/l35 0 2024-03-26T18:54:35.141 INFO:tasks.workunit.client.0.smithi102.stdout:3/497: dwrite df/d61/d55/f7c [0,4194304] 0 2024-03-26T18:54:35.144 INFO:tasks.workunit.client.0.smithi102.stdout:2/287: link d3/d11/d15/d31/l61 d3/d11/d27/l66 0 2024-03-26T18:54:35.145 INFO:tasks.workunit.client.0.smithi102.stdout:2/288: dread d3/d11/d15/d31/d39/f64 [0,4194304] 0 2024-03-26T18:54:35.148 INFO:tasks.workunit.client.0.smithi102.stdout:4/250: dread d1/d25/d39/f4e [0,4194304] 0 2024-03-26T18:54:35.150 INFO:tasks.workunit.client.1.smithi190.stdout:6/591: dwrite d17/d28/d2b/fc4 [0,4194304] 0 2024-03-26T18:54:35.154 INFO:tasks.workunit.client.1.smithi190.stdout:8/689: unlink d9/dd/d23/d36/d37/d3a/d61/f93 0 2024-03-26T18:54:35.154 INFO:tasks.workunit.client.1.smithi190.stdout:8/690: write f6 [1353970,102487] 0 2024-03-26T18:54:35.155 INFO:tasks.workunit.client.1.smithi190.stdout:1/567: dread d3/da/d45/f48 [0,4194304] 0 2024-03-26T18:54:35.158 INFO:tasks.workunit.client.0.smithi102.stdout:0/355: mknod d6/de/d13/d4a/c72 0 2024-03-26T18:54:35.158 INFO:tasks.workunit.client.0.smithi102.stdout:0/356: fsync d6/d3e/d54/f61 0 2024-03-26T18:54:35.159 INFO:tasks.workunit.client.1.smithi190.stdout:3/531: sync 2024-03-26T18:54:35.161 INFO:tasks.workunit.client.1.smithi190.stdout:5/538: rename d2/d17/d1b/d60/d63/d85/d89 to d2/d17/d1b/d60/db8 0 2024-03-26T18:54:35.162 INFO:tasks.workunit.client.0.smithi102.stdout:8/358: rename d0/d9/d65 to d0/d3c/d6f 0 2024-03-26T18:54:35.164 INFO:tasks.workunit.client.1.smithi190.stdout:5/539: dread d2/d17/f44 [0,4194304] 0 2024-03-26T18:54:35.164 INFO:tasks.workunit.client.1.smithi190.stdout:5/540: creat d2/d17/d1b/fb9 x:0 0 0 2024-03-26T18:54:35.165 INFO:tasks.workunit.client.0.smithi102.stdout:1/320: link la de/d1f/l71 0 2024-03-26T18:54:35.167 INFO:tasks.workunit.client.1.smithi190.stdout:9/883: sync 2024-03-26T18:54:35.167 INFO:tasks.workunit.client.1.smithi190.stdout:2/640: sync 2024-03-26T18:54:35.168 INFO:tasks.workunit.client.0.smithi102.stdout:5/552: truncate d2/f17 498192 0 2024-03-26T18:54:35.168 INFO:tasks.workunit.client.0.smithi102.stdout:5/553: write d2/d25/d2a/d4f/f96 [325901,118625] 0 2024-03-26T18:54:35.168 INFO:tasks.workunit.client.0.smithi102.stdout:5/554: write d2/d25/d2a/f59 [645778,114285] 0 2024-03-26T18:54:35.171 INFO:tasks.workunit.client.1.smithi190.stdout:6/592: truncate d17/d28/d74/d97/fb9 271022 0 2024-03-26T18:54:35.171 INFO:tasks.workunit.client.1.smithi190.stdout:6/593: fdatasync d17/d28/d46/f7f 0 2024-03-26T18:54:35.172 INFO:tasks.workunit.client.1.smithi190.stdout:6/594: chown d17/d28/d38/f45 57 1 2024-03-26T18:54:35.172 INFO:tasks.workunit.client.1.smithi190.stdout:6/595: chown d17/c4a 671515 1 2024-03-26T18:54:35.175 INFO:tasks.workunit.client.1.smithi190.stdout:8/691: mknod d9/d39/d79/ce6 0 2024-03-26T18:54:35.176 INFO:tasks.workunit.client.0.smithi102.stdout:2/289: mkdir d3/d2d/d67 0 2024-03-26T18:54:35.178 INFO:tasks.workunit.client.1.smithi190.stdout:1/568: symlink d3/d9/da3/lbe 0 2024-03-26T18:54:35.182 INFO:tasks.workunit.client.1.smithi190.stdout:7/692: rename d3/d6/d9/d1f/d41/d65/d90/d92/d9b/deb to d3/d6/d9/d1f/d93/dfb/d102 0 2024-03-26T18:54:35.182 INFO:tasks.workunit.client.1.smithi190.stdout:7/693: fsync d3/d6/d9/d14/d59/f61 0 2024-03-26T18:54:35.182 INFO:tasks.workunit.client.1.smithi190.stdout:7/694: chown d3/d6/d9/d1f/d4c/d71/dcc/ce4 2423695 1 2024-03-26T18:54:35.183 INFO:tasks.workunit.client.0.smithi102.stdout:4/251: dwrite d1/d17/d1a/f4b [0,4194304] 0 2024-03-26T18:54:35.184 INFO:tasks.workunit.client.1.smithi190.stdout:3/532: dwrite d1/d3/df/d18/d35/d41/d51/f93 [0,4194304] 0 2024-03-26T18:54:35.184 INFO:tasks.workunit.client.1.smithi190.stdout:3/533: readlink d1/l1e 0 2024-03-26T18:54:35.184 INFO:tasks.workunit.client.0.smithi102.stdout:4/252: readlink d1/d17/l1b 0 2024-03-26T18:54:35.185 INFO:tasks.workunit.client.0.smithi102.stdout:0/357: creat d6/de/d13/d25/d4f/f73 x:0 0 0 2024-03-26T18:54:35.189 INFO:tasks.workunit.client.1.smithi190.stdout:2/641: truncate d6/d9/d10/f2d 3232928 0 2024-03-26T18:54:35.189 INFO:tasks.workunit.client.1.smithi190.stdout:2/642: creat d6/d9/d3a/fe4 x:0 0 0 2024-03-26T18:54:35.190 INFO:tasks.workunit.client.0.smithi102.stdout:3/498: rename df/f9c to df/d1c/d70/fb6 0 2024-03-26T18:54:35.192 INFO:tasks.workunit.client.1.smithi190.stdout:9/884: mkdir d12/d2d/d1d/d23/d2b/da0/d138 0 2024-03-26T18:54:35.192 INFO:tasks.workunit.client.0.smithi102.stdout:8/359: stat d0/d9/c17 0 2024-03-26T18:54:35.196 INFO:tasks.workunit.client.1.smithi190.stdout:6/596: dwrite d17/d28/d46/f56 [0,4194304] 0 2024-03-26T18:54:35.196 INFO:tasks.workunit.client.1.smithi190.stdout:8/692: unlink d9/dd/d15/f4b 0 2024-03-26T18:54:35.197 INFO:tasks.workunit.client.0.smithi102.stdout:1/321: truncate f3 654934 0 2024-03-26T18:54:35.198 INFO:tasks.workunit.client.0.smithi102.stdout:0/358: read d6/de/d13/d1f/f27 [179193,19105] 0 2024-03-26T18:54:35.200 INFO:tasks.workunit.client.1.smithi190.stdout:1/569: creat d3/da/d2e/d59/fbf x:0 0 0 2024-03-26T18:54:35.200 INFO:tasks.workunit.client.1.smithi190.stdout:1/570: write d3/d5f/f71 [2086246,130849] 0 2024-03-26T18:54:35.203 INFO:tasks.workunit.client.1.smithi190.stdout:7/695: creat d3/d6/d9/d14/d36/f103 x:0 0 0 2024-03-26T18:54:35.204 INFO:tasks.workunit.client.1.smithi190.stdout:7/696: chown d3/d6/d9/d14/d59/f61 6158523 1 2024-03-26T18:54:35.204 INFO:tasks.workunit.client.1.smithi190.stdout:7/697: write d3/f19 [1227746,87613] 0 2024-03-26T18:54:35.204 INFO:tasks.workunit.client.1.smithi190.stdout:7/698: chown d3/d6/d9/d1f/d41/c68 3 1 2024-03-26T18:54:35.204 INFO:tasks.workunit.client.1.smithi190.stdout:7/699: creat d3/d6/d9/d14/d59/d73/d8b/f104 x:0 0 0 2024-03-26T18:54:35.205 INFO:tasks.workunit.client.0.smithi102.stdout:5/555: truncate d2/d14/d1a/f61 475850 0 2024-03-26T18:54:35.213 INFO:tasks.workunit.client.0.smithi102.stdout:4/253: mknod d1/d25/d39/d3e/c50 0 2024-03-26T18:54:35.213 INFO:tasks.workunit.client.0.smithi102.stdout:4/254: creat d1/d25/d39/f51 x:0 0 0 2024-03-26T18:54:35.214 INFO:tasks.workunit.client.1.smithi190.stdout:6/597: write d17/d28/d2b/f60 [1786494,80935] 0 2024-03-26T18:54:35.214 INFO:tasks.workunit.client.1.smithi190.stdout:6/598: fdatasync d17/d28/d2c/d77/fac 0 2024-03-26T18:54:35.214 INFO:tasks.workunit.client.1.smithi190.stdout:6/599: truncate d17/d28/d80/fc0 577579 0 2024-03-26T18:54:35.214 INFO:tasks.workunit.client.1.smithi190.stdout:6/600: read d17/d28/d2c/d77/f8d [2302504,101483] 0 2024-03-26T18:54:35.215 INFO:tasks.workunit.client.1.smithi190.stdout:6/601: write d17/f25 [4647987,46916] 0 2024-03-26T18:54:35.215 INFO:tasks.workunit.client.0.smithi102.stdout:3/499: creat df/d8b/fb7 x:0 0 0 2024-03-26T18:54:35.216 INFO:tasks.workunit.client.0.smithi102.stdout:2/290: dwrite d3/d6/f1d [4194304,4194304] 0 2024-03-26T18:54:35.216 INFO:tasks.workunit.client.0.smithi102.stdout:1/322: write de/d2e/f49 [733448,81120] 0 2024-03-26T18:54:35.216 INFO:tasks.workunit.client.0.smithi102.stdout:1/323: chown de/d16/d1d/d2a/d2c/d43/f46 4031 1 2024-03-26T18:54:35.218 INFO:tasks.workunit.client.0.smithi102.stdout:0/359: link d6/de/d13/d4a/f68 d6/de/d13/d4a/f74 0 2024-03-26T18:54:35.218 INFO:tasks.workunit.client.0.smithi102.stdout:0/360: write d6/de/d13/d1f/d39/f29 [3121526,69557] 0 2024-03-26T18:54:35.219 INFO:tasks.workunit.client.0.smithi102.stdout:5/556: creat d2/d25/d5b/d72/fc5 x:0 0 0 2024-03-26T18:54:35.222 INFO:tasks.workunit.client.0.smithi102.stdout:4/255: mknod d1/d25/d39/c52 0 2024-03-26T18:54:35.225 INFO:tasks.workunit.client.0.smithi102.stdout:3/500: rename df/d29/d31/d77/d8a/f97 to df/d29/d31/d36/d45/daf/fb8 0 2024-03-26T18:54:35.228 INFO:tasks.workunit.client.0.smithi102.stdout:4/256: write d1/f31 [1320973,19557] 0 2024-03-26T18:54:35.228 INFO:tasks.workunit.client.0.smithi102.stdout:2/291: creat d3/d2d/f68 x:0 0 0 2024-03-26T18:54:35.228 INFO:tasks.workunit.client.0.smithi102.stdout:4/257: dread - d1/d27/f4d zero size 2024-03-26T18:54:35.229 INFO:tasks.workunit.client.0.smithi102.stdout:7/280: sync 2024-03-26T18:54:35.229 INFO:tasks.workunit.client.0.smithi102.stdout:9/249: sync 2024-03-26T18:54:35.229 INFO:tasks.workunit.client.0.smithi102.stdout:7/281: readlink d7/d12/l2e 0 2024-03-26T18:54:35.235 INFO:tasks.workunit.client.0.smithi102.stdout:0/361: symlink d6/d3e/l75 0 2024-03-26T18:54:35.236 INFO:tasks.workunit.client.1.smithi190.stdout:2/643: dwrite d6/d21/d61/fc2 [0,4194304] 0 2024-03-26T18:54:35.236 INFO:tasks.workunit.client.1.smithi190.stdout:2/644: read - d6/d9/d10/f59 zero size 2024-03-26T18:54:35.236 INFO:tasks.workunit.client.1.smithi190.stdout:5/541: rmdir d2/d17 39 2024-03-26T18:54:35.236 INFO:tasks.workunit.client.1.smithi190.stdout:3/534: dwrite d1/d3/d52/d70/f85 [0,4194304] 0 2024-03-26T18:54:35.236 INFO:tasks.workunit.client.1.smithi190.stdout:3/535: write d1/d3/f34 [1497115,38180] 0 2024-03-26T18:54:35.237 INFO:tasks.workunit.client.0.smithi102.stdout:8/360: dwrite d0/d3/d31/f59 [0,4194304] 0 2024-03-26T18:54:35.237 INFO:tasks.workunit.client.1.smithi190.stdout:9/885: dwrite d12/d2d/fd9 [0,4194304] 0 2024-03-26T18:54:35.237 INFO:tasks.workunit.client.1.smithi190.stdout:9/886: fdatasync d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/db3/fca 0 2024-03-26T18:54:35.237 INFO:tasks.workunit.client.1.smithi190.stdout:8/693: dwrite d9/da2/f2b [0,4194304] 0 2024-03-26T18:54:35.238 INFO:tasks.workunit.client.1.smithi190.stdout:1/571: creat d3/da/dd/d2f/d5b/d5d/fc0 x:0 0 0 2024-03-26T18:54:35.243 INFO:tasks.workunit.client.0.smithi102.stdout:9/250: dread df/d1a/f52 [0,4194304] 0 2024-03-26T18:54:35.243 INFO:tasks.workunit.client.0.smithi102.stdout:9/251: fsync df/d1a/d23/d4e/f4f 0 2024-03-26T18:54:35.243 INFO:tasks.workunit.client.0.smithi102.stdout:9/252: dread - df/d1a/d1d/f3a zero size 2024-03-26T18:54:35.243 INFO:tasks.workunit.client.1.smithi190.stdout:7/700: mkdir d3/d6/d9/d1f/d93/d105 0 2024-03-26T18:54:35.244 INFO:tasks.workunit.client.0.smithi102.stdout:1/324: rename de/d16/d1d/d2a/c6f to de/d16/d1d/d2a/d2c/d56/d6e/c72 0 2024-03-26T18:54:35.245 INFO:tasks.workunit.client.1.smithi190.stdout:6/602: getdents d17/d28/d46/d6a 0 2024-03-26T18:54:35.245 INFO:tasks.workunit.client.1.smithi190.stdout:6/603: chown d17/d28/d2c/d77/c7a 189155 1 2024-03-26T18:54:35.246 INFO:tasks.workunit.client.1.smithi190.stdout:8/694: dread d9/dd/f59 [0,4194304] 0 2024-03-26T18:54:35.251 INFO:tasks.workunit.client.0.smithi102.stdout:4/258: dwrite d1/d17/d1a/f4f [0,4194304] 0 2024-03-26T18:54:35.252 INFO:tasks.workunit.client.0.smithi102.stdout:3/501: unlink df/f2e 0 2024-03-26T18:54:35.252 INFO:tasks.workunit.client.0.smithi102.stdout:3/502: chown df/d8b 764 1 2024-03-26T18:54:35.252 INFO:tasks.workunit.client.1.smithi190.stdout:7/701: dread d3/f29 [0,4194304] 0 2024-03-26T18:54:35.252 INFO:tasks.workunit.client.1.smithi190.stdout:7/702: fsync d3/d6/fac 0 2024-03-26T18:54:35.252 INFO:tasks.workunit.client.1.smithi190.stdout:7/703: fdatasync d3/d6/d9/d1f/d4c/f7f 0 2024-03-26T18:54:35.252 INFO:tasks.workunit.client.1.smithi190.stdout:7/704: chown d3/d6/d9/d1f/dab 966 1 2024-03-26T18:54:35.253 INFO:tasks.workunit.client.0.smithi102.stdout:2/292: mkdir d3/d2d/d69 0 2024-03-26T18:54:35.254 INFO:tasks.workunit.client.0.smithi102.stdout:2/293: stat d3/d2a/l3a 0 2024-03-26T18:54:35.260 INFO:tasks.workunit.client.1.smithi190.stdout:8/695: dread d9/d39/d67/fa3 [0,4194304] 0 2024-03-26T18:54:35.261 INFO:tasks.workunit.client.0.smithi102.stdout:2/294: dread d3/d6/f16 [0,4194304] 0 2024-03-26T18:54:35.266 INFO:tasks.workunit.client.1.smithi190.stdout:8/696: dread d9/da2/d73/fb0 [0,4194304] 0 2024-03-26T18:54:35.282 INFO:tasks.workunit.client.1.smithi190.stdout:7/705: dwrite d3/f19 [0,4194304] 0 2024-03-26T18:54:35.283 INFO:tasks.workunit.client.1.smithi190.stdout:8/697: dwrite d9/da2/dc4/f66 [0,4194304] 0 2024-03-26T18:54:35.284 INFO:tasks.workunit.client.1.smithi190.stdout:8/698: write d9/da2/dc4/f53 [323360,110088] 0 2024-03-26T18:54:35.299 INFO:tasks.workunit.client.1.smithi190.stdout:2/645: link d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d7c/dac/cc5 d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d7c/ce5 0 2024-03-26T18:54:35.299 INFO:tasks.workunit.client.1.smithi190.stdout:2/646: truncate d6/d9/d45/d6b/f80 126736 0 2024-03-26T18:54:35.300 INFO:tasks.workunit.client.0.smithi102.stdout:7/282: mknod d7/d4e/c52 0 2024-03-26T18:54:35.300 INFO:tasks.workunit.client.0.smithi102.stdout:7/283: truncate d7/d10/d27/d34/f48 513540 0 2024-03-26T18:54:35.300 INFO:tasks.workunit.client.0.smithi102.stdout:7/284: stat d7/c3e 0 2024-03-26T18:54:35.302 INFO:tasks.workunit.client.0.smithi102.stdout:6/309: sync 2024-03-26T18:54:35.307 INFO:tasks.workunit.client.1.smithi190.stdout:8/699: dwrite d9/dd/d15/f30 [0,4194304] 0 2024-03-26T18:54:35.317 INFO:tasks.workunit.client.0.smithi102.stdout:8/361: symlink d0/d3/l70 0 2024-03-26T18:54:35.317 INFO:tasks.workunit.client.0.smithi102.stdout:8/362: stat d0/d3/d26/c47 0 2024-03-26T18:54:35.317 INFO:tasks.workunit.client.0.smithi102.stdout:8/363: stat d0/d3/d26 0 2024-03-26T18:54:35.320 INFO:tasks.workunit.client.0.smithi102.stdout:9/253: symlink df/d1a/d23/d29/d45/l61 0 2024-03-26T18:54:35.320 INFO:tasks.workunit.client.0.smithi102.stdout:9/254: fsync df/d1a/d23/d4e/f53 0 2024-03-26T18:54:35.324 INFO:tasks.workunit.client.0.smithi102.stdout:5/557: rename d2/d25/d2a/db4/lbe to d2/d25/d75/db9/db3/lc6 0 2024-03-26T18:54:35.324 INFO:tasks.workunit.client.0.smithi102.stdout:9/255: dread df/f27 [0,4194304] 0 2024-03-26T18:54:35.325 INFO:tasks.workunit.client.0.smithi102.stdout:7/285: dwrite d7/d10/d17/f37 [0,4194304] 0 2024-03-26T18:54:35.325 INFO:tasks.workunit.client.1.smithi190.stdout:8/700: dwrite d9/dd/d23/d36/d37/d3a/d61/fd6 [0,4194304] 0 2024-03-26T18:54:35.326 INFO:tasks.workunit.client.1.smithi190.stdout:1/572: truncate d3/fab 440467 0 2024-03-26T18:54:35.326 INFO:tasks.workunit.client.1.smithi190.stdout:1/573: creat d3/d9/db/fc1 x:0 0 0 2024-03-26T18:54:35.326 INFO:tasks.workunit.client.1.smithi190.stdout:1/574: fsync d3/da/f24 0 2024-03-26T18:54:35.326 INFO:tasks.workunit.client.1.smithi190.stdout:1/575: stat d3/d9/db/fc1 0 2024-03-26T18:54:35.326 INFO:tasks.workunit.client.1.smithi190.stdout:1/576: readlink d3/da/dd/d2f/d34/l6e 0 2024-03-26T18:54:35.327 INFO:tasks.workunit.client.0.smithi102.stdout:1/325: creat de/f73 x:0 0 0 2024-03-26T18:54:35.327 INFO:tasks.workunit.client.0.smithi102.stdout:1/326: fdatasync f8 0 2024-03-26T18:54:35.337 INFO:tasks.workunit.client.0.smithi102.stdout:3/503: mknod df/d29/d31/d36/d3c/da7/cb9 0 2024-03-26T18:54:35.337 INFO:tasks.workunit.client.0.smithi102.stdout:3/504: stat df/d61/d55/f7c 0 2024-03-26T18:54:35.337 INFO:tasks.workunit.client.0.smithi102.stdout:7/286: dread d7/d10/d27/f33 [0,4194304] 0 2024-03-26T18:54:35.341 INFO:tasks.workunit.client.0.smithi102.stdout:2/295: creat d3/d2d/d69/f6a x:0 0 0 2024-03-26T18:54:35.366 INFO:tasks.workunit.client.0.smithi102.stdout:7/287: dwrite d7/f41 [0,4194304] 0 2024-03-26T18:54:35.366 INFO:tasks.workunit.client.0.smithi102.stdout:3/505: dwrite df/d29/d31/d36/d3c/f3f [0,4194304] 0 2024-03-26T18:54:35.366 INFO:tasks.workunit.client.0.smithi102.stdout:3/506: dread df/d29/d3a/f85 [0,4194304] 0 2024-03-26T18:54:35.369 INFO:tasks.workunit.client.1.smithi190.stdout:5/542: rename d2/d17/d1b/db6/d24/d3b/f47 to d2/d17/d98/d3a/fba 0 2024-03-26T18:54:35.382 INFO:tasks.workunit.client.0.smithi102.stdout:7/288: dwrite d7/d12/d4d/f4f [0,4194304] 0 2024-03-26T18:54:35.382 INFO:tasks.workunit.client.0.smithi102.stdout:7/289: truncate d7/f41 5010092 0 2024-03-26T18:54:35.382 INFO:tasks.workunit.client.0.smithi102.stdout:7/290: truncate d7/d12/f38 430389 0 2024-03-26T18:54:35.409 INFO:tasks.workunit.client.0.smithi102.stdout:6/310: mkdir d2/df/d35/d60 0 2024-03-26T18:54:35.409 INFO:tasks.workunit.client.0.smithi102.stdout:6/311: creat d2/df/d35/d43/f61 x:0 0 0 2024-03-26T18:54:35.413 INFO:tasks.workunit.client.0.smithi102.stdout:8/364: rmdir d0/d3/d2c 39 2024-03-26T18:54:35.416 INFO:tasks.workunit.client.0.smithi102.stdout:4/259: rename d1/d17/d1f/f48 to d1/d17/d1a/f53 0 2024-03-26T18:54:35.421 INFO:tasks.workunit.client.0.smithi102.stdout:9/256: creat df/d1a/d1d/f62 x:0 0 0 2024-03-26T18:54:35.421 INFO:tasks.workunit.client.0.smithi102.stdout:9/257: dread - df/d1a/f24 zero size 2024-03-26T18:54:35.428 INFO:tasks.workunit.client.0.smithi102.stdout:2/296: truncate d3/d6/f2b 649160 0 2024-03-26T18:54:35.428 INFO:tasks.workunit.client.0.smithi102.stdout:2/297: chown d3/d11/d15/d31/d39/f64 3969 1 2024-03-26T18:54:35.429 INFO:tasks.workunit.client.0.smithi102.stdout:2/298: read d3/d11/d15/d31/d39/f64 [137084,13871] 0 2024-03-26T18:54:35.429 INFO:tasks.workunit.client.0.smithi102.stdout:2/299: truncate d3/d11/d15/d31/f60 906422 0 2024-03-26T18:54:35.435 INFO:tasks.workunit.client.0.smithi102.stdout:3/507: mknod df/d61/d55/cba 0 2024-03-26T18:54:35.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:35 smithi190 ceph-mon[35043]: pgmap v245: 65 pgs: 65 active+clean; 1.5 GiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 147 MiB/s rd, 185 MiB/s wr, 730 op/s 2024-03-26T18:54:35.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:35 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:35.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:35 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:35.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:35 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:54:35.438 INFO:tasks.workunit.client.1.smithi190.stdout:7/706: mknod d3/d6/d9/d1f/d93/dfb/d102/c106 0 2024-03-26T18:54:35.440 INFO:tasks.workunit.client.0.smithi102.stdout:2/300: dread d3/d11/d15/d31/d39/f5a [0,4194304] 0 2024-03-26T18:54:35.440 INFO:tasks.workunit.client.0.smithi102.stdout:2/301: truncate d3/d6/f1b 919168 0 2024-03-26T18:54:35.440 INFO:tasks.workunit.client.0.smithi102.stdout:2/302: creat d3/d11/d27/f6b x:0 0 0 2024-03-26T18:54:35.442 INFO:tasks.workunit.client.0.smithi102.stdout:7/291: creat d7/d44/f53 x:0 0 0 2024-03-26T18:54:35.447 INFO:tasks.workunit.client.0.smithi102.stdout:8/365: rename d0/d3/d2c/l68 to d0/d3/d31/l71 0 2024-03-26T18:54:35.447 INFO:tasks.workunit.client.0.smithi102.stdout:8/366: readlink d0/d3/d26/d19/l50 0 2024-03-26T18:54:35.448 INFO:tasks.workunit.client.0.smithi102.stdout:2/303: dread d3/fb [0,4194304] 0 2024-03-26T18:54:35.448 INFO:tasks.workunit.client.0.smithi102.stdout:4/260: truncate d1/f31 3194326 0 2024-03-26T18:54:35.449 INFO:tasks.workunit.client.1.smithi190.stdout:7/707: dread d3/d6/f30 [0,4194304] 0 2024-03-26T18:54:35.449 INFO:tasks.workunit.client.1.smithi190.stdout:7/708: stat d3/d6/d9/d1f/d4c/d71/dcc/ce4 0 2024-03-26T18:54:35.455 INFO:tasks.workunit.client.0.smithi102.stdout:9/258: unlink cd 0 2024-03-26T18:54:35.458 INFO:tasks.workunit.client.0.smithi102.stdout:1/327: getdents de/d16/d1d/d2a 0 2024-03-26T18:54:35.458 INFO:tasks.workunit.client.0.smithi102.stdout:1/328: fsync de/d16/d1d/d2a/f5d 0 2024-03-26T18:54:35.458 INFO:tasks.workunit.client.0.smithi102.stdout:1/329: truncate f9 3370450 0 2024-03-26T18:54:35.460 INFO:tasks.workunit.client.0.smithi102.stdout:4/261: dread d1/d17/d1a/d21/f2c [8388608,4194304] 0 2024-03-26T18:54:35.462 INFO:tasks.workunit.client.0.smithi102.stdout:3/508: mkdir df/d8b/dbb 0 2024-03-26T18:54:35.462 INFO:tasks.workunit.client.0.smithi102.stdout:3/509: write df/d29/d31/d36/d3c/f3f [4809366,89602] 0 2024-03-26T18:54:35.462 INFO:tasks.workunit.client.0.smithi102.stdout:3/510: fdatasync df/d1c/d34/f37 0 2024-03-26T18:54:35.462 INFO:tasks.workunit.client.0.smithi102.stdout:3/511: dread - df/d8b/fb7 zero size 2024-03-26T18:54:35.464 INFO:tasks.workunit.client.0.smithi102.stdout:7/292: stat d7/ld 0 2024-03-26T18:54:35.465 INFO:tasks.workunit.client.0.smithi102.stdout:7/293: readlink d7/ld 0 2024-03-26T18:54:35.468 INFO:tasks.workunit.client.0.smithi102.stdout:8/367: rename d0/d9/c13 to d0/d15/d1b/c72 0 2024-03-26T18:54:35.469 INFO:tasks.workunit.client.0.smithi102.stdout:4/262: dread d1/f9 [0,4194304] 0 2024-03-26T18:54:35.469 INFO:tasks.workunit.client.0.smithi102.stdout:4/263: readlink d1/d17/d2a/l3a 0 2024-03-26T18:54:35.471 INFO:tasks.workunit.client.0.smithi102.stdout:2/304: getdents d3/d11/d15/d31/d58 0 2024-03-26T18:54:35.474 INFO:tasks.workunit.client.0.smithi102.stdout:9/259: mkdir df/d46/d63 0 2024-03-26T18:54:35.474 INFO:tasks.workunit.client.0.smithi102.stdout:9/260: dread - df/d1a/f5e zero size 2024-03-26T18:54:35.474 INFO:tasks.workunit.client.0.smithi102.stdout:9/261: stat df/d1a/d1d/f32 0 2024-03-26T18:54:35.476 INFO:tasks.workunit.client.0.smithi102.stdout:1/330: creat de/d16/d1d/d2a/f74 x:0 0 0 2024-03-26T18:54:35.482 INFO:tasks.workunit.client.0.smithi102.stdout:7/294: link d7/d10/d17/l23 d7/d2a/l54 0 2024-03-26T18:54:35.489 INFO:tasks.workunit.client.0.smithi102.stdout:4/264: rmdir d1/d25/d39/d47 0 2024-03-26T18:54:35.489 INFO:tasks.workunit.client.0.smithi102.stdout:8/368: dwrite d0/d15/f16 [0,4194304] 0 2024-03-26T18:54:35.489 INFO:tasks.workunit.client.0.smithi102.stdout:8/369: dread - d0/d3/d26/d19/d27/f5b zero size 2024-03-26T18:54:35.489 INFO:tasks.workunit.client.0.smithi102.stdout:8/370: creat d0/d3/d26/d19/d27/f73 x:0 0 0 2024-03-26T18:54:35.489 INFO:tasks.workunit.client.0.smithi102.stdout:8/371: write d0/d3c/d66/f6c [473290,22629] 0 2024-03-26T18:54:35.492 INFO:tasks.workunit.client.0.smithi102.stdout:9/262: creat df/d1a/d57/f64 x:0 0 0 2024-03-26T18:54:35.493 INFO:tasks.workunit.client.0.smithi102.stdout:9/263: truncate df/d1a/d23/d29/f34 638714 0 2024-03-26T18:54:35.495 INFO:tasks.workunit.client.0.smithi102.stdout:1/331: creat de/d16/d1d/d2a/f75 x:0 0 0 2024-03-26T18:54:35.496 INFO:tasks.workunit.client.0.smithi102.stdout:7/295: rmdir d7/d12/d2b 39 2024-03-26T18:54:35.499 INFO:tasks.workunit.client.1.smithi190.stdout:8/701: read d9/fa [4035575,80747] 0 2024-03-26T18:54:35.499 INFO:tasks.workunit.client.1.smithi190.stdout:3/536: rename d1/d3/df/d18/d57/d91/d8c/f8d to d1/de/d4e/fa0 0 2024-03-26T18:54:35.499 INFO:tasks.workunit.client.1.smithi190.stdout:8/702: chown d9/da2/f63 20839 1 2024-03-26T18:54:35.499 INFO:tasks.workunit.client.1.smithi190.stdout:3/537: truncate d1/d3/df/d18/d35/d41/d51/f96 829659 0 2024-03-26T18:54:35.499 INFO:tasks.workunit.client.1.smithi190.stdout:3/538: fsync d1/f11 0 2024-03-26T18:54:35.500 INFO:tasks.workunit.client.0.smithi102.stdout:4/265: creat d1/d17/d2a/f54 x:0 0 0 2024-03-26T18:54:35.501 INFO:tasks.workunit.client.1.smithi190.stdout:5/543: getdents d2/d17/d1b/db6/d24/d3b/da5 0 2024-03-26T18:54:35.504 INFO:tasks.workunit.client.0.smithi102.stdout:3/512: rename df/d29/l6b to df/d29/d31/d36/d3c/d6f/d75/lbc 0 2024-03-26T18:54:35.504 INFO:tasks.workunit.client.0.smithi102.stdout:3/513: write df/d1c/d34/f37 [427173,112488] 0 2024-03-26T18:54:35.504 INFO:tasks.workunit.client.0.smithi102.stdout:3/514: stat df/d29/f2c 0 2024-03-26T18:54:35.504 INFO:tasks.workunit.client.0.smithi102.stdout:3/515: chown df/d29/l2b 752 1 2024-03-26T18:54:35.507 INFO:tasks.workunit.client.1.smithi190.stdout:7/709: mknod d3/d6/d9/d1f/d41/d50/c107 0 2024-03-26T18:54:35.511 INFO:tasks.workunit.client.0.smithi102.stdout:8/372: link d0/d9/c17 d0/d3/d2c/d2f/c74 0 2024-03-26T18:54:35.511 INFO:tasks.workunit.client.0.smithi102.stdout:8/373: dread - d0/d15/f67 zero size 2024-03-26T18:54:35.511 INFO:tasks.workunit.client.0.smithi102.stdout:8/374: read d0/d3/d2c/f42 [62922,111522] 0 2024-03-26T18:54:35.515 INFO:tasks.workunit.client.0.smithi102.stdout:9/264: rename df/d1a/d33 to df/d1a/d65 0 2024-03-26T18:54:35.515 INFO:tasks.workunit.client.0.smithi102.stdout:9/265: creat df/d1a/f66 x:0 0 0 2024-03-26T18:54:35.516 INFO:tasks.workunit.client.0.smithi102.stdout:5/558: symlink d2/d25/d75/db9/lc7 0 2024-03-26T18:54:35.519 INFO:tasks.workunit.client.0.smithi102.stdout:3/516: creat df/d29/d31/d77/d8a/fbd x:0 0 0 2024-03-26T18:54:35.522 INFO:tasks.workunit.client.0.smithi102.stdout:8/375: stat d0/d3/d26/f1f 0 2024-03-26T18:54:35.525 INFO:tasks.workunit.client.0.smithi102.stdout:4/266: dwrite d1/d17/d1a/f53 [4194304,4194304] 0 2024-03-26T18:54:35.525 INFO:tasks.workunit.client.0.smithi102.stdout:4/267: write d1/d17/d1f/f2d [1189522,62774] 0 2024-03-26T18:54:35.525 INFO:tasks.workunit.client.0.smithi102.stdout:7/296: dwrite d7/f20 [4194304,4194304] 0 2024-03-26T18:54:35.531 INFO:tasks.workunit.client.0.smithi102.stdout:8/376: dread d0/d3/d2c/d2f/f53 [0,4194304] 0 2024-03-26T18:54:35.535 INFO:tasks.workunit.client.0.smithi102.stdout:1/332: mknod de/d16/d1d/d2a/d2c/d56/d5f/c76 0 2024-03-26T18:54:35.535 INFO:tasks.workunit.client.0.smithi102.stdout:7/297: dread f6 [0,4194304] 0 2024-03-26T18:54:35.535 INFO:tasks.workunit.client.0.smithi102.stdout:7/298: write f6 [946404,101974] 0 2024-03-26T18:54:35.536 INFO:tasks.workunit.client.0.smithi102.stdout:7/299: write d7/d10/d27/d34/f48 [6174,8375] 0 2024-03-26T18:54:35.536 INFO:tasks.workunit.client.0.smithi102.stdout:5/559: write d2/d14/d1a/f1b [4155690,84462] 0 2024-03-26T18:54:35.536 INFO:tasks.workunit.client.1.smithi190.stdout:2/647: rename d6/d9/d3a/d3d/d2e/cbe to d6/ce6 0 2024-03-26T18:54:35.537 INFO:tasks.workunit.client.1.smithi190.stdout:2/648: read - d6/d9/d3a/fda zero size 2024-03-26T18:54:35.537 INFO:tasks.workunit.client.1.smithi190.stdout:2/649: chown d6/ce6 74371 1 2024-03-26T18:54:35.542 INFO:tasks.workunit.client.0.smithi102.stdout:4/268: unlink d1/d17/d1a/f1c 0 2024-03-26T18:54:35.545 INFO:tasks.workunit.client.1.smithi190.stdout:5/544: creat d2/d17/d1b/d8e/fbb x:0 0 0 2024-03-26T18:54:35.545 INFO:tasks.workunit.client.1.smithi190.stdout:5/545: write d2/d17/d98/d3a/f55 [916670,17210] 0 2024-03-26T18:54:35.545 INFO:tasks.workunit.client.0.smithi102.stdout:3/517: dwrite df/d29/d31/d36/d3c/f6a [0,4194304] 0 2024-03-26T18:54:35.545 INFO:tasks.workunit.client.0.smithi102.stdout:3/518: write df/d29/d31/f6c [697280,75259] 0 2024-03-26T18:54:35.547 INFO:tasks.workunit.client.1.smithi190.stdout:7/710: creat d3/d6/d9/d1f/d41/d65/d90/d92/d9b/dc4/f108 x:0 0 0 2024-03-26T18:54:35.551 INFO:tasks.workunit.client.0.smithi102.stdout:4/269: write d1/d27/f3c [3249884,80998] 0 2024-03-26T18:54:35.561 INFO:tasks.workunit.client.0.smithi102.stdout:8/377: mkdir d0/d3/d2c/d58/d75 0 2024-03-26T18:54:35.562 INFO:tasks.workunit.client.1.smithi190.stdout:1/577: unlink d3/da/dd/d95/f46 0 2024-03-26T18:54:35.562 INFO:tasks.workunit.client.1.smithi190.stdout:1/578: creat d3/da/d45/fc2 x:0 0 0 2024-03-26T18:54:35.563 INFO:tasks.workunit.client.0.smithi102.stdout:1/333: truncate de/f14 609406 0 2024-03-26T18:54:35.563 INFO:tasks.workunit.client.0.smithi102.stdout:1/334: chown de/d16/d1d/d2a/d2c/f66 645 1 2024-03-26T18:54:35.563 INFO:tasks.workunit.client.0.smithi102.stdout:1/335: fdatasync de/d16/d1d/d2a/d2c/f66 0 2024-03-26T18:54:35.567 INFO:tasks.workunit.client.0.smithi102.stdout:7/300: creat d7/d10/d27/f55 x:0 0 0 2024-03-26T18:54:35.571 INFO:tasks.workunit.client.0.smithi102.stdout:5/560: rename d2/d14/f2f to d2/d2c/d6f/db8/fc8 0 2024-03-26T18:54:35.581 INFO:tasks.workunit.client.0.smithi102.stdout:8/378: mkdir d0/d3/d26/d19/d76 0 2024-03-26T18:54:35.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:35 smithi102 ceph-mon[29002]: pgmap v245: 65 pgs: 65 active+clean; 1.5 GiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 147 MiB/s rd, 185 MiB/s wr, 730 op/s 2024-03-26T18:54:35.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:35 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:35.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:35 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:35.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:35 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:54:35.653 INFO:tasks.workunit.client.0.smithi102.stdout:1/336: link de/d16/d1d/d2a/f74 de/d16/d1d/d61/f77 0 2024-03-26T18:54:35.654 INFO:tasks.workunit.client.1.smithi190.stdout:5/546: rename d2/d17/d1b/db6/d88/d8d to d2/d17/d1b/db6/d88/dbc 0 2024-03-26T18:54:35.659 INFO:tasks.workunit.client.0.smithi102.stdout:7/301: mkdir d7/d31/d56 0 2024-03-26T18:54:35.659 INFO:tasks.workunit.client.1.smithi190.stdout:7/711: mkdir d3/d6/d9/d1f/d41/d65/d109 0 2024-03-26T18:54:35.661 INFO:tasks.workunit.client.1.smithi190.stdout:1/579: getdents d3/da/dd/d2f/d33/d9c 0 2024-03-26T18:54:35.661 INFO:tasks.workunit.client.1.smithi190.stdout:1/580: fdatasync d3/da/dd/f8c 0 2024-03-26T18:54:35.661 INFO:tasks.workunit.client.1.smithi190.stdout:1/581: dread - d3/da/d45/f49 zero size 2024-03-26T18:54:35.661 INFO:tasks.workunit.client.1.smithi190.stdout:1/582: chown d3/da/f94 13672600 1 2024-03-26T18:54:35.663 INFO:tasks.workunit.client.0.smithi102.stdout:3/519: rename df/d29/d31/f6c to df/d1c/d70/fbe 0 2024-03-26T18:54:35.666 INFO:tasks.workunit.client.1.smithi190.stdout:5/547: truncate d2/d17/d1b/db6/d24/f61 2230952 0 2024-03-26T18:54:35.666 INFO:tasks.workunit.client.1.smithi190.stdout:5/548: stat d2/d17/d62/d64/f70 0 2024-03-26T18:54:35.666 INFO:tasks.workunit.client.1.smithi190.stdout:5/549: write d2/f4 [1560293,124798] 0 2024-03-26T18:54:35.666 INFO:tasks.workunit.client.1.smithi190.stdout:5/550: write d2/d17/d1b/db6/f54 [3713920,96110] 0 2024-03-26T18:54:35.667 INFO:tasks.workunit.client.0.smithi102.stdout:5/561: rmdir d2/d2c/d6f 39 2024-03-26T18:54:35.667 INFO:tasks.workunit.client.0.smithi102.stdout:5/562: stat d2/d25/f26 0 2024-03-26T18:54:35.667 INFO:tasks.workunit.client.1.smithi190.stdout:5/551: fdatasync d2/d17/f44 0 2024-03-26T18:54:35.670 INFO:tasks.workunit.client.1.smithi190.stdout:7/712: truncate d3/f43 456689 0 2024-03-26T18:54:35.670 INFO:tasks.workunit.client.1.smithi190.stdout:7/713: creat d3/d6/d9/d14/d36/f10a x:0 0 0 2024-03-26T18:54:35.671 INFO:tasks.workunit.client.1.smithi190.stdout:7/714: fdatasync d3/d1b/f66 0 2024-03-26T18:54:35.690 INFO:tasks.workunit.client.0.smithi102.stdout:3/520: unlink df/d29/d3a/f85 0 2024-03-26T18:54:35.690 INFO:tasks.workunit.client.0.smithi102.stdout:3/521: stat df/d1c/c5c 0 2024-03-26T18:54:35.691 INFO:tasks.workunit.client.0.smithi102.stdout:5/563: mkdir d2/d25/d75/db9/d43/d9e/d82/dc9 0 2024-03-26T18:54:35.692 INFO:tasks.workunit.client.0.smithi102.stdout:8/379: link d0/d3c/l6e d0/d9/l77 0 2024-03-26T18:54:35.693 INFO:tasks.workunit.client.0.smithi102.stdout:1/337: dwrite de/d16/f51 [0,4194304] 0 2024-03-26T18:54:35.695 INFO:tasks.workunit.client.0.smithi102.stdout:7/302: dwrite d7/d10/d27/d34/f49 [0,4194304] 0 2024-03-26T18:54:35.695 INFO:tasks.workunit.client.0.smithi102.stdout:7/303: chown d7/d10/f1c 8732076 1 2024-03-26T18:54:35.695 INFO:tasks.workunit.client.0.smithi102.stdout:7/304: truncate d7/d2a/f30 1755444 0 2024-03-26T18:54:35.695 INFO:tasks.workunit.client.0.smithi102.stdout:7/305: chown d7/ff 77710 1 2024-03-26T18:54:35.695 INFO:tasks.workunit.client.0.smithi102.stdout:7/306: write d7/d12/f38 [787906,50003] 0 2024-03-26T18:54:35.695 INFO:tasks.workunit.client.0.smithi102.stdout:7/307: readlink d7/d10/d17/l23 0 2024-03-26T18:54:35.695 INFO:tasks.workunit.client.0.smithi102.stdout:7/308: truncate d7/d2a/f4a 129780 0 2024-03-26T18:54:35.696 INFO:tasks.workunit.client.1.smithi190.stdout:1/583: dwrite d3/da/d3a/f50 [0,4194304] 0 2024-03-26T18:54:35.696 INFO:tasks.workunit.client.1.smithi190.stdout:1/584: stat d3/da/dd/d95/ca4 0 2024-03-26T18:54:35.696 INFO:tasks.workunit.client.1.smithi190.stdout:1/585: truncate d3/da/d3a/f55 702235 0 2024-03-26T18:54:35.696 INFO:tasks.workunit.client.1.smithi190.stdout:1/586: readlink d3/d9/db/l2d 0 2024-03-26T18:54:35.696 INFO:tasks.workunit.client.1.smithi190.stdout:7/715: dwrite d3/f2f [0,4194304] 0 2024-03-26T18:54:35.697 INFO:tasks.workunit.client.0.smithi102.stdout:3/522: rename df/d29/d31/d36/d3c/f6a to df/d1c/fbf 0 2024-03-26T18:54:35.697 INFO:tasks.workunit.client.0.smithi102.stdout:5/564: symlink d2/d25/d2a/d4f/da9/lca 0 2024-03-26T18:54:35.697 INFO:tasks.workunit.client.0.smithi102.stdout:5/565: write d2/d25/d5b/d72/f22 [3508871,116524] 0 2024-03-26T18:54:35.708 INFO:tasks.workunit.client.1.smithi190.stdout:5/552: rmdir d2/d17 39 2024-03-26T18:54:35.708 INFO:tasks.workunit.client.1.smithi190.stdout:5/553: chown d2/d17/d1b/d30/l34 48956 1 2024-03-26T18:54:35.708 INFO:tasks.workunit.client.1.smithi190.stdout:5/554: fdatasync d2/d17/d1b/d60/d63/fa3 0 2024-03-26T18:54:35.708 INFO:tasks.workunit.client.1.smithi190.stdout:5/555: fdatasync d2/d17/f21 0 2024-03-26T18:54:35.712 INFO:tasks.workunit.client.1.smithi190.stdout:5/556: symlink d2/d17/d1b/d8e/lbd 0 2024-03-26T18:54:35.714 INFO:tasks.workunit.client.0.smithi102.stdout:7/309: mknod d7/d12/d51/c57 0 2024-03-26T18:54:35.714 INFO:tasks.workunit.client.0.smithi102.stdout:7/310: truncate d7/d10/d27/d34/f49 4616145 0 2024-03-26T18:54:35.714 INFO:tasks.workunit.client.0.smithi102.stdout:7/311: fsync d7/ff 0 2024-03-26T18:54:35.717 INFO:tasks.workunit.client.0.smithi102.stdout:7/312: creat d7/d12/d51/f58 x:0 0 0 2024-03-26T18:54:35.728 INFO:tasks.workunit.client.0.smithi102.stdout:9/266: rmdir df/d1a 39 2024-03-26T18:54:35.730 INFO:tasks.workunit.client.1.smithi190.stdout:2/650: creat d6/d9/d3a/d3d/d1c/fe7 x:0 0 0 2024-03-26T18:54:35.733 INFO:tasks.workunit.client.1.smithi190.stdout:2/651: mkdir d6/d9/d3a/d3d/d1c/d8d/de8 0 2024-03-26T18:54:35.733 INFO:tasks.workunit.client.1.smithi190.stdout:2/652: write d6/d9/d3a/d3d/d2e/f9a [3425627,67345] 0 2024-03-26T18:54:35.734 INFO:tasks.workunit.client.0.smithi102.stdout:9/267: creat df/d46/f67 x:0 0 0 2024-03-26T18:54:35.736 INFO:tasks.workunit.client.1.smithi190.stdout:5/557: read d2/d17/d1b/db6/f7f [428646,5935] 0 2024-03-26T18:54:35.736 INFO:tasks.workunit.client.1.smithi190.stdout:5/558: write d2/d17/d1b/db6/d68/f75 [726219,69938] 0 2024-03-26T18:54:35.736 INFO:tasks.workunit.client.1.smithi190.stdout:5/559: chown d2/le 166541 1 2024-03-26T18:54:35.736 INFO:tasks.workunit.client.1.smithi190.stdout:5/560: truncate d2/d17/d98/d3a/faa 1000412 0 2024-03-26T18:54:35.737 INFO:tasks.workunit.client.1.smithi190.stdout:2/653: symlink d6/d9/d3a/le9 0 2024-03-26T18:54:35.738 INFO:tasks.workunit.client.0.smithi102.stdout:9/268: symlink df/d1a/d57/l68 0 2024-03-26T18:54:35.739 INFO:tasks.workunit.client.0.smithi102.stdout:5/566: dwrite d2/d25/d2a/d4f/fb2 [0,4194304] 0 2024-03-26T18:54:35.739 INFO:tasks.workunit.client.0.smithi102.stdout:5/567: write d2/d25/d5b/d72/fc5 [973415,73329] 0 2024-03-26T18:54:35.740 INFO:tasks.workunit.client.1.smithi190.stdout:7/716: dwrite d3/d1b/d57/ff5 [0,4194304] 0 2024-03-26T18:54:35.740 INFO:tasks.workunit.client.1.smithi190.stdout:7/717: creat d3/d1b/dd3/f10b x:0 0 0 2024-03-26T18:54:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:7/313: dwrite d7/d10/d27/d34/f48 [0,4194304] 0 2024-03-26T18:54:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:8/380: dwrite d0/d3/d26/d5c/f62 [0,4194304] 0 2024-03-26T18:54:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:7/314: readlink d7/ld 0 2024-03-26T18:54:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:7/315: creat d7/d10/d17/f59 x:0 0 0 2024-03-26T18:54:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:7/316: chown d7/d4e/c52 99840 1 2024-03-26T18:54:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:3/523: dwrite df/d29/d31/d36/d45/daf/fb8 [0,4194304] 0 2024-03-26T18:54:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:3/524: stat df/f25 0 2024-03-26T18:54:35.742 INFO:tasks.workunit.client.1.smithi190.stdout:1/587: dwrite d3/f29 [4194304,4194304] 0 2024-03-26T18:54:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:9/269: symlink df/d1a/d65/l69 0 2024-03-26T18:54:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:9/270: chown df/d1a/d1d/l4b 0 1 2024-03-26T18:54:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:9/271: creat df/d46/f6a x:0 0 0 2024-03-26T18:54:35.745 INFO:tasks.workunit.client.0.smithi102.stdout:0/362: symlink d6/de/d13/d1f/l76 0 2024-03-26T18:54:35.747 INFO:tasks.workunit.client.0.smithi102.stdout:2/305: mknod d3/d11/c6c 0 2024-03-26T18:54:35.747 INFO:tasks.workunit.client.0.smithi102.stdout:2/306: truncate d3/d2a/f2f 4854150 0 2024-03-26T18:54:35.747 INFO:tasks.workunit.client.0.smithi102.stdout:2/307: chown d3/d2a/d5f 18357 1 2024-03-26T18:54:35.749 INFO:tasks.workunit.client.0.smithi102.stdout:5/568: unlink d2/lc 0 2024-03-26T18:54:35.751 INFO:tasks.workunit.client.1.smithi190.stdout:2/654: dread d6/d21/d61/fc2 [0,4194304] 0 2024-03-26T18:54:35.751 INFO:tasks.workunit.client.0.smithi102.stdout:3/525: write df/d29/d31/d36/d3c/d6f/f7d [4007857,108935] 0 2024-03-26T18:54:35.751 INFO:tasks.workunit.client.1.smithi190.stdout:3/539: mknod d1/d3/df/d18/d35/ca1 0 2024-03-26T18:54:35.754 INFO:tasks.workunit.client.1.smithi190.stdout:7/718: creat d3/d6/d9/d14/f10c x:0 0 0 2024-03-26T18:54:35.755 INFO:tasks.workunit.client.1.smithi190.stdout:7/719: dread - d3/d6/d9/d1f/d41/d65/d90/d92/fa1 zero size 2024-03-26T18:54:35.759 INFO:tasks.workunit.client.1.smithi190.stdout:1/588: creat d3/d73/d88/fc3 x:0 0 0 2024-03-26T18:54:35.759 INFO:tasks.workunit.client.1.smithi190.stdout:1/589: dread - d3/d73/d88/fad zero size 2024-03-26T18:54:35.759 INFO:tasks.workunit.client.1.smithi190.stdout:1/590: write d3/da/dd/d2f/d33/fac [5044538,41757] 0 2024-03-26T18:54:35.760 INFO:tasks.workunit.client.0.smithi102.stdout:7/317: symlink d7/d10/d17/l5a 0 2024-03-26T18:54:35.761 INFO:tasks.workunit.client.1.smithi190.stdout:1/591: mkdir d3/da/dd/d76/dc4 0 2024-03-26T18:54:35.761 INFO:tasks.workunit.client.1.smithi190.stdout:1/592: stat d3/l75 0 2024-03-26T18:54:35.761 INFO:tasks.workunit.client.0.smithi102.stdout:7/318: read d7/d12/f22 [362368,44268] 0 2024-03-26T18:54:35.762 INFO:tasks.workunit.client.0.smithi102.stdout:7/319: read d7/d10/f14 [358547,90892] 0 2024-03-26T18:54:35.763 INFO:tasks.workunit.client.0.smithi102.stdout:7/320: write d7/d10/f14 [670621,54888] 0 2024-03-26T18:54:35.763 INFO:tasks.workunit.client.0.smithi102.stdout:7/321: stat d7/d10/d17/l1e 0 2024-03-26T18:54:35.763 INFO:tasks.workunit.client.0.smithi102.stdout:7/322: chown d7/d12/f43 1915999 1 2024-03-26T18:54:35.763 INFO:tasks.workunit.client.1.smithi190.stdout:2/655: dread d6/f36 [0,4194304] 0 2024-03-26T18:54:35.765 INFO:tasks.workunit.client.1.smithi190.stdout:3/540: dread d1/f4 [0,4194304] 0 2024-03-26T18:54:35.766 INFO:tasks.workunit.client.1.smithi190.stdout:3/541: read d1/de/d1c/f83 [58888,35365] 0 2024-03-26T18:54:35.766 INFO:tasks.workunit.client.1.smithi190.stdout:3/542: chown d1/d3/df/d18/d1b/f9b 95382720 1 2024-03-26T18:54:35.768 INFO:tasks.workunit.client.0.smithi102.stdout:2/308: read d3/d20/f4c [3821962,43741] 0 2024-03-26T18:54:35.769 INFO:tasks.workunit.client.1.smithi190.stdout:2/656: mkdir d6/d9/d3a/d3d/d1c/d8d/dc7/dd1/dea 0 2024-03-26T18:54:35.769 INFO:tasks.workunit.client.1.smithi190.stdout:2/657: chown d6/d9/d10/f87 90 1 2024-03-26T18:54:35.770 INFO:tasks.workunit.client.0.smithi102.stdout:0/363: getdents d6/d2d 0 2024-03-26T18:54:35.773 INFO:tasks.workunit.client.1.smithi190.stdout:2/658: mkdir d6/d9/d45/d6b/deb 0 2024-03-26T18:54:35.773 INFO:tasks.workunit.client.0.smithi102.stdout:8/381: dwrite d0/d15/f6b [0,4194304] 0 2024-03-26T18:54:35.773 INFO:tasks.workunit.client.0.smithi102.stdout:8/382: write d0/d3c/d66/f6c [589680,106154] 0 2024-03-26T18:54:35.774 INFO:tasks.workunit.client.1.smithi190.stdout:5/561: dwrite d2/d17/d1b/db6/f2d [0,4194304] 0 2024-03-26T18:54:35.774 INFO:tasks.workunit.client.0.smithi102.stdout:5/569: rename d2/d25/d75/db9/l4a to d2/lcb 0 2024-03-26T18:54:35.780 INFO:tasks.workunit.client.0.smithi102.stdout:3/526: mkdir df/d29/d31/dc0 0 2024-03-26T18:54:35.781 INFO:tasks.workunit.client.1.smithi190.stdout:2/659: dread d6/d9/d10/d5f/fad [0,4194304] 0 2024-03-26T18:54:35.782 INFO:tasks.workunit.client.0.smithi102.stdout:7/323: mknod d7/d44/c5b 0 2024-03-26T18:54:35.782 INFO:tasks.workunit.client.0.smithi102.stdout:7/324: write d7/d12/f38 [1379485,1745] 0 2024-03-26T18:54:35.782 INFO:tasks.workunit.client.0.smithi102.stdout:7/325: fsync d7/d12/f43 0 2024-03-26T18:54:35.782 INFO:tasks.workunit.client.0.smithi102.stdout:7/326: creat d7/d10/d17/f5c x:0 0 0 2024-03-26T18:54:35.782 INFO:tasks.workunit.client.0.smithi102.stdout:7/327: write d7/d2a/f4a [733780,124538] 0 2024-03-26T18:54:35.782 INFO:tasks.workunit.client.0.smithi102.stdout:7/328: creat d7/d10/d17/f5d x:0 0 0 2024-03-26T18:54:35.783 INFO:tasks.workunit.client.1.smithi190.stdout:3/543: dwrite d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:35.784 INFO:tasks.workunit.client.0.smithi102.stdout:2/309: rename d3/d2a/f40 to d3/d2a/f6d 0 2024-03-26T18:54:35.787 INFO:tasks.workunit.client.1.smithi190.stdout:5/562: creat d2/d17/d1b/fbe x:0 0 0 2024-03-26T18:54:35.790 INFO:tasks.workunit.client.0.smithi102.stdout:0/364: mkdir d6/d3e/d54/d77 0 2024-03-26T18:54:35.791 INFO:tasks.workunit.client.1.smithi190.stdout:3/544: link d1/de/d1c/f83 d1/d3/d52/fa2 0 2024-03-26T18:54:35.794 INFO:tasks.workunit.client.1.smithi190.stdout:5/563: truncate d2/d17/f21 1134845 0 2024-03-26T18:54:35.795 INFO:tasks.workunit.client.1.smithi190.stdout:5/564: chown d2/d17/d98/l49 1700828 1 2024-03-26T18:54:35.795 INFO:tasks.workunit.client.1.smithi190.stdout:5/565: chown d2/d17/d1b/d8e/l92 3 1 2024-03-26T18:54:35.795 INFO:tasks.workunit.client.1.smithi190.stdout:5/566: fsync d2/f4 0 2024-03-26T18:54:35.796 INFO:tasks.workunit.client.1.smithi190.stdout:3/545: mknod d1/de/ca3 0 2024-03-26T18:54:35.800 INFO:tasks.workunit.client.0.smithi102.stdout:7/329: dwrite d7/d10/f1c [0,4194304] 0 2024-03-26T18:54:35.800 INFO:tasks.workunit.client.0.smithi102.stdout:7/330: chown d7/d12/l3f 197 1 2024-03-26T18:54:35.800 INFO:tasks.workunit.client.0.smithi102.stdout:7/331: readlink d7/d12/l3f 0 2024-03-26T18:54:35.804 INFO:tasks.workunit.client.0.smithi102.stdout:8/383: link d0/d3/d31/f40 d0/d3/d2c/d2f/f78 0 2024-03-26T18:54:35.804 INFO:tasks.workunit.client.0.smithi102.stdout:8/384: write d0/d3/d31/f4f [1063974,117225] 0 2024-03-26T18:54:35.805 INFO:tasks.workunit.client.0.smithi102.stdout:5/570: mknod d2/d25/d75/db9/d43/d9e/d82/dc9/ccc 0 2024-03-26T18:54:35.805 INFO:tasks.workunit.client.0.smithi102.stdout:5/571: fsync d2/d25/d5b/d72/f57 0 2024-03-26T18:54:35.805 INFO:tasks.workunit.client.0.smithi102.stdout:5/572: chown d2/d2c/d6f 2097597124 1 2024-03-26T18:54:35.814 INFO:tasks.workunit.client.0.smithi102.stdout:2/310: mkdir d3/d65/d6e 0 2024-03-26T18:54:35.815 INFO:tasks.workunit.client.0.smithi102.stdout:2/311: stat d3/d11/d15/d31/l45 0 2024-03-26T18:54:35.815 INFO:tasks.workunit.client.0.smithi102.stdout:0/365: mknod d6/c78 0 2024-03-26T18:54:35.819 INFO:tasks.workunit.client.0.smithi102.stdout:8/385: link d0/d9/c56 d0/d3/d2c/d2f/c79 0 2024-03-26T18:54:35.819 INFO:tasks.workunit.client.0.smithi102.stdout:8/386: write d0/d3/f32 [957692,69111] 0 2024-03-26T18:54:35.823 INFO:tasks.workunit.client.0.smithi102.stdout:3/527: getdents df/d61 0 2024-03-26T18:54:35.823 INFO:tasks.workunit.client.0.smithi102.stdout:3/528: chown df/f1b 5 1 2024-03-26T18:54:35.824 INFO:tasks.workunit.client.0.smithi102.stdout:8/387: write d0/d3/d2c/f34 [1278466,27388] 0 2024-03-26T18:54:35.826 INFO:tasks.workunit.client.0.smithi102.stdout:2/312: mknod d3/d11/d15/d31/d58/c6f 0 2024-03-26T18:54:35.829 INFO:tasks.workunit.client.0.smithi102.stdout:3/529: mknod df/d1c/d70/cc1 0 2024-03-26T18:54:35.829 INFO:tasks.workunit.client.0.smithi102.stdout:3/530: fsync df/d1c/d70/fb6 0 2024-03-26T18:54:35.832 INFO:tasks.workunit.client.0.smithi102.stdout:3/531: creat df/d29/d31/d36/d3c/fc2 x:0 0 0 2024-03-26T18:54:35.833 INFO:tasks.workunit.client.0.smithi102.stdout:0/366: dwrite d6/de/d13/d1f/d55/d57/f5e [0,4194304] 0 2024-03-26T18:54:35.833 INFO:tasks.workunit.client.0.smithi102.stdout:0/367: write d6/de/d13/d1f/d39/f29 [2872536,77810] 0 2024-03-26T18:54:35.833 INFO:tasks.workunit.client.0.smithi102.stdout:0/368: chown d6/de/d13/d25/d4f/c6b 16388 1 2024-03-26T18:54:35.834 INFO:tasks.workunit.client.0.smithi102.stdout:0/369: write d6/d2d/d3d/f44 [3752905,127626] 0 2024-03-26T18:54:35.834 INFO:tasks.workunit.client.0.smithi102.stdout:9/272: getdents df/d1a/d65 0 2024-03-26T18:54:35.834 INFO:tasks.workunit.client.0.smithi102.stdout:9/273: chown df/d1a/d65 4522258 1 2024-03-26T18:54:35.841 INFO:tasks.workunit.client.0.smithi102.stdout:0/370: symlink d6/d3e/d54/l79 0 2024-03-26T18:54:35.842 INFO:tasks.workunit.client.0.smithi102.stdout:9/274: creat df/d1a/d23/d3e/f6b x:0 0 0 2024-03-26T18:54:35.845 INFO:tasks.workunit.client.0.smithi102.stdout:0/371: dread d6/f51 [0,4194304] 0 2024-03-26T18:54:35.849 INFO:tasks.workunit.client.0.smithi102.stdout:0/372: symlink d6/de/d13/d1f/l7a 0 2024-03-26T18:54:35.849 INFO:tasks.workunit.client.0.smithi102.stdout:0/373: dread - d6/de/d13/d25/d4f/f73 zero size 2024-03-26T18:54:35.871 INFO:tasks.workunit.client.0.smithi102.stdout:2/313: dwrite d3/d2a/f37 [0,4194304] 0 2024-03-26T18:54:35.871 INFO:tasks.workunit.client.0.smithi102.stdout:2/314: readlink d3/d6/ld 0 2024-03-26T18:54:35.873 INFO:tasks.workunit.client.0.smithi102.stdout:8/388: dwrite d0/d3/d31/f59 [0,4194304] 0 2024-03-26T18:54:35.873 INFO:tasks.workunit.client.0.smithi102.stdout:8/389: getdents d0/d3c/d6f 0 2024-03-26T18:54:35.873 INFO:tasks.workunit.client.0.smithi102.stdout:8/390: write d0/d3/d2c/f42 [1003739,37618] 0 2024-03-26T18:54:35.873 INFO:tasks.workunit.client.0.smithi102.stdout:9/275: dwrite f7 [0,4194304] 0 2024-03-26T18:54:35.873 INFO:tasks.workunit.client.0.smithi102.stdout:9/276: fdatasync df/d1a/d23/f2c 0 2024-03-26T18:54:35.873 INFO:tasks.workunit.client.0.smithi102.stdout:9/277: readlink df/d1a/d23/d29/l41 0 2024-03-26T18:54:35.873 INFO:tasks.workunit.client.0.smithi102.stdout:8/391: write d0/d3/d31/f5f [116370,480] 0 2024-03-26T18:54:35.873 INFO:tasks.workunit.client.0.smithi102.stdout:0/374: dwrite d6/de/d13/d1f/f27 [4194304,4194304] 0 2024-03-26T18:54:35.873 INFO:tasks.workunit.client.0.smithi102.stdout:0/375: chown d6 490177715 1 2024-03-26T18:54:35.873 INFO:tasks.workunit.client.0.smithi102.stdout:0/376: read d6/fb [4603019,74165] 0 2024-03-26T18:54:35.873 INFO:tasks.workunit.client.0.smithi102.stdout:3/532: dwrite df/d1c/d34/f8e [0,4194304] 0 2024-03-26T18:54:35.874 INFO:tasks.workunit.client.0.smithi102.stdout:7/332: truncate d7/d2a/f30 346794 0 2024-03-26T18:54:35.874 INFO:tasks.workunit.client.0.smithi102.stdout:7/333: readlink d7/ld 0 2024-03-26T18:54:35.875 INFO:tasks.workunit.client.0.smithi102.stdout:2/315: rename d3/d6/l22 to d3/d2a/d5f/l70 0 2024-03-26T18:54:35.882 INFO:tasks.workunit.client.0.smithi102.stdout:9/278: mknod df/d1a/d65/c6c 0 2024-03-26T18:54:35.883 INFO:tasks.workunit.client.0.smithi102.stdout:8/392: link d0/d3/d31/f40 d0/d3/d26/d19/f7a 0 2024-03-26T18:54:35.886 INFO:tasks.workunit.client.0.smithi102.stdout:3/533: symlink df/d29/d31/lc3 0 2024-03-26T18:54:35.887 INFO:tasks.workunit.client.0.smithi102.stdout:0/377: write d6/de/d13/d1f/d55/d57/f1c [8734,47174] 0 2024-03-26T18:54:35.887 INFO:tasks.workunit.client.0.smithi102.stdout:0/378: readlink d6/l53 0 2024-03-26T18:54:35.889 INFO:tasks.workunit.client.0.smithi102.stdout:2/316: truncate d3/f8 352886 0 2024-03-26T18:54:35.889 INFO:tasks.workunit.client.0.smithi102.stdout:2/317: truncate d3/d11/d15/d31/f60 4711323 0 2024-03-26T18:54:35.891 INFO:tasks.workunit.client.0.smithi102.stdout:9/279: creat df/d1a/d57/f6d x:0 0 0 2024-03-26T18:54:35.891 INFO:tasks.workunit.client.0.smithi102.stdout:9/280: chown df/d46 624721238 1 2024-03-26T18:54:35.893 INFO:tasks.workunit.client.0.smithi102.stdout:7/334: dwrite d7/d10/d17/f59 [0,4194304] 0 2024-03-26T18:54:35.894 INFO:tasks.workunit.client.0.smithi102.stdout:8/393: rename d0/d15/d1b/f4c to d0/d3/d2c/d4e/f7b 0 2024-03-26T18:54:35.895 INFO:tasks.workunit.client.0.smithi102.stdout:2/318: mknod d3/d11/d15/d46/c71 0 2024-03-26T18:54:35.895 INFO:tasks.workunit.client.0.smithi102.stdout:2/319: write d3/d20/f4c [3648225,33258] 0 2024-03-26T18:54:35.900 INFO:tasks.workunit.client.0.smithi102.stdout:7/335: rename d7/d12/d4d/f4f to d7/d12/d4d/f5e 0 2024-03-26T18:54:35.903 INFO:tasks.workunit.client.0.smithi102.stdout:7/336: mknod d7/d10/c5f 0 2024-03-26T18:54:35.904 INFO:tasks.workunit.client.0.smithi102.stdout:7/337: creat d7/d2a/f60 x:0 0 0 2024-03-26T18:54:35.907 INFO:tasks.workunit.client.0.smithi102.stdout:9/281: write df/f3b [2329959,32467] 0 2024-03-26T18:54:35.930 INFO:tasks.workunit.client.0.smithi102.stdout:3/534: dwrite df/f26 [4194304,4194304] 0 2024-03-26T18:54:35.930 INFO:tasks.workunit.client.0.smithi102.stdout:8/394: dwrite d0/d3/d31/f3a [0,4194304] 0 2024-03-26T18:54:35.930 INFO:tasks.workunit.client.0.smithi102.stdout:0/379: dwrite d6/d2d/d3d/f40 [0,4194304] 0 2024-03-26T18:54:35.935 INFO:tasks.workunit.client.0.smithi102.stdout:0/380: dread d6/de/d13/d1f/f27 [0,4194304] 0 2024-03-26T18:54:35.935 INFO:tasks.workunit.client.0.smithi102.stdout:3/535: dread df/d29/f33 [0,4194304] 0 2024-03-26T18:54:35.935 INFO:tasks.workunit.client.0.smithi102.stdout:2/320: dwrite d3/f1c [0,4194304] 0 2024-03-26T18:54:35.935 INFO:tasks.workunit.client.0.smithi102.stdout:3/536: fdatasync df/d29/d31/d36/f7f 0 2024-03-26T18:54:35.935 INFO:tasks.workunit.client.0.smithi102.stdout:2/321: write d3/d6/f2b [1321919,89849] 0 2024-03-26T18:54:35.945 INFO:tasks.workunit.client.0.smithi102.stdout:9/282: creat df/d1a/f6e x:0 0 0 2024-03-26T18:54:35.945 INFO:tasks.workunit.client.0.smithi102.stdout:9/283: creat df/d1a/f6f x:0 0 0 2024-03-26T18:54:35.945 INFO:tasks.workunit.client.0.smithi102.stdout:6/312: sync 2024-03-26T18:54:35.945 INFO:tasks.workunit.client.0.smithi102.stdout:6/313: chown d2/d3d/l55 2 1 2024-03-26T18:54:35.946 INFO:tasks.workunit.client.0.smithi102.stdout:6/314: chown d2/d3/c5f 28 1 2024-03-26T18:54:35.946 INFO:tasks.workunit.client.0.smithi102.stdout:6/315: chown d2/d7/c37 1617 1 2024-03-26T18:54:35.970 INFO:tasks.workunit.client.1.smithi190.stdout:9/887: link d12/d2d/d1d/d23/f69 d12/d2d/d1d/d23/d83/dc9/f139 0 2024-03-26T18:54:35.972 INFO:tasks.workunit.client.1.smithi190.stdout:8/703: rename d9/d39/d79/f7d to d9/da2/d87/fe7 0 2024-03-26T18:54:35.974 INFO:tasks.workunit.client.1.smithi190.stdout:9/888: creat d12/d2d/d1d/da1/dba/dbc/f13a x:0 0 0 2024-03-26T18:54:35.974 INFO:tasks.workunit.client.1.smithi190.stdout:9/889: truncate d12/d2d/d1d/d23/f33 5203151 0 2024-03-26T18:54:35.975 INFO:tasks.workunit.client.1.smithi190.stdout:8/704: link d9/d48/d49/c55 d9/d39/d67/dc1/ce8 0 2024-03-26T18:54:35.977 INFO:tasks.workunit.client.1.smithi190.stdout:8/705: creat d9/d39/d79/fe9 x:0 0 0 2024-03-26T18:54:35.977 INFO:tasks.workunit.client.1.smithi190.stdout:8/706: read - d9/dd/d23/d36/d37/d3a/d61/daf/db3/fe2 zero size 2024-03-26T18:54:35.978 INFO:tasks.workunit.client.1.smithi190.stdout:8/707: mkdir d9/dd/d23/dea 0 2024-03-26T18:54:35.978 INFO:tasks.workunit.client.1.smithi190.stdout:8/708: chown d9/d48/f50 127295 1 2024-03-26T18:54:35.987 INFO:tasks.workunit.client.1.smithi190.stdout:9/890: write d12/d2d/fd9 [3772506,48043] 0 2024-03-26T18:54:35.988 INFO:tasks.workunit.client.1.smithi190.stdout:9/891: fdatasync d12/d2d/d1d/da1/dba/dbc/f13a 0 2024-03-26T18:54:35.988 INFO:tasks.workunit.client.0.smithi102.stdout:8/395: mknod d0/d15/c7c 0 2024-03-26T18:54:35.990 INFO:tasks.workunit.client.1.smithi190.stdout:9/892: symlink d12/d2d/d1d/d23/d83/l13b 0 2024-03-26T18:54:35.990 INFO:tasks.workunit.client.0.smithi102.stdout:0/381: mkdir d6/de/d13/d25/d7b 0 2024-03-26T18:54:35.990 INFO:tasks.workunit.client.1.smithi190.stdout:9/893: symlink d12/d2d/d1d/d23/d2b/da0/d138/l13c 0 2024-03-26T18:54:35.993 INFO:tasks.workunit.client.1.smithi190.stdout:0/707: sync 2024-03-26T18:54:35.993 INFO:tasks.workunit.client.1.smithi190.stdout:6/604: sync 2024-03-26T18:54:35.993 INFO:tasks.workunit.client.1.smithi190.stdout:4/657: sync 2024-03-26T18:54:35.993 INFO:tasks.workunit.client.1.smithi190.stdout:2/660: sync 2024-03-26T18:54:35.993 INFO:tasks.workunit.client.1.smithi190.stdout:1/593: sync 2024-03-26T18:54:35.993 INFO:tasks.workunit.client.1.smithi190.stdout:1/594: getdents d3/da/db2 0 2024-03-26T18:54:35.996 INFO:tasks.workunit.client.0.smithi102.stdout:3/537: mknod df/d1c/d34/d9b/cc4 0 2024-03-26T18:54:35.999 INFO:tasks.workunit.client.0.smithi102.stdout:2/322: mknod d3/d2a/d5f/c72 0 2024-03-26T18:54:36.002 INFO:tasks.workunit.client.0.smithi102.stdout:7/338: dwrite d7/d10/d17/f59 [4194304,4194304] 0 2024-03-26T18:54:36.005 INFO:tasks.workunit.client.1.smithi190.stdout:0/708: dread d0/dd/f12 [0,4194304] 0 2024-03-26T18:54:36.006 INFO:tasks.workunit.client.1.smithi190.stdout:0/709: dread - d0/dd/d20/d1c/d46/da7/df0/da2/fc0 zero size 2024-03-26T18:54:36.006 INFO:tasks.workunit.client.1.smithi190.stdout:0/710: getdents d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d66/dcb/de7 0 2024-03-26T18:54:36.008 INFO:tasks.workunit.client.0.smithi102.stdout:3/538: dread fd [0,4194304] 0 2024-03-26T18:54:36.009 INFO:tasks.workunit.client.0.smithi102.stdout:9/284: symlink df/d1a/l70 0 2024-03-26T18:54:36.012 INFO:tasks.workunit.client.0.smithi102.stdout:6/316: symlink d2/d12/l62 0 2024-03-26T18:54:36.014 INFO:tasks.workunit.client.0.smithi102.stdout:8/396: creat d0/d9/f7d x:0 0 0 2024-03-26T18:54:36.015 INFO:tasks.workunit.client.0.smithi102.stdout:4/270: sync 2024-03-26T18:54:36.015 INFO:tasks.workunit.client.1.smithi190.stdout:9/894: dwrite f11 [0,4194304] 0 2024-03-26T18:54:36.016 INFO:tasks.workunit.client.0.smithi102.stdout:1/338: sync 2024-03-26T18:54:36.017 INFO:tasks.workunit.client.0.smithi102.stdout:0/382: mkdir d6/de/d13/d25/d7b/d7c 0 2024-03-26T18:54:36.022 INFO:tasks.workunit.client.0.smithi102.stdout:7/339: symlink d7/d31/l61 0 2024-03-26T18:54:36.022 INFO:tasks.workunit.client.0.smithi102.stdout:7/340: readlink d7/l25 0 2024-03-26T18:54:36.023 INFO:tasks.workunit.client.0.smithi102.stdout:0/383: dread d6/f51 [0,4194304] 0 2024-03-26T18:54:36.025 INFO:tasks.workunit.client.0.smithi102.stdout:3/539: creat df/d29/d31/d77/d8a/dae/fc5 x:0 0 0 2024-03-26T18:54:36.028 INFO:tasks.workunit.client.0.smithi102.stdout:2/323: dwrite d3/d6/ff [8388608,4194304] 0 2024-03-26T18:54:36.029 INFO:tasks.workunit.client.0.smithi102.stdout:9/285: truncate df/f17 1294664 0 2024-03-26T18:54:36.034 INFO:tasks.workunit.client.0.smithi102.stdout:7/341: mkdir d7/d12/d4d/d62 0 2024-03-26T18:54:36.034 INFO:tasks.workunit.client.0.smithi102.stdout:7/342: truncate d7/d10/d27/d34/f48 5156613 0 2024-03-26T18:54:36.036 INFO:tasks.workunit.client.0.smithi102.stdout:3/540: symlink df/d29/d31/d77/d8a/lc6 0 2024-03-26T18:54:36.036 INFO:tasks.workunit.client.0.smithi102.stdout:3/541: stat df/d1c/d34/c89 0 2024-03-26T18:54:36.036 INFO:tasks.workunit.client.0.smithi102.stdout:5/573: sync 2024-03-26T18:54:36.037 INFO:tasks.workunit.client.0.smithi102.stdout:6/317: rmdir d2/df/d35/d43 39 2024-03-26T18:54:36.037 INFO:tasks.workunit.client.0.smithi102.stdout:6/318: dread - d2/d7/d3c/f3e zero size 2024-03-26T18:54:36.038 INFO:tasks.workunit.client.0.smithi102.stdout:4/271: getdents d1/d17/d1a/d21 0 2024-03-26T18:54:36.038 INFO:tasks.workunit.client.0.smithi102.stdout:4/272: chown d1/d25/d39/d3e/d4c 67368258 1 2024-03-26T18:54:36.038 INFO:tasks.workunit.client.0.smithi102.stdout:4/273: creat d1/d27/f55 x:0 0 0 2024-03-26T18:54:36.039 INFO:tasks.workunit.client.0.smithi102.stdout:5/574: write d2/d14/f70 [570440,101034] 0 2024-03-26T18:54:36.039 INFO:tasks.workunit.client.0.smithi102.stdout:5/575: write d2/d25/d2a/f9f [449547,77572] 0 2024-03-26T18:54:36.040 INFO:tasks.workunit.client.0.smithi102.stdout:6/319: mkdir d2/d3d/d53/d63 0 2024-03-26T18:54:36.042 INFO:tasks.workunit.client.0.smithi102.stdout:8/397: dwrite d0/d3/d26/d19/d27/f5b [0,4194304] 0 2024-03-26T18:54:36.042 INFO:tasks.workunit.client.0.smithi102.stdout:8/398: dread - d0/f4d zero size 2024-03-26T18:54:36.042 INFO:tasks.workunit.client.0.smithi102.stdout:1/339: dwrite de/d16/d1d/d2a/f75 [0,4194304] 0 2024-03-26T18:54:36.042 INFO:tasks.workunit.client.0.smithi102.stdout:1/340: stat de/c12 0 2024-03-26T18:54:36.042 INFO:tasks.workunit.client.0.smithi102.stdout:1/341: chown de/d16/f51 3680646 1 2024-03-26T18:54:36.044 INFO:tasks.workunit.client.1.smithi190.stdout:6/605: mknod d17/d28/d80/da8/cc9 0 2024-03-26T18:54:36.044 INFO:tasks.workunit.client.1.smithi190.stdout:6/606: write d17/d28/d2c/f47 [849267,11828] 0 2024-03-26T18:54:36.048 INFO:tasks.workunit.client.0.smithi102.stdout:5/576: rename d2/d25/d2a/d4f/f9c to d2/fcd 0 2024-03-26T18:54:36.048 INFO:tasks.workunit.client.0.smithi102.stdout:5/577: write d2/d14/d1a/f41 [559453,85907] 0 2024-03-26T18:54:36.054 INFO:tasks.workunit.client.0.smithi102.stdout:1/342: write de/d16/f5e [441627,34445] 0 2024-03-26T18:54:36.054 INFO:tasks.workunit.client.0.smithi102.stdout:1/343: read de/d2e/f49 [4022055,32163] 0 2024-03-26T18:54:36.054 INFO:tasks.workunit.client.1.smithi190.stdout:0/711: rename d0/dd5/dee/ff1 to d0/dd5/dee/f101 0 2024-03-26T18:54:36.056 INFO:tasks.workunit.client.1.smithi190.stdout:6/607: chown d17/l26 0 1 2024-03-26T18:54:36.057 INFO:tasks.workunit.client.1.smithi190.stdout:6/608: truncate d17/f25 4890304 0 2024-03-26T18:54:36.065 INFO:tasks.workunit.client.1.smithi190.stdout:2/661: getdents d6/d9 0 2024-03-26T18:54:36.065 INFO:tasks.workunit.client.1.smithi190.stdout:2/662: creat d6/d9/d3a/d3d/d2e/d31/fec x:0 0 0 2024-03-26T18:54:36.066 INFO:tasks.workunit.client.0.smithi102.stdout:6/320: dread d2/f3a [0,4194304] 0 2024-03-26T18:54:36.066 INFO:tasks.workunit.client.0.smithi102.stdout:5/578: unlink d2/f6a 0 2024-03-26T18:54:36.067 INFO:tasks.workunit.client.0.smithi102.stdout:6/321: creat d2/df/f64 x:0 0 0 2024-03-26T18:54:36.067 INFO:tasks.workunit.client.0.smithi102.stdout:6/322: fdatasync d2/d3d/d53/d5c/f5d 0 2024-03-26T18:54:36.067 INFO:tasks.workunit.client.0.smithi102.stdout:6/323: write d2/df/f2b [67425,19597] 0 2024-03-26T18:54:36.068 INFO:tasks.workunit.client.1.smithi190.stdout:1/595: rmdir d3/da/dd/d2f/d5b 39 2024-03-26T18:54:36.068 INFO:tasks.workunit.client.1.smithi190.stdout:1/596: write d3/da/dd/fb6 [475266,83462] 0 2024-03-26T18:54:36.068 INFO:tasks.workunit.client.1.smithi190.stdout:1/597: stat d3/da/dd/d95/d52 0 2024-03-26T18:54:36.068 INFO:tasks.workunit.client.1.smithi190.stdout:1/598: creat d3/d9/db/fc5 x:0 0 0 2024-03-26T18:54:36.068 INFO:tasks.workunit.client.1.smithi190.stdout:1/599: chown d3/da/ca8 2999 1 2024-03-26T18:54:36.068 INFO:tasks.workunit.client.0.smithi102.stdout:6/324: creat d2/d3d/d53/d5c/f65 x:0 0 0 2024-03-26T18:54:36.068 INFO:tasks.workunit.client.0.smithi102.stdout:5/579: symlink d2/d2c/lce 0 2024-03-26T18:54:36.069 INFO:tasks.workunit.client.1.smithi190.stdout:0/712: dread d0/dd/d20/d1c/d46/da7/df0/d4b/fd3 [0,4194304] 0 2024-03-26T18:54:36.073 INFO:tasks.workunit.client.0.smithi102.stdout:6/325: unlink d2/d12/f45 0 2024-03-26T18:54:36.073 INFO:tasks.workunit.client.0.smithi102.stdout:6/326: creat d2/d3d/d53/f66 x:0 0 0 2024-03-26T18:54:36.075 INFO:tasks.workunit.client.1.smithi190.stdout:2/663: read d6/d9/d3a/d3d/d2e/f51 [2636549,105852] 0 2024-03-26T18:54:36.083 INFO:tasks.workunit.client.1.smithi190.stdout:1/600: write d3/d9/f2a [7520139,61890] 0 2024-03-26T18:54:36.083 INFO:tasks.workunit.client.0.smithi102.stdout:5/580: dread d2/f11 [0,4194304] 0 2024-03-26T18:54:36.084 INFO:tasks.workunit.client.0.smithi102.stdout:5/581: dread - d2/d25/d75/db9/d43/d9e/f8c zero size 2024-03-26T18:54:36.085 INFO:tasks.workunit.client.0.smithi102.stdout:6/327: dread d2/f21 [0,4194304] 0 2024-03-26T18:54:36.086 INFO:tasks.workunit.client.1.smithi190.stdout:0/713: dread d0/f1f [0,4194304] 0 2024-03-26T18:54:36.086 INFO:tasks.workunit.client.1.smithi190.stdout:1/601: truncate d3/f53 3091599 0 2024-03-26T18:54:36.087 INFO:tasks.workunit.client.0.smithi102.stdout:6/328: link d2/d7/f51 d2/d7/f67 0 2024-03-26T18:54:36.087 INFO:tasks.workunit.client.0.smithi102.stdout:6/329: readlink d2/d12/l62 0 2024-03-26T18:54:36.088 INFO:tasks.workunit.client.1.smithi190.stdout:0/714: mknod d0/dd/d20/d1c/d8b/d8d/d8f/c102 0 2024-03-26T18:54:36.089 INFO:tasks.workunit.client.1.smithi190.stdout:1/602: creat d3/da/d3a/d99/fc6 x:0 0 0 2024-03-26T18:54:36.090 INFO:tasks.workunit.client.1.smithi190.stdout:1/603: readlink d3/d9/l17 0 2024-03-26T18:54:36.091 INFO:tasks.workunit.client.1.smithi190.stdout:2/664: read d6/d9/f38 [1572536,94698] 0 2024-03-26T18:54:36.091 INFO:tasks.workunit.client.1.smithi190.stdout:0/715: creat d0/dd/d20/d1c/d46/d58/de4/f103 x:0 0 0 2024-03-26T18:54:36.092 INFO:tasks.workunit.client.1.smithi190.stdout:2/665: dread d6/d21/d61/d6c/fc4 [0,4194304] 0 2024-03-26T18:54:36.093 INFO:tasks.workunit.client.1.smithi190.stdout:1/604: creat d3/da/dd/d2f/d5b/db5/fc7 x:0 0 0 2024-03-26T18:54:36.094 INFO:tasks.workunit.client.0.smithi102.stdout:0/384: dwrite d6/fd [0,4194304] 0 2024-03-26T18:54:36.094 INFO:tasks.workunit.client.0.smithi102.stdout:0/385: stat d6/de/d13/d1f/l76 0 2024-03-26T18:54:36.095 INFO:tasks.workunit.client.0.smithi102.stdout:9/286: dwrite df/d1a/f5e [0,4194304] 0 2024-03-26T18:54:36.096 INFO:tasks.workunit.client.1.smithi190.stdout:0/716: mkdir d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104 0 2024-03-26T18:54:36.097 INFO:tasks.workunit.client.0.smithi102.stdout:1/344: dwrite de/f19 [4194304,4194304] 0 2024-03-26T18:54:36.097 INFO:tasks.workunit.client.0.smithi102.stdout:1/345: write f8 [8525544,125314] 0 2024-03-26T18:54:36.097 INFO:tasks.workunit.client.0.smithi102.stdout:3/542: dwrite fd [0,4194304] 0 2024-03-26T18:54:36.097 INFO:tasks.workunit.client.1.smithi190.stdout:6/609: dwrite d17/d28/d2c/d50/f8f [0,4194304] 0 2024-03-26T18:54:36.098 INFO:tasks.workunit.client.0.smithi102.stdout:8/399: dwrite d0/d3c/f43 [0,4194304] 0 2024-03-26T18:54:36.098 INFO:tasks.workunit.client.0.smithi102.stdout:8/400: write d0/d3/d2c/f34 [2883472,102866] 0 2024-03-26T18:54:36.098 INFO:tasks.workunit.client.0.smithi102.stdout:8/401: creat d0/d3/d2c/d2f/f7e x:0 0 0 2024-03-26T18:54:36.098 INFO:tasks.workunit.client.0.smithi102.stdout:9/287: creat df/d1a/d23/f71 x:0 0 0 2024-03-26T18:54:36.100 INFO:tasks.workunit.client.1.smithi190.stdout:0/717: mkdir d0/d105 0 2024-03-26T18:54:36.102 INFO:tasks.workunit.client.0.smithi102.stdout:4/274: getdents d1/d27 0 2024-03-26T18:54:36.103 INFO:tasks.workunit.client.1.smithi190.stdout:2/666: dread d6/f7a [0,4194304] 0 2024-03-26T18:54:36.106 INFO:tasks.workunit.client.1.smithi190.stdout:0/718: mkdir d0/dd5/d106 0 2024-03-26T18:54:36.111 INFO:tasks.workunit.client.0.smithi102.stdout:0/386: read d6/de/d13/d1f/f3b [4087325,82230] 0 2024-03-26T18:54:36.112 INFO:tasks.workunit.client.1.smithi190.stdout:0/719: truncate d0/dd/d20/d1c/d46/da7/df0/f7d 1384783 0 2024-03-26T18:54:36.121 INFO:tasks.workunit.client.0.smithi102.stdout:4/275: dread d1/d17/d1a/f4b [0,4194304] 0 2024-03-26T18:54:36.124 INFO:tasks.workunit.client.0.smithi102.stdout:6/330: dwrite d2/d12/f2d [0,4194304] 0 2024-03-26T18:54:36.124 INFO:tasks.workunit.client.0.smithi102.stdout:6/331: dread - d2/d7/d3c/f48 zero size 2024-03-26T18:54:36.124 INFO:tasks.workunit.client.1.smithi190.stdout:2/667: rmdir d6/d9/d48 39 2024-03-26T18:54:36.127 INFO:tasks.workunit.client.1.smithi190.stdout:0/720: getdents d0/dd/d20/d1c/d46/da7/df0 0 2024-03-26T18:54:36.129 INFO:tasks.workunit.client.1.smithi190.stdout:2/668: creat d6/d21/d61/d6c/db4/fed x:0 0 0 2024-03-26T18:54:36.129 INFO:tasks.workunit.client.1.smithi190.stdout:2/669: truncate d6/d9/fe2 98357 0 2024-03-26T18:54:36.135 INFO:tasks.workunit.client.1.smithi190.stdout:0/721: write d0/dd/d20/d1c/d46/da7/df0/da2/d80/fb2 [2165561,58477] 0 2024-03-26T18:54:36.135 INFO:tasks.workunit.client.1.smithi190.stdout:1/605: dwrite d3/d73/faa [0,4194304] 0 2024-03-26T18:54:36.136 INFO:tasks.workunit.client.1.smithi190.stdout:1/606: chown d3/d9/d9f/ca2 983479 1 2024-03-26T18:54:36.136 INFO:tasks.workunit.client.1.smithi190.stdout:6/610: dwrite d17/d28/d80/f88 [0,4194304] 0 2024-03-26T18:54:36.136 INFO:tasks.workunit.client.0.smithi102.stdout:3/543: dwrite df/d29/d31/d36/d45/f60 [8388608,4194304] 0 2024-03-26T18:54:36.144 INFO:tasks.workunit.client.1.smithi190.stdout:2/670: creat d6/d21/dd0/fee x:0 0 0 2024-03-26T18:54:36.146 INFO:tasks.workunit.client.1.smithi190.stdout:0/722: rmdir d0/dd/d20/d1c/d46/da7/df0 39 2024-03-26T18:54:36.147 INFO:tasks.workunit.client.0.smithi102.stdout:9/288: mkdir df/d1a/d1d/d72 0 2024-03-26T18:54:36.148 INFO:tasks.workunit.client.0.smithi102.stdout:4/276: dwrite d1/f8 [0,4194304] 0 2024-03-26T18:54:36.150 INFO:tasks.workunit.client.0.smithi102.stdout:1/346: rmdir de/d16/d1d/d2a/d2c/d43 39 2024-03-26T18:54:36.154 INFO:tasks.workunit.client.0.smithi102.stdout:0/387: unlink d6/de/d13/d4a/l4d 0 2024-03-26T18:54:36.154 INFO:tasks.workunit.client.0.smithi102.stdout:0/388: truncate d6/de/f15 4608224 0 2024-03-26T18:54:36.156 INFO:tasks.workunit.client.1.smithi190.stdout:2/671: dread d6/d9/d3a/d3d/d1c/d8d/dc7/f43 [0,4194304] 0 2024-03-26T18:54:36.157 INFO:tasks.workunit.client.0.smithi102.stdout:6/332: rename d2/d7/l1e to d2/df/d35/d43/l68 0 2024-03-26T18:54:36.158 INFO:tasks.workunit.client.1.smithi190.stdout:6/611: creat d17/d28/d2b/fca x:0 0 0 2024-03-26T18:54:36.162 INFO:tasks.workunit.client.0.smithi102.stdout:3/544: creat df/d29/fc7 x:0 0 0 2024-03-26T18:54:36.162 INFO:tasks.workunit.client.1.smithi190.stdout:6/612: write d17/d28/d2c/db8/f54 [643524,128184] 0 2024-03-26T18:54:36.164 INFO:tasks.workunit.client.1.smithi190.stdout:0/723: symlink d0/dd5/dee/df5/l107 0 2024-03-26T18:54:36.166 INFO:tasks.workunit.client.0.smithi102.stdout:8/402: getdents d0/d15/d5d 0 2024-03-26T18:54:36.168 INFO:tasks.workunit.client.0.smithi102.stdout:9/289: dread df/d1a/d23/f48 [4194304,4194304] 0 2024-03-26T18:54:36.168 INFO:tasks.workunit.client.0.smithi102.stdout:9/290: stat cc 0 2024-03-26T18:54:36.168 INFO:tasks.workunit.client.0.smithi102.stdout:9/291: fsync df/d1a/d23/d29/d45/f56 0 2024-03-26T18:54:36.170 INFO:tasks.workunit.client.0.smithi102.stdout:4/277: creat d1/d17/d1a/d21/f56 x:0 0 0 2024-03-26T18:54:36.170 INFO:tasks.workunit.client.0.smithi102.stdout:4/278: creat d1/d17/d1a/f57 x:0 0 0 2024-03-26T18:54:36.171 INFO:tasks.workunit.client.0.smithi102.stdout:8/403: dread d0/d3/d31/f4f [4194304,4194304] 0 2024-03-26T18:54:36.171 INFO:tasks.workunit.client.0.smithi102.stdout:8/404: stat d0/d3/d2c/d4e/f7b 0 2024-03-26T18:54:36.171 INFO:tasks.workunit.client.0.smithi102.stdout:8/405: read d0/d3/d26/d19/d27/f45 [435158,106052] 0 2024-03-26T18:54:36.171 INFO:tasks.workunit.client.0.smithi102.stdout:1/347: write de/d16/d1d/f4d [2002076,23363] 0 2024-03-26T18:54:36.173 INFO:tasks.workunit.client.1.smithi190.stdout:2/672: getdents d6/d21/d61 0 2024-03-26T18:54:36.176 INFO:tasks.workunit.client.1.smithi190.stdout:6/613: creat d17/d28/d74/d97/fcb x:0 0 0 2024-03-26T18:54:36.179 INFO:tasks.workunit.client.1.smithi190.stdout:2/673: symlink d6/lef 0 2024-03-26T18:54:36.182 INFO:tasks.workunit.client.1.smithi190.stdout:2/674: rename d6/d9/d3a/d3d/d1c/d8d/dc7/l2a to d6/d9/d3a/d3d/d1c/d8d/dc7/lf0 0 2024-03-26T18:54:36.183 INFO:tasks.workunit.client.0.smithi102.stdout:8/406: dread d0/d3/f6 [0,4194304] 0 2024-03-26T18:54:36.183 INFO:tasks.workunit.client.0.smithi102.stdout:8/407: read d0/d3c/d66/f6c [133627,38601] 0 2024-03-26T18:54:36.184 INFO:tasks.workunit.client.0.smithi102.stdout:0/389: dwrite d6/de/d13/d25/d31/d45/f48 [0,4194304] 0 2024-03-26T18:54:36.184 INFO:tasks.workunit.client.0.smithi102.stdout:0/390: readlink d6/de/d13/d25/d4f/l6f 0 2024-03-26T18:54:36.185 INFO:tasks.workunit.client.0.smithi102.stdout:6/333: dwrite d2/d3/f38 [4194304,4194304] 0 2024-03-26T18:54:36.185 INFO:tasks.workunit.client.0.smithi102.stdout:6/334: read - d2/f34 zero size 2024-03-26T18:54:36.185 INFO:tasks.workunit.client.0.smithi102.stdout:6/335: creat d2/d3d/d53/f69 x:0 0 0 2024-03-26T18:54:36.186 INFO:tasks.workunit.client.1.smithi190.stdout:2/675: symlink d6/d9/d3a/d3d/d1c/d8d/dc7/d72/db1/lf1 0 2024-03-26T18:54:36.186 INFO:tasks.workunit.client.1.smithi190.stdout:2/676: chown d6/d9/d3a/d3d/d1c/d8d/dc7/f64 3735 1 2024-03-26T18:54:36.190 INFO:tasks.workunit.client.1.smithi190.stdout:2/677: creat d6/d9/d45/d6b/d93/dc0/ff2 x:0 0 0 2024-03-26T18:54:36.190 INFO:tasks.workunit.client.1.smithi190.stdout:6/614: write d17/d28/d80/f81 [988408,123822] 0 2024-03-26T18:54:36.208 INFO:tasks.workunit.client.1.smithi190.stdout:1/607: dwrite d3/da/d3a/f55 [0,4194304] 0 2024-03-26T18:54:36.217 INFO:tasks.workunit.client.1.smithi190.stdout:6/615: mkdir d17/d28/d2c/db8/dcc 0 2024-03-26T18:54:36.225 INFO:tasks.workunit.client.1.smithi190.stdout:6/616: write f9 [2261775,83352] 0 2024-03-26T18:54:36.225 INFO:tasks.workunit.client.1.smithi190.stdout:6/617: chown d17/d28/d2b/fca 0 1 2024-03-26T18:54:36.226 INFO:tasks.workunit.client.1.smithi190.stdout:6/618: write d17/d28/d2c/f47 [94792,85157] 0 2024-03-26T18:54:36.239 INFO:tasks.workunit.client.0.smithi102.stdout:3/545: mknod df/d29/d31/d36/d45/daf/cc8 0 2024-03-26T18:54:36.239 INFO:tasks.workunit.client.0.smithi102.stdout:3/546: write df/d29/d3a/f6d [276797,82601] 0 2024-03-26T18:54:36.239 INFO:tasks.workunit.client.0.smithi102.stdout:3/547: chown df/d1c/d34/c38 35857406 1 2024-03-26T18:54:36.240 INFO:tasks.workunit.client.0.smithi102.stdout:9/292: truncate df/f30 295703 0 2024-03-26T18:54:36.241 INFO:tasks.workunit.client.1.smithi190.stdout:7/720: rename d3/d6/d9/d1f/d41 to d3/d1b/dd3/d10d 0 2024-03-26T18:54:36.242 INFO:tasks.workunit.client.1.smithi190.stdout:4/658: mknod d7/d8/d1c/ce1 0 2024-03-26T18:54:36.244 INFO:tasks.workunit.client.1.smithi190.stdout:7/721: symlink d3/d6/d9/d14/d59/d73/d8b/l10e 0 2024-03-26T18:54:36.245 INFO:tasks.workunit.client.1.smithi190.stdout:7/722: creat d3/f10f x:0 0 0 2024-03-26T18:54:36.245 INFO:tasks.workunit.client.1.smithi190.stdout:7/723: dread - d3/f10f zero size 2024-03-26T18:54:36.246 INFO:tasks.workunit.client.0.smithi102.stdout:4/279: truncate d1/f31 2938993 0 2024-03-26T18:54:36.246 INFO:tasks.workunit.client.0.smithi102.stdout:3/548: write df/d29/d31/d36/f50 [4093702,118578] 0 2024-03-26T18:54:36.249 INFO:tasks.workunit.client.1.smithi190.stdout:5/567: sync 2024-03-26T18:54:36.250 INFO:tasks.workunit.client.1.smithi190.stdout:5/568: chown d2/d17/d98/d3a/d3c/l50 320138 1 2024-03-26T18:54:36.250 INFO:tasks.workunit.client.0.smithi102.stdout:1/348: mkdir de/d16/d1d/d2a/d2c/d56/d78 0 2024-03-26T18:54:36.252 INFO:tasks.workunit.client.1.smithi190.stdout:7/724: write d3/f43 [567591,16687] 0 2024-03-26T18:54:36.253 INFO:tasks.workunit.client.0.smithi102.stdout:8/408: creat d0/d3/d2c/f7f x:0 0 0 2024-03-26T18:54:36.254 INFO:tasks.workunit.client.0.smithi102.stdout:3/549: dread df/d1c/f2a [0,4194304] 0 2024-03-26T18:54:36.256 INFO:tasks.workunit.client.1.smithi190.stdout:7/725: dread d3/d6/d9/fcd [0,4194304] 0 2024-03-26T18:54:36.257 INFO:tasks.workunit.client.1.smithi190.stdout:7/726: creat d3/d6/d9/d1f/f110 x:0 0 0 2024-03-26T18:54:36.267 INFO:tasks.workunit.client.1.smithi190.stdout:3/546: rename d1/d3/df/d18/d35/d41/l4c to d1/d3/df/d18/d35/d41/d51/d9c/la4 0 2024-03-26T18:54:36.267 INFO:tasks.workunit.client.1.smithi190.stdout:7/727: write d3/d1b/d57/ff5 [1518287,113971] 0 2024-03-26T18:54:36.269 INFO:tasks.workunit.client.1.smithi190.stdout:5/569: creat d2/d17/d62/d64/fbf x:0 0 0 2024-03-26T18:54:36.270 INFO:tasks.workunit.client.1.smithi190.stdout:5/570: fsync d2/d17/f7b 0 2024-03-26T18:54:36.270 INFO:tasks.workunit.client.1.smithi190.stdout:8/709: rename d9/da2/d73/c83 to d9/d39/d9e/ceb 0 2024-03-26T18:54:36.270 INFO:tasks.workunit.client.0.smithi102.stdout:6/336: mkdir d2/d3d/d53/d63/d6a 0 2024-03-26T18:54:36.273 INFO:tasks.workunit.client.1.smithi190.stdout:9/895: rename d12/d2d/d1d/f3a to d12/d2d/d1d/d23/d83/dc9/dd4/f13d 0 2024-03-26T18:54:36.273 INFO:tasks.workunit.client.1.smithi190.stdout:9/896: getdents d12/d2d/d1d/d23/d8e 0 2024-03-26T18:54:36.274 INFO:tasks.workunit.client.0.smithi102.stdout:0/391: mkdir d6/de/d13/d1f/d55/d57/d7d 0 2024-03-26T18:54:36.275 INFO:tasks.workunit.client.0.smithi102.stdout:0/392: write d6/fb [4110218,98889] 0 2024-03-26T18:54:36.275 INFO:tasks.workunit.client.0.smithi102.stdout:0/393: truncate d6/de/d13/d4a/f68 735899 0 2024-03-26T18:54:36.275 INFO:tasks.workunit.client.0.smithi102.stdout:3/550: dwrite df/d61/f52 [0,4194304] 0 2024-03-26T18:54:36.275 INFO:tasks.workunit.client.0.smithi102.stdout:3/551: stat df/fb1 0 2024-03-26T18:54:36.275 INFO:tasks.workunit.client.0.smithi102.stdout:3/552: fdatasync df/f1b 0 2024-03-26T18:54:36.281 INFO:tasks.workunit.client.1.smithi190.stdout:7/728: dread d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/f8a [0,4194304] 0 2024-03-26T18:54:36.285 INFO:tasks.workunit.client.0.smithi102.stdout:9/293: rename df/d1a/d23/f2c to df/d1a/d65/f73 0 2024-03-26T18:54:36.285 INFO:tasks.workunit.client.0.smithi102.stdout:9/294: readlink df/d1a/d1d/l1f 0 2024-03-26T18:54:36.285 INFO:tasks.workunit.client.0.smithi102.stdout:9/295: fsync df/d1a/f52 0 2024-03-26T18:54:36.289 INFO:tasks.workunit.client.0.smithi102.stdout:4/280: creat d1/d25/d39/d3e/f58 x:0 0 0 2024-03-26T18:54:36.292 INFO:tasks.workunit.client.0.smithi102.stdout:5/582: sync 2024-03-26T18:54:36.292 INFO:tasks.workunit.client.0.smithi102.stdout:7/343: sync 2024-03-26T18:54:36.292 INFO:tasks.workunit.client.0.smithi102.stdout:5/583: stat d2/d25/d2a/l98 0 2024-03-26T18:54:36.292 INFO:tasks.workunit.client.0.smithi102.stdout:5/584: chown d2/f7 121260 1 2024-03-26T18:54:36.292 INFO:tasks.workunit.client.0.smithi102.stdout:5/585: dread - d2/d25/d5b/d76/f9a zero size 2024-03-26T18:54:36.292 INFO:tasks.workunit.client.0.smithi102.stdout:2/324: sync 2024-03-26T18:54:36.292 INFO:tasks.workunit.client.0.smithi102.stdout:5/586: chown d2/d25/d2a/db4 74 1 2024-03-26T18:54:36.293 INFO:tasks.workunit.client.1.smithi190.stdout:8/710: dwrite d9/d39/d9e/fc5 [0,4194304] 0 2024-03-26T18:54:36.294 INFO:tasks.workunit.client.0.smithi102.stdout:1/349: unlink de/d2e/f49 0 2024-03-26T18:54:36.297 INFO:tasks.workunit.client.1.smithi190.stdout:8/711: link d9/dd/d23/d36/d37/d3a/lab d9/d39/d9e/lec 0 2024-03-26T18:54:36.298 INFO:tasks.workunit.client.1.smithi190.stdout:8/712: mknod d9/dd/d23/db1/ced 0 2024-03-26T18:54:36.298 INFO:tasks.workunit.client.0.smithi102.stdout:0/394: dwrite d6/d3e/d54/f58 [0,4194304] 0 2024-03-26T18:54:36.300 INFO:tasks.workunit.client.1.smithi190.stdout:3/547: symlink d1/de/la5 0 2024-03-26T18:54:36.300 INFO:tasks.workunit.client.1.smithi190.stdout:8/713: link d9/dd/d23/d36/d37/d3a/d61/fb6 d9/da2/dc4/fee 0 2024-03-26T18:54:36.303 INFO:tasks.workunit.client.1.smithi190.stdout:5/571: truncate d2/d17/d62/fad 538343 0 2024-03-26T18:54:36.303 INFO:tasks.workunit.client.1.smithi190.stdout:8/714: mknod d9/dd/d23/d36/d37/d3a/d43/dc2/cef 0 2024-03-26T18:54:36.303 INFO:tasks.workunit.client.1.smithi190.stdout:8/715: stat d9/d48/d49/le0 0 2024-03-26T18:54:36.305 INFO:tasks.workunit.client.0.smithi102.stdout:3/553: symlink df/d1c/d34/lc9 0 2024-03-26T18:54:36.305 INFO:tasks.workunit.client.1.smithi190.stdout:0/724: rename d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/f5f to d0/dd/d20/d1c/d46/d58/de4/f108 0 2024-03-26T18:54:36.307 INFO:tasks.workunit.client.0.smithi102.stdout:4/281: link d1/d27/c46 d1/d17/d1a/c59 0 2024-03-26T18:54:36.310 INFO:tasks.workunit.client.1.smithi190.stdout:0/725: unlink d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/lbd 0 2024-03-26T18:54:36.310 INFO:tasks.workunit.client.0.smithi102.stdout:7/344: creat d7/d31/d56/f63 x:0 0 0 2024-03-26T18:54:36.310 INFO:tasks.workunit.client.0.smithi102.stdout:7/345: truncate d7/d10/d27/f55 797493 0 2024-03-26T18:54:36.312 INFO:tasks.workunit.client.1.smithi190.stdout:1/608: rename d3/da/dd/d2f/d5b/d83/l8b to d3/da/dd/d95/d52/lc8 0 2024-03-26T18:54:36.313 INFO:tasks.workunit.client.0.smithi102.stdout:5/587: mkdir d2/d25/d2a/dbb/dcf 0 2024-03-26T18:54:36.313 INFO:tasks.workunit.client.0.smithi102.stdout:1/350: truncate de/d16/d1d/d2a/f65 684187 0 2024-03-26T18:54:36.314 INFO:tasks.workunit.client.1.smithi190.stdout:8/716: dread d9/d48/f50 [0,4194304] 0 2024-03-26T18:54:36.314 INFO:tasks.workunit.client.1.smithi190.stdout:8/717: chown d9/da2/c19 2145 1 2024-03-26T18:54:36.315 INFO:tasks.workunit.client.0.smithi102.stdout:8/409: mkdir d0/d3/d26/d80 0 2024-03-26T18:54:36.317 INFO:tasks.workunit.client.1.smithi190.stdout:4/659: rename d7/fd6 to d7/d8/d14/d4b/d7d/da1/fe2 0 2024-03-26T18:54:36.317 INFO:tasks.workunit.client.1.smithi190.stdout:1/609: symlink d3/lc9 0 2024-03-26T18:54:36.317 INFO:tasks.workunit.client.1.smithi190.stdout:1/610: dread - d3/da/d3a/d99/fc6 zero size 2024-03-26T18:54:36.317 INFO:tasks.workunit.client.1.smithi190.stdout:1/611: truncate d3/d9/db/fc1 416396 0 2024-03-26T18:54:36.318 INFO:tasks.workunit.client.1.smithi190.stdout:8/718: creat d9/dd/d23/d36/d37/d3a/d61/daf/db3/db5/ff0 x:0 0 0 2024-03-26T18:54:36.319 INFO:tasks.workunit.client.1.smithi190.stdout:5/572: read d2/f3 [5913032,26759] 0 2024-03-26T18:54:36.319 INFO:tasks.workunit.client.1.smithi190.stdout:7/729: dwrite d3/d6/d9/d14/d59/d73/d8b/f104 [0,4194304] 0 2024-03-26T18:54:36.319 INFO:tasks.workunit.client.0.smithi102.stdout:0/395: rename d6/de/d13/f3f to d6/de/d13/f7e 0 2024-03-26T18:54:36.319 INFO:tasks.workunit.client.0.smithi102.stdout:0/396: write d6/de/d13/d4a/f68 [1595149,62117] 0 2024-03-26T18:54:36.320 INFO:tasks.workunit.client.1.smithi190.stdout:9/897: rename d12/d2d/d1d/d23/l65 to d12/d2d/dfc/l13e 0 2024-03-26T18:54:36.321 INFO:tasks.workunit.client.0.smithi102.stdout:9/296: dwrite df/d1a/f24 [0,4194304] 0 2024-03-26T18:54:36.321 INFO:tasks.workunit.client.0.smithi102.stdout:2/325: dwrite d3/d11/d27/f6b [0,4194304] 0 2024-03-26T18:54:36.324 INFO:tasks.workunit.client.0.smithi102.stdout:4/282: dread d1/d17/d1a/f53 [4194304,4194304] 0 2024-03-26T18:54:36.325 INFO:tasks.workunit.client.0.smithi102.stdout:3/554: rename df/d61/l4b to df/d29/d31/d36/d3c/d6f/lca 0 2024-03-26T18:54:36.326 INFO:tasks.workunit.client.1.smithi190.stdout:7/730: creat d3/d1b/dd3/f111 x:0 0 0 2024-03-26T18:54:36.327 INFO:tasks.workunit.client.0.smithi102.stdout:8/410: dread d0/d3/d2c/d33/f49 [0,4194304] 0 2024-03-26T18:54:36.328 INFO:tasks.workunit.client.1.smithi190.stdout:8/719: rename d9/da2/d73/fb0 to d9/d48/d49/ff1 0 2024-03-26T18:54:36.328 INFO:tasks.workunit.client.0.smithi102.stdout:5/588: mknod d2/d25/d2a/d4f/cd0 0 2024-03-26T18:54:36.335 INFO:tasks.workunit.client.0.smithi102.stdout:0/397: mkdir d6/de/d13/d1f/d55/d57/d7d/d7f 0 2024-03-26T18:54:36.335 INFO:tasks.workunit.client.0.smithi102.stdout:0/398: chown d6/de/d13/d1f/f3b 51118765 1 2024-03-26T18:54:36.349 INFO:tasks.workunit.client.1.smithi190.stdout:1/612: rmdir d3/da 39 2024-03-26T18:54:36.351 INFO:tasks.workunit.client.1.smithi190.stdout:5/573: dread d2/f15 [0,4194304] 0 2024-03-26T18:54:36.352 INFO:tasks.workunit.client.0.smithi102.stdout:3/555: truncate df/d61/d55/f7c 4149095 0 2024-03-26T18:54:36.352 INFO:tasks.workunit.client.0.smithi102.stdout:3/556: chown df/d1c/d34/d9b 287597 1 2024-03-26T18:54:36.357 INFO:tasks.workunit.client.0.smithi102.stdout:7/346: dwrite d7/d31/d56/f63 [0,4194304] 0 2024-03-26T18:54:36.358 INFO:tasks.workunit.client.0.smithi102.stdout:5/589: rename d2/d25/d5b/d72/f22 to d2/d25/d2a/db4/fd1 0 2024-03-26T18:54:36.358 INFO:tasks.workunit.client.0.smithi102.stdout:5/590: fdatasync d2/d25/d2a/f2e 0 2024-03-26T18:54:36.360 INFO:tasks.workunit.client.0.smithi102.stdout:4/283: getdents d1/d25/d39 0 2024-03-26T18:54:36.360 INFO:tasks.workunit.client.1.smithi190.stdout:0/726: dwrite d0/dd/d20/d1c/d46/d58/de4/f103 [0,4194304] 0 2024-03-26T18:54:36.361 INFO:tasks.workunit.client.1.smithi190.stdout:5/574: creat d2/d17/d1b/db6/d24/d3b/da7/fc0 x:0 0 0 2024-03-26T18:54:36.361 INFO:tasks.workunit.client.1.smithi190.stdout:3/548: dwrite d1/f5 [0,4194304] 0 2024-03-26T18:54:36.361 INFO:tasks.workunit.client.1.smithi190.stdout:3/549: truncate d1/d3/f4b 4350922 0 2024-03-26T18:54:36.365 INFO:tasks.workunit.client.0.smithi102.stdout:7/347: link d7/cc d7/d12/c64 0 2024-03-26T18:54:36.365 INFO:tasks.workunit.client.0.smithi102.stdout:7/348: chown d7/d31/d56/f63 238686482 1 2024-03-26T18:54:36.368 INFO:tasks.workunit.client.0.smithi102.stdout:5/591: creat d2/d25/d75/db9/d43/d9e/daf/fd2 x:0 0 0 2024-03-26T18:54:36.369 INFO:tasks.workunit.client.1.smithi190.stdout:5/575: link d2/d17/d62/d64/fbf d2/d17/d1b/db6/fc1 0 2024-03-26T18:54:36.370 INFO:tasks.workunit.client.0.smithi102.stdout:4/284: symlink d1/d25/d39/l5a 0 2024-03-26T18:54:36.371 INFO:tasks.workunit.client.1.smithi190.stdout:9/898: dwrite d12/d2d/fd9 [4194304,4194304] 0 2024-03-26T18:54:36.373 INFO:tasks.workunit.client.1.smithi190.stdout:3/550: creat d1/d3/df/d18/d35/d41/d51/d9c/fa6 x:0 0 0 2024-03-26T18:54:36.375 INFO:tasks.workunit.client.1.smithi190.stdout:4/660: dwrite d7/d8/d14/d23/fdc [0,4194304] 0 2024-03-26T18:54:36.375 INFO:tasks.workunit.client.1.smithi190.stdout:9/899: creat d12/d80/db2/d112/f13f x:0 0 0 2024-03-26T18:54:36.375 INFO:tasks.workunit.client.1.smithi190.stdout:9/900: read - d12/d2d/d1d/da1/dba/f120 zero size 2024-03-26T18:54:36.375 INFO:tasks.workunit.client.1.smithi190.stdout:9/901: chown d12 1 1 2024-03-26T18:54:36.375 INFO:tasks.workunit.client.1.smithi190.stdout:4/661: write d7/f9 [5193398,71802] 0 2024-03-26T18:54:36.376 INFO:tasks.workunit.client.1.smithi190.stdout:5/576: unlink d2/fb0 0 2024-03-26T18:54:36.376 INFO:tasks.workunit.client.1.smithi190.stdout:9/902: mknod d12/d2d/d1d/d23/c140 0 2024-03-26T18:54:36.378 INFO:tasks.workunit.client.1.smithi190.stdout:3/551: rename d1/d3/df/f74 to d1/de/d8a/fa7 0 2024-03-26T18:54:36.380 INFO:tasks.workunit.client.0.smithi102.stdout:7/349: creat d7/d12/d4d/d62/f65 x:0 0 0 2024-03-26T18:54:36.380 INFO:tasks.workunit.client.0.smithi102.stdout:5/592: read d2/d25/f26 [1054978,12460] 0 2024-03-26T18:54:36.381 INFO:tasks.workunit.client.1.smithi190.stdout:5/577: rmdir d2/d17/d62 39 2024-03-26T18:54:36.381 INFO:tasks.workunit.client.1.smithi190.stdout:9/903: dread d12/d2d/d1d/d54/f88 [0,4194304] 0 2024-03-26T18:54:36.383 INFO:tasks.workunit.client.1.smithi190.stdout:9/904: creat d12/d2d/d1d/d23/d83/dc9/dd4/d71/f141 x:0 0 0 2024-03-26T18:54:36.383 INFO:tasks.workunit.client.1.smithi190.stdout:9/905: write d12/f28 [4752163,51472] 0 2024-03-26T18:54:36.383 INFO:tasks.workunit.client.1.smithi190.stdout:9/906: chown d12/d2d/d1d/da1/dba/dd2/lfa 134686 1 2024-03-26T18:54:36.383 INFO:tasks.workunit.client.1.smithi190.stdout:3/552: creat d1/d3/d5a/fa8 x:0 0 0 2024-03-26T18:54:36.384 INFO:tasks.workunit.client.0.smithi102.stdout:5/593: symlink d2/d2c/d6f/ld3 0 2024-03-26T18:54:36.385 INFO:tasks.workunit.client.1.smithi190.stdout:8/720: rmdir d9/dd/d23/d36/d37/d3a/d43/dc2 39 2024-03-26T18:54:36.385 INFO:tasks.workunit.client.1.smithi190.stdout:8/721: truncate d9/d39/f6d 515120 0 2024-03-26T18:54:36.385 INFO:tasks.workunit.client.1.smithi190.stdout:5/578: symlink d2/d17/d1b/db6/d88/dbc/lc2 0 2024-03-26T18:54:36.385 INFO:tasks.workunit.client.0.smithi102.stdout:4/285: rmdir d1/d17/d1f 39 2024-03-26T18:54:36.388 INFO:tasks.workunit.client.1.smithi190.stdout:8/722: mkdir d9/dd/d23/d36/d37/d3a/d43/dc2/df2 0 2024-03-26T18:54:36.388 INFO:tasks.workunit.client.0.smithi102.stdout:4/286: symlink d1/d27/l5b 0 2024-03-26T18:54:36.390 INFO:tasks.workunit.client.1.smithi190.stdout:8/723: truncate d9/dd/d15/f7a 766867 0 2024-03-26T18:54:36.390 INFO:tasks.workunit.client.1.smithi190.stdout:3/553: getdents d1 0 2024-03-26T18:54:36.390 INFO:tasks.workunit.client.1.smithi190.stdout:3/554: chown d1/de/d81 33 1 2024-03-26T18:54:36.391 INFO:tasks.workunit.client.0.smithi102.stdout:4/287: rename d1/d17/l1b to d1/d17/d2a/l5c 0 2024-03-26T18:54:36.392 INFO:tasks.workunit.client.0.smithi102.stdout:4/288: mkdir d1/d17/d1f/d5d 0 2024-03-26T18:54:36.393 INFO:tasks.workunit.client.1.smithi190.stdout:4/662: dread d7/d8/d14/d23/fdc [0,4194304] 0 2024-03-26T18:54:36.393 INFO:tasks.workunit.client.1.smithi190.stdout:4/663: chown d7/d8/d14/d5c/f83 410016502 1 2024-03-26T18:54:36.394 INFO:tasks.workunit.client.1.smithi190.stdout:3/555: dread d1/de/d1c/d36/f40 [0,4194304] 0 2024-03-26T18:54:36.394 INFO:tasks.workunit.client.1.smithi190.stdout:3/556: creat d1/d3/d5a/fa9 x:0 0 0 2024-03-26T18:54:36.396 INFO:tasks.workunit.client.0.smithi102.stdout:9/297: dwrite df/d1a/d23/f5d [0,4194304] 0 2024-03-26T18:54:36.396 INFO:tasks.workunit.client.0.smithi102.stdout:0/399: dwrite d6/de/d13/d25/d31/d45/f48 [0,4194304] 0 2024-03-26T18:54:36.396 INFO:tasks.workunit.client.0.smithi102.stdout:0/400: fdatasync d6/de/d13/f7e 0 2024-03-26T18:54:36.397 INFO:tasks.workunit.client.1.smithi190.stdout:7/731: dwrite d3/d6/f8c [0,4194304] 0 2024-03-26T18:54:36.397 INFO:tasks.workunit.client.1.smithi190.stdout:7/732: truncate d3/d6/d9/d14/d36/f10a 228863 0 2024-03-26T18:54:36.397 INFO:tasks.workunit.client.1.smithi190.stdout:7/733: write d3/f16 [1930430,21978] 0 2024-03-26T18:54:36.397 INFO:tasks.workunit.client.1.smithi190.stdout:7/734: write d3/d6/d9/d14/d59/d73/d8b/f101 [571666,19349] 0 2024-03-26T18:54:36.398 INFO:tasks.workunit.client.1.smithi190.stdout:1/613: dwrite d3/da/dd/f8c [4194304,4194304] 0 2024-03-26T18:54:36.398 INFO:tasks.workunit.client.1.smithi190.stdout:1/614: truncate d3/d9/f2a 8424615 0 2024-03-26T18:54:36.400 INFO:tasks.workunit.client.0.smithi102.stdout:5/594: dread d2/d25/d75/db9/d43/d9e/d82/fa2 [0,4194304] 0 2024-03-26T18:54:36.400 INFO:tasks.workunit.client.0.smithi102.stdout:5/595: truncate d2/d25/d2a/f9f 1007903 0 2024-03-26T18:54:36.400 INFO:tasks.workunit.client.0.smithi102.stdout:5/596: fdatasync d2/d14/d1a/f1b 0 2024-03-26T18:54:36.400 INFO:tasks.workunit.client.0.smithi102.stdout:5/597: write d2/d25/db0/fbd [1046506,51839] 0 2024-03-26T18:54:36.401 INFO:tasks.workunit.client.1.smithi190.stdout:0/727: dwrite d0/dd/d20/d1c/d46/da7/df0/f7d [0,4194304] 0 2024-03-26T18:54:36.401 INFO:tasks.workunit.client.0.smithi102.stdout:7/350: dread d7/d12/d4d/f5e [0,4194304] 0 2024-03-26T18:54:36.401 INFO:tasks.workunit.client.0.smithi102.stdout:3/557: dwrite df/d1c/d34/f37 [0,4194304] 0 2024-03-26T18:54:36.401 INFO:tasks.workunit.client.0.smithi102.stdout:5/598: write d2/d25/d2a/d4f/f96 [220107,16393] 0 2024-03-26T18:54:36.403 INFO:tasks.workunit.client.1.smithi190.stdout:0/728: dread d0/dd/d20/d1c/d4e/f7e [0,4194304] 0 2024-03-26T18:54:36.403 INFO:tasks.workunit.client.1.smithi190.stdout:0/729: write d0/f1f [2344410,46538] 0 2024-03-26T18:54:36.408 INFO:tasks.workunit.client.0.smithi102.stdout:2/326: write d3/d11/d27/f6b [4929024,83881] 0 2024-03-26T18:54:36.410 INFO:tasks.workunit.client.0.smithi102.stdout:0/401: mkdir d6/d2d/d3d/d80 0 2024-03-26T18:54:36.410 INFO:tasks.workunit.client.0.smithi102.stdout:0/402: chown d6/de/d13/d1f/d55/d57/d7d 14908 1 2024-03-26T18:54:36.410 INFO:tasks.workunit.client.0.smithi102.stdout:0/403: creat d6/de/d13/d25/d4f/d66/f81 x:0 0 0 2024-03-26T18:54:36.410 INFO:tasks.workunit.client.0.smithi102.stdout:0/404: dread - d6/de/d13/d25/d4f/d66/f6e zero size 2024-03-26T18:54:36.414 INFO:tasks.workunit.client.0.smithi102.stdout:7/351: truncate d7/d10/d17/f59 4821246 0 2024-03-26T18:54:36.414 INFO:tasks.workunit.client.0.smithi102.stdout:7/352: creat d7/d12/d4d/d62/f66 x:0 0 0 2024-03-26T18:54:36.416 INFO:tasks.workunit.client.1.smithi190.stdout:8/724: rename d9/d39/f6f to d9/da2/dc4/ff3 0 2024-03-26T18:54:36.416 INFO:tasks.workunit.client.1.smithi190.stdout:8/725: fdatasync d9/da2/f63 0 2024-03-26T18:54:36.417 INFO:tasks.workunit.client.1.smithi190.stdout:4/664: rename d7/d1d/d42/cde to d7/d8/d1c/ce3 0 2024-03-26T18:54:36.418 INFO:tasks.workunit.client.1.smithi190.stdout:8/726: symlink d9/da2/lf4 0 2024-03-26T18:54:36.420 INFO:tasks.workunit.client.0.smithi102.stdout:5/599: readlink d2/d25/d75/db9/l45 0 2024-03-26T18:54:36.421 INFO:tasks.workunit.client.1.smithi190.stdout:7/735: rename d3/d6/d9/d1f/dab to d3/d1b/dd3/d10d/d65/d90/d112 0 2024-03-26T18:54:36.422 INFO:tasks.workunit.client.1.smithi190.stdout:4/665: mkdir d7/dc8/dd5/de4 0 2024-03-26T18:54:36.422 INFO:tasks.workunit.client.1.smithi190.stdout:4/666: write d7/d8/d14/f52 [911213,94452] 0 2024-03-26T18:54:36.423 INFO:tasks.workunit.client.1.smithi190.stdout:4/667: chown c6 561318 1 2024-03-26T18:54:36.424 INFO:tasks.workunit.client.1.smithi190.stdout:4/668: creat d7/d8/d14/d5c/fe5 x:0 0 0 2024-03-26T18:54:36.424 INFO:tasks.workunit.client.1.smithi190.stdout:4/669: stat d7/d1d/l3e 0 2024-03-26T18:54:36.424 INFO:tasks.workunit.client.1.smithi190.stdout:4/670: chown d7/d8/d1c/d20/c6d 436996005 1 2024-03-26T18:54:36.425 INFO:tasks.workunit.client.1.smithi190.stdout:9/907: dwrite d12/d2d/d1d/d23/d83/dc9/dd4/d71/f75 [4194304,4194304] 0 2024-03-26T18:54:36.425 INFO:tasks.workunit.client.1.smithi190.stdout:7/736: dread d3/d6/d9/d14/f18 [0,4194304] 0 2024-03-26T18:54:36.425 INFO:tasks.workunit.client.1.smithi190.stdout:7/737: fsync d3/d1b/dd3/d10d/d50/f60 0 2024-03-26T18:54:36.425 INFO:tasks.workunit.client.0.smithi102.stdout:2/327: dread d3/d11/d15/d46/f47 [8388608,4194304] 0 2024-03-26T18:54:36.425 INFO:tasks.workunit.client.0.smithi102.stdout:2/328: truncate d3/d11/d15/f2e 475888 0 2024-03-26T18:54:36.426 INFO:tasks.workunit.client.1.smithi190.stdout:9/908: getdents d12/d2d/d1d/d23/d83/dc9/dd4/d3f/d55 0 2024-03-26T18:54:36.427 INFO:tasks.workunit.client.1.smithi190.stdout:0/730: mkdir d0/dd/d20/d1c/d46/d58/d109 0 2024-03-26T18:54:36.428 INFO:tasks.workunit.client.1.smithi190.stdout:7/738: symlink d3/d6/d9/d1f/d93/d105/l113 0 2024-03-26T18:54:36.429 INFO:tasks.workunit.client.1.smithi190.stdout:5/579: dwrite d2/d17/f44 [0,4194304] 0 2024-03-26T18:54:36.430 INFO:tasks.workunit.client.1.smithi190.stdout:9/909: creat d12/f142 x:0 0 0 2024-03-26T18:54:36.430 INFO:tasks.workunit.client.0.smithi102.stdout:3/558: dwrite df/d29/d31/d36/f7f [0,4194304] 0 2024-03-26T18:54:36.430 INFO:tasks.workunit.client.0.smithi102.stdout:7/353: dread d7/d10/d17/f37 [0,4194304] 0 2024-03-26T18:54:36.431 INFO:tasks.workunit.client.1.smithi190.stdout:7/739: mknod d3/d6/d9/d14/d59/dbf/c114 0 2024-03-26T18:54:36.433 INFO:tasks.workunit.client.0.smithi102.stdout:0/405: rename d6/de/d13/d25/d7b to d6/de/d13/d25/d31/d45/d82 0 2024-03-26T18:54:36.435 INFO:tasks.workunit.client.1.smithi190.stdout:0/731: symlink d0/dd5/dee/l10a 0 2024-03-26T18:54:36.435 INFO:tasks.workunit.client.1.smithi190.stdout:0/732: truncate d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/f9b 4236464 0 2024-03-26T18:54:36.439 INFO:tasks.workunit.client.0.smithi102.stdout:9/298: dwrite df/d1a/d23/d4e/f53 [0,4194304] 0 2024-03-26T18:54:36.439 INFO:tasks.workunit.client.0.smithi102.stdout:9/299: chown df/d1a/l3c 81 1 2024-03-26T18:54:36.439 INFO:tasks.workunit.client.0.smithi102.stdout:9/300: readlink df/d1a/d1d/l55 0 2024-03-26T18:54:36.439 INFO:tasks.workunit.client.1.smithi190.stdout:3/557: rmdir d1/d3/df/d18/d35/d41/d51/d9c 39 2024-03-26T18:54:36.439 INFO:tasks.workunit.client.1.smithi190.stdout:5/580: dread d2/d17/d1b/db6/d24/f61 [0,4194304] 0 2024-03-26T18:54:36.440 INFO:tasks.workunit.client.0.smithi102.stdout:2/329: link d3/f13 d3/d2d/d69/f73 0 2024-03-26T18:54:36.440 INFO:tasks.workunit.client.0.smithi102.stdout:2/330: chown d3/f13 0 1 2024-03-26T18:54:36.442 INFO:tasks.workunit.client.1.smithi190.stdout:4/671: write d7/d8/d14/d5c/f97 [3215303,128166] 0 2024-03-26T18:54:36.442 INFO:tasks.workunit.client.1.smithi190.stdout:1/615: dwrite d3/da/dd/d95/f8e [0,4194304] 0 2024-03-26T18:54:36.444 INFO:tasks.workunit.client.1.smithi190.stdout:9/910: write d12/d2d/d1d/d54/f8a [2378690,8971] 0 2024-03-26T18:54:36.445 INFO:tasks.workunit.client.1.smithi190.stdout:9/911: write d12/d2d/d1d/d23/d83/f7c [7473,68984] 0 2024-03-26T18:54:36.445 INFO:tasks.workunit.client.1.smithi190.stdout:0/733: mknod d0/dd/d20/d61/c10b 0 2024-03-26T18:54:36.447 INFO:tasks.workunit.client.1.smithi190.stdout:4/672: mknod d7/d8/d14/d6e/ce6 0 2024-03-26T18:54:36.448 INFO:tasks.workunit.client.0.smithi102.stdout:3/559: stat df/c68 0 2024-03-26T18:54:36.450 INFO:tasks.workunit.client.0.smithi102.stdout:7/354: mkdir d7/d10/d67 0 2024-03-26T18:54:36.451 INFO:tasks.workunit.client.1.smithi190.stdout:9/912: link d12/l8c d12/d2d/d1d/d23/d2b/da0/d138/l143 0 2024-03-26T18:54:36.455 INFO:tasks.workunit.client.1.smithi190.stdout:0/734: rename d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/fea to d0/dd/d20/d1c/d46/da7/df0/da2/d80/f10c 0 2024-03-26T18:54:36.456 INFO:tasks.workunit.client.0.smithi102.stdout:5/600: dwrite d2/d14/d1a/f41 [0,4194304] 0 2024-03-26T18:54:36.457 INFO:tasks.workunit.client.0.smithi102.stdout:5/601: truncate d2/d25/d2a/d4f/fc2 797018 0 2024-03-26T18:54:36.460 INFO:tasks.workunit.client.1.smithi190.stdout:1/616: dread d3/da/dd/f72 [0,4194304] 0 2024-03-26T18:54:36.464 INFO:tasks.workunit.client.0.smithi102.stdout:9/301: rename df/d1a/f2d to df/d1a/d23/d60/f74 0 2024-03-26T18:54:36.464 INFO:tasks.workunit.client.0.smithi102.stdout:9/302: chown df/d1a/d57/f6d 11 1 2024-03-26T18:54:36.464 INFO:tasks.workunit.client.0.smithi102.stdout:9/303: chown df/d46/l49 96255 1 2024-03-26T18:54:36.465 INFO:tasks.workunit.client.1.smithi190.stdout:1/617: dread d3/da/dd/d95/d52/fba [0,4194304] 0 2024-03-26T18:54:36.465 INFO:tasks.workunit.client.1.smithi190.stdout:8/727: dwrite d9/da2/dc4/f35 [0,4194304] 0 2024-03-26T18:54:36.467 INFO:tasks.workunit.client.1.smithi190.stdout:0/735: dread d0/dd/d20/d1c/d46/d7c/faa [0,4194304] 0 2024-03-26T18:54:36.478 INFO:tasks.workunit.client.1.smithi190.stdout:7/740: dwrite d3/d6/f91 [0,4194304] 0 2024-03-26T18:54:36.484 INFO:tasks.workunit.client.1.smithi190.stdout:8/728: mknod d9/dd/d23/cf5 0 2024-03-26T18:54:36.486 INFO:tasks.workunit.client.0.smithi102.stdout:7/355: dwrite d7/d10/d17/f35 [4194304,4194304] 0 2024-03-26T18:54:36.487 INFO:tasks.workunit.client.0.smithi102.stdout:7/356: stat d7/d31/f45 0 2024-03-26T18:54:36.487 INFO:tasks.workunit.client.0.smithi102.stdout:0/406: dwrite d6/de/f28 [4194304,4194304] 0 2024-03-26T18:54:36.487 INFO:tasks.workunit.client.0.smithi102.stdout:0/407: dread - d6/de/d13/d25/d31/d45/f65 zero size 2024-03-26T18:54:36.487 INFO:tasks.workunit.client.0.smithi102.stdout:0/408: write d6/f11 [8396129,34049] 0 2024-03-26T18:54:36.487 INFO:tasks.workunit.client.0.smithi102.stdout:0/409: dread - d6/de/d13/d1f/f59 zero size 2024-03-26T18:54:36.487 INFO:tasks.workunit.client.1.smithi190.stdout:1/618: symlink d3/da/dd/d95/d52/lca 0 2024-03-26T18:54:36.487 INFO:tasks.workunit.client.1.smithi190.stdout:4/673: dwrite d7/d1d/fb5 [0,4194304] 0 2024-03-26T18:54:36.488 INFO:tasks.workunit.client.1.smithi190.stdout:3/558: dwrite d1/d3/df/d18/d35/d41/d51/f96 [0,4194304] 0 2024-03-26T18:54:36.490 INFO:tasks.workunit.client.0.smithi102.stdout:3/560: creat df/d29/d31/d36/d3c/da7/fcb x:0 0 0 2024-03-26T18:54:36.490 INFO:tasks.workunit.client.0.smithi102.stdout:3/561: truncate df/d29/fc7 920192 0 2024-03-26T18:54:36.490 INFO:tasks.workunit.client.0.smithi102.stdout:3/562: creat df/d8b/fcc x:0 0 0 2024-03-26T18:54:36.490 INFO:tasks.workunit.client.0.smithi102.stdout:3/563: chown df/d29/l2b 266 1 2024-03-26T18:54:36.492 INFO:tasks.workunit.client.1.smithi190.stdout:0/736: symlink d0/dd/d20/d1c/d8b/d8d/db7/de6/l10d 0 2024-03-26T18:54:36.497 INFO:tasks.workunit.client.0.smithi102.stdout:5/602: mknod d2/d25/d75/db9/d43/d9e/d82/dc9/cd4 0 2024-03-26T18:54:36.501 INFO:tasks.workunit.client.1.smithi190.stdout:0/737: dread d0/dd/d20/d1c/d46/da7/df0/da2/fa5 [0,4194304] 0 2024-03-26T18:54:36.501 INFO:tasks.workunit.client.1.smithi190.stdout:0/738: write d0/dd/d20/d1c/d46/d58/de4/f108 [4893525,68478] 0 2024-03-26T18:54:36.502 INFO:tasks.workunit.client.1.smithi190.stdout:9/913: creat d12/d80/f144 x:0 0 0 2024-03-26T18:54:36.502 INFO:tasks.workunit.client.0.smithi102.stdout:9/304: rename df/d1a/d65/d43 to df/d1a/d23/d3e/d75 0 2024-03-26T18:54:36.502 INFO:tasks.workunit.client.0.smithi102.stdout:9/305: write df/f17 [1941598,48039] 0 2024-03-26T18:54:36.507 INFO:tasks.workunit.client.1.smithi190.stdout:7/741: rename d3/d6/d9/d1f/f3d to d3/d1b/dd2/f115 0 2024-03-26T18:54:36.509 INFO:tasks.workunit.client.1.smithi190.stdout:4/674: creat d7/d8/d14/d23/d28/d9e/fe7 x:0 0 0 2024-03-26T18:54:36.510 INFO:tasks.workunit.client.1.smithi190.stdout:7/742: mknod d3/d6/d9/d1f/d4c/d71/dcc/c116 0 2024-03-26T18:54:36.511 INFO:tasks.workunit.client.0.smithi102.stdout:7/357: dwrite d7/d10/d17/f1b [0,4194304] 0 2024-03-26T18:54:36.517 INFO:tasks.workunit.client.1.smithi190.stdout:1/619: dwrite d3/d9/db/f23 [0,4194304] 0 2024-03-26T18:54:36.518 INFO:tasks.workunit.client.1.smithi190.stdout:3/559: rename d1/d3/d52/f90 to d1/de/d9f/faa 0 2024-03-26T18:54:36.524 INFO:tasks.workunit.client.1.smithi190.stdout:4/675: rename d7/d1d/fc6 to d7/d8/d14/d23/d28/d38/dd3/fe8 0 2024-03-26T18:54:36.524 INFO:tasks.workunit.client.0.smithi102.stdout:0/410: mkdir d6/d2d/d3d/d80/d83 0 2024-03-26T18:54:36.526 INFO:tasks.workunit.client.1.smithi190.stdout:4/676: unlink d7/fb9 0 2024-03-26T18:54:36.527 INFO:tasks.workunit.client.0.smithi102.stdout:7/358: dread d7/d12/d2b/f47 [0,4194304] 0 2024-03-26T18:54:36.528 INFO:tasks.workunit.client.1.smithi190.stdout:4/677: creat d7/d8/d14/d4b/d7d/da1/fe9 x:0 0 0 2024-03-26T18:54:36.528 INFO:tasks.workunit.client.1.smithi190.stdout:4/678: stat d7/d8/d14/d5c/fe5 0 2024-03-26T18:54:36.528 INFO:tasks.workunit.client.1.smithi190.stdout:4/679: dread - d7/d1d/d5e/fce zero size 2024-03-26T18:54:36.528 INFO:tasks.workunit.client.1.smithi190.stdout:8/729: dwrite d9/d48/d49/fc0 [0,4194304] 0 2024-03-26T18:54:36.528 INFO:tasks.workunit.client.0.smithi102.stdout:9/306: creat df/d1a/d1d/f76 x:0 0 0 2024-03-26T18:54:36.529 INFO:tasks.workunit.client.1.smithi190.stdout:4/680: creat d7/d8/d14/d23/d28/d38/fea x:0 0 0 2024-03-26T18:54:36.531 INFO:tasks.workunit.client.0.smithi102.stdout:0/411: mkdir d6/de/d13/d25/d31/d45/d82/d84 0 2024-03-26T18:54:36.533 INFO:tasks.workunit.client.0.smithi102.stdout:7/359: link d7/fb d7/d10/f68 0 2024-03-26T18:54:36.533 INFO:tasks.workunit.client.0.smithi102.stdout:7/360: chown d7/d12/c46 20322544 1 2024-03-26T18:54:36.533 INFO:tasks.workunit.client.1.smithi190.stdout:8/730: mkdir d9/da2/df6 0 2024-03-26T18:54:36.534 INFO:tasks.workunit.client.0.smithi102.stdout:9/307: creat df/d1a/d23/d4e/f77 x:0 0 0 2024-03-26T18:54:36.534 INFO:tasks.workunit.client.0.smithi102.stdout:9/308: dread - df/d1a/d23/d4e/f77 zero size 2024-03-26T18:54:36.534 INFO:tasks.workunit.client.1.smithi190.stdout:3/560: dread d1/de/f29 [0,4194304] 0 2024-03-26T18:54:36.535 INFO:tasks.workunit.client.1.smithi190.stdout:3/561: dread - d1/d3/df/d18/d1b/f9b zero size 2024-03-26T18:54:36.535 INFO:tasks.workunit.client.1.smithi190.stdout:4/681: dread d7/d1d/d5e/f76 [0,4194304] 0 2024-03-26T18:54:36.535 INFO:tasks.workunit.client.1.smithi190.stdout:4/682: truncate d7/d8/d14/d23/d28/d38/fcf 1390140 0 2024-03-26T18:54:36.535 INFO:tasks.workunit.client.1.smithi190.stdout:4/683: readlink d7/d1d/lc1 0 2024-03-26T18:54:36.535 INFO:tasks.workunit.client.0.smithi102.stdout:3/564: rename df/d29/d31/d36/d3c/da7 to df/d1c/d70/dcd 0 2024-03-26T18:54:36.535 INFO:tasks.workunit.client.0.smithi102.stdout:7/361: write d7/d12/f22 [634577,22152] 0 2024-03-26T18:54:36.536 INFO:tasks.workunit.client.0.smithi102.stdout:7/362: fdatasync d7/d12/f43 0 2024-03-26T18:54:36.536 INFO:tasks.workunit.client.0.smithi102.stdout:7/363: chown d7/d10/f68 7 1 2024-03-26T18:54:36.536 INFO:tasks.workunit.client.0.smithi102.stdout:7/364: dread - d7/d31/f45 zero size 2024-03-26T18:54:36.537 INFO:tasks.workunit.client.1.smithi190.stdout:4/684: rename d7/d8/d14/d23/d28/d38/f62 to d7/d8/d14/d23/feb 0 2024-03-26T18:54:36.537 INFO:tasks.workunit.client.1.smithi190.stdout:4/685: chown d7/d1d/f26 58 1 2024-03-26T18:54:36.538 INFO:tasks.workunit.client.1.smithi190.stdout:3/562: mknod d1/d3/d52/cab 0 2024-03-26T18:54:36.539 INFO:tasks.workunit.client.0.smithi102.stdout:3/565: rmdir df/d1c/d34 39 2024-03-26T18:54:36.540 INFO:tasks.workunit.client.1.smithi190.stdout:3/563: read d1/de/f1d [3087499,57573] 0 2024-03-26T18:54:36.543 INFO:tasks.workunit.client.1.smithi190.stdout:6/619: sync 2024-03-26T18:54:36.543 INFO:tasks.workunit.client.1.smithi190.stdout:2/678: sync 2024-03-26T18:54:36.543 INFO:tasks.workunit.client.1.smithi190.stdout:6/620: write d17/d28/d74/d97/fbc [420167,53554] 0 2024-03-26T18:54:36.544 INFO:tasks.workunit.client.1.smithi190.stdout:0/739: dwrite d0/dd/d20/d1c/d46/da7/df0/f2f [0,4194304] 0 2024-03-26T18:54:36.544 INFO:tasks.workunit.client.0.smithi102.stdout:5/603: rename d2/f3c to d2/d25/d75/db9/fd5 0 2024-03-26T18:54:36.546 INFO:tasks.workunit.client.1.smithi190.stdout:6/621: mknod d17/d28/d2c/ccd 0 2024-03-26T18:54:36.546 INFO:tasks.workunit.client.1.smithi190.stdout:6/622: write d17/d28/d2c/d50/f7e [1266829,47598] 0 2024-03-26T18:54:36.549 INFO:tasks.workunit.client.1.smithi190.stdout:8/731: dread d9/dd/d23/d36/d37/d3a/d61/fd6 [0,4194304] 0 2024-03-26T18:54:36.550 INFO:tasks.workunit.client.0.smithi102.stdout:5/604: link d2/d2c/d6f/ld3 d2/d2c/d6f/ld6 0 2024-03-26T18:54:36.551 INFO:tasks.workunit.client.1.smithi190.stdout:4/686: write d7/d8/faa [405603,42421] 0 2024-03-26T18:54:36.551 INFO:tasks.workunit.client.1.smithi190.stdout:0/740: mkdir d0/dd5/d10e 0 2024-03-26T18:54:36.551 INFO:tasks.workunit.client.1.smithi190.stdout:9/914: dwrite d12/d2d/d1d/d54/ff8 [0,4194304] 0 2024-03-26T18:54:36.551 INFO:tasks.workunit.client.1.smithi190.stdout:0/741: write d0/dd/d20/d1c/d8b/d8d/fd2 [392359,69458] 0 2024-03-26T18:54:36.552 INFO:tasks.workunit.client.1.smithi190.stdout:1/620: dwrite f2 [0,4194304] 0 2024-03-26T18:54:36.552 INFO:tasks.workunit.client.0.smithi102.stdout:2/331: dwrite d3/d6/f1b [0,4194304] 0 2024-03-26T18:54:36.552 INFO:tasks.workunit.client.0.smithi102.stdout:2/332: chown d3/d20/f4c 4827454 1 2024-03-26T18:54:36.553 INFO:tasks.workunit.client.1.smithi190.stdout:6/623: link d17/d28/d46/d6a/fa4 d17/d28/d2c/d50/fce 0 2024-03-26T18:54:36.553 INFO:tasks.workunit.client.1.smithi190.stdout:6/624: fdatasync d17/d28/d2b/f60 0 2024-03-26T18:54:36.553 INFO:tasks.workunit.client.1.smithi190.stdout:8/732: symlink d9/dd/d23/d36/d37/d3a/d61/daf/db3/db5/de3/lf7 0 2024-03-26T18:54:36.553 INFO:tasks.workunit.client.1.smithi190.stdout:8/733: chown d9/dd/d23/d36/d37/d3a/d43/l64 470 1 2024-03-26T18:54:36.556 INFO:tasks.workunit.client.0.smithi102.stdout:9/309: rename df/f42 to df/d46/d63/f78 0 2024-03-26T18:54:36.557 INFO:tasks.workunit.client.1.smithi190.stdout:3/564: write d1/de/d4e/f54 [1637903,87475] 0 2024-03-26T18:54:36.557 INFO:tasks.workunit.client.1.smithi190.stdout:4/687: link d7/d8/d1c/l25 d7/d8/d14/d6e/lec 0 2024-03-26T18:54:36.557 INFO:tasks.workunit.client.1.smithi190.stdout:4/688: readlink d7/d8/d14/d4b/l51 0 2024-03-26T18:54:36.559 INFO:tasks.workunit.client.0.smithi102.stdout:5/605: truncate d2/d25/f2b 1727940 0 2024-03-26T18:54:36.559 INFO:tasks.workunit.client.1.smithi190.stdout:9/915: creat d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/db3/f145 x:0 0 0 2024-03-26T18:54:36.562 INFO:tasks.workunit.client.1.smithi190.stdout:8/734: rmdir d9/d39/d79/dad 39 2024-03-26T18:54:36.563 INFO:tasks.workunit.client.1.smithi190.stdout:1/621: link d3/da/dd/d2f/f7b d3/da/d2e/d59/dae/fcb 0 2024-03-26T18:54:36.564 INFO:tasks.workunit.client.1.smithi190.stdout:4/689: symlink d7/d8/d14/d23/d28/d9e/led 0 2024-03-26T18:54:36.565 INFO:tasks.workunit.client.1.smithi190.stdout:6/625: creat d17/d28/d38/d79/fcf x:0 0 0 2024-03-26T18:54:36.566 INFO:tasks.workunit.client.1.smithi190.stdout:9/916: unlink d12/d2d/d1d/d23/d83/dc9/dd4/l137 0 2024-03-26T18:54:36.568 INFO:tasks.workunit.client.1.smithi190.stdout:8/735: creat d9/d39/d79/dad/ff8 x:0 0 0 2024-03-26T18:54:36.569 INFO:tasks.workunit.client.1.smithi190.stdout:1/622: mkdir d3/d73/d88/dcc 0 2024-03-26T18:54:36.570 INFO:tasks.workunit.client.1.smithi190.stdout:1/623: write d3/da/f1f [5213700,65965] 0 2024-03-26T18:54:36.572 INFO:tasks.workunit.client.0.smithi102.stdout:9/310: symlink df/d1a/d23/d60/l79 0 2024-03-26T18:54:36.573 INFO:tasks.workunit.client.1.smithi190.stdout:9/917: truncate d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/d10e/f110 933117 0 2024-03-26T18:54:36.573 INFO:tasks.workunit.client.1.smithi190.stdout:3/565: dread d1/d3/df/f19 [0,4194304] 0 2024-03-26T18:54:36.573 INFO:tasks.workunit.client.1.smithi190.stdout:8/736: getdents d9/dd/d23/d36/d37/d3a/d61/daf/db3 0 2024-03-26T18:54:36.573 INFO:tasks.workunit.client.0.smithi102.stdout:9/311: getdents df/d1a/d65 0 2024-03-26T18:54:36.573 INFO:tasks.workunit.client.0.smithi102.stdout:9/312: chown df/d1a/d1d/l37 5 1 2024-03-26T18:54:36.574 INFO:tasks.workunit.client.1.smithi190.stdout:0/742: write d0/dd/d20/f78 [1896286,125738] 0 2024-03-26T18:54:36.574 INFO:tasks.workunit.client.1.smithi190.stdout:0/743: write d0/dd/d20/d1c/d8b/d8d/db7/de6/fe9 [292907,99447] 0 2024-03-26T18:54:36.574 INFO:tasks.workunit.client.1.smithi190.stdout:6/626: dread d17/d28/d2c/db8/f54 [0,4194304] 0 2024-03-26T18:54:36.575 INFO:tasks.workunit.client.1.smithi190.stdout:3/566: creat d1/d3/df/d18/d35/fac x:0 0 0 2024-03-26T18:54:36.575 INFO:tasks.workunit.client.1.smithi190.stdout:9/918: rmdir d12/d2d/d1d/d54 39 2024-03-26T18:54:36.577 INFO:tasks.workunit.client.0.smithi102.stdout:9/313: truncate df/f40 2750923 0 2024-03-26T18:54:36.577 INFO:tasks.workunit.client.0.smithi102.stdout:9/314: fdatasync df/d1a/d1d/f3a 0 2024-03-26T18:54:36.577 INFO:tasks.workunit.client.0.smithi102.stdout:9/315: write df/d1a/f24 [4467946,46074] 0 2024-03-26T18:54:36.578 INFO:tasks.workunit.client.1.smithi190.stdout:0/744: mknod d0/dd/d20/d1c/d46/da7/df0/d87/d93/dc4/dd8/c10f 0 2024-03-26T18:54:36.579 INFO:tasks.workunit.client.1.smithi190.stdout:6/627: mknod d17/d28/d38/d79/cd0 0 2024-03-26T18:54:36.579 INFO:tasks.workunit.client.1.smithi190.stdout:6/628: stat d17/d28/d46/d6a 0 2024-03-26T18:54:36.579 INFO:tasks.workunit.client.1.smithi190.stdout:6/629: chown d17/d28/d2c/d77/fbb 108294906 1 2024-03-26T18:54:36.582 INFO:tasks.workunit.client.1.smithi190.stdout:4/690: write d7/d8/d14/d23/feb [2630441,72147] 0 2024-03-26T18:54:36.583 INFO:tasks.workunit.client.1.smithi190.stdout:3/567: unlink d1/d3/d5a/f63 0 2024-03-26T18:54:36.583 INFO:tasks.workunit.client.0.smithi102.stdout:9/316: unlink df/d1a/l2b 0 2024-03-26T18:54:36.583 INFO:tasks.workunit.client.1.smithi190.stdout:4/691: rmdir d7/d8/d14/d23/d28/d38/d96 39 2024-03-26T18:54:36.584 INFO:tasks.workunit.client.1.smithi190.stdout:4/692: rmdir d7/d8/d14/d23 39 2024-03-26T18:54:36.584 INFO:tasks.workunit.client.1.smithi190.stdout:6/630: rmdir d17/d28/d8b 39 2024-03-26T18:54:36.584 INFO:tasks.workunit.client.1.smithi190.stdout:6/631: creat d17/d28/d38/d79/fd1 x:0 0 0 2024-03-26T18:54:36.584 INFO:tasks.workunit.client.0.smithi102.stdout:9/317: dread df/f30 [0,4194304] 0 2024-03-26T18:54:36.586 INFO:tasks.workunit.client.1.smithi190.stdout:4/693: creat d7/d8/d14/d23/d28/fee x:0 0 0 2024-03-26T18:54:36.586 INFO:tasks.workunit.client.1.smithi190.stdout:3/568: mkdir d1/de/d81/dad 0 2024-03-26T18:54:36.586 INFO:tasks.workunit.client.1.smithi190.stdout:3/569: chown d1/de/d8a/c8e 21 1 2024-03-26T18:54:36.587 INFO:tasks.workunit.client.1.smithi190.stdout:4/694: rmdir d7/d8/d14/d23/d28/d38/dd3 39 2024-03-26T18:54:36.588 INFO:tasks.workunit.client.1.smithi190.stdout:6/632: creat d17/d28/d8b/d95/fd2 x:0 0 0 2024-03-26T18:54:36.589 INFO:tasks.workunit.client.1.smithi190.stdout:6/633: rmdir d17/d28 39 2024-03-26T18:54:36.592 INFO:tasks.workunit.client.1.smithi190.stdout:9/919: dread d12/d2d/d1d/d23/d83/dc9/dd4/d3f/f81 [0,4194304] 0 2024-03-26T18:54:36.592 INFO:tasks.workunit.client.1.smithi190.stdout:9/920: chown d12/d2d/d1d/da1 459 1 2024-03-26T18:54:36.592 INFO:tasks.workunit.client.1.smithi190.stdout:9/921: truncate d12/d2d/d1d/d23/d83/d91/fae 97511 0 2024-03-26T18:54:36.592 INFO:tasks.workunit.client.0.smithi102.stdout:7/365: dwrite d7/d10/f14 [0,4194304] 0 2024-03-26T18:54:36.593 INFO:tasks.workunit.client.0.smithi102.stdout:7/366: dread d7/d10/d27/f55 [0,4194304] 0 2024-03-26T18:54:36.593 INFO:tasks.workunit.client.0.smithi102.stdout:2/333: dwrite d3/f4 [0,4194304] 0 2024-03-26T18:54:36.594 INFO:tasks.workunit.client.0.smithi102.stdout:3/566: dwrite df/fb1 [0,4194304] 0 2024-03-26T18:54:36.594 INFO:tasks.workunit.client.0.smithi102.stdout:3/567: chown df/d29/d3a/l78 9553521 1 2024-03-26T18:54:36.594 INFO:tasks.workunit.client.1.smithi190.stdout:2/679: dwrite d6/d21/d61/fc2 [0,4194304] 0 2024-03-26T18:54:36.594 INFO:tasks.workunit.client.1.smithi190.stdout:2/680: chown d6/f9b 6373 1 2024-03-26T18:54:36.595 INFO:tasks.workunit.client.1.smithi190.stdout:9/922: rmdir d12/d2d/d1d/d54/d92/de3/dff 39 2024-03-26T18:54:36.597 INFO:tasks.workunit.client.0.smithi102.stdout:5/606: dwrite d2/d25/d5b/fc4 [0,4194304] 0 2024-03-26T18:54:36.600 INFO:tasks.workunit.client.0.smithi102.stdout:5/607: dread d2/d25/d5b/f94 [0,4194304] 0 2024-03-26T18:54:36.601 INFO:tasks.workunit.client.1.smithi190.stdout:9/923: creat d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/d10e/f146 x:0 0 0 2024-03-26T18:54:36.602 INFO:tasks.workunit.client.0.smithi102.stdout:7/367: mkdir d7/d10/d67/d69 0 2024-03-26T18:54:36.605 INFO:tasks.workunit.client.0.smithi102.stdout:8/411: sync 2024-03-26T18:54:36.606 INFO:tasks.workunit.client.0.smithi102.stdout:1/351: sync 2024-03-26T18:54:36.606 INFO:tasks.workunit.client.0.smithi102.stdout:6/337: sync 2024-03-26T18:54:36.606 INFO:tasks.workunit.client.0.smithi102.stdout:6/338: truncate d2/f33 381687 0 2024-03-26T18:54:36.609 INFO:tasks.workunit.client.1.smithi190.stdout:2/681: rmdir d6/d9/d3a/d3d/d55 39 2024-03-26T18:54:36.612 INFO:tasks.workunit.client.0.smithi102.stdout:3/568: mkdir df/d1c/d34/da5/dce 0 2024-03-26T18:54:36.612 INFO:tasks.workunit.client.0.smithi102.stdout:6/339: read d2/d7/f67 [599151,84535] 0 2024-03-26T18:54:36.618 INFO:tasks.workunit.client.0.smithi102.stdout:5/608: unlink d2/l3 0 2024-03-26T18:54:36.618 INFO:tasks.workunit.client.1.smithi190.stdout:8/737: dwrite d9/da2/dc4/f5c [0,4194304] 0 2024-03-26T18:54:36.619 INFO:tasks.workunit.client.0.smithi102.stdout:7/368: unlink d7/d12/d51/c57 0 2024-03-26T18:54:36.621 INFO:tasks.workunit.client.0.smithi102.stdout:4/289: sync 2024-03-26T18:54:36.622 INFO:tasks.workunit.client.1.smithi190.stdout:9/924: dread d12/d2d/d1d/d23/f33 [0,4194304] 0 2024-03-26T18:54:36.623 INFO:tasks.workunit.client.1.smithi190.stdout:2/682: dread d6/f36 [0,4194304] 0 2024-03-26T18:54:36.623 INFO:tasks.workunit.client.0.smithi102.stdout:1/352: creat de/d16/d1d/d2a/d2c/d56/d78/f79 x:0 0 0 2024-03-26T18:54:36.624 INFO:tasks.workunit.client.0.smithi102.stdout:5/609: symlink d2/ld7 0 2024-03-26T18:54:36.624 INFO:tasks.workunit.client.0.smithi102.stdout:5/610: readlink d2/d25/l34 0 2024-03-26T18:54:36.627 INFO:tasks.workunit.client.0.smithi102.stdout:7/369: mkdir d7/d12/d51/d6a 0 2024-03-26T18:54:36.627 INFO:tasks.workunit.client.0.smithi102.stdout:7/370: stat d7/d2a/f4a 0 2024-03-26T18:54:36.630 INFO:tasks.workunit.client.0.smithi102.stdout:6/340: write d2/d12/f2d [336397,47664] 0 2024-03-26T18:54:36.635 INFO:tasks.workunit.client.0.smithi102.stdout:1/353: creat de/d16/d1d/d2a/d2c/d56/d6e/f7a x:0 0 0 2024-03-26T18:54:36.636 INFO:tasks.workunit.client.0.smithi102.stdout:5/611: mkdir d2/d25/d75/db9/d43/d9e/d82/dc9/dd8 0 2024-03-26T18:54:36.637 INFO:tasks.workunit.client.1.smithi190.stdout:8/738: dread d9/d48/f50 [4194304,4194304] 0 2024-03-26T18:54:36.637 INFO:tasks.workunit.client.0.smithi102.stdout:1/354: dread de/f3f [0,4194304] 0 2024-03-26T18:54:36.637 INFO:tasks.workunit.client.1.smithi190.stdout:8/739: creat d9/dd/d23/db1/ff9 x:0 0 0 2024-03-26T18:54:36.639 INFO:tasks.workunit.client.0.smithi102.stdout:6/341: link d2/d7/c37 d2/d3d/d53/d5c/c6b 0 2024-03-26T18:54:36.641 INFO:tasks.workunit.client.0.smithi102.stdout:8/412: rename d0/d3/d26/d19/d27 to d0/d15/d81 0 2024-03-26T18:54:36.643 INFO:tasks.workunit.client.1.smithi190.stdout:9/925: dread d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/d10e/f7e [0,4194304] 0 2024-03-26T18:54:36.645 INFO:tasks.workunit.client.1.smithi190.stdout:9/926: creat d12/d2d/d1d/d23/de5/f147 x:0 0 0 2024-03-26T18:54:36.647 INFO:tasks.workunit.client.1.smithi190.stdout:3/570: dwrite d1/de/f73 [0,4194304] 0 2024-03-26T18:54:36.649 INFO:tasks.workunit.client.1.smithi190.stdout:0/745: dwrite d0/dd/d20/fbe [0,4194304] 0 2024-03-26T18:54:36.653 INFO:tasks.workunit.client.0.smithi102.stdout:6/342: link d2/df/f64 d2/d3/f6c 0 2024-03-26T18:54:36.654 INFO:tasks.workunit.client.0.smithi102.stdout:2/334: dwrite d3/d6/ff [8388608,4194304] 0 2024-03-26T18:54:36.654 INFO:tasks.workunit.client.0.smithi102.stdout:2/335: write d3/d6/fa [1597330,100342] 0 2024-03-26T18:54:36.656 INFO:tasks.workunit.client.1.smithi190.stdout:4/695: dwrite d7/d8/d14/d23/d28/d38/fcf [0,4194304] 0 2024-03-26T18:54:36.656 INFO:tasks.workunit.client.1.smithi190.stdout:4/696: read d7/f49 [200360,58919] 0 2024-03-26T18:54:36.656 INFO:tasks.workunit.client.1.smithi190.stdout:4/697: readlink d7/d1d/d42/d4a/l8f 0 2024-03-26T18:54:36.656 INFO:tasks.workunit.client.1.smithi190.stdout:4/698: readlink d7/d8/d14/d4b/l51 0 2024-03-26T18:54:36.657 INFO:tasks.workunit.client.1.smithi190.stdout:4/699: write d7/d8/d14/d5c/f97 [2728293,11246] 0 2024-03-26T18:54:36.657 INFO:tasks.workunit.client.0.smithi102.stdout:3/569: rename df/d61/f52 to df/d8b/dbb/fcf 0 2024-03-26T18:54:36.657 INFO:tasks.workunit.client.0.smithi102.stdout:3/570: stat df/d1c/d34/d53/l8f 0 2024-03-26T18:54:36.657 INFO:tasks.workunit.client.0.smithi102.stdout:3/571: creat df/d29/d31/d77/d8a/fd0 x:0 0 0 2024-03-26T18:54:36.657 INFO:tasks.workunit.client.0.smithi102.stdout:3/572: creat df/d8b/fd1 x:0 0 0 2024-03-26T18:54:36.658 INFO:tasks.workunit.client.0.smithi102.stdout:8/413: creat d0/d3/d26/d19/f82 x:0 0 0 2024-03-26T18:54:36.659 INFO:tasks.workunit.client.1.smithi190.stdout:9/927: dread d12/f4c [0,4194304] 0 2024-03-26T18:54:36.659 INFO:tasks.workunit.client.1.smithi190.stdout:4/700: mkdir d7/dc8/dd5/de4/def 0 2024-03-26T18:54:36.660 INFO:tasks.workunit.client.0.smithi102.stdout:5/612: creat d2/d25/d75/db9/db3/fd9 x:0 0 0 2024-03-26T18:54:36.661 INFO:tasks.workunit.client.1.smithi190.stdout:9/928: creat d12/d2d/d1d/d23/d2b/da0/d138/f148 x:0 0 0 2024-03-26T18:54:36.662 INFO:tasks.workunit.client.0.smithi102.stdout:1/355: getdents de/d2e 0 2024-03-26T18:54:36.662 INFO:tasks.workunit.client.0.smithi102.stdout:1/356: truncate de/d1f/f3a 2045341 0 2024-03-26T18:54:36.662 INFO:tasks.workunit.client.0.smithi102.stdout:1/357: fsync de/d1f/f33 0 2024-03-26T18:54:36.662 INFO:tasks.workunit.client.0.smithi102.stdout:1/358: fsync de/f2d 0 2024-03-26T18:54:36.663 INFO:tasks.workunit.client.1.smithi190.stdout:4/701: mkdir d7/d8/d14/d6e/df0 0 2024-03-26T18:54:36.664 INFO:tasks.workunit.client.0.smithi102.stdout:7/371: rmdir d7/d10 39 2024-03-26T18:54:36.664 INFO:tasks.workunit.client.0.smithi102.stdout:7/372: fsync d7/d10/d27/d34/f49 0 2024-03-26T18:54:36.664 INFO:tasks.workunit.client.0.smithi102.stdout:6/343: write d2/d3/f38 [6359600,51856] 0 2024-03-26T18:54:36.664 INFO:tasks.workunit.client.0.smithi102.stdout:6/344: chown d2/df/d35/d60 893904 1 2024-03-26T18:54:36.664 INFO:tasks.workunit.client.0.smithi102.stdout:6/345: write d2/df/d35/d43/f61 [671184,118043] 0 2024-03-26T18:54:36.664 INFO:tasks.workunit.client.0.smithi102.stdout:6/346: fdatasync d2/d7/d3c/f3e 0 2024-03-26T18:54:36.665 INFO:tasks.workunit.client.1.smithi190.stdout:0/746: write d0/dd/d20/d1c/d46/da7/df0/da2/d80/f10c [2209325,38384] 0 2024-03-26T18:54:36.665 INFO:tasks.workunit.client.1.smithi190.stdout:6/634: dwrite d17/d28/d2c/d63/fa0 [0,4194304] 0 2024-03-26T18:54:36.665 INFO:tasks.workunit.client.1.smithi190.stdout:9/929: creat d12/d2d/d1d/d23/d2b/da0/f149 x:0 0 0 2024-03-26T18:54:36.666 INFO:tasks.workunit.client.0.smithi102.stdout:4/290: dwrite d1/d17/d1a/d21/f2c [0,4194304] 0 2024-03-26T18:54:36.666 INFO:tasks.workunit.client.0.smithi102.stdout:2/336: mkdir d3/d74 0 2024-03-26T18:54:36.666 INFO:tasks.workunit.client.1.smithi190.stdout:4/702: rename d7/d8/d14/d23/d28/f8a to d7/d8/d1c/ff1 0 2024-03-26T18:54:36.667 INFO:tasks.workunit.client.1.smithi190.stdout:9/930: rmdir d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa 39 2024-03-26T18:54:36.667 INFO:tasks.workunit.client.1.smithi190.stdout:9/931: chown d12/d2d/d1d/fd3 241765 1 2024-03-26T18:54:36.668 INFO:tasks.workunit.client.0.smithi102.stdout:9/318: dwrite df/f40 [0,4194304] 0 2024-03-26T18:54:36.668 INFO:tasks.workunit.client.0.smithi102.stdout:9/319: fdatasync df/d46/f67 0 2024-03-26T18:54:36.668 INFO:tasks.workunit.client.0.smithi102.stdout:9/320: write df/d46/f6a [263433,65135] 0 2024-03-26T18:54:36.668 INFO:tasks.workunit.client.0.smithi102.stdout:1/359: dread fc [0,4194304] 0 2024-03-26T18:54:36.669 INFO:tasks.workunit.client.1.smithi190.stdout:3/571: rename d1/d3/df/d18/d35/d41/d51 to d1/d3/df/d18/d35/d41/dae 0 2024-03-26T18:54:36.669 INFO:tasks.workunit.client.1.smithi190.stdout:8/740: dwrite d9/d39/d79/f7e [0,4194304] 0 2024-03-26T18:54:36.669 INFO:tasks.workunit.client.1.smithi190.stdout:4/703: rmdir d7/d8/d14/d4b/db2 39 2024-03-26T18:54:36.669 INFO:tasks.workunit.client.1.smithi190.stdout:4/704: stat d7/d8/d14/f19 0 2024-03-26T18:54:36.669 INFO:tasks.workunit.client.1.smithi190.stdout:2/683: dwrite d6/d9/d3a/f83 [0,4194304] 0 2024-03-26T18:54:36.669 INFO:tasks.workunit.client.1.smithi190.stdout:9/932: getdents d12/d2d/d1d/da1/dba/dd2 0 2024-03-26T18:54:36.670 INFO:tasks.workunit.client.1.smithi190.stdout:8/741: fsync f6 0 2024-03-26T18:54:36.673 INFO:tasks.workunit.client.1.smithi190.stdout:0/747: symlink d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/l110 0 2024-03-26T18:54:36.674 INFO:tasks.workunit.client.1.smithi190.stdout:4/705: link d7/d8/l5d d7/d8/d1c/lf2 0 2024-03-26T18:54:36.674 INFO:tasks.workunit.client.1.smithi190.stdout:9/933: dread d12/d2d/d1d/fd3 [0,4194304] 0 2024-03-26T18:54:36.674 INFO:tasks.workunit.client.1.smithi190.stdout:9/934: fsync d12/d2d/f9f 0 2024-03-26T18:54:36.676 INFO:tasks.workunit.client.1.smithi190.stdout:2/684: mkdir d6/d9/d45/df3 0 2024-03-26T18:54:36.678 INFO:tasks.workunit.client.1.smithi190.stdout:8/742: mkdir d9/dd/d23/d88/dfa 0 2024-03-26T18:54:36.678 INFO:tasks.workunit.client.1.smithi190.stdout:8/743: chown d9/da2/df6 1994 1 2024-03-26T18:54:36.678 INFO:tasks.workunit.client.0.smithi102.stdout:8/414: rename d0/c5 to d0/d15/d81/c83 0 2024-03-26T18:54:36.678 INFO:tasks.workunit.client.0.smithi102.stdout:6/347: dread d2/d7/f1d [0,4194304] 0 2024-03-26T18:54:36.682 INFO:tasks.workunit.client.1.smithi190.stdout:0/748: dread d0/dd/d20/d1c/d46/da7/df0/f2b [4194304,4194304] 0 2024-03-26T18:54:36.682 INFO:tasks.workunit.client.1.smithi190.stdout:0/749: chown d0/dd/d20/fdf 2621 1 2024-03-26T18:54:36.682 INFO:tasks.workunit.client.1.smithi190.stdout:0/750: chown d0/dd/d20/d1c/d46/da7 5390 1 2024-03-26T18:54:36.682 INFO:tasks.workunit.client.1.smithi190.stdout:4/706: creat d7/dc8/dd5/de4/ff3 x:0 0 0 2024-03-26T18:54:36.682 INFO:tasks.workunit.client.1.smithi190.stdout:0/751: dread d0/dd/d20/d1c/d4e/f9d [0,4194304] 0 2024-03-26T18:54:36.682 INFO:tasks.workunit.client.1.smithi190.stdout:9/935: rename d12/d2d/d1d/d23/d83/d91/c109 to d12/d2d/d1d/d54/c14a 0 2024-03-26T18:54:36.683 INFO:tasks.workunit.client.1.smithi190.stdout:2/685: creat d6/d21/dd8/ff4 x:0 0 0 2024-03-26T18:54:36.683 INFO:tasks.workunit.client.0.smithi102.stdout:5/613: creat d2/d25/d2a/dbb/dcf/fda x:0 0 0 2024-03-26T18:54:36.683 INFO:tasks.workunit.client.0.smithi102.stdout:7/373: symlink d7/d4e/l6b 0 2024-03-26T18:54:36.684 INFO:tasks.workunit.client.1.smithi190.stdout:3/572: creat d1/d3/df/d18/d57/d91/d8c/faf x:0 0 0 2024-03-26T18:54:36.684 INFO:tasks.workunit.client.1.smithi190.stdout:3/573: fdatasync d1/d3/df/f2a 0 2024-03-26T18:54:36.684 INFO:tasks.workunit.client.1.smithi190.stdout:8/744: mkdir d9/d39/d79/dad/dfb 0 2024-03-26T18:54:36.686 INFO:tasks.workunit.client.1.smithi190.stdout:6/635: mkdir d17/dd3 0 2024-03-26T18:54:36.686 INFO:tasks.workunit.client.1.smithi190.stdout:2/686: dread d6/d9/d3a/d3d/d1c/d8d/fab [0,4194304] 0 2024-03-26T18:54:36.688 INFO:tasks.workunit.client.1.smithi190.stdout:2/687: mknod d6/d9/d3a/d3d/d1c/d8d/dc7/dd1/cf5 0 2024-03-26T18:54:36.690 INFO:tasks.workunit.client.1.smithi190.stdout:3/574: rename d1/d3/df/f2a to d1/de/d81/dad/fb0 0 2024-03-26T18:54:36.692 INFO:tasks.workunit.client.0.smithi102.stdout:9/321: dwrite df/d1a/d23/d4e/f4f [0,4194304] 0 2024-03-26T18:54:36.693 INFO:tasks.workunit.client.1.smithi190.stdout:4/707: dread d7/d1d/f26 [0,4194304] 0 2024-03-26T18:54:36.695 INFO:tasks.workunit.client.1.smithi190.stdout:6/636: mknod d17/dd3/cd4 0 2024-03-26T18:54:36.696 INFO:tasks.workunit.client.1.smithi190.stdout:0/752: write d0/dd/d20/d1c/d46/da7/df0/f2b [1133944,124197] 0 2024-03-26T18:54:36.697 INFO:tasks.workunit.client.0.smithi102.stdout:2/337: unlink d3/d6/f23 0 2024-03-26T18:54:36.697 INFO:tasks.workunit.client.0.smithi102.stdout:2/338: truncate d3/d11/f50 833862 0 2024-03-26T18:54:36.697 INFO:tasks.workunit.client.1.smithi190.stdout:9/936: dread d12/d2d/d1d/d23/d83/dc9/dd4/fe8 [0,4194304] 0 2024-03-26T18:54:36.699 INFO:tasks.workunit.client.1.smithi190.stdout:2/688: dread d6/d9/d10/db7/dcf/fd4 [0,4194304] 0 2024-03-26T18:54:36.699 INFO:tasks.workunit.client.0.smithi102.stdout:4/291: write d1/d25/d39/f4e [2360666,100291] 0 2024-03-26T18:54:36.701 INFO:tasks.workunit.client.1.smithi190.stdout:8/745: read d9/d39/d79/f7e [1251741,62769] 0 2024-03-26T18:54:36.702 INFO:tasks.workunit.client.1.smithi190.stdout:6/637: unlink d17/d28/d2c/db8/c6d 0 2024-03-26T18:54:36.705 INFO:tasks.workunit.client.1.smithi190.stdout:9/937: symlink d12/d2d/d1d/da1/dba/dd2/l14b 0 2024-03-26T18:54:36.705 INFO:tasks.workunit.client.1.smithi190.stdout:9/938: readlink d12/l4a 0 2024-03-26T18:54:36.706 INFO:tasks.workunit.client.1.smithi190.stdout:0/753: creat d0/dd/d20/d1c/d46/da7/df0/f111 x:0 0 0 2024-03-26T18:54:36.706 INFO:tasks.workunit.client.1.smithi190.stdout:2/689: rmdir d6/d9/d3a/d3d/d67/d9e/da3 39 2024-03-26T18:54:36.709 INFO:tasks.workunit.client.1.smithi190.stdout:8/746: mkdir d9/da2/d73/dfc 0 2024-03-26T18:54:36.710 INFO:tasks.workunit.client.0.smithi102.stdout:9/322: read df/d1a/d23/d4e/f53 [324988,77263] 0 2024-03-26T18:54:36.710 INFO:tasks.workunit.client.1.smithi190.stdout:9/939: rename d12/d2d/d1d/d54/c60 to d12/d2d/d1d/d54/d92/deb/c14c 0 2024-03-26T18:54:36.711 INFO:tasks.workunit.client.1.smithi190.stdout:2/690: symlink d6/d9/d45/lf6 0 2024-03-26T18:54:36.711 INFO:tasks.workunit.client.1.smithi190.stdout:2/691: chown d6/d21/dd8 43154 1 2024-03-26T18:54:36.712 INFO:tasks.workunit.client.0.smithi102.stdout:6/348: rmdir d2/d5a 0 2024-03-26T18:54:36.713 INFO:tasks.workunit.client.1.smithi190.stdout:6/638: read d17/f87 [664913,93735] 0 2024-03-26T18:54:36.713 INFO:tasks.workunit.client.1.smithi190.stdout:8/747: link d9/dd/d23/d36/d37/d3a/l44 d9/dd/d15/lfd 0 2024-03-26T18:54:36.714 INFO:tasks.workunit.client.1.smithi190.stdout:0/754: creat d0/dd/d20/d1c/d46/da7/df0/da2/f112 x:0 0 0 2024-03-26T18:54:36.715 INFO:tasks.workunit.client.1.smithi190.stdout:2/692: creat d6/d9/d45/ff7 x:0 0 0 2024-03-26T18:54:36.716 INFO:tasks.workunit.client.0.smithi102.stdout:7/374: mknod d7/d10/d27/c6c 0 2024-03-26T18:54:36.718 INFO:tasks.workunit.client.1.smithi190.stdout:0/755: link d0/dd/d20/d1c/d46/da7/df0/da2/f9e d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/f113 0 2024-03-26T18:54:36.718 INFO:tasks.workunit.client.1.smithi190.stdout:0/756: read d0/dd/d20/d1c/d46/f5d [1392298,5712] 0 2024-03-26T18:54:36.718 INFO:tasks.workunit.client.1.smithi190.stdout:2/693: creat d6/d9/d3a/d3d/d67/d9e/ff8 x:0 0 0 2024-03-26T18:54:36.722 INFO:tasks.workunit.client.1.smithi190.stdout:0/757: truncate d0/dd/d20/d1c/d46/da7/df0/da2/fc6 2755626 0 2024-03-26T18:54:36.724 INFO:tasks.workunit.client.0.smithi102.stdout:9/323: symlink df/d1a/d65/l7a 0 2024-03-26T18:54:36.724 INFO:tasks.workunit.client.1.smithi190.stdout:3/575: dwrite d1/d3/df/f19 [0,4194304] 0 2024-03-26T18:54:36.724 INFO:tasks.workunit.client.1.smithi190.stdout:3/576: creat d1/d3/df/d18/d1b/fb1 x:0 0 0 2024-03-26T18:54:36.726 INFO:tasks.workunit.client.1.smithi190.stdout:0/758: unlink d0/dd/d20/d1c/d8b/d8d/db7/fba 0 2024-03-26T18:54:36.728 INFO:tasks.workunit.client.0.smithi102.stdout:8/415: dwrite d0/d3/d26/f22 [4194304,4194304] 0 2024-03-26T18:54:36.728 INFO:tasks.workunit.client.0.smithi102.stdout:8/416: chown d0/d3/d31/c37 206643 1 2024-03-26T18:54:36.728 INFO:tasks.workunit.client.1.smithi190.stdout:4/708: dwrite d7/d1d/d5e/f81 [0,4194304] 0 2024-03-26T18:54:36.728 INFO:tasks.workunit.client.1.smithi190.stdout:4/709: creat d7/d8/d14/d23/d28/d38/ff4 x:0 0 0 2024-03-26T18:54:36.729 INFO:tasks.workunit.client.1.smithi190.stdout:4/710: write d7/d1d/d5e/d98/fc9 [633393,107993] 0 2024-03-26T18:54:36.729 INFO:tasks.workunit.client.0.smithi102.stdout:6/349: unlink d2/l13 0 2024-03-26T18:54:36.730 INFO:tasks.workunit.client.1.smithi190.stdout:4/711: symlink d7/d8/d14/d4b/lf5 0 2024-03-26T18:54:36.731 INFO:tasks.workunit.client.0.smithi102.stdout:5/614: rmdir d2/d25/d75/db9/d43/d9e/d8b 39 2024-03-26T18:54:36.731 INFO:tasks.workunit.client.0.smithi102.stdout:5/615: stat d2/d25/d75/db9/d43/d9e/d82/fa2 0 2024-03-26T18:54:36.731 INFO:tasks.workunit.client.1.smithi190.stdout:4/712: symlink d7/d8/d14/d6e/d6f/lf6 0 2024-03-26T18:54:36.734 INFO:tasks.workunit.client.0.smithi102.stdout:7/375: unlink d7/d2a/l54 0 2024-03-26T18:54:36.738 INFO:tasks.workunit.client.1.smithi190.stdout:0/759: creat d0/dd/d20/d1c/d46/d7c/f114 x:0 0 0 2024-03-26T18:54:36.740 INFO:tasks.workunit.client.0.smithi102.stdout:9/324: creat df/d1a/d23/d29/d45/f7b x:0 0 0 2024-03-26T18:54:36.740 INFO:tasks.workunit.client.0.smithi102.stdout:9/325: creat df/d1a/d23/d4e/f7c x:0 0 0 2024-03-26T18:54:36.743 INFO:tasks.workunit.client.0.smithi102.stdout:6/350: mknod d2/df/d35/c6d 0 2024-03-26T18:54:36.743 INFO:tasks.workunit.client.0.smithi102.stdout:6/351: write d2/d3d/d53/f66 [662760,27520] 0 2024-03-26T18:54:36.744 INFO:tasks.workunit.client.1.smithi190.stdout:3/577: rmdir d1/d3/d52 39 2024-03-26T18:54:36.744 INFO:tasks.workunit.client.1.smithi190.stdout:3/578: chown d1/d3/d5a/d6e 4 1 2024-03-26T18:54:36.746 INFO:tasks.workunit.client.0.smithi102.stdout:5/616: creat d2/d25/d75/db9/d43/fdb x:0 0 0 2024-03-26T18:54:36.747 INFO:tasks.workunit.client.1.smithi190.stdout:4/713: read d7/d8/d14/f48 [284945,8942] 0 2024-03-26T18:54:36.747 INFO:tasks.workunit.client.1.smithi190.stdout:4/714: creat d7/d8/d14/d5c/ff7 x:0 0 0 2024-03-26T18:54:36.747 INFO:tasks.workunit.client.1.smithi190.stdout:0/760: creat d0/dd/d20/d1c/d46/da7/df0/d87/d93/dc4/dd8/f115 x:0 0 0 2024-03-26T18:54:36.747 INFO:tasks.workunit.client.1.smithi190.stdout:0/761: truncate d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/f82 475523 0 2024-03-26T18:54:36.748 INFO:tasks.workunit.client.1.smithi190.stdout:6/639: dwrite d17/d28/d80/da8/fb2 [4194304,4194304] 0 2024-03-26T18:54:36.749 INFO:tasks.workunit.client.0.smithi102.stdout:7/376: rename d7/cc to d7/d12/d4b/c6d 0 2024-03-26T18:54:36.751 INFO:tasks.workunit.client.1.smithi190.stdout:4/715: mkdir d7/d8/d14/d4b/d7d/da1/df8 0 2024-03-26T18:54:36.751 INFO:tasks.workunit.client.1.smithi190.stdout:3/579: rename d1/d3/df/d31/c45 to d1/d3/df/cb2 0 2024-03-26T18:54:36.751 INFO:tasks.workunit.client.1.smithi190.stdout:3/580: stat d1/d3/df/d18/d35/fac 0 2024-03-26T18:54:36.754 INFO:tasks.workunit.client.0.smithi102.stdout:4/292: link d1/d25/d39/d3e/f58 d1/d25/d39/f5e 0 2024-03-26T18:54:36.754 INFO:tasks.workunit.client.1.smithi190.stdout:4/716: rename d7/d8/d14/d23/d28/fee to d7/dc8/dd5/de4/def/ff9 0 2024-03-26T18:54:36.755 INFO:tasks.workunit.client.1.smithi190.stdout:4/717: rmdir d7/dc8/dd5/de4/def 39 2024-03-26T18:54:36.755 INFO:tasks.workunit.client.1.smithi190.stdout:0/762: mkdir d0/dd/d20/d1c/d8b/d116 0 2024-03-26T18:54:36.756 INFO:tasks.workunit.client.1.smithi190.stdout:4/718: rmdir d7/d1d/d41 39 2024-03-26T18:54:36.756 INFO:tasks.workunit.client.1.smithi190.stdout:4/719: dread - d7/d8/d14/d6e/d6f/f93 zero size 2024-03-26T18:54:36.757 INFO:tasks.workunit.client.0.smithi102.stdout:9/326: dread df/d1a/d23/f5d [0,4194304] 0 2024-03-26T18:54:36.757 INFO:tasks.workunit.client.0.smithi102.stdout:9/327: dread - df/d1a/d23/d4e/f7c zero size 2024-03-26T18:54:36.757 INFO:tasks.workunit.client.1.smithi190.stdout:8/748: dwrite d9/dd/d23/d36/d37/d3a/d61/f91 [0,4194304] 0 2024-03-26T18:54:36.757 INFO:tasks.workunit.client.1.smithi190.stdout:9/940: dwrite d12/d2d/f31 [0,4194304] 0 2024-03-26T18:54:36.757 INFO:tasks.workunit.client.0.smithi102.stdout:8/417: dwrite d0/d3/d31/f40 [0,4194304] 0 2024-03-26T18:54:36.758 INFO:tasks.workunit.client.1.smithi190.stdout:6/640: symlink d17/d28/d46/d6a/ld5 0 2024-03-26T18:54:36.759 INFO:tasks.workunit.client.1.smithi190.stdout:4/720: rename d7/d8/d14/d5c/fda to d7/d1d/d5e/d98/ffa 0 2024-03-26T18:54:36.760 INFO:tasks.workunit.client.0.smithi102.stdout:6/352: creat d2/df/d35/f6e x:0 0 0 2024-03-26T18:54:36.761 INFO:tasks.workunit.client.1.smithi190.stdout:3/581: mkdir d1/d3/d52/d70/db3 0 2024-03-26T18:54:36.765 INFO:tasks.workunit.client.1.smithi190.stdout:0/763: creat d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/f117 x:0 0 0 2024-03-26T18:54:36.767 INFO:tasks.workunit.client.0.smithi102.stdout:5/617: rename d2/d14/dae to d2/d25/d2a/d4f/ddc 0 2024-03-26T18:54:36.768 INFO:tasks.workunit.client.1.smithi190.stdout:8/749: mkdir d9/dd/d23/dea/dfe 0 2024-03-26T18:54:36.773 INFO:tasks.workunit.client.1.smithi190.stdout:9/941: link d12/d2d/cde d12/d2d/d1d/d54/d92/de3/dff/c14d 0 2024-03-26T18:54:36.773 INFO:tasks.workunit.client.1.smithi190.stdout:9/942: stat d12/d2d/d1d/d23/de5/c132 0 2024-03-26T18:54:36.777 INFO:tasks.workunit.client.1.smithi190.stdout:3/582: mkdir d1/d3/df/d18/db4 0 2024-03-26T18:54:36.777 INFO:tasks.workunit.client.1.smithi190.stdout:3/583: dread - d1/d3/df/d18/d35/f39 zero size 2024-03-26T18:54:36.777 INFO:tasks.workunit.client.1.smithi190.stdout:3/584: chown d1/d3 0 1 2024-03-26T18:54:36.781 INFO:tasks.workunit.client.1.smithi190.stdout:4/721: truncate d7/f13 821044 0 2024-03-26T18:54:36.782 INFO:tasks.workunit.client.1.smithi190.stdout:8/750: creat d9/dd/d23/fff x:0 0 0 2024-03-26T18:54:36.782 INFO:tasks.workunit.client.1.smithi190.stdout:3/585: rename d1/d3/df/d18/d1b/f21 to d1/fb5 0 2024-03-26T18:54:36.784 INFO:tasks.workunit.client.1.smithi190.stdout:4/722: rmdir d7/d8/d14/d4b/db2 39 2024-03-26T18:54:36.785 INFO:tasks.workunit.client.1.smithi190.stdout:3/586: symlink d1/d3/df/d18/d35/d41/dae/d86/lb6 0 2024-03-26T18:54:36.787 INFO:tasks.workunit.client.0.smithi102.stdout:8/418: unlink d0/d3/d31/l71 0 2024-03-26T18:54:36.789 INFO:tasks.workunit.client.0.smithi102.stdout:6/353: creat d2/d3d/d53/d63/d6a/f6f x:0 0 0 2024-03-26T18:54:36.792 INFO:tasks.workunit.client.0.smithi102.stdout:7/377: rename d7/d10/f68 to d7/d12/d2b/d50/f6e 0 2024-03-26T18:54:36.795 INFO:tasks.workunit.client.0.smithi102.stdout:8/419: mknod d0/d3c/d66/c84 0 2024-03-26T18:54:36.795 INFO:tasks.workunit.client.1.smithi190.stdout:3/587: dread d1/de/d81/dad/fb0 [0,4194304] 0 2024-03-26T18:54:36.796 INFO:tasks.workunit.client.0.smithi102.stdout:6/354: chown d2/d7/f8 2 1 2024-03-26T18:54:36.798 INFO:tasks.workunit.client.0.smithi102.stdout:4/293: rename d1/f31 to d1/d27/f5f 0 2024-03-26T18:54:36.799 INFO:tasks.workunit.client.0.smithi102.stdout:8/420: link d0/d15/d81/f5b d0/d3/d2c/f85 0 2024-03-26T18:54:36.800 INFO:tasks.workunit.client.0.smithi102.stdout:6/355: unlink d2/d3d/d53/d5c/f5e 0 2024-03-26T18:54:36.800 INFO:tasks.workunit.client.0.smithi102.stdout:6/356: dread - d2/d3d/d53/d5c/f65 zero size 2024-03-26T18:54:36.801 INFO:tasks.workunit.client.0.smithi102.stdout:4/294: creat d1/f60 x:0 0 0 2024-03-26T18:54:36.801 INFO:tasks.workunit.client.0.smithi102.stdout:4/295: creat d1/d17/d2a/f61 x:0 0 0 2024-03-26T18:54:36.801 INFO:tasks.workunit.client.0.smithi102.stdout:6/357: creat d2/d3d/f70 x:0 0 0 2024-03-26T18:54:36.805 INFO:tasks.workunit.client.0.smithi102.stdout:7/378: read d7/d10/d27/f33 [2058998,104664] 0 2024-03-26T18:54:36.805 INFO:tasks.workunit.client.0.smithi102.stdout:6/358: read d2/d7/f1d [4882279,122930] 0 2024-03-26T18:54:36.805 INFO:tasks.workunit.client.0.smithi102.stdout:6/359: truncate d2/f34 133363 0 2024-03-26T18:54:36.809 INFO:tasks.workunit.client.1.smithi190.stdout:6/641: dwrite d17/d28/d2b/f3c [0,4194304] 0 2024-03-26T18:54:36.809 INFO:tasks.workunit.client.1.smithi190.stdout:6/642: chown d17/f19 7 1 2024-03-26T18:54:36.810 INFO:tasks.workunit.client.0.smithi102.stdout:9/328: dwrite df/f3b [4194304,4194304] 0 2024-03-26T18:54:36.811 INFO:tasks.workunit.client.1.smithi190.stdout:6/643: mknod d17/d28/d2c/d77/cd6 0 2024-03-26T18:54:36.813 INFO:tasks.workunit.client.0.smithi102.stdout:8/421: dread d0/d3/d2c/f85 [0,4194304] 0 2024-03-26T18:54:36.813 INFO:tasks.workunit.client.0.smithi102.stdout:8/422: truncate d0/d3/d26/d19/f82 591783 0 2024-03-26T18:54:36.815 INFO:tasks.workunit.client.1.smithi190.stdout:6/644: mkdir d17/d28/d46/dd7 0 2024-03-26T18:54:36.815 INFO:tasks.workunit.client.1.smithi190.stdout:6/645: truncate d17/d28/d38/d79/fd1 990636 0 2024-03-26T18:54:36.820 INFO:tasks.workunit.client.1.smithi190.stdout:6/646: dread d17/d28/d2c/f47 [0,4194304] 0 2024-03-26T18:54:36.821 INFO:tasks.workunit.client.1.smithi190.stdout:0/764: dwrite d0/dd/d20/d1c/d8b/d8d/db7/de6/fe9 [0,4194304] 0 2024-03-26T18:54:36.821 INFO:tasks.workunit.client.1.smithi190.stdout:6/647: write d17/d28/d74/d97/fb9 [659668,130538] 0 2024-03-26T18:54:36.825 INFO:tasks.workunit.client.0.smithi102.stdout:9/329: write df/d1a/d23/f48 [7237955,65476] 0 2024-03-26T18:54:36.826 INFO:tasks.workunit.client.1.smithi190.stdout:4/723: dwrite d7/d8/d14/d4b/fc4 [0,4194304] 0 2024-03-26T18:54:36.828 INFO:tasks.workunit.client.0.smithi102.stdout:6/360: dwrite d2/d3d/d53/d5c/f5d [0,4194304] 0 2024-03-26T18:54:36.828 INFO:tasks.workunit.client.0.smithi102.stdout:6/361: truncate d2/df/d35/f6e 812245 0 2024-03-26T18:54:36.828 INFO:tasks.workunit.client.0.smithi102.stdout:6/362: write d2/f33 [933154,91137] 0 2024-03-26T18:54:36.829 INFO:tasks.workunit.client.0.smithi102.stdout:4/296: dwrite d1/d17/d1a/f57 [0,4194304] 0 2024-03-26T18:54:36.829 INFO:tasks.workunit.client.1.smithi190.stdout:8/751: dwrite d9/dd/d23/d36/d37/d3a/d43/f80 [0,4194304] 0 2024-03-26T18:54:36.829 INFO:tasks.workunit.client.1.smithi190.stdout:8/752: dread - d9/dd/d23/d88/fe4 zero size 2024-03-26T18:54:36.830 INFO:tasks.workunit.client.1.smithi190.stdout:4/724: mknod d7/d8/d14/d4b/d7d/cfb 0 2024-03-26T18:54:36.830 INFO:tasks.workunit.client.1.smithi190.stdout:4/725: truncate d7/d8/d14/d4b/d7d/da1/fe9 422782 0 2024-03-26T18:54:36.830 INFO:tasks.workunit.client.1.smithi190.stdout:8/753: mkdir d9/dd/d100 0 2024-03-26T18:54:36.831 INFO:tasks.workunit.client.1.smithi190.stdout:4/726: getdents d7/d8/d14/d23 0 2024-03-26T18:54:36.832 INFO:tasks.workunit.client.1.smithi190.stdout:4/727: fsync d7/d8/d1c/d20/f78 0 2024-03-26T18:54:36.832 INFO:tasks.workunit.client.1.smithi190.stdout:4/728: write d7/d8/d14/d23/d28/d38/f71 [1073685,23936] 0 2024-03-26T18:54:36.832 INFO:tasks.workunit.client.1.smithi190.stdout:4/729: fdatasync d7/d1d/d5e/fce 0 2024-03-26T18:54:36.833 INFO:tasks.workunit.client.1.smithi190.stdout:4/730: creat d7/d8/d14/d6e/df0/ffc x:0 0 0 2024-03-26T18:54:36.833 INFO:tasks.workunit.client.1.smithi190.stdout:4/731: dread - d7/dc8/fd9 zero size 2024-03-26T18:54:36.833 INFO:tasks.workunit.client.1.smithi190.stdout:4/732: write d7/d8/d14/f94 [617162,57134] 0 2024-03-26T18:54:36.833 INFO:tasks.workunit.client.1.smithi190.stdout:4/733: chown d7/d8/d1c/ff1 47183913 1 2024-03-26T18:54:36.833 INFO:tasks.workunit.client.1.smithi190.stdout:4/734: chown d7/d8/d14/d23/d28/d38/dd3/fe8 1 1 2024-03-26T18:54:36.833 INFO:tasks.workunit.client.1.smithi190.stdout:4/735: fdatasync d7/d1d/f32 0 2024-03-26T18:54:36.838 INFO:tasks.workunit.client.1.smithi190.stdout:8/754: dread d9/dd/f2d [0,4194304] 0 2024-03-26T18:54:36.839 INFO:tasks.workunit.client.1.smithi190.stdout:8/755: write d9/dd/f78 [963760,53943] 0 2024-03-26T18:54:36.840 INFO:tasks.workunit.client.1.smithi190.stdout:8/756: symlink d9/dd/d23/dea/dfe/l101 0 2024-03-26T18:54:36.841 INFO:tasks.workunit.client.1.smithi190.stdout:8/757: truncate d9/d48/d49/fc0 949649 0 2024-03-26T18:54:36.844 INFO:tasks.workunit.client.1.smithi190.stdout:8/758: dread d9/d39/f6d [0,4194304] 0 2024-03-26T18:54:36.845 INFO:tasks.workunit.client.1.smithi190.stdout:8/759: getdents d9/d48/d49 0 2024-03-26T18:54:36.849 INFO:tasks.workunit.client.1.smithi190.stdout:8/760: dread d9/dd/f2d [0,4194304] 0 2024-03-26T18:54:36.849 INFO:tasks.workunit.client.1.smithi190.stdout:8/761: fdatasync f6 0 2024-03-26T18:54:36.852 INFO:tasks.workunit.client.1.smithi190.stdout:4/736: dwrite d7/d8/f22 [0,4194304] 0 2024-03-26T18:54:36.852 INFO:tasks.workunit.client.1.smithi190.stdout:4/737: truncate d7/d8/d14/d5c/ff7 868061 0 2024-03-26T18:54:36.853 INFO:tasks.workunit.client.1.smithi190.stdout:4/738: mkdir d7/d8/d14/d23/d28/d38/dd3/dfd 0 2024-03-26T18:54:36.854 INFO:tasks.workunit.client.1.smithi190.stdout:4/739: mkdir d7/d1d/d42/d4a/dfe 0 2024-03-26T18:54:36.856 INFO:tasks.workunit.client.1.smithi190.stdout:4/740: creat d7/d1d/fff x:0 0 0 2024-03-26T18:54:36.856 INFO:tasks.workunit.client.1.smithi190.stdout:4/741: chown d7/d8/d14/d23/d28/d38/c65 6 1 2024-03-26T18:54:36.856 INFO:tasks.workunit.client.1.smithi190.stdout:4/742: write d7/d8/d14/f19 [2159375,35709] 0 2024-03-26T18:54:36.857 INFO:tasks.workunit.client.1.smithi190.stdout:4/743: symlink d7/d8/l100 0 2024-03-26T18:54:36.866 INFO:tasks.workunit.client.1.smithi190.stdout:8/762: dwrite d9/d39/f6d [0,4194304] 0 2024-03-26T18:54:36.868 INFO:tasks.workunit.client.1.smithi190.stdout:8/763: dread d9/d39/d67/f94 [0,4194304] 0 2024-03-26T18:54:36.869 INFO:tasks.workunit.client.1.smithi190.stdout:8/764: truncate d9/fa 2685187 0 2024-03-26T18:54:36.871 INFO:tasks.workunit.client.1.smithi190.stdout:8/765: symlink d9/da2/dc4/l102 0 2024-03-26T18:54:36.871 INFO:tasks.workunit.client.1.smithi190.stdout:8/766: write d9/dd/d23/d36/d37/db8/fd9 [917796,47828] 0 2024-03-26T18:54:36.873 INFO:tasks.workunit.client.1.smithi190.stdout:8/767: mknod d9/d39/d79/dad/c103 0 2024-03-26T18:54:36.874 INFO:tasks.workunit.client.1.smithi190.stdout:8/768: creat d9/dd/d23/db1/f104 x:0 0 0 2024-03-26T18:54:36.874 INFO:tasks.workunit.client.1.smithi190.stdout:8/769: readlink d9/da2/d87/ld1 0 2024-03-26T18:54:36.875 INFO:tasks.workunit.client.1.smithi190.stdout:8/770: truncate d9/dd/d15/f60 1877755 0 2024-03-26T18:54:36.877 INFO:tasks.workunit.client.0.smithi102.stdout:8/423: creat d0/d3/d2c/d58/d75/f86 x:0 0 0 2024-03-26T18:54:36.877 INFO:tasks.workunit.client.1.smithi190.stdout:8/771: mknod d9/d39/d67/c105 0 2024-03-26T18:54:36.878 INFO:tasks.workunit.client.1.smithi190.stdout:8/772: symlink d9/dd/d23/d36/d37/d3a/d61/daf/l106 0 2024-03-26T18:54:36.879 INFO:tasks.workunit.client.0.smithi102.stdout:9/330: mkdir df/d1a/d1d/d72/d7d 0 2024-03-26T18:54:36.879 INFO:tasks.workunit.client.0.smithi102.stdout:9/331: fdatasync df/d1a/f1c 0 2024-03-26T18:54:36.879 INFO:tasks.workunit.client.0.smithi102.stdout:9/332: creat df/d1a/d23/f7e x:0 0 0 2024-03-26T18:54:36.880 INFO:tasks.workunit.client.1.smithi190.stdout:8/773: rmdir d9/dd/d23/d36/d37/d3a/d61 39 2024-03-26T18:54:36.880 INFO:tasks.workunit.client.1.smithi190.stdout:8/774: write d9/d39/d67/fa3 [5073295,81139] 0 2024-03-26T18:54:36.880 INFO:tasks.workunit.client.1.smithi190.stdout:8/775: chown d9/d39/d9e/ceb 189 1 2024-03-26T18:54:36.880 INFO:tasks.workunit.client.1.smithi190.stdout:8/776: write d9/d39/d79/f7e [3299576,106855] 0 2024-03-26T18:54:36.880 INFO:tasks.workunit.client.1.smithi190.stdout:0/765: mkdir d0/dd/d20/d1c/d8b/d8d/db7/d118 0 2024-03-26T18:54:36.881 INFO:tasks.workunit.client.1.smithi190.stdout:8/777: creat d9/dd/d23/d36/d37/d3a/d43/dc2/f107 x:0 0 0 2024-03-26T18:54:36.886 INFO:tasks.workunit.client.1.smithi190.stdout:0/766: mknod d0/dd5/d106/c119 0 2024-03-26T18:54:36.886 INFO:tasks.workunit.client.1.smithi190.stdout:8/778: dread d9/da2/d73/f81 [0,4194304] 0 2024-03-26T18:54:36.886 INFO:tasks.workunit.client.1.smithi190.stdout:8/779: mkdir d9/da2/d73/d108 0 2024-03-26T18:54:36.886 INFO:tasks.workunit.client.0.smithi102.stdout:6/363: mkdir d2/df/d71 0 2024-03-26T18:54:36.889 INFO:tasks.workunit.client.1.smithi190.stdout:0/767: creat d0/dd/d20/d1c/d8b/d8d/db7/d118/f11a x:0 0 0 2024-03-26T18:54:36.905 INFO:tasks.workunit.client.0.smithi102.stdout:0/412: sync 2024-03-26T18:54:36.906 INFO:tasks.workunit.client.0.smithi102.stdout:9/333: mkdir df/d1a/d7f 0 2024-03-26T18:54:36.906 INFO:tasks.workunit.client.0.smithi102.stdout:9/334: fdatasync df/d1a/d23/d4e/f7c 0 2024-03-26T18:54:36.906 INFO:tasks.workunit.client.0.smithi102.stdout:9/335: stat df/d1a/f6f 0 2024-03-26T18:54:36.908 INFO:tasks.workunit.client.0.smithi102.stdout:4/297: symlink d1/d25/d39/l62 0 2024-03-26T18:54:36.908 INFO:tasks.workunit.client.0.smithi102.stdout:4/298: chown d1/d25/c37 1270543067 1 2024-03-26T18:54:36.909 INFO:tasks.workunit.client.0.smithi102.stdout:9/336: symlink df/d1a/d23/d60/l80 0 2024-03-26T18:54:36.909 INFO:tasks.workunit.client.0.smithi102.stdout:9/337: truncate df/d1a/d23/d29/d45/f56 992312 0 2024-03-26T18:54:36.909 INFO:tasks.workunit.client.0.smithi102.stdout:9/338: write df/d1a/d1d/f26 [958810,68686] 0 2024-03-26T18:54:36.910 INFO:tasks.workunit.client.0.smithi102.stdout:9/339: write df/d1a/f6f [412999,49611] 0 2024-03-26T18:54:36.910 INFO:tasks.workunit.client.0.smithi102.stdout:9/340: stat df/d1a/d23/d4e 0 2024-03-26T18:54:36.910 INFO:tasks.workunit.client.0.smithi102.stdout:9/341: fsync df/d1a/d23/d3e/f6b 0 2024-03-26T18:54:36.910 INFO:tasks.workunit.client.0.smithi102.stdout:6/364: unlink d2/d7/l4a 0 2024-03-26T18:54:36.911 INFO:tasks.workunit.client.1.smithi190.stdout:8/780: dwrite d9/d39/d9e/fc5 [0,4194304] 0 2024-03-26T18:54:36.912 INFO:tasks.workunit.client.1.smithi190.stdout:8/781: dread - d9/d39/f82 zero size 2024-03-26T18:54:36.912 INFO:tasks.workunit.client.0.smithi102.stdout:8/424: rename d0/d3c/d66 to d0/d3/d2c/d33/d87 0 2024-03-26T18:54:36.913 INFO:tasks.workunit.client.0.smithi102.stdout:9/342: mkdir df/d1a/d81 0 2024-03-26T18:54:36.914 INFO:tasks.workunit.client.1.smithi190.stdout:0/768: dwrite d0/dd/d20/d1c/d46/d58/de4/f103 [4194304,4194304] 0 2024-03-26T18:54:36.914 INFO:tasks.workunit.client.1.smithi190.stdout:8/782: mkdir d9/da2/d87/d109 0 2024-03-26T18:54:36.914 INFO:tasks.workunit.client.1.smithi190.stdout:8/783: readlink d9/d39/d79/ldf 0 2024-03-26T18:54:36.914 INFO:tasks.workunit.client.1.smithi190.stdout:8/784: mkdir d9/da2/d73/d10a 0 2024-03-26T18:54:36.915 INFO:tasks.workunit.client.0.smithi102.stdout:6/365: mkdir d2/df/d72 0 2024-03-26T18:54:36.915 INFO:tasks.workunit.client.0.smithi102.stdout:6/366: getdents d2/df/d35/d60 0 2024-03-26T18:54:36.917 INFO:tasks.workunit.client.0.smithi102.stdout:8/425: creat d0/d3/d26/d5c/f88 x:0 0 0 2024-03-26T18:54:36.919 INFO:tasks.workunit.client.0.smithi102.stdout:9/343: creat df/f82 x:0 0 0 2024-03-26T18:54:36.920 INFO:tasks.workunit.client.0.smithi102.stdout:6/367: symlink d2/d3d/l73 0 2024-03-26T18:54:36.920 INFO:tasks.workunit.client.0.smithi102.stdout:6/368: fdatasync d2/f47 0 2024-03-26T18:54:36.920 INFO:tasks.workunit.client.0.smithi102.stdout:6/369: write d2/d3d/d53/f5b [46954,50972] 0 2024-03-26T18:54:36.920 INFO:tasks.workunit.client.0.smithi102.stdout:6/370: dread - d2/d3d/d53/f69 zero size 2024-03-26T18:54:36.920 INFO:tasks.workunit.client.0.smithi102.stdout:6/371: chown d2/d3 6731 1 2024-03-26T18:54:36.921 INFO:tasks.workunit.client.0.smithi102.stdout:8/426: mkdir d0/d15/d5d/d89 0 2024-03-26T18:54:36.924 INFO:tasks.workunit.client.0.smithi102.stdout:9/344: mknod df/d1a/d1d/d72/c83 0 2024-03-26T18:54:36.924 INFO:tasks.workunit.client.1.smithi190.stdout:8/785: write d9/da2/d87/fe1 [1870754,58366] 0 2024-03-26T18:54:36.926 INFO:tasks.workunit.client.1.smithi190.stdout:8/786: rename d9/da9 to d9/dd/d23/d10b 0 2024-03-26T18:54:36.927 INFO:tasks.workunit.client.0.smithi102.stdout:6/372: truncate d2/df/f2b 79664 0 2024-03-26T18:54:36.927 INFO:tasks.workunit.client.0.smithi102.stdout:6/373: readlink d2/df/l18 0 2024-03-26T18:54:36.927 INFO:tasks.workunit.client.0.smithi102.stdout:6/374: creat d2/d3d/d53/f74 x:0 0 0 2024-03-26T18:54:36.928 INFO:tasks.workunit.client.0.smithi102.stdout:0/413: rename d6/d3e/d54 to d6/de/d13/d4a/d85 0 2024-03-26T18:54:36.930 INFO:tasks.workunit.client.1.smithi190.stdout:8/787: mkdir d9/d48/d10c 0 2024-03-26T18:54:36.930 INFO:tasks.workunit.client.1.smithi190.stdout:8/788: readlink d9/dd/d23/d36/d37/db8/ldc 0 2024-03-26T18:54:36.931 INFO:tasks.workunit.client.0.smithi102.stdout:6/375: mkdir d2/df/d35/d43/d75 0 2024-03-26T18:54:36.931 INFO:tasks.workunit.client.0.smithi102.stdout:6/376: creat d2/d3d/d53/f76 x:0 0 0 2024-03-26T18:54:36.931 INFO:tasks.workunit.client.0.smithi102.stdout:6/377: write d2/df/f27 [1023157,125151] 0 2024-03-26T18:54:36.931 INFO:tasks.workunit.client.0.smithi102.stdout:6/378: fdatasync d2/f3a 0 2024-03-26T18:54:36.931 INFO:tasks.workunit.client.0.smithi102.stdout:6/379: fdatasync d2/df/f2b 0 2024-03-26T18:54:36.931 INFO:tasks.workunit.client.1.smithi190.stdout:8/789: rename d9/d39/d79/fe9 to d9/dd/d23/d36/d37/d3a/d43/d8f/dd2/f10d 0 2024-03-26T18:54:36.932 INFO:tasks.workunit.client.0.smithi102.stdout:8/427: read d0/d3/d2c/d2f/f78 [3923790,67051] 0 2024-03-26T18:54:36.932 INFO:tasks.workunit.client.0.smithi102.stdout:4/299: rename d1/f60 to d1/d17/d1a/d21/f63 0 2024-03-26T18:54:36.935 INFO:tasks.workunit.client.0.smithi102.stdout:0/414: rmdir d6/de/d13/d25/d31/d45/d82/d7c 0 2024-03-26T18:54:36.939 INFO:tasks.workunit.client.0.smithi102.stdout:6/380: mkdir d2/d3d/d53/d77 0 2024-03-26T18:54:36.939 INFO:tasks.workunit.client.0.smithi102.stdout:6/381: fsync d2/df/d35/d43/f61 0 2024-03-26T18:54:36.939 INFO:tasks.workunit.client.0.smithi102.stdout:6/382: chown d2/d12 0 1 2024-03-26T18:54:36.940 INFO:tasks.workunit.client.0.smithi102.stdout:4/300: link d1/f26 d1/d17/d1a/d21/f64 0 2024-03-26T18:54:36.942 INFO:tasks.workunit.client.0.smithi102.stdout:0/415: link d6/de/d13/d1f/l7a d6/de/d13/d4a/l86 0 2024-03-26T18:54:36.943 INFO:tasks.workunit.client.0.smithi102.stdout:0/416: truncate d6/de/d13/d25/d4f/d66/f6e 205303 0 2024-03-26T18:54:36.945 INFO:tasks.workunit.client.0.smithi102.stdout:4/301: unlink d1/l15 0 2024-03-26T18:54:36.945 INFO:tasks.workunit.client.0.smithi102.stdout:4/302: write d1/d27/f3c [4277658,110326] 0 2024-03-26T18:54:36.947 INFO:tasks.workunit.client.0.smithi102.stdout:9/345: rename df/d46/d63 to df/d84 0 2024-03-26T18:54:36.949 INFO:tasks.workunit.client.0.smithi102.stdout:6/383: mkdir d2/d7/d3c/d78 0 2024-03-26T18:54:36.953 INFO:tasks.workunit.client.0.smithi102.stdout:4/303: rename d1/c30 to d1/d25/d39/d3e/c65 0 2024-03-26T18:54:36.956 INFO:tasks.workunit.client.0.smithi102.stdout:3/573: sync 2024-03-26T18:54:36.956 INFO:tasks.workunit.client.0.smithi102.stdout:3/574: creat df/d29/d31/d77/d8a/fd2 x:0 0 0 2024-03-26T18:54:36.958 INFO:tasks.workunit.client.0.smithi102.stdout:4/304: creat d1/d25/d39/f66 x:0 0 0 2024-03-26T18:54:36.960 INFO:tasks.workunit.client.0.smithi102.stdout:0/417: dread d6/f1a [4194304,4194304] 0 2024-03-26T18:54:36.960 INFO:tasks.workunit.client.0.smithi102.stdout:8/428: dwrite d0/d3/d2c/d4e/f52 [0,4194304] 0 2024-03-26T18:54:36.961 INFO:tasks.workunit.client.0.smithi102.stdout:3/575: creat df/d1c/d34/da5/fd3 x:0 0 0 2024-03-26T18:54:36.964 INFO:tasks.workunit.client.0.smithi102.stdout:0/418: rename d6/de/d13/d1f/d39 to d6/d3e/d87 0 2024-03-26T18:54:36.968 INFO:tasks.workunit.client.0.smithi102.stdout:9/346: dwrite df/d1a/d23/f71 [0,4194304] 0 2024-03-26T18:54:36.971 INFO:tasks.workunit.client.0.smithi102.stdout:3/576: rename df/d8b/db3 to df/d1c/d34/d53/dd4 0 2024-03-26T18:54:36.971 INFO:tasks.workunit.client.0.smithi102.stdout:3/577: fdatasync df/d29/d31/d36/f50 0 2024-03-26T18:54:36.971 INFO:tasks.workunit.client.0.smithi102.stdout:3/578: read - df/d1c/d70/dcd/fcb zero size 2024-03-26T18:54:36.977 INFO:tasks.workunit.client.0.smithi102.stdout:4/305: dread d1/f9 [0,4194304] 0 2024-03-26T18:54:36.977 INFO:tasks.workunit.client.0.smithi102.stdout:4/306: dread - d1/d25/d39/f66 zero size 2024-03-26T18:54:36.978 INFO:tasks.workunit.client.0.smithi102.stdout:3/579: mkdir df/d8b/dd5 0 2024-03-26T18:54:36.981 INFO:tasks.workunit.client.0.smithi102.stdout:9/347: rename df/l15 to df/d1a/d23/d3e/d75/l85 0 2024-03-26T18:54:36.981 INFO:tasks.workunit.client.0.smithi102.stdout:9/348: readlink df/d1a/d23/d29/l41 0 2024-03-26T18:54:36.981 INFO:tasks.workunit.client.0.smithi102.stdout:9/349: creat df/d1a/f86 x:0 0 0 2024-03-26T18:54:36.981 INFO:tasks.workunit.client.0.smithi102.stdout:9/350: write df/d1a/d23/d4e/f53 [677910,46121] 0 2024-03-26T18:54:36.982 INFO:tasks.workunit.client.0.smithi102.stdout:4/307: mknod d1/d27/c67 0 2024-03-26T18:54:36.983 INFO:tasks.workunit.client.0.smithi102.stdout:3/580: read df/d1c/f2a [483792,93258] 0 2024-03-26T18:54:36.983 INFO:tasks.workunit.client.0.smithi102.stdout:3/581: fdatasync df/d8b/fb7 0 2024-03-26T18:54:36.984 INFO:tasks.workunit.client.0.smithi102.stdout:6/384: dwrite d2/d7/f51 [0,4194304] 0 2024-03-26T18:54:36.985 INFO:tasks.workunit.client.0.smithi102.stdout:9/351: mknod df/d1a/d1d/d72/d7d/c87 0 2024-03-26T18:54:36.986 INFO:tasks.workunit.client.0.smithi102.stdout:4/308: rmdir d1/d27 39 2024-03-26T18:54:36.988 INFO:tasks.workunit.client.0.smithi102.stdout:6/385: write d2/d7/f46 [4484803,114477] 0 2024-03-26T18:54:36.988 INFO:tasks.workunit.client.0.smithi102.stdout:6/386: truncate d2/d3d/d53/f74 446861 0 2024-03-26T18:54:36.989 INFO:tasks.workunit.client.0.smithi102.stdout:9/352: mkdir df/d1a/d81/d88 0 2024-03-26T18:54:36.990 INFO:tasks.workunit.client.0.smithi102.stdout:4/309: creat d1/d25/d39/d3e/d4c/f68 x:0 0 0 2024-03-26T18:54:36.992 INFO:tasks.workunit.client.0.smithi102.stdout:6/387: unlink d2/df/d35/c4e 0 2024-03-26T18:54:36.993 INFO:tasks.workunit.client.0.smithi102.stdout:9/353: symlink df/d1a/d81/l89 0 2024-03-26T18:54:36.995 INFO:tasks.workunit.client.0.smithi102.stdout:9/354: rename df/d1a/d1d/f76 to df/d1a/d65/f8a 0 2024-03-26T18:54:36.995 INFO:tasks.workunit.client.0.smithi102.stdout:9/355: creat df/d1a/d57/f8b x:0 0 0 2024-03-26T18:54:36.996 INFO:tasks.workunit.client.0.smithi102.stdout:4/310: read d1/f9 [2092005,92647] 0 2024-03-26T18:54:36.997 INFO:tasks.workunit.client.0.smithi102.stdout:9/356: dread df/d1a/d23/d60/f74 [0,4194304] 0 2024-03-26T18:54:36.997 INFO:tasks.workunit.client.0.smithi102.stdout:9/357: chown df/d1a/l3c 173583 1 2024-03-26T18:54:36.997 INFO:tasks.workunit.client.0.smithi102.stdout:9/358: readlink df/d1a/d1d/l37 0 2024-03-26T18:54:36.997 INFO:tasks.workunit.client.0.smithi102.stdout:9/359: dread - df/d1a/d23/d29/f4c zero size 2024-03-26T18:54:37.013 INFO:tasks.workunit.client.0.smithi102.stdout:3/582: dwrite df/d61/d55/f95 [0,4194304] 0 2024-03-26T18:54:37.013 INFO:tasks.workunit.client.0.smithi102.stdout:0/419: dwrite d6/de/f28 [4194304,4194304] 0 2024-03-26T18:54:37.013 INFO:tasks.workunit.client.0.smithi102.stdout:6/388: dread d2/f52 [0,4194304] 0 2024-03-26T18:54:37.014 INFO:tasks.workunit.client.1.smithi190.stdout:2/694: sync 2024-03-26T18:54:37.014 INFO:tasks.workunit.client.1.smithi190.stdout:5/581: sync 2024-03-26T18:54:37.014 INFO:tasks.workunit.client.1.smithi190.stdout:2/695: chown d6/d9/d3a/d3d/d1c/d8d/fab 125 1 2024-03-26T18:54:37.014 INFO:tasks.workunit.client.0.smithi102.stdout:4/311: dread d1/f7 [0,4194304] 0 2024-03-26T18:54:37.015 INFO:tasks.workunit.client.0.smithi102.stdout:8/429: dwrite d0/d15/f6b [0,4194304] 0 2024-03-26T18:54:37.015 INFO:tasks.workunit.client.0.smithi102.stdout:4/312: fdatasync d1/d17/d1a/f57 0 2024-03-26T18:54:37.015 INFO:tasks.workunit.client.1.smithi190.stdout:2/696: unlink d6/d9/d3a/d3d/f22 0 2024-03-26T18:54:37.015 INFO:tasks.workunit.client.1.smithi190.stdout:2/697: fdatasync d6/d9/d10/f87 0 2024-03-26T18:54:37.015 INFO:tasks.workunit.client.1.smithi190.stdout:2/698: chown d6/d9/d10/f13 7277 1 2024-03-26T18:54:37.015 INFO:tasks.workunit.client.1.smithi190.stdout:2/699: truncate d6/d9/f75 458684 0 2024-03-26T18:54:37.016 INFO:tasks.workunit.client.1.smithi190.stdout:7/743: sync 2024-03-26T18:54:37.016 INFO:tasks.workunit.client.1.smithi190.stdout:7/744: chown d3/d6/d9/f40 6591792 1 2024-03-26T18:54:37.016 INFO:tasks.workunit.client.1.smithi190.stdout:1/624: sync 2024-03-26T18:54:37.016 INFO:tasks.workunit.client.1.smithi190.stdout:1/625: write d3/da/f47 [4576147,31504] 0 2024-03-26T18:54:37.016 INFO:tasks.workunit.client.1.smithi190.stdout:2/700: mknod d6/d9/d3a/d3d/d67/cf9 0 2024-03-26T18:54:37.017 INFO:tasks.workunit.client.1.smithi190.stdout:2/701: mkdir d6/d9/d45/df3/dfa 0 2024-03-26T18:54:37.018 INFO:tasks.workunit.client.1.smithi190.stdout:2/702: read - d6/d9/d3a/d3d/d55/fc6 zero size 2024-03-26T18:54:37.019 INFO:tasks.workunit.client.1.smithi190.stdout:2/703: unlink d6/f3c 0 2024-03-26T18:54:37.019 INFO:tasks.workunit.client.1.smithi190.stdout:2/704: write d6/d9/d3a/d3d/d2e/f6e [495052,109660] 0 2024-03-26T18:54:37.020 INFO:tasks.workunit.client.1.smithi190.stdout:5/582: symlink d2/d17/d98/d3a/d3c/lc3 0 2024-03-26T18:54:37.023 INFO:tasks.workunit.client.0.smithi102.stdout:3/583: truncate fd 1965256 0 2024-03-26T18:54:37.025 INFO:tasks.workunit.client.0.smithi102.stdout:6/389: symlink d2/d3d/d53/d63/d6a/l79 0 2024-03-26T18:54:37.025 INFO:tasks.workunit.client.0.smithi102.stdout:6/390: chown d2/df/d35/f6e 74156431 1 2024-03-26T18:54:37.025 INFO:tasks.workunit.client.1.smithi190.stdout:1/626: truncate d3/da/d2e/d59/dae/fcb 3862062 0 2024-03-26T18:54:37.025 INFO:tasks.workunit.client.1.smithi190.stdout:5/583: dread d2/d17/f21 [0,4194304] 0 2024-03-26T18:54:37.025 INFO:tasks.workunit.client.1.smithi190.stdout:5/584: chown d2/d17/d1b/db6/d24/d3b/l4b 52202794 1 2024-03-26T18:54:37.026 INFO:tasks.workunit.client.0.smithi102.stdout:6/391: fsync d2/d7/d3c/f3e 0 2024-03-26T18:54:37.028 INFO:tasks.workunit.client.0.smithi102.stdout:4/313: dread d1/d17/d1a/f4b [0,4194304] 0 2024-03-26T18:54:37.028 INFO:tasks.workunit.client.0.smithi102.stdout:4/314: chown d1/d17/d2a/f54 5806 1 2024-03-26T18:54:37.028 INFO:tasks.workunit.client.0.smithi102.stdout:4/315: dread - d1/d17/d2a/f54 zero size 2024-03-26T18:54:37.028 INFO:tasks.workunit.client.0.smithi102.stdout:4/316: stat d1/f9 0 2024-03-26T18:54:37.028 INFO:tasks.workunit.client.1.smithi190.stdout:7/745: read d3/f16 [1496773,80171] 0 2024-03-26T18:54:37.029 INFO:tasks.workunit.client.1.smithi190.stdout:1/627: mknod d3/da/daf/ccd 0 2024-03-26T18:54:37.030 INFO:tasks.workunit.client.1.smithi190.stdout:5/585: link d2/d17/d1b/l5c d2/d17/d62/lc4 0 2024-03-26T18:54:37.030 INFO:tasks.workunit.client.1.smithi190.stdout:5/586: read d2/d17/f21 [324251,14187] 0 2024-03-26T18:54:37.030 INFO:tasks.workunit.client.1.smithi190.stdout:5/587: write d2/d17/d1b/db6/d24/d3b/f56 [4355263,53739] 0 2024-03-26T18:54:37.030 INFO:tasks.workunit.client.1.smithi190.stdout:5/588: fsync d2/d17/d98/d3a/f55 0 2024-03-26T18:54:37.030 INFO:tasks.workunit.client.1.smithi190.stdout:7/746: mkdir d3/d1b/dd3/d10d/d50/d117 0 2024-03-26T18:54:37.030 INFO:tasks.workunit.client.1.smithi190.stdout:7/747: chown d3/d6/d9/d1f/d4c/d71/ce3 4103 1 2024-03-26T18:54:37.031 INFO:tasks.workunit.client.1.smithi190.stdout:7/748: truncate d3/d1b/dd3/d10d/fb7 1705554 0 2024-03-26T18:54:37.031 INFO:tasks.workunit.client.1.smithi190.stdout:7/749: chown d3/d6/d9/d14/l38 1 1 2024-03-26T18:54:37.033 INFO:tasks.workunit.client.1.smithi190.stdout:7/750: creat d3/d1b/dd3/d10d/d50/f118 x:0 0 0 2024-03-26T18:54:37.033 INFO:tasks.workunit.client.1.smithi190.stdout:5/589: creat d2/d17/d1b/d60/fc5 x:0 0 0 2024-03-26T18:54:37.033 INFO:tasks.workunit.client.1.smithi190.stdout:5/590: chown d2/d17/d1b/db6/d96 1695 1 2024-03-26T18:54:37.034 INFO:tasks.workunit.client.1.smithi190.stdout:7/751: mkdir d3/d1b/dd3/d10d/d65/d109/d119 0 2024-03-26T18:54:37.035 INFO:tasks.workunit.client.0.smithi102.stdout:3/584: mkdir df/d61/dd6 0 2024-03-26T18:54:37.035 INFO:tasks.workunit.client.0.smithi102.stdout:9/360: dwrite df/d1a/f52 [0,4194304] 0 2024-03-26T18:54:37.035 INFO:tasks.workunit.client.0.smithi102.stdout:9/361: chown df/d1a/l44 422 1 2024-03-26T18:54:37.035 INFO:tasks.workunit.client.0.smithi102.stdout:9/362: rename df/d1a/d23/d60 to df/d1a/d23/d60/d8c 22 2024-03-26T18:54:37.036 INFO:tasks.workunit.client.1.smithi190.stdout:7/752: mknod d3/d6/d9/d1f/d93/c11a 0 2024-03-26T18:54:37.036 INFO:tasks.workunit.client.0.smithi102.stdout:9/363: fdatasync df/f30 0 2024-03-26T18:54:37.037 INFO:tasks.workunit.client.1.smithi190.stdout:7/753: mkdir d3/d1b/dd3/d10d/d65/d109/d11b 0 2024-03-26T18:54:37.038 INFO:tasks.workunit.client.1.smithi190.stdout:1/628: rmdir d3/da/db2 0 2024-03-26T18:54:37.040 INFO:tasks.workunit.client.1.smithi190.stdout:7/754: rename d3/d1b/dd3/f111 to d3/d1b/dd3/d10d/d65/d90/d92/d9b/f11c 0 2024-03-26T18:54:37.043 INFO:tasks.workunit.client.0.smithi102.stdout:3/585: getdents df/d29/d3a 0 2024-03-26T18:54:37.043 INFO:tasks.workunit.client.0.smithi102.stdout:3/586: write df/d1c/f2a [256340,48892] 0 2024-03-26T18:54:37.043 INFO:tasks.workunit.client.0.smithi102.stdout:3/587: chown df/d29/d31/d77/d8a/fbd 3 1 2024-03-26T18:54:37.045 INFO:tasks.workunit.client.1.smithi190.stdout:1/629: rename d3/da/dd/d2f/d34/c98 to d3/cce 0 2024-03-26T18:54:37.048 INFO:tasks.workunit.client.0.smithi102.stdout:3/588: creat df/d29/d3a/fd7 x:0 0 0 2024-03-26T18:54:37.048 INFO:tasks.workunit.client.0.smithi102.stdout:3/589: write df/d1c/d34/f74 [702033,40683] 0 2024-03-26T18:54:37.050 INFO:tasks.workunit.client.1.smithi190.stdout:1/630: mknod d3/d73/ccf 0 2024-03-26T18:54:37.052 INFO:tasks.workunit.client.1.smithi190.stdout:1/631: symlink d3/da/d3a/d99/ld0 0 2024-03-26T18:54:37.053 INFO:tasks.workunit.client.1.smithi190.stdout:1/632: rename d3/d5f/fbb to d3/da/dd/fd1 0 2024-03-26T18:54:37.055 INFO:tasks.workunit.client.0.smithi102.stdout:7/379: sync 2024-03-26T18:54:37.055 INFO:tasks.workunit.client.0.smithi102.stdout:1/360: sync 2024-03-26T18:54:37.055 INFO:tasks.workunit.client.0.smithi102.stdout:7/380: chown d7/d10/d17/l19 891 1 2024-03-26T18:54:37.056 INFO:tasks.workunit.client.0.smithi102.stdout:7/381: dread d7/d10/d27/f55 [0,4194304] 0 2024-03-26T18:54:37.057 INFO:tasks.workunit.client.0.smithi102.stdout:5/618: sync 2024-03-26T18:54:37.057 INFO:tasks.workunit.client.0.smithi102.stdout:2/339: sync 2024-03-26T18:54:37.057 INFO:tasks.workunit.client.0.smithi102.stdout:5/619: write d2/d2c/f4c [924171,97479] 0 2024-03-26T18:54:37.058 INFO:tasks.workunit.client.0.smithi102.stdout:3/590: dread df/d29/d31/d36/d45/daf/fb8 [0,4194304] 0 2024-03-26T18:54:37.058 INFO:tasks.workunit.client.1.smithi190.stdout:1/633: mknod d3/da/dd/d2f/d34/cd2 0 2024-03-26T18:54:37.058 INFO:tasks.workunit.client.1.smithi190.stdout:1/634: fsync d3/da/dd/d2f/d5b/f63 0 2024-03-26T18:54:37.058 INFO:tasks.workunit.client.1.smithi190.stdout:1/635: write d3/da/d45/f49 [335143,81920] 0 2024-03-26T18:54:37.059 INFO:tasks.workunit.client.0.smithi102.stdout:2/340: creat d3/f75 x:0 0 0 2024-03-26T18:54:37.060 INFO:tasks.workunit.client.1.smithi190.stdout:1/636: mknod d3/da/dd/d2f/d5b/d83/cd3 0 2024-03-26T18:54:37.064 INFO:tasks.workunit.client.0.smithi102.stdout:3/591: creat df/d29/d31/d36/d45/fd8 x:0 0 0 2024-03-26T18:54:37.064 INFO:tasks.workunit.client.0.smithi102.stdout:3/592: write df/d29/d31/d77/d8a/dae/fc5 [969676,117849] 0 2024-03-26T18:54:37.066 INFO:tasks.workunit.client.1.smithi190.stdout:1/637: mkdir d3/d9/db/dd4 0 2024-03-26T18:54:37.066 INFO:tasks.workunit.client.1.smithi190.stdout:1/638: write d3/da/dd/d2f/d34/f35 [4308834,20509] 0 2024-03-26T18:54:37.066 INFO:tasks.workunit.client.0.smithi102.stdout:3/593: truncate df/d29/f2c 2745251 0 2024-03-26T18:54:37.068 INFO:tasks.workunit.client.0.smithi102.stdout:7/382: write d7/d12/d4d/f5e [2001311,27866] 0 2024-03-26T18:54:37.069 INFO:tasks.workunit.client.1.smithi190.stdout:1/639: getdents d3/da/d45 0 2024-03-26T18:54:37.069 INFO:tasks.workunit.client.1.smithi190.stdout:1/640: write d3/da/f78 [1001334,126746] 0 2024-03-26T18:54:37.071 INFO:tasks.workunit.client.0.smithi102.stdout:7/383: symlink d7/d12/d4d/l6f 0 2024-03-26T18:54:37.073 INFO:tasks.workunit.client.1.smithi190.stdout:1/641: truncate f2 6019888 0 2024-03-26T18:54:37.073 INFO:tasks.workunit.client.0.smithi102.stdout:7/384: rename d7/d10/d27/c6c to d7/d12/d2b/c70 0 2024-03-26T18:54:37.073 INFO:tasks.workunit.client.0.smithi102.stdout:1/361: dread de/d16/d1d/f1e [0,4194304] 0 2024-03-26T18:54:37.074 INFO:tasks.workunit.client.1.smithi190.stdout:5/591: write d2/d17/d1b/db6/d24/f84 [2960676,16406] 0 2024-03-26T18:54:37.074 INFO:tasks.workunit.client.1.smithi190.stdout:5/592: dread - d2/d17/d98/d3a/d3c/f6a zero size 2024-03-26T18:54:37.074 INFO:tasks.workunit.client.0.smithi102.stdout:1/362: unlink de/d2e/c68 0 2024-03-26T18:54:37.076 INFO:tasks.workunit.client.1.smithi190.stdout:5/593: link d2/d17/f4f d2/d17/d1b/db6/fc6 0 2024-03-26T18:54:37.076 INFO:tasks.workunit.client.1.smithi190.stdout:5/594: write d2/d17/d98/d3a/f6b [966575,62875] 0 2024-03-26T18:54:37.076 INFO:tasks.workunit.client.1.smithi190.stdout:5/595: truncate d2/d17/d1b/fb7 372729 0 2024-03-26T18:54:37.077 INFO:tasks.workunit.client.0.smithi102.stdout:7/385: mknod d7/c71 0 2024-03-26T18:54:37.077 INFO:tasks.workunit.client.0.smithi102.stdout:0/420: dwrite d6/de/d13/d25/d31/d45/f65 [0,4194304] 0 2024-03-26T18:54:37.078 INFO:tasks.workunit.client.0.smithi102.stdout:1/363: rename de/d2e/l4e to de/d2e/l7b 0 2024-03-26T18:54:37.078 INFO:tasks.workunit.client.1.smithi190.stdout:5/596: write d2/d17/d1b/db6/f38 [434475,11816] 0 2024-03-26T18:54:37.078 INFO:tasks.workunit.client.0.smithi102.stdout:7/386: getdents d7/d4e 0 2024-03-26T18:54:37.081 INFO:tasks.workunit.client.1.smithi190.stdout:5/597: link d2/d17/d1b/d30/f73 d2/d17/d1b/db6/fc7 0 2024-03-26T18:54:37.081 INFO:tasks.workunit.client.1.smithi190.stdout:5/598: dread - d2/d17/d98/d3a/d3c/f6a zero size 2024-03-26T18:54:37.082 INFO:tasks.workunit.client.0.smithi102.stdout:0/421: truncate d6/de/d13/d1f/d55/d57/f1c 4670387 0 2024-03-26T18:54:37.085 INFO:tasks.workunit.client.1.smithi190.stdout:5/599: unlink d2/d17/d1b/l20 0 2024-03-26T18:54:37.086 INFO:tasks.workunit.client.0.smithi102.stdout:0/422: rmdir d6 39 2024-03-26T18:54:37.087 INFO:tasks.workunit.client.0.smithi102.stdout:0/423: mkdir d6/de/d13/d25/d31/d45/d82/d84/d88 0 2024-03-26T18:54:37.088 INFO:tasks.workunit.client.1.smithi190.stdout:5/600: dread d2/d17/d1b/d60/d63/f6e [0,4194304] 0 2024-03-26T18:54:37.088 INFO:tasks.workunit.client.1.smithi190.stdout:5/601: chown d2/l18 4 1 2024-03-26T18:54:37.089 INFO:tasks.workunit.client.1.smithi190.stdout:5/602: write d2/d17/d62/f7e [352395,9401] 0 2024-03-26T18:54:37.089 INFO:tasks.workunit.client.1.smithi190.stdout:5/603: creat d2/d17/d1b/fc8 x:0 0 0 2024-03-26T18:54:37.089 INFO:tasks.workunit.client.1.smithi190.stdout:5/604: readlink d2/d17/d1b/db6/l32 0 2024-03-26T18:54:37.089 INFO:tasks.workunit.client.0.smithi102.stdout:0/424: rename d6/d3e/l75 to d6/d3e/l89 0 2024-03-26T18:54:37.090 INFO:tasks.workunit.client.0.smithi102.stdout:0/425: creat d6/de/d13/d25/d31/d45/d82/f8a x:0 0 0 2024-03-26T18:54:37.092 INFO:tasks.workunit.client.1.smithi190.stdout:5/605: mkdir d2/d17/d1b/db6/d88/dc9 0 2024-03-26T18:54:37.092 INFO:tasks.workunit.client.1.smithi190.stdout:5/606: chown d2/d17/d1b/l5c 102756 1 2024-03-26T18:54:37.098 INFO:tasks.workunit.client.0.smithi102.stdout:3/594: dread df/d1c/d34/f8e [0,4194304] 0 2024-03-26T18:54:37.111 INFO:tasks.workunit.client.0.smithi102.stdout:8/430: dwrite d0/d15/d5d/f61 [0,4194304] 0 2024-03-26T18:54:37.112 INFO:tasks.workunit.client.0.smithi102.stdout:0/426: dread d6/de/d13/d1f/d55/d57/f5e [0,4194304] 0 2024-03-26T18:54:37.113 INFO:tasks.workunit.client.1.smithi190.stdout:7/755: dwrite d3/d6/d9/d14/d36/f10a [0,4194304] 0 2024-03-26T18:54:37.114 INFO:tasks.workunit.client.0.smithi102.stdout:6/392: dwrite d2/d7/f30 [0,4194304] 0 2024-03-26T18:54:37.114 INFO:tasks.workunit.client.0.smithi102.stdout:4/317: dwrite d1/d27/f55 [0,4194304] 0 2024-03-26T18:54:37.114 INFO:tasks.workunit.client.1.smithi190.stdout:1/642: dwrite d3/da/dd/d95/f8e [0,4194304] 0 2024-03-26T18:54:37.114 INFO:tasks.workunit.client.1.smithi190.stdout:1/643: write d3/d9/f3c [793116,106476] 0 2024-03-26T18:54:37.115 INFO:tasks.workunit.client.0.smithi102.stdout:9/364: dwrite df/d1a/d23/d29/f31 [0,4194304] 0 2024-03-26T18:54:37.115 INFO:tasks.workunit.client.0.smithi102.stdout:8/431: symlink d0/d3/d31/l8a 0 2024-03-26T18:54:37.115 INFO:tasks.workunit.client.0.smithi102.stdout:8/432: chown d0/d3/l7 6 1 2024-03-26T18:54:37.115 INFO:tasks.workunit.client.0.smithi102.stdout:8/433: write d0/d15/f4b [1256807,89468] 0 2024-03-26T18:54:37.115 INFO:tasks.workunit.client.0.smithi102.stdout:7/387: dwrite d7/d10/d17/f35 [8388608,4194304] 0 2024-03-26T18:54:37.116 INFO:tasks.workunit.client.0.smithi102.stdout:0/427: creat d6/de/d13/d1f/d55/d57/d7d/d7f/f8b x:0 0 0 2024-03-26T18:54:37.116 INFO:tasks.workunit.client.0.smithi102.stdout:0/428: write d6/de/d13/f7e [2996127,96695] 0 2024-03-26T18:54:37.116 INFO:tasks.workunit.client.1.smithi190.stdout:7/756: dread d3/d6/d9/d14/d36/f48 [0,4194304] 0 2024-03-26T18:54:37.116 INFO:tasks.workunit.client.1.smithi190.stdout:7/757: write d3/d6/d9/d14/f10c [361675,46014] 0 2024-03-26T18:54:37.116 INFO:tasks.workunit.client.1.smithi190.stdout:7/758: readlink d3/d1b/l64 0 2024-03-26T18:54:37.117 INFO:tasks.workunit.client.0.smithi102.stdout:1/364: dwrite de/d16/d1d/f1e [0,4194304] 0 2024-03-26T18:54:37.117 INFO:tasks.workunit.client.0.smithi102.stdout:1/365: truncate de/d16/f41 723281 0 2024-03-26T18:54:37.117 INFO:tasks.workunit.client.0.smithi102.stdout:2/341: dwrite d3/d6/fe [0,4194304] 0 2024-03-26T18:54:37.121 INFO:tasks.workunit.client.0.smithi102.stdout:6/393: write d2/d7/f36 [4979920,39300] 0 2024-03-26T18:54:37.126 INFO:tasks.workunit.client.0.smithi102.stdout:3/595: dwrite df/d1c/d70/fb6 [0,4194304] 0 2024-03-26T18:54:37.126 INFO:tasks.workunit.client.0.smithi102.stdout:3/596: dread - df/d1c/d70/fa6 zero size 2024-03-26T18:54:37.128 INFO:tasks.workunit.client.1.smithi190.stdout:7/759: write d3/d1b/dd3/d10d/d50/f52 [4006897,3204] 0 2024-03-26T18:54:37.129 INFO:tasks.workunit.client.1.smithi190.stdout:5/607: dread d2/d17/d98/d3a/f55 [0,4194304] 0 2024-03-26T18:54:37.129 INFO:tasks.workunit.client.1.smithi190.stdout:5/608: write d2/d17/f4f [1585779,118172] 0 2024-03-26T18:54:37.129 INFO:tasks.workunit.client.1.smithi190.stdout:5/609: fsync d2/d17/d1b/db6/d24/f84 0 2024-03-26T18:54:37.129 INFO:tasks.workunit.client.0.smithi102.stdout:4/318: creat d1/d25/f69 x:0 0 0 2024-03-26T18:54:37.130 INFO:tasks.workunit.client.0.smithi102.stdout:4/319: creat d1/d25/d39/d3e/d4c/f6a x:0 0 0 2024-03-26T18:54:37.130 INFO:tasks.workunit.client.0.smithi102.stdout:4/320: write d1/d25/d39/d3e/d4c/f68 [150755,58989] 0 2024-03-26T18:54:37.131 INFO:tasks.workunit.client.1.smithi190.stdout:7/760: mkdir d3/d1b/d57/dd1/d11d 0 2024-03-26T18:54:37.133 INFO:tasks.workunit.client.0.smithi102.stdout:9/365: creat df/d1a/d23/d3e/d75/f8d x:0 0 0 2024-03-26T18:54:37.133 INFO:tasks.workunit.client.1.smithi190.stdout:7/761: link d3/lef d3/d6/d9/d1f/d4c/d71/dcc/l11e 0 2024-03-26T18:54:37.135 INFO:tasks.workunit.client.0.smithi102.stdout:8/434: dwrite d0/d3c/f43 [0,4194304] 0 2024-03-26T18:54:37.140 INFO:tasks.workunit.client.0.smithi102.stdout:7/388: mkdir d7/d10/d27/d34/d72 0 2024-03-26T18:54:37.140 INFO:tasks.workunit.client.0.smithi102.stdout:7/389: write f6 [1681171,124462] 0 2024-03-26T18:54:37.140 INFO:tasks.workunit.client.0.smithi102.stdout:7/390: fdatasync d7/d12/d51/f58 0 2024-03-26T18:54:37.141 INFO:tasks.workunit.client.1.smithi190.stdout:5/610: rename d2/d17/d1b/db6/f38 to d2/d17/d62/fca 0 2024-03-26T18:54:37.143 INFO:tasks.workunit.client.1.smithi190.stdout:7/762: dread d3/d1b/dd2/f115 [0,4194304] 0 2024-03-26T18:54:37.143 INFO:tasks.workunit.client.1.smithi190.stdout:7/763: read d3/d1b/dd3/d10d/d50/f52 [673106,1730] 0 2024-03-26T18:54:37.144 INFO:tasks.workunit.client.1.smithi190.stdout:7/764: chown d3/d6/d9/le5 131491038 1 2024-03-26T18:54:37.145 INFO:tasks.workunit.client.0.smithi102.stdout:0/429: rename d6/de/d13/d4a/d85/d77 to d6/d3e/d69/d8c 0 2024-03-26T18:54:37.150 INFO:tasks.workunit.client.1.smithi190.stdout:5/611: creat d2/fcb x:0 0 0 2024-03-26T18:54:37.151 INFO:tasks.workunit.client.1.smithi190.stdout:5/612: dread d2/d17/d98/d3a/faa [0,4194304] 0 2024-03-26T18:54:37.153 INFO:tasks.workunit.client.0.smithi102.stdout:2/342: creat d3/d6/d1a/f76 x:0 0 0 2024-03-26T18:54:37.153 INFO:tasks.workunit.client.1.smithi190.stdout:5/613: dread d2/d17/d62/f82 [0,4194304] 0 2024-03-26T18:54:37.156 INFO:tasks.workunit.client.0.smithi102.stdout:6/394: truncate d2/d3/f2a 3239255 0 2024-03-26T18:54:37.163 INFO:tasks.workunit.client.0.smithi102.stdout:3/597: mknod df/d61/d55/cd9 0 2024-03-26T18:54:37.168 INFO:tasks.workunit.client.1.smithi190.stdout:9/943: sync 2024-03-26T18:54:37.169 INFO:tasks.workunit.client.0.smithi102.stdout:2/343: dwrite d3/d2d/f68 [0,4194304] 0 2024-03-26T18:54:37.169 INFO:tasks.workunit.client.0.smithi102.stdout:2/344: stat d3/d2d/c5b 0 2024-03-26T18:54:37.169 INFO:tasks.workunit.client.0.smithi102.stdout:2/345: stat d3/d11/d15/d46/c71 0 2024-03-26T18:54:37.170 INFO:tasks.workunit.client.0.smithi102.stdout:2/346: chown d3/d11/d15/d31/l45 31818907 1 2024-03-26T18:54:37.170 INFO:tasks.workunit.client.0.smithi102.stdout:2/347: write d3/d11/d27/f57 [543407,26235] 0 2024-03-26T18:54:37.170 INFO:tasks.workunit.client.0.smithi102.stdout:2/348: readlink d3/d11/d27/l66 0 2024-03-26T18:54:37.170 INFO:tasks.workunit.client.1.smithi190.stdout:9/944: truncate d12/d2d/fd9 314836 0 2024-03-26T18:54:37.171 INFO:tasks.workunit.client.1.smithi190.stdout:9/945: symlink d12/d2d/d1d/d23/d83/dc9/l14e 0 2024-03-26T18:54:37.172 INFO:tasks.workunit.client.1.smithi190.stdout:9/946: mkdir d12/d2d/d1d/d23/d83/dc9/dd4/d71/d129/d14f 0 2024-03-26T18:54:37.173 INFO:tasks.workunit.client.1.smithi190.stdout:9/947: creat d12/d2d/d1d/d23/d83/dc9/dd4/d71/da7/f150 x:0 0 0 2024-03-26T18:54:37.173 INFO:tasks.workunit.client.1.smithi190.stdout:9/948: symlink d12/d2d/d1d/d54/d92/l151 0 2024-03-26T18:54:37.174 INFO:tasks.workunit.client.1.smithi190.stdout:9/949: creat d12/d2d/d1d/d23/d130/f152 x:0 0 0 2024-03-26T18:54:37.175 INFO:tasks.workunit.client.1.smithi190.stdout:3/588: sync 2024-03-26T18:54:37.176 INFO:tasks.workunit.client.1.smithi190.stdout:3/589: fdatasync d1/d3/d5a/fa9 0 2024-03-26T18:54:37.176 INFO:tasks.workunit.client.1.smithi190.stdout:9/950: link d12/d2d/d1d/d23/d83/dc9/dd4/d3f/cdc d12/d2d/d1d/d23/d2b/c153 0 2024-03-26T18:54:37.176 INFO:tasks.workunit.client.1.smithi190.stdout:5/614: rmdir d2/d17/d1b/d60 39 2024-03-26T18:54:37.178 INFO:tasks.workunit.client.0.smithi102.stdout:8/435: creat d0/d3c/d6f/f8b x:0 0 0 2024-03-26T18:54:37.180 INFO:tasks.workunit.client.1.smithi190.stdout:3/590: unlink d1/d3/df/d18/d57/c7e 0 2024-03-26T18:54:37.180 INFO:tasks.workunit.client.1.smithi190.stdout:3/591: read d1/d3/df/d31/d5b/f6b [1734681,35531] 0 2024-03-26T18:54:37.181 INFO:tasks.workunit.client.1.smithi190.stdout:1/644: write d3/da/d45/f48 [187922,51652] 0 2024-03-26T18:54:37.182 INFO:tasks.workunit.client.0.smithi102.stdout:6/395: dwrite d2/d7/d3c/f48 [0,4194304] 0 2024-03-26T18:54:37.182 INFO:tasks.workunit.client.0.smithi102.stdout:6/396: chown d2/d7 1031648 1 2024-03-26T18:54:37.182 INFO:tasks.workunit.client.0.smithi102.stdout:6/397: readlink d2/d12/l49 0 2024-03-26T18:54:37.182 INFO:tasks.workunit.client.0.smithi102.stdout:6/398: read - d2/df/d35/d43/f57 zero size 2024-03-26T18:54:37.183 INFO:tasks.workunit.client.1.smithi190.stdout:5/615: creat d2/d17/d1b/db6/d24/d3b/da5/fcc x:0 0 0 2024-03-26T18:54:37.183 INFO:tasks.workunit.client.1.smithi190.stdout:5/616: read d2/d17/d1b/d60/d63/f6e [858495,84691] 0 2024-03-26T18:54:37.185 INFO:tasks.workunit.client.0.smithi102.stdout:2/349: write d3/d6/fe [2794435,13571] 0 2024-03-26T18:54:37.186 INFO:tasks.workunit.client.0.smithi102.stdout:6/399: dread d2/f33 [0,4194304] 0 2024-03-26T18:54:37.186 INFO:tasks.workunit.client.0.smithi102.stdout:7/391: link d7/d10/d17/l19 d7/d44/l73 0 2024-03-26T18:54:37.186 INFO:tasks.workunit.client.0.smithi102.stdout:7/392: chown d7/d4e/c52 34 1 2024-03-26T18:54:37.186 INFO:tasks.workunit.client.0.smithi102.stdout:7/393: creat d7/d2a/f74 x:0 0 0 2024-03-26T18:54:37.188 INFO:tasks.workunit.client.0.smithi102.stdout:8/436: dread d0/d3/d2c/d33/f49 [0,4194304] 0 2024-03-26T18:54:37.188 INFO:tasks.workunit.client.1.smithi190.stdout:3/592: symlink d1/d3/df/d18/d1b/lb7 0 2024-03-26T18:54:37.191 INFO:tasks.workunit.client.1.smithi190.stdout:5/617: mkdir d2/d17/d1b/d60/d63/dcd 0 2024-03-26T18:54:37.191 INFO:tasks.workunit.client.1.smithi190.stdout:0/769: sync 2024-03-26T18:54:37.191 INFO:tasks.workunit.client.1.smithi190.stdout:2/705: sync 2024-03-26T18:54:37.191 INFO:tasks.workunit.client.1.smithi190.stdout:6/648: sync 2024-03-26T18:54:37.191 INFO:tasks.workunit.client.1.smithi190.stdout:4/744: sync 2024-03-26T18:54:37.191 INFO:tasks.workunit.client.1.smithi190.stdout:8/790: sync 2024-03-26T18:54:37.192 INFO:tasks.workunit.client.1.smithi190.stdout:8/791: getdents d9/dd/d23/dea/dfe 0 2024-03-26T18:54:37.192 INFO:tasks.workunit.client.1.smithi190.stdout:8/792: chown d9/dd/d23/d88/ld7 63665 1 2024-03-26T18:54:37.192 INFO:tasks.workunit.client.1.smithi190.stdout:8/793: chown d9/dd/d23/d36/d37/d3a/lab 31271 1 2024-03-26T18:54:37.193 INFO:tasks.workunit.client.1.smithi190.stdout:8/794: rename d9/d39/d67/ca7 to d9/d39/c10e 0 2024-03-26T18:54:37.193 INFO:tasks.workunit.client.1.smithi190.stdout:2/706: write d6/d9/d10/f87 [863548,62096] 0 2024-03-26T18:54:37.195 INFO:tasks.workunit.client.1.smithi190.stdout:3/593: symlink d1/d3/df/d18/d35/d41/dae/d86/lb8 0 2024-03-26T18:54:37.201 INFO:tasks.workunit.client.1.smithi190.stdout:0/770: creat d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d75/f11b x:0 0 0 2024-03-26T18:54:37.201 INFO:tasks.workunit.client.1.smithi190.stdout:0/771: chown d0/dd/d20/d1c/d46/da7/lc2 5738109 1 2024-03-26T18:54:37.201 INFO:tasks.workunit.client.1.smithi190.stdout:0/772: stat d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/c33 0 2024-03-26T18:54:37.204 INFO:tasks.workunit.client.1.smithi190.stdout:6/649: creat d17/d28/d80/fd8 x:0 0 0 2024-03-26T18:54:37.207 INFO:tasks.workunit.client.1.smithi190.stdout:3/594: symlink d1/de/lb9 0 2024-03-26T18:54:37.209 INFO:tasks.workunit.client.1.smithi190.stdout:0/773: mkdir d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d66/d11c 0 2024-03-26T18:54:37.210 INFO:tasks.workunit.client.0.smithi102.stdout:1/366: rmdir de/d16/d1d/d2a 39 2024-03-26T18:54:37.212 INFO:tasks.workunit.client.1.smithi190.stdout:2/707: dread d6/d9/d3a/d3d/d2e/f5c [0,4194304] 0 2024-03-26T18:54:37.212 INFO:tasks.workunit.client.1.smithi190.stdout:3/595: creat d1/de/fba x:0 0 0 2024-03-26T18:54:37.213 INFO:tasks.workunit.client.1.smithi190.stdout:8/795: dread d9/d48/f58 [0,4194304] 0 2024-03-26T18:54:37.213 INFO:tasks.workunit.client.0.smithi102.stdout:3/598: creat df/d29/d31/d36/d3c/d6f/fda x:0 0 0 2024-03-26T18:54:37.214 INFO:tasks.workunit.client.1.smithi190.stdout:8/796: symlink d9/da2/dc4/l10f 0 2024-03-26T18:54:37.214 INFO:tasks.workunit.client.1.smithi190.stdout:8/797: write d9/da2/d87/fe7 [4317320,103730] 0 2024-03-26T18:54:37.215 INFO:tasks.workunit.client.1.smithi190.stdout:0/774: creat d0/dd/d20/d1c/d46/da7/df0/d87/f11d x:0 0 0 2024-03-26T18:54:37.216 INFO:tasks.workunit.client.1.smithi190.stdout:0/775: fsync d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d66/f94 0 2024-03-26T18:54:37.216 INFO:tasks.workunit.client.1.smithi190.stdout:0/776: chown d0/dd/d20/d1c/d46/da7/df0/d4b/lb0 0 1 2024-03-26T18:54:37.216 INFO:tasks.workunit.client.1.smithi190.stdout:0/777: write d0/dd/d20/fdf [633865,68351] 0 2024-03-26T18:54:37.217 INFO:tasks.workunit.client.0.smithi102.stdout:4/321: mkdir d1/d6b 0 2024-03-26T18:54:37.217 INFO:tasks.workunit.client.0.smithi102.stdout:4/322: chown d1/d17/d1a 3050 1 2024-03-26T18:54:37.217 INFO:tasks.workunit.client.1.smithi190.stdout:4/745: write d7/d1d/d42/fbe [3895213,57530] 0 2024-03-26T18:54:37.219 INFO:tasks.workunit.client.1.smithi190.stdout:4/746: mknod d7/d8/d14/d23/d28/d38/dd3/dfd/c101 0 2024-03-26T18:54:37.222 INFO:tasks.workunit.client.1.smithi190.stdout:5/618: dwrite d2/f6 [0,4194304] 0 2024-03-26T18:54:37.222 INFO:tasks.workunit.client.1.smithi190.stdout:0/778: unlink d0/dd/ld0 0 2024-03-26T18:54:37.228 INFO:tasks.workunit.client.1.smithi190.stdout:2/708: write d6/d9/d10/d5f/fad [2657748,7805] 0 2024-03-26T18:54:37.228 INFO:tasks.workunit.client.1.smithi190.stdout:2/709: dread - d6/d9/d10/db7/f49 zero size 2024-03-26T18:54:37.228 INFO:tasks.workunit.client.0.smithi102.stdout:6/400: creat d2/df/d72/f7a x:0 0 0 2024-03-26T18:54:37.231 INFO:tasks.workunit.client.0.smithi102.stdout:0/430: dwrite d6/de/d13/d1f/d55/d57/f30 [0,4194304] 0 2024-03-26T18:54:37.233 INFO:tasks.workunit.client.1.smithi190.stdout:2/710: unlink d6/d9/d3a/c46 0 2024-03-26T18:54:37.233 INFO:tasks.workunit.client.1.smithi190.stdout:2/711: write d6/d9/d45/d6b/fb0 [1044449,98658] 0 2024-03-26T18:54:37.233 INFO:tasks.workunit.client.1.smithi190.stdout:5/619: dread d2/d17/d1b/db6/fc7 [0,4194304] 0 2024-03-26T18:54:37.233 INFO:tasks.workunit.client.1.smithi190.stdout:5/620: chown d2/c13 1 1 2024-03-26T18:54:37.233 INFO:tasks.workunit.client.1.smithi190.stdout:5/621: truncate d2/d17/d1b/db6/d88/f8b 569078 0 2024-03-26T18:54:37.234 INFO:tasks.workunit.client.1.smithi190.stdout:5/622: fdatasync d2/d17/d98/d3a/d3c/f94 0 2024-03-26T18:54:37.235 INFO:tasks.workunit.client.0.smithi102.stdout:8/437: link d0/d15/f16 d0/d3/d2c/d33/d87/f8c 0 2024-03-26T18:54:37.241 INFO:tasks.workunit.client.1.smithi190.stdout:5/623: unlink d2/d17/d1b/l37 0 2024-03-26T18:54:37.241 INFO:tasks.workunit.client.1.smithi190.stdout:5/624: stat d2/d17/d1b/db6/d24/d3b/f56 0 2024-03-26T18:54:37.241 INFO:tasks.workunit.client.1.smithi190.stdout:5/625: truncate d2/d17/d1b/fc8 217770 0 2024-03-26T18:54:37.242 INFO:tasks.workunit.client.0.smithi102.stdout:0/431: dread d6/d2d/f38 [0,4194304] 0 2024-03-26T18:54:37.242 INFO:tasks.workunit.client.0.smithi102.stdout:0/432: readlink d6/l10 0 2024-03-26T18:54:37.246 INFO:tasks.workunit.client.1.smithi190.stdout:8/798: write d9/dd/d15/f30 [1002259,75111] 0 2024-03-26T18:54:37.247 INFO:tasks.workunit.client.1.smithi190.stdout:8/799: mkdir d9/dd/d23/d36/d37/db9/d110 0 2024-03-26T18:54:37.251 INFO:tasks.workunit.client.0.smithi102.stdout:3/599: creat df/d61/dd6/fdb x:0 0 0 2024-03-26T18:54:37.253 INFO:tasks.workunit.client.1.smithi190.stdout:5/626: mknod d2/d17/d1b/d60/d63/d85/cce 0 2024-03-26T18:54:37.253 INFO:tasks.workunit.client.0.smithi102.stdout:6/401: stat d2/d12/l49 0 2024-03-26T18:54:37.254 INFO:tasks.workunit.client.1.smithi190.stdout:5/627: dread d2/d17/d1b/d60/d63/f6e [0,4194304] 0 2024-03-26T18:54:37.257 INFO:tasks.workunit.client.1.smithi190.stdout:6/650: dwrite d17/d28/d74/fae [0,4194304] 0 2024-03-26T18:54:37.257 INFO:tasks.workunit.client.1.smithi190.stdout:6/651: write d17/d28/d2c/fb7 [836221,10477] 0 2024-03-26T18:54:37.257 INFO:tasks.workunit.client.1.smithi190.stdout:6/652: write d17/d28/d2c/f41 [5784569,87952] 0 2024-03-26T18:54:37.257 INFO:tasks.workunit.client.0.smithi102.stdout:0/433: mkdir d6/de/d13/d8d 0 2024-03-26T18:54:37.259 INFO:tasks.workunit.client.0.smithi102.stdout:9/366: rename df/d1a/d23/d29/d45 to df/d1a/d23/d4e/d8e 0 2024-03-26T18:54:37.260 INFO:tasks.workunit.client.0.smithi102.stdout:3/600: dread df/d29/f2c [0,4194304] 0 2024-03-26T18:54:37.261 INFO:tasks.workunit.client.0.smithi102.stdout:0/434: creat d6/de/d13/d25/d31/f8e x:0 0 0 2024-03-26T18:54:37.266 INFO:tasks.workunit.client.0.smithi102.stdout:9/367: unlink df/d1a/d65/l69 0 2024-03-26T18:54:37.266 INFO:tasks.workunit.client.0.smithi102.stdout:9/368: readlink df/d1a/l44 0 2024-03-26T18:54:37.266 INFO:tasks.workunit.client.0.smithi102.stdout:9/369: creat df/d1a/d57/f8f x:0 0 0 2024-03-26T18:54:37.266 INFO:tasks.workunit.client.0.smithi102.stdout:9/370: dread - df/d1a/d23/d4e/f77 zero size 2024-03-26T18:54:37.266 INFO:tasks.workunit.client.0.smithi102.stdout:9/371: fdatasync df/d1a/d1d/f3a 0 2024-03-26T18:54:37.266 INFO:tasks.workunit.client.0.smithi102.stdout:9/372: write df/d1a/d65/f73 [1333888,85841] 0 2024-03-26T18:54:37.266 INFO:tasks.workunit.client.0.smithi102.stdout:9/373: dread - df/d46/f67 zero size 2024-03-26T18:54:37.266 INFO:tasks.workunit.client.0.smithi102.stdout:9/374: dread - df/d1a/d57/f64 zero size 2024-03-26T18:54:37.269 INFO:tasks.workunit.client.1.smithi190.stdout:4/747: dwrite d7/d8/d14/d23/d28/d9e/fe7 [0,4194304] 0 2024-03-26T18:54:37.270 INFO:tasks.workunit.client.0.smithi102.stdout:3/601: truncate df/d29/d31/d36/f7f 2445525 0 2024-03-26T18:54:37.271 INFO:tasks.workunit.client.1.smithi190.stdout:4/748: symlink d7/d1d/d42/l102 0 2024-03-26T18:54:37.271 INFO:tasks.workunit.client.1.smithi190.stdout:4/749: readlink d7/d1d/lc1 0 2024-03-26T18:54:37.271 INFO:tasks.workunit.client.1.smithi190.stdout:4/750: chown d7/dc8 68817 1 2024-03-26T18:54:37.272 INFO:tasks.workunit.client.1.smithi190.stdout:4/751: creat d7/d8/d14/d23/d28/d38/d96/f103 x:0 0 0 2024-03-26T18:54:37.273 INFO:tasks.workunit.client.0.smithi102.stdout:0/435: mkdir d6/d3e/d69/d8c/d8f 0 2024-03-26T18:54:37.273 INFO:tasks.workunit.client.0.smithi102.stdout:4/323: dwrite d1/d17/d2a/f54 [0,4194304] 0 2024-03-26T18:54:37.274 INFO:tasks.workunit.client.0.smithi102.stdout:4/324: write d1/d17/d1a/d21/f63 [839994,81302] 0 2024-03-26T18:54:37.274 INFO:tasks.workunit.client.1.smithi190.stdout:4/752: mknod d7/d8/d14/dc2/c104 0 2024-03-26T18:54:37.276 INFO:tasks.workunit.client.0.smithi102.stdout:2/350: dwrite d3/d6/f1d [4194304,4194304] 0 2024-03-26T18:54:37.276 INFO:tasks.workunit.client.0.smithi102.stdout:2/351: fdatasync d3/d20/f30 0 2024-03-26T18:54:37.277 INFO:tasks.workunit.client.0.smithi102.stdout:3/602: truncate df/d61/d55/f95 1899599 0 2024-03-26T18:54:37.278 INFO:tasks.workunit.client.0.smithi102.stdout:3/603: readlink df/d29/d31/d36/d3c/l42 0 2024-03-26T18:54:37.278 INFO:tasks.workunit.client.0.smithi102.stdout:3/604: chown df/d29/d31/d36/d3c/d6f/d75/lbc 836 1 2024-03-26T18:54:37.278 INFO:tasks.workunit.client.0.smithi102.stdout:3/605: chown df/d61/f58 381790 1 2024-03-26T18:54:37.278 INFO:tasks.workunit.client.0.smithi102.stdout:3/606: dread - df/d1c/f81 zero size 2024-03-26T18:54:37.281 INFO:tasks.workunit.client.0.smithi102.stdout:3/607: creat df/fdc x:0 0 0 2024-03-26T18:54:37.281 INFO:tasks.workunit.client.0.smithi102.stdout:3/608: creat df/d1c/d34/d53/fdd x:0 0 0 2024-03-26T18:54:37.281 INFO:tasks.workunit.client.0.smithi102.stdout:3/609: creat df/d1c/d34/da5/fde x:0 0 0 2024-03-26T18:54:37.281 INFO:tasks.workunit.client.0.smithi102.stdout:3/610: truncate df/f1b 13530000 0 2024-03-26T18:54:37.284 INFO:tasks.workunit.client.1.smithi190.stdout:4/753: write d7/d1d/f26 [3943,29712] 0 2024-03-26T18:54:37.288 INFO:tasks.workunit.client.1.smithi190.stdout:4/754: getdents d7/d8/d14/d5c 0 2024-03-26T18:54:37.289 INFO:tasks.workunit.client.0.smithi102.stdout:0/436: dread d6/fb [0,4194304] 0 2024-03-26T18:54:37.289 INFO:tasks.workunit.client.0.smithi102.stdout:0/437: creat d6/de/d13/d25/d4f/d66/f90 x:0 0 0 2024-03-26T18:54:37.289 INFO:tasks.workunit.client.0.smithi102.stdout:0/438: write d6/de/f15 [3546515,34926] 0 2024-03-26T18:54:37.294 INFO:tasks.workunit.client.1.smithi190.stdout:0/779: dwrite d0/dd/d20/d1c/d4e/f4f [0,4194304] 0 2024-03-26T18:54:37.294 INFO:tasks.workunit.client.0.smithi102.stdout:8/438: dwrite d0/d15/d81/f29 [4194304,4194304] 0 2024-03-26T18:54:37.294 INFO:tasks.workunit.client.0.smithi102.stdout:8/439: chown d0/d9/c17 13 1 2024-03-26T18:54:37.294 INFO:tasks.workunit.client.0.smithi102.stdout:0/439: creat d6/f91 x:0 0 0 2024-03-26T18:54:37.296 INFO:tasks.workunit.client.0.smithi102.stdout:3/611: dread df/d29/d31/d36/d45/f60 [0,4194304] 0 2024-03-26T18:54:37.296 INFO:tasks.workunit.client.0.smithi102.stdout:2/352: write d3/d2a/f4f [3853483,55408] 0 2024-03-26T18:54:37.299 INFO:tasks.workunit.client.0.smithi102.stdout:8/440: creat d0/d3/d2c/d33/f8d x:0 0 0 2024-03-26T18:54:37.299 INFO:tasks.workunit.client.0.smithi102.stdout:8/441: creat d0/d3/d2c/f8e x:0 0 0 2024-03-26T18:54:37.302 INFO:tasks.workunit.client.0.smithi102.stdout:1/367: dwrite de/d16/d1d/d2a/f4c [0,4194304] 0 2024-03-26T18:54:37.304 INFO:tasks.workunit.client.0.smithi102.stdout:0/440: truncate d6/de/d13/d25/d31/f4e 3580685 0 2024-03-26T18:54:37.304 INFO:tasks.workunit.client.0.smithi102.stdout:0/441: chown d6/de/d13/d25/d31/d45/d82/d84 2980649 1 2024-03-26T18:54:37.304 INFO:tasks.workunit.client.0.smithi102.stdout:9/375: dwrite df/d1a/f86 [0,4194304] 0 2024-03-26T18:54:37.304 INFO:tasks.workunit.client.0.smithi102.stdout:0/442: truncate d6/de/d13/d25/d4f/d66/f90 327378 0 2024-03-26T18:54:37.304 INFO:tasks.workunit.client.0.smithi102.stdout:9/376: chown df/d1a/d23/d3e/d75/c51 467051 1 2024-03-26T18:54:37.307 INFO:tasks.workunit.client.1.smithi190.stdout:0/780: creat d0/dd/d20/d1c/d8b/d8d/db7/f11e x:0 0 0 2024-03-26T18:54:37.307 INFO:tasks.workunit.client.1.smithi190.stdout:8/800: rmdir d9/dd/d23 39 2024-03-26T18:54:37.309 INFO:tasks.workunit.client.0.smithi102.stdout:6/402: dwrite d2/d3/f2a [0,4194304] 0 2024-03-26T18:54:37.309 INFO:tasks.workunit.client.0.smithi102.stdout:6/403: fsync d2/d12/f2d 0 2024-03-26T18:54:37.310 INFO:tasks.workunit.client.1.smithi190.stdout:8/801: creat d9/dd/d23/db1/f111 x:0 0 0 2024-03-26T18:54:37.311 INFO:tasks.workunit.client.0.smithi102.stdout:8/442: rename d0/d3/d2c/d2f/f53 to d0/d3/d2c/d2f/f8f 0 2024-03-26T18:54:37.311 INFO:tasks.workunit.client.0.smithi102.stdout:8/443: write d0/d3/f32 [1184975,52857] 0 2024-03-26T18:54:37.315 INFO:tasks.workunit.client.1.smithi190.stdout:2/712: dwrite d6/d9/d10/d5f/fad [0,4194304] 0 2024-03-26T18:54:37.316 INFO:tasks.workunit.client.1.smithi190.stdout:2/713: chown d6/d9/d45 8519 1 2024-03-26T18:54:37.316 INFO:tasks.workunit.client.1.smithi190.stdout:2/714: truncate d6/d9/d3a/d3d/d2e/d31/fec 358776 0 2024-03-26T18:54:37.316 INFO:tasks.workunit.client.0.smithi102.stdout:1/368: truncate de/d1f/f48 89226 0 2024-03-26T18:54:37.317 INFO:tasks.workunit.client.0.smithi102.stdout:6/404: dread d2/df/f27 [0,4194304] 0 2024-03-26T18:54:37.317 INFO:tasks.workunit.client.0.smithi102.stdout:4/325: dwrite d1/d25/d39/f66 [0,4194304] 0 2024-03-26T18:54:37.319 INFO:tasks.workunit.client.1.smithi190.stdout:2/715: mkdir d6/d9/d3a/d3d/d1c/d8d/de8/dfb 0 2024-03-26T18:54:37.319 INFO:tasks.workunit.client.1.smithi190.stdout:2/716: chown d6/d9/d10/l15 32234 1 2024-03-26T18:54:37.321 INFO:tasks.workunit.client.1.smithi190.stdout:2/717: mkdir d6/d9/d3a/d3d/d67/d9e/dfc 0 2024-03-26T18:54:37.323 INFO:tasks.workunit.client.0.smithi102.stdout:2/353: dread d3/d2d/f68 [0,4194304] 0 2024-03-26T18:54:37.324 INFO:tasks.workunit.client.0.smithi102.stdout:6/405: dread d2/df/d35/d43/f61 [0,4194304] 0 2024-03-26T18:54:37.326 INFO:tasks.workunit.client.0.smithi102.stdout:9/377: link df/d1a/d23/d4e/d8e/f7b df/d1a/d1d/d72/f90 0 2024-03-26T18:54:37.328 INFO:tasks.workunit.client.1.smithi190.stdout:0/781: write d0/dd/d20/d1c/d46/d7c/fa9 [1844152,41588] 0 2024-03-26T18:54:37.329 INFO:tasks.workunit.client.0.smithi102.stdout:2/354: truncate d3/d6/fa 1081298 0 2024-03-26T18:54:37.332 INFO:tasks.workunit.client.1.smithi190.stdout:0/782: rmdir d0/dd/d20/d1c/d8b/d8d 39 2024-03-26T18:54:37.343 INFO:tasks.workunit.client.1.smithi190.stdout:6/653: dwrite d17/d28/d2c/fb7 [0,4194304] 0 2024-03-26T18:54:37.344 INFO:tasks.workunit.client.1.smithi190.stdout:4/755: dwrite d7/d8/d14/d5c/f83 [0,4194304] 0 2024-03-26T18:54:37.348 INFO:tasks.workunit.client.1.smithi190.stdout:2/718: read d6/d9/d3a/f3b [912114,119771] 0 2024-03-26T18:54:37.348 INFO:tasks.workunit.client.1.smithi190.stdout:6/654: link d17/d28/d2c/d50/f8f d17/d28/d80/da8/fd9 0 2024-03-26T18:54:37.349 INFO:tasks.workunit.client.1.smithi190.stdout:2/719: rmdir d6/d9/d3a/d3d/d1c/d8d/dc7/d72/db1 39 2024-03-26T18:54:37.351 INFO:tasks.workunit.client.0.smithi102.stdout:0/443: dwrite d6/de/f4b [0,4194304] 0 2024-03-26T18:54:37.351 INFO:tasks.workunit.client.0.smithi102.stdout:0/444: creat d6/de/d13/d25/d4f/f92 x:0 0 0 2024-03-26T18:54:37.351 INFO:tasks.workunit.client.1.smithi190.stdout:2/720: mkdir d6/dbc/dfd 0 2024-03-26T18:54:37.351 INFO:tasks.workunit.client.1.smithi190.stdout:6/655: mkdir d17/d28/d8b/d95/dda 0 2024-03-26T18:54:37.351 INFO:tasks.workunit.client.1.smithi190.stdout:6/656: fdatasync d17/d28/d38/d79/f8e 0 2024-03-26T18:54:37.352 INFO:tasks.workunit.client.1.smithi190.stdout:6/657: dread - d17/d28/d8b/d95/fc6 zero size 2024-03-26T18:54:37.353 INFO:tasks.workunit.client.1.smithi190.stdout:2/721: rename d6/d9/d3a/d3d/d67/d9e/da3/dae to d6/d9/d3a/d3d/d1c/dfe 0 2024-03-26T18:54:37.353 INFO:tasks.workunit.client.1.smithi190.stdout:2/722: readlink d6/d9/d45/lf6 0 2024-03-26T18:54:37.354 INFO:tasks.workunit.client.1.smithi190.stdout:2/723: getdents d6/d9/d48 0 2024-03-26T18:54:37.354 INFO:tasks.workunit.client.1.smithi190.stdout:6/658: rmdir d17/d28/d74/d7b 39 2024-03-26T18:54:37.356 INFO:tasks.workunit.client.1.smithi190.stdout:2/724: dread d6/d9/d45/d6b/f80 [0,4194304] 0 2024-03-26T18:54:37.356 INFO:tasks.workunit.client.0.smithi102.stdout:3/612: dwrite df/f7b [0,4194304] 0 2024-03-26T18:54:37.357 INFO:tasks.workunit.client.1.smithi190.stdout:2/725: mknod d6/cff 0 2024-03-26T18:54:37.357 INFO:tasks.workunit.client.1.smithi190.stdout:2/726: dread - d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d8b/fa1 zero size 2024-03-26T18:54:37.359 INFO:tasks.workunit.client.1.smithi190.stdout:2/727: symlink d6/d21/dd8/l100 0 2024-03-26T18:54:37.359 INFO:tasks.workunit.client.1.smithi190.stdout:2/728: getdents d6/d9/d3a/d3d/d67/d9e/dfc 0 2024-03-26T18:54:37.359 INFO:tasks.workunit.client.1.smithi190.stdout:2/729: write d6/d9/d3a/d3d/d55/f92 [5193690,29501] 0 2024-03-26T18:54:37.359 INFO:tasks.workunit.client.1.smithi190.stdout:8/802: dwrite d9/dd/d23/f3e [0,4194304] 0 2024-03-26T18:54:37.361 INFO:tasks.workunit.client.1.smithi190.stdout:2/730: creat d6/d9/d10/db7/dcf/f101 x:0 0 0 2024-03-26T18:54:37.361 INFO:tasks.workunit.client.1.smithi190.stdout:2/731: fdatasync d6/d21/dd0/fee 0 2024-03-26T18:54:37.363 INFO:tasks.workunit.client.1.smithi190.stdout:4/756: read d7/d8/d14/d23/d28/d9e/fe7 [1680158,16267] 0 2024-03-26T18:54:37.365 INFO:tasks.workunit.client.1.smithi190.stdout:8/803: creat d9/dd/d23/d36/d37/d3a/d43/dc2/df2/f112 x:0 0 0 2024-03-26T18:54:37.366 INFO:tasks.workunit.client.1.smithi190.stdout:8/804: getdents d9/da2/d73/dfc 0 2024-03-26T18:54:37.367 INFO:tasks.workunit.client.1.smithi190.stdout:8/805: truncate d9/d48/f50 7596220 0 2024-03-26T18:54:37.368 INFO:tasks.workunit.client.1.smithi190.stdout:8/806: creat d9/d39/d9c/f113 x:0 0 0 2024-03-26T18:54:37.369 INFO:tasks.workunit.client.1.smithi190.stdout:8/807: dread - d9/dd/d23/db1/f111 zero size 2024-03-26T18:54:37.369 INFO:tasks.workunit.client.1.smithi190.stdout:8/808: chown d9/d39/d67/c6b 16136 1 2024-03-26T18:54:37.369 INFO:tasks.workunit.client.0.smithi102.stdout:0/445: dread d6/d2d/d3d/f40 [0,4194304] 0 2024-03-26T18:54:37.371 INFO:tasks.workunit.client.1.smithi190.stdout:8/809: mkdir d9/d39/d9e/d114 0 2024-03-26T18:54:37.371 INFO:tasks.workunit.client.1.smithi190.stdout:8/810: fsync d9/dd/f2d 0 2024-03-26T18:54:37.372 INFO:tasks.workunit.client.1.smithi190.stdout:8/811: symlink d9/dd/d23/d36/d37/d3a/l115 0 2024-03-26T18:54:37.372 INFO:tasks.workunit.client.1.smithi190.stdout:8/812: chown d9/dd/d15/cdd 27 1 2024-03-26T18:54:37.372 INFO:tasks.workunit.client.1.smithi190.stdout:8/813: dread - d9/dd/d23/d88/fe4 zero size 2024-03-26T18:54:37.372 INFO:tasks.workunit.client.1.smithi190.stdout:8/814: fdatasync d9/d48/d49/ff1 0 2024-03-26T18:54:37.373 INFO:tasks.workunit.client.0.smithi102.stdout:3/613: dread fd [0,4194304] 0 2024-03-26T18:54:37.373 INFO:tasks.workunit.client.0.smithi102.stdout:3/614: fdatasync df/d29/d31/f79 0 2024-03-26T18:54:37.375 INFO:tasks.workunit.client.1.smithi190.stdout:6/659: dread f1 [0,4194304] 0 2024-03-26T18:54:37.376 INFO:tasks.workunit.client.0.smithi102.stdout:0/446: mkdir d6/de/d93 0 2024-03-26T18:54:37.377 INFO:tasks.workunit.client.0.smithi102.stdout:3/615: creat df/fdf x:0 0 0 2024-03-26T18:54:37.378 INFO:tasks.workunit.client.0.smithi102.stdout:3/616: creat df/d1c/fe0 x:0 0 0 2024-03-26T18:54:37.379 INFO:tasks.workunit.client.0.smithi102.stdout:3/617: stat df/d29/d3a/f6d 0 2024-03-26T18:54:37.380 INFO:tasks.workunit.client.1.smithi190.stdout:4/757: write d7/f35 [3985561,113534] 0 2024-03-26T18:54:37.382 INFO:tasks.workunit.client.1.smithi190.stdout:4/758: link d7/d1d/d42/d4a/lb8 d7/dc8/dd5/l105 0 2024-03-26T18:54:37.383 INFO:tasks.workunit.client.1.smithi190.stdout:6/660: dread d17/d28/d2c/db8/f54 [0,4194304] 0 2024-03-26T18:54:37.383 INFO:tasks.workunit.client.1.smithi190.stdout:4/759: symlink d7/d8/d14/d23/d28/d38/d96/d9a/l106 0 2024-03-26T18:54:37.384 INFO:tasks.workunit.client.0.smithi102.stdout:8/444: dread d0/d3/f32 [0,4194304] 0 2024-03-26T18:54:37.384 INFO:tasks.workunit.client.1.smithi190.stdout:6/661: mkdir d17/d28/d46/dd7/ddb 0 2024-03-26T18:54:37.384 INFO:tasks.workunit.client.1.smithi190.stdout:6/662: chown d17/d28/f2d 2006 1 2024-03-26T18:54:37.385 INFO:tasks.workunit.client.1.smithi190.stdout:4/760: symlink d7/d8/d14/d5c/l107 0 2024-03-26T18:54:37.385 INFO:tasks.workunit.client.1.smithi190.stdout:4/761: fsync d7/d1d/d42/fbe 0 2024-03-26T18:54:37.385 INFO:tasks.workunit.client.1.smithi190.stdout:4/762: read d7/d8/d1c/f39 [1833434,54025] 0 2024-03-26T18:54:37.388 INFO:tasks.workunit.client.1.smithi190.stdout:6/663: dread d17/d28/d38/f89 [0,4194304] 0 2024-03-26T18:54:37.389 INFO:tasks.workunit.client.0.smithi102.stdout:9/378: dwrite df/d1a/d23/d3e/f6b [0,4194304] 0 2024-03-26T18:54:37.390 INFO:tasks.workunit.client.1.smithi190.stdout:8/815: write d9/d39/d67/fba [3760852,108495] 0 2024-03-26T18:54:37.391 INFO:tasks.workunit.client.1.smithi190.stdout:2/732: dwrite d6/d9/fe2 [0,4194304] 0 2024-03-26T18:54:37.392 INFO:tasks.workunit.client.1.smithi190.stdout:2/733: chown d6/d9/d3a/d3d/d2e/d86 0 1 2024-03-26T18:54:37.392 INFO:tasks.workunit.client.1.smithi190.stdout:8/816: rename d9/d39/d67/dc1 to d9/dd/d23/d36/d37/db8/d116 0 2024-03-26T18:54:37.393 INFO:tasks.workunit.client.0.smithi102.stdout:9/379: mknod df/d1a/d23/d3e/d75/c91 0 2024-03-26T18:54:37.393 INFO:tasks.workunit.client.0.smithi102.stdout:2/355: dwrite d3/d6/f1b [0,4194304] 0 2024-03-26T18:54:37.393 INFO:tasks.workunit.client.0.smithi102.stdout:2/356: write d3/d6/f1b [5121264,103568] 0 2024-03-26T18:54:37.393 INFO:tasks.workunit.client.0.smithi102.stdout:2/357: readlink d3/d11/d15/d31/l61 0 2024-03-26T18:54:37.398 INFO:tasks.workunit.client.1.smithi190.stdout:8/817: mknod d9/dd/d23/d88/dfa/c117 0 2024-03-26T18:54:37.398 INFO:tasks.workunit.client.1.smithi190.stdout:8/818: dread - d9/dd/d23/d36/d37/d3a/d61/daf/db3/fe2 zero size 2024-03-26T18:54:37.398 INFO:tasks.workunit.client.1.smithi190.stdout:8/819: chown d9/dd/d23/d36/d37/d3a/d43/l7f 19 1 2024-03-26T18:54:37.399 INFO:tasks.workunit.client.0.smithi102.stdout:1/369: dwrite de/d16/f40 [0,4194304] 0 2024-03-26T18:54:37.399 INFO:tasks.workunit.client.0.smithi102.stdout:1/370: dread de/d1f/f3a [0,4194304] 0 2024-03-26T18:54:37.399 INFO:tasks.workunit.client.0.smithi102.stdout:1/371: truncate de/d16/d1d/d2a/d2c/f3c 919316 0 2024-03-26T18:54:37.399 INFO:tasks.workunit.client.0.smithi102.stdout:2/358: rename d3/d6/d1a/f76 to d3/d74/f77 0 2024-03-26T18:54:37.399 INFO:tasks.workunit.client.0.smithi102.stdout:2/359: chown d3/d2d/f68 4 1 2024-03-26T18:54:37.401 INFO:tasks.workunit.client.0.smithi102.stdout:3/618: write df/d1c/d34/f8e [834673,47017] 0 2024-03-26T18:54:37.401 INFO:tasks.workunit.client.0.smithi102.stdout:3/619: truncate df/d29/d31/d77/d8a/fbd 538541 0 2024-03-26T18:54:37.401 INFO:tasks.workunit.client.0.smithi102.stdout:4/326: dwrite d1/f9 [0,4194304] 0 2024-03-26T18:54:37.401 INFO:tasks.workunit.client.0.smithi102.stdout:4/327: chown d1/l1e 5366059 1 2024-03-26T18:54:37.402 INFO:tasks.workunit.client.1.smithi190.stdout:0/783: dwrite d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d75/f11b [0,4194304] 0 2024-03-26T18:54:37.402 INFO:tasks.workunit.client.1.smithi190.stdout:0/784: stat d0/c7 0 2024-03-26T18:54:37.402 INFO:tasks.workunit.client.1.smithi190.stdout:4/763: dread d7/d1d/f3c [0,4194304] 0 2024-03-26T18:54:37.405 INFO:tasks.workunit.client.1.smithi190.stdout:0/785: dread d0/dd/d20/d1c/d4e/f9d [0,4194304] 0 2024-03-26T18:54:37.406 INFO:tasks.workunit.client.0.smithi102.stdout:2/360: truncate d3/d11/d27/f57 193345 0 2024-03-26T18:54:37.406 INFO:tasks.workunit.client.0.smithi102.stdout:8/445: read d0/d3/d2c/d33/d87/f8c [523583,126661] 0 2024-03-26T18:54:37.406 INFO:tasks.workunit.client.0.smithi102.stdout:8/446: dread - d0/d9/f7d zero size 2024-03-26T18:54:37.406 INFO:tasks.workunit.client.0.smithi102.stdout:8/447: creat d0/d3/d26/d19/f90 x:0 0 0 2024-03-26T18:54:37.410 INFO:tasks.workunit.client.1.smithi190.stdout:2/734: read d6/d9/d3a/d3d/d1c/d8d/dc7/f43 [1679683,77135] 0 2024-03-26T18:54:37.410 INFO:tasks.workunit.client.1.smithi190.stdout:2/735: write d6/d9/d10/d5f/fa2 [1388670,20526] 0 2024-03-26T18:54:37.413 INFO:tasks.workunit.client.1.smithi190.stdout:2/736: creat d6/d9/d3a/d3d/d1c/d8d/dc7/dd1/f102 x:0 0 0 2024-03-26T18:54:37.415 INFO:tasks.workunit.client.0.smithi102.stdout:3/620: symlink df/d61/d55/le1 0 2024-03-26T18:54:37.416 INFO:tasks.workunit.client.0.smithi102.stdout:0/447: dwrite d6/d2d/f38 [0,4194304] 0 2024-03-26T18:54:37.416 INFO:tasks.workunit.client.0.smithi102.stdout:0/448: chown d6/de/d13/d1f/d55/d57/d7d/d7f 282 1 2024-03-26T18:54:37.416 INFO:tasks.workunit.client.1.smithi190.stdout:2/737: link d6/d9/d3a/d3d/d1c/d8d/dc7/l42 d6/d9/d10/d5f/l103 0 2024-03-26T18:54:37.419 INFO:tasks.workunit.client.1.smithi190.stdout:2/738: getdents d6/d9/d3a 0 2024-03-26T18:54:37.423 INFO:tasks.workunit.client.1.smithi190.stdout:8/820: dread d9/dd/d23/d36/d37/d3a/d43/f80 [0,4194304] 0 2024-03-26T18:54:37.423 INFO:tasks.workunit.client.0.smithi102.stdout:3/621: dread df/d1c/f2a [0,4194304] 0 2024-03-26T18:54:37.424 INFO:tasks.workunit.client.0.smithi102.stdout:2/361: rename d3/d11/d15/d31/d58 to d3/d20/d78 0 2024-03-26T18:54:37.424 INFO:tasks.workunit.client.1.smithi190.stdout:0/786: dread d0/dd/d20/d1c/d46/da7/df0/d4b/fc5 [0,4194304] 0 2024-03-26T18:54:37.424 INFO:tasks.workunit.client.1.smithi190.stdout:0/787: write d0/dd/d20/f9f [61243,46760] 0 2024-03-26T18:54:37.427 INFO:tasks.workunit.client.0.smithi102.stdout:0/449: write d6/d2d/d3d/f40 [2699295,104548] 0 2024-03-26T18:54:37.427 INFO:tasks.workunit.client.0.smithi102.stdout:8/448: symlink d0/d3/d26/d19/d76/l91 0 2024-03-26T18:54:37.429 INFO:tasks.workunit.client.1.smithi190.stdout:0/788: rename d0/dd/ffa to d0/dd5/dee/f11f 0 2024-03-26T18:54:37.433 INFO:tasks.workunit.client.1.smithi190.stdout:0/789: mknod d0/dd/d20/d1c/d46/c120 0 2024-03-26T18:54:37.433 INFO:tasks.workunit.client.1.smithi190.stdout:0/790: chown d0/dd/d20/d1c/d8b/d8d/d8f/dfe 7591038 1 2024-03-26T18:54:37.433 INFO:tasks.workunit.client.1.smithi190.stdout:0/791: fsync d0/dd/d20/d1c/d46/da7/df0/d87/d93/db5/dcc/fe3 0 2024-03-26T18:54:37.434 INFO:tasks.workunit.client.0.smithi102.stdout:4/328: mkdir d1/d6c 0 2024-03-26T18:54:37.434 INFO:tasks.workunit.client.0.smithi102.stdout:4/329: chown d1/d17/d1a/d21/c3d 15927 1 2024-03-26T18:54:37.434 INFO:tasks.workunit.client.0.smithi102.stdout:4/330: chown d1 2120 1 2024-03-26T18:54:37.435 INFO:tasks.workunit.client.0.smithi102.stdout:3/622: mknod df/d61/d55/ce2 0 2024-03-26T18:54:37.436 INFO:tasks.workunit.client.0.smithi102.stdout:3/623: creat df/d29/fe3 x:0 0 0 2024-03-26T18:54:37.436 INFO:tasks.workunit.client.0.smithi102.stdout:9/380: dwrite df/d84/f78 [0,4194304] 0 2024-03-26T18:54:37.436 INFO:tasks.workunit.client.0.smithi102.stdout:9/381: truncate df/d1a/d23/d3e/d75/f4a 1021324 0 2024-03-26T18:54:37.437 INFO:tasks.workunit.client.0.smithi102.stdout:2/362: mknod d3/c79 0 2024-03-26T18:54:37.438 INFO:tasks.workunit.client.0.smithi102.stdout:0/450: mknod d6/de/d13/d1f/d55/c94 0 2024-03-26T18:54:37.438 INFO:tasks.workunit.client.0.smithi102.stdout:0/451: creat d6/de/d13/d25/d4f/d66/f95 x:0 0 0 2024-03-26T18:54:37.439 INFO:tasks.workunit.client.0.smithi102.stdout:0/452: fsync d6/de/d13/d1f/d55/d57/d7d/d7f/f8b 0 2024-03-26T18:54:37.441 INFO:tasks.workunit.client.0.smithi102.stdout:8/449: link d0/d15/d5d/f61 d0/d3/d2c/f92 0 2024-03-26T18:54:37.441 INFO:tasks.workunit.client.0.smithi102.stdout:4/331: stat d1/d25/d39/l44 0 2024-03-26T18:54:37.445 INFO:tasks.workunit.client.0.smithi102.stdout:9/382: symlink df/d1a/d23/d29/l92 0 2024-03-26T18:54:37.445 INFO:tasks.workunit.client.1.smithi190.stdout:7/765: sync 2024-03-26T18:54:37.445 INFO:tasks.workunit.client.1.smithi190.stdout:7/766: fdatasync d3/d1b/dd3/d10d/d65/d90/d92/fbc 0 2024-03-26T18:54:37.446 INFO:tasks.workunit.client.0.smithi102.stdout:2/363: truncate d3/d11/d15/d46/f47 732766 0 2024-03-26T18:54:37.446 INFO:tasks.workunit.client.1.smithi190.stdout:7/767: creat d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/d87/f11f x:0 0 0 2024-03-26T18:54:37.448 INFO:tasks.workunit.client.1.smithi190.stdout:7/768: link d3/d6/d9/d1f/d4c/fe6 d3/d6/d9/d1f/d4c/d71/df6/f120 0 2024-03-26T18:54:37.448 INFO:tasks.workunit.client.0.smithi102.stdout:3/624: dread df/d1c/d70/f87 [0,4194304] 0 2024-03-26T18:54:37.448 INFO:tasks.workunit.client.0.smithi102.stdout:3/625: write df/d61/dd6/fdb [1025853,47477] 0 2024-03-26T18:54:37.449 INFO:tasks.workunit.client.0.smithi102.stdout:8/450: rename d0/d3/f6 to d0/d15/d81/f93 0 2024-03-26T18:54:37.449 INFO:tasks.workunit.client.0.smithi102.stdout:8/451: chown d0/d3/d2c/d4e 14 1 2024-03-26T18:54:37.449 INFO:tasks.workunit.client.0.smithi102.stdout:1/372: dwrite de/d16/d1d/d2a/d2c/d56/d6e/f7a [0,4194304] 0 2024-03-26T18:54:37.449 INFO:tasks.workunit.client.0.smithi102.stdout:8/452: creat d0/d3/d31/f94 x:0 0 0 2024-03-26T18:54:37.450 INFO:tasks.workunit.client.0.smithi102.stdout:9/383: creat df/d1a/d65/f93 x:0 0 0 2024-03-26T18:54:37.450 INFO:tasks.workunit.client.0.smithi102.stdout:9/384: creat df/d1a/d57/f94 x:0 0 0 2024-03-26T18:54:37.450 INFO:tasks.workunit.client.1.smithi190.stdout:7/769: link d3/d1b/dd3/d10d/d65/d90/d112/ffe d3/d1b/f121 0 2024-03-26T18:54:37.450 INFO:tasks.workunit.client.1.smithi190.stdout:7/770: chown d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/d8f/fcf 0 1 2024-03-26T18:54:37.451 INFO:tasks.workunit.client.1.smithi190.stdout:7/771: write d3/fe [814676,32696] 0 2024-03-26T18:54:37.451 INFO:tasks.workunit.client.0.smithi102.stdout:2/364: creat d3/d20/d78/f7a x:0 0 0 2024-03-26T18:54:37.451 INFO:tasks.workunit.client.1.smithi190.stdout:7/772: write d3/d6/fd [4367889,21470] 0 2024-03-26T18:54:37.451 INFO:tasks.workunit.client.1.smithi190.stdout:7/773: fsync d3/d6/d9/d14/d36/f10a 0 2024-03-26T18:54:37.452 INFO:tasks.workunit.client.0.smithi102.stdout:0/453: getdents d6/de/d13/d25/d31/d45/d82 0 2024-03-26T18:54:37.453 INFO:tasks.workunit.client.0.smithi102.stdout:3/626: symlink df/d29/d31/d36/d45/daf/le4 0 2024-03-26T18:54:37.455 INFO:tasks.workunit.client.0.smithi102.stdout:7/394: sync 2024-03-26T18:54:37.456 INFO:tasks.workunit.client.0.smithi102.stdout:7/395: write d7/d12/f22 [1330226,61135] 0 2024-03-26T18:54:37.456 INFO:tasks.workunit.client.0.smithi102.stdout:7/396: fsync f6 0 2024-03-26T18:54:37.456 INFO:tasks.workunit.client.1.smithi190.stdout:7/774: creat d3/d1b/d57/f122 x:0 0 0 2024-03-26T18:54:37.456 INFO:tasks.workunit.client.1.smithi190.stdout:4/764: dwrite d7/d8/d14/f5b [0,4194304] 0 2024-03-26T18:54:37.456 INFO:tasks.workunit.client.1.smithi190.stdout:2/739: dwrite d6/d9/d10/f34 [0,4194304] 0 2024-03-26T18:54:37.456 INFO:tasks.workunit.client.1.smithi190.stdout:2/740: truncate d6/d9/d3a/d3d/d1c/d8d/fab 1456344 0 2024-03-26T18:54:37.458 INFO:tasks.workunit.client.1.smithi190.stdout:0/792: dwrite d0/dd/d20/d1c/d46/d7c/fa9 [0,4194304] 0 2024-03-26T18:54:37.458 INFO:tasks.workunit.client.1.smithi190.stdout:0/793: chown d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d66/ca6 3173 1 2024-03-26T18:54:37.459 INFO:tasks.workunit.client.0.smithi102.stdout:1/373: dread de/d16/d1d/f4d [0,4194304] 0 2024-03-26T18:54:37.459 INFO:tasks.workunit.client.0.smithi102.stdout:8/453: unlink d0/d3/d26/d19/d76/l91 0 2024-03-26T18:54:37.460 INFO:tasks.workunit.client.1.smithi190.stdout:7/775: dread d3/f29 [0,4194304] 0 2024-03-26T18:54:37.461 INFO:tasks.workunit.client.1.smithi190.stdout:2/741: mknod d6/d9/d3a/d3d/d1c/d8d/c104 0 2024-03-26T18:54:37.461 INFO:tasks.workunit.client.1.smithi190.stdout:2/742: chown d6/d9/d3a/d3d/d67/ldf 67339 1 2024-03-26T18:54:37.462 INFO:tasks.workunit.client.1.smithi190.stdout:4/765: dread d7/d8/d14/f52 [0,4194304] 0 2024-03-26T18:54:37.463 INFO:tasks.workunit.client.1.smithi190.stdout:7/776: rename d3/d6/d9/d1f/d4c/d71/ce2 to d3/d1b/dd3/d10d/c123 0 2024-03-26T18:54:37.463 INFO:tasks.workunit.client.0.smithi102.stdout:9/385: symlink df/d1a/d23/d4e/d8e/l95 0 2024-03-26T18:54:37.463 INFO:tasks.workunit.client.0.smithi102.stdout:9/386: stat df/d1a/d65 0 2024-03-26T18:54:37.463 INFO:tasks.workunit.client.0.smithi102.stdout:9/387: write df/d1a/d23/d4e/f7c [955820,39657] 0 2024-03-26T18:54:37.463 INFO:tasks.workunit.client.0.smithi102.stdout:9/388: fsync df/d1a/d57/f8b 0 2024-03-26T18:54:37.463 INFO:tasks.workunit.client.0.smithi102.stdout:5/620: sync 2024-03-26T18:54:37.464 INFO:tasks.workunit.client.1.smithi190.stdout:7/777: mkdir d3/d1b/dd3/d10d/d65/d109/d11b/d124 0 2024-03-26T18:54:37.468 INFO:tasks.workunit.client.0.smithi102.stdout:2/365: mkdir d3/d11/d15/d46/d7b 0 2024-03-26T18:54:37.472 INFO:tasks.workunit.client.0.smithi102.stdout:3/627: mkdir df/d61/de5 0 2024-03-26T18:54:37.472 INFO:tasks.workunit.client.0.smithi102.stdout:3/628: dread - df/d29/d31/d36/d3c/d6f/d75/f94 zero size 2024-03-26T18:54:37.472 INFO:tasks.workunit.client.0.smithi102.stdout:3/629: creat df/d61/dd6/fe6 x:0 0 0 2024-03-26T18:54:37.472 INFO:tasks.workunit.client.0.smithi102.stdout:3/630: truncate df/d61/f58 174239 0 2024-03-26T18:54:37.476 INFO:tasks.workunit.client.0.smithi102.stdout:7/397: symlink d7/d10/d27/d34/l75 0 2024-03-26T18:54:37.478 INFO:tasks.workunit.client.0.smithi102.stdout:1/374: creat de/d16/d1d/d61/f7c x:0 0 0 2024-03-26T18:54:37.478 INFO:tasks.workunit.client.0.smithi102.stdout:1/375: fsync de/d16/d1d/f4d 0 2024-03-26T18:54:37.478 INFO:tasks.workunit.client.0.smithi102.stdout:1/376: chown de/d16/d1d/d2a/d2c/d56/d5f 30041 1 2024-03-26T18:54:37.478 INFO:tasks.workunit.client.1.smithi190.stdout:7/778: dread d3/d1b/dd3/d10d/d65/d90/d92/d9b/fb0 [0,4194304] 0 2024-03-26T18:54:37.479 INFO:tasks.workunit.client.0.smithi102.stdout:9/389: truncate df/d1a/d65/f73 1292971 0 2024-03-26T18:54:37.479 INFO:tasks.workunit.client.0.smithi102.stdout:9/390: readlink df/d1a/d1d/l55 0 2024-03-26T18:54:37.482 INFO:tasks.workunit.client.1.smithi190.stdout:0/794: dread d0/dd/d20/d61/f7b [4194304,4194304] 0 2024-03-26T18:54:37.482 INFO:tasks.workunit.client.0.smithi102.stdout:3/631: getdents df/d29/d31/d36 0 2024-03-26T18:54:37.484 INFO:tasks.workunit.client.1.smithi190.stdout:7/779: creat d3/d6/d9/d1f/d93/f125 x:0 0 0 2024-03-26T18:54:37.485 INFO:tasks.workunit.client.0.smithi102.stdout:9/391: symlink df/d1a/d1d/l96 0 2024-03-26T18:54:37.485 INFO:tasks.workunit.client.0.smithi102.stdout:5/621: link d2/d25/d75/db9/l46 d2/d25/d5b/ldd 0 2024-03-26T18:54:37.490 INFO:tasks.workunit.client.0.smithi102.stdout:2/366: write d3/f1c [950596,13769] 0 2024-03-26T18:54:37.491 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:37 smithi190 ceph-mon[35043]: pgmap v246: 65 pgs: 65 active+clean; 2.2 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 231 MiB/s rd, 278 MiB/s wr, 723 op/s 2024-03-26T18:54:37.493 INFO:tasks.workunit.client.0.smithi102.stdout:7/398: write d7/f20 [3191135,28564] 0 2024-03-26T18:54:37.494 INFO:tasks.workunit.client.1.smithi190.stdout:2/743: dread d6/f36 [0,4194304] 0 2024-03-26T18:54:37.498 INFO:tasks.workunit.client.0.smithi102.stdout:2/367: mkdir d3/d20/d78/d7c 0 2024-03-26T18:54:37.498 INFO:tasks.workunit.client.0.smithi102.stdout:2/368: write d3/d2a/f6d [4756618,115342] 0 2024-03-26T18:54:37.498 INFO:tasks.workunit.client.0.smithi102.stdout:3/632: dread df/d29/d31/d36/f7f [0,4194304] 0 2024-03-26T18:54:37.499 INFO:tasks.workunit.client.0.smithi102.stdout:3/633: creat df/d61/d55/fe7 x:0 0 0 2024-03-26T18:54:37.499 INFO:tasks.workunit.client.0.smithi102.stdout:2/369: rmdir d3/d6/d1a 39 2024-03-26T18:54:37.499 INFO:tasks.workunit.client.0.smithi102.stdout:2/370: chown d3/d6/d1a/d25/c5e 10048 1 2024-03-26T18:54:37.499 INFO:tasks.workunit.client.1.smithi190.stdout:2/744: link d6/d9/d10/d5f/f7f d6/dbc/f105 0 2024-03-26T18:54:37.500 INFO:tasks.workunit.client.0.smithi102.stdout:1/377: dread de/d16/d1d/f1e [0,4194304] 0 2024-03-26T18:54:37.501 INFO:tasks.workunit.client.0.smithi102.stdout:1/378: fsync de/d16/f40 0 2024-03-26T18:54:37.501 INFO:tasks.workunit.client.0.smithi102.stdout:3/634: creat df/d29/d31/dc0/fe8 x:0 0 0 2024-03-26T18:54:37.501 INFO:tasks.workunit.client.0.smithi102.stdout:3/635: write df/d29/d31/d77/d8a/fbd [713135,81633] 0 2024-03-26T18:54:37.501 INFO:tasks.workunit.client.0.smithi102.stdout:3/636: creat df/d1c/fe9 x:0 0 0 2024-03-26T18:54:37.502 INFO:tasks.workunit.client.0.smithi102.stdout:3/637: fsync df/d29/d3a/fd7 0 2024-03-26T18:54:37.502 INFO:tasks.workunit.client.1.smithi190.stdout:4/766: dwrite d7/d1d/f3c [0,4194304] 0 2024-03-26T18:54:37.502 INFO:tasks.workunit.client.1.smithi190.stdout:4/767: readlink d7/d1d/lc1 0 2024-03-26T18:54:37.502 INFO:tasks.workunit.client.1.smithi190.stdout:4/768: chown d7/d8/d1c/l25 589598 1 2024-03-26T18:54:37.503 INFO:tasks.workunit.client.0.smithi102.stdout:2/371: creat d3/d6/d1a/f7d x:0 0 0 2024-03-26T18:54:37.503 INFO:tasks.workunit.client.0.smithi102.stdout:8/454: dwrite d0/d15/d81/f73 [0,4194304] 0 2024-03-26T18:54:37.503 INFO:tasks.workunit.client.0.smithi102.stdout:8/455: chown d0/d3/d2c/d2f/f78 175088520 1 2024-03-26T18:54:37.506 INFO:tasks.workunit.client.1.smithi190.stdout:4/769: symlink d7/d1d/d42/d4a/l108 0 2024-03-26T18:54:37.506 INFO:tasks.workunit.client.1.smithi190.stdout:4/770: dread - d7/d8/d14/d4b/d7d/fd0 zero size 2024-03-26T18:54:37.507 INFO:tasks.workunit.client.0.smithi102.stdout:0/454: dwrite d6/de/f4b [4194304,4194304] 0 2024-03-26T18:54:37.508 INFO:tasks.workunit.client.0.smithi102.stdout:3/638: mknod df/d29/d31/d36/d3c/d6f/d75/cea 0 2024-03-26T18:54:37.509 INFO:tasks.workunit.client.0.smithi102.stdout:2/372: creat d3/d20/d78/d7c/f7e x:0 0 0 2024-03-26T18:54:37.509 INFO:tasks.workunit.client.1.smithi190.stdout:4/771: creat d7/d1d/d41/d74/f109 x:0 0 0 2024-03-26T18:54:37.511 INFO:tasks.workunit.client.0.smithi102.stdout:8/456: truncate d0/d3c/f43 4068938 0 2024-03-26T18:54:37.513 INFO:tasks.workunit.client.0.smithi102.stdout:3/639: chown df/d29/d3a/d93/cab 452 1 2024-03-26T18:54:37.513 INFO:tasks.workunit.client.0.smithi102.stdout:3/640: stat df/d29/d31/d36/d3c/d6f/d75/cea 0 2024-03-26T18:54:37.513 INFO:tasks.workunit.client.0.smithi102.stdout:3/641: chown df/d29/d31/d36/d45/f65 852 1 2024-03-26T18:54:37.513 INFO:tasks.workunit.client.0.smithi102.stdout:2/373: mkdir d3/d11/d15/d31/d7f 0 2024-03-26T18:54:37.514 INFO:tasks.workunit.client.0.smithi102.stdout:2/374: creat d3/d20/f80 x:0 0 0 2024-03-26T18:54:37.515 INFO:tasks.workunit.client.0.smithi102.stdout:1/379: dread de/d16/f35 [0,4194304] 0 2024-03-26T18:54:37.517 INFO:tasks.workunit.client.1.smithi190.stdout:0/795: dread d0/dd/d20/d1c/d46/da7/df0/d87/d93/db5/dcc/fdc [0,4194304] 0 2024-03-26T18:54:37.517 INFO:tasks.workunit.client.1.smithi190.stdout:0/796: dread - d0/dd/d20/d1c/d46/d7c/f114 zero size 2024-03-26T18:54:37.517 INFO:tasks.workunit.client.1.smithi190.stdout:0/797: write d0/dd/d20/d1c/d46/da7/df0/da2/fa5 [4950107,16816] 0 2024-03-26T18:54:37.518 INFO:tasks.workunit.client.0.smithi102.stdout:2/375: mknod d3/d2d/c81 0 2024-03-26T18:54:37.518 INFO:tasks.workunit.client.0.smithi102.stdout:2/376: readlink d3/d2d/l42 0 2024-03-26T18:54:37.518 INFO:tasks.workunit.client.0.smithi102.stdout:2/377: readlink d3/d11/d15/d31/l61 0 2024-03-26T18:54:37.518 INFO:tasks.workunit.client.0.smithi102.stdout:2/378: read - d3/d74/f77 zero size 2024-03-26T18:54:37.518 INFO:tasks.workunit.client.0.smithi102.stdout:2/379: fdatasync d3/f13 0 2024-03-26T18:54:37.518 INFO:tasks.workunit.client.0.smithi102.stdout:2/380: chown d3/d6/cc 1582030 1 2024-03-26T18:54:37.521 INFO:tasks.workunit.client.0.smithi102.stdout:1/380: unlink fb 0 2024-03-26T18:54:37.524 INFO:tasks.workunit.client.0.smithi102.stdout:5/622: dwrite d2/d25/d5b/d72/f57 [0,4194304] 0 2024-03-26T18:54:37.526 INFO:tasks.workunit.client.0.smithi102.stdout:1/381: creat de/d16/d1d/d61/d6a/f7d x:0 0 0 2024-03-26T18:54:37.527 INFO:tasks.workunit.client.1.smithi190.stdout:0/798: unlink d0/dd/d20/d1c/d46/d58/de4/f108 0 2024-03-26T18:54:37.527 INFO:tasks.workunit.client.1.smithi190.stdout:7/780: dwrite d3/d1b/dd2/f115 [0,4194304] 0 2024-03-26T18:54:37.528 INFO:tasks.workunit.client.1.smithi190.stdout:7/781: symlink d3/d1b/dd3/d10d/d65/d90/d92/l126 0 2024-03-26T18:54:37.528 INFO:tasks.workunit.client.1.smithi190.stdout:7/782: dread - d3/d1b/dd3/d10d/d65/d90/d92/d9b/dc4/f108 zero size 2024-03-26T18:54:37.528 INFO:tasks.workunit.client.0.smithi102.stdout:5/623: creat d2/d25/d5b/fde x:0 0 0 2024-03-26T18:54:37.528 INFO:tasks.workunit.client.0.smithi102.stdout:5/624: write d2/d25/d75/db9/d43/d9e/d82/fa2 [1959316,78040] 0 2024-03-26T18:54:37.530 INFO:tasks.workunit.client.1.smithi190.stdout:7/783: creat d3/d6/d9/d1f/d4c/d71/df6/f127 x:0 0 0 2024-03-26T18:54:37.532 INFO:tasks.workunit.client.0.smithi102.stdout:5/625: dread d2/d25/d5b/f94 [0,4194304] 0 2024-03-26T18:54:37.532 INFO:tasks.workunit.client.1.smithi190.stdout:7/784: mknod d3/d6/c128 0 2024-03-26T18:54:37.533 INFO:tasks.workunit.client.1.smithi190.stdout:7/785: rmdir d3/d1b/dd3/d10d/d65/d109 39 2024-03-26T18:54:37.534 INFO:tasks.workunit.client.0.smithi102.stdout:5/626: creat d2/d2c/d6f/db8/fdf x:0 0 0 2024-03-26T18:54:37.534 INFO:tasks.workunit.client.1.smithi190.stdout:7/786: getdents d3/d1b/dd3/d10d/d50/dde/dfd 0 2024-03-26T18:54:37.536 INFO:tasks.workunit.client.0.smithi102.stdout:5/627: rename d2/d14/d1a/l44 to d2/d2c/d6f/le0 0 2024-03-26T18:54:37.537 INFO:tasks.workunit.client.0.smithi102.stdout:5/628: stat d2/d25/d75/db9/l46 0 2024-03-26T18:54:37.540 INFO:tasks.workunit.client.0.smithi102.stdout:5/629: dread d2/d14/d1a/f61 [0,4194304] 0 2024-03-26T18:54:37.540 INFO:tasks.workunit.client.0.smithi102.stdout:5/630: fsync d2/d25/d75/db9/db3/fd9 0 2024-03-26T18:54:37.540 INFO:tasks.workunit.client.1.smithi190.stdout:7/787: link d3/d6/d9/l4b d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/d8f/l129 0 2024-03-26T18:54:37.541 INFO:tasks.workunit.client.1.smithi190.stdout:7/788: dread - d3/d6/d9/d1f/d4c/d71/df6/f127 zero size 2024-03-26T18:54:37.541 INFO:tasks.workunit.client.1.smithi190.stdout:7/789: write d3/f43 [1371349,130369] 0 2024-03-26T18:54:37.541 INFO:tasks.workunit.client.0.smithi102.stdout:5/631: mknod d2/d25/d2a/dbb/ce1 0 2024-03-26T18:54:37.542 INFO:tasks.workunit.client.0.smithi102.stdout:5/632: chown d2/d2c/d6f/l7e 1797 1 2024-03-26T18:54:37.543 INFO:tasks.workunit.client.0.smithi102.stdout:5/633: rmdir d2/d25/db0 39 2024-03-26T18:54:37.543 INFO:tasks.workunit.client.0.smithi102.stdout:5/634: write d2/d2c/f58 [2172523,14189] 0 2024-03-26T18:54:37.544 INFO:tasks.workunit.client.0.smithi102.stdout:5/635: rmdir d2/d25/d75/db9/d43/d9e/d82/dc9 39 2024-03-26T18:54:37.544 INFO:tasks.workunit.client.0.smithi102.stdout:9/392: dwrite df/d1a/d65/f93 [0,4194304] 0 2024-03-26T18:54:37.545 INFO:tasks.workunit.client.0.smithi102.stdout:5/636: link d2/d25/d2a/d4f/da9/lca d2/d25/le2 0 2024-03-26T18:54:37.549 INFO:tasks.workunit.client.0.smithi102.stdout:9/393: rename df/d1a/d65/c6c to df/d1a/c97 0 2024-03-26T18:54:37.550 INFO:tasks.workunit.client.0.smithi102.stdout:5/637: mknod d2/d25/db0/ce3 0 2024-03-26T18:54:37.553 INFO:tasks.workunit.client.0.smithi102.stdout:9/394: link df/d46/f67 df/d1a/d23/d60/f98 0 2024-03-26T18:54:37.555 INFO:tasks.workunit.client.1.smithi190.stdout:2/745: dwrite d6/d21/d61/d6c/f9c [0,4194304] 0 2024-03-26T18:54:37.555 INFO:tasks.workunit.client.1.smithi190.stdout:2/746: write d6/d9/d10/f87 [898835,9604] 0 2024-03-26T18:54:37.555 INFO:tasks.workunit.client.1.smithi190.stdout:2/747: getdents d6/d21/d61 0 2024-03-26T18:54:37.555 INFO:tasks.workunit.client.1.smithi190.stdout:9/951: sync 2024-03-26T18:54:37.555 INFO:tasks.workunit.client.1.smithi190.stdout:9/952: fdatasync d12/d2d/d1d/da1/dba/fd0 0 2024-03-26T18:54:37.556 INFO:tasks.workunit.client.0.smithi102.stdout:3/642: dwrite df/d1c/fe0 [0,4194304] 0 2024-03-26T18:54:37.556 INFO:tasks.workunit.client.0.smithi102.stdout:3/643: truncate df/d61/d55/fe7 594736 0 2024-03-26T18:54:37.557 INFO:tasks.workunit.client.0.smithi102.stdout:3/644: fsync df/d29/d31/d36/f7f 0 2024-03-26T18:54:37.557 INFO:tasks.workunit.client.0.smithi102.stdout:3/645: truncate df/d8b/fd1 431611 0 2024-03-26T18:54:37.559 INFO:tasks.workunit.client.1.smithi190.stdout:2/748: truncate d6/d9/d3a/d3d/fbf 2885745 0 2024-03-26T18:54:37.559 INFO:tasks.workunit.client.1.smithi190.stdout:2/749: chown d6/d21/dd8/ff4 83769 1 2024-03-26T18:54:37.563 INFO:tasks.workunit.client.1.smithi190.stdout:9/953: creat d12/d2d/d1d/d23/d83/dc9/dd4/d3f/f154 x:0 0 0 2024-03-26T18:54:37.564 INFO:tasks.workunit.client.1.smithi190.stdout:2/750: mknod d6/d9/d45/df3/c106 0 2024-03-26T18:54:37.564 INFO:tasks.workunit.client.1.smithi190.stdout:2/751: fdatasync d6/d9/d3a/d3d/d1c/d8d/dc7/dd1/f102 0 2024-03-26T18:54:37.565 INFO:tasks.workunit.client.1.smithi190.stdout:2/752: mknod d6/d9/d3a/d3d/d1c/d8d/de8/dfb/c107 0 2024-03-26T18:54:37.565 INFO:tasks.workunit.client.1.smithi190.stdout:2/753: chown d6/d9/d3a/d3d/d1c/fe7 113286 1 2024-03-26T18:54:37.566 INFO:tasks.workunit.client.1.smithi190.stdout:9/954: truncate f8 7576520 0 2024-03-26T18:54:37.567 INFO:tasks.workunit.client.0.smithi102.stdout:5/638: read d2/d25/d75/db9/d43/d9e/d82/f8f [1216871,69260] 0 2024-03-26T18:54:37.568 INFO:tasks.workunit.client.1.smithi190.stdout:9/955: creat d12/d80/db2/de1/f155 x:0 0 0 2024-03-26T18:54:37.581 INFO:tasks.workunit.client.1.smithi190.stdout:4/772: dwrite d7/d8/d14/d4b/d7d/f89 [0,4194304] 0 2024-03-26T18:54:37.582 INFO:tasks.workunit.client.0.smithi102.stdout:0/455: dwrite d6/de/f28 [4194304,4194304] 0 2024-03-26T18:54:37.583 INFO:tasks.workunit.client.1.smithi190.stdout:1/645: sync 2024-03-26T18:54:37.583 INFO:tasks.workunit.client.0.smithi102.stdout:0/456: link d6/l10 d6/de/d13/d25/l96 0 2024-03-26T18:54:37.584 INFO:tasks.workunit.client.1.smithi190.stdout:4/773: mknod d7/d1d/d5e/d98/c10a 0 2024-03-26T18:54:37.584 INFO:tasks.workunit.client.1.smithi190.stdout:4/774: fsync d7/d1d/f32 0 2024-03-26T18:54:37.584 INFO:tasks.workunit.client.1.smithi190.stdout:4/775: chown d7/d8/d14/dc2/c104 127 1 2024-03-26T18:54:37.584 INFO:tasks.workunit.client.1.smithi190.stdout:9/956: dread f11 [0,4194304] 0 2024-03-26T18:54:37.584 INFO:tasks.workunit.client.0.smithi102.stdout:0/457: mknod d6/de/d13/d25/c97 0 2024-03-26T18:54:37.585 INFO:tasks.workunit.client.1.smithi190.stdout:4/776: mkdir d7/d8/d14/d4b/d7d/da1/d10b 0 2024-03-26T18:54:37.587 INFO:tasks.workunit.client.1.smithi190.stdout:9/957: creat d12/d2d/d1d/d23/d2b/f156 x:0 0 0 2024-03-26T18:54:37.587 INFO:tasks.workunit.client.1.smithi190.stdout:9/958: chown d12/d2d/d1d/cb6 13510 1 2024-03-26T18:54:37.588 INFO:tasks.workunit.client.1.smithi190.stdout:4/777: rename d7/d8/d14/dc2/c104 to d7/d8/d14/d6e/df0/c10c 0 2024-03-26T18:54:37.589 INFO:tasks.workunit.client.1.smithi190.stdout:9/959: mknod d12/d2d/d1d/da1/dba/dd2/c157 0 2024-03-26T18:54:37.592 INFO:tasks.workunit.client.1.smithi190.stdout:9/960: mkdir d12/d2d/d1d/d23/d83/dc9/dd4/d71/d129/d158 0 2024-03-26T18:54:37.598 INFO:tasks.workunit.client.1.smithi190.stdout:9/961: rename d12/d2d/dfc/d119 to d12/d2d/d1d/d23/d83/dc9/dd4/d71/d129/d158/d159 0 2024-03-26T18:54:37.598 INFO:tasks.workunit.client.1.smithi190.stdout:9/962: chown d12/d2d/d1d/d23/d83/dc9/dd4/d71/l134 202 1 2024-03-26T18:54:37.598 INFO:tasks.workunit.client.1.smithi190.stdout:9/963: creat d12/f15a x:0 0 0 2024-03-26T18:54:37.606 INFO:tasks.workunit.client.0.smithi102.stdout:8/457: dwrite d0/d3/d31/f4a [0,4194304] 0 2024-03-26T18:54:37.606 INFO:tasks.workunit.client.0.smithi102.stdout:8/458: read - d0/d3/d2c/d2f/f60 zero size 2024-03-26T18:54:37.609 INFO:tasks.workunit.client.0.smithi102.stdout:8/459: creat d0/d3c/d6f/f95 x:0 0 0 2024-03-26T18:54:37.610 INFO:tasks.workunit.client.0.smithi102.stdout:8/460: mkdir d0/d3c/d96 0 2024-03-26T18:54:37.619 INFO:tasks.workunit.client.1.smithi190.stdout:0/799: dwrite d0/dd/d20/d1c/d46/da7/df0/da2/f9e [0,4194304] 0 2024-03-26T18:54:37.619 INFO:tasks.workunit.client.1.smithi190.stdout:0/800: fdatasync d0/dd/f22 0 2024-03-26T18:54:37.619 INFO:tasks.workunit.client.1.smithi190.stdout:0/801: stat d0/dd/d20/d1c/d8b/c38 0 2024-03-26T18:54:37.620 INFO:tasks.workunit.client.0.smithi102.stdout:1/382: dwrite de/d16/d1d/d2a/d2c/d56/d78/f79 [0,4194304] 0 2024-03-26T18:54:37.622 INFO:tasks.workunit.client.1.smithi190.stdout:0/802: truncate d0/dd/d20/d1c/d46/da7/df0/da2/d80/fb2 4031042 0 2024-03-26T18:54:37.624 INFO:tasks.workunit.client.0.smithi102.stdout:1/383: link de/d1f/c63 de/d16/d1d/d2a/d2c/c7e 0 2024-03-26T18:54:37.624 INFO:tasks.workunit.client.0.smithi102.stdout:1/384: chown f9 11284 1 2024-03-26T18:54:37.624 INFO:tasks.workunit.client.0.smithi102.stdout:1/385: readlink de/d16/d1d/d61/l6c 0 2024-03-26T18:54:37.625 INFO:tasks.workunit.client.0.smithi102.stdout:1/386: truncate de/d16/d1d/d61/f7c 409383 0 2024-03-26T18:54:37.625 INFO:tasks.workunit.client.0.smithi102.stdout:1/387: read - de/d16/f54 zero size 2024-03-26T18:54:37.625 INFO:tasks.workunit.client.0.smithi102.stdout:1/388: creat de/d1f/f7f x:0 0 0 2024-03-26T18:54:37.625 INFO:tasks.workunit.client.0.smithi102.stdout:1/389: dread - de/d16/f34 zero size 2024-03-26T18:54:37.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:37 smithi102 ceph-mon[29002]: pgmap v246: 65 pgs: 65 active+clean; 2.2 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 231 MiB/s rd, 278 MiB/s wr, 723 op/s 2024-03-26T18:54:37.625 INFO:tasks.workunit.client.0.smithi102.stdout:1/390: truncate de/f3f 669045 0 2024-03-26T18:54:37.626 INFO:tasks.workunit.client.0.smithi102.stdout:1/391: stat de/d16/d1d/c37 0 2024-03-26T18:54:37.627 INFO:tasks.workunit.client.1.smithi190.stdout:9/964: write d12/f14 [524221,49651] 0 2024-03-26T18:54:37.627 INFO:tasks.workunit.client.1.smithi190.stdout:9/965: read - d12/d2d/d1d/d23/d83/dc9/f10a zero size 2024-03-26T18:54:37.627 INFO:tasks.workunit.client.1.smithi190.stdout:7/790: dwrite d3/d6/d9/d14/d59/d73/d8b/f101 [0,4194304] 0 2024-03-26T18:54:37.627 INFO:tasks.workunit.client.0.smithi102.stdout:1/392: mkdir de/d16/d1d/d2a/d2c/d56/d78/d80 0 2024-03-26T18:54:37.628 INFO:tasks.workunit.client.0.smithi102.stdout:9/395: dwrite df/d1a/d57/f64 [0,4194304] 0 2024-03-26T18:54:37.628 INFO:tasks.workunit.client.0.smithi102.stdout:3/646: dwrite df/f67 [0,4194304] 0 2024-03-26T18:54:37.628 INFO:tasks.workunit.client.0.smithi102.stdout:9/396: read - df/d1a/d23/d3e/d75/f8d zero size 2024-03-26T18:54:37.629 INFO:tasks.workunit.client.0.smithi102.stdout:1/393: rename de/d1f/f48 to de/d16/d1d/d2a/d2c/d56/d6e/f81 0 2024-03-26T18:54:37.631 INFO:tasks.workunit.client.1.smithi190.stdout:7/791: write d3/d1b/dd3/d10d/d65/f7d [566863,127276] 0 2024-03-26T18:54:37.633 INFO:tasks.workunit.client.1.smithi190.stdout:7/792: rmdir d3/d6 39 2024-03-26T18:54:37.636 INFO:tasks.workunit.client.0.smithi102.stdout:3/647: write df/d61/dd6/fdb [557139,99132] 0 2024-03-26T18:54:37.636 INFO:tasks.workunit.client.0.smithi102.stdout:3/648: truncate f7 4111784 0 2024-03-26T18:54:37.636 INFO:tasks.workunit.client.1.smithi190.stdout:7/793: dread d3/d6/d9/d14/d36/f70 [0,4194304] 0 2024-03-26T18:54:37.638 INFO:tasks.workunit.client.1.smithi190.stdout:7/794: mkdir d3/d6/d9/d14/d36/d12a 0 2024-03-26T18:54:37.638 INFO:tasks.workunit.client.1.smithi190.stdout:7/795: readlink d3/d1b/d98/lb3 0 2024-03-26T18:54:37.638 INFO:tasks.workunit.client.0.smithi102.stdout:9/397: rmdir df/d84 39 2024-03-26T18:54:37.638 INFO:tasks.workunit.client.0.smithi102.stdout:9/398: write df/d1a/d23/d4e/f53 [4774924,126243] 0 2024-03-26T18:54:37.638 INFO:tasks.workunit.client.0.smithi102.stdout:9/399: creat df/d1a/d23/d3e/f99 x:0 0 0 2024-03-26T18:54:37.638 INFO:tasks.workunit.client.0.smithi102.stdout:9/400: readlink df/d1a/d65/l7a 0 2024-03-26T18:54:37.639 INFO:tasks.workunit.client.0.smithi102.stdout:9/401: creat df/d1a/f9a x:0 0 0 2024-03-26T18:54:37.641 INFO:tasks.workunit.client.1.smithi190.stdout:7/796: rename d3/f2b to d3/d6/d9/f12b 0 2024-03-26T18:54:37.641 INFO:tasks.workunit.client.1.smithi190.stdout:7/797: truncate d3/d6/d9/d14/d59/d73/d8b/f101 5180988 0 2024-03-26T18:54:37.641 INFO:tasks.workunit.client.1.smithi190.stdout:7/798: chown d3/d6/d9/d1f/d4c/d71/df6/f127 2940 1 2024-03-26T18:54:37.642 INFO:tasks.workunit.client.0.smithi102.stdout:5/639: rmdir d2 39 2024-03-26T18:54:37.643 INFO:tasks.workunit.client.1.smithi190.stdout:7/799: mknod d3/d1b/dd3/c12c 0 2024-03-26T18:54:37.643 INFO:tasks.workunit.client.1.smithi190.stdout:2/754: dwrite d6/d9/d3a/d3d/d2e/f51 [0,4194304] 0 2024-03-26T18:54:37.645 INFO:tasks.workunit.client.1.smithi190.stdout:2/755: mkdir d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d7c/d108 0 2024-03-26T18:54:37.645 INFO:tasks.workunit.client.1.smithi190.stdout:2/756: chown d6/d9/d45/ff7 1504 1 2024-03-26T18:54:37.646 INFO:tasks.workunit.client.0.smithi102.stdout:9/402: unlink ce 0 2024-03-26T18:54:37.648 INFO:tasks.workunit.client.1.smithi190.stdout:1/646: dwrite d3/f53 [0,4194304] 0 2024-03-26T18:54:37.650 INFO:tasks.workunit.client.1.smithi190.stdout:7/800: rename d3/d1b/d98 to d3/d1b/dd3/d10d/d65/d90/d92/d12d 0 2024-03-26T18:54:37.651 INFO:tasks.workunit.client.1.smithi190.stdout:2/757: mknod d6/d21/c109 0 2024-03-26T18:54:37.651 INFO:tasks.workunit.client.1.smithi190.stdout:2/758: fsync d6/d21/fb5 0 2024-03-26T18:54:37.652 INFO:tasks.workunit.client.1.smithi190.stdout:7/801: dread d3/d1b/dd3/d10d/f55 [0,4194304] 0 2024-03-26T18:54:37.652 INFO:tasks.workunit.client.1.smithi190.stdout:2/759: mkdir d6/d9/d3a/d3d/d67/d9e/d10a 0 2024-03-26T18:54:37.652 INFO:tasks.workunit.client.1.smithi190.stdout:2/760: read - d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d7c/fcb zero size 2024-03-26T18:54:37.653 INFO:tasks.workunit.client.1.smithi190.stdout:7/802: symlink d3/d1b/dd2/l12e 0 2024-03-26T18:54:37.654 INFO:tasks.workunit.client.1.smithi190.stdout:7/803: read - d3/d6/d9/d1f/f110 zero size 2024-03-26T18:54:37.654 INFO:tasks.workunit.client.1.smithi190.stdout:7/804: creat d3/d1b/d57/f12f x:0 0 0 2024-03-26T18:54:37.654 INFO:tasks.workunit.client.1.smithi190.stdout:7/805: stat d3/d6/d9/d1f/d4c 0 2024-03-26T18:54:37.654 INFO:tasks.workunit.client.1.smithi190.stdout:7/806: stat d3/f19 0 2024-03-26T18:54:37.654 INFO:tasks.workunit.client.1.smithi190.stdout:7/807: read d3/d6/d9/d14/d36/f3c [3553980,56942] 0 2024-03-26T18:54:37.654 INFO:tasks.workunit.client.1.smithi190.stdout:2/761: dread d6/d9/d45/d6b/f80 [0,4194304] 0 2024-03-26T18:54:37.656 INFO:tasks.workunit.client.1.smithi190.stdout:7/808: unlink d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/d8f/fcf 0 2024-03-26T18:54:37.656 INFO:tasks.workunit.client.1.smithi190.stdout:7/809: write d3/d6/d9/d14/d36/f72 [1079037,9054] 0 2024-03-26T18:54:37.657 INFO:tasks.workunit.client.1.smithi190.stdout:2/762: getdents d6/d9/d3a/d3d/d2e/d86 0 2024-03-26T18:54:37.659 INFO:tasks.workunit.client.1.smithi190.stdout:2/763: unlink d6/d9/d3a/d3d/d55/l99 0 2024-03-26T18:54:37.661 INFO:tasks.workunit.client.1.smithi190.stdout:4/778: dwrite d7/d8/d14/d23/feb [0,4194304] 0 2024-03-26T18:54:37.661 INFO:tasks.workunit.client.1.smithi190.stdout:2/764: mkdir d6/d9/d3a/d3d/d2e/d86/d10b 0 2024-03-26T18:54:37.661 INFO:tasks.workunit.client.1.smithi190.stdout:2/765: readlink d6/d9/d3a/d3d/d1c/l25 0 2024-03-26T18:54:37.661 INFO:tasks.workunit.client.1.smithi190.stdout:4/779: write d7/d1d/d41/d74/f109 [940787,32238] 0 2024-03-26T18:54:37.661 INFO:tasks.workunit.client.1.smithi190.stdout:2/766: write d6/d9/d45/d6b/d93/dc0/ff2 [504969,105776] 0 2024-03-26T18:54:37.661 INFO:tasks.workunit.client.1.smithi190.stdout:0/803: dwrite d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/fca [0,4194304] 0 2024-03-26T18:54:37.662 INFO:tasks.workunit.client.0.smithi102.stdout:0/458: dwrite d6/d2d/d3d/f44 [0,4194304] 0 2024-03-26T18:54:37.662 INFO:tasks.workunit.client.0.smithi102.stdout:0/459: read - d6/f91 zero size 2024-03-26T18:54:37.662 INFO:tasks.workunit.client.1.smithi190.stdout:2/767: symlink d6/d9/d10/db7/dcf/l10c 0 2024-03-26T18:54:37.663 INFO:tasks.workunit.client.1.smithi190.stdout:2/768: stat d6/lef 0 2024-03-26T18:54:37.663 INFO:tasks.workunit.client.1.smithi190.stdout:2/769: chown d6/d21/dd0 2491 1 2024-03-26T18:54:37.663 INFO:tasks.workunit.client.0.smithi102.stdout:0/460: mknod d6/de/d13/d25/d31/c98 0 2024-03-26T18:54:37.665 INFO:tasks.workunit.client.1.smithi190.stdout:2/770: unlink d6/d9/d3a/fda 0 2024-03-26T18:54:37.667 INFO:tasks.workunit.client.1.smithi190.stdout:0/804: rename d0/dd/d20/d1c/d8b/d8d/db7/de6 to d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/d121 0 2024-03-26T18:54:37.667 INFO:tasks.workunit.client.1.smithi190.stdout:2/771: unlink d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d7c/f7d 0 2024-03-26T18:54:37.667 INFO:tasks.workunit.client.1.smithi190.stdout:2/772: read d6/d9/d45/d6b/fb0 [1057698,18932] 0 2024-03-26T18:54:37.669 INFO:tasks.workunit.client.1.smithi190.stdout:0/805: mknod d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/c122 0 2024-03-26T18:54:37.671 INFO:tasks.workunit.client.1.smithi190.stdout:0/806: rename d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/l70 to d0/dd/d20/d1c/d8b/d8d/dc8/l123 0 2024-03-26T18:54:37.673 INFO:tasks.workunit.client.1.smithi190.stdout:0/807: creat d0/dd/d20/d1c/d46/da7/df0/da2/d80/f124 x:0 0 0 2024-03-26T18:54:37.673 INFO:tasks.workunit.client.1.smithi190.stdout:0/808: write d0/dd/d20/d1c/d46/da7/df0/da2/fc6 [998819,113947] 0 2024-03-26T18:54:37.675 INFO:tasks.workunit.client.1.smithi190.stdout:4/780: write d7/d8/d14/d4b/f75 [287200,19940] 0 2024-03-26T18:54:37.676 INFO:tasks.workunit.client.1.smithi190.stdout:4/781: dread d7/d8/d1c/d20/f47 [0,4194304] 0 2024-03-26T18:54:37.679 INFO:tasks.workunit.client.1.smithi190.stdout:4/782: creat d7/d8/d14/d23/d28/d38/dd3/dfd/f10d x:0 0 0 2024-03-26T18:54:37.681 INFO:tasks.workunit.client.1.smithi190.stdout:3/596: sync 2024-03-26T18:54:37.681 INFO:tasks.workunit.client.1.smithi190.stdout:6/664: sync 2024-03-26T18:54:37.681 INFO:tasks.workunit.client.1.smithi190.stdout:8/821: sync 2024-03-26T18:54:37.681 INFO:tasks.workunit.client.1.smithi190.stdout:5/628: sync 2024-03-26T18:54:37.681 INFO:tasks.workunit.client.1.smithi190.stdout:3/597: write d1/d3/d52/fa2 [530759,63082] 0 2024-03-26T18:54:37.682 INFO:tasks.workunit.client.1.smithi190.stdout:9/966: dwrite d12/d2d/d1d/d23/d83/dc9/dd4/d71/f141 [0,4194304] 0 2024-03-26T18:54:37.683 INFO:tasks.workunit.client.1.smithi190.stdout:8/822: symlink d9/d39/d67/l118 0 2024-03-26T18:54:37.684 INFO:tasks.workunit.client.0.smithi102.stdout:6/406: sync 2024-03-26T18:54:37.685 INFO:tasks.workunit.client.1.smithi190.stdout:5/629: unlink d2/d17/d1b/db6/d88/c9d 0 2024-03-26T18:54:37.686 INFO:tasks.workunit.client.0.smithi102.stdout:6/407: readlink d2/d7/le 0 2024-03-26T18:54:37.686 INFO:tasks.workunit.client.1.smithi190.stdout:6/665: creat d17/d28/d2c/d63/fdc x:0 0 0 2024-03-26T18:54:37.691 INFO:tasks.workunit.client.1.smithi190.stdout:3/598: creat d1/d3/df/fbb x:0 0 0 2024-03-26T18:54:37.691 INFO:tasks.workunit.client.1.smithi190.stdout:3/599: truncate d1/d3/df/d18/d35/d41/dae/d9c/fa6 278822 0 2024-03-26T18:54:37.691 INFO:tasks.workunit.client.1.smithi190.stdout:3/600: chown d1/de/d81/dad/fb0 4304 1 2024-03-26T18:54:37.691 INFO:tasks.workunit.client.1.smithi190.stdout:6/666: mknod d17/d28/d8b/cdd 0 2024-03-26T18:54:37.691 INFO:tasks.workunit.client.1.smithi190.stdout:7/810: dread d3/d6/d9/d1f/f20 [4194304,4194304] 0 2024-03-26T18:54:37.692 INFO:tasks.workunit.client.1.smithi190.stdout:7/811: chown d3/d6/d9/d1f/d93/d105 89559814 1 2024-03-26T18:54:37.693 INFO:tasks.workunit.client.1.smithi190.stdout:7/812: rmdir d3/d6/d9/d1f/d93 39 2024-03-26T18:54:37.694 INFO:tasks.workunit.client.1.smithi190.stdout:3/601: rename d1/d3/d52/d70/f85 to d1/d3/df/d18/d57/d91/fbc 0 2024-03-26T18:54:37.697 INFO:tasks.workunit.client.1.smithi190.stdout:7/813: creat d3/d1b/dd3/d10d/d65/d109/f130 x:0 0 0 2024-03-26T18:54:37.698 INFO:tasks.workunit.client.1.smithi190.stdout:3/602: getdents d1/d3/df/d31/d5b 0 2024-03-26T18:54:37.698 INFO:tasks.workunit.client.1.smithi190.stdout:3/603: stat d1/d3/df/d18/d35/d65/c67 0 2024-03-26T18:54:37.702 INFO:tasks.workunit.client.1.smithi190.stdout:7/814: rmdir d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9 39 2024-03-26T18:54:37.702 INFO:tasks.workunit.client.1.smithi190.stdout:7/815: fdatasync d3/d6/d9/d14/f33 0 2024-03-26T18:54:37.702 INFO:tasks.workunit.client.1.smithi190.stdout:5/630: write d2/d17/d1b/db6/f77 [3135007,7460] 0 2024-03-26T18:54:37.703 INFO:tasks.workunit.client.1.smithi190.stdout:5/631: unlink d2/d17/d98/d3a/d3c/c8a 0 2024-03-26T18:54:37.703 INFO:tasks.workunit.client.1.smithi190.stdout:5/632: write d2/d17/d1b/db6/d24/f84 [119830,88942] 0 2024-03-26T18:54:37.703 INFO:tasks.workunit.client.1.smithi190.stdout:7/816: rmdir d3/d1b/dd3/d10d/d50 39 2024-03-26T18:54:37.703 INFO:tasks.workunit.client.1.smithi190.stdout:3/604: rmdir d1/d3/df/d18/d57/d91/d8c 39 2024-03-26T18:54:37.703 INFO:tasks.workunit.client.1.smithi190.stdout:4/783: write d7/d8/f22 [843846,115609] 0 2024-03-26T18:54:37.703 INFO:tasks.workunit.client.1.smithi190.stdout:7/817: symlink d3/d1b/dd3/d10d/d65/d90/l131 0 2024-03-26T18:54:37.703 INFO:tasks.workunit.client.0.smithi102.stdout:5/640: dwrite d2/f3a [0,4194304] 0 2024-03-26T18:54:37.703 INFO:tasks.workunit.client.0.smithi102.stdout:1/394: dwrite de/d1f/f53 [4194304,4194304] 0 2024-03-26T18:54:37.704 INFO:tasks.workunit.client.1.smithi190.stdout:4/784: mkdir d7/d8/d14/d6e/d6f/d10e 0 2024-03-26T18:54:37.704 INFO:tasks.workunit.client.1.smithi190.stdout:3/605: rename d1/d3/df/d31/d5b/f6b to d1/d3/d52/d70/db3/fbd 0 2024-03-26T18:54:37.704 INFO:tasks.workunit.client.1.smithi190.stdout:7/818: dread d3/d6/d9/d14/d36/f70 [0,4194304] 0 2024-03-26T18:54:37.705 INFO:tasks.workunit.client.1.smithi190.stdout:7/819: creat d3/d6/d9/d14/f132 x:0 0 0 2024-03-26T18:54:37.706 INFO:tasks.workunit.client.1.smithi190.stdout:3/606: mknod d1/d3/df/d18/d35/cbe 0 2024-03-26T18:54:37.707 INFO:tasks.workunit.client.0.smithi102.stdout:5/641: symlink d2/d25/d2a/db4/le4 0 2024-03-26T18:54:37.708 INFO:tasks.workunit.client.0.smithi102.stdout:1/395: creat de/d16/d1d/f82 x:0 0 0 2024-03-26T18:54:37.708 INFO:tasks.workunit.client.0.smithi102.stdout:1/396: creat de/f83 x:0 0 0 2024-03-26T18:54:37.709 INFO:tasks.workunit.client.0.smithi102.stdout:1/397: chown de/d16/d1d/d2a/d2c/f60 587089 1 2024-03-26T18:54:37.709 INFO:tasks.workunit.client.0.smithi102.stdout:1/398: chown de/d16/d1d/d61 1315000 1 2024-03-26T18:54:37.709 INFO:tasks.workunit.client.0.smithi102.stdout:1/399: fsync f9 0 2024-03-26T18:54:37.709 INFO:tasks.workunit.client.0.smithi102.stdout:3/649: dwrite df/d29/d31/d36/f50 [0,4194304] 0 2024-03-26T18:54:37.709 INFO:tasks.workunit.client.0.smithi102.stdout:3/650: creat df/d29/d31/d36/d3c/feb x:0 0 0 2024-03-26T18:54:37.709 INFO:tasks.workunit.client.1.smithi190.stdout:3/607: creat d1/d3/fbf x:0 0 0 2024-03-26T18:54:37.713 INFO:tasks.workunit.client.1.smithi190.stdout:5/633: write d2/d17/d1b/f58 [25174,128648] 0 2024-03-26T18:54:37.716 INFO:tasks.workunit.client.1.smithi190.stdout:5/634: creat d2/d17/d98/d3a/d3c/fcf x:0 0 0 2024-03-26T18:54:37.716 INFO:tasks.workunit.client.1.smithi190.stdout:1/647: dwrite d3/da/f47 [0,4194304] 0 2024-03-26T18:54:37.717 INFO:tasks.workunit.client.0.smithi102.stdout:3/651: symlink df/d1c/d34/d9b/lec 0 2024-03-26T18:54:37.718 INFO:tasks.workunit.client.1.smithi190.stdout:5/635: symlink d2/d17/d62/d64/ld0 0 2024-03-26T18:54:37.718 INFO:tasks.workunit.client.1.smithi190.stdout:5/636: chown d2/d17/d1b/d60/d63/d85/cce 568380 1 2024-03-26T18:54:37.719 INFO:tasks.workunit.client.1.smithi190.stdout:1/648: symlink d3/d9/da3/ld5 0 2024-03-26T18:54:37.721 INFO:tasks.workunit.client.1.smithi190.stdout:5/637: mkdir d2/d17/d1b/db6/d24/dd1 0 2024-03-26T18:54:37.722 INFO:tasks.workunit.client.1.smithi190.stdout:4/785: read d7/d8/d14/d5c/f83 [445770,97356] 0 2024-03-26T18:54:37.722 INFO:tasks.workunit.client.0.smithi102.stdout:3/652: getdents df/d29/d31/dc0 0 2024-03-26T18:54:37.722 INFO:tasks.workunit.client.0.smithi102.stdout:3/653: chown df/c68 495933012 1 2024-03-26T18:54:37.724 INFO:tasks.workunit.client.0.smithi102.stdout:5/642: write d2/d25/d75/db9/d43/d9e/f68 [3511343,25461] 0 2024-03-26T18:54:37.724 INFO:tasks.workunit.client.0.smithi102.stdout:5/643: readlink d2/d25/d75/db9/d43/d9e/l73 0 2024-03-26T18:54:37.724 INFO:tasks.workunit.client.0.smithi102.stdout:5/644: truncate d2/d25/f26 2202051 0 2024-03-26T18:54:37.724 INFO:tasks.workunit.client.0.smithi102.stdout:3/654: link df/d61/d55/c6e df/d61/dd6/ced 0 2024-03-26T18:54:37.726 INFO:tasks.workunit.client.0.smithi102.stdout:5/645: mknod d2/d25/d75/db9/d43/ce5 0 2024-03-26T18:54:37.726 INFO:tasks.workunit.client.0.smithi102.stdout:0/461: dwrite d6/de/d13/f5a [0,4194304] 0 2024-03-26T18:54:37.727 INFO:tasks.workunit.client.0.smithi102.stdout:9/403: dwrite df/d1a/d23/d4e/f7c [0,4194304] 0 2024-03-26T18:54:37.728 INFO:tasks.workunit.client.0.smithi102.stdout:5/646: mkdir d2/d25/de6 0 2024-03-26T18:54:37.741 INFO:tasks.workunit.client.0.smithi102.stdout:9/404: mknod df/d1a/d65/c9b 0 2024-03-26T18:54:37.743 INFO:tasks.workunit.client.0.smithi102.stdout:5/647: mkdir d2/d14/de7 0 2024-03-26T18:54:37.744 INFO:tasks.workunit.client.0.smithi102.stdout:0/462: getdents d6/d2d/d3d 0 2024-03-26T18:54:37.744 INFO:tasks.workunit.client.0.smithi102.stdout:0/463: chown d6/c23 3433364 1 2024-03-26T18:54:37.750 INFO:tasks.workunit.client.1.smithi190.stdout:4/786: read d7/d8/d14/d5c/ff7 [609,118069] 0 2024-03-26T18:54:37.750 INFO:tasks.workunit.client.1.smithi190.stdout:4/787: stat d7/d8/d14/d5c/f83 0 2024-03-26T18:54:37.752 INFO:tasks.workunit.client.0.smithi102.stdout:9/405: write df/d1a/f86 [2237548,109991] 0 2024-03-26T18:54:37.752 INFO:tasks.workunit.client.0.smithi102.stdout:9/406: chown df/d1a/l50 726916982 1 2024-03-26T18:54:37.752 INFO:tasks.workunit.client.0.smithi102.stdout:9/407: dread - df/d1a/d23/d29/f4c zero size 2024-03-26T18:54:37.753 INFO:tasks.workunit.client.0.smithi102.stdout:9/408: dread df/d1a/d23/d4e/d8e/f56 [0,4194304] 0 2024-03-26T18:54:37.756 INFO:tasks.workunit.client.0.smithi102.stdout:9/409: rmdir df/d46 39 2024-03-26T18:54:37.756 INFO:tasks.workunit.client.0.smithi102.stdout:9/410: dread - df/d1a/d23/d4e/f77 zero size 2024-03-26T18:54:37.756 INFO:tasks.workunit.client.0.smithi102.stdout:9/411: write df/f82 [66858,8481] 0 2024-03-26T18:54:37.757 INFO:tasks.workunit.client.0.smithi102.stdout:9/412: mkdir df/d1a/d23/d29/d9c 0 2024-03-26T18:54:37.757 INFO:tasks.workunit.client.0.smithi102.stdout:9/413: read - df/d1a/d23/f7e zero size 2024-03-26T18:54:37.759 INFO:tasks.workunit.client.0.smithi102.stdout:9/414: symlink df/d46/l9d 0 2024-03-26T18:54:37.761 INFO:tasks.workunit.client.0.smithi102.stdout:1/400: dread de/d16/d1d/d2a/d2c/d56/d6e/f7a [0,4194304] 0 2024-03-26T18:54:37.761 INFO:tasks.workunit.client.0.smithi102.stdout:1/401: readlink de/d1f/l71 0 2024-03-26T18:54:37.763 INFO:tasks.workunit.client.1.smithi190.stdout:2/773: dwrite d6/d9/d45/d6b/d93/dc0/ff2 [0,4194304] 0 2024-03-26T18:54:37.764 INFO:tasks.workunit.client.0.smithi102.stdout:1/402: rename de/l39 to de/d16/d1d/d2a/d2c/l84 0 2024-03-26T18:54:37.766 INFO:tasks.workunit.client.0.smithi102.stdout:1/403: creat de/d16/d1d/d2a/d2c/d56/d78/d80/f85 x:0 0 0 2024-03-26T18:54:37.766 INFO:tasks.workunit.client.0.smithi102.stdout:1/404: truncate de/d16/f34 225362 0 2024-03-26T18:54:37.767 INFO:tasks.workunit.client.1.smithi190.stdout:0/809: dwrite d0/f13 [0,4194304] 0 2024-03-26T18:54:37.769 INFO:tasks.workunit.client.1.smithi190.stdout:0/810: mkdir d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/d125 0 2024-03-26T18:54:37.769 INFO:tasks.workunit.client.1.smithi190.stdout:2/774: creat d6/d9/d10/d5f/f10d x:0 0 0 2024-03-26T18:54:37.773 INFO:tasks.workunit.client.1.smithi190.stdout:0/811: rmdir d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/d121 39 2024-03-26T18:54:37.773 INFO:tasks.workunit.client.1.smithi190.stdout:1/649: write d3/d73/d88/fad [2284105,60662] 0 2024-03-26T18:54:37.773 INFO:tasks.workunit.client.1.smithi190.stdout:1/650: chown d3/da/dd/d2f/d33/d9c/fb7 12939193 1 2024-03-26T18:54:37.779 INFO:tasks.workunit.client.1.smithi190.stdout:2/775: rename d6/d9/d3a/d3d/d67 to d6/d9/d3a/d3d/d2e/d10e 0 2024-03-26T18:54:37.779 INFO:tasks.workunit.client.1.smithi190.stdout:2/776: truncate d6/d9/d3a/d3d/d1c/d8d/dc7/fba 1004136 0 2024-03-26T18:54:37.788 INFO:tasks.workunit.client.1.smithi190.stdout:9/967: dwrite d12/d2d/d1d/d23/de5/ff4 [0,4194304] 0 2024-03-26T18:54:37.789 INFO:tasks.workunit.client.0.smithi102.stdout:6/408: dwrite d2/d3d/d53/f5b [0,4194304] 0 2024-03-26T18:54:37.789 INFO:tasks.workunit.client.0.smithi102.stdout:6/409: chown d2/d3d/d53/f66 3986 1 2024-03-26T18:54:37.798 INFO:tasks.workunit.client.1.smithi190.stdout:3/608: getdents d1/d3 0 2024-03-26T18:54:37.798 INFO:tasks.workunit.client.1.smithi190.stdout:3/609: chown d1/d3/df/c78 399 1 2024-03-26T18:54:37.798 INFO:tasks.workunit.client.1.smithi190.stdout:3/610: creat d1/d3/df/fc0 x:0 0 0 2024-03-26T18:54:37.800 INFO:tasks.workunit.client.0.smithi102.stdout:6/410: creat d2/d7/d3c/d78/f7b x:0 0 0 2024-03-26T18:54:37.800 INFO:tasks.workunit.client.0.smithi102.stdout:6/411: dread - d2/df/d35/d43/f57 zero size 2024-03-26T18:54:37.802 INFO:tasks.workunit.client.1.smithi190.stdout:3/611: creat d1/de/d81/fc1 x:0 0 0 2024-03-26T18:54:37.804 INFO:tasks.workunit.client.0.smithi102.stdout:6/412: creat d2/d3d/d53/d63/f7c x:0 0 0 2024-03-26T18:54:37.804 INFO:tasks.workunit.client.0.smithi102.stdout:0/464: dwrite d6/d3e/f6a [0,4194304] 0 2024-03-26T18:54:37.805 INFO:tasks.workunit.client.0.smithi102.stdout:6/413: truncate d2/d3/f38 4166856 0 2024-03-26T18:54:37.805 INFO:tasks.workunit.client.0.smithi102.stdout:6/414: creat d2/d3d/f7d x:0 0 0 2024-03-26T18:54:37.807 INFO:tasks.workunit.client.1.smithi190.stdout:9/968: read d12/d2d/d1d/d23/d83/dc9/dd4/d3f/d55/f62 [1428980,90441] 0 2024-03-26T18:54:37.809 INFO:tasks.workunit.client.1.smithi190.stdout:9/969: rename d12/d80/db2/de1/f155 to d12/d2d/f15b 0 2024-03-26T18:54:37.809 INFO:tasks.workunit.client.1.smithi190.stdout:9/970: fdatasync d12/d2d/f39 0 2024-03-26T18:54:37.809 INFO:tasks.workunit.client.1.smithi190.stdout:9/971: write d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/f12e [329242,59264] 0 2024-03-26T18:54:37.809 INFO:tasks.workunit.client.1.smithi190.stdout:9/972: getdents d12/d2d/d1d/d23/d83/dc9/dd4/d3f/d55 0 2024-03-26T18:54:37.814 INFO:tasks.workunit.client.1.smithi190.stdout:7/820: dwrite d3/d6/f8c [0,4194304] 0 2024-03-26T18:54:37.815 INFO:tasks.workunit.client.1.smithi190.stdout:7/821: creat d3/d1b/dd3/d10d/d50/dde/f133 x:0 0 0 2024-03-26T18:54:37.820 INFO:tasks.workunit.client.1.smithi190.stdout:3/612: write d1/fb [6659132,125865] 0 2024-03-26T18:54:37.821 INFO:tasks.workunit.client.1.smithi190.stdout:4/788: dwrite d7/d1d/fb5 [0,4194304] 0 2024-03-26T18:54:37.822 INFO:tasks.workunit.client.0.smithi102.stdout:0/465: dread d6/d2d/f38 [0,4194304] 0 2024-03-26T18:54:37.823 INFO:tasks.workunit.client.1.smithi190.stdout:0/812: dwrite d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/f95 [0,4194304] 0 2024-03-26T18:54:37.823 INFO:tasks.workunit.client.1.smithi190.stdout:0/813: write d0/dd/d20/d1c/d8b/d8d/fd2 [1217475,67183] 0 2024-03-26T18:54:37.823 INFO:tasks.workunit.client.1.smithi190.stdout:0/814: write d0/dd/d20/d1c/d46/da7/df0/f111 [348818,116570] 0 2024-03-26T18:54:37.824 INFO:tasks.workunit.client.0.smithi102.stdout:2/381: sync 2024-03-26T18:54:37.824 INFO:tasks.workunit.client.0.smithi102.stdout:4/332: sync 2024-03-26T18:54:37.824 INFO:tasks.workunit.client.0.smithi102.stdout:7/399: sync 2024-03-26T18:54:37.824 INFO:tasks.workunit.client.1.smithi190.stdout:1/651: dwrite d3/d9/f1c [0,4194304] 0 2024-03-26T18:54:37.826 INFO:tasks.workunit.client.0.smithi102.stdout:2/382: mkdir d3/d82 0 2024-03-26T18:54:37.826 INFO:tasks.workunit.client.1.smithi190.stdout:9/973: write d12/d2d/d1d/d23/d83/dc9/dd4/d71/f141 [2098052,6716] 0 2024-03-26T18:54:37.827 INFO:tasks.workunit.client.1.smithi190.stdout:5/638: dwrite d2/d17/d62/d64/fbf [0,4194304] 0 2024-03-26T18:54:37.828 INFO:tasks.workunit.client.0.smithi102.stdout:7/400: creat d7/d10/d27/d34/d72/f76 x:0 0 0 2024-03-26T18:54:37.828 INFO:tasks.workunit.client.0.smithi102.stdout:7/401: chown d7/d12/d2b 3005 1 2024-03-26T18:54:37.829 INFO:tasks.workunit.client.1.smithi190.stdout:4/789: rename d7/d8/d1c/d20 to d7/d8/d14/d23/d28/d38/d96/d10f 0 2024-03-26T18:54:37.829 INFO:tasks.workunit.client.0.smithi102.stdout:5/648: dwrite d2/d25/d75/db9/f5c [0,4194304] 0 2024-03-26T18:54:37.830 INFO:tasks.workunit.client.0.smithi102.stdout:2/383: creat d3/d11/d15/d31/f83 x:0 0 0 2024-03-26T18:54:37.831 INFO:tasks.workunit.client.0.smithi102.stdout:9/415: dwrite df/d1a/f6e [0,4194304] 0 2024-03-26T18:54:37.832 INFO:tasks.workunit.client.0.smithi102.stdout:9/416: dread df/d1a/d23/d3e/d75/f4a [0,4194304] 0 2024-03-26T18:54:37.832 INFO:tasks.workunit.client.0.smithi102.stdout:9/417: readlink l5 0 2024-03-26T18:54:37.833 INFO:tasks.workunit.client.1.smithi190.stdout:4/790: truncate d7/d8/d1c/f9f 3922502 0 2024-03-26T18:54:37.833 INFO:tasks.workunit.client.1.smithi190.stdout:1/652: mknod d3/da/dd/d2f/d33/d9c/cd6 0 2024-03-26T18:54:37.837 INFO:tasks.workunit.client.1.smithi190.stdout:7/822: getdents d3/d1b/dd3/d10d/d50/dde 0 2024-03-26T18:54:37.839 INFO:tasks.workunit.client.0.smithi102.stdout:5/649: mknod d2/d25/d75/db9/db3/ce8 0 2024-03-26T18:54:37.840 INFO:tasks.workunit.client.1.smithi190.stdout:7/823: creat d3/d1b/dd3/d10d/d65/d90/d92/d12d/f134 x:0 0 0 2024-03-26T18:54:37.840 INFO:tasks.workunit.client.0.smithi102.stdout:4/333: write d1/d17/d1a/f53 [781913,42203] 0 2024-03-26T18:54:37.841 INFO:tasks.workunit.client.1.smithi190.stdout:7/824: dread d3/d6/d9/d14/f18 [0,4194304] 0 2024-03-26T18:54:37.841 INFO:tasks.workunit.client.1.smithi190.stdout:7/825: truncate d3/d6/d9/d1f/d4c/d71/df6/f127 259045 0 2024-03-26T18:54:37.841 INFO:tasks.workunit.client.1.smithi190.stdout:7/826: chown d3/f19 407639889 1 2024-03-26T18:54:37.842 INFO:tasks.workunit.client.1.smithi190.stdout:2/777: dwrite d6/d9/d3a/f3b [0,4194304] 0 2024-03-26T18:54:37.842 INFO:tasks.workunit.client.1.smithi190.stdout:2/778: fsync d6/d9/d10/db7/dcf/f101 0 2024-03-26T18:54:37.843 INFO:tasks.workunit.client.0.smithi102.stdout:2/384: rename d3/d65 to d3/d20/d78/d84 0 2024-03-26T18:54:37.843 INFO:tasks.workunit.client.0.smithi102.stdout:6/415: fsync d2/d3d/d53/d63/f7c 0 2024-03-26T18:54:37.845 INFO:tasks.workunit.client.1.smithi190.stdout:5/639: link d2/d17/d98/d3a/d3c/lc3 d2/d17/d1b/db6/d24/d3b/da7/ld2 0 2024-03-26T18:54:37.845 INFO:tasks.workunit.client.1.smithi190.stdout:5/640: chown d2/f14 63 1 2024-03-26T18:54:37.846 INFO:tasks.workunit.client.1.smithi190.stdout:4/791: dread d7/d8/d1c/f1f [0,4194304] 0 2024-03-26T18:54:37.846 INFO:tasks.workunit.client.1.smithi190.stdout:4/792: write d7/d8/d1c/f1f [2506580,62255] 0 2024-03-26T18:54:37.846 INFO:tasks.workunit.client.1.smithi190.stdout:4/793: fsync d7/d8/d14/d5c/fe5 0 2024-03-26T18:54:37.846 INFO:tasks.workunit.client.1.smithi190.stdout:4/794: creat d7/d8/d14/d4b/d7d/da1/f110 x:0 0 0 2024-03-26T18:54:37.848 INFO:tasks.workunit.client.1.smithi190.stdout:0/815: rename d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/fdb to d0/dd/d20/d1c/d46/da7/df0/d4b/f126 0 2024-03-26T18:54:37.848 INFO:tasks.workunit.client.1.smithi190.stdout:0/816: fdatasync d0/dd/d20/d1c/d46/da7/df0/da2/fc0 0 2024-03-26T18:54:37.848 INFO:tasks.workunit.client.1.smithi190.stdout:0/817: read - d0/dd/d20/d1c/d8b/fd9 zero size 2024-03-26T18:54:37.848 INFO:tasks.workunit.client.1.smithi190.stdout:4/795: symlink d7/d1d/d41/l111 0 2024-03-26T18:54:37.850 INFO:tasks.workunit.client.1.smithi190.stdout:1/653: mkdir d3/d9/db/dd4/dd7 0 2024-03-26T18:54:37.850 INFO:tasks.workunit.client.1.smithi190.stdout:2/779: rename d6/d9/d3a/d3d/d2e/f9a to d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d8b/f10f 0 2024-03-26T18:54:37.851 INFO:tasks.workunit.client.0.smithi102.stdout:9/418: creat df/d84/f9e x:0 0 0 2024-03-26T18:54:37.852 INFO:tasks.workunit.client.1.smithi190.stdout:2/780: dread d6/d9/f75 [0,4194304] 0 2024-03-26T18:54:37.853 INFO:tasks.workunit.client.1.smithi190.stdout:2/781: creat d6/d9/d48/da0/f110 x:0 0 0 2024-03-26T18:54:37.854 INFO:tasks.workunit.client.0.smithi102.stdout:5/650: mkdir d2/d25/d2a/d4f/ddc/de9 0 2024-03-26T18:54:37.855 INFO:tasks.workunit.client.1.smithi190.stdout:0/818: creat d0/dd/d20/d1c/d46/da7/df0/d87/d93/db5/f127 x:0 0 0 2024-03-26T18:54:37.855 INFO:tasks.workunit.client.1.smithi190.stdout:0/819: dread - d0/dd/d20/d1c/d46/da7/df0/da2/d80/f124 zero size 2024-03-26T18:54:37.856 INFO:tasks.workunit.client.0.smithi102.stdout:4/334: mkdir d1/d17/d1a/d21/d6d 0 2024-03-26T18:54:37.857 INFO:tasks.workunit.client.1.smithi190.stdout:7/827: dread d3/d1b/dd3/d10d/d50/f52 [0,4194304] 0 2024-03-26T18:54:37.857 INFO:tasks.workunit.client.1.smithi190.stdout:5/641: rename d2/d17/d1b/d60/d63/d85/l9b to d2/d17/d1b/d60/d63/dcd/ld3 0 2024-03-26T18:54:37.858 INFO:tasks.workunit.client.0.smithi102.stdout:2/385: link d3/f1c d3/d20/d78/d7c/f85 0 2024-03-26T18:54:37.861 INFO:tasks.workunit.client.0.smithi102.stdout:6/416: symlink d2/df/d35/d43/d75/l7e 0 2024-03-26T18:54:37.861 INFO:tasks.workunit.client.1.smithi190.stdout:2/782: rename d6/d9/d10/f2c to d6/d9/d3a/d3d/d2e/d86/d10b/f111 0 2024-03-26T18:54:37.862 INFO:tasks.workunit.client.0.smithi102.stdout:4/335: creat d1/d27/f6e x:0 0 0 2024-03-26T18:54:37.863 INFO:tasks.workunit.client.1.smithi190.stdout:2/783: creat d6/d9/d45/d6b/deb/f112 x:0 0 0 2024-03-26T18:54:37.865 INFO:tasks.workunit.client.1.smithi190.stdout:2/784: mknod d6/d21/dd0/c113 0 2024-03-26T18:54:37.868 INFO:tasks.workunit.client.1.smithi190.stdout:2/785: rename d6/d9/d3a/d3d/c6a to d6/d9/d3a/d3d/d2e/d10e/d9e/dfc/c114 0 2024-03-26T18:54:37.870 INFO:tasks.workunit.client.1.smithi190.stdout:1/654: dread d3/da/d2e/f4a [0,4194304] 0 2024-03-26T18:54:37.870 INFO:tasks.workunit.client.1.smithi190.stdout:1/655: chown d3/da/f47 6521 1 2024-03-26T18:54:37.870 INFO:tasks.workunit.client.1.smithi190.stdout:1/656: dread - d3/da/dd/d2f/d5b/db5/fc7 zero size 2024-03-26T18:54:37.870 INFO:tasks.workunit.client.1.smithi190.stdout:1/657: write d3/da/dd/d95/fb8 [225303,4990] 0 2024-03-26T18:54:37.872 INFO:tasks.workunit.client.1.smithi190.stdout:1/658: rmdir d3/da/dd/d95/d52 39 2024-03-26T18:54:37.872 INFO:tasks.workunit.client.1.smithi190.stdout:1/659: fdatasync f2 0 2024-03-26T18:54:37.872 INFO:tasks.workunit.client.1.smithi190.stdout:1/660: dread - d3/da/dd/d2f/d5b/f90 zero size 2024-03-26T18:54:37.872 INFO:tasks.workunit.client.1.smithi190.stdout:1/661: truncate d3/d73/d88/fad 5164563 0 2024-03-26T18:54:37.874 INFO:tasks.workunit.client.1.smithi190.stdout:3/613: dwrite d1/d3/df/d18/d35/d41/dae/f93 [0,4194304] 0 2024-03-26T18:54:37.875 INFO:tasks.workunit.client.1.smithi190.stdout:3/614: readlink d1/l23 0 2024-03-26T18:54:37.875 INFO:tasks.workunit.client.1.smithi190.stdout:3/615: creat d1/de/d8a/fc2 x:0 0 0 2024-03-26T18:54:37.875 INFO:tasks.workunit.client.1.smithi190.stdout:3/616: chown d1/de/d1c/d36/f3a 93 1 2024-03-26T18:54:37.875 INFO:tasks.workunit.client.1.smithi190.stdout:1/662: unlink d3/da/dd/d95/ca4 0 2024-03-26T18:54:37.878 INFO:tasks.workunit.client.0.smithi102.stdout:7/402: dwrite d7/d12/d4d/d62/f66 [0,4194304] 0 2024-03-26T18:54:37.878 INFO:tasks.workunit.client.0.smithi102.stdout:7/403: fsync d7/d31/f45 0 2024-03-26T18:54:37.880 INFO:tasks.workunit.client.0.smithi102.stdout:4/336: dread d1/d27/f3c [0,4194304] 0 2024-03-26T18:54:37.880 INFO:tasks.workunit.client.0.smithi102.stdout:4/337: chown d1/d25/d39/f5e 16751443 1 2024-03-26T18:54:37.881 INFO:tasks.workunit.client.1.smithi190.stdout:3/617: unlink d1/d3/df/c78 0 2024-03-26T18:54:37.881 INFO:tasks.workunit.client.1.smithi190.stdout:4/796: dread d7/d8/d14/d23/d28/d38/fcf [0,4194304] 0 2024-03-26T18:54:37.886 INFO:tasks.workunit.client.1.smithi190.stdout:4/797: creat d7/d8/d14/d5c/f112 x:0 0 0 2024-03-26T18:54:37.886 INFO:tasks.workunit.client.1.smithi190.stdout:4/798: chown d7/d1d/d42/l85 0 1 2024-03-26T18:54:37.886 INFO:tasks.workunit.client.1.smithi190.stdout:0/820: write d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/f95 [1623880,81597] 0 2024-03-26T18:54:37.887 INFO:tasks.workunit.client.1.smithi190.stdout:9/974: dwrite d12/d2d/f9f [0,4194304] 0 2024-03-26T18:54:37.887 INFO:tasks.workunit.client.1.smithi190.stdout:3/618: symlink d1/d3/df/d18/d35/d65/lc3 0 2024-03-26T18:54:37.888 INFO:tasks.workunit.client.0.smithi102.stdout:7/404: mknod d7/d12/c77 0 2024-03-26T18:54:37.888 INFO:tasks.workunit.client.0.smithi102.stdout:7/405: write d7/d10/d27/f33 [3322985,103135] 0 2024-03-26T18:54:37.889 INFO:tasks.workunit.client.0.smithi102.stdout:7/406: write d7/d10/d27/d34/d72/f76 [417823,63222] 0 2024-03-26T18:54:37.889 INFO:tasks.workunit.client.0.smithi102.stdout:7/407: chown d7/d10/f1c 2132 1 2024-03-26T18:54:37.890 INFO:tasks.workunit.client.1.smithi190.stdout:2/786: dread d6/d9/d48/f7e [0,4194304] 0 2024-03-26T18:54:37.890 INFO:tasks.workunit.client.0.smithi102.stdout:7/408: mkdir d7/d10/d67/d69/d78 0 2024-03-26T18:54:37.890 INFO:tasks.workunit.client.0.smithi102.stdout:7/409: dread - d7/d12/d51/f58 zero size 2024-03-26T18:54:37.893 INFO:tasks.workunit.client.0.smithi102.stdout:9/419: dwrite df/d1a/d1d/f26 [0,4194304] 0 2024-03-26T18:54:37.894 INFO:tasks.workunit.client.1.smithi190.stdout:0/821: mkdir d0/dd/d128 0 2024-03-26T18:54:37.897 INFO:tasks.workunit.client.1.smithi190.stdout:3/619: mkdir d1/d3/df/d31/dc4 0 2024-03-26T18:54:37.898 INFO:tasks.workunit.client.1.smithi190.stdout:1/663: dread d3/da/f24 [4194304,4194304] 0 2024-03-26T18:54:37.898 INFO:tasks.workunit.client.1.smithi190.stdout:9/975: unlink d12/d2d/d1d/d23/d83/d91/fae 0 2024-03-26T18:54:37.899 INFO:tasks.workunit.client.1.smithi190.stdout:2/787: getdents d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d7c/dac 0 2024-03-26T18:54:37.899 INFO:tasks.workunit.client.1.smithi190.stdout:2/788: fdatasync d6/d9/d10/f85 0 2024-03-26T18:54:37.902 INFO:tasks.workunit.client.1.smithi190.stdout:3/620: link d1/d3/df/d18/d35/d41/dae/f7a d1/d3/df/d31/dc4/fc5 0 2024-03-26T18:54:37.902 INFO:tasks.workunit.client.1.smithi190.stdout:7/828: dwrite d3/d6/d9/d14/f18 [0,4194304] 0 2024-03-26T18:54:37.902 INFO:tasks.workunit.client.1.smithi190.stdout:1/664: creat d3/da/dd/d95/d52/fd8 x:0 0 0 2024-03-26T18:54:37.903 INFO:tasks.workunit.client.0.smithi102.stdout:5/651: dwrite d2/f3a [0,4194304] 0 2024-03-26T18:54:37.903 INFO:tasks.workunit.client.0.smithi102.stdout:5/652: chown d2/d2c/d6f/f7a 3 1 2024-03-26T18:54:37.903 INFO:tasks.workunit.client.1.smithi190.stdout:9/976: unlink d12/d2d/d1d/d54/l9d 0 2024-03-26T18:54:37.904 INFO:tasks.workunit.client.1.smithi190.stdout:5/642: dwrite d2/f4 [0,4194304] 0 2024-03-26T18:54:37.907 INFO:tasks.workunit.client.1.smithi190.stdout:3/621: rename d1/d47/l7f to d1/d3/df/d18/d35/d41/d61/lc6 0 2024-03-26T18:54:37.907 INFO:tasks.workunit.client.1.smithi190.stdout:1/665: mkdir d3/dd9 0 2024-03-26T18:54:37.908 INFO:tasks.workunit.client.1.smithi190.stdout:7/829: mknod d3/d6/d9/d14/d59/d73/c135 0 2024-03-26T18:54:37.911 INFO:tasks.workunit.client.1.smithi190.stdout:9/977: symlink d12/d2d/dfc/l15c 0 2024-03-26T18:54:37.912 INFO:tasks.workunit.client.1.smithi190.stdout:7/830: unlink d3/d6/d9/d1f/d4c/cf3 0 2024-03-26T18:54:37.912 INFO:tasks.workunit.client.1.smithi190.stdout:7/831: readlink d3/d1b/dd3/d10d/d65/d90/l131 0 2024-03-26T18:54:37.913 INFO:tasks.workunit.client.1.smithi190.stdout:9/978: mknod d12/d2d/dfc/c15d 0 2024-03-26T18:54:37.913 INFO:tasks.workunit.client.1.smithi190.stdout:9/979: dread - d12/d2d/d1d/d23/d83/dc9/dd4/d3f/ffd zero size 2024-03-26T18:54:37.915 INFO:tasks.workunit.client.1.smithi190.stdout:7/832: symlink d3/d1b/d57/dd1/l136 0 2024-03-26T18:54:37.917 INFO:tasks.workunit.client.0.smithi102.stdout:5/653: dread d2/d25/d75/db9/d43/d9e/d82/fa2 [0,4194304] 0 2024-03-26T18:54:37.917 INFO:tasks.workunit.client.1.smithi190.stdout:7/833: getdents d3/d1b/dd3/d10d/d65/d109/d11b 0 2024-03-26T18:54:37.918 INFO:tasks.workunit.client.0.smithi102.stdout:6/417: dwrite d2/d3d/d53/d5c/f65 [0,4194304] 0 2024-03-26T18:54:37.919 INFO:tasks.workunit.client.1.smithi190.stdout:9/980: dread d12/faf [0,4194304] 0 2024-03-26T18:54:37.919 INFO:tasks.workunit.client.1.smithi190.stdout:7/834: symlink d3/d1b/d57/dd1/l137 0 2024-03-26T18:54:37.919 INFO:tasks.workunit.client.1.smithi190.stdout:7/835: chown d3/d6/d9/l10 2 1 2024-03-26T18:54:37.922 INFO:tasks.workunit.client.1.smithi190.stdout:9/981: symlink d12/d2d/d1d/d23/de5/d122/d12c/l15e 0 2024-03-26T18:54:37.923 INFO:tasks.workunit.client.0.smithi102.stdout:5/654: rename d2/d25/d2a to d2/dea 0 2024-03-26T18:54:37.924 INFO:tasks.workunit.client.0.smithi102.stdout:5/655: dread d2/d25/d75/db9/d43/f92 [0,4194304] 0 2024-03-26T18:54:37.924 INFO:tasks.workunit.client.1.smithi190.stdout:7/836: symlink d3/d6/d9/d1f/d93/dfb/d102/l138 0 2024-03-26T18:54:37.925 INFO:tasks.workunit.client.0.smithi102.stdout:6/418: symlink d2/df/d35/d43/d75/l7f 0 2024-03-26T18:54:37.926 INFO:tasks.workunit.client.0.smithi102.stdout:5/656: mknod d2/d2c/d6f/db8/ceb 0 2024-03-26T18:54:37.928 INFO:tasks.workunit.client.1.smithi190.stdout:7/837: rename d3/d6/d9/f45 to d3/d1b/dd3/d10d/d65/d109/d119/f139 0 2024-03-26T18:54:37.928 INFO:tasks.workunit.client.1.smithi190.stdout:7/838: chown d3/d6/d9/d1f/d4c/d71/df6/f127 3845 1 2024-03-26T18:54:37.928 INFO:tasks.workunit.client.1.smithi190.stdout:7/839: chown d3/d1b/f66 55008967 1 2024-03-26T18:54:37.929 INFO:tasks.workunit.client.0.smithi102.stdout:6/419: mkdir d2/df/d71/d80 0 2024-03-26T18:54:37.929 INFO:tasks.workunit.client.0.smithi102.stdout:6/420: stat d2/df/f64 0 2024-03-26T18:54:37.930 INFO:tasks.workunit.client.0.smithi102.stdout:6/421: chown d2/df/d35/d43/d75 4394872 1 2024-03-26T18:54:37.930 INFO:tasks.workunit.client.0.smithi102.stdout:6/422: dread - d2/d3d/f70 zero size 2024-03-26T18:54:37.930 INFO:tasks.workunit.client.1.smithi190.stdout:7/840: mknod d3/d1b/dd3/d10d/d50/d117/c13a 0 2024-03-26T18:54:37.931 INFO:tasks.workunit.client.0.smithi102.stdout:5/657: mkdir d2/d25/d75/db9/d43/d9e/d82/dc9/dd8/dec 0 2024-03-26T18:54:37.932 INFO:tasks.workunit.client.0.smithi102.stdout:5/658: readlink d2/l88 0 2024-03-26T18:54:37.932 INFO:tasks.workunit.client.0.smithi102.stdout:5/659: chown d2/d14/c33 10659444 1 2024-03-26T18:54:37.940 INFO:tasks.workunit.client.1.smithi190.stdout:9/982: dread d12/d2d/d1d/d23/d83/dc9/dd4/f6c [0,4194304] 0 2024-03-26T18:54:37.940 INFO:tasks.workunit.client.1.smithi190.stdout:9/983: chown d12/d2d/d1d/d23/d83/dc9/dd4/d71/f85 142246339 1 2024-03-26T18:54:37.940 INFO:tasks.workunit.client.1.smithi190.stdout:4/799: dwrite d7/d1d/fb5 [0,4194304] 0 2024-03-26T18:54:37.944 INFO:tasks.workunit.client.1.smithi190.stdout:4/800: mkdir d7/d8/d14/d4b/d7d/da1/d113 0 2024-03-26T18:54:37.947 INFO:tasks.workunit.client.1.smithi190.stdout:2/789: link d6/d9/d3a/d3d/d2e/d86/d10b/f111 d6/d9/d3a/d3d/d2e/d10e/d9e/d10a/f115 0 2024-03-26T18:54:37.947 INFO:tasks.workunit.client.1.smithi190.stdout:2/790: dread - d6/d9/d3a/d3d/f57 zero size 2024-03-26T18:54:37.949 INFO:tasks.workunit.client.1.smithi190.stdout:4/801: getdents d7/d8/d14/d23/d28 0 2024-03-26T18:54:37.951 INFO:tasks.workunit.client.1.smithi190.stdout:4/802: rmdir d7/d8/d14/d4b/d7d 39 2024-03-26T18:54:37.958 INFO:tasks.workunit.client.0.smithi102.stdout:4/338: dwrite d1/d17/d1a/d21/f56 [0,4194304] 0 2024-03-26T18:54:37.962 INFO:tasks.workunit.client.0.smithi102.stdout:4/339: mkdir d1/d17/d1f/d5d/d6f 0 2024-03-26T18:54:37.962 INFO:tasks.workunit.client.0.smithi102.stdout:4/340: write d1/d27/f4d [317930,19578] 0 2024-03-26T18:54:37.964 INFO:tasks.workunit.client.1.smithi190.stdout:2/791: dread d6/d9/d10/db7/dcf/fd4 [0,4194304] 0 2024-03-26T18:54:37.964 INFO:tasks.workunit.client.1.smithi190.stdout:2/792: fdatasync d6/f8f 0 2024-03-26T18:54:37.965 INFO:tasks.workunit.client.1.smithi190.stdout:2/793: fdatasync d6/d9/d3a/d3d/d55/f92 0 2024-03-26T18:54:37.969 INFO:tasks.workunit.client.0.smithi102.stdout:5/660: dread d2/d25/d75/db9/d43/d9e/d8b/fab [0,4194304] 0 2024-03-26T18:54:37.971 INFO:tasks.workunit.client.1.smithi190.stdout:1/666: dwrite d3/da/dd/d2f/f5e [4194304,4194304] 0 2024-03-26T18:54:37.973 INFO:tasks.workunit.client.0.smithi102.stdout:4/341: dread d1/d17/d1a/f4f [0,4194304] 0 2024-03-26T18:54:37.973 INFO:tasks.workunit.client.0.smithi102.stdout:4/342: dread - d1/d25/f69 zero size 2024-03-26T18:54:37.974 INFO:tasks.workunit.client.1.smithi190.stdout:4/803: dread d7/d8/d14/f19 [4194304,4194304] 0 2024-03-26T18:54:37.974 INFO:tasks.workunit.client.1.smithi190.stdout:5/643: dwrite d2/d17/d62/d64/f70 [0,4194304] 0 2024-03-26T18:54:37.975 INFO:tasks.workunit.client.0.smithi102.stdout:4/343: rename d1/d17/d2a/c42 to d1/d17/d1a/d21/d6d/c70 0 2024-03-26T18:54:37.975 INFO:tasks.workunit.client.0.smithi102.stdout:4/344: write d1/d17/d1a/d21/f64 [1383167,127122] 0 2024-03-26T18:54:37.975 INFO:tasks.workunit.client.1.smithi190.stdout:5/644: write d2/d17/d1b/db6/d88/f8b [284285,62627] 0 2024-03-26T18:54:37.976 INFO:tasks.workunit.client.1.smithi190.stdout:4/804: mknod d7/d8/d14/d23/d28/d38/d96/d9a/c114 0 2024-03-26T18:54:37.978 INFO:tasks.workunit.client.1.smithi190.stdout:4/805: truncate d7/d8/d14/d23/d28/d38/f71 955700 0 2024-03-26T18:54:37.978 INFO:tasks.workunit.client.1.smithi190.stdout:4/806: stat d7/d8/d14/d23/l8e 0 2024-03-26T18:54:37.978 INFO:tasks.workunit.client.1.smithi190.stdout:4/807: write d7/d8/f2a [2930785,90185] 0 2024-03-26T18:54:37.979 INFO:tasks.workunit.client.0.smithi102.stdout:4/345: unlink d1/f7 0 2024-03-26T18:54:37.980 INFO:tasks.workunit.client.1.smithi190.stdout:4/808: creat d7/d1d/d42/f115 x:0 0 0 2024-03-26T18:54:37.982 INFO:tasks.workunit.client.0.smithi102.stdout:4/346: creat d1/d17/d1a/d21/d6d/f71 x:0 0 0 2024-03-26T18:54:37.983 INFO:tasks.workunit.client.1.smithi190.stdout:4/809: rename d7/d8/d14/d6e/ce6 to d7/d1d/d5e/c116 0 2024-03-26T18:54:37.983 INFO:tasks.workunit.client.1.smithi190.stdout:4/810: write d7/d8/d14/d4b/d7d/da1/f110 [387298,84632] 0 2024-03-26T18:54:37.983 INFO:tasks.workunit.client.1.smithi190.stdout:4/811: truncate d7/d1d/fff 713549 0 2024-03-26T18:54:37.990 INFO:tasks.workunit.client.1.smithi190.stdout:1/667: dread d3/d9/db/f32 [4194304,4194304] 0 2024-03-26T18:54:37.993 INFO:tasks.workunit.client.0.smithi102.stdout:9/420: dwrite df/d1a/d57/f8f [0,4194304] 0 2024-03-26T18:54:37.994 INFO:tasks.workunit.client.1.smithi190.stdout:3/622: dwrite d1/d3/df/d18/d57/d91/fbc [0,4194304] 0 2024-03-26T18:54:37.996 INFO:tasks.workunit.client.1.smithi190.stdout:0/822: dwrite d0/dd/d20/d1c/d8b/d8d/db7/d118/f11a [0,4194304] 0 2024-03-26T18:54:37.997 INFO:tasks.workunit.client.1.smithi190.stdout:3/623: symlink d1/d3/df/d18/d35/lc7 0 2024-03-26T18:54:37.999 INFO:tasks.workunit.client.1.smithi190.stdout:1/668: write f2 [4440522,109321] 0 2024-03-26T18:54:37.999 INFO:tasks.workunit.client.1.smithi190.stdout:1/669: dread - d3/da/dd/d2f/d5b/d5d/fc0 zero size 2024-03-26T18:54:38.000 INFO:tasks.workunit.client.0.smithi102.stdout:7/410: dwrite d7/d31/d56/f63 [0,4194304] 0 2024-03-26T18:54:38.000 INFO:tasks.workunit.client.0.smithi102.stdout:7/411: chown d7/d10 19 1 2024-03-26T18:54:38.001 INFO:tasks.workunit.client.1.smithi190.stdout:0/823: creat d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/f129 x:0 0 0 2024-03-26T18:54:38.001 INFO:tasks.workunit.client.1.smithi190.stdout:9/984: dwrite d12/d80/f144 [0,4194304] 0 2024-03-26T18:54:38.002 INFO:tasks.workunit.client.0.smithi102.stdout:9/421: dread df/f40 [0,4194304] 0 2024-03-26T18:54:38.003 INFO:tasks.workunit.client.0.smithi102.stdout:9/422: stat df/d1a/d23/d3e/d75 0 2024-03-26T18:54:38.003 INFO:tasks.workunit.client.0.smithi102.stdout:9/423: creat df/d1a/d23/d3e/f9f x:0 0 0 2024-03-26T18:54:38.003 INFO:tasks.workunit.client.1.smithi190.stdout:1/670: mkdir d3/da/dd/d95/dda 0 2024-03-26T18:54:38.004 INFO:tasks.workunit.client.1.smithi190.stdout:1/671: chown d3/da/dd/d95/dda 1617665 1 2024-03-26T18:54:38.005 INFO:tasks.workunit.client.0.smithi102.stdout:7/412: mkdir d7/d31/d79 0 2024-03-26T18:54:38.005 INFO:tasks.workunit.client.0.smithi102.stdout:7/413: fdatasync d7/d12/d4d/d62/f65 0 2024-03-26T18:54:38.005 INFO:tasks.workunit.client.0.smithi102.stdout:7/414: truncate d7/d10/d17/f5c 728947 0 2024-03-26T18:54:38.005 INFO:tasks.workunit.client.0.smithi102.stdout:7/415: write d7/d2a/f60 [504581,69996] 0 2024-03-26T18:54:38.006 INFO:tasks.workunit.client.1.smithi190.stdout:0/824: creat d0/dd/d20/d1c/d8b/d8d/d8f/f12a x:0 0 0 2024-03-26T18:54:38.006 INFO:tasks.workunit.client.0.smithi102.stdout:9/424: symlink df/d1a/d1d/la0 0 2024-03-26T18:54:38.007 INFO:tasks.workunit.client.0.smithi102.stdout:9/425: write df/d1a/d65/f73 [2207170,39060] 0 2024-03-26T18:54:38.007 INFO:tasks.workunit.client.1.smithi190.stdout:1/672: dread d3/d9/db/f32 [4194304,4194304] 0 2024-03-26T18:54:38.008 INFO:tasks.workunit.client.1.smithi190.stdout:0/825: mkdir d0/dd/d20/d12b 0 2024-03-26T18:54:38.008 INFO:tasks.workunit.client.1.smithi190.stdout:1/673: write d3/fab [844505,92324] 0 2024-03-26T18:54:38.008 INFO:tasks.workunit.client.1.smithi190.stdout:1/674: write d3/da/dd/d2f/d5b/d5d/fc0 [907588,36564] 0 2024-03-26T18:54:38.008 INFO:tasks.workunit.client.1.smithi190.stdout:7/841: dwrite d3/d1b/f5e [0,4194304] 0 2024-03-26T18:54:38.009 INFO:tasks.workunit.client.1.smithi190.stdout:2/794: dwrite d6/d9/d10/fd9 [0,4194304] 0 2024-03-26T18:54:38.014 INFO:tasks.workunit.client.1.smithi190.stdout:7/842: rename d3/d6/d9/d1f/d93/f125 to d3/d6/d9/d1f/d4c/d71/dcc/f13b 0 2024-03-26T18:54:38.014 INFO:tasks.workunit.client.1.smithi190.stdout:7/843: fsync d3/d1b/dd3/d10d/d50/f89 0 2024-03-26T18:54:38.016 INFO:tasks.workunit.client.1.smithi190.stdout:7/844: link d3/d1b/d57/dd1/l136 d3/d6/d9/d1f/d93/dfb/d102/d99/l13c 0 2024-03-26T18:54:38.016 INFO:tasks.workunit.client.1.smithi190.stdout:7/845: fdatasync d3/d6/d9/d14/d36/f72 0 2024-03-26T18:54:38.017 INFO:tasks.workunit.client.0.smithi102.stdout:9/426: write df/d1a/f1c [1228884,94596] 0 2024-03-26T18:54:38.017 INFO:tasks.workunit.client.1.smithi190.stdout:3/624: write d1/d3/f10 [5075302,13142] 0 2024-03-26T18:54:38.018 INFO:tasks.workunit.client.1.smithi190.stdout:7/846: symlink d3/d6/l13d 0 2024-03-26T18:54:38.019 INFO:tasks.workunit.client.0.smithi102.stdout:9/427: unlink df/d46/l49 0 2024-03-26T18:54:38.020 INFO:tasks.workunit.client.1.smithi190.stdout:9/985: dread d12/d2d/d1d/da1/dba/fd0 [0,4194304] 0 2024-03-26T18:54:38.020 INFO:tasks.workunit.client.1.smithi190.stdout:9/986: write d12/d2d/d1d/d23/d83/dc9/dd4/d3f/f154 [694207,63428] 0 2024-03-26T18:54:38.021 INFO:tasks.workunit.client.1.smithi190.stdout:5/645: dwrite d2/d17/d1b/db6/d88/f8c [0,4194304] 0 2024-03-26T18:54:38.021 INFO:tasks.workunit.client.1.smithi190.stdout:4/812: dwrite d7/d8/d14/d23/d28/d38/d96/f103 [0,4194304] 0 2024-03-26T18:54:38.021 INFO:tasks.workunit.client.1.smithi190.stdout:4/813: fsync d7/dc8/dd5/de4/def/ff9 0 2024-03-26T18:54:38.021 INFO:tasks.workunit.client.1.smithi190.stdout:4/814: dread - d7/d8/d14/d23/d28/d38/fea zero size 2024-03-26T18:54:38.022 INFO:tasks.workunit.client.1.smithi190.stdout:2/795: dread d6/d9/d3a/d3d/d1c/d8d/dc7/f43 [0,4194304] 0 2024-03-26T18:54:38.022 INFO:tasks.workunit.client.0.smithi102.stdout:9/428: mknod df/d1a/d23/d4e/d8e/ca1 0 2024-03-26T18:54:38.023 INFO:tasks.workunit.client.0.smithi102.stdout:9/429: truncate df/d1a/d57/f94 251238 0 2024-03-26T18:54:38.024 INFO:tasks.workunit.client.0.smithi102.stdout:4/347: dwrite d1/d25/f69 [0,4194304] 0 2024-03-26T18:54:38.024 INFO:tasks.workunit.client.0.smithi102.stdout:6/423: dwrite d2/f21 [0,4194304] 0 2024-03-26T18:54:38.024 INFO:tasks.workunit.client.0.smithi102.stdout:6/424: creat d2/d7/d3c/d78/f81 x:0 0 0 2024-03-26T18:54:38.024 INFO:tasks.workunit.client.0.smithi102.stdout:6/425: readlink d2/df/d35/d43/d75/l7f 0 2024-03-26T18:54:38.024 INFO:tasks.workunit.client.0.smithi102.stdout:9/430: creat df/d1a/d81/fa2 x:0 0 0 2024-03-26T18:54:38.025 INFO:tasks.workunit.client.0.smithi102.stdout:9/431: write df/d1a/d23/d4e/d8e/f56 [427992,51756] 0 2024-03-26T18:54:38.026 INFO:tasks.workunit.client.1.smithi190.stdout:9/987: mknod d12/d2d/dfc/d100/c15f 0 2024-03-26T18:54:38.026 INFO:tasks.workunit.client.1.smithi190.stdout:1/675: truncate d3/da/dd/d95/fb8 75484 0 2024-03-26T18:54:38.028 INFO:tasks.workunit.client.1.smithi190.stdout:4/815: mknod d7/d8/d14/d4b/d7d/da1/d113/c117 0 2024-03-26T18:54:38.028 INFO:tasks.workunit.client.1.smithi190.stdout:5/646: creat d2/d17/d1b/d60/d63/dcd/fd4 x:0 0 0 2024-03-26T18:54:38.029 INFO:tasks.workunit.client.1.smithi190.stdout:3/625: dread d1/d3/df/d18/d57/d91/fbc [0,4194304] 0 2024-03-26T18:54:38.029 INFO:tasks.workunit.client.1.smithi190.stdout:3/626: readlink d1/d3/df/d18/d57/d91/l84 0 2024-03-26T18:54:38.030 INFO:tasks.workunit.client.1.smithi190.stdout:3/627: write d1/de/f73 [4921600,37960] 0 2024-03-26T18:54:38.031 INFO:tasks.workunit.client.1.smithi190.stdout:2/796: rename d6/d9/d3a/d3d/d55/fa5 to d6/d9/d3a/d3d/d55/f116 0 2024-03-26T18:54:38.032 INFO:tasks.workunit.client.1.smithi190.stdout:1/676: creat d3/d9/db/dd4/dd7/fdb x:0 0 0 2024-03-26T18:54:38.032 INFO:tasks.workunit.client.1.smithi190.stdout:7/847: getdents d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/d87 0 2024-03-26T18:54:38.034 INFO:tasks.workunit.client.0.smithi102.stdout:7/416: dwrite d7/d10/d17/f37 [0,4194304] 0 2024-03-26T18:54:38.034 INFO:tasks.workunit.client.0.smithi102.stdout:7/417: chown d7/d31/d79 242841 1 2024-03-26T18:54:38.034 INFO:tasks.workunit.client.0.smithi102.stdout:7/418: write d7/d12/f38 [802847,68805] 0 2024-03-26T18:54:38.038 INFO:tasks.workunit.client.1.smithi190.stdout:2/797: creat d6/f117 x:0 0 0 2024-03-26T18:54:38.039 INFO:tasks.workunit.client.1.smithi190.stdout:7/848: mkdir d3/d6/d9/d14/d59/d73/d13e 0 2024-03-26T18:54:38.040 INFO:tasks.workunit.client.1.smithi190.stdout:5/647: dread d2/d17/d1b/db6/f77 [0,4194304] 0 2024-03-26T18:54:38.040 INFO:tasks.workunit.client.1.smithi190.stdout:5/648: write d2/d17/d1b/db6/d24/d3b/f46 [4363745,60382] 0 2024-03-26T18:54:38.040 INFO:tasks.workunit.client.1.smithi190.stdout:1/677: creat d3/da/dd/d95/dda/fdc x:0 0 0 2024-03-26T18:54:38.043 INFO:tasks.workunit.client.0.smithi102.stdout:4/348: dread d1/d17/d1a/d21/f2c [8388608,4194304] 0 2024-03-26T18:54:38.043 INFO:tasks.workunit.client.0.smithi102.stdout:6/426: symlink d2/df/d35/d60/l82 0 2024-03-26T18:54:38.044 INFO:tasks.workunit.client.1.smithi190.stdout:7/849: getdents d3/d1b/dd3/d10d 0 2024-03-26T18:54:38.047 INFO:tasks.workunit.client.0.smithi102.stdout:8/461: sync 2024-03-26T18:54:38.050 INFO:tasks.workunit.client.0.smithi102.stdout:9/432: link c8 df/d1a/d23/d3e/d75/ca3 0 2024-03-26T18:54:38.050 INFO:tasks.workunit.client.0.smithi102.stdout:9/433: write df/d1a/d23/d60/f74 [581461,91986] 0 2024-03-26T18:54:38.053 INFO:tasks.workunit.client.0.smithi102.stdout:7/419: creat d7/d12/d51/d6a/f7a x:0 0 0 2024-03-26T18:54:38.055 INFO:tasks.workunit.client.0.smithi102.stdout:4/349: unlink d1/d25/c37 0 2024-03-26T18:54:38.055 INFO:tasks.workunit.client.0.smithi102.stdout:4/350: read d1/d25/d39/f4e [484268,11124] 0 2024-03-26T18:54:38.057 INFO:tasks.workunit.client.1.smithi190.stdout:6/667: sync 2024-03-26T18:54:38.057 INFO:tasks.workunit.client.1.smithi190.stdout:8/823: sync 2024-03-26T18:54:38.057 INFO:tasks.workunit.client.1.smithi190.stdout:6/668: fdatasync d17/d28/d74/d97/fcb 0 2024-03-26T18:54:38.057 INFO:tasks.workunit.client.1.smithi190.stdout:8/824: chown d9/dd/d23/d36/d37/d3a/d61/daf/db3/fe2 3426 1 2024-03-26T18:54:38.058 INFO:tasks.workunit.client.0.smithi102.stdout:8/462: unlink d0/d3/d2c/d33/f8d 0 2024-03-26T18:54:38.064 INFO:tasks.workunit.client.0.smithi102.stdout:9/434: link df/d1a/d23/d3e/d75/ca3 df/d1a/d65/ca4 0 2024-03-26T18:54:38.064 INFO:tasks.workunit.client.0.smithi102.stdout:9/435: creat df/d1a/d23/d3e/fa5 x:0 0 0 2024-03-26T18:54:38.064 INFO:tasks.workunit.client.0.smithi102.stdout:9/436: chown df/f30 2797 1 2024-03-26T18:54:38.065 INFO:tasks.workunit.client.0.smithi102.stdout:6/427: dread d2/d3/f2a [0,4194304] 0 2024-03-26T18:54:38.065 INFO:tasks.workunit.client.1.smithi190.stdout:9/988: dwrite d12/d2d/d1d/d23/d130/f152 [0,4194304] 0 2024-03-26T18:54:38.070 INFO:tasks.workunit.client.1.smithi190.stdout:8/825: write d9/dd/f99 [1075217,71608] 0 2024-03-26T18:54:38.072 INFO:tasks.workunit.client.1.smithi190.stdout:7/850: dwrite d3/d6/d9/d1f/d93/dfb/d102/d99/fd6 [0,4194304] 0 2024-03-26T18:54:38.073 INFO:tasks.workunit.client.1.smithi190.stdout:1/678: dwrite d3/da/dd/d2f/d5b/f63 [4194304,4194304] 0 2024-03-26T18:54:38.073 INFO:tasks.workunit.client.1.smithi190.stdout:3/628: dwrite d1/de/d1c/d36/f58 [0,4194304] 0 2024-03-26T18:54:38.073 INFO:tasks.workunit.client.1.smithi190.stdout:4/816: dwrite d7/d8/faa [4194304,4194304] 0 2024-03-26T18:54:38.074 INFO:tasks.workunit.client.1.smithi190.stdout:8/826: rename d9/d39/d9e/ccc to d9/da2/df6/c119 0 2024-03-26T18:54:38.074 INFO:tasks.workunit.client.0.smithi102.stdout:6/428: dread d2/d3/f2a [0,4194304] 0 2024-03-26T18:54:38.074 INFO:tasks.workunit.client.0.smithi102.stdout:6/429: creat d2/d3d/d53/f83 x:0 0 0 2024-03-26T18:54:38.080 INFO:tasks.workunit.client.0.smithi102.stdout:7/420: getdents d7/d31 0 2024-03-26T18:54:38.080 INFO:tasks.workunit.client.0.smithi102.stdout:7/421: fsync d7/d31/d56/f63 0 2024-03-26T18:54:38.080 INFO:tasks.workunit.client.0.smithi102.stdout:7/422: chown d7/d10 81 1 2024-03-26T18:54:38.081 INFO:tasks.workunit.client.1.smithi190.stdout:4/817: creat d7/d8/d14/d4b/db2/f118 x:0 0 0 2024-03-26T18:54:38.085 INFO:tasks.workunit.client.1.smithi190.stdout:8/827: rename d9/dd/d23/d88/dfa/c117 to d9/d39/d79/dad/c11a 0 2024-03-26T18:54:38.085 INFO:tasks.workunit.client.1.smithi190.stdout:8/828: stat d9/dd/d23/dea/dfe/l101 0 2024-03-26T18:54:38.087 INFO:tasks.workunit.client.1.smithi190.stdout:5/649: rmdir d2 39 2024-03-26T18:54:38.088 INFO:tasks.workunit.client.1.smithi190.stdout:5/650: stat d2/d17/d1b/d60/d63/dcd/fd4 0 2024-03-26T18:54:38.088 INFO:tasks.workunit.client.1.smithi190.stdout:4/818: creat d7/d8/d14/d4b/d7d/f119 x:0 0 0 2024-03-26T18:54:38.088 INFO:tasks.workunit.client.1.smithi190.stdout:4/819: creat d7/d8/d14/d23/d28/d38/d96/f11a x:0 0 0 2024-03-26T18:54:38.089 INFO:tasks.workunit.client.1.smithi190.stdout:6/669: dwrite d17/d28/d46/d6a/fa4 [0,4194304] 0 2024-03-26T18:54:38.089 INFO:tasks.workunit.client.1.smithi190.stdout:6/670: fsync d17/d28/d2c/d50/f67 0 2024-03-26T18:54:38.091 INFO:tasks.workunit.client.1.smithi190.stdout:8/829: creat d9/dd/d23/d36/d37/d3a/d61/daf/db3/f11b x:0 0 0 2024-03-26T18:54:38.092 INFO:tasks.workunit.client.1.smithi190.stdout:8/830: fdatasync d9/dd/d23/fff 0 2024-03-26T18:54:38.093 INFO:tasks.workunit.client.1.smithi190.stdout:1/679: creat d3/da/dd/d76/dc4/fdd x:0 0 0 2024-03-26T18:54:38.093 INFO:tasks.workunit.client.1.smithi190.stdout:7/851: dread d3/d1b/dd3/d10d/d65/d90/d92/d9b/fb0 [0,4194304] 0 2024-03-26T18:54:38.094 INFO:tasks.workunit.client.0.smithi102.stdout:6/430: truncate d2/d7/d3c/f48 3621274 0 2024-03-26T18:54:38.094 INFO:tasks.workunit.client.0.smithi102.stdout:6/431: write d2/df/f2b [196269,73769] 0 2024-03-26T18:54:38.095 INFO:tasks.workunit.client.0.smithi102.stdout:9/437: getdents df/d84 0 2024-03-26T18:54:38.098 INFO:tasks.workunit.client.0.smithi102.stdout:7/423: rename f6 to d7/f7b 0 2024-03-26T18:54:38.098 INFO:tasks.workunit.client.1.smithi190.stdout:4/820: link d7/d8/cd7 d7/d8/d14/d23/c11b 0 2024-03-26T18:54:38.099 INFO:tasks.workunit.client.1.smithi190.stdout:4/821: creat d7/d8/d14/d23/d28/d38/d96/f11c x:0 0 0 2024-03-26T18:54:38.099 INFO:tasks.workunit.client.1.smithi190.stdout:4/822: write d7/d8/d14/d4b/d7d/fd0 [574257,105887] 0 2024-03-26T18:54:38.099 INFO:tasks.workunit.client.1.smithi190.stdout:4/823: chown d7/d1d/l92 0 1 2024-03-26T18:54:38.099 INFO:tasks.workunit.client.1.smithi190.stdout:4/824: rename d7/d8 to d7/d8/d14/d4b/d11d 22 2024-03-26T18:54:38.099 INFO:tasks.workunit.client.0.smithi102.stdout:3/655: sync 2024-03-26T18:54:38.101 INFO:tasks.workunit.client.0.smithi102.stdout:3/656: truncate df/f26 11890317 0 2024-03-26T18:54:38.102 INFO:tasks.workunit.client.0.smithi102.stdout:3/657: mkdir df/d29/d31/d77/d8a/dae/dee 0 2024-03-26T18:54:38.103 INFO:tasks.workunit.client.0.smithi102.stdout:3/658: symlink df/d29/d31/d36/d3c/d6f/lef 0 2024-03-26T18:54:38.104 INFO:tasks.workunit.client.1.smithi190.stdout:5/651: dread d2/d17/d1b/f76 [0,4194304] 0 2024-03-26T18:54:38.104 INFO:tasks.workunit.client.1.smithi190.stdout:5/652: truncate d2/d17/d1b/d60/d63/fa3 801770 0 2024-03-26T18:54:38.106 INFO:tasks.workunit.client.1.smithi190.stdout:4/825: mknod d7/d1d/d42/d4a/c11e 0 2024-03-26T18:54:38.106 INFO:tasks.workunit.client.1.smithi190.stdout:4/826: chown d7/d8/d14/d23/d28/d38/d96/d10f 23 1 2024-03-26T18:54:38.106 INFO:tasks.workunit.client.1.smithi190.stdout:4/827: fsync d7/f35 0 2024-03-26T18:54:38.106 INFO:tasks.workunit.client.1.smithi190.stdout:4/828: fsync d7/d8/d14/d6e/d6f/f7e 0 2024-03-26T18:54:38.107 INFO:tasks.workunit.client.0.smithi102.stdout:8/463: getdents d0/d3/d2c/d33 0 2024-03-26T18:54:38.107 INFO:tasks.workunit.client.0.smithi102.stdout:8/464: write d0/d15/f16 [4168258,111986] 0 2024-03-26T18:54:38.107 INFO:tasks.workunit.client.1.smithi190.stdout:4/829: mkdir d7/d8/d14/d4b/db2/d11f 0 2024-03-26T18:54:38.110 INFO:tasks.workunit.client.1.smithi190.stdout:4/830: unlink d7/d8/d14/d23/d28/d38/fcf 0 2024-03-26T18:54:38.113 INFO:tasks.workunit.client.0.smithi102.stdout:8/465: creat d0/f97 x:0 0 0 2024-03-26T18:54:38.114 INFO:tasks.workunit.client.0.smithi102.stdout:4/351: dwrite d1/d25/d39/f51 [0,4194304] 0 2024-03-26T18:54:38.114 INFO:tasks.workunit.client.1.smithi190.stdout:4/831: creat d7/d1d/d5e/d98/f120 x:0 0 0 2024-03-26T18:54:38.117 INFO:tasks.workunit.client.1.smithi190.stdout:4/832: unlink d7/d8/d14/d23/d28/l36 0 2024-03-26T18:54:38.117 INFO:tasks.workunit.client.1.smithi190.stdout:4/833: chown d7/d8/d1c/c45 515690 1 2024-03-26T18:54:38.119 INFO:tasks.workunit.client.1.smithi190.stdout:4/834: mknod d7/d8/d14/d6e/df0/c121 0 2024-03-26T18:54:38.121 INFO:tasks.workunit.client.1.smithi190.stdout:4/835: getdents d7/dc8/dd5/de4/def 0 2024-03-26T18:54:38.121 INFO:tasks.workunit.client.1.smithi190.stdout:4/836: dread - d7/d8/d14/d23/d28/d38/dd3/fe8 zero size 2024-03-26T18:54:38.121 INFO:tasks.workunit.client.1.smithi190.stdout:4/837: dread - d7/d8/d14/d23/d28/d38/d96/fdf zero size 2024-03-26T18:54:38.121 INFO:tasks.workunit.client.0.smithi102.stdout:1/405: sync 2024-03-26T18:54:38.122 INFO:tasks.workunit.client.0.smithi102.stdout:1/406: chown de/d1f/c63 157890 1 2024-03-26T18:54:38.123 INFO:tasks.workunit.client.1.smithi190.stdout:4/838: mknod d7/d8/d14/d23/d28/d38/c122 0 2024-03-26T18:54:38.123 INFO:tasks.workunit.client.1.smithi190.stdout:4/839: chown d7/d8/d14/d23/d28/d38/d96/d10f/c6d 77 1 2024-03-26T18:54:38.123 INFO:tasks.workunit.client.1.smithi190.stdout:4/840: write d7/d1d/d42/d4a/fa6 [402175,92924] 0 2024-03-26T18:54:38.123 INFO:tasks.workunit.client.1.smithi190.stdout:4/841: creat d7/d1d/d42/f123 x:0 0 0 2024-03-26T18:54:38.124 INFO:tasks.workunit.client.0.smithi102.stdout:1/407: rename f7 to de/d16/d1d/d2a/d2c/d43/f86 0 2024-03-26T18:54:38.124 INFO:tasks.workunit.client.0.smithi102.stdout:1/408: fdatasync de/d16/d1d/d2a/f75 0 2024-03-26T18:54:38.126 INFO:tasks.workunit.client.1.smithi190.stdout:4/842: getdents d7/dc8 0 2024-03-26T18:54:38.127 INFO:tasks.workunit.client.0.smithi102.stdout:1/409: rename de/d16/d1d/d2a/d2c/f66 to de/d16/d1d/f87 0 2024-03-26T18:54:38.129 INFO:tasks.workunit.client.1.smithi190.stdout:4/843: mkdir d7/d8/d14/d23/d28/d124 0 2024-03-26T18:54:38.130 INFO:tasks.workunit.client.0.smithi102.stdout:6/432: dread d2/d12/f3f [0,4194304] 0 2024-03-26T18:54:38.131 INFO:tasks.workunit.client.0.smithi102.stdout:6/433: dread - d2/d7/f54 zero size 2024-03-26T18:54:38.132 INFO:tasks.workunit.client.0.smithi102.stdout:6/434: readlink d2/d12/l49 0 2024-03-26T18:54:38.132 INFO:tasks.workunit.client.0.smithi102.stdout:6/435: creat d2/d3d/d53/d63/f84 x:0 0 0 2024-03-26T18:54:38.132 INFO:tasks.workunit.client.0.smithi102.stdout:6/436: stat d2/df/l32 0 2024-03-26T18:54:38.132 INFO:tasks.workunit.client.0.smithi102.stdout:6/437: read - d2/df/f64 zero size 2024-03-26T18:54:38.132 INFO:tasks.workunit.client.0.smithi102.stdout:6/438: readlink d2/d12/l2f 0 2024-03-26T18:54:38.132 INFO:tasks.workunit.client.0.smithi102.stdout:6/439: chown d2/d3d/d53/f69 3802 1 2024-03-26T18:54:38.132 INFO:tasks.workunit.client.0.smithi102.stdout:6/440: dread - d2/d3d/f7d zero size 2024-03-26T18:54:38.132 INFO:tasks.workunit.client.0.smithi102.stdout:6/441: fsync d2/d3d/d53/d63/d6a/f6f 0 2024-03-26T18:54:38.132 INFO:tasks.workunit.client.1.smithi190.stdout:4/844: rmdir d7/d8/d14/d5c 39 2024-03-26T18:54:38.133 INFO:tasks.workunit.client.1.smithi190.stdout:4/845: chown d7/d8/d14/d5c/fe5 0 1 2024-03-26T18:54:38.133 INFO:tasks.workunit.client.0.smithi102.stdout:6/442: rename d2/f3a to d2/df/d35/d43/f85 0 2024-03-26T18:54:38.133 INFO:tasks.workunit.client.0.smithi102.stdout:6/443: chown d2/f34 180604 1 2024-03-26T18:54:38.133 INFO:tasks.workunit.client.0.smithi102.stdout:6/444: write d2/f34 [548553,112843] 0 2024-03-26T18:54:38.134 INFO:tasks.workunit.client.1.smithi190.stdout:4/846: mknod d7/d1d/d42/d4a/c125 0 2024-03-26T18:54:38.137 INFO:tasks.workunit.client.1.smithi190.stdout:4/847: mknod d7/d8/d14/d4b/d7d/da1/c126 0 2024-03-26T18:54:38.137 INFO:tasks.workunit.client.1.smithi190.stdout:4/848: readlink d7/d1d/l92 0 2024-03-26T18:54:38.140 INFO:tasks.workunit.client.0.smithi102.stdout:8/466: read d0/d3/f28 [2769091,70705] 0 2024-03-26T18:54:38.141 INFO:tasks.workunit.client.0.smithi102.stdout:8/467: getdents d0/d3/d2c/d33/d87 0 2024-03-26T18:54:38.141 INFO:tasks.workunit.client.0.smithi102.stdout:8/468: write d0/d3/d26/d5c/f88 [249014,94827] 0 2024-03-26T18:54:38.141 INFO:tasks.workunit.client.1.smithi190.stdout:9/989: dwrite d12/d2d/d1d/f8d [0,4194304] 0 2024-03-26T18:54:38.142 INFO:tasks.workunit.client.1.smithi190.stdout:9/990: fsync d12/d6d/f128 0 2024-03-26T18:54:38.142 INFO:tasks.workunit.client.1.smithi190.stdout:9/991: write d12/d2d/d1d/d23/d2b/da0/f149 [782904,59381] 0 2024-03-26T18:54:38.142 INFO:tasks.workunit.client.1.smithi190.stdout:3/629: dwrite d1/d3/df/d18/f30 [0,4194304] 0 2024-03-26T18:54:38.144 INFO:tasks.workunit.client.1.smithi190.stdout:3/630: symlink d1/d3/lc8 0 2024-03-26T18:54:38.144 INFO:tasks.workunit.client.1.smithi190.stdout:3/631: stat d1/de/f29 0 2024-03-26T18:54:38.146 INFO:tasks.workunit.client.1.smithi190.stdout:3/632: symlink d1/d3/df/d18/db4/lc9 0 2024-03-26T18:54:38.146 INFO:tasks.workunit.client.1.smithi190.stdout:3/633: truncate d1/de/d9f/faa 808307 0 2024-03-26T18:54:38.155 INFO:tasks.workunit.client.1.smithi190.stdout:7/852: dwrite d3/d1b/dd3/d10d/d50/dde/fe0 [0,4194304] 0 2024-03-26T18:54:38.155 INFO:tasks.workunit.client.1.smithi190.stdout:7/853: stat d3/d6/d9/d1f/d93/lc8 0 2024-03-26T18:54:38.156 INFO:tasks.workunit.client.0.smithi102.stdout:7/424: write d7/fb [1550969,110111] 0 2024-03-26T18:54:38.158 INFO:tasks.workunit.client.1.smithi190.stdout:7/854: rmdir d3/d6/d9/d14/d59/d73/d13e 0 2024-03-26T18:54:38.159 INFO:tasks.workunit.client.1.smithi190.stdout:7/855: mkdir d3/d6/d9/d1f/d4c/d71/dcc/d13f 0 2024-03-26T18:54:38.161 INFO:tasks.workunit.client.1.smithi190.stdout:7/856: link d3/d1b/d57/dd1/l136 d3/d1b/dd3/d10d/d50/dde/l140 0 2024-03-26T18:54:38.162 INFO:tasks.workunit.client.1.smithi190.stdout:7/857: write d3/d1b/dd3/d10d/d65/d109/d119/f139 [5840225,530] 0 2024-03-26T18:54:38.162 INFO:tasks.workunit.client.1.smithi190.stdout:7/858: dread - d3/d1b/dd3/d10d/d50/f118 zero size 2024-03-26T18:54:38.162 INFO:tasks.workunit.client.1.smithi190.stdout:7/859: write d3/d1b/dd3/d10d/d65/d90/d92/fbc [389978,24241] 0 2024-03-26T18:54:38.162 INFO:tasks.workunit.client.1.smithi190.stdout:7/860: dread - d3/d1b/d57/f12f zero size 2024-03-26T18:54:38.163 INFO:tasks.workunit.client.1.smithi190.stdout:7/861: creat d3/d1b/dd3/d10d/d50/dde/dfd/f141 x:0 0 0 2024-03-26T18:54:38.163 INFO:tasks.workunit.client.1.smithi190.stdout:3/634: unlink d1/de/d1c/d36/f5d 0 2024-03-26T18:54:38.165 INFO:tasks.workunit.client.1.smithi190.stdout:7/862: link d3/d1b/l64 d3/d1b/dd3/d10d/d50/l142 0 2024-03-26T18:54:38.165 INFO:tasks.workunit.client.0.smithi102.stdout:7/425: unlink d7/d10/d27/d34/f48 0 2024-03-26T18:54:38.166 INFO:tasks.workunit.client.1.smithi190.stdout:7/863: symlink d3/d6/d9/l143 0 2024-03-26T18:54:38.166 INFO:tasks.workunit.client.1.smithi190.stdout:7/864: mkdir d3/d1b/dd3/d10d/d65/d144 0 2024-03-26T18:54:38.167 INFO:tasks.workunit.client.0.smithi102.stdout:7/426: rename d7/c3e to d7/d10/d67/d69/d78/c7c 0 2024-03-26T18:54:38.171 INFO:tasks.workunit.client.1.smithi190.stdout:8/831: dwrite d9/dd/d23/d36/d37/d3a/d61/fd6 [0,4194304] 0 2024-03-26T18:54:38.171 INFO:tasks.workunit.client.1.smithi190.stdout:8/832: chown d9/d39/cb2 7987811 1 2024-03-26T18:54:38.174 INFO:tasks.workunit.client.1.smithi190.stdout:5/653: dwrite d2/d17/d1b/fb9 [0,4194304] 0 2024-03-26T18:54:38.177 INFO:tasks.workunit.client.1.smithi190.stdout:3/635: dread d1/de/d1c/f7b [0,4194304] 0 2024-03-26T18:54:38.177 INFO:tasks.workunit.client.1.smithi190.stdout:3/636: chown d1/d3/df 98024 1 2024-03-26T18:54:38.181 INFO:tasks.workunit.client.1.smithi190.stdout:5/654: creat d2/d17/d1b/d60/db8/fd5 x:0 0 0 2024-03-26T18:54:38.183 INFO:tasks.workunit.client.0.smithi102.stdout:8/469: dread d0/d15/d81/f29 [4194304,4194304] 0 2024-03-26T18:54:38.183 INFO:tasks.workunit.client.0.smithi102.stdout:8/470: chown d0/d3c/d6f 942 1 2024-03-26T18:54:38.183 INFO:tasks.workunit.client.1.smithi190.stdout:3/637: mkdir d1/d3/d5a/d6e/dca 0 2024-03-26T18:54:38.185 INFO:tasks.workunit.client.1.smithi190.stdout:5/655: rmdir d2/d17/d1b/d60/d63 39 2024-03-26T18:54:38.185 INFO:tasks.workunit.client.1.smithi190.stdout:6/671: dwrite d17/d28/d46/f56 [0,4194304] 0 2024-03-26T18:54:38.186 INFO:tasks.workunit.client.1.smithi190.stdout:2/798: dwrite d6/d9/f75 [0,4194304] 0 2024-03-26T18:54:38.186 INFO:tasks.workunit.client.0.smithi102.stdout:8/471: mkdir d0/d3c/d96/d98 0 2024-03-26T18:54:38.186 INFO:tasks.workunit.client.0.smithi102.stdout:8/472: fdatasync d0/d15/d81/f45 0 2024-03-26T18:54:38.186 INFO:tasks.workunit.client.0.smithi102.stdout:0/466: sync 2024-03-26T18:54:38.186 INFO:tasks.workunit.client.0.smithi102.stdout:2/386: sync 2024-03-26T18:54:38.187 INFO:tasks.workunit.client.1.smithi190.stdout:3/638: dread d1/de/f69 [0,4194304] 0 2024-03-26T18:54:38.187 INFO:tasks.workunit.client.0.smithi102.stdout:8/473: symlink d0/d15/d1b/l99 0 2024-03-26T18:54:38.188 INFO:tasks.workunit.client.0.smithi102.stdout:5/661: sync 2024-03-26T18:54:38.191 INFO:tasks.workunit.client.0.smithi102.stdout:2/387: rename d3/d2a/l3a to d3/d6/l86 0 2024-03-26T18:54:38.191 INFO:tasks.workunit.client.0.smithi102.stdout:2/388: write d3/f1c [3801302,30942] 0 2024-03-26T18:54:38.192 INFO:tasks.workunit.client.1.smithi190.stdout:2/799: mkdir d6/d9/d3a/d3d/d2e/d10e/d9e/dfc/d118 0 2024-03-26T18:54:38.193 INFO:tasks.workunit.client.0.smithi102.stdout:1/410: getdents de/d16/d1d/d2a/d2c/d43 0 2024-03-26T18:54:38.193 INFO:tasks.workunit.client.0.smithi102.stdout:1/411: dread - de/f83 zero size 2024-03-26T18:54:38.193 INFO:tasks.workunit.client.0.smithi102.stdout:1/412: chown de/d16/d1d/d2a/d2c/d56/d6e/f81 229 1 2024-03-26T18:54:38.193 INFO:tasks.workunit.client.0.smithi102.stdout:1/413: chown de/d2e 7253 1 2024-03-26T18:54:38.193 INFO:tasks.workunit.client.0.smithi102.stdout:1/414: write de/d16/f29 [1216760,29237] 0 2024-03-26T18:54:38.193 INFO:tasks.workunit.client.0.smithi102.stdout:1/415: chown de/d2e/c45 22880 1 2024-03-26T18:54:38.197 INFO:tasks.workunit.client.0.smithi102.stdout:8/474: mkdir d0/d15/d9a 0 2024-03-26T18:54:38.207 INFO:tasks.workunit.client.0.smithi102.stdout:8/475: link d0/d3/d2c/d33/d87/c84 d0/d15/d9a/c9b 0 2024-03-26T18:54:38.207 INFO:tasks.workunit.client.0.smithi102.stdout:2/389: rmdir d3/d11/d27 39 2024-03-26T18:54:38.208 INFO:tasks.workunit.client.1.smithi190.stdout:6/672: write d17/d28/d2c/d77/fbb [2553330,94595] 0 2024-03-26T18:54:38.209 INFO:tasks.workunit.client.0.smithi102.stdout:0/467: write d6/de/d13/d4a/d85/f61 [5063116,4180] 0 2024-03-26T18:54:38.209 INFO:tasks.workunit.client.0.smithi102.stdout:8/476: creat d0/d3/d26/d19/d76/f9c x:0 0 0 2024-03-26T18:54:38.209 INFO:tasks.workunit.client.0.smithi102.stdout:0/468: creat d6/de/d13/d4a/f99 x:0 0 0 2024-03-26T18:54:38.211 INFO:tasks.workunit.client.0.smithi102.stdout:2/390: truncate d3/d20/f62 4034401 0 2024-03-26T18:54:38.212 INFO:tasks.workunit.client.0.smithi102.stdout:2/391: write d3/d11/d15/d31/d39/f64 [389314,15975] 0 2024-03-26T18:54:38.212 INFO:tasks.workunit.client.0.smithi102.stdout:9/438: dwrite df/d1a/f5e [0,4194304] 0 2024-03-26T18:54:38.215 INFO:tasks.workunit.client.1.smithi190.stdout:2/800: dread d6/d9/d3a/d3d/d2e/d86/d10b/f111 [0,4194304] 0 2024-03-26T18:54:38.215 INFO:tasks.workunit.client.1.smithi190.stdout:2/801: write d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d8b/f10f [2119995,14047] 0 2024-03-26T18:54:38.215 INFO:tasks.workunit.client.1.smithi190.stdout:2/802: dread - d6/d9/d3a/fe4 zero size 2024-03-26T18:54:38.215 INFO:tasks.workunit.client.1.smithi190.stdout:2/803: dread - d6/d21/d61/d6c/fcc zero size 2024-03-26T18:54:38.216 INFO:tasks.workunit.client.0.smithi102.stdout:9/439: dread df/d1a/d23/d29/f31 [4194304,4194304] 0 2024-03-26T18:54:38.216 INFO:tasks.workunit.client.0.smithi102.stdout:3/659: dwrite df/f1b [4194304,4194304] 0 2024-03-26T18:54:38.217 INFO:tasks.workunit.client.0.smithi102.stdout:4/352: dwrite d1/d25/d39/d3e/d4c/f6a [0,4194304] 0 2024-03-26T18:54:38.217 INFO:tasks.workunit.client.1.smithi190.stdout:6/673: getdents d17/dd3 0 2024-03-26T18:54:38.218 INFO:tasks.workunit.client.1.smithi190.stdout:4/849: dwrite d7/d8/d14/f94 [0,4194304] 0 2024-03-26T18:54:38.218 INFO:tasks.workunit.client.1.smithi190.stdout:4/850: stat d7/d8/d14/d4b/d7d/da1/d113/c117 0 2024-03-26T18:54:38.218 INFO:tasks.workunit.client.1.smithi190.stdout:4/851: fdatasync d7/d1d/fb5 0 2024-03-26T18:54:38.221 INFO:tasks.workunit.client.0.smithi102.stdout:8/477: dread d0/d3/d31/f4f [0,4194304] 0 2024-03-26T18:54:38.222 INFO:tasks.workunit.client.1.smithi190.stdout:6/674: creat d17/d28/d46/dd7/ddb/fde x:0 0 0 2024-03-26T18:54:38.223 INFO:tasks.workunit.client.0.smithi102.stdout:9/440: creat df/fa6 x:0 0 0 2024-03-26T18:54:38.224 INFO:tasks.workunit.client.1.smithi190.stdout:4/852: mkdir d7/d8/d127 0 2024-03-26T18:54:38.224 INFO:tasks.workunit.client.1.smithi190.stdout:4/853: fsync d7/d8/d14/d23/d28/d38/dd3/dfd/f10d 0 2024-03-26T18:54:38.224 INFO:tasks.workunit.client.1.smithi190.stdout:6/675: truncate d17/d28/d80/f88 4259972 0 2024-03-26T18:54:38.226 INFO:tasks.workunit.client.0.smithi102.stdout:3/660: dread df/d1c/f2a [0,4194304] 0 2024-03-26T18:54:38.227 INFO:tasks.workunit.client.1.smithi190.stdout:6/676: truncate d17/f21 3302303 0 2024-03-26T18:54:38.228 INFO:tasks.workunit.client.0.smithi102.stdout:4/353: rename d1/d17/d2a/l3a to d1/d17/d1a/d21/l72 0 2024-03-26T18:54:38.229 INFO:tasks.workunit.client.0.smithi102.stdout:9/441: symlink df/d1a/d23/d29/d9c/la7 0 2024-03-26T18:54:38.230 INFO:tasks.workunit.client.0.smithi102.stdout:7/427: dwrite d7/d10/f14 [0,4194304] 0 2024-03-26T18:54:38.230 INFO:tasks.workunit.client.0.smithi102.stdout:7/428: dread - d7/d12/d51/f58 zero size 2024-03-26T18:54:38.231 INFO:tasks.workunit.client.1.smithi190.stdout:9/992: dwrite d12/d6d/f128 [0,4194304] 0 2024-03-26T18:54:38.232 INFO:tasks.workunit.client.1.smithi190.stdout:7/865: dwrite d3/fc [0,4194304] 0 2024-03-26T18:54:38.234 INFO:tasks.workunit.client.0.smithi102.stdout:8/478: write d0/d15/d5d/f61 [110572,112123] 0 2024-03-26T18:54:38.234 INFO:tasks.workunit.client.0.smithi102.stdout:8/479: creat d0/d3/d2c/d58/d75/f9d x:0 0 0 2024-03-26T18:54:38.236 INFO:tasks.workunit.client.0.smithi102.stdout:4/354: mknod d1/d17/d1a/d21/d6d/c73 0 2024-03-26T18:54:38.238 INFO:tasks.workunit.client.0.smithi102.stdout:9/442: rmdir df/d1a/d1d/d72 39 2024-03-26T18:54:38.238 INFO:tasks.workunit.client.0.smithi102.stdout:9/443: write df/f30 [1295166,125856] 0 2024-03-26T18:54:38.240 INFO:tasks.workunit.client.1.smithi190.stdout:9/993: creat d12/d2d/d1d/d23/d2b/da0/d138/f160 x:0 0 0 2024-03-26T18:54:38.241 INFO:tasks.workunit.client.0.smithi102.stdout:9/444: mkdir df/d46/da8 0 2024-03-26T18:54:38.241 INFO:tasks.workunit.client.1.smithi190.stdout:7/866: creat d3/d6/d9/d1f/d4c/d71/dcc/d13f/f145 x:0 0 0 2024-03-26T18:54:38.243 INFO:tasks.workunit.client.0.smithi102.stdout:9/445: creat df/d84/fa9 x:0 0 0 2024-03-26T18:54:38.253 INFO:tasks.workunit.client.1.smithi190.stdout:8/833: dwrite d9/da2/f2b [4194304,4194304] 0 2024-03-26T18:54:38.253 INFO:tasks.workunit.client.1.smithi190.stdout:8/834: write d9/dd/d23/d36/d37/d3a/d43/dc2/f107 [720922,37108] 0 2024-03-26T18:54:38.254 INFO:tasks.workunit.client.1.smithi190.stdout:8/835: mknod d9/dd/d15/c11c 0 2024-03-26T18:54:38.258 INFO:tasks.workunit.client.1.smithi190.stdout:5/656: dwrite d2/d17/d1b/db6/f28 [0,4194304] 0 2024-03-26T18:54:38.258 INFO:tasks.workunit.client.1.smithi190.stdout:5/657: fsync d2/d17/d1b/d60/d63/f6e 0 2024-03-26T18:54:38.259 INFO:tasks.workunit.client.0.smithi102.stdout:9/446: dread df/d1a/f86 [0,4194304] 0 2024-03-26T18:54:38.260 INFO:tasks.workunit.client.1.smithi190.stdout:7/867: write d3/d6/f8c [1541785,109031] 0 2024-03-26T18:54:38.261 INFO:tasks.workunit.client.1.smithi190.stdout:5/658: creat d2/d17/d62/d64/fd6 x:0 0 0 2024-03-26T18:54:38.261 INFO:tasks.workunit.client.1.smithi190.stdout:5/659: write d2/f15 [5094782,88643] 0 2024-03-26T18:54:38.261 INFO:tasks.workunit.client.1.smithi190.stdout:7/868: mkdir d3/d1b/dd3/d10d/d65/d109/d11b/d146 0 2024-03-26T18:54:38.262 INFO:tasks.workunit.client.0.smithi102.stdout:9/447: creat df/d1a/d23/d29/d9c/faa x:0 0 0 2024-03-26T18:54:38.262 INFO:tasks.workunit.client.0.smithi102.stdout:9/448: creat df/d1a/d23/d29/d9c/fab x:0 0 0 2024-03-26T18:54:38.264 INFO:tasks.workunit.client.1.smithi190.stdout:5/660: mkdir d2/d17/d62/dd7 0 2024-03-26T18:54:38.265 INFO:tasks.workunit.client.1.smithi190.stdout:9/994: dread d12/d2d/f9f [0,4194304] 0 2024-03-26T18:54:38.267 INFO:tasks.workunit.client.1.smithi190.stdout:7/869: link d3/d6/d9/d1f/f22 d3/d1b/dd3/d10d/d65/d90/f147 0 2024-03-26T18:54:38.267 INFO:tasks.workunit.client.0.smithi102.stdout:9/449: rename df/d1a/d23/d3e/d75/c47 to df/d1a/d1d/d72/d7d/cac 0 2024-03-26T18:54:38.269 INFO:tasks.workunit.client.1.smithi190.stdout:5/661: mknod d2/d17/d98/d3a/cd8 0 2024-03-26T18:54:38.271 INFO:tasks.workunit.client.1.smithi190.stdout:9/995: dread d12/fc6 [0,4194304] 0 2024-03-26T18:54:38.273 INFO:tasks.workunit.client.1.smithi190.stdout:9/996: symlink d12/d2d/d1d/d23/d83/dc9/dd4/d71/daa/d10e/l161 0 2024-03-26T18:54:38.285 INFO:tasks.workunit.client.1.smithi190.stdout:3/639: dwrite d1/de/d81/dad/fb0 [0,4194304] 0 2024-03-26T18:54:38.286 INFO:tasks.workunit.client.0.smithi102.stdout:1/416: dwrite de/d16/d1d/d61/d6a/f7d [0,4194304] 0 2024-03-26T18:54:38.289 INFO:tasks.workunit.client.1.smithi190.stdout:3/640: symlink d1/d3/df/d18/d35/lcb 0 2024-03-26T18:54:38.289 INFO:tasks.workunit.client.1.smithi190.stdout:3/641: creat d1/de/fcc x:0 0 0 2024-03-26T18:54:38.292 INFO:tasks.workunit.client.1.smithi190.stdout:3/642: mknod d1/d3/d52/d70/ccd 0 2024-03-26T18:54:38.308 INFO:tasks.workunit.client.1.smithi190.stdout:7/870: dread d3/d6/f4e [0,4194304] 0 2024-03-26T18:54:38.309 INFO:tasks.workunit.client.0.smithi102.stdout:9/450: write df/d1a/d23/f5d [3854476,10206] 0 2024-03-26T18:54:38.311 INFO:tasks.workunit.client.0.smithi102.stdout:5/662: dwrite d2/d25/d75/db9/f89 [0,4194304] 0 2024-03-26T18:54:38.311 INFO:tasks.workunit.client.1.smithi190.stdout:7/871: unlink d3/d6/d9/d1f/d93/dfb/d102/d99/fa3 0 2024-03-26T18:54:38.313 INFO:tasks.workunit.client.1.smithi190.stdout:7/872: mkdir d3/d6/d9/d14/d59/dbf/d148 0 2024-03-26T18:54:38.315 INFO:tasks.workunit.client.1.smithi190.stdout:2/804: dwrite d6/d9/d45/d6b/d93/dc0/fe3 [0,4194304] 0 2024-03-26T18:54:38.318 INFO:tasks.workunit.client.0.smithi102.stdout:5/663: dread d2/d2c/d6f/db8/fc8 [0,4194304] 0 2024-03-26T18:54:38.323 INFO:tasks.workunit.client.1.smithi190.stdout:7/873: rmdir d3/d1b/dd3/d10d/d65/d109/d119 39 2024-03-26T18:54:38.323 INFO:tasks.workunit.client.1.smithi190.stdout:7/874: chown d3/d1b/f2c 5590403 1 2024-03-26T18:54:38.323 INFO:tasks.workunit.client.1.smithi190.stdout:7/875: chown d3/d1b/dd3/d10d/d65 616977 1 2024-03-26T18:54:38.327 INFO:tasks.workunit.client.1.smithi190.stdout:2/805: mknod d6/d9/d48/da0/c119 0 2024-03-26T18:54:38.327 INFO:tasks.workunit.client.1.smithi190.stdout:8/836: dwrite d9/d39/d67/f94 [0,4194304] 0 2024-03-26T18:54:38.328 INFO:tasks.workunit.client.0.smithi102.stdout:8/480: dwrite d0/d3/d2c/f8e [0,4194304] 0 2024-03-26T18:54:38.328 INFO:tasks.workunit.client.0.smithi102.stdout:8/481: write d0/d3/d26/f22 [9198936,129507] 0 2024-03-26T18:54:38.328 INFO:tasks.workunit.client.0.smithi102.stdout:0/469: dwrite d6/de/d13/d4a/f68 [0,4194304] 0 2024-03-26T18:54:38.328 INFO:tasks.workunit.client.0.smithi102.stdout:7/429: dwrite d7/d10/d17/f35 [8388608,4194304] 0 2024-03-26T18:54:38.329 INFO:tasks.workunit.client.0.smithi102.stdout:2/392: dwrite d3/d11/f55 [0,4194304] 0 2024-03-26T18:54:38.331 INFO:tasks.workunit.client.0.smithi102.stdout:8/482: creat d0/d3/d2c/d33/d87/f9e x:0 0 0 2024-03-26T18:54:38.331 INFO:tasks.workunit.client.0.smithi102.stdout:3/661: dwrite df/f67 [0,4194304] 0 2024-03-26T18:54:38.332 INFO:tasks.workunit.client.0.smithi102.stdout:5/664: dread d2/f7 [0,4194304] 0 2024-03-26T18:54:38.332 INFO:tasks.workunit.client.0.smithi102.stdout:5/665: truncate d2/dea/dbb/dcf/fda 346146 0 2024-03-26T18:54:38.332 INFO:tasks.workunit.client.0.smithi102.stdout:5/666: chown d2/dea/d4f/fc2 3 1 2024-03-26T18:54:38.332 INFO:tasks.workunit.client.0.smithi102.stdout:4/355: dwrite d1/d17/d2a/f54 [4194304,4194304] 0 2024-03-26T18:54:38.332 INFO:tasks.workunit.client.0.smithi102.stdout:5/667: fdatasync d2/d14/d1a/f61 0 2024-03-26T18:54:38.333 INFO:tasks.workunit.client.0.smithi102.stdout:5/668: stat d2/d14/d1a/l21 0 2024-03-26T18:54:38.333 INFO:tasks.workunit.client.0.smithi102.stdout:4/356: fdatasync d1/d25/d39/d3e/d4c/f6a 0 2024-03-26T18:54:38.333 INFO:tasks.workunit.client.1.smithi190.stdout:7/876: dread d3/d1b/dd3/d10d/d50/f89 [0,4194304] 0 2024-03-26T18:54:38.334 INFO:tasks.workunit.client.0.smithi102.stdout:3/662: dread df/d29/d31/d36/d3c/d6f/d75/f88 [0,4194304] 0 2024-03-26T18:54:38.334 INFO:tasks.workunit.client.0.smithi102.stdout:3/663: fsync df/fb1 0 2024-03-26T18:54:38.334 INFO:tasks.workunit.client.1.smithi190.stdout:8/837: mknod d9/da2/c11d 0 2024-03-26T18:54:38.336 INFO:tasks.workunit.client.1.smithi190.stdout:5/662: dwrite d2/d17/d62/f7e [0,4194304] 0 2024-03-26T18:54:38.336 INFO:tasks.workunit.client.1.smithi190.stdout:3/643: dwrite d1/de/f69 [0,4194304] 0 2024-03-26T18:54:38.338 INFO:tasks.workunit.client.1.smithi190.stdout:9/997: dwrite d12/d2d/f3e [0,4194304] 0 2024-03-26T18:54:38.338 INFO:tasks.workunit.client.1.smithi190.stdout:9/998: readlink d12/d2d/d1d/d23/d83/dc9/dd4/d3f/l76 0 2024-03-26T18:54:38.338 INFO:tasks.workunit.client.0.smithi102.stdout:9/451: dwrite df/fa6 [0,4194304] 0 2024-03-26T18:54:38.338 INFO:tasks.workunit.client.0.smithi102.stdout:7/430: truncate d7/d12/f38 268738 0 2024-03-26T18:54:38.340 INFO:tasks.workunit.client.1.smithi190.stdout:7/877: dread d3/d6/d9/d14/f7e [0,4194304] 0 2024-03-26T18:54:38.341 INFO:tasks.workunit.client.1.smithi190.stdout:5/663: truncate d2/d17/d1b/db6/d96/fa4 710542 0 2024-03-26T18:54:38.342 INFO:tasks.workunit.client.0.smithi102.stdout:2/393: rmdir d3/d6/d1a/d25 39 2024-03-26T18:54:38.345 INFO:tasks.workunit.client.1.smithi190.stdout:9/999: mkdir d12/d2d/d1d/d23/d8e/d162 0 2024-03-26T18:54:38.345 INFO:tasks.workunit.client.1.smithi190.stdout:5/664: dread d2/d17/d98/d3a/f52 [0,4194304] 0 2024-03-26T18:54:38.347 INFO:tasks.workunit.client.1.smithi190.stdout:7/878: mknod d3/d1b/dd3/d10d/d65/d90/c149 0 2024-03-26T18:54:38.348 INFO:tasks.workunit.client.0.smithi102.stdout:8/483: mkdir d0/d3/d26/d19/d9f 0 2024-03-26T18:54:38.348 INFO:tasks.workunit.client.0.smithi102.stdout:8/484: stat d0/d3/d31/c39 0 2024-03-26T18:54:38.349 INFO:tasks.workunit.client.0.smithi102.stdout:3/664: write df/f67 [291467,64752] 0 2024-03-26T18:54:38.349 INFO:tasks.workunit.client.0.smithi102.stdout:3/665: stat df/d29/d31/d36/d45 0 2024-03-26T18:54:38.350 INFO:tasks.workunit.client.1.smithi190.stdout:7/879: creat d3/d1b/dd3/d10d/d65/d109/d11b/f14a x:0 0 0 2024-03-26T18:54:38.350 INFO:tasks.workunit.client.1.smithi190.stdout:5/665: creat d2/d17/d1b/db6/d88/fd9 x:0 0 0 2024-03-26T18:54:38.350 INFO:tasks.workunit.client.1.smithi190.stdout:5/666: fsync d2/f16 0 2024-03-26T18:54:38.350 INFO:tasks.workunit.client.1.smithi190.stdout:5/667: chown d2/d17/d1b/d60/d63/d85 21 1 2024-03-26T18:54:38.350 INFO:tasks.workunit.client.1.smithi190.stdout:5/668: fsync d2/d17/f4f 0 2024-03-26T18:54:38.350 INFO:tasks.workunit.client.1.smithi190.stdout:5/669: truncate d2/d17/d1b/db6/fc1 4525017 0 2024-03-26T18:54:38.350 INFO:tasks.workunit.client.1.smithi190.stdout:7/880: mkdir d3/d6/d9/d1f/d14b 0 2024-03-26T18:54:38.351 INFO:tasks.workunit.client.1.smithi190.stdout:7/881: read - d3/d1b/dd3/d10d/d50/f60 zero size 2024-03-26T18:54:38.351 INFO:tasks.workunit.client.1.smithi190.stdout:5/670: symlink d2/d17/d1b/db6/lda 0 2024-03-26T18:54:38.351 INFO:tasks.workunit.client.1.smithi190.stdout:8/838: truncate d9/d48/f58 3721709 0 2024-03-26T18:54:38.352 INFO:tasks.workunit.client.1.smithi190.stdout:8/839: write d9/d39/d9c/f113 [529973,49587] 0 2024-03-26T18:54:38.352 INFO:tasks.workunit.client.1.smithi190.stdout:5/671: symlink d2/d17/d1b/d60/db8/ldb 0 2024-03-26T18:54:38.353 INFO:tasks.workunit.client.1.smithi190.stdout:8/840: symlink d9/d39/d79/l11e 0 2024-03-26T18:54:38.353 INFO:tasks.workunit.client.1.smithi190.stdout:8/841: chown d9/d39/d9c/f113 143761 1 2024-03-26T18:54:38.353 INFO:tasks.workunit.client.0.smithi102.stdout:5/669: truncate d2/d14/d1a/f1b 1537949 0 2024-03-26T18:54:38.354 INFO:tasks.workunit.client.1.smithi190.stdout:5/672: mkdir d2/d17/d1b/d60/d63/d85/ddc 0 2024-03-26T18:54:38.356 INFO:tasks.workunit.client.1.smithi190.stdout:5/673: truncate d2/d17/d98/d3a/f6b 658111 0 2024-03-26T18:54:38.356 INFO:tasks.workunit.client.1.smithi190.stdout:5/674: truncate d2/d17/d1b/d60/fc5 249444 0 2024-03-26T18:54:38.358 INFO:tasks.workunit.client.0.smithi102.stdout:2/394: symlink d3/d11/d15/l87 0 2024-03-26T18:54:38.358 INFO:tasks.workunit.client.1.smithi190.stdout:5/675: truncate d2/d17/d1b/fc8 1935 0 2024-03-26T18:54:38.361 INFO:tasks.workunit.client.1.smithi190.stdout:5/676: mkdir d2/d17/ddd 0 2024-03-26T18:54:38.361 INFO:tasks.workunit.client.1.smithi190.stdout:5/677: fdatasync d2/d17/d62/fca 0 2024-03-26T18:54:38.361 INFO:tasks.workunit.client.1.smithi190.stdout:5/678: read d2/d17/d1b/f58 [1411385,97692] 0 2024-03-26T18:54:38.362 INFO:tasks.workunit.client.0.smithi102.stdout:8/485: mkdir d0/d15/da0 0 2024-03-26T18:54:38.362 INFO:tasks.workunit.client.0.smithi102.stdout:8/486: truncate d0/d15/f4b 1721397 0 2024-03-26T18:54:38.362 INFO:tasks.workunit.client.0.smithi102.stdout:8/487: fsync d0/d15/d81/f73 0 2024-03-26T18:54:38.362 INFO:tasks.workunit.client.0.smithi102.stdout:8/488: creat d0/d15/fa1 x:0 0 0 2024-03-26T18:54:38.362 INFO:tasks.workunit.client.1.smithi190.stdout:7/882: dread d3/d1b/dd3/d10d/d65/d90/d92/d12d/fc5 [0,4194304] 0 2024-03-26T18:54:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:3/666: mknod df/d1c/d70/dcd/cf0 0 2024-03-26T18:54:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:3/667: readlink df/d29/d31/lc3 0 2024-03-26T18:54:38.364 INFO:tasks.workunit.client.1.smithi190.stdout:5/679: truncate d2/d17/d1b/db6/d24/d3b/f46 4055712 0 2024-03-26T18:54:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:0/470: rename d6/de/d13/d4a to d6/de/d13/d9a 0 2024-03-26T18:54:38.366 INFO:tasks.workunit.client.1.smithi190.stdout:7/883: getdents d3/d1b/dd3 0 2024-03-26T18:54:38.366 INFO:tasks.workunit.client.1.smithi190.stdout:2/806: dwrite d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d8b/fa8 [0,4194304] 0 2024-03-26T18:54:38.366 INFO:tasks.workunit.client.1.smithi190.stdout:2/807: creat d6/d9/d10/db7/dcf/f11a x:0 0 0 2024-03-26T18:54:38.366 INFO:tasks.workunit.client.1.smithi190.stdout:2/808: chown d6 20 1 2024-03-26T18:54:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:8/489: symlink d0/d15/la2 0 2024-03-26T18:54:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:8/490: write d0/d3/d2c/f7f [860757,112923] 0 2024-03-26T18:54:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:8/491: chown d0/d15/c7c 24188266 1 2024-03-26T18:54:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:3/668: link df/d1c/d34/f37 df/d29/d3a/d93/ff1 0 2024-03-26T18:54:38.370 INFO:tasks.workunit.client.1.smithi190.stdout:7/884: creat d3/d1b/dd3/d10d/d65/d109/d119/f14c x:0 0 0 2024-03-26T18:54:38.371 INFO:tasks.workunit.client.0.smithi102.stdout:9/452: dread df/d1a/d23/d3e/f6b [0,4194304] 0 2024-03-26T18:54:38.371 INFO:tasks.workunit.client.0.smithi102.stdout:9/453: fsync df/d1a/d57/f94 0 2024-03-26T18:54:38.371 INFO:tasks.workunit.client.0.smithi102.stdout:8/492: dread d0/d3/d26/d5c/f88 [0,4194304] 0 2024-03-26T18:54:38.371 INFO:tasks.workunit.client.0.smithi102.stdout:0/471: rmdir d6/de/d13/d8d 0 2024-03-26T18:54:38.374 INFO:tasks.workunit.client.0.smithi102.stdout:9/454: creat df/d1a/d7f/fad x:0 0 0 2024-03-26T18:54:38.374 INFO:tasks.workunit.client.0.smithi102.stdout:9/455: creat df/d1a/d65/fae x:0 0 0 2024-03-26T18:54:38.376 INFO:tasks.workunit.client.0.smithi102.stdout:8/493: rename d0/d3c/l6e to d0/d3/d26/d5c/la3 0 2024-03-26T18:54:38.377 INFO:tasks.workunit.client.1.smithi190.stdout:2/809: stat d6/d9/d45/d6b/c76 0 2024-03-26T18:54:38.377 INFO:tasks.workunit.client.1.smithi190.stdout:2/810: write d6/d9/d10/f8c [764710,52685] 0 2024-03-26T18:54:38.378 INFO:tasks.workunit.client.1.smithi190.stdout:7/885: mkdir d3/d14d 0 2024-03-26T18:54:38.378 INFO:tasks.workunit.client.0.smithi102.stdout:0/472: creat d6/de/d13/d1f/d71/f9b x:0 0 0 2024-03-26T18:54:38.379 INFO:tasks.workunit.client.0.smithi102.stdout:0/473: dread - d6/f91 zero size 2024-03-26T18:54:38.379 INFO:tasks.workunit.client.1.smithi190.stdout:2/811: truncate d6/d9/d10/d5f/fad 1612534 0 2024-03-26T18:54:38.379 INFO:tasks.workunit.client.1.smithi190.stdout:2/812: fsync d6/d21/d61/d6c/f9c 0 2024-03-26T18:54:38.379 INFO:tasks.workunit.client.1.smithi190.stdout:7/886: mkdir d3/d1b/dd3/d10d/d65/d90/d92/d9b/d14e 0 2024-03-26T18:54:38.381 INFO:tasks.workunit.client.0.smithi102.stdout:9/456: link df/d1a/d23/d29/l92 df/d1a/d57/laf 0 2024-03-26T18:54:38.382 INFO:tasks.workunit.client.1.smithi190.stdout:7/887: creat d3/d1b/dd3/d10d/f14f x:0 0 0 2024-03-26T18:54:38.383 INFO:tasks.workunit.client.0.smithi102.stdout:0/474: unlink d6/de/d13/d9a/f74 0 2024-03-26T18:54:38.386 INFO:tasks.workunit.client.0.smithi102.stdout:0/475: mknod d6/de/d13/d25/d31/d45/d82/d84/d88/c9c 0 2024-03-26T18:54:38.387 INFO:tasks.workunit.client.0.smithi102.stdout:0/476: unlink d6/fb 0 2024-03-26T18:54:38.389 INFO:tasks.workunit.client.0.smithi102.stdout:8/494: dread d0/d3c/f43 [0,4194304] 0 2024-03-26T18:54:38.390 INFO:tasks.workunit.client.0.smithi102.stdout:0/477: mknod d6/d3e/d69/d8c/d8f/c9d 0 2024-03-26T18:54:38.391 INFO:tasks.workunit.client.0.smithi102.stdout:7/431: dwrite d7/d12/f43 [0,4194304] 0 2024-03-26T18:54:38.392 INFO:tasks.workunit.client.1.smithi190.stdout:8/842: dwrite d9/dd/d23/d36/d37/d3a/d61/f91 [0,4194304] 0 2024-03-26T18:54:38.392 INFO:tasks.workunit.client.1.smithi190.stdout:8/843: write d9/dd/d23/db1/ff9 [750692,92222] 0 2024-03-26T18:54:38.392 INFO:tasks.workunit.client.1.smithi190.stdout:8/844: truncate d9/f1a 4998247 0 2024-03-26T18:54:38.392 INFO:tasks.workunit.client.1.smithi190.stdout:8/845: read d9/dd/f78 [1315,53481] 0 2024-03-26T18:54:38.393 INFO:tasks.workunit.client.0.smithi102.stdout:7/432: unlink d7/d10/d67/d69/d78/c7c 0 2024-03-26T18:54:38.395 INFO:tasks.workunit.client.0.smithi102.stdout:7/433: creat d7/d10/d17/f7d x:0 0 0 2024-03-26T18:54:38.396 INFO:tasks.workunit.client.0.smithi102.stdout:7/434: symlink d7/d12/d4d/d62/l7e 0 2024-03-26T18:54:38.399 INFO:tasks.workunit.client.0.smithi102.stdout:7/435: getdents d7 0 2024-03-26T18:54:38.400 INFO:tasks.workunit.client.1.smithi190.stdout:7/888: rmdir d3/d1b/dd3/d10d/d65/d109/d119 39 2024-03-26T18:54:38.400 INFO:tasks.workunit.client.0.smithi102.stdout:7/436: mknod d7/d12/d51/d6a/c7f 0 2024-03-26T18:54:38.403 INFO:tasks.workunit.client.0.smithi102.stdout:7/437: rename d7/d31/d56/f63 to d7/d10/d67/f80 0 2024-03-26T18:54:38.403 INFO:tasks.workunit.client.0.smithi102.stdout:7/438: truncate d7/d10/d27/d34/f49 4954708 0 2024-03-26T18:54:38.403 INFO:tasks.workunit.client.1.smithi190.stdout:7/889: link d3/d6/d9/d14/d36/f72 d3/d1b/f150 0 2024-03-26T18:54:38.403 INFO:tasks.workunit.client.1.smithi190.stdout:7/890: fdatasync d3/d6/d9/d14/d59/d73/d8b/f104 0 2024-03-26T18:54:38.404 INFO:tasks.workunit.client.0.smithi102.stdout:2/395: dwrite d3/f19 [0,4194304] 0 2024-03-26T18:54:38.404 INFO:tasks.workunit.client.0.smithi102.stdout:2/396: dread - d3/d2d/d69/f6a zero size 2024-03-26T18:54:38.404 INFO:tasks.workunit.client.0.smithi102.stdout:7/439: getdents d7/d12/d4b 0 2024-03-26T18:54:38.405 INFO:tasks.workunit.client.1.smithi190.stdout:7/891: rename d3/d1b/dd3/d10d/d50/dde/l140 to d3/d1b/dd3/d10d/d50/dde/dfd/l151 0 2024-03-26T18:54:38.407 INFO:tasks.workunit.client.1.smithi190.stdout:7/892: mkdir d3/d1b/dd3/d10d/d65/d109/d152 0 2024-03-26T18:54:38.407 INFO:tasks.workunit.client.1.smithi190.stdout:7/893: chown d3/d6/d9/d1f/d93/dfb/d102/c106 18097884 1 2024-03-26T18:54:38.407 INFO:tasks.workunit.client.1.smithi190.stdout:7/894: write d3/d1b/dd3/d10d/d65/d90/d92/d12d/f134 [630512,39407] 0 2024-03-26T18:54:38.407 INFO:tasks.workunit.client.1.smithi190.stdout:7/895: chown d3/d6/d9/d14/d59/d73/c135 24730689 1 2024-03-26T18:54:38.409 INFO:tasks.workunit.client.0.smithi102.stdout:0/478: write d6/de/d13/d9a/d85/f61 [3551717,21799] 0 2024-03-26T18:54:38.410 INFO:tasks.workunit.client.1.smithi190.stdout:7/896: rename d3/d6/c2d to d3/d1b/d57/dd1/c153 0 2024-03-26T18:54:38.410 INFO:tasks.workunit.client.0.smithi102.stdout:2/397: symlink d3/d20/d78/l88 0 2024-03-26T18:54:38.411 INFO:tasks.workunit.client.0.smithi102.stdout:5/670: dwrite d2/d14/f1c [0,4194304] 0 2024-03-26T18:54:38.411 INFO:tasks.workunit.client.0.smithi102.stdout:8/495: write d0/d3/d2c/d4e/f52 [524880,62569] 0 2024-03-26T18:54:38.411 INFO:tasks.workunit.client.0.smithi102.stdout:9/457: dwrite df/d1a/d1d/d72/f90 [0,4194304] 0 2024-03-26T18:54:38.411 INFO:tasks.workunit.client.0.smithi102.stdout:9/458: chown df/d1a/d65/l7a 2 1 2024-03-26T18:54:38.412 INFO:tasks.workunit.client.1.smithi190.stdout:7/897: creat d3/d6/d9/d14/d59/dbf/d148/f154 x:0 0 0 2024-03-26T18:54:38.412 INFO:tasks.workunit.client.1.smithi190.stdout:7/898: truncate d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/f8a 4947766 0 2024-03-26T18:54:38.413 INFO:tasks.workunit.client.0.smithi102.stdout:0/479: symlink d6/l9e 0 2024-03-26T18:54:38.413 INFO:tasks.workunit.client.0.smithi102.stdout:0/480: chown d6/de/d13/d1f/d55/d57/f30 217254316 1 2024-03-26T18:54:38.414 INFO:tasks.workunit.client.0.smithi102.stdout:2/398: getdents d3/d11 0 2024-03-26T18:54:38.415 INFO:tasks.workunit.client.1.smithi190.stdout:7/899: mkdir d3/d1b/dd3/d10d/d65/d109/d11b/d124/d155 0 2024-03-26T18:54:38.415 INFO:tasks.workunit.client.1.smithi190.stdout:7/900: getdents d3/d6/d9/d14/d36/d12a 0 2024-03-26T18:54:38.415 INFO:tasks.workunit.client.1.smithi190.stdout:7/901: write d3/d1b/dd3/d10d/d65/d90/d92/d12d/f134 [1531955,50180] 0 2024-03-26T18:54:38.415 INFO:tasks.workunit.client.1.smithi190.stdout:7/902: fdatasync d3/f29 0 2024-03-26T18:54:38.418 INFO:tasks.workunit.client.1.smithi190.stdout:7/903: mkdir d3/d6/d9/d14/d59/d73/d8b/d156 0 2024-03-26T18:54:38.418 INFO:tasks.workunit.client.1.smithi190.stdout:8/846: dwrite d9/dd/d23/d36/d37/d3a/d61/f72 [0,4194304] 0 2024-03-26T18:54:38.421 INFO:tasks.workunit.client.0.smithi102.stdout:8/496: unlink d0/d3/d26/d19/f90 0 2024-03-26T18:54:38.421 INFO:tasks.workunit.client.1.smithi190.stdout:7/904: rename d3/d1b/dd3/d10d/d65/d90/d112 to d3/d1b/dd3/d10d/d65/d109/d11b/d157 0 2024-03-26T18:54:38.421 INFO:tasks.workunit.client.1.smithi190.stdout:7/905: write d3/d1b/dd3/ff0 [184668,1584] 0 2024-03-26T18:54:38.422 INFO:tasks.workunit.client.0.smithi102.stdout:8/497: write d0/d3/d26/d19/f82 [579950,4248] 0 2024-03-26T18:54:38.422 INFO:tasks.workunit.client.0.smithi102.stdout:8/498: stat d0/d15/c7c 0 2024-03-26T18:54:38.422 INFO:tasks.workunit.client.0.smithi102.stdout:8/499: write d0/d3/d2c/f92 [264040,8505] 0 2024-03-26T18:54:38.423 INFO:tasks.workunit.client.1.smithi190.stdout:7/906: creat d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/f158 x:0 0 0 2024-03-26T18:54:38.424 INFO:tasks.workunit.client.1.smithi190.stdout:7/907: truncate d3/d6/d9/d14/d59/d73/d8b/ffa 442193 0 2024-03-26T18:54:38.424 INFO:tasks.workunit.client.0.smithi102.stdout:9/459: unlink l5 0 2024-03-26T18:54:38.424 INFO:tasks.workunit.client.1.smithi190.stdout:2/813: dread d6/d9/d10/f2d [0,4194304] 0 2024-03-26T18:54:38.425 INFO:tasks.workunit.client.1.smithi190.stdout:2/814: read d6/d9/d3a/d3d/d2e/d31/fec [301191,81397] 0 2024-03-26T18:54:38.425 INFO:tasks.workunit.client.0.smithi102.stdout:7/440: dwrite d7/d12/d51/d6a/f7a [0,4194304] 0 2024-03-26T18:54:38.425 INFO:tasks.workunit.client.0.smithi102.stdout:7/441: getdents d7/d31 0 2024-03-26T18:54:38.426 INFO:tasks.workunit.client.0.smithi102.stdout:7/442: write d7/d12/f38 [995248,9645] 0 2024-03-26T18:54:38.426 INFO:tasks.workunit.client.0.smithi102.stdout:7/443: creat d7/d10/d17/f81 x:0 0 0 2024-03-26T18:54:38.426 INFO:tasks.workunit.client.1.smithi190.stdout:7/908: mkdir d3/d1b/dd3/d10d/d50/d159 0 2024-03-26T18:54:38.426 INFO:tasks.workunit.client.1.smithi190.stdout:7/909: fdatasync d3/d6/f8c 0 2024-03-26T18:54:38.429 INFO:tasks.workunit.client.0.smithi102.stdout:5/671: mkdir d2/d25/d75/db9/ded 0 2024-03-26T18:54:38.430 INFO:tasks.workunit.client.1.smithi190.stdout:2/815: creat d6/d21/dd8/f11b x:0 0 0 2024-03-26T18:54:38.433 INFO:tasks.workunit.client.0.smithi102.stdout:6/445: sync 2024-03-26T18:54:38.434 INFO:tasks.workunit.client.0.smithi102.stdout:3/669: getdents df/d29/d3a/d93 0 2024-03-26T18:54:38.434 INFO:tasks.workunit.client.0.smithi102.stdout:3/670: write df/d61/d55/f90 [448297,62963] 0 2024-03-26T18:54:38.434 INFO:tasks.workunit.client.0.smithi102.stdout:3/671: dread - df/d29/d31/fb5 zero size 2024-03-26T18:54:38.435 INFO:tasks.workunit.client.1.smithi190.stdout:2/816: link d6/d9/d48/da0/f110 d6/d9/d3a/d3d/f11c 0 2024-03-26T18:54:38.437 INFO:tasks.workunit.client.1.smithi190.stdout:7/910: getdents d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/d8f 0 2024-03-26T18:54:38.437 INFO:tasks.workunit.client.1.smithi190.stdout:7/911: write d3/d1b/dd3/d10d/d50/f52 [344538,114194] 0 2024-03-26T18:54:38.439 INFO:tasks.workunit.client.1.smithi190.stdout:7/912: mkdir d3/d1b/dd3/d10d/d65/d90/d92/d9b/dc4/d15a 0 2024-03-26T18:54:38.442 INFO:tasks.workunit.client.0.smithi102.stdout:0/481: getdents d6/de/d13 0 2024-03-26T18:54:38.443 INFO:tasks.workunit.client.0.smithi102.stdout:5/672: rmdir d2/d14/de7 0 2024-03-26T18:54:38.443 INFO:tasks.workunit.client.0.smithi102.stdout:5/673: read d2/dea/f9f [532424,10258] 0 2024-03-26T18:54:38.444 INFO:tasks.workunit.client.1.smithi190.stdout:0/826: sync 2024-03-26T18:54:38.444 INFO:tasks.workunit.client.1.smithi190.stdout:0/827: dread - d0/dd/d20/d1c/d46/da7/df0/d87/d93/dc4/dd8/f115 zero size 2024-03-26T18:54:38.448 INFO:tasks.workunit.client.1.smithi190.stdout:0/828: creat d0/dd/d20/d1c/d8b/d8d/d8f/dfe/f12c x:0 0 0 2024-03-26T18:54:38.449 INFO:tasks.workunit.client.0.smithi102.stdout:8/500: getdents d0/d3/d2c/d2f 0 2024-03-26T18:54:38.449 INFO:tasks.workunit.client.0.smithi102.stdout:8/501: creat d0/d3c/fa4 x:0 0 0 2024-03-26T18:54:38.449 INFO:tasks.workunit.client.0.smithi102.stdout:8/502: write d0/d3c/d6f/f8b [915869,68111] 0 2024-03-26T18:54:38.450 INFO:tasks.workunit.client.1.smithi190.stdout:8/847: dwrite d9/dd/d23/d36/d37/d3a/fb7 [0,4194304] 0 2024-03-26T18:54:38.450 INFO:tasks.workunit.client.1.smithi190.stdout:8/848: stat d9/da2/dc4/ff3 0 2024-03-26T18:54:38.451 INFO:tasks.workunit.client.0.smithi102.stdout:3/672: mkdir df/d29/d31/d77/d8a/dae/dee/df2 0 2024-03-26T18:54:38.451 INFO:tasks.workunit.client.0.smithi102.stdout:3/673: write df/d61/dd6/fe6 [121096,8054] 0 2024-03-26T18:54:38.452 INFO:tasks.workunit.client.1.smithi190.stdout:8/849: mknod d9/d48/c11f 0 2024-03-26T18:54:38.453 INFO:tasks.workunit.client.1.smithi190.stdout:8/850: creat d9/dd/d23/dea/dfe/f120 x:0 0 0 2024-03-26T18:54:38.454 INFO:tasks.workunit.client.1.smithi190.stdout:0/829: creat d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/f12d x:0 0 0 2024-03-26T18:54:38.455 INFO:tasks.workunit.client.1.smithi190.stdout:8/851: rmdir d9/da2 39 2024-03-26T18:54:38.456 INFO:tasks.workunit.client.0.smithi102.stdout:5/674: mkdir d2/dea/d4f/da9/dee 0 2024-03-26T18:54:38.459 INFO:tasks.workunit.client.0.smithi102.stdout:0/482: link d6/l53 d6/de/d13/d1f/d55/d57/l9f 0 2024-03-26T18:54:38.460 INFO:tasks.workunit.client.0.smithi102.stdout:9/460: dwrite df/d1a/f3f [0,4194304] 0 2024-03-26T18:54:38.461 INFO:tasks.workunit.client.0.smithi102.stdout:6/446: dwrite d2/d12/f2d [0,4194304] 0 2024-03-26T18:54:38.463 INFO:tasks.workunit.client.0.smithi102.stdout:5/675: rename d2/d25/d75/db9/d43/d9e/d82/dc9/dd8 to d2/d25/d75/def 0 2024-03-26T18:54:38.466 INFO:tasks.workunit.client.0.smithi102.stdout:0/483: unlink d6/de/d13/d25/d4f/d66/f81 0 2024-03-26T18:54:38.466 INFO:tasks.workunit.client.1.smithi190.stdout:8/852: read d9/da2/d87/fe7 [852694,10223] 0 2024-03-26T18:54:38.466 INFO:tasks.workunit.client.1.smithi190.stdout:8/853: write d9/dd/d23/db1/ff9 [1671360,105775] 0 2024-03-26T18:54:38.468 INFO:tasks.workunit.client.1.smithi190.stdout:7/913: dwrite d3/d1b/dd3/d10d/d50/f69 [0,4194304] 0 2024-03-26T18:54:38.468 INFO:tasks.workunit.client.1.smithi190.stdout:7/914: write d3/d1b/dd3/d10d/d65/d90/d92/d9b/fd9 [895659,48978] 0 2024-03-26T18:54:38.468 INFO:tasks.workunit.client.1.smithi190.stdout:8/854: rename d9/dd/d15/f45 to d9/dd/d23/d36/d37/db9/f121 0 2024-03-26T18:54:38.468 INFO:tasks.workunit.client.1.smithi190.stdout:8/855: chown d9/d48/d49/fbe 12361 1 2024-03-26T18:54:38.470 INFO:tasks.workunit.client.0.smithi102.stdout:8/503: dwrite d0/d3/f28 [4194304,4194304] 0 2024-03-26T18:54:38.470 INFO:tasks.workunit.client.0.smithi102.stdout:8/504: write d0/d3/d2c/d33/d87/f6c [963337,55530] 0 2024-03-26T18:54:38.471 INFO:tasks.workunit.client.0.smithi102.stdout:6/447: unlink d2/d3d/d53/d5c/f5d 0 2024-03-26T18:54:38.478 INFO:tasks.workunit.client.0.smithi102.stdout:0/484: creat d6/d2d/d3d/fa0 x:0 0 0 2024-03-26T18:54:38.479 INFO:tasks.workunit.client.1.smithi190.stdout:7/915: rmdir d3/d1b/dd3/d10d/d50 39 2024-03-26T18:54:38.480 INFO:tasks.workunit.client.0.smithi102.stdout:8/505: truncate d0/d3/d31/f3a 3697232 0 2024-03-26T18:54:38.480 INFO:tasks.workunit.client.0.smithi102.stdout:8/506: write d0/d3/d31/f59 [4514378,58174] 0 2024-03-26T18:54:38.481 INFO:tasks.workunit.client.0.smithi102.stdout:0/485: mknod d6/de/d13/ca1 0 2024-03-26T18:54:38.481 INFO:tasks.workunit.client.0.smithi102.stdout:0/486: creat d6/d2d/d3d/fa2 x:0 0 0 2024-03-26T18:54:38.483 INFO:tasks.workunit.client.0.smithi102.stdout:6/448: rename d2/d7/f30 to d2/d7/f86 0 2024-03-26T18:54:38.486 INFO:tasks.workunit.client.0.smithi102.stdout:0/487: mknod d6/de/ca3 0 2024-03-26T18:54:38.486 INFO:tasks.workunit.client.1.smithi190.stdout:7/916: rename d3/d6/d9/d1f/d4c/fe6 to d3/d1b/f15b 0 2024-03-26T18:54:38.487 INFO:tasks.workunit.client.0.smithi102.stdout:6/449: truncate d2/f2e 1027686 0 2024-03-26T18:54:38.489 INFO:tasks.workunit.client.0.smithi102.stdout:0/488: truncate d6/de/d13/f7e 3091457 0 2024-03-26T18:54:38.489 INFO:tasks.workunit.client.0.smithi102.stdout:0/489: fsync d6/f7 0 2024-03-26T18:54:38.489 INFO:tasks.workunit.client.0.smithi102.stdout:0/490: read - d6/de/d13/d25/d31/f8e zero size 2024-03-26T18:54:38.494 INFO:tasks.workunit.client.0.smithi102.stdout:6/450: mknod d2/d7/c87 0 2024-03-26T18:54:38.494 INFO:tasks.workunit.client.0.smithi102.stdout:6/451: write d2/df/d35/d43/f57 [916441,52477] 0 2024-03-26T18:54:38.498 INFO:tasks.workunit.client.1.smithi190.stdout:8/856: dwrite d9/dd/d23/db1/f111 [0,4194304] 0 2024-03-26T18:54:38.500 INFO:tasks.workunit.client.0.smithi102.stdout:5/676: getdents d2/dea/d4f/da9 0 2024-03-26T18:54:38.500 INFO:tasks.workunit.client.0.smithi102.stdout:5/677: getdents d2/d25/de6 0 2024-03-26T18:54:38.500 INFO:tasks.workunit.client.0.smithi102.stdout:5/678: write d2/f12 [1288099,97453] 0 2024-03-26T18:54:38.500 INFO:tasks.workunit.client.0.smithi102.stdout:5/679: chown d2/d25/d75/db9/db3/lbc 1201 1 2024-03-26T18:54:38.500 INFO:tasks.workunit.client.0.smithi102.stdout:8/507: dwrite d0/d3/d26/d5c/f88 [0,4194304] 0 2024-03-26T18:54:38.501 INFO:tasks.workunit.client.0.smithi102.stdout:5/680: rename d2/d25/db0/fbd to d2/d25/d5b/ff0 0 2024-03-26T18:54:38.501 INFO:tasks.workunit.client.0.smithi102.stdout:5/681: dread - d2/d25/d5b/d72/f66 zero size 2024-03-26T18:54:38.502 INFO:tasks.workunit.client.0.smithi102.stdout:5/682: fsync d2/dea/d4f/f52 0 2024-03-26T18:54:38.503 INFO:tasks.workunit.client.0.smithi102.stdout:5/683: read d2/dea/f2e [58914,15156] 0 2024-03-26T18:54:38.504 INFO:tasks.workunit.client.1.smithi190.stdout:8/857: symlink d9/dd/d23/dea/dfe/l122 0 2024-03-26T18:54:38.515 INFO:tasks.workunit.client.1.smithi190.stdout:7/917: dwrite d3/d1b/d57/ff5 [0,4194304] 0 2024-03-26T18:54:38.516 INFO:tasks.workunit.client.1.smithi190.stdout:7/918: truncate d3/d6/d9/f75 4380508 0 2024-03-26T18:54:38.516 INFO:tasks.workunit.client.1.smithi190.stdout:7/919: stat d3/d1b/dd3/d10d/d65/d109/d119/f14c 0 2024-03-26T18:54:38.516 INFO:tasks.workunit.client.1.smithi190.stdout:7/920: truncate d3/d6/f91 1363139 0 2024-03-26T18:54:38.518 INFO:tasks.workunit.client.1.smithi190.stdout:7/921: mknod d3/d6/d9/d14/d59/dbf/c15c 0 2024-03-26T18:54:38.525 INFO:tasks.workunit.client.0.smithi102.stdout:6/452: dwrite d2/d3d/d53/f69 [0,4194304] 0 2024-03-26T18:54:38.526 INFO:tasks.workunit.client.0.smithi102.stdout:0/491: dwrite d6/de/d13/d1f/d55/d57/d7d/d7f/f8b [0,4194304] 0 2024-03-26T18:54:38.527 INFO:tasks.workunit.client.0.smithi102.stdout:0/492: dread - d6/de/d13/d25/d4f/d66/f95 zero size 2024-03-26T18:54:38.527 INFO:tasks.workunit.client.0.smithi102.stdout:0/493: readlink d6/l10 0 2024-03-26T18:54:38.527 INFO:tasks.workunit.client.1.smithi190.stdout:8/858: dwrite d9/da2/f63 [0,4194304] 0 2024-03-26T18:54:38.530 INFO:tasks.workunit.client.1.smithi190.stdout:8/859: rename d9/da2/d87 to d9/dd/d123 0 2024-03-26T18:54:38.531 INFO:tasks.workunit.client.1.smithi190.stdout:8/860: mknod d9/dd/d23/d88/c124 0 2024-03-26T18:54:38.531 INFO:tasks.workunit.client.1.smithi190.stdout:8/861: write d9/d39/d67/fba [2816261,56522] 0 2024-03-26T18:54:38.534 INFO:tasks.workunit.client.1.smithi190.stdout:7/922: dwrite d3/d1b/f15b [0,4194304] 0 2024-03-26T18:54:38.539 INFO:tasks.workunit.client.1.smithi190.stdout:7/923: creat d3/d1b/dd3/f15d x:0 0 0 2024-03-26T18:54:38.546 INFO:tasks.workunit.client.0.smithi102.stdout:8/508: unlink d0/d15/c7c 0 2024-03-26T18:54:38.546 INFO:tasks.workunit.client.0.smithi102.stdout:8/509: fdatasync d0/d3/d31/f59 0 2024-03-26T18:54:38.546 INFO:tasks.workunit.client.1.smithi190.stdout:7/924: read d3/d6/d9/f24 [277225,112502] 0 2024-03-26T18:54:38.547 INFO:tasks.workunit.client.0.smithi102.stdout:5/684: unlink d2/d2c/f5a 0 2024-03-26T18:54:38.553 INFO:tasks.workunit.client.0.smithi102.stdout:0/494: mkdir d6/de/d13/d25/da4 0 2024-03-26T18:54:38.555 INFO:tasks.workunit.client.0.smithi102.stdout:5/685: symlink d2/d25/de6/lf1 0 2024-03-26T18:54:38.556 INFO:tasks.workunit.client.0.smithi102.stdout:6/453: dwrite d2/d7/f8 [4194304,4194304] 0 2024-03-26T18:54:38.557 INFO:tasks.workunit.client.0.smithi102.stdout:0/495: symlink d6/de/d13/d1f/d55/la5 0 2024-03-26T18:54:38.560 INFO:tasks.workunit.client.0.smithi102.stdout:5/686: unlink d2/d25/d5b/d72/l99 0 2024-03-26T18:54:38.563 INFO:tasks.workunit.client.0.smithi102.stdout:0/496: creat d6/de/d13/d25/d31/d45/d82/d84/d88/fa6 x:0 0 0 2024-03-26T18:54:38.563 INFO:tasks.workunit.client.0.smithi102.stdout:0/497: write d6/de/d13/d25/d31/d45/d82/f8a [116775,89301] 0 2024-03-26T18:54:38.565 INFO:tasks.workunit.client.1.smithi190.stdout:8/862: dwrite d9/d48/f58 [4194304,4194304] 0 2024-03-26T18:54:38.567 INFO:tasks.workunit.client.0.smithi102.stdout:0/498: unlink d6/de/d13/d1f/c49 0 2024-03-26T18:54:38.567 INFO:tasks.workunit.client.0.smithi102.stdout:0/499: fdatasync d6/de/d13/d25/d31/f50 0 2024-03-26T18:54:38.567 INFO:tasks.workunit.client.1.smithi190.stdout:8/863: unlink d9/dd/f5a 0 2024-03-26T18:54:38.567 INFO:tasks.workunit.client.1.smithi190.stdout:8/864: readlink d9/dd/d23/d36/d37/d3a/l44 0 2024-03-26T18:54:38.568 INFO:tasks.workunit.client.0.smithi102.stdout:0/500: rename d6/de/ca3 to d6/d3e/d69/d8c/d8f/ca7 0 2024-03-26T18:54:38.569 INFO:tasks.workunit.client.0.smithi102.stdout:5/687: dread d2/f1d [0,4194304] 0 2024-03-26T18:54:38.569 INFO:tasks.workunit.client.0.smithi102.stdout:0/501: creat d6/de/d13/d1f/fa8 x:0 0 0 2024-03-26T18:54:38.569 INFO:tasks.workunit.client.0.smithi102.stdout:0/502: stat d6/f11 0 2024-03-26T18:54:38.570 INFO:tasks.workunit.client.0.smithi102.stdout:5/688: truncate d2/dea/f9f 610357 0 2024-03-26T18:54:38.575 INFO:tasks.workunit.client.0.smithi102.stdout:5/689: symlink d2/d25/d75/lf2 0 2024-03-26T18:54:38.577 INFO:tasks.workunit.client.0.smithi102.stdout:5/690: creat d2/dea/d4f/ddc/ff3 x:0 0 0 2024-03-26T18:54:38.581 INFO:tasks.workunit.client.1.smithi190.stdout:1/680: sync 2024-03-26T18:54:38.581 INFO:tasks.workunit.client.1.smithi190.stdout:6/677: sync 2024-03-26T18:54:38.581 INFO:tasks.workunit.client.1.smithi190.stdout:1/681: chown d3/da/d3a/da5 2 1 2024-03-26T18:54:38.581 INFO:tasks.workunit.client.1.smithi190.stdout:6/678: read d17/d28/d2b/f60 [2030267,59167] 0 2024-03-26T18:54:38.582 INFO:tasks.workunit.client.1.smithi190.stdout:4/854: sync 2024-03-26T18:54:38.582 INFO:tasks.workunit.client.1.smithi190.stdout:7/925: dwrite d3/d6/d9/d14/d59/d73/d8b/f104 [0,4194304] 0 2024-03-26T18:54:38.583 INFO:tasks.workunit.client.0.smithi102.stdout:8/510: dwrite d0/d3c/f43 [0,4194304] 0 2024-03-26T18:54:38.583 INFO:tasks.workunit.client.1.smithi190.stdout:1/682: mknod d3/d5f/cde 0 2024-03-26T18:54:38.584 INFO:tasks.workunit.client.1.smithi190.stdout:1/683: dread - d3/da/dd/d76/dc4/fdd zero size 2024-03-26T18:54:38.584 INFO:tasks.workunit.client.1.smithi190.stdout:6/679: mkdir d17/d28/d38/d79/ddf 0 2024-03-26T18:54:38.584 INFO:tasks.workunit.client.1.smithi190.stdout:6/680: readlink d17/d28/d2c/l76 0 2024-03-26T18:54:38.585 INFO:tasks.workunit.client.0.smithi102.stdout:5/691: dread d2/dea/f59 [0,4194304] 0 2024-03-26T18:54:38.585 INFO:tasks.workunit.client.0.smithi102.stdout:6/454: dwrite d2/d3d/d53/f69 [0,4194304] 0 2024-03-26T18:54:38.586 INFO:tasks.workunit.client.1.smithi190.stdout:7/926: link d3/d1b/dd3/d10d/d65/d90/d92/d9b/ca8 d3/d1b/dd3/d10d/d65/d90/d92/c15e 0 2024-03-26T18:54:38.586 INFO:tasks.workunit.client.1.smithi190.stdout:1/684: getdents d3/d9 0 2024-03-26T18:54:38.586 INFO:tasks.workunit.client.1.smithi190.stdout:1/685: readlink d3/da/dd/d95/l82 0 2024-03-26T18:54:38.586 INFO:tasks.workunit.client.1.smithi190.stdout:1/686: chown d3/d9/db/f32 968 1 2024-03-26T18:54:38.587 INFO:tasks.workunit.client.0.smithi102.stdout:8/511: creat d0/d3/d2c/d2f/fa5 x:0 0 0 2024-03-26T18:54:38.587 INFO:tasks.workunit.client.0.smithi102.stdout:8/512: creat d0/d3/d2c/d2f/fa6 x:0 0 0 2024-03-26T18:54:38.587 INFO:tasks.workunit.client.1.smithi190.stdout:7/927: dread d3/d6/d9/d1f/fa2 [0,4194304] 0 2024-03-26T18:54:38.587 INFO:tasks.workunit.client.1.smithi190.stdout:7/928: fdatasync d3/f16 0 2024-03-26T18:54:38.588 INFO:tasks.workunit.client.0.smithi102.stdout:5/692: symlink d2/d14/lf4 0 2024-03-26T18:54:38.589 INFO:tasks.workunit.client.1.smithi190.stdout:7/929: creat d3/d6/d9/d1f/d93/f15f x:0 0 0 2024-03-26T18:54:38.589 INFO:tasks.workunit.client.1.smithi190.stdout:7/930: dread - d3/d1b/dd3/d10d/d65/d109/d11b/f14a zero size 2024-03-26T18:54:38.589 INFO:tasks.workunit.client.1.smithi190.stdout:1/687: dread d3/da/dd/fb6 [0,4194304] 0 2024-03-26T18:54:38.589 INFO:tasks.workunit.client.1.smithi190.stdout:7/931: symlink d3/d6/d9/d1f/l160 0 2024-03-26T18:54:38.591 INFO:tasks.workunit.client.1.smithi190.stdout:7/932: rename d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/d87/f11f to d3/d6/d9/d1f/d4c/d71/dcc/d13f/f161 0 2024-03-26T18:54:38.592 INFO:tasks.workunit.client.1.smithi190.stdout:7/933: creat d3/d1b/dd3/d10d/d65/d144/f162 x:0 0 0 2024-03-26T18:54:38.594 INFO:tasks.workunit.client.1.smithi190.stdout:7/934: creat d3/d6/d9/d1f/d4c/d71/dcc/f163 x:0 0 0 2024-03-26T18:54:38.594 INFO:tasks.workunit.client.1.smithi190.stdout:4/855: dread d7/d8/d1c/f9f [0,4194304] 0 2024-03-26T18:54:38.595 INFO:tasks.workunit.client.1.smithi190.stdout:4/856: truncate d7/dc8/fd9 1009010 0 2024-03-26T18:54:38.595 INFO:tasks.workunit.client.1.smithi190.stdout:7/935: rmdir d3/d1b/dd3/d10d/d65/d109/d11b/d124 39 2024-03-26T18:54:38.595 INFO:tasks.workunit.client.1.smithi190.stdout:7/936: chown d3/d1b/dd3/d10d/d65/d90/d92/d9b 147578 1 2024-03-26T18:54:38.595 INFO:tasks.workunit.client.0.smithi102.stdout:8/513: rename d0/d15/d81 to d0/d3/d2c/da7 0 2024-03-26T18:54:38.596 INFO:tasks.workunit.client.1.smithi190.stdout:7/937: mknod d3/d6/d9/d1f/d4c/d71/c164 0 2024-03-26T18:54:38.596 INFO:tasks.workunit.client.0.smithi102.stdout:5/693: symlink d2/d25/d75/db9/d43/d9e/d82/dc9/lf5 0 2024-03-26T18:54:38.597 INFO:tasks.workunit.client.1.smithi190.stdout:8/865: dwrite d9/dd/d23/d36/d37/d3a/d61/daf/db3/fe2 [0,4194304] 0 2024-03-26T18:54:38.597 INFO:tasks.workunit.client.0.smithi102.stdout:6/455: rmdir d2/d3d/d53/d63/d6a 39 2024-03-26T18:54:38.598 INFO:tasks.workunit.client.0.smithi102.stdout:0/503: dwrite d6/de/d13/d1f/d71/f9b [0,4194304] 0 2024-03-26T18:54:38.600 INFO:tasks.workunit.client.0.smithi102.stdout:8/514: rename d0/d3/d26/d5c/f88 to d0/d3/d2c/d33/d87/fa8 0 2024-03-26T18:54:38.600 INFO:tasks.workunit.client.0.smithi102.stdout:1/417: sync 2024-03-26T18:54:38.600 INFO:tasks.workunit.client.1.smithi190.stdout:7/938: link d3/d1b/dd3/d10d/d50/f89 d3/d1b/dd3/d10d/d50/dde/f165 0 2024-03-26T18:54:38.601 INFO:tasks.workunit.client.0.smithi102.stdout:6/456: creat d2/d3d/d53/d77/f88 x:0 0 0 2024-03-26T18:54:38.603 INFO:tasks.workunit.client.1.smithi190.stdout:8/866: mkdir d9/dd/d23/d88/d125 0 2024-03-26T18:54:38.606 INFO:tasks.workunit.client.0.smithi102.stdout:0/504: symlink d6/de/d13/d1f/d55/d57/d7d/d7f/la9 0 2024-03-26T18:54:38.606 INFO:tasks.workunit.client.0.smithi102.stdout:0/505: chown d6/d2d/l6d 1412 1 2024-03-26T18:54:38.606 INFO:tasks.workunit.client.0.smithi102.stdout:0/506: chown d6/de/d13 14243969 1 2024-03-26T18:54:38.606 INFO:tasks.workunit.client.0.smithi102.stdout:0/507: fsync d6/de/d13/f7e 0 2024-03-26T18:54:38.609 INFO:tasks.workunit.client.0.smithi102.stdout:5/694: rename d2/d25/d75/db9/f89 to d2/d14/ff6 0 2024-03-26T18:54:38.610 INFO:tasks.workunit.client.0.smithi102.stdout:1/418: creat de/d16/d1d/d2a/f88 x:0 0 0 2024-03-26T18:54:38.612 INFO:tasks.workunit.client.0.smithi102.stdout:0/508: readlink d6/de/d13/d1f/l7a 0 2024-03-26T18:54:38.615 INFO:tasks.workunit.client.1.smithi190.stdout:1/688: dwrite d3/da/f47 [0,4194304] 0 2024-03-26T18:54:38.616 INFO:tasks.workunit.client.1.smithi190.stdout:7/939: dread d3/d1b/d57/ff5 [0,4194304] 0 2024-03-26T18:54:38.616 INFO:tasks.workunit.client.0.smithi102.stdout:6/457: rename d2/d7/d3c/d78/f81 to d2/df/d72/f89 0 2024-03-26T18:54:38.616 INFO:tasks.workunit.client.0.smithi102.stdout:6/458: creat d2/d3d/f8a x:0 0 0 2024-03-26T18:54:38.617 INFO:tasks.workunit.client.1.smithi190.stdout:7/940: mknod d3/d1b/dd2/c166 0 2024-03-26T18:54:38.617 INFO:tasks.workunit.client.0.smithi102.stdout:1/419: creat de/d16/d1d/d2a/d2c/d56/d6e/f89 x:0 0 0 2024-03-26T18:54:38.620 INFO:tasks.workunit.client.0.smithi102.stdout:0/509: mkdir d6/de/d13/d25/d31/daa 0 2024-03-26T18:54:38.620 INFO:tasks.workunit.client.0.smithi102.stdout:0/510: creat d6/de/d13/d25/d31/d45/d82/d84/d88/fab x:0 0 0 2024-03-26T18:54:38.623 INFO:tasks.workunit.client.1.smithi190.stdout:1/689: mknod d3/da/d45/cdf 0 2024-03-26T18:54:38.623 INFO:tasks.workunit.client.1.smithi190.stdout:4/857: dwrite d7/d8/d14/f94 [0,4194304] 0 2024-03-26T18:54:38.624 INFO:tasks.workunit.client.0.smithi102.stdout:6/459: mknod d2/df/d35/d43/d75/c8b 0 2024-03-26T18:54:38.624 INFO:tasks.workunit.client.0.smithi102.stdout:6/460: chown d2/d12/l49 1885037 1 2024-03-26T18:54:38.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:38 smithi102 ceph-mon[29002]: pgmap v247: 65 pgs: 65 active+clean; 2.2 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 210 MiB/s rd, 249 MiB/s wr, 384 op/s 2024-03-26T18:54:38.631 INFO:tasks.workunit.client.0.smithi102.stdout:0/511: unlink d6/de/d13/f5a 0 2024-03-26T18:54:38.633 INFO:tasks.workunit.client.0.smithi102.stdout:6/461: creat d2/df/d35/d43/d75/f8c x:0 0 0 2024-03-26T18:54:38.633 INFO:tasks.workunit.client.0.smithi102.stdout:6/462: truncate d2/d3d/d53/d77/f88 11821 0 2024-03-26T18:54:38.634 INFO:tasks.workunit.client.0.smithi102.stdout:0/512: truncate d6/de/d13/d25/d4f/d66/f90 160704 0 2024-03-26T18:54:38.636 INFO:tasks.workunit.client.1.smithi190.stdout:4/858: write d7/d8/f2a [4565086,31522] 0 2024-03-26T18:54:38.637 INFO:tasks.workunit.client.0.smithi102.stdout:6/463: symlink d2/d7/l8d 0 2024-03-26T18:54:38.637 INFO:tasks.workunit.client.0.smithi102.stdout:6/464: readlink d2/df/l18 0 2024-03-26T18:54:38.637 INFO:tasks.workunit.client.0.smithi102.stdout:6/465: creat d2/df/f8e x:0 0 0 2024-03-26T18:54:38.637 INFO:tasks.workunit.client.0.smithi102.stdout:6/466: write d2/f33 [1722721,8034] 0 2024-03-26T18:54:38.638 INFO:tasks.workunit.client.1.smithi190.stdout:4/859: creat d7/d8/d14/d6e/f128 x:0 0 0 2024-03-26T18:54:38.639 INFO:tasks.workunit.client.1.smithi190.stdout:1/690: write d3/d9/f2a [1057950,124171] 0 2024-03-26T18:54:38.639 INFO:tasks.workunit.client.1.smithi190.stdout:1/691: write d3/da/dd/d95/d52/fba [1174505,35783] 0 2024-03-26T18:54:38.639 INFO:tasks.workunit.client.1.smithi190.stdout:4/860: symlink d7/d8/d14/d4b/d7d/da1/df8/l129 0 2024-03-26T18:54:38.641 INFO:tasks.workunit.client.1.smithi190.stdout:4/861: creat d7/d8/d14/d4b/d7d/da1/d113/f12a x:0 0 0 2024-03-26T18:54:38.641 INFO:tasks.workunit.client.1.smithi190.stdout:1/692: creat d3/da/dd/fe0 x:0 0 0 2024-03-26T18:54:38.647 INFO:tasks.workunit.client.1.smithi190.stdout:8/867: dwrite d9/dd/d23/fff [0,4194304] 0 2024-03-26T18:54:38.648 INFO:tasks.workunit.client.0.smithi102.stdout:5/695: dwrite d2/d14/d1a/f1b [0,4194304] 0 2024-03-26T18:54:38.648 INFO:tasks.workunit.client.0.smithi102.stdout:0/513: write d6/de/f4b [7595255,46725] 0 2024-03-26T18:54:38.649 INFO:tasks.workunit.client.1.smithi190.stdout:7/941: dwrite d3/d1b/d57/dd1/fea [0,4194304] 0 2024-03-26T18:54:38.651 INFO:tasks.workunit.client.0.smithi102.stdout:1/420: dwrite de/f73 [0,4194304] 0 2024-03-26T18:54:38.651 INFO:tasks.workunit.client.0.smithi102.stdout:5/696: dread d2/dea/f9f [0,4194304] 0 2024-03-26T18:54:38.652 INFO:tasks.workunit.client.0.smithi102.stdout:5/697: read d2/d25/f81 [43619,74801] 0 2024-03-26T18:54:38.653 INFO:tasks.workunit.client.0.smithi102.stdout:1/421: read de/d1f/f3a [667546,5034] 0 2024-03-26T18:54:38.655 INFO:tasks.workunit.client.0.smithi102.stdout:5/698: mknod d2/dea/d4f/ddc/de9/cf7 0 2024-03-26T18:54:38.656 INFO:tasks.workunit.client.0.smithi102.stdout:1/422: creat de/d16/d1d/d2a/d2c/f8a x:0 0 0 2024-03-26T18:54:38.657 INFO:tasks.workunit.client.0.smithi102.stdout:1/423: chown de/d1f/c69 0 1 2024-03-26T18:54:38.658 INFO:tasks.workunit.client.0.smithi102.stdout:1/424: creat de/d16/d1d/d2a/d2c/d56/d5f/f8b x:0 0 0 2024-03-26T18:54:38.658 INFO:tasks.workunit.client.0.smithi102.stdout:1/425: dread - de/d16/d1d/f87 zero size 2024-03-26T18:54:38.659 INFO:tasks.workunit.client.1.smithi190.stdout:8/868: write d9/dd/f2d [1769276,27107] 0 2024-03-26T18:54:38.660 INFO:tasks.workunit.client.0.smithi102.stdout:1/426: rename de/d16/f34 to de/d16/d1d/d2a/d2c/d56/d78/f8c 0 2024-03-26T18:54:38.661 INFO:tasks.workunit.client.0.smithi102.stdout:1/427: truncate de/d16/d1d/d2a/d2c/f8a 867160 0 2024-03-26T18:54:38.661 INFO:tasks.workunit.client.1.smithi190.stdout:8/869: creat d9/dd/d15/dd3/f126 x:0 0 0 2024-03-26T18:54:38.669 INFO:tasks.workunit.client.1.smithi190.stdout:7/942: write d3/d6/d9/d14/fc2 [808725,105862] 0 2024-03-26T18:54:38.670 INFO:tasks.workunit.client.1.smithi190.stdout:7/943: write d3/d1b/dd3/d10d/d50/f89 [104913,26037] 0 2024-03-26T18:54:38.672 INFO:tasks.workunit.client.1.smithi190.stdout:7/944: creat d3/d6/d9/d1f/d93/dfb/f167 x:0 0 0 2024-03-26T18:54:38.674 INFO:tasks.workunit.client.1.smithi190.stdout:4/862: write d7/f35 [7080360,107531] 0 2024-03-26T18:54:38.675 INFO:tasks.workunit.client.1.smithi190.stdout:7/945: getdents d3/d6/d9/d1f/d4c/d71/dcc/d13f 0 2024-03-26T18:54:38.675 INFO:tasks.workunit.client.1.smithi190.stdout:7/946: fdatasync d3/f29 0 2024-03-26T18:54:38.675 INFO:tasks.workunit.client.1.smithi190.stdout:7/947: write d3/d1b/dd3/d10d/f44 [1778060,27598] 0 2024-03-26T18:54:38.675 INFO:tasks.workunit.client.1.smithi190.stdout:7/948: truncate d3/d6/d9/d14/d36/f48 773946 0 2024-03-26T18:54:38.676 INFO:tasks.workunit.client.0.smithi102.stdout:1/428: write de/f19 [7673911,95486] 0 2024-03-26T18:54:38.679 INFO:tasks.workunit.client.1.smithi190.stdout:4/863: symlink d7/d8/d14/d23/d28/d38/d96/d9a/l12b 0 2024-03-26T18:54:38.680 INFO:tasks.workunit.client.1.smithi190.stdout:4/864: stat d7/d8/d14/d23/d28/d9e 0 2024-03-26T18:54:38.681 INFO:tasks.workunit.client.1.smithi190.stdout:1/693: rmdir d3 39 2024-03-26T18:54:38.681 INFO:tasks.workunit.client.1.smithi190.stdout:1/694: readlink d3/d9/l10 0 2024-03-26T18:54:38.681 INFO:tasks.workunit.client.1.smithi190.stdout:1/695: fsync d3/da/dd/d95/dda/fdc 0 2024-03-26T18:54:38.682 INFO:tasks.workunit.client.1.smithi190.stdout:7/949: link l2 d3/d6/d9/d1f/d14b/l168 0 2024-03-26T18:54:38.682 INFO:tasks.workunit.client.1.smithi190.stdout:7/950: fsync d3/d1b/dd3/d10d/d65/d109/d11b/f14a 0 2024-03-26T18:54:38.682 INFO:tasks.workunit.client.1.smithi190.stdout:4/865: mknod d7/d8/c12c 0 2024-03-26T18:54:38.683 INFO:tasks.workunit.client.1.smithi190.stdout:4/866: write d7/d8/d14/d4b/d7d/fd0 [1294896,69684] 0 2024-03-26T18:54:38.683 INFO:tasks.workunit.client.1.smithi190.stdout:4/867: getdents d7/d1d/d42/d4a/dfe 0 2024-03-26T18:54:38.684 INFO:tasks.workunit.client.1.smithi190.stdout:7/951: symlink d3/d1b/dd3/d10d/d65/d90/l169 0 2024-03-26T18:54:38.684 INFO:tasks.workunit.client.1.smithi190.stdout:3/644: sync 2024-03-26T18:54:38.685 INFO:tasks.workunit.client.1.smithi190.stdout:4/868: mknod d7/d8/d14/d23/d28/d38/dd3/dfd/c12d 0 2024-03-26T18:54:38.686 INFO:tasks.workunit.client.1.smithi190.stdout:4/869: write d7/d8/f2a [9420310,76064] 0 2024-03-26T18:54:38.686 INFO:tasks.workunit.client.1.smithi190.stdout:3/645: symlink d1/d3/df/d31/lce 0 2024-03-26T18:54:38.687 INFO:tasks.workunit.client.1.smithi190.stdout:3/646: creat d1/d3/df/d18/d35/d41/dae/d9c/fcf x:0 0 0 2024-03-26T18:54:38.687 INFO:tasks.workunit.client.1.smithi190.stdout:4/870: creat d7/d8/d127/f12e x:0 0 0 2024-03-26T18:54:38.688 INFO:tasks.workunit.client.0.smithi102.stdout:6/467: dwrite d2/d3d/d53/f76 [0,4194304] 0 2024-03-26T18:54:38.689 INFO:tasks.workunit.client.1.smithi190.stdout:4/871: link d7/d1d/l3e d7/d1d/d42/d4a/dfe/l12f 0 2024-03-26T18:54:38.689 INFO:tasks.workunit.client.1.smithi190.stdout:4/872: creat d7/d1d/d5e/d98/f130 x:0 0 0 2024-03-26T18:54:38.692 INFO:tasks.workunit.client.0.smithi102.stdout:5/699: dwrite d2/d25/d75/db9/d43/d9e/daf/fd2 [0,4194304] 0 2024-03-26T18:54:38.693 INFO:tasks.workunit.client.1.smithi190.stdout:1/696: dread d3/da/dd/d2f/d5b/f70 [0,4194304] 0 2024-03-26T18:54:38.693 INFO:tasks.workunit.client.1.smithi190.stdout:1/697: dread - d3/d9/db/fc5 zero size 2024-03-26T18:54:38.694 INFO:tasks.workunit.client.0.smithi102.stdout:5/700: unlink d2/dea/db4/fd1 0 2024-03-26T18:54:38.694 INFO:tasks.workunit.client.0.smithi102.stdout:5/701: stat d2/d2c/f58 0 2024-03-26T18:54:38.695 INFO:tasks.workunit.client.0.smithi102.stdout:5/702: read d2/f12 [1541376,79096] 0 2024-03-26T18:54:38.695 INFO:tasks.workunit.client.0.smithi102.stdout:5/703: creat d2/d25/db0/ff8 x:0 0 0 2024-03-26T18:54:38.697 INFO:tasks.workunit.client.0.smithi102.stdout:5/704: dread d2/dea/f9f [0,4194304] 0 2024-03-26T18:54:38.698 INFO:tasks.workunit.client.0.smithi102.stdout:5/705: dread d2/dea/dbb/dcf/fda [0,4194304] 0 2024-03-26T18:54:38.698 INFO:tasks.workunit.client.1.smithi190.stdout:7/952: write d3/d1b/d57/dd1/fea [3341302,47929] 0 2024-03-26T18:54:38.702 INFO:tasks.workunit.client.1.smithi190.stdout:3/647: read d1/de/f69 [3002856,9445] 0 2024-03-26T18:54:38.702 INFO:tasks.workunit.client.1.smithi190.stdout:3/648: chown d1/d3/df/d18/d35/d89/c9a 41866375 1 2024-03-26T18:54:38.703 INFO:tasks.workunit.client.0.smithi102.stdout:1/429: dwrite de/d16/d1d/d2a/d2c/d56/d6e/f81 [0,4194304] 0 2024-03-26T18:54:38.703 INFO:tasks.workunit.client.0.smithi102.stdout:0/514: dwrite d6/de/d13/d25/d4f/f92 [0,4194304] 0 2024-03-26T18:54:38.704 INFO:tasks.workunit.client.0.smithi102.stdout:6/468: dread d2/d7/f51 [0,4194304] 0 2024-03-26T18:54:38.704 INFO:tasks.workunit.client.0.smithi102.stdout:1/430: rmdir de/d16/d1d/d2a/d2c/d56 39 2024-03-26T18:54:38.705 INFO:tasks.workunit.client.1.smithi190.stdout:3/649: symlink d1/de/d81/ld0 0 2024-03-26T18:54:38.705 INFO:tasks.workunit.client.0.smithi102.stdout:2/399: sync 2024-03-26T18:54:38.706 INFO:tasks.workunit.client.0.smithi102.stdout:2/400: write d3/d11/d15/d31/d39/f64 [1016989,31393] 0 2024-03-26T18:54:38.706 INFO:tasks.workunit.client.0.smithi102.stdout:7/444: sync 2024-03-26T18:54:38.706 INFO:tasks.workunit.client.0.smithi102.stdout:4/357: sync 2024-03-26T18:54:38.706 INFO:tasks.workunit.client.1.smithi190.stdout:4/873: write d7/d8/d14/d4b/f4c [1241135,9020] 0 2024-03-26T18:54:38.708 INFO:tasks.workunit.client.1.smithi190.stdout:3/650: mkdir d1/d3/df/d31/dd1 0 2024-03-26T18:54:38.708 INFO:tasks.workunit.client.1.smithi190.stdout:3/651: fsync d1/d3/df/d18/d35/f60 0 2024-03-26T18:54:38.709 INFO:tasks.workunit.client.1.smithi190.stdout:8/870: dwrite d9/d39/f82 [0,4194304] 0 2024-03-26T18:54:38.709 INFO:tasks.workunit.client.1.smithi190.stdout:8/871: chown d9/d48/f62 16614093 1 2024-03-26T18:54:38.709 INFO:tasks.workunit.client.0.smithi102.stdout:0/515: mkdir d6/de/d93/dac 0 2024-03-26T18:54:38.709 INFO:tasks.workunit.client.0.smithi102.stdout:0/516: write d6/d2d/d3d/f40 [1893027,125213] 0 2024-03-26T18:54:38.710 INFO:tasks.workunit.client.1.smithi190.stdout:5/680: sync 2024-03-26T18:54:38.710 INFO:tasks.workunit.client.0.smithi102.stdout:6/469: mknod d2/df/d35/d60/c8f 0 2024-03-26T18:54:38.711 INFO:tasks.workunit.client.0.smithi102.stdout:5/706: dread d2/d25/d5b/d72/f42 [0,4194304] 0 2024-03-26T18:54:38.711 INFO:tasks.workunit.client.0.smithi102.stdout:5/707: stat d2/d25/d5b/ff0 0 2024-03-26T18:54:38.711 INFO:tasks.workunit.client.0.smithi102.stdout:5/708: chown d2/dea/d4f/c90 995945858 1 2024-03-26T18:54:38.711 INFO:tasks.workunit.client.0.smithi102.stdout:4/358: dread d1/d27/f4d [0,4194304] 0 2024-03-26T18:54:38.711 INFO:tasks.workunit.client.0.smithi102.stdout:4/359: readlink d1/d17/d2a/l40 0 2024-03-26T18:54:38.712 INFO:tasks.workunit.client.0.smithi102.stdout:1/431: link de/d16/d1d/d2a/d2c/f8a de/f8d 0 2024-03-26T18:54:38.715 INFO:tasks.workunit.client.1.smithi190.stdout:5/681: creat d2/d17/d1b/d30/fde x:0 0 0 2024-03-26T18:54:38.717 INFO:tasks.workunit.client.0.smithi102.stdout:2/401: rename d3/d11/d15/f48 to d3/d2d/f89 0 2024-03-26T18:54:38.720 INFO:tasks.workunit.client.0.smithi102.stdout:7/445: mkdir d7/d10/d27/d34/d82 0 2024-03-26T18:54:38.722 INFO:tasks.workunit.client.0.smithi102.stdout:5/709: symlink d2/d25/d75/db9/ded/lf9 0 2024-03-26T18:54:38.723 INFO:tasks.workunit.client.1.smithi190.stdout:8/872: truncate d9/d39/d67/f95 364835 0 2024-03-26T18:54:38.723 INFO:tasks.workunit.client.0.smithi102.stdout:4/360: mkdir d1/d25/d39/d3e/d74 0 2024-03-26T18:54:38.725 INFO:tasks.workunit.client.0.smithi102.stdout:1/432: mkdir de/d16/d1d/d2a/d2c/d56/d78/d80/d8e 0 2024-03-26T18:54:38.725 INFO:tasks.workunit.client.0.smithi102.stdout:1/433: dread - de/f83 zero size 2024-03-26T18:54:38.730 INFO:tasks.workunit.client.0.smithi102.stdout:6/470: rename d2/d12/l2f to d2/df/d35/d43/l90 0 2024-03-26T18:54:38.730 INFO:tasks.workunit.client.0.smithi102.stdout:6/471: mkdir d2/df/d71/d80/d91 0 2024-03-26T18:54:38.731 INFO:tasks.workunit.client.1.smithi190.stdout:7/953: dwrite d3/d1b/dd3/d10d/d50/f60 [0,4194304] 0 2024-03-26T18:54:38.731 INFO:tasks.workunit.client.1.smithi190.stdout:4/874: dwrite d7/d8/d14/d23/d28/d38/d96/f11c [0,4194304] 0 2024-03-26T18:54:38.736 INFO:tasks.workunit.client.1.smithi190.stdout:7/954: creat d3/d1b/dd3/d10d/d65/d109/d11b/d124/d155/f16a x:0 0 0 2024-03-26T18:54:38.736 INFO:tasks.workunit.client.1.smithi190.stdout:7/955: write d3/f29 [353798,43853] 0 2024-03-26T18:54:38.738 INFO:tasks.workunit.client.1.smithi190.stdout:1/698: dwrite f2 [4194304,4194304] 0 2024-03-26T18:54:38.738 INFO:tasks.workunit.client.1.smithi190.stdout:1/699: stat d3/d73/ccf 0 2024-03-26T18:54:38.743 INFO:tasks.workunit.client.1.smithi190.stdout:7/956: creat d3/d1b/dd3/d10d/d50/d117/f16b x:0 0 0 2024-03-26T18:54:38.743 INFO:tasks.workunit.client.1.smithi190.stdout:7/957: chown d3/d1b/dd3/d10d/d65/d90/d92/fbc 5 1 2024-03-26T18:54:38.743 INFO:tasks.workunit.client.1.smithi190.stdout:3/652: dread d1/f5 [0,4194304] 0 2024-03-26T18:54:38.743 INFO:tasks.workunit.client.1.smithi190.stdout:3/653: truncate d1/d3/fbf 768582 0 2024-03-26T18:54:38.746 INFO:tasks.workunit.client.1.smithi190.stdout:7/958: link d3/d1b/dd3/d10d/f55 d3/d6/d9/d14/d59/f16c 0 2024-03-26T18:54:38.751 INFO:tasks.workunit.client.1.smithi190.stdout:7/959: unlink d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/d8f/ld7 0 2024-03-26T18:54:38.751 INFO:tasks.workunit.client.1.smithi190.stdout:7/960: chown d3/f7c 110093277 1 2024-03-26T18:54:38.752 INFO:tasks.workunit.client.1.smithi190.stdout:7/961: fdatasync d3/d6/d9/d1f/d4c/fd4 0 2024-03-26T18:54:38.758 INFO:tasks.workunit.client.1.smithi190.stdout:5/682: dwrite d2/d17/d1b/db6/f77 [0,4194304] 0 2024-03-26T18:54:38.759 INFO:tasks.workunit.client.1.smithi190.stdout:4/875: dread d7/d8/d14/f52 [0,4194304] 0 2024-03-26T18:54:38.759 INFO:tasks.workunit.client.1.smithi190.stdout:3/654: write d1/f5 [2037047,82182] 0 2024-03-26T18:54:38.762 INFO:tasks.workunit.client.1.smithi190.stdout:4/876: creat d7/d8/f131 x:0 0 0 2024-03-26T18:54:38.762 INFO:tasks.workunit.client.1.smithi190.stdout:4/877: read - d7/d8/d14/d5c/f112 zero size 2024-03-26T18:54:38.763 INFO:tasks.workunit.client.0.smithi102.stdout:0/517: dread d6/de/d13/d9a/f68 [0,4194304] 0 2024-03-26T18:54:38.764 INFO:tasks.workunit.client.0.smithi102.stdout:0/518: chown d6/de/d13/d25/d31 444 1 2024-03-26T18:54:38.767 INFO:tasks.workunit.client.1.smithi190.stdout:0/830: sync 2024-03-26T18:54:38.767 INFO:tasks.workunit.client.1.smithi190.stdout:2/817: sync 2024-03-26T18:54:38.767 INFO:tasks.workunit.client.1.smithi190.stdout:2/818: chown d6/d21 7174 1 2024-03-26T18:54:38.767 INFO:tasks.workunit.client.0.smithi102.stdout:0/519: symlink d6/de/lad 0 2024-03-26T18:54:38.767 INFO:tasks.workunit.client.0.smithi102.stdout:0/520: creat d6/de/d13/d25/d31/d45/d82/d84/d88/fae x:0 0 0 2024-03-26T18:54:38.769 INFO:tasks.workunit.client.1.smithi190.stdout:2/819: mkdir d6/d9/d3a/d3d/d2e/d10e/d9e/dfc/d11d 0 2024-03-26T18:54:38.770 INFO:tasks.workunit.client.1.smithi190.stdout:0/831: rmdir d0/dd/d20/d1c/d8b/d8d/d8f/dfe 39 2024-03-26T18:54:38.773 INFO:tasks.workunit.client.1.smithi190.stdout:2/820: mknod d6/d9/d45/d6b/d93/dc0/c11e 0 2024-03-26T18:54:38.773 INFO:tasks.workunit.client.1.smithi190.stdout:2/821: chown d6/d9/d45/f74 202585 1 2024-03-26T18:54:38.775 INFO:tasks.workunit.client.1.smithi190.stdout:2/822: creat d6/d9/d3a/d3d/d2e/d10e/d9e/da3/f11f x:0 0 0 2024-03-26T18:54:38.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:38 smithi190 ceph-mon[35043]: pgmap v247: 65 pgs: 65 active+clean; 2.2 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 210 MiB/s rd, 249 MiB/s wr, 384 op/s 2024-03-26T18:54:38.783 INFO:tasks.workunit.client.1.smithi190.stdout:5/683: write d2/d17/d1b/f36 [2830217,66696] 0 2024-03-26T18:54:38.787 INFO:tasks.workunit.client.1.smithi190.stdout:2/823: write d6/d9/d3a/d3d/d2e/d31/f91 [3536678,39408] 0 2024-03-26T18:54:38.795 INFO:tasks.workunit.client.0.smithi102.stdout:2/402: dwrite d3/d11/f55 [0,4194304] 0 2024-03-26T18:54:38.795 INFO:tasks.workunit.client.0.smithi102.stdout:7/446: dwrite d7/d10/d27/d34/f49 [0,4194304] 0 2024-03-26T18:54:38.795 INFO:tasks.workunit.client.0.smithi102.stdout:7/447: chown d7/d10/d27/d34/d72 1219631040 1 2024-03-26T18:54:38.795 INFO:tasks.workunit.client.0.smithi102.stdout:7/448: chown d7/ff 6599872 1 2024-03-26T18:54:38.795 INFO:tasks.workunit.client.0.smithi102.stdout:7/449: chown d7/d12/d2b 412 1 2024-03-26T18:54:38.796 INFO:tasks.workunit.client.0.smithi102.stdout:2/403: link d3/d20/c2c d3/d11/d15/d31/d39/c8a 0 2024-03-26T18:54:38.797 INFO:tasks.workunit.client.0.smithi102.stdout:4/361: dwrite d1/d25/d39/d3e/d4c/f6a [0,4194304] 0 2024-03-26T18:54:38.797 INFO:tasks.workunit.client.0.smithi102.stdout:4/362: chown d1/d17/d1a/f4f 55664 1 2024-03-26T18:54:38.798 INFO:tasks.workunit.client.0.smithi102.stdout:2/404: creat d3/d11/d15/d46/f8b x:0 0 0 2024-03-26T18:54:38.798 INFO:tasks.workunit.client.1.smithi190.stdout:0/832: dread d0/dd/d20/d1c/d46/da7/df0/da2/d80/f10c [0,4194304] 0 2024-03-26T18:54:38.800 INFO:tasks.workunit.client.1.smithi190.stdout:0/833: mkdir d0/dd/d20/d1c/d46/d58/de4/d12e 0 2024-03-26T18:54:38.801 INFO:tasks.workunit.client.0.smithi102.stdout:4/363: dread d1/d25/d39/d3e/d4c/f68 [0,4194304] 0 2024-03-26T18:54:38.802 INFO:tasks.workunit.client.1.smithi190.stdout:5/684: write d2/d17/d1b/db6/fc1 [2008475,123730] 0 2024-03-26T18:54:38.802 INFO:tasks.workunit.client.1.smithi190.stdout:5/685: write d2/d17/d1b/fbe [698873,34712] 0 2024-03-26T18:54:38.802 INFO:tasks.workunit.client.1.smithi190.stdout:0/834: write d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/faf [141653,129649] 0 2024-03-26T18:54:38.802 INFO:tasks.workunit.client.0.smithi102.stdout:4/364: mkdir d1/d75 0 2024-03-26T18:54:38.804 INFO:tasks.workunit.client.1.smithi190.stdout:0/835: mknod d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/d121/c12f 0 2024-03-26T18:54:38.804 INFO:tasks.workunit.client.1.smithi190.stdout:0/836: creat d0/dd/d20/d1c/d46/da7/df0/d87/d93/dc4/dd8/f130 x:0 0 0 2024-03-26T18:54:38.805 INFO:tasks.workunit.client.0.smithi102.stdout:4/365: getdents d1/d17/d2a 0 2024-03-26T18:54:38.805 INFO:tasks.workunit.client.1.smithi190.stdout:0/837: dread d0/dd/d20/d1c/d4e/f9d [0,4194304] 0 2024-03-26T18:54:38.806 INFO:tasks.workunit.client.1.smithi190.stdout:0/838: write d0/dd5/dee/f101 [1028740,52203] 0 2024-03-26T18:54:38.806 INFO:tasks.workunit.client.1.smithi190.stdout:0/839: write d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/f36 [5650261,30843] 0 2024-03-26T18:54:38.807 INFO:tasks.workunit.client.1.smithi190.stdout:0/840: truncate d0/dd/d20/d1c/d8b/d8d/db7/d118/f11a 2509770 0 2024-03-26T18:54:38.807 INFO:tasks.workunit.client.1.smithi190.stdout:0/841: read - d0/dd/d20/d1c/d46/da7/df0/d87/fad zero size 2024-03-26T18:54:38.807 INFO:tasks.workunit.client.1.smithi190.stdout:0/842: truncate d0/dd/d20/d1c/d8b/d8d/d8f/f12a 131662 0 2024-03-26T18:54:38.808 INFO:tasks.workunit.client.1.smithi190.stdout:0/843: symlink d0/dd/d20/d1c/d46/da7/df0/d87/d93/dc4/l131 0 2024-03-26T18:54:38.808 INFO:tasks.workunit.client.1.smithi190.stdout:8/873: dwrite d9/dd/f99 [0,4194304] 0 2024-03-26T18:54:38.810 INFO:tasks.workunit.client.1.smithi190.stdout:0/844: creat d0/dd5/f132 x:0 0 0 2024-03-26T18:54:38.810 INFO:tasks.workunit.client.1.smithi190.stdout:7/962: dwrite d3/d6/d9/d14/fc9 [0,4194304] 0 2024-03-26T18:54:38.810 INFO:tasks.workunit.client.1.smithi190.stdout:7/963: write d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/d8f/fda [4712596,91412] 0 2024-03-26T18:54:38.810 INFO:tasks.workunit.client.1.smithi190.stdout:7/964: creat d3/d1b/dd3/d10d/d65/d144/f16d x:0 0 0 2024-03-26T18:54:38.810 INFO:tasks.workunit.client.1.smithi190.stdout:7/965: chown d3/d6/d9/d1f/fa2 2 1 2024-03-26T18:54:38.810 INFO:tasks.workunit.client.1.smithi190.stdout:7/966: fsync d3/d1b/dd3/d10d/d50/f60 0 2024-03-26T18:54:38.810 INFO:tasks.workunit.client.1.smithi190.stdout:5/686: dread d2/d17/d1b/db6/d24/f84 [0,4194304] 0 2024-03-26T18:54:38.810 INFO:tasks.workunit.client.1.smithi190.stdout:5/687: write d2/d17/d1b/db6/d88/f8b [364345,90765] 0 2024-03-26T18:54:38.811 INFO:tasks.workunit.client.1.smithi190.stdout:0/845: symlink d0/dd/d20/d1c/d46/d58/de4/d12e/l133 0 2024-03-26T18:54:38.811 INFO:tasks.workunit.client.1.smithi190.stdout:8/874: symlink d9/dd/l127 0 2024-03-26T18:54:38.812 INFO:tasks.workunit.client.0.smithi102.stdout:2/405: read d3/d2a/f2f [1983252,113273] 0 2024-03-26T18:54:38.812 INFO:tasks.workunit.client.0.smithi102.stdout:2/406: chown d3/d74 3 1 2024-03-26T18:54:38.812 INFO:tasks.workunit.client.0.smithi102.stdout:2/407: truncate d3/d6/fe 5115244 0 2024-03-26T18:54:38.815 INFO:tasks.workunit.client.1.smithi190.stdout:8/875: creat d9/da2/d73/d108/f128 x:0 0 0 2024-03-26T18:54:38.816 INFO:tasks.workunit.client.1.smithi190.stdout:5/688: mknod d2/d17/d1b/db6/d88/cdf 0 2024-03-26T18:54:38.817 INFO:tasks.workunit.client.1.smithi190.stdout:0/846: rmdir d0 39 2024-03-26T18:54:38.818 INFO:tasks.workunit.client.1.smithi190.stdout:5/689: creat d2/d17/d98/d3a/fe0 x:0 0 0 2024-03-26T18:54:38.818 INFO:tasks.workunit.client.0.smithi102.stdout:7/450: write d7/d12/d2b/f47 [3173688,82557] 0 2024-03-26T18:54:38.822 INFO:tasks.workunit.client.0.smithi102.stdout:0/521: dwrite d6/de/d13/d25/d31/d45/d82/d84/d88/fae [0,4194304] 0 2024-03-26T18:54:38.824 INFO:tasks.workunit.client.1.smithi190.stdout:4/878: dwrite d7/d8/d14/d23/d28/d38/d96/d10f/f78 [0,4194304] 0 2024-03-26T18:54:38.824 INFO:tasks.workunit.client.1.smithi190.stdout:4/879: chown d7/d8/d14/d4b/db2/f118 2 1 2024-03-26T18:54:38.824 INFO:tasks.workunit.client.0.smithi102.stdout:6/472: dwrite d2/df/d35/f6e [0,4194304] 0 2024-03-26T18:54:38.825 INFO:tasks.workunit.client.1.smithi190.stdout:3/655: dwrite d1/d3/df/d18/d1b/fb1 [0,4194304] 0 2024-03-26T18:54:38.826 INFO:tasks.workunit.client.0.smithi102.stdout:0/522: truncate d6/de/f15 2320967 0 2024-03-26T18:54:38.826 INFO:tasks.workunit.client.0.smithi102.stdout:5/710: dwrite d2/f17 [0,4194304] 0 2024-03-26T18:54:38.833 INFO:tasks.workunit.client.1.smithi190.stdout:3/656: symlink d1/d3/df/d31/dc4/ld2 0 2024-03-26T18:54:38.833 INFO:tasks.workunit.client.0.smithi102.stdout:4/366: dread d1/d25/f69 [0,4194304] 0 2024-03-26T18:54:38.835 INFO:tasks.workunit.client.1.smithi190.stdout:4/880: dread d7/d1d/d42/d4a/fa6 [0,4194304] 0 2024-03-26T18:54:38.835 INFO:tasks.workunit.client.1.smithi190.stdout:4/881: write d7/d8/d1c/ff1 [312554,22918] 0 2024-03-26T18:54:38.835 INFO:tasks.workunit.client.1.smithi190.stdout:4/882: write d7/d8/faa [1977717,80525] 0 2024-03-26T18:54:38.837 INFO:tasks.workunit.client.1.smithi190.stdout:3/657: mkdir d1/dd3 0 2024-03-26T18:54:38.839 INFO:tasks.workunit.client.0.smithi102.stdout:6/473: dread d2/d7/f20 [0,4194304] 0 2024-03-26T18:54:38.839 INFO:tasks.workunit.client.0.smithi102.stdout:6/474: fdatasync d2/d3d/d53/f5b 0 2024-03-26T18:54:38.839 INFO:tasks.workunit.client.0.smithi102.stdout:6/475: chown d2/d7/d3c/f3e 1380571 1 2024-03-26T18:54:38.840 INFO:tasks.workunit.client.0.smithi102.stdout:0/523: symlink d6/d3e/d87/laf 0 2024-03-26T18:54:38.840 INFO:tasks.workunit.client.0.smithi102.stdout:0/524: getdents d6/de/d13/d25/da4 0 2024-03-26T18:54:38.841 INFO:tasks.workunit.client.1.smithi190.stdout:4/883: mknod d7/c132 0 2024-03-26T18:54:38.842 INFO:tasks.workunit.client.1.smithi190.stdout:3/658: getdents d1/d3/df/d18/d35/d41/d61 0 2024-03-26T18:54:38.842 INFO:tasks.workunit.client.0.smithi102.stdout:3/674: sync 2024-03-26T18:54:38.842 INFO:tasks.workunit.client.0.smithi102.stdout:9/461: sync 2024-03-26T18:54:38.842 INFO:tasks.workunit.client.0.smithi102.stdout:9/462: stat df/d1a/d23/d29/d9c 0 2024-03-26T18:54:38.842 INFO:tasks.workunit.client.0.smithi102.stdout:3/675: write df/d61/fb0 [89603,103790] 0 2024-03-26T18:54:38.846 INFO:tasks.workunit.client.1.smithi190.stdout:3/659: creat d1/de/d9f/fd4 x:0 0 0 2024-03-26T18:54:38.846 INFO:tasks.workunit.client.1.smithi190.stdout:3/660: dread - d1/de/d9f/fd4 zero size 2024-03-26T18:54:38.846 INFO:tasks.workunit.client.1.smithi190.stdout:3/661: chown d1/de/d81/ld0 14390148 1 2024-03-26T18:54:38.846 INFO:tasks.workunit.client.0.smithi102.stdout:2/408: dwrite d3/d2a/f6d [0,4194304] 0 2024-03-26T18:54:38.847 INFO:tasks.workunit.client.0.smithi102.stdout:5/711: symlink d2/d25/d75/db9/d43/d9e/daf/lfa 0 2024-03-26T18:54:38.848 INFO:tasks.workunit.client.0.smithi102.stdout:4/367: mkdir d1/d25/d76 0 2024-03-26T18:54:38.848 INFO:tasks.workunit.client.0.smithi102.stdout:4/368: stat d1/d17/d1a/d21/f2c 0 2024-03-26T18:54:38.848 INFO:tasks.workunit.client.0.smithi102.stdout:4/369: chown d1/d25/d39/d3e/f58 129333 1 2024-03-26T18:54:38.849 INFO:tasks.workunit.client.0.smithi102.stdout:4/370: stat d1/f9 0 2024-03-26T18:54:38.850 INFO:tasks.workunit.client.1.smithi190.stdout:3/662: unlink d1/de/c49 0 2024-03-26T18:54:38.850 INFO:tasks.workunit.client.1.smithi190.stdout:3/663: fdatasync d1/d3/df/d18/d35/d41/dae/f96 0 2024-03-26T18:54:38.851 INFO:tasks.workunit.client.0.smithi102.stdout:0/525: unlink d6/de/d13/d25/d4f/f92 0 2024-03-26T18:54:38.852 INFO:tasks.workunit.client.1.smithi190.stdout:0/847: dread d0/dd/d20/d61/fac [0,4194304] 0 2024-03-26T18:54:38.853 INFO:tasks.workunit.client.0.smithi102.stdout:9/463: mkdir df/d1a/d23/d4e/db0 0 2024-03-26T18:54:38.853 INFO:tasks.workunit.client.0.smithi102.stdout:9/464: truncate df/d1a/d23/f25 455970 0 2024-03-26T18:54:38.854 INFO:tasks.workunit.client.1.smithi190.stdout:3/664: chown d1/de/d4e/l5f 18844553 1 2024-03-26T18:54:38.854 INFO:tasks.workunit.client.1.smithi190.stdout:3/665: write d1/f5 [813052,91112] 0 2024-03-26T18:54:38.854 INFO:tasks.workunit.client.1.smithi190.stdout:3/666: truncate d1/de/d8a/f95 412742 0 2024-03-26T18:54:38.856 INFO:tasks.workunit.client.0.smithi102.stdout:5/712: truncate d2/d14/ff6 2706920 0 2024-03-26T18:54:38.862 INFO:tasks.workunit.client.0.smithi102.stdout:6/476: rename d2/d3/f2a to d2/d7/f92 0 2024-03-26T18:54:38.862 INFO:tasks.workunit.client.0.smithi102.stdout:9/465: read df/d1a/d23/d4e/f7c [3322939,16613] 0 2024-03-26T18:54:38.863 INFO:tasks.workunit.client.0.smithi102.stdout:9/466: chown df/d1a/d23/d29 70935337 1 2024-03-26T18:54:38.863 INFO:tasks.workunit.client.0.smithi102.stdout:3/676: write df/d1c/fbf [3350172,127000] 0 2024-03-26T18:54:38.863 INFO:tasks.workunit.client.0.smithi102.stdout:3/677: chown df/d29/d31/d36/d3c/l42 1 1 2024-03-26T18:54:38.864 INFO:tasks.workunit.client.0.smithi102.stdout:2/409: rename d3/d6/d1a/d25/l4d to d3/d82/l8c 0 2024-03-26T18:54:38.866 INFO:tasks.workunit.client.0.smithi102.stdout:6/477: mknod d2/d3d/d53/d77/c93 0 2024-03-26T18:54:38.867 INFO:tasks.workunit.client.1.smithi190.stdout:6/681: sync 2024-03-26T18:54:38.871 INFO:tasks.workunit.client.0.smithi102.stdout:4/371: rename d1/d17/d1a/d21/d6d to d1/d17/d1a/d21/d77 0 2024-03-26T18:54:38.871 INFO:tasks.workunit.client.0.smithi102.stdout:4/372: stat d1/d27/f55 0 2024-03-26T18:54:38.872 INFO:tasks.workunit.client.0.smithi102.stdout:8/515: sync 2024-03-26T18:54:38.872 INFO:tasks.workunit.client.0.smithi102.stdout:8/516: read - d0/d3c/fa4 zero size 2024-03-26T18:54:38.873 INFO:tasks.workunit.client.1.smithi190.stdout:6/682: mkdir d17/d28/d38/de0 0 2024-03-26T18:54:38.876 INFO:tasks.workunit.client.1.smithi190.stdout:6/683: rmdir d17/d28/d46/dd7 39 2024-03-26T18:54:38.880 INFO:tasks.workunit.client.1.smithi190.stdout:6/684: symlink d17/d28/d2c/db8/dcc/le1 0 2024-03-26T18:54:38.881 INFO:tasks.workunit.client.1.smithi190.stdout:6/685: mknod d17/d28/d2c/d77/ce2 0 2024-03-26T18:54:38.885 INFO:tasks.workunit.client.1.smithi190.stdout:3/667: read d1/d3/f13 [3902597,60190] 0 2024-03-26T18:54:38.885 INFO:tasks.workunit.client.1.smithi190.stdout:3/668: write d1/d3/d52/fa2 [1514654,41713] 0 2024-03-26T18:54:38.894 INFO:tasks.workunit.client.0.smithi102.stdout:2/410: write d3/d20/f30 [1118161,20525] 0 2024-03-26T18:54:38.896 INFO:tasks.workunit.client.0.smithi102.stdout:2/411: unlink d3/d2d/d69/f73 0 2024-03-26T18:54:38.897 INFO:tasks.workunit.client.1.smithi190.stdout:6/686: dread d17/d28/d2c/d50/f8f [0,4194304] 0 2024-03-26T18:54:38.898 INFO:tasks.workunit.client.0.smithi102.stdout:2/412: rename d3/d2d/c81 to d3/d20/d78/d84/c8d 0 2024-03-26T18:54:38.899 INFO:tasks.workunit.client.0.smithi102.stdout:2/413: write d3/d11/d15/d31/f60 [4669293,36782] 0 2024-03-26T18:54:38.900 INFO:tasks.workunit.client.1.smithi190.stdout:6/687: creat d17/d28/d2c/d77/fe3 x:0 0 0 2024-03-26T18:54:38.900 INFO:tasks.workunit.client.0.smithi102.stdout:2/414: mknod d3/d6/d1a/c8e 0 2024-03-26T18:54:38.901 INFO:tasks.workunit.client.1.smithi190.stdout:0/848: read d0/dd/d20/d1c/d8b/d8d/db7/d118/f11a [328889,24220] 0 2024-03-26T18:54:38.901 INFO:tasks.workunit.client.1.smithi190.stdout:0/849: read - d0/dd/d20/d1c/d46/da7/df0/d87/fad zero size 2024-03-26T18:54:38.901 INFO:tasks.workunit.client.0.smithi102.stdout:2/415: mkdir d3/d11/d8f 0 2024-03-26T18:54:38.901 INFO:tasks.workunit.client.0.smithi102.stdout:2/416: read d3/d6/f1d [729558,8208] 0 2024-03-26T18:54:38.902 INFO:tasks.workunit.client.1.smithi190.stdout:7/967: dwrite d3/d6/d9/d1f/f22 [0,4194304] 0 2024-03-26T18:54:38.903 INFO:tasks.workunit.client.1.smithi190.stdout:6/688: unlink d17/d28/d46/c49 0 2024-03-26T18:54:38.903 INFO:tasks.workunit.client.1.smithi190.stdout:6/689: fsync d17/d28/d2c/d77/f8d 0 2024-03-26T18:54:38.903 INFO:tasks.workunit.client.1.smithi190.stdout:3/669: dread d1/de/d1c/f7b [0,4194304] 0 2024-03-26T18:54:38.904 INFO:tasks.workunit.client.1.smithi190.stdout:0/850: unlink d0/dd/f45 0 2024-03-26T18:54:38.905 INFO:tasks.workunit.client.0.smithi102.stdout:2/417: creat d3/d2a/d5f/f90 x:0 0 0 2024-03-26T18:54:38.907 INFO:tasks.workunit.client.1.smithi190.stdout:6/690: link d17/l1f d17/d28/d46/dd7/le4 0 2024-03-26T18:54:38.909 INFO:tasks.workunit.client.1.smithi190.stdout:3/670: truncate d1/d3/f34 1156239 0 2024-03-26T18:54:38.909 INFO:tasks.workunit.client.1.smithi190.stdout:3/671: fdatasync d1/d3/d5a/fa8 0 2024-03-26T18:54:38.911 INFO:tasks.workunit.client.0.smithi102.stdout:4/373: dread d1/d17/d1f/f2d [0,4194304] 0 2024-03-26T18:54:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:4/374: write d1/d17/d1a/d21/d77/f71 [278163,4127] 0 2024-03-26T18:54:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:4/375: truncate d1/d17/d1a/d21/d77/f71 290809 0 2024-03-26T18:54:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:4/376: chown d1/d25/d76 40056727 1 2024-03-26T18:54:38.918 INFO:tasks.workunit.client.0.smithi102.stdout:7/451: dwrite d7/d10/d17/f37 [0,4194304] 0 2024-03-26T18:54:38.934 INFO:tasks.workunit.client.1.smithi190.stdout:8/876: dwrite d9/dd/d15/f60 [0,4194304] 0 2024-03-26T18:54:38.934 INFO:tasks.workunit.client.1.smithi190.stdout:8/877: write d9/dd/d23/d36/d37/d3a/d43/d8f/dd2/f10d [544660,115538] 0 2024-03-26T18:54:38.938 INFO:tasks.workunit.client.1.smithi190.stdout:8/878: mknod d9/dd/d123/d109/c129 0 2024-03-26T18:54:38.938 INFO:tasks.workunit.client.1.smithi190.stdout:8/879: mknod d9/d39/d9e/c12a 0 2024-03-26T18:54:38.938 INFO:tasks.workunit.client.1.smithi190.stdout:8/880: chown d9/dd/d23/d36/d37/d3a/d61/daf/db3/db5/ff0 545 1 2024-03-26T18:54:38.940 INFO:tasks.workunit.client.1.smithi190.stdout:8/881: rename d9/dd/d23/d36 to d9/da2/d73/d108/d12b 0 2024-03-26T18:54:38.940 INFO:tasks.workunit.client.1.smithi190.stdout:8/882: write d9/da2/d73/d108/d12b/d37/d3a/d61/daf/db3/db5/ff0 [526966,122540] 0 2024-03-26T18:54:38.940 INFO:tasks.workunit.client.1.smithi190.stdout:8/883: creat d9/d39/d79/f12c x:0 0 0 2024-03-26T18:54:38.941 INFO:tasks.workunit.client.1.smithi190.stdout:8/884: chown l1 53 1 2024-03-26T18:54:38.947 INFO:tasks.workunit.client.1.smithi190.stdout:4/884: dwrite d7/d8/d14/d4b/db2/f118 [0,4194304] 0 2024-03-26T18:54:38.949 INFO:tasks.workunit.client.1.smithi190.stdout:5/690: dwrite d2/d17/d1b/db6/f7f [0,4194304] 0 2024-03-26T18:54:38.950 INFO:tasks.workunit.client.1.smithi190.stdout:5/691: creat d2/d17/d1b/db6/d24/d3b/da7/fe1 x:0 0 0 2024-03-26T18:54:38.952 INFO:tasks.workunit.client.1.smithi190.stdout:5/692: truncate d2/d17/d1b/db6/d24/d3b/faf 826655 0 2024-03-26T18:54:38.953 INFO:tasks.workunit.client.1.smithi190.stdout:2/824: dwrite d6/d9/d45/d6b/d93/dc0/fe3 [0,4194304] 0 2024-03-26T18:54:38.953 INFO:tasks.workunit.client.1.smithi190.stdout:5/693: symlink d2/d17/d98/d3a/d3c/le2 0 2024-03-26T18:54:38.956 INFO:tasks.workunit.client.1.smithi190.stdout:5/694: mknod d2/d17/d62/ce3 0 2024-03-26T18:54:38.957 INFO:tasks.workunit.client.0.smithi102.stdout:7/452: write d7/f9 [2201588,40741] 0 2024-03-26T18:54:38.958 INFO:tasks.workunit.client.1.smithi190.stdout:2/825: truncate d6/d9/d3a/d3d/d1c/f54 85043 0 2024-03-26T18:54:38.958 INFO:tasks.workunit.client.1.smithi190.stdout:2/826: truncate d6/d9/d48/f68 581939 0 2024-03-26T18:54:38.959 INFO:tasks.workunit.client.1.smithi190.stdout:7/968: dwrite d3/d6/f91 [0,4194304] 0 2024-03-26T18:54:38.959 INFO:tasks.workunit.client.1.smithi190.stdout:7/969: write d3/d1b/dd3/d10d/d65/d109/d119/f14c [380738,120885] 0 2024-03-26T18:54:38.959 INFO:tasks.workunit.client.0.smithi102.stdout:0/526: dwrite d6/de/d13/d25/d4f/d66/f90 [0,4194304] 0 2024-03-26T18:54:38.960 INFO:tasks.workunit.client.0.smithi102.stdout:2/418: dwrite d3/d6/f1b [0,4194304] 0 2024-03-26T18:54:38.960 INFO:tasks.workunit.client.0.smithi102.stdout:2/419: chown d3/d2a/f2f 39 1 2024-03-26T18:54:38.960 INFO:tasks.workunit.client.0.smithi102.stdout:7/453: mkdir d7/d12/d4b/d83 0 2024-03-26T18:54:38.960 INFO:tasks.workunit.client.0.smithi102.stdout:7/454: write d7/f20 [1251217,101003] 0 2024-03-26T18:54:38.961 INFO:tasks.workunit.client.0.smithi102.stdout:7/455: readlink d7/ld 0 2024-03-26T18:54:38.961 INFO:tasks.workunit.client.1.smithi190.stdout:2/827: rename d6/d9/d45/d6b/c76 to d6/d9/d3a/d3d/d2e/d10e/d9e/dfc/c120 0 2024-03-26T18:54:38.961 INFO:tasks.workunit.client.1.smithi190.stdout:2/828: write d6/d21/d61/d6c/db4/fed [636707,66127] 0 2024-03-26T18:54:38.961 INFO:tasks.workunit.client.1.smithi190.stdout:2/829: truncate d6/d9/d10/d5f/fad 2414315 0 2024-03-26T18:54:38.961 INFO:tasks.workunit.client.0.smithi102.stdout:9/467: dwrite df/f27 [0,4194304] 0 2024-03-26T18:54:38.962 INFO:tasks.workunit.client.1.smithi190.stdout:8/885: dwrite d9/dd/d23/d88/fe4 [0,4194304] 0 2024-03-26T18:54:38.962 INFO:tasks.workunit.client.0.smithi102.stdout:0/527: getdents d6/de 0 2024-03-26T18:54:38.964 INFO:tasks.workunit.client.1.smithi190.stdout:3/672: dread d1/de/d1c/f83 [0,4194304] 0 2024-03-26T18:54:38.964 INFO:tasks.workunit.client.1.smithi190.stdout:3/673: getdents d1/d3/df/d18/d35/d41/d61 0 2024-03-26T18:54:38.964 INFO:tasks.workunit.client.0.smithi102.stdout:6/478: dwrite d2/d7/d3c/f48 [0,4194304] 0 2024-03-26T18:54:38.964 INFO:tasks.workunit.client.0.smithi102.stdout:6/479: dread - d2/df/f8e zero size 2024-03-26T18:54:38.964 INFO:tasks.workunit.client.0.smithi102.stdout:6/480: chown d2/df/l15 525 1 2024-03-26T18:54:38.965 INFO:tasks.workunit.client.1.smithi190.stdout:0/851: dwrite d0/dd/d20/d1c/d46/da7/df0/f111 [0,4194304] 0 2024-03-26T18:54:38.965 INFO:tasks.workunit.client.1.smithi190.stdout:0/852: chown d0/dd/d20/d1c/d46/d58/de4/d12e 46529 1 2024-03-26T18:54:38.965 INFO:tasks.workunit.client.1.smithi190.stdout:2/830: creat d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d8b/f121 x:0 0 0 2024-03-26T18:54:38.966 INFO:tasks.workunit.client.0.smithi102.stdout:3/678: dwrite df/d29/d31/d36/d3c/d6f/d75/f88 [0,4194304] 0 2024-03-26T18:54:38.966 INFO:tasks.workunit.client.0.smithi102.stdout:2/420: stat d3/d11/l3f 0 2024-03-26T18:54:38.966 INFO:tasks.workunit.client.0.smithi102.stdout:5/713: dwrite d2/d25/d5b/d76/fac [0,4194304] 0 2024-03-26T18:54:38.967 INFO:tasks.workunit.client.0.smithi102.stdout:8/517: dwrite d0/d3/d2c/d58/d75/f86 [0,4194304] 0 2024-03-26T18:54:38.968 INFO:tasks.workunit.client.1.smithi190.stdout:8/886: rename d9/d48/d49/ff1 to d9/dd/d23/dea/dfe/f12d 0 2024-03-26T18:54:38.968 INFO:tasks.workunit.client.1.smithi190.stdout:3/674: getdents d1/d3/df/d31 0 2024-03-26T18:54:38.977 INFO:tasks.workunit.client.1.smithi190.stdout:2/831: creat d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d7c/f122 x:0 0 0 2024-03-26T18:54:38.977 INFO:tasks.workunit.client.1.smithi190.stdout:2/832: readlink d6/d9/d10/d5f/l103 0 2024-03-26T18:54:38.979 INFO:tasks.workunit.client.0.smithi102.stdout:9/468: dread df/d1a/f3f [0,4194304] 0 2024-03-26T18:54:38.981 INFO:tasks.workunit.client.1.smithi190.stdout:3/675: symlink d1/d3/df/d18/d35/d41/ld5 0 2024-03-26T18:54:38.981 INFO:tasks.workunit.client.1.smithi190.stdout:3/676: readlink d1/d3/d5a/d6e/l82 0 2024-03-26T18:54:38.982 INFO:tasks.workunit.client.0.smithi102.stdout:6/481: link d2/df/d72/f89 d2/df/d35/d43/f94 0 2024-03-26T18:54:38.982 INFO:tasks.workunit.client.0.smithi102.stdout:6/482: fdatasync d2/f34 0 2024-03-26T18:54:38.983 INFO:tasks.workunit.client.1.smithi190.stdout:4/885: dwrite d7/d8/d14/d5c/f83 [0,4194304] 0 2024-03-26T18:54:38.985 INFO:tasks.workunit.client.1.smithi190.stdout:0/853: getdents d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b 0 2024-03-26T18:54:38.991 INFO:tasks.workunit.client.1.smithi190.stdout:6/691: getdents d17/d28/d38 0 2024-03-26T18:54:38.992 INFO:tasks.workunit.client.1.smithi190.stdout:3/677: mknod d1/dd3/cd6 0 2024-03-26T18:54:38.992 INFO:tasks.workunit.client.1.smithi190.stdout:3/678: write d1/de/d1c/d36/f40 [555463,24509] 0 2024-03-26T18:54:38.992 INFO:tasks.workunit.client.0.smithi102.stdout:3/679: rmdir df/d1c/d34/d53/dd4 0 2024-03-26T18:54:38.994 INFO:tasks.workunit.client.1.smithi190.stdout:0/854: dread d0/dd/d20/d1c/f29 [0,4194304] 0 2024-03-26T18:54:38.995 INFO:tasks.workunit.client.0.smithi102.stdout:2/421: rename d3/d6/fe to d3/d2d/d69/f91 0 2024-03-26T18:54:38.995 INFO:tasks.workunit.client.0.smithi102.stdout:2/422: creat d3/d11/d15/d46/f92 x:0 0 0 2024-03-26T18:54:38.995 INFO:tasks.workunit.client.0.smithi102.stdout:2/423: fdatasync d3/d20/d78/f7a 0 2024-03-26T18:54:38.995 INFO:tasks.workunit.client.0.smithi102.stdout:2/424: stat d3/d11/f55 0 2024-03-26T18:54:38.996 INFO:tasks.workunit.client.1.smithi190.stdout:4/886: rmdir d7/d8/d14/d23/d28/d38 39 2024-03-26T18:54:38.996 INFO:tasks.workunit.client.1.smithi190.stdout:4/887: dread - d7/d8/d14/d6e/df0/ffc zero size 2024-03-26T18:54:38.996 INFO:tasks.workunit.client.1.smithi190.stdout:4/888: chown d7/d8/d1c/f7a 112205 1 2024-03-26T18:54:38.996 INFO:tasks.workunit.client.1.smithi190.stdout:2/833: dread d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d8b/f10f [0,4194304] 0 2024-03-26T18:54:38.998 INFO:tasks.workunit.client.1.smithi190.stdout:4/889: mknod d7/c133 0 2024-03-26T18:54:38.999 INFO:tasks.workunit.client.1.smithi190.stdout:4/890: chown d7/d1d/d41/d74/l90 49892 1 2024-03-26T18:54:39.000 INFO:tasks.workunit.client.1.smithi190.stdout:6/692: symlink d17/d28/d2c/le5 0 2024-03-26T18:54:39.004 INFO:tasks.workunit.client.1.smithi190.stdout:5/695: dwrite d2/d17/d1b/db6/d96/fa4 [0,4194304] 0 2024-03-26T18:54:39.004 INFO:tasks.workunit.client.1.smithi190.stdout:5/696: fsync d2/d17/d98/d3a/f52 0 2024-03-26T18:54:39.009 INFO:tasks.workunit.client.0.smithi102.stdout:5/714: creat d2/d25/d75/ffb x:0 0 0 2024-03-26T18:54:39.014 INFO:tasks.workunit.client.1.smithi190.stdout:6/693: read d17/d28/d38/fb1 [397654,90459] 0 2024-03-26T18:54:39.016 INFO:tasks.workunit.client.0.smithi102.stdout:7/456: dwrite d7/d10/d17/f5d [0,4194304] 0 2024-03-26T18:54:39.016 INFO:tasks.workunit.client.0.smithi102.stdout:7/457: truncate d7/d10/d27/d34/f49 5948081 0 2024-03-26T18:54:39.019 INFO:tasks.workunit.client.1.smithi190.stdout:7/970: dwrite d3/d1b/d57/f12f [0,4194304] 0 2024-03-26T18:54:39.019 INFO:tasks.workunit.client.0.smithi102.stdout:8/518: creat d0/d3/d26/d19/d9f/fa9 x:0 0 0 2024-03-26T18:54:39.020 INFO:tasks.workunit.client.1.smithi190.stdout:5/697: dread d2/d17/d98/d3a/d3c/f94 [0,4194304] 0 2024-03-26T18:54:39.022 INFO:tasks.workunit.client.0.smithi102.stdout:6/483: link d2/d3d/d53/d77/f88 d2/d3d/d53/d63/d6a/f95 0 2024-03-26T18:54:39.022 INFO:tasks.workunit.client.0.smithi102.stdout:6/484: creat d2/df/f96 x:0 0 0 2024-03-26T18:54:39.022 INFO:tasks.workunit.client.0.smithi102.stdout:6/485: write d2/d7/f54 [375597,78314] 0 2024-03-26T18:54:39.022 INFO:tasks.workunit.client.0.smithi102.stdout:6/486: stat d2/d3d/d53/d77 0 2024-03-26T18:54:39.026 INFO:tasks.workunit.client.0.smithi102.stdout:3/680: rename df/d29/d31/d36/d3c/d6f/c8d to df/d29/d31/d77/d8a/dae/cf3 0 2024-03-26T18:54:39.027 INFO:tasks.workunit.client.0.smithi102.stdout:7/458: dread d7/d10/d17/f1b [0,4194304] 0 2024-03-26T18:54:39.028 INFO:tasks.workunit.client.1.smithi190.stdout:5/698: unlink d2/d17/d62/f82 0 2024-03-26T18:54:39.028 INFO:tasks.workunit.client.1.smithi190.stdout:5/699: chown d2/d17/d1b/db6/d24/d3b/c59 41037 1 2024-03-26T18:54:39.029 INFO:tasks.workunit.client.1.smithi190.stdout:5/700: write d2/d17/d1b/f72 [1558084,78690] 0 2024-03-26T18:54:39.029 INFO:tasks.workunit.client.0.smithi102.stdout:2/425: mkdir d3/d82/d93 0 2024-03-26T18:54:39.030 INFO:tasks.workunit.client.0.smithi102.stdout:5/715: mkdir d2/dfc 0 2024-03-26T18:54:39.035 INFO:tasks.workunit.client.1.smithi190.stdout:7/971: write d3/d6/d9/d14/f18 [2249685,514] 0 2024-03-26T18:54:39.036 INFO:tasks.workunit.client.0.smithi102.stdout:7/459: unlink d7/d12/d2b/d50/f6e 0 2024-03-26T18:54:39.036 INFO:tasks.workunit.client.0.smithi102.stdout:9/469: dwrite df/d1a/f3f [0,4194304] 0 2024-03-26T18:54:39.036 INFO:tasks.workunit.client.0.smithi102.stdout:9/470: readlink df/d1a/d1d/l4b 0 2024-03-26T18:54:39.036 INFO:tasks.workunit.client.0.smithi102.stdout:9/471: chown df/d1a/c1e 383481 1 2024-03-26T18:54:39.037 INFO:tasks.workunit.client.0.smithi102.stdout:2/426: getdents d3/d11/d27 0 2024-03-26T18:54:39.037 INFO:tasks.workunit.client.0.smithi102.stdout:2/427: chown d3/d11/d15/f2e 63724 1 2024-03-26T18:54:39.038 INFO:tasks.workunit.client.1.smithi190.stdout:7/972: rename d3/d6/d9/d1f/d93/dfb/f167 to d3/d6/f16e 0 2024-03-26T18:54:39.038 INFO:tasks.workunit.client.1.smithi190.stdout:7/973: fsync d3/d1b/d57/f122 0 2024-03-26T18:54:39.038 INFO:tasks.workunit.client.1.smithi190.stdout:7/974: truncate d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/d8f/fb1 161387 0 2024-03-26T18:54:39.039 INFO:tasks.workunit.client.0.smithi102.stdout:3/681: rename df/f25 to df/d29/d3a/ff4 0 2024-03-26T18:54:39.041 INFO:tasks.workunit.client.0.smithi102.stdout:7/460: unlink d7/d10/f1c 0 2024-03-26T18:54:39.043 INFO:tasks.workunit.client.1.smithi190.stdout:5/701: write d2/f4 [2381112,67081] 0 2024-03-26T18:54:39.043 INFO:tasks.workunit.client.1.smithi190.stdout:5/702: truncate d2/d17/d1b/db6/d24/d3b/da5/fcc 631120 0 2024-03-26T18:54:39.043 INFO:tasks.workunit.client.1.smithi190.stdout:8/887: dwrite d9/d39/d79/f7e [0,4194304] 0 2024-03-26T18:54:39.043 INFO:tasks.workunit.client.1.smithi190.stdout:8/888: write d9/da2/d73/d108/d12b/d37/d3a/d61/daf/db3/db5/ff0 [1392190,46719] 0 2024-03-26T18:54:39.044 INFO:tasks.workunit.client.1.smithi190.stdout:1/700: sync 2024-03-26T18:54:39.044 INFO:tasks.workunit.client.1.smithi190.stdout:7/975: dread d3/d6/d9/d14/d36/f72 [0,4194304] 0 2024-03-26T18:54:39.044 INFO:tasks.workunit.client.0.smithi102.stdout:9/472: rename df/d84/fa9 to df/d1a/d57/fb1 0 2024-03-26T18:54:39.044 INFO:tasks.workunit.client.0.smithi102.stdout:9/473: creat df/d1a/d23/fb2 x:0 0 0 2024-03-26T18:54:39.046 INFO:tasks.workunit.client.1.smithi190.stdout:8/889: link d9/da2/d73/d108/d12b/d37/d3a/lab d9/da2/l12e 0 2024-03-26T18:54:39.046 INFO:tasks.workunit.client.1.smithi190.stdout:7/976: getdents d3/d1b/dd3/d10d/d65/d90/d92 0 2024-03-26T18:54:39.047 INFO:tasks.workunit.client.1.smithi190.stdout:5/703: mkdir d2/d17/d1b/db6/d88/dbc/de4 0 2024-03-26T18:54:39.048 INFO:tasks.workunit.client.1.smithi190.stdout:8/890: creat d9/da2/f12f x:0 0 0 2024-03-26T18:54:39.049 INFO:tasks.workunit.client.0.smithi102.stdout:7/461: mkdir d7/d10/d84 0 2024-03-26T18:54:39.049 INFO:tasks.workunit.client.0.smithi102.stdout:2/428: write d3/d6/f1b [3231042,88240] 0 2024-03-26T18:54:39.049 INFO:tasks.workunit.client.0.smithi102.stdout:0/528: dwrite d6/de/f4b [0,4194304] 0 2024-03-26T18:54:39.050 INFO:tasks.workunit.client.1.smithi190.stdout:1/701: rmdir d3/d9/da3 39 2024-03-26T18:54:39.050 INFO:tasks.workunit.client.1.smithi190.stdout:1/702: readlink d3/d8f/lb1 0 2024-03-26T18:54:39.051 INFO:tasks.workunit.client.1.smithi190.stdout:8/891: creat d9/da2/d73/d108/d12b/d37/db8/f130 x:0 0 0 2024-03-26T18:54:39.054 INFO:tasks.workunit.client.1.smithi190.stdout:8/892: creat d9/dd/d23/f131 x:0 0 0 2024-03-26T18:54:39.054 INFO:tasks.workunit.client.0.smithi102.stdout:9/474: rename df/d1a/c97 to df/d1a/d1d/d72/d7d/cb3 0 2024-03-26T18:54:39.054 INFO:tasks.workunit.client.0.smithi102.stdout:0/529: rename d6/de/d13/d25/d31 to d6/de/d13/d25/d31/d45/d82/db0 22 2024-03-26T18:54:39.054 INFO:tasks.workunit.client.0.smithi102.stdout:0/530: chown d6/de/d13/ca1 5950 1 2024-03-26T18:54:39.055 INFO:tasks.workunit.client.0.smithi102.stdout:7/462: symlink d7/d12/d4b/l85 0 2024-03-26T18:54:39.056 INFO:tasks.workunit.client.0.smithi102.stdout:5/716: dread d2/d25/d75/db9/d43/f4e [0,4194304] 0 2024-03-26T18:54:39.057 INFO:tasks.workunit.client.1.smithi190.stdout:8/893: link d9/d39/d67/c89 d9/da2/d73/c132 0 2024-03-26T18:54:39.059 INFO:tasks.workunit.client.0.smithi102.stdout:9/475: rename df/d1a/d23/d3e/f6b to df/d46/fb4 0 2024-03-26T18:54:39.059 INFO:tasks.workunit.client.0.smithi102.stdout:9/476: fdatasync df/d1a/d23/d29/f34 0 2024-03-26T18:54:39.060 INFO:tasks.workunit.client.1.smithi190.stdout:0/855: dwrite d0/dd/d20/d61/f9c [0,4194304] 0 2024-03-26T18:54:39.062 INFO:tasks.workunit.client.0.smithi102.stdout:0/531: link d6/d3e/l89 d6/d2d/d3d/lb1 0 2024-03-26T18:54:39.063 INFO:tasks.workunit.client.0.smithi102.stdout:7/463: mknod d7/d44/c86 0 2024-03-26T18:54:39.066 INFO:tasks.workunit.client.1.smithi190.stdout:6/694: dwrite d17/d28/d2c/d50/f9c [0,4194304] 0 2024-03-26T18:54:39.067 INFO:tasks.workunit.client.1.smithi190.stdout:3/679: dwrite d1/de/d4e/f8f [4194304,4194304] 0 2024-03-26T18:54:39.067 INFO:tasks.workunit.client.0.smithi102.stdout:9/477: link df/d46/f67 df/d1a/d23/fb5 0 2024-03-26T18:54:39.068 INFO:tasks.workunit.client.0.smithi102.stdout:9/478: truncate df/d1a/d65/f8a 309232 0 2024-03-26T18:54:39.068 INFO:tasks.workunit.client.1.smithi190.stdout:2/834: dwrite d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d7c/f122 [0,4194304] 0 2024-03-26T18:54:39.069 INFO:tasks.workunit.client.0.smithi102.stdout:8/519: dwrite d0/d3/d26/f22 [8388608,4194304] 0 2024-03-26T18:54:39.070 INFO:tasks.workunit.client.0.smithi102.stdout:0/532: mkdir d6/de/d13/d25/d31/d45/d82/d84/d88/db2 0 2024-03-26T18:54:39.073 INFO:tasks.workunit.client.1.smithi190.stdout:0/856: rename d0/dd/d20/d1c/d46/da7/df0/d4b/lb0 to d0/dd5/dee/dfc/l134 0 2024-03-26T18:54:39.073 INFO:tasks.workunit.client.1.smithi190.stdout:0/857: creat d0/dd/d20/d1c/d46/da7/df0/d87/d93/db5/f135 x:0 0 0 2024-03-26T18:54:39.073 INFO:tasks.workunit.client.0.smithi102.stdout:3/682: getdents df/d1c/d34/d53 0 2024-03-26T18:54:39.074 INFO:tasks.workunit.client.1.smithi190.stdout:3/680: dread d1/d3/df/d18/d35/d41/f72 [0,4194304] 0 2024-03-26T18:54:39.074 INFO:tasks.workunit.client.0.smithi102.stdout:2/429: write d3/d11/f55 [1516923,9272] 0 2024-03-26T18:54:39.075 INFO:tasks.workunit.client.0.smithi102.stdout:9/479: symlink df/d1a/d23/d29/d9c/lb6 0 2024-03-26T18:54:39.080 INFO:tasks.workunit.client.1.smithi190.stdout:2/835: rmdir d6/d9/d45/df3/dfa 0 2024-03-26T18:54:39.081 INFO:tasks.workunit.client.0.smithi102.stdout:8/520: symlink d0/d3c/d96/laa 0 2024-03-26T18:54:39.082 INFO:tasks.workunit.client.0.smithi102.stdout:7/464: dread d7/d10/d17/f37 [0,4194304] 0 2024-03-26T18:54:39.082 INFO:tasks.workunit.client.0.smithi102.stdout:7/465: read d7/f9 [1461318,11113] 0 2024-03-26T18:54:39.083 INFO:tasks.workunit.client.0.smithi102.stdout:1/434: sync 2024-03-26T18:54:39.084 INFO:tasks.workunit.client.1.smithi190.stdout:8/894: dread d9/da2/d73/d108/d12b/d37/d3a/d43/d8f/dd2/f10d [0,4194304] 0 2024-03-26T18:54:39.084 INFO:tasks.workunit.client.1.smithi190.stdout:8/895: chown d9/d39/lce 29414 1 2024-03-26T18:54:39.084 INFO:tasks.workunit.client.0.smithi102.stdout:0/533: rename d6/l9e to d6/de/d13/d25/da4/lb3 0 2024-03-26T18:54:39.084 INFO:tasks.workunit.client.0.smithi102.stdout:0/534: creat d6/d2d/d3d/fb4 x:0 0 0 2024-03-26T18:54:39.085 INFO:tasks.workunit.client.0.smithi102.stdout:0/535: write d6/fd [2314596,106877] 0 2024-03-26T18:54:39.085 INFO:tasks.workunit.client.0.smithi102.stdout:6/487: dwrite d2/df/d72/f89 [0,4194304] 0 2024-03-26T18:54:39.085 INFO:tasks.workunit.client.0.smithi102.stdout:6/488: creat d2/d3/f97 x:0 0 0 2024-03-26T18:54:39.087 INFO:tasks.workunit.client.0.smithi102.stdout:3/683: truncate df/d61/d55/f7c 2113334 0 2024-03-26T18:54:39.087 INFO:tasks.workunit.client.0.smithi102.stdout:3/684: dread - df/d29/d31/dc0/fe8 zero size 2024-03-26T18:54:39.088 INFO:tasks.workunit.client.1.smithi190.stdout:7/977: dwrite d3/d1b/dd3/d10d/d65/d90/d92/d12d/fc5 [0,4194304] 0 2024-03-26T18:54:39.088 INFO:tasks.workunit.client.1.smithi190.stdout:7/978: readlink d3/d6/d9/l143 0 2024-03-26T18:54:39.092 INFO:tasks.workunit.client.1.smithi190.stdout:3/681: write d1/d3/df/d18/d35/d41/dae/f96 [3980962,8266] 0 2024-03-26T18:54:39.094 INFO:tasks.workunit.client.1.smithi190.stdout:5/704: dread d2/d17/d1b/f72 [0,4194304] 0 2024-03-26T18:54:39.094 INFO:tasks.workunit.client.1.smithi190.stdout:5/705: chown d2/d17/d1b/db6/d24/d3b/c78 41808990 1 2024-03-26T18:54:39.094 INFO:tasks.workunit.client.1.smithi190.stdout:7/979: truncate d3/d1b/f3b 129206 0 2024-03-26T18:54:39.095 INFO:tasks.workunit.client.1.smithi190.stdout:0/858: truncate d0/dd/d20/d1c/d46/da7/df0/f7d 3115019 0 2024-03-26T18:54:39.095 INFO:tasks.workunit.client.1.smithi190.stdout:5/706: read d2/d17/d98/d3a/f55 [826318,22476] 0 2024-03-26T18:54:39.095 INFO:tasks.workunit.client.0.smithi102.stdout:8/521: creat d0/d3c/d6f/fab x:0 0 0 2024-03-26T18:54:39.096 INFO:tasks.workunit.client.1.smithi190.stdout:7/980: write d3/d6/d9/d14/d36/f70 [100453,22177] 0 2024-03-26T18:54:39.098 INFO:tasks.workunit.client.0.smithi102.stdout:0/536: dread d6/f51 [0,4194304] 0 2024-03-26T18:54:39.098 INFO:tasks.workunit.client.0.smithi102.stdout:0/537: truncate d6/d2d/d3d/fa2 339851 0 2024-03-26T18:54:39.098 INFO:tasks.workunit.client.0.smithi102.stdout:0/538: creat d6/de/d13/d25/d31/fb5 x:0 0 0 2024-03-26T18:54:39.099 INFO:tasks.workunit.client.0.smithi102.stdout:0/539: chown d6/de/d13/d9a/l86 1 1 2024-03-26T18:54:39.099 INFO:tasks.workunit.client.0.smithi102.stdout:7/466: link d7/d2a/f4a d7/d10/d17/f87 0 2024-03-26T18:54:39.099 INFO:tasks.workunit.client.0.smithi102.stdout:7/467: chown d7/d31/l61 37546035 1 2024-03-26T18:54:39.099 INFO:tasks.workunit.client.0.smithi102.stdout:7/468: chown d7/d2a/f30 74948332 1 2024-03-26T18:54:39.100 INFO:tasks.workunit.client.1.smithi190.stdout:1/703: dwrite d3/da/dd/d2f/f5e [0,4194304] 0 2024-03-26T18:54:39.100 INFO:tasks.workunit.client.1.smithi190.stdout:1/704: chown d3/d9/da3/ld5 18052 1 2024-03-26T18:54:39.103 INFO:tasks.workunit.client.1.smithi190.stdout:8/896: write d9/d39/f6d [3281692,9195] 0 2024-03-26T18:54:39.104 INFO:tasks.workunit.client.0.smithi102.stdout:5/717: dwrite d2/f1d [0,4194304] 0 2024-03-26T18:54:39.104 INFO:tasks.workunit.client.0.smithi102.stdout:5/718: creat d2/d25/db0/ffd x:0 0 0 2024-03-26T18:54:39.110 INFO:tasks.workunit.client.1.smithi190.stdout:6/695: dwrite f8 [0,4194304] 0 2024-03-26T18:54:39.110 INFO:tasks.workunit.client.0.smithi102.stdout:9/480: dwrite df/d1a/d1d/f62 [0,4194304] 0 2024-03-26T18:54:39.110 INFO:tasks.workunit.client.0.smithi102.stdout:9/481: fdatasync df/d1a/d23/d29/f34 0 2024-03-26T18:54:39.113 INFO:tasks.workunit.client.1.smithi190.stdout:7/981: dread d3/d6/d9/d1f/d4c/f54 [0,4194304] 0 2024-03-26T18:54:39.113 INFO:tasks.workunit.client.1.smithi190.stdout:2/836: dwrite d6/d9/d48/f68 [0,4194304] 0 2024-03-26T18:54:39.114 INFO:tasks.workunit.client.1.smithi190.stdout:3/682: stat d1/d3/d5a/d6e/l77 0 2024-03-26T18:54:39.115 INFO:tasks.workunit.client.1.smithi190.stdout:7/982: getdents d3/d6/d9/d1f/d93/dfb/d102 0 2024-03-26T18:54:39.116 INFO:tasks.workunit.client.0.smithi102.stdout:5/719: write d2/d25/d5b/d72/f57 [1851091,84531] 0 2024-03-26T18:54:39.116 INFO:tasks.workunit.client.0.smithi102.stdout:5/720: write d2/d25/db0/ff8 [768660,119183] 0 2024-03-26T18:54:39.118 INFO:tasks.workunit.client.1.smithi190.stdout:5/707: creat d2/d17/fe5 x:0 0 0 2024-03-26T18:54:39.126 INFO:tasks.workunit.client.0.smithi102.stdout:6/489: rename d2/df/d35/c58 to d2/df/d72/c98 0 2024-03-26T18:54:39.127 INFO:tasks.workunit.client.1.smithi190.stdout:1/705: symlink d3/da/d2e/d59/dae/le1 0 2024-03-26T18:54:39.128 INFO:tasks.workunit.client.1.smithi190.stdout:0/859: dwrite d0/dd/d20/d1c/d46/da7/df0/da2/d80/f124 [0,4194304] 0 2024-03-26T18:54:39.128 INFO:tasks.workunit.client.0.smithi102.stdout:3/685: mknod df/d29/d31/d77/d8a/dae/dee/df2/cf5 0 2024-03-26T18:54:39.129 INFO:tasks.workunit.client.0.smithi102.stdout:3/686: write df/d1c/d34/f8e [1719062,72179] 0 2024-03-26T18:54:39.131 INFO:tasks.workunit.client.0.smithi102.stdout:4/377: sync 2024-03-26T18:54:39.131 INFO:tasks.workunit.client.0.smithi102.stdout:4/378: fsync d1/d17/d1a/d21/f56 0 2024-03-26T18:54:39.132 INFO:tasks.workunit.client.1.smithi190.stdout:6/696: mknod d17/d28/d2c/db8/ce6 0 2024-03-26T18:54:39.132 INFO:tasks.workunit.client.1.smithi190.stdout:6/697: readlink d17/d28/d38/d79/l96 0 2024-03-26T18:54:39.132 INFO:tasks.workunit.client.1.smithi190.stdout:6/698: chown d17/d28/d2c/d50/lb6 8040 1 2024-03-26T18:54:39.132 INFO:tasks.workunit.client.1.smithi190.stdout:6/699: write d17/d28/d2c/d77/fbb [382783,21490] 0 2024-03-26T18:54:39.132 INFO:tasks.workunit.client.1.smithi190.stdout:6/700: truncate d17/d28/d46/fba 1017501 0 2024-03-26T18:54:39.138 INFO:tasks.workunit.client.0.smithi102.stdout:8/522: mkdir d0/d15/d5d/dac 0 2024-03-26T18:54:39.138 INFO:tasks.workunit.client.1.smithi190.stdout:3/683: mkdir d1/de/d81/dd7 0 2024-03-26T18:54:39.139 INFO:tasks.workunit.client.1.smithi190.stdout:3/684: fdatasync d1/d3/d5a/fa9 0 2024-03-26T18:54:39.139 INFO:tasks.workunit.client.1.smithi190.stdout:3/685: readlink d1/l1e 0 2024-03-26T18:54:39.139 INFO:tasks.workunit.client.1.smithi190.stdout:1/706: creat d3/d8f/fe2 x:0 0 0 2024-03-26T18:54:39.143 INFO:tasks.workunit.client.1.smithi190.stdout:3/686: dread d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:39.143 INFO:tasks.workunit.client.1.smithi190.stdout:3/687: write d1/d3/df/d18/d35/d41/dae/d9c/fcf [701007,72171] 0 2024-03-26T18:54:39.144 INFO:tasks.workunit.client.0.smithi102.stdout:7/469: dwrite d7/d12/f3c [0,4194304] 0 2024-03-26T18:54:39.147 INFO:tasks.workunit.client.1.smithi190.stdout:8/897: dwrite d9/f1a [0,4194304] 0 2024-03-26T18:54:39.147 INFO:tasks.workunit.client.1.smithi190.stdout:3/688: write d1/f15 [922020,98156] 0 2024-03-26T18:54:39.147 INFO:tasks.workunit.client.0.smithi102.stdout:7/470: dread d7/d12/f38 [0,4194304] 0 2024-03-26T18:54:39.147 INFO:tasks.workunit.client.0.smithi102.stdout:8/523: dread d0/d3/d2c/d33/d87/f8c [0,4194304] 0 2024-03-26T18:54:39.149 INFO:tasks.workunit.client.1.smithi190.stdout:0/860: rename d0/dd/d20/d1c/d46/da7/df0/d4b/fd3 to d0/dd/d20/d1c/d8b/d116/f136 0 2024-03-26T18:54:39.149 INFO:tasks.workunit.client.1.smithi190.stdout:0/861: write d0/dd/f22 [4891921,108834] 0 2024-03-26T18:54:39.158 INFO:tasks.workunit.client.0.smithi102.stdout:1/435: rmdir de/d16/d1d/d2a/d2c 39 2024-03-26T18:54:39.158 INFO:tasks.workunit.client.0.smithi102.stdout:1/436: chown de/d16/d1d/d2a/d2c/f60 1049657703 1 2024-03-26T18:54:39.158 INFO:tasks.workunit.client.1.smithi190.stdout:1/707: creat d3/da/d2e/d59/dae/fe3 x:0 0 0 2024-03-26T18:54:39.159 INFO:tasks.workunit.client.1.smithi190.stdout:2/837: dwrite d6/d9/d45/f74 [0,4194304] 0 2024-03-26T18:54:39.160 INFO:tasks.workunit.client.0.smithi102.stdout:6/490: dwrite d2/df/d35/d43/f61 [0,4194304] 0 2024-03-26T18:54:39.160 INFO:tasks.workunit.client.0.smithi102.stdout:1/437: write de/f3e [653651,60190] 0 2024-03-26T18:54:39.161 INFO:tasks.workunit.client.0.smithi102.stdout:2/430: truncate d3/d11/f55 3725263 0 2024-03-26T18:54:39.162 INFO:tasks.workunit.client.1.smithi190.stdout:7/983: dwrite d3/f7c [0,4194304] 0 2024-03-26T18:54:39.162 INFO:tasks.workunit.client.1.smithi190.stdout:5/708: dwrite d2/d17/d1b/fb7 [0,4194304] 0 2024-03-26T18:54:39.162 INFO:tasks.workunit.client.1.smithi190.stdout:5/709: truncate d2/d17/d1b/db6/d24/d3b/da7/fc0 1026642 0 2024-03-26T18:54:39.163 INFO:tasks.workunit.client.1.smithi190.stdout:5/710: chown d2/l2b 12 1 2024-03-26T18:54:39.166 INFO:tasks.workunit.client.1.smithi190.stdout:2/838: symlink d6/d9/d3a/d3d/d1c/d8d/dc7/l123 0 2024-03-26T18:54:39.166 INFO:tasks.workunit.client.1.smithi190.stdout:3/689: mkdir d1/dd8 0 2024-03-26T18:54:39.166 INFO:tasks.workunit.client.1.smithi190.stdout:3/690: stat d1/d3/df/d18/d35/cbe 0 2024-03-26T18:54:39.166 INFO:tasks.workunit.client.1.smithi190.stdout:7/984: mknod d3/d1b/dd3/d10d/d65/d90/c16f 0 2024-03-26T18:54:39.168 INFO:tasks.workunit.client.0.smithi102.stdout:9/482: mkdir df/d1a/d1d/d72/d7d/db7 0 2024-03-26T18:54:39.168 INFO:tasks.workunit.client.0.smithi102.stdout:9/483: creat df/d1a/d23/fb8 x:0 0 0 2024-03-26T18:54:39.170 INFO:tasks.workunit.client.0.smithi102.stdout:5/721: creat d2/d25/ffe x:0 0 0 2024-03-26T18:54:39.173 INFO:tasks.workunit.client.1.smithi190.stdout:0/862: creat d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/f137 x:0 0 0 2024-03-26T18:54:39.173 INFO:tasks.workunit.client.1.smithi190.stdout:3/691: dread d1/f26 [0,4194304] 0 2024-03-26T18:54:39.173 INFO:tasks.workunit.client.1.smithi190.stdout:2/839: unlink d6/c19 0 2024-03-26T18:54:39.173 INFO:tasks.workunit.client.1.smithi190.stdout:2/840: write d6/d9/d3a/d3d/d2e/d31/fec [1266090,18800] 0 2024-03-26T18:54:39.174 INFO:tasks.workunit.client.0.smithi102.stdout:4/379: mknod d1/d17/d1f/d5d/c78 0 2024-03-26T18:54:39.174 INFO:tasks.workunit.client.0.smithi102.stdout:4/380: readlink d1/d17/d1a/d21/l41 0 2024-03-26T18:54:39.175 INFO:tasks.workunit.client.0.smithi102.stdout:0/540: getdents d6/d3e 0 2024-03-26T18:54:39.178 INFO:tasks.workunit.client.1.smithi190.stdout:5/711: mknod d2/d17/d62/d64/ce6 0 2024-03-26T18:54:39.178 INFO:tasks.workunit.client.1.smithi190.stdout:5/712: creat d2/d17/d62/d64/fe7 x:0 0 0 2024-03-26T18:54:39.180 INFO:tasks.workunit.client.1.smithi190.stdout:2/841: rename d6/d9/d3a/d3d/d2e/d31/fec to d6/d9/d3a/d3d/d1c/dfe/f124 0 2024-03-26T18:54:39.180 INFO:tasks.workunit.client.0.smithi102.stdout:8/524: creat d0/d3/d26/fad x:0 0 0 2024-03-26T18:54:39.180 INFO:tasks.workunit.client.0.smithi102.stdout:8/525: fsync d0/d3/d2c/f92 0 2024-03-26T18:54:39.182 INFO:tasks.workunit.client.1.smithi190.stdout:2/842: chown d6/d9/d3a/d3d/d2e/l81 2133413 1 2024-03-26T18:54:39.185 INFO:tasks.workunit.client.1.smithi190.stdout:2/843: creat d6/d9/d3a/d3d/d2e/d10e/d9e/d10a/f125 x:0 0 0 2024-03-26T18:54:39.185 INFO:tasks.workunit.client.1.smithi190.stdout:3/692: mkdir d1/d3/df/d31/dc4/dd9 0 2024-03-26T18:54:39.185 INFO:tasks.workunit.client.1.smithi190.stdout:3/693: chown d1/d3/df/d31/dc4/dd9 54943704 1 2024-03-26T18:54:39.186 INFO:tasks.workunit.client.1.smithi190.stdout:3/694: readlink d1/d3/df/d18/d1b/l37 0 2024-03-26T18:54:39.186 INFO:tasks.workunit.client.1.smithi190.stdout:0/863: dread d0/dd/d20/d1c/d46/f5d [0,4194304] 0 2024-03-26T18:54:39.187 INFO:tasks.workunit.client.0.smithi102.stdout:6/491: truncate d2/d7/f92 2907123 0 2024-03-26T18:54:39.188 INFO:tasks.workunit.client.1.smithi190.stdout:5/713: dread d2/d17/d62/d64/f70 [0,4194304] 0 2024-03-26T18:54:39.189 INFO:tasks.workunit.client.1.smithi190.stdout:0/864: creat d0/d105/f138 x:0 0 0 2024-03-26T18:54:39.189 INFO:tasks.workunit.client.1.smithi190.stdout:0/865: creat d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d75/f139 x:0 0 0 2024-03-26T18:54:39.189 INFO:tasks.workunit.client.0.smithi102.stdout:1/438: truncate de/d16/d1d/d2a/d2c/d43/f86 1098136 0 2024-03-26T18:54:39.189 INFO:tasks.workunit.client.0.smithi102.stdout:1/439: write de/d16/d1d/f82 [1018484,65568] 0 2024-03-26T18:54:39.192 INFO:tasks.workunit.client.1.smithi190.stdout:5/714: rmdir d2 39 2024-03-26T18:54:39.193 INFO:tasks.workunit.client.0.smithi102.stdout:9/484: creat df/d1a/d1d/d72/d7d/fb9 x:0 0 0 2024-03-26T18:54:39.193 INFO:tasks.workunit.client.0.smithi102.stdout:9/485: write df/d1a/d57/f94 [456563,37495] 0 2024-03-26T18:54:39.193 INFO:tasks.workunit.client.1.smithi190.stdout:8/898: dwrite d9/dd/d23/d88/fe4 [0,4194304] 0 2024-03-26T18:54:39.193 INFO:tasks.workunit.client.1.smithi190.stdout:8/899: write d9/dd/d23/f3e [5026067,24591] 0 2024-03-26T18:54:39.193 INFO:tasks.workunit.client.1.smithi190.stdout:8/900: fdatasync d9/da2/d73/f81 0 2024-03-26T18:54:39.193 INFO:tasks.workunit.client.0.smithi102.stdout:5/722: write d2/d25/d75/db9/d43/d9e/d8b/fab [5702064,123643] 0 2024-03-26T18:54:39.194 INFO:tasks.workunit.client.0.smithi102.stdout:3/687: mknod df/cf6 0 2024-03-26T18:54:39.195 INFO:tasks.workunit.client.0.smithi102.stdout:3/688: readlink df/l14 0 2024-03-26T18:54:39.195 INFO:tasks.workunit.client.1.smithi190.stdout:8/901: readlink l5 0 2024-03-26T18:54:39.196 INFO:tasks.workunit.client.1.smithi190.stdout:5/715: symlink d2/d17/d1b/d60/d63/d85/le8 0 2024-03-26T18:54:39.196 INFO:tasks.workunit.client.1.smithi190.stdout:5/716: chown d2/d17/d1b/db6/d24/dd1 526477 1 2024-03-26T18:54:39.197 INFO:tasks.workunit.client.0.smithi102.stdout:4/381: truncate d1/d25/f69 2216796 0 2024-03-26T18:54:39.197 INFO:tasks.workunit.client.0.smithi102.stdout:4/382: chown d1/d17/d1a/f4b 44 1 2024-03-26T18:54:39.197 INFO:tasks.workunit.client.0.smithi102.stdout:4/383: creat d1/d17/d1a/d21/f79 x:0 0 0 2024-03-26T18:54:39.197 INFO:tasks.workunit.client.1.smithi190.stdout:8/902: mknod d9/d39/d79/dad/c133 0 2024-03-26T18:54:39.197 INFO:tasks.workunit.client.1.smithi190.stdout:0/866: rmdir d0/dd/d20/d1c/d8b/d8d/dc8 39 2024-03-26T18:54:39.197 INFO:tasks.workunit.client.1.smithi190.stdout:8/903: readlink d9/da2/dc4/l102 0 2024-03-26T18:54:39.198 INFO:tasks.workunit.client.0.smithi102.stdout:0/541: getdents d6/de/d13/d9a 0 2024-03-26T18:54:39.198 INFO:tasks.workunit.client.0.smithi102.stdout:8/526: truncate d0/d3/f28 8110348 0 2024-03-26T18:54:39.199 INFO:tasks.workunit.client.1.smithi190.stdout:8/904: link d9/da2/dc4/f5c d9/d39/d9e/d114/f134 0 2024-03-26T18:54:39.199 INFO:tasks.workunit.client.0.smithi102.stdout:4/384: dread d1/d17/d1a/f4f [0,4194304] 0 2024-03-26T18:54:39.200 INFO:tasks.workunit.client.1.smithi190.stdout:0/867: mknod d0/dd/d20/d1c/d46/da7/df0/d87/d93/dc4/c13a 0 2024-03-26T18:54:39.200 INFO:tasks.workunit.client.1.smithi190.stdout:8/905: getdents d9/d39/d79 0 2024-03-26T18:54:39.202 INFO:tasks.workunit.client.0.smithi102.stdout:1/440: mknod de/d16/d1d/d61/d6a/c8f 0 2024-03-26T18:54:39.202 INFO:tasks.workunit.client.0.smithi102.stdout:1/441: chown de/f2d 274502 1 2024-03-26T18:54:39.202 INFO:tasks.workunit.client.0.smithi102.stdout:1/442: readlink de/d16/d1d/d2a/d2c/l5b 0 2024-03-26T18:54:39.202 INFO:tasks.workunit.client.0.smithi102.stdout:1/443: write de/f3f [1365035,107237] 0 2024-03-26T18:54:39.202 INFO:tasks.workunit.client.0.smithi102.stdout:1/444: truncate de/d16/d1d/d2a/d2c/d56/d78/d80/f85 123626 0 2024-03-26T18:54:39.202 INFO:tasks.workunit.client.0.smithi102.stdout:1/445: readlink de/d16/d1d/d61/l70 0 2024-03-26T18:54:39.202 INFO:tasks.workunit.client.0.smithi102.stdout:9/486: truncate df/d1a/d57/f64 1517480 0 2024-03-26T18:54:39.203 INFO:tasks.workunit.client.0.smithi102.stdout:5/723: getdents d2/dea/dbb 0 2024-03-26T18:54:39.203 INFO:tasks.workunit.client.0.smithi102.stdout:5/724: write d2/d25/d75/ffb [634983,17096] 0 2024-03-26T18:54:39.203 INFO:tasks.workunit.client.0.smithi102.stdout:3/689: getdents df/d61 0 2024-03-26T18:54:39.203 INFO:tasks.workunit.client.0.smithi102.stdout:3/690: chown df/d29/d31/d77 788 1 2024-03-26T18:54:39.208 INFO:tasks.workunit.client.0.smithi102.stdout:4/385: mkdir d1/d27/d7a 0 2024-03-26T18:54:39.208 INFO:tasks.workunit.client.0.smithi102.stdout:4/386: chown d1/d25/d39/d3e/d74 9340 1 2024-03-26T18:54:39.209 INFO:tasks.workunit.client.0.smithi102.stdout:4/387: readlink d1/d25/d39/l5a 0 2024-03-26T18:54:39.209 INFO:tasks.workunit.client.0.smithi102.stdout:4/388: readlink d1/d17/d2a/l5c 0 2024-03-26T18:54:39.209 INFO:tasks.workunit.client.0.smithi102.stdout:6/492: getdents d2/d3d/d53 0 2024-03-26T18:54:39.209 INFO:tasks.workunit.client.0.smithi102.stdout:6/493: readlink d2/d12/l62 0 2024-03-26T18:54:39.211 INFO:tasks.workunit.client.0.smithi102.stdout:1/446: truncate de/f22 1339711 0 2024-03-26T18:54:39.212 INFO:tasks.workunit.client.1.smithi190.stdout:2/844: write d6/d9/d3a/d3d/d1c/d8d/dc7/f64 [2524651,17711] 0 2024-03-26T18:54:39.213 INFO:tasks.workunit.client.0.smithi102.stdout:9/487: rename df/d1a/d23/d3e/fa5 to df/d1a/d7f/fba 0 2024-03-26T18:54:39.213 INFO:tasks.workunit.client.0.smithi102.stdout:0/542: mknod d6/de/d13/d1f/d55/d57/cb6 0 2024-03-26T18:54:39.213 INFO:tasks.workunit.client.0.smithi102.stdout:0/543: write d6/de/f15 [2039631,22816] 0 2024-03-26T18:54:39.214 INFO:tasks.workunit.client.1.smithi190.stdout:2/845: symlink d6/d21/d61/d6c/l126 0 2024-03-26T18:54:39.214 INFO:tasks.workunit.client.0.smithi102.stdout:4/389: mknod d1/d17/d2a/c7b 0 2024-03-26T18:54:39.215 INFO:tasks.workunit.client.1.smithi190.stdout:2/846: symlink d6/d9/d45/d6b/d93/dc0/l127 0 2024-03-26T18:54:39.216 INFO:tasks.workunit.client.0.smithi102.stdout:6/494: link d2/d7/d3c/f48 d2/df/d72/f99 0 2024-03-26T18:54:39.217 INFO:tasks.workunit.client.1.smithi190.stdout:2/847: truncate d6/d9/d45/d6b/f80 1092628 0 2024-03-26T18:54:39.218 INFO:tasks.workunit.client.0.smithi102.stdout:4/390: mknod d1/d17/d1f/c7c 0 2024-03-26T18:54:39.220 INFO:tasks.workunit.client.1.smithi190.stdout:2/848: unlink d6/d9/l1b 0 2024-03-26T18:54:39.220 INFO:tasks.workunit.client.0.smithi102.stdout:6/495: mknod d2/d3d/d53/d77/c9a 0 2024-03-26T18:54:39.220 INFO:tasks.workunit.client.0.smithi102.stdout:6/496: creat d2/df/d35/d43/d75/f9b x:0 0 0 2024-03-26T18:54:39.220 INFO:tasks.workunit.client.0.smithi102.stdout:6/497: truncate d2/d3d/f70 30416 0 2024-03-26T18:54:39.221 INFO:tasks.workunit.client.0.smithi102.stdout:6/498: readlink d2/l24 0 2024-03-26T18:54:39.221 INFO:tasks.workunit.client.0.smithi102.stdout:4/391: unlink d1/d25/d39/d3e/c50 0 2024-03-26T18:54:39.221 INFO:tasks.workunit.client.0.smithi102.stdout:4/392: write d1/d17/d1a/d21/d77/f71 [985368,34434] 0 2024-03-26T18:54:39.221 INFO:tasks.workunit.client.0.smithi102.stdout:4/393: chown d1/d17/d1a/f4b 136 1 2024-03-26T18:54:39.225 INFO:tasks.workunit.client.1.smithi190.stdout:2/849: rename d6/d9/d3a/d3d/d1c/d8d/dc7/dd1/dea to d6/d9/d45/d6b/deb/d128 0 2024-03-26T18:54:39.225 INFO:tasks.workunit.client.0.smithi102.stdout:8/527: write d0/d3/f21 [557777,85834] 0 2024-03-26T18:54:39.226 INFO:tasks.workunit.client.1.smithi190.stdout:1/708: dwrite d3/da/d2e/f30 [0,4194304] 0 2024-03-26T18:54:39.227 INFO:tasks.workunit.client.1.smithi190.stdout:2/850: symlink d6/d9/d3a/d3d/d55/l129 0 2024-03-26T18:54:39.229 INFO:tasks.workunit.client.1.smithi190.stdout:2/851: creat d6/d9/d3a/d3d/d2e/d31/f12a x:0 0 0 2024-03-26T18:54:39.231 INFO:tasks.workunit.client.0.smithi102.stdout:7/471: dwrite d7/d10/d27/f33 [0,4194304] 0 2024-03-26T18:54:39.231 INFO:tasks.workunit.client.0.smithi102.stdout:7/472: fdatasync d7/d12/d51/d6a/f7a 0 2024-03-26T18:54:39.231 INFO:tasks.workunit.client.0.smithi102.stdout:7/473: creat d7/d12/d4d/f88 x:0 0 0 2024-03-26T18:54:39.231 INFO:tasks.workunit.client.0.smithi102.stdout:7/474: chown d7/d2a/f30 4568032 1 2024-03-26T18:54:39.231 INFO:tasks.workunit.client.1.smithi190.stdout:2/852: creat d6/d9/d3a/d3d/d1c/dfe/f12b x:0 0 0 2024-03-26T18:54:39.232 INFO:tasks.workunit.client.0.smithi102.stdout:5/725: dread d2/d2c/f8d [0,4194304] 0 2024-03-26T18:54:39.235 INFO:tasks.workunit.client.1.smithi190.stdout:2/853: link d6/d9/d3a/d3d/d2e/d86/f8a d6/d9/d3a/d3d/d2e/d10e/d9e/d10a/f12c 0 2024-03-26T18:54:39.238 INFO:tasks.workunit.client.0.smithi102.stdout:5/726: symlink d2/dea/d4f/ddc/lff 0 2024-03-26T18:54:39.239 INFO:tasks.workunit.client.1.smithi190.stdout:2/854: symlink d6/d9/d3a/d3d/d2e/d10e/d9e/dfc/d118/l12d 0 2024-03-26T18:54:39.239 INFO:tasks.workunit.client.1.smithi190.stdout:2/855: fdatasync d6/dbc/f105 0 2024-03-26T18:54:39.242 INFO:tasks.workunit.client.1.smithi190.stdout:7/985: dwrite d3/d1b/dd3/d10d/d50/dde/dfd/f141 [0,4194304] 0 2024-03-26T18:54:39.242 INFO:tasks.workunit.client.1.smithi190.stdout:7/986: stat d3/d6/d9/d14/d36/d12a 0 2024-03-26T18:54:39.243 INFO:tasks.workunit.client.0.smithi102.stdout:5/727: unlink d2/d25/d75/db9/lc7 0 2024-03-26T18:54:39.244 INFO:tasks.workunit.client.1.smithi190.stdout:7/987: creat d3/d1b/dd3/d10d/d65/d90/f170 x:0 0 0 2024-03-26T18:54:39.246 INFO:tasks.workunit.client.1.smithi190.stdout:7/988: mknod d3/d1b/dd3/d10d/d65/d90/d92/d9b/de9/d87/c171 0 2024-03-26T18:54:39.246 INFO:tasks.workunit.client.0.smithi102.stdout:5/728: unlink d2/l4b 0 2024-03-26T18:54:39.246 INFO:tasks.workunit.client.0.smithi102.stdout:5/729: chown d2/f10 6151 1 2024-03-26T18:54:39.247 INFO:tasks.workunit.client.1.smithi190.stdout:7/989: mkdir d3/d1b/dd3/d10d/d65/d109/d11b/d157/d172 0 2024-03-26T18:54:39.247 INFO:tasks.workunit.client.1.smithi190.stdout:7/990: write d3/d1b/dd3/d10d/f55 [1639641,39741] 0 2024-03-26T18:54:39.247 INFO:tasks.workunit.client.1.smithi190.stdout:7/991: rmdir d3/d1b/dd3/d10d/d65/d109/d119 39 2024-03-26T18:54:39.247 INFO:tasks.workunit.client.1.smithi190.stdout:7/992: dread - d3/d6/d9/d1f/d4c/fe7 zero size 2024-03-26T18:54:39.248 INFO:tasks.workunit.client.1.smithi190.stdout:7/993: symlink d3/d1b/dd3/d10d/d65/d90/l173 0 2024-03-26T18:54:39.249 INFO:tasks.workunit.client.0.smithi102.stdout:5/730: unlink d2/f12 0 2024-03-26T18:54:39.250 INFO:tasks.workunit.client.1.smithi190.stdout:7/994: creat d3/d1b/dd3/d10d/d50/dde/f174 x:0 0 0 2024-03-26T18:54:39.250 INFO:tasks.workunit.client.1.smithi190.stdout:7/995: dread - d3/d6/d9/d1f/f110 zero size 2024-03-26T18:54:39.251 INFO:tasks.workunit.client.1.smithi190.stdout:7/996: symlink d3/d1b/dd3/d10d/l175 0 2024-03-26T18:54:39.252 INFO:tasks.workunit.client.1.smithi190.stdout:7/997: rename d3/d1b/dd2 to d3/d6/d9/d14/d36/d12a/d176 0 2024-03-26T18:54:39.254 INFO:tasks.workunit.client.0.smithi102.stdout:2/431: dwrite d3/d6/f2b [0,4194304] 0 2024-03-26T18:54:39.254 INFO:tasks.workunit.client.1.smithi190.stdout:2/856: dread d6/d9/d3a/d3d/d2e/f51 [0,4194304] 0 2024-03-26T18:54:39.254 INFO:tasks.workunit.client.1.smithi190.stdout:7/998: mknod d3/d6/d9/d1f/d4c/c177 0 2024-03-26T18:54:39.255 INFO:tasks.workunit.client.0.smithi102.stdout:2/432: creat d3/d20/d78/d84/d6e/f94 x:0 0 0 2024-03-26T18:54:39.257 INFO:tasks.workunit.client.1.smithi190.stdout:3/695: dwrite d1/d3/df/d18/d35/d41/dae/f93 [0,4194304] 0 2024-03-26T18:54:39.257 INFO:tasks.workunit.client.1.smithi190.stdout:3/696: write d1/de/d81/fc1 [989897,45873] 0 2024-03-26T18:54:39.257 INFO:tasks.workunit.client.1.smithi190.stdout:3/697: chown d1/d3/d52/d70 148 1 2024-03-26T18:54:39.257 INFO:tasks.workunit.client.1.smithi190.stdout:8/906: dwrite d9/d39/d67/fa3 [4194304,4194304] 0 2024-03-26T18:54:39.258 INFO:tasks.workunit.client.0.smithi102.stdout:9/488: dwrite df/d1a/d23/d3e/d75/f4a [0,4194304] 0 2024-03-26T18:54:39.259 INFO:tasks.workunit.client.1.smithi190.stdout:7/999: rename d3/d1b/dd3/d10d/d65/d90/l131 to d3/d1b/dd3/d10d/d65/l178 0 2024-03-26T18:54:39.262 INFO:tasks.workunit.client.1.smithi190.stdout:2/857: rmdir d6/d9/d10/db7/dcf 39 2024-03-26T18:54:39.264 INFO:tasks.workunit.client.1.smithi190.stdout:0/868: dwrite d0/dd/d20/d1c/d46/da7/df0/f7a [4194304,4194304] 0 2024-03-26T18:54:39.264 INFO:tasks.workunit.client.1.smithi190.stdout:0/869: readlink d0/dd/d20/d1c/d8b/l8e 0 2024-03-26T18:54:39.264 INFO:tasks.workunit.client.1.smithi190.stdout:0/870: write d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/f9b [4362071,8960] 0 2024-03-26T18:54:39.265 INFO:tasks.workunit.client.0.smithi102.stdout:9/489: creat df/d46/da8/fbb x:0 0 0 2024-03-26T18:54:39.268 INFO:tasks.workunit.client.1.smithi190.stdout:2/858: mkdir d6/d9/d12e 0 2024-03-26T18:54:39.271 INFO:tasks.workunit.client.1.smithi190.stdout:2/859: rename d6/f8f to d6/d9/d3a/d3d/d2e/d10e/f12f 0 2024-03-26T18:54:39.271 INFO:tasks.workunit.client.1.smithi190.stdout:2/860: creat d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d8b/f130 x:0 0 0 2024-03-26T18:54:39.271 INFO:tasks.workunit.client.1.smithi190.stdout:3/698: dread d1/fb5 [4194304,4194304] 0 2024-03-26T18:54:39.273 INFO:tasks.workunit.client.1.smithi190.stdout:8/907: read d9/da2/dc4/fee [2434583,13950] 0 2024-03-26T18:54:39.273 INFO:tasks.workunit.client.1.smithi190.stdout:8/908: chown d9/dd/l127 231186 1 2024-03-26T18:54:39.273 INFO:tasks.workunit.client.0.smithi102.stdout:4/394: rmdir d1/d17/d1a 39 2024-03-26T18:54:39.273 INFO:tasks.workunit.client.0.smithi102.stdout:1/447: dwrite de/d1f/f6d [0,4194304] 0 2024-03-26T18:54:39.275 INFO:tasks.workunit.client.0.smithi102.stdout:7/475: dwrite d7/d10/d17/f35 [4194304,4194304] 0 2024-03-26T18:54:39.277 INFO:tasks.workunit.client.0.smithi102.stdout:9/490: mkdir df/dbc 0 2024-03-26T18:54:39.277 INFO:tasks.workunit.client.1.smithi190.stdout:2/861: rename d6/d9/d10/d5f/fad to d6/d9/d45/d6b/deb/f131 0 2024-03-26T18:54:39.277 INFO:tasks.workunit.client.1.smithi190.stdout:2/862: fdatasync d6/d9/d45/d6b/f80 0 2024-03-26T18:54:39.277 INFO:tasks.workunit.client.1.smithi190.stdout:2/863: creat d6/d9/d48/da0/f132 x:0 0 0 2024-03-26T18:54:39.277 INFO:tasks.workunit.client.0.smithi102.stdout:6/499: dwrite d2/df/d35/d43/f85 [0,4194304] 0 2024-03-26T18:54:39.278 INFO:tasks.workunit.client.0.smithi102.stdout:6/500: stat d2/d3/f38 0 2024-03-26T18:54:39.280 INFO:tasks.workunit.client.0.smithi102.stdout:8/528: dwrite d0/d9/f2b [0,4194304] 0 2024-03-26T18:54:39.280 INFO:tasks.workunit.client.0.smithi102.stdout:5/731: dwrite d2/dea/f59 [0,4194304] 0 2024-03-26T18:54:39.284 INFO:tasks.workunit.client.0.smithi102.stdout:4/395: mknod d1/d17/d1f/d5d/c7d 0 2024-03-26T18:54:39.284 INFO:tasks.workunit.client.0.smithi102.stdout:4/396: write d1/d17/d2a/f61 [139368,58488] 0 2024-03-26T18:54:39.286 INFO:tasks.workunit.client.0.smithi102.stdout:1/448: symlink de/d2e/l90 0 2024-03-26T18:54:39.286 INFO:tasks.workunit.client.0.smithi102.stdout:1/449: chown fc 4463 1 2024-03-26T18:54:39.289 INFO:tasks.workunit.client.1.smithi190.stdout:0/871: dwrite d0/dd/d20/d1c/d46/da7/df0/d87/d93/db5/dcc/fe3 [0,4194304] 0 2024-03-26T18:54:39.297 INFO:tasks.workunit.client.0.smithi102.stdout:6/501: rename d2/c42 to d2/df/c9c 0 2024-03-26T18:54:39.297 INFO:tasks.workunit.client.1.smithi190.stdout:3/699: read d1/de/d4e/f54 [4281161,56875] 0 2024-03-26T18:54:39.297 INFO:tasks.workunit.client.0.smithi102.stdout:4/397: write d1/d17/d1a/f53 [4677414,85180] 0 2024-03-26T18:54:39.300 INFO:tasks.workunit.client.1.smithi190.stdout:0/872: mknod d0/dd/d128/c13b 0 2024-03-26T18:54:39.300 INFO:tasks.workunit.client.1.smithi190.stdout:0/873: creat d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/f13c x:0 0 0 2024-03-26T18:54:39.300 INFO:tasks.workunit.client.1.smithi190.stdout:0/874: chown d0/dd/d20/d1c/d8b/d8d/dc8/fce 37711 1 2024-03-26T18:54:39.303 INFO:tasks.workunit.client.1.smithi190.stdout:3/700: getdents d1/d47 0 2024-03-26T18:54:39.309 INFO:tasks.workunit.client.1.smithi190.stdout:3/701: rename d1/dd3/cd6 to d1/de/d81/dd7/cda 0 2024-03-26T18:54:39.309 INFO:tasks.workunit.client.1.smithi190.stdout:3/702: chown d1/d3/df/f19 244669422 1 2024-03-26T18:54:39.314 INFO:tasks.workunit.client.0.smithi102.stdout:4/398: dread d1/d17/d2a/f54 [4194304,4194304] 0 2024-03-26T18:54:39.319 INFO:tasks.workunit.client.0.smithi102.stdout:1/450: rmdir de/d2e/d62 39 2024-03-26T18:54:39.322 INFO:tasks.workunit.client.0.smithi102.stdout:6/502: link d2/d3d/d53/d63/d6a/l79 d2/d3d/d53/l9d 0 2024-03-26T18:54:39.322 INFO:tasks.workunit.client.0.smithi102.stdout:6/503: chown d2/df/d35/d60/l82 10239 1 2024-03-26T18:54:39.324 INFO:tasks.workunit.client.0.smithi102.stdout:4/399: mknod d1/d17/d1a/d21/c7e 0 2024-03-26T18:54:39.324 INFO:tasks.workunit.client.0.smithi102.stdout:4/400: dread - d1/d25/d39/f5e zero size 2024-03-26T18:54:39.325 INFO:tasks.workunit.client.1.smithi190.stdout:8/909: dwrite d9/d39/d79/f12c [0,4194304] 0 2024-03-26T18:54:39.325 INFO:tasks.workunit.client.1.smithi190.stdout:8/910: chown d9/c9d 659 1 2024-03-26T18:54:39.327 INFO:tasks.workunit.client.0.smithi102.stdout:6/504: truncate d2/d3d/d53/f66 146727 0 2024-03-26T18:54:39.328 INFO:tasks.workunit.client.1.smithi190.stdout:8/911: unlink d9/dd/d23/db1/ff9 0 2024-03-26T18:54:39.328 INFO:tasks.workunit.client.0.smithi102.stdout:9/491: dwrite df/d1a/d23/f5d [0,4194304] 0 2024-03-26T18:54:39.329 INFO:tasks.workunit.client.1.smithi190.stdout:8/912: mknod d9/dd/d123/c135 0 2024-03-26T18:54:39.330 INFO:tasks.workunit.client.0.smithi102.stdout:4/401: creat d1/d25/d39/f7f x:0 0 0 2024-03-26T18:54:39.330 INFO:tasks.workunit.client.0.smithi102.stdout:4/402: fdatasync d1/d17/d1a/d21/f2c 0 2024-03-26T18:54:39.330 INFO:tasks.workunit.client.0.smithi102.stdout:4/403: write d1/d27/f6e [1006772,97690] 0 2024-03-26T18:54:39.330 INFO:tasks.workunit.client.0.smithi102.stdout:4/404: fsync d1/d25/f69 0 2024-03-26T18:54:39.330 INFO:tasks.workunit.client.0.smithi102.stdout:4/405: chown d1/d25/d39/l62 1506619875 1 2024-03-26T18:54:39.330 INFO:tasks.workunit.client.0.smithi102.stdout:4/406: stat d1/d27/l5b 0 2024-03-26T18:54:39.331 INFO:tasks.workunit.client.1.smithi190.stdout:4/891: sync 2024-03-26T18:54:39.331 INFO:tasks.workunit.client.1.smithi190.stdout:4/892: chown d7/d8/d14/d23/d28/d38/d96/d10f/f47 1748 1 2024-03-26T18:54:39.331 INFO:tasks.workunit.client.1.smithi190.stdout:4/893: chown d7/d8/d14/d6e/lec 3 1 2024-03-26T18:54:39.333 INFO:tasks.workunit.client.0.smithi102.stdout:9/492: getdents df/d46 0 2024-03-26T18:54:39.333 INFO:tasks.workunit.client.1.smithi190.stdout:4/894: rmdir d7/d8/d14/d6e/d6f/d10e 0 2024-03-26T18:54:39.333 INFO:tasks.workunit.client.1.smithi190.stdout:4/895: chown d7/d1d/d5e/d98/lbb 50496354 1 2024-03-26T18:54:39.333 INFO:tasks.workunit.client.1.smithi190.stdout:4/896: truncate d7/d8/d14/d23/d28/d38/d96/f103 5190136 0 2024-03-26T18:54:39.333 INFO:tasks.workunit.client.1.smithi190.stdout:4/897: chown d7/d8/d14/d23/d28/d38/d96/d10f/d40/f6a 1147020936 1 2024-03-26T18:54:39.334 INFO:tasks.workunit.client.1.smithi190.stdout:6/701: sync 2024-03-26T18:54:39.337 INFO:tasks.workunit.client.1.smithi190.stdout:6/702: rmdir d17/d28/d2c/db8 39 2024-03-26T18:54:39.342 INFO:tasks.workunit.client.0.smithi102.stdout:4/407: read d1/d27/f55 [43020,117463] 0 2024-03-26T18:54:39.345 INFO:tasks.workunit.client.1.smithi190.stdout:2/864: dwrite d6/d9/d3a/d3d/f11c [0,4194304] 0 2024-03-26T18:54:39.346 INFO:tasks.workunit.client.1.smithi190.stdout:2/865: write d6/d9/d10/db7/dcf/f11a [54429,38552] 0 2024-03-26T18:54:39.346 INFO:tasks.workunit.client.1.smithi190.stdout:8/913: read d9/f1a [2146091,126890] 0 2024-03-26T18:54:39.346 INFO:tasks.workunit.client.0.smithi102.stdout:8/529: dwrite d0/d3/d2c/d2f/fa5 [0,4194304] 0 2024-03-26T18:54:39.347 INFO:tasks.workunit.client.0.smithi102.stdout:5/732: dwrite d2/d25/d75/db9/d43/d9e/daf/fd2 [0,4194304] 0 2024-03-26T18:54:39.347 INFO:tasks.workunit.client.0.smithi102.stdout:5/733: fsync d2/d25/d75/db9/f5c 0 2024-03-26T18:54:39.347 INFO:tasks.workunit.client.0.smithi102.stdout:8/530: truncate d0/f4d 723243 0 2024-03-26T18:54:39.347 INFO:tasks.workunit.client.0.smithi102.stdout:8/531: write d0/d3/d2c/f42 [12679,16483] 0 2024-03-26T18:54:39.348 INFO:tasks.workunit.client.1.smithi190.stdout:4/898: read d7/d8/d14/d23/d28/d38/d96/f11c [2770747,45326] 0 2024-03-26T18:54:39.349 INFO:tasks.workunit.client.1.smithi190.stdout:2/866: unlink d6/cbb 0 2024-03-26T18:54:39.351 INFO:tasks.workunit.client.0.smithi102.stdout:5/734: link d2/c19 d2/d2c/d6f/c100 0 2024-03-26T18:54:39.351 INFO:tasks.workunit.client.1.smithi190.stdout:2/867: truncate d6/d9/d10/db7/dcf/fd4 3862152 0 2024-03-26T18:54:39.352 INFO:tasks.workunit.client.1.smithi190.stdout:2/868: truncate d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d8b/f121 390610 0 2024-03-26T18:54:39.352 INFO:tasks.workunit.client.1.smithi190.stdout:2/869: chown d6/d9/d48/da0 8517674 1 2024-03-26T18:54:39.352 INFO:tasks.workunit.client.0.smithi102.stdout:5/735: truncate d2/f11 3426972 0 2024-03-26T18:54:39.352 INFO:tasks.workunit.client.0.smithi102.stdout:5/736: fsync d2/dea/d4f/fb2 0 2024-03-26T18:54:39.354 INFO:tasks.workunit.client.0.smithi102.stdout:5/737: unlink d2/dea/l98 0 2024-03-26T18:54:39.355 INFO:tasks.workunit.client.1.smithi190.stdout:6/703: write d17/d28/d2b/f3c [3771658,38600] 0 2024-03-26T18:54:39.355 INFO:tasks.workunit.client.1.smithi190.stdout:6/704: chown d17/d28/d8b/d95/dda 8744 1 2024-03-26T18:54:39.355 INFO:tasks.workunit.client.1.smithi190.stdout:6/705: chown d17/d28/d46/caa 1754 1 2024-03-26T18:54:39.356 INFO:tasks.workunit.client.0.smithi102.stdout:5/738: link d2/d25/d75/db9/d43/d9e/c80 d2/d25/d75/c101 0 2024-03-26T18:54:39.356 INFO:tasks.workunit.client.0.smithi102.stdout:5/739: readlink d2/d2c/d6f/l7e 0 2024-03-26T18:54:39.356 INFO:tasks.workunit.client.0.smithi102.stdout:5/740: truncate d2/d2c/d6f/f7a 42981 0 2024-03-26T18:54:39.359 INFO:tasks.workunit.client.0.smithi102.stdout:4/408: dread d1/d17/d1a/d21/f56 [0,4194304] 0 2024-03-26T18:54:39.359 INFO:tasks.workunit.client.1.smithi190.stdout:3/703: dwrite d1/de/f29 [0,4194304] 0 2024-03-26T18:54:39.360 INFO:tasks.workunit.client.1.smithi190.stdout:3/704: chown d1/l3e 25036730 1 2024-03-26T18:54:39.361 INFO:tasks.workunit.client.0.smithi102.stdout:8/532: dread d0/d3/d2c/da7/f5b [0,4194304] 0 2024-03-26T18:54:39.361 INFO:tasks.workunit.client.0.smithi102.stdout:1/451: dwrite de/d16/f40 [0,4194304] 0 2024-03-26T18:54:39.362 INFO:tasks.workunit.client.1.smithi190.stdout:0/875: dwrite d0/dd/d20/d1c/d46/da7/df0/f54 [0,4194304] 0 2024-03-26T18:54:39.362 INFO:tasks.workunit.client.1.smithi190.stdout:0/876: stat d0/dd5/d10e 0 2024-03-26T18:54:39.362 INFO:tasks.workunit.client.1.smithi190.stdout:0/877: stat d0/dd/d20/d1c/d4e/f7e 0 2024-03-26T18:54:39.363 INFO:tasks.workunit.client.1.smithi190.stdout:6/706: link d17/d28/d46/l9e d17/d28/d74/le7 0 2024-03-26T18:54:39.366 INFO:tasks.workunit.client.1.smithi190.stdout:8/914: dread d9/d39/d67/fa3 [0,4194304] 0 2024-03-26T18:54:39.367 INFO:tasks.workunit.client.1.smithi190.stdout:3/705: symlink d1/de/d1c/ldb 0 2024-03-26T18:54:39.370 INFO:tasks.workunit.client.0.smithi102.stdout:5/741: symlink d2/d14/l102 0 2024-03-26T18:54:39.372 INFO:tasks.workunit.client.0.smithi102.stdout:6/505: dwrite d2/d7/f36 [4194304,4194304] 0 2024-03-26T18:54:39.372 INFO:tasks.workunit.client.0.smithi102.stdout:6/506: chown d2/d7/f51 28 1 2024-03-26T18:54:39.373 INFO:tasks.workunit.client.0.smithi102.stdout:6/507: creat d2/d3d/d53/d77/f9e x:0 0 0 2024-03-26T18:54:39.373 INFO:tasks.workunit.client.0.smithi102.stdout:1/452: rename de/d16/d1d/d61/f77 to de/d2e/d62/f91 0 2024-03-26T18:54:39.374 INFO:tasks.workunit.client.0.smithi102.stdout:9/493: dwrite df/d1a/d23/fb5 [0,4194304] 0 2024-03-26T18:54:39.374 INFO:tasks.workunit.client.0.smithi102.stdout:8/533: symlink d0/d15/d5d/d89/lae 0 2024-03-26T18:54:39.375 INFO:tasks.workunit.client.1.smithi190.stdout:2/870: dread d6/d9/d10/d5f/fa2 [8388608,4194304] 0 2024-03-26T18:54:39.375 INFO:tasks.workunit.client.1.smithi190.stdout:4/899: dwrite d7/d1d/d5e/f76 [0,4194304] 0 2024-03-26T18:54:39.376 INFO:tasks.workunit.client.1.smithi190.stdout:2/871: dread d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d8b/f121 [0,4194304] 0 2024-03-26T18:54:39.376 INFO:tasks.workunit.client.1.smithi190.stdout:2/872: readlink d6/d9/d3a/d3d/d2e/d31/lb2 0 2024-03-26T18:54:39.377 INFO:tasks.workunit.client.0.smithi102.stdout:5/742: creat d2/dea/f103 x:0 0 0 2024-03-26T18:54:39.383 INFO:tasks.workunit.client.0.smithi102.stdout:6/508: write d2/d7/f86 [3325364,123063] 0 2024-03-26T18:54:39.385 INFO:tasks.workunit.client.1.smithi190.stdout:0/878: rename d0/dd/d20/d1c/d46/d58/lf3 to d0/dd/d20/d1c/d46/da7/df0/d87/d93/dc4/dd8/l13d 0 2024-03-26T18:54:39.388 INFO:tasks.workunit.client.1.smithi190.stdout:6/707: mkdir d17/d28/de8 0 2024-03-26T18:54:39.389 INFO:tasks.workunit.client.1.smithi190.stdout:8/915: rename d9/dd/d15/dd3 to d9/d39/d9e/d136 0 2024-03-26T18:54:39.389 INFO:tasks.workunit.client.0.smithi102.stdout:4/409: dwrite d1/d17/d1f/f2d [0,4194304] 0 2024-03-26T18:54:39.394 INFO:tasks.workunit.client.1.smithi190.stdout:2/873: dread d6/d9/d3a/d3d/d1c/d8d/dc7/f58 [0,4194304] 0 2024-03-26T18:54:39.394 INFO:tasks.workunit.client.1.smithi190.stdout:4/900: dwrite d7/d8/d1c/ff1 [0,4194304] 0 2024-03-26T18:54:39.394 INFO:tasks.workunit.client.1.smithi190.stdout:4/901: chown d7/d8/d14/d4b/d7d/da1/fa9 5234 1 2024-03-26T18:54:39.394 INFO:tasks.workunit.client.1.smithi190.stdout:4/902: creat d7/d8/d14/d4b/d7d/f134 x:0 0 0 2024-03-26T18:54:39.397 INFO:tasks.workunit.client.1.smithi190.stdout:0/879: link d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/c62 d0/dd5/dee/df5/c13e 0 2024-03-26T18:54:39.397 INFO:tasks.workunit.client.1.smithi190.stdout:0/880: truncate d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/faf 1237881 0 2024-03-26T18:54:39.397 INFO:tasks.workunit.client.1.smithi190.stdout:2/874: mkdir d6/d9/d3a/d3d/d1c/d8d/d133 0 2024-03-26T18:54:39.397 INFO:tasks.workunit.client.0.smithi102.stdout:9/494: rename df/d1a/d57/f94 to df/d1a/d23/d3e/d75/fbd 0 2024-03-26T18:54:39.398 INFO:tasks.workunit.client.1.smithi190.stdout:4/903: dread d7/d8/d14/d4b/f4c [4194304,4194304] 0 2024-03-26T18:54:39.400 INFO:tasks.workunit.client.1.smithi190.stdout:6/708: creat d17/d28/d2c/d50/fe9 x:0 0 0 2024-03-26T18:54:39.401 INFO:tasks.workunit.client.0.smithi102.stdout:8/534: creat d0/d3/faf x:0 0 0 2024-03-26T18:54:39.401 INFO:tasks.workunit.client.0.smithi102.stdout:8/535: fsync d0/d3/d2c/f85 0 2024-03-26T18:54:39.401 INFO:tasks.workunit.client.0.smithi102.stdout:8/536: truncate d0/d15/f67 394376 0 2024-03-26T18:54:39.402 INFO:tasks.workunit.client.1.smithi190.stdout:3/706: rename d1/d47/c5e to d1/de/d1c/cdc 0 2024-03-26T18:54:39.403 INFO:tasks.workunit.client.0.smithi102.stdout:5/743: creat d2/d14/f104 x:0 0 0 2024-03-26T18:54:39.404 INFO:tasks.workunit.client.1.smithi190.stdout:6/709: creat d17/fea x:0 0 0 2024-03-26T18:54:39.409 INFO:tasks.workunit.client.1.smithi190.stdout:3/707: rename d1/de/f29 to d1/de/d9f/fdd 0 2024-03-26T18:54:39.411 INFO:tasks.workunit.client.1.smithi190.stdout:4/904: dread d7/d8/d14/d4b/db2/f118 [0,4194304] 0 2024-03-26T18:54:39.412 INFO:tasks.workunit.client.1.smithi190.stdout:6/710: symlink d17/d28/d38/d79/ddf/leb 0 2024-03-26T18:54:39.413 INFO:tasks.workunit.client.1.smithi190.stdout:6/711: write d17/d28/d2c/d50/f7e [486987,48633] 0 2024-03-26T18:54:39.414 INFO:tasks.workunit.client.1.smithi190.stdout:6/712: fdatasync d17/d28/d46/f7f 0 2024-03-26T18:54:39.414 INFO:tasks.workunit.client.1.smithi190.stdout:6/713: chown d17/d28/d46/caa 89584776 1 2024-03-26T18:54:39.416 INFO:tasks.workunit.client.1.smithi190.stdout:3/708: mkdir d1/d3/df/d18/d35/d89/dde 0 2024-03-26T18:54:39.417 INFO:tasks.workunit.client.1.smithi190.stdout:5/717: sync 2024-03-26T18:54:39.417 INFO:tasks.workunit.client.1.smithi190.stdout:1/709: sync 2024-03-26T18:54:39.418 INFO:tasks.workunit.client.1.smithi190.stdout:6/714: creat d17/d28/d8b/d95/dda/fec x:0 0 0 2024-03-26T18:54:39.418 INFO:tasks.workunit.client.1.smithi190.stdout:6/715: readlink d17/d28/d80/l85 0 2024-03-26T18:54:39.419 INFO:tasks.workunit.client.0.smithi102.stdout:1/453: creat de/d16/d1d/d2a/d67/f92 x:0 0 0 2024-03-26T18:54:39.419 INFO:tasks.workunit.client.0.smithi102.stdout:6/509: dwrite d2/df/d72/f7a [0,4194304] 0 2024-03-26T18:54:39.421 INFO:tasks.workunit.client.1.smithi190.stdout:3/709: truncate d1/d3/df/d18/d35/f60 3912585 0 2024-03-26T18:54:39.421 INFO:tasks.workunit.client.1.smithi190.stdout:3/710: write d1/de/f69 [1843119,127118] 0 2024-03-26T18:54:39.421 INFO:tasks.workunit.client.0.smithi102.stdout:9/495: mkdir df/d1a/d65/dbe 0 2024-03-26T18:54:39.424 INFO:tasks.workunit.client.1.smithi190.stdout:1/710: truncate d3/da/dd/d2f/d34/f69 4849582 0 2024-03-26T18:54:39.426 INFO:tasks.workunit.client.0.smithi102.stdout:8/537: rename d0/d3/d31/f5f to d0/d15/fb0 0 2024-03-26T18:54:39.426 INFO:tasks.workunit.client.0.smithi102.stdout:8/538: fdatasync d0/d3/d2c/d33/d87/fa8 0 2024-03-26T18:54:39.427 INFO:tasks.workunit.client.1.smithi190.stdout:6/716: mknod d17/d28/d74/d97/ced 0 2024-03-26T18:54:39.429 INFO:tasks.workunit.client.0.smithi102.stdout:1/454: mknod de/d16/d1d/d2a/d2c/d56/d78/d80/c93 0 2024-03-26T18:54:39.430 INFO:tasks.workunit.client.1.smithi190.stdout:1/711: dread d3/d9/db/f32 [0,4194304] 0 2024-03-26T18:54:39.431 INFO:tasks.workunit.client.1.smithi190.stdout:6/717: symlink d17/d28/d8b/d95/dda/lee 0 2024-03-26T18:54:39.432 INFO:tasks.workunit.client.1.smithi190.stdout:6/718: write d17/d28/d2c/d50/f65 [317018,22499] 0 2024-03-26T18:54:39.432 INFO:tasks.workunit.client.1.smithi190.stdout:6/719: chown d17/dd3 870 1 2024-03-26T18:54:39.432 INFO:tasks.workunit.client.1.smithi190.stdout:6/720: dread - d17/d28/d74/fb4 zero size 2024-03-26T18:54:39.432 INFO:tasks.workunit.client.1.smithi190.stdout:6/721: fdatasync d17/d28/d80/fd8 0 2024-03-26T18:54:39.433 INFO:tasks.workunit.client.1.smithi190.stdout:1/712: creat d3/d73/d88/dcc/fe4 x:0 0 0 2024-03-26T18:54:39.433 INFO:tasks.workunit.client.1.smithi190.stdout:1/713: dread - d3/da/d3a/d99/fc6 zero size 2024-03-26T18:54:39.433 INFO:tasks.workunit.client.1.smithi190.stdout:0/881: dwrite d0/dd/d20/f78 [0,4194304] 0 2024-03-26T18:54:39.434 INFO:tasks.workunit.client.1.smithi190.stdout:8/916: dwrite d9/d39/d79/dad/ff8 [0,4194304] 0 2024-03-26T18:54:39.435 INFO:tasks.workunit.client.0.smithi102.stdout:6/510: dread d2/d7/d3c/f48 [0,4194304] 0 2024-03-26T18:54:39.435 INFO:tasks.workunit.client.0.smithi102.stdout:9/496: creat df/d1a/d1d/d72/fbf x:0 0 0 2024-03-26T18:54:39.435 INFO:tasks.workunit.client.0.smithi102.stdout:9/497: chown df/d1a/d57 2 1 2024-03-26T18:54:39.435 INFO:tasks.workunit.client.0.smithi102.stdout:9/498: fsync df/d1a/d65/f73 0 2024-03-26T18:54:39.436 INFO:tasks.workunit.client.1.smithi190.stdout:2/875: dwrite d6/d9/d3a/f94 [4194304,4194304] 0 2024-03-26T18:54:39.436 INFO:tasks.workunit.client.1.smithi190.stdout:2/876: chown d6/lef 401 1 2024-03-26T18:54:39.436 INFO:tasks.workunit.client.1.smithi190.stdout:3/711: write d1/d3/df/d18/d35/d41/dae/f93 [1890444,59443] 0 2024-03-26T18:54:39.436 INFO:tasks.workunit.client.1.smithi190.stdout:2/877: fsync d6/d9/d10/f59 0 2024-03-26T18:54:39.436 INFO:tasks.workunit.client.1.smithi190.stdout:3/712: fsync d1/d3/df/d18/d35/f39 0 2024-03-26T18:54:39.436 INFO:tasks.workunit.client.1.smithi190.stdout:6/722: rename d17/d28/d2c/d77/f8d to d17/dd3/fef 0 2024-03-26T18:54:39.438 INFO:tasks.workunit.client.1.smithi190.stdout:4/905: dwrite d7/d8/d14/d23/d28/d38/d96/d10f/f3f [4194304,4194304] 0 2024-03-26T18:54:39.438 INFO:tasks.workunit.client.0.smithi102.stdout:5/744: rename d2/d25/db0/cb5 to d2/dea/d4f/c105 0 2024-03-26T18:54:39.439 INFO:tasks.workunit.client.1.smithi190.stdout:2/878: creat d6/d9/d3a/d3d/d1c/d8d/de8/dfb/f134 x:0 0 0 2024-03-26T18:54:39.439 INFO:tasks.workunit.client.1.smithi190.stdout:2/879: write d6/d9/d3a/d3d/d2e/d31/f12a [839807,63201] 0 2024-03-26T18:54:39.439 INFO:tasks.workunit.client.1.smithi190.stdout:0/882: unlink d0/dd/d20/d1c/d46/da7/df0/d87/d93/dc4/dd8/f130 0 2024-03-26T18:54:39.440 INFO:tasks.workunit.client.1.smithi190.stdout:4/906: truncate d7/d8/d14/f19 2157036 0 2024-03-26T18:54:39.441 INFO:tasks.workunit.client.0.smithi102.stdout:8/539: symlink d0/d15/d1b/lb1 0 2024-03-26T18:54:39.441 INFO:tasks.workunit.client.1.smithi190.stdout:2/880: getdents d6/d9/d3a/d3d/d2e/d10e/d9e/d10a 0 2024-03-26T18:54:39.442 INFO:tasks.workunit.client.1.smithi190.stdout:1/714: write d3/d9/db/f32 [2516318,68452] 0 2024-03-26T18:54:39.444 INFO:tasks.workunit.client.1.smithi190.stdout:6/723: symlink d17/d28/d2b/lf0 0 2024-03-26T18:54:39.445 INFO:tasks.workunit.client.0.smithi102.stdout:6/511: symlink d2/d12/l9f 0 2024-03-26T18:54:39.445 INFO:tasks.workunit.client.0.smithi102.stdout:6/512: fdatasync d2/d7/f1d 0 2024-03-26T18:54:39.446 INFO:tasks.workunit.client.1.smithi190.stdout:4/907: creat d7/dc8/dd5/f135 x:0 0 0 2024-03-26T18:54:39.446 INFO:tasks.workunit.client.1.smithi190.stdout:5/718: dwrite d2/f14 [0,4194304] 0 2024-03-26T18:54:39.447 INFO:tasks.workunit.client.1.smithi190.stdout:5/719: chown d2/d17/d1b/d60/d63/fa3 435150028 1 2024-03-26T18:54:39.447 INFO:tasks.workunit.client.1.smithi190.stdout:4/908: mknod d7/d8/d127/c136 0 2024-03-26T18:54:39.449 INFO:tasks.workunit.client.1.smithi190.stdout:4/909: symlink d7/d8/l137 0 2024-03-26T18:54:39.449 INFO:tasks.workunit.client.1.smithi190.stdout:4/910: truncate d7/d8/d14/d4b/d7d/da1/f110 488059 0 2024-03-26T18:54:39.449 INFO:tasks.workunit.client.1.smithi190.stdout:4/911: write d7/d8/d14/d23/d28/d38/d96/fdf [171300,92205] 0 2024-03-26T18:54:39.449 INFO:tasks.workunit.client.1.smithi190.stdout:4/912: write d7/d8/d14/d4b/d7d/da1/fe9 [763326,126597] 0 2024-03-26T18:54:39.450 INFO:tasks.workunit.client.1.smithi190.stdout:0/883: rename d0/dd/d20/d1c/d46/da7/df0/d87/d93/db5/f135 to d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/f13f 0 2024-03-26T18:54:39.450 INFO:tasks.workunit.client.1.smithi190.stdout:0/884: write d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/f13f [780043,120612] 0 2024-03-26T18:54:39.450 INFO:tasks.workunit.client.1.smithi190.stdout:0/885: write d0/dd/d20/d1c/d8b/d8d/db7/d118/f11a [3009247,15030] 0 2024-03-26T18:54:39.453 INFO:tasks.workunit.client.0.smithi102.stdout:9/499: unlink df/d1a/d57/l68 0 2024-03-26T18:54:39.454 INFO:tasks.workunit.client.1.smithi190.stdout:3/713: write d1/de/d9f/fdd [2392519,24872] 0 2024-03-26T18:54:39.454 INFO:tasks.workunit.client.1.smithi190.stdout:3/714: chown d1/d3/d5a/d6e/l77 4 1 2024-03-26T18:54:39.455 INFO:tasks.workunit.client.1.smithi190.stdout:5/720: symlink d2/d17/d1b/db6/d68/le9 0 2024-03-26T18:54:39.456 INFO:tasks.workunit.client.0.smithi102.stdout:8/540: truncate d0/d3c/d6f/f8b 51630 0 2024-03-26T18:54:39.456 INFO:tasks.workunit.client.0.smithi102.stdout:8/541: dread - d0/d3/faf zero size 2024-03-26T18:54:39.458 INFO:tasks.workunit.client.0.smithi102.stdout:5/745: dread d2/f11 [0,4194304] 0 2024-03-26T18:54:39.459 INFO:tasks.workunit.client.0.smithi102.stdout:6/513: link d2/d3/l26 d2/df/d72/la0 0 2024-03-26T18:54:39.460 INFO:tasks.workunit.client.0.smithi102.stdout:6/514: chown d2/d3d/d53/d63 979588 1 2024-03-26T18:54:39.462 INFO:tasks.workunit.client.1.smithi190.stdout:0/886: rename d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/c33 to d0/dd5/dee/c140 0 2024-03-26T18:54:39.464 INFO:tasks.workunit.client.0.smithi102.stdout:9/500: creat df/d1a/d65/dbe/fc0 x:0 0 0 2024-03-26T18:54:39.465 INFO:tasks.workunit.client.1.smithi190.stdout:3/715: getdents d1/d3/d5a/d6e 0 2024-03-26T18:54:39.465 INFO:tasks.workunit.client.1.smithi190.stdout:3/716: chown d1/de/d9f 4047692 1 2024-03-26T18:54:39.465 INFO:tasks.workunit.client.0.smithi102.stdout:3/691: sync 2024-03-26T18:54:39.467 INFO:tasks.workunit.client.0.smithi102.stdout:8/542: rename d0/d3/d31/c39 to d0/d3/cb2 0 2024-03-26T18:54:39.468 INFO:tasks.workunit.client.1.smithi190.stdout:5/721: symlink d2/d17/d1b/lea 0 2024-03-26T18:54:39.468 INFO:tasks.workunit.client.0.smithi102.stdout:3/692: dread df/d61/d55/f90 [0,4194304] 0 2024-03-26T18:54:39.468 INFO:tasks.workunit.client.0.smithi102.stdout:3/693: stat df/d61/dd6 0 2024-03-26T18:54:39.469 INFO:tasks.workunit.client.0.smithi102.stdout:6/515: mkdir d2/df/d35/da1 0 2024-03-26T18:54:39.470 INFO:tasks.workunit.client.0.smithi102.stdout:3/694: dread df/d1c/d34/f74 [0,4194304] 0 2024-03-26T18:54:39.471 INFO:tasks.workunit.client.1.smithi190.stdout:0/887: link d0/c4a d0/dd/d20/d1c/d46/da7/df0/d4b/c141 0 2024-03-26T18:54:39.472 INFO:tasks.workunit.client.0.smithi102.stdout:9/501: mkdir df/d1a/d23/d4e/dc1 0 2024-03-26T18:54:39.475 INFO:tasks.workunit.client.1.smithi190.stdout:3/717: truncate d1/f2c 1040242 0 2024-03-26T18:54:39.475 INFO:tasks.workunit.client.1.smithi190.stdout:3/718: chown d1/d3/df/d18/d35/d41/dae/f96 489144862 1 2024-03-26T18:54:39.476 INFO:tasks.workunit.client.0.smithi102.stdout:8/543: creat d0/d15/da0/fb3 x:0 0 0 2024-03-26T18:54:39.479 INFO:tasks.workunit.client.0.smithi102.stdout:6/516: symlink d2/d12/la2 0 2024-03-26T18:54:39.479 INFO:tasks.workunit.client.0.smithi102.stdout:6/517: write d2/df/f2b [545606,93065] 0 2024-03-26T18:54:39.479 INFO:tasks.workunit.client.0.smithi102.stdout:6/518: truncate d2/df/f8e 557972 0 2024-03-26T18:54:39.479 INFO:tasks.workunit.client.0.smithi102.stdout:6/519: creat d2/d3d/d53/d77/fa3 x:0 0 0 2024-03-26T18:54:39.479 INFO:tasks.workunit.client.1.smithi190.stdout:5/722: rmdir d2/d17/d1b/db6 39 2024-03-26T18:54:39.479 INFO:tasks.workunit.client.1.smithi190.stdout:8/917: dwrite d9/d39/f6d [0,4194304] 0 2024-03-26T18:54:39.480 INFO:tasks.workunit.client.1.smithi190.stdout:4/913: dwrite d7/f35 [4194304,4194304] 0 2024-03-26T18:54:39.480 INFO:tasks.workunit.client.0.smithi102.stdout:0/544: sync 2024-03-26T18:54:39.480 INFO:tasks.workunit.client.0.smithi102.stdout:0/545: readlink d6/de/d13/d25/da4/lb3 0 2024-03-26T18:54:39.481 INFO:tasks.workunit.client.0.smithi102.stdout:9/502: symlink df/d1a/d23/d29/lc2 0 2024-03-26T18:54:39.481 INFO:tasks.workunit.client.0.smithi102.stdout:9/503: readlink df/d1a/d57/laf 0 2024-03-26T18:54:39.481 INFO:tasks.workunit.client.0.smithi102.stdout:9/504: write df/d1a/d1d/d72/d7d/fb9 [246931,101703] 0 2024-03-26T18:54:39.482 INFO:tasks.workunit.client.1.smithi190.stdout:2/881: dwrite d6/d9/d3a/d3d/d2e/f5c [0,4194304] 0 2024-03-26T18:54:39.482 INFO:tasks.workunit.client.1.smithi190.stdout:2/882: fdatasync d6/d9/d48/da0/f110 0 2024-03-26T18:54:39.482 INFO:tasks.workunit.client.1.smithi190.stdout:2/883: write d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d8b/f130 [661218,49947] 0 2024-03-26T18:54:39.485 INFO:tasks.workunit.client.1.smithi190.stdout:8/918: dread d9/da2/dc4/f53 [0,4194304] 0 2024-03-26T18:54:39.485 INFO:tasks.workunit.client.0.smithi102.stdout:9/505: dread df/d1a/d23/d4e/f53 [4194304,4194304] 0 2024-03-26T18:54:39.487 INFO:tasks.workunit.client.0.smithi102.stdout:5/746: rename d2/d14/d1a/f91 to d2/d25/d75/db9/d43/d9e/d82/f106 0 2024-03-26T18:54:39.489 INFO:tasks.workunit.client.0.smithi102.stdout:8/544: creat d0/d3c/d6f/fb4 x:0 0 0 2024-03-26T18:54:39.489 INFO:tasks.workunit.client.0.smithi102.stdout:8/545: write d0/d3c/d6f/fb4 [304402,89514] 0 2024-03-26T18:54:39.489 INFO:tasks.workunit.client.0.smithi102.stdout:8/546: readlink d0/d3/d2c/d33/l3f 0 2024-03-26T18:54:39.490 INFO:tasks.workunit.client.0.smithi102.stdout:8/547: write d0/d3/faf [495437,22868] 0 2024-03-26T18:54:39.490 INFO:tasks.workunit.client.1.smithi190.stdout:0/888: dread d0/dd/d20/d1c/d46/d58/de4/f103 [4194304,4194304] 0 2024-03-26T18:54:39.491 INFO:tasks.workunit.client.0.smithi102.stdout:3/695: dread df/d8b/dbb/fcf [0,4194304] 0 2024-03-26T18:54:39.493 INFO:tasks.workunit.client.1.smithi190.stdout:3/719: dread d1/d3/f4b [0,4194304] 0 2024-03-26T18:54:39.494 INFO:tasks.workunit.client.1.smithi190.stdout:2/884: dread d6/d9/d45/d6b/deb/f131 [0,4194304] 0 2024-03-26T18:54:39.494 INFO:tasks.workunit.client.1.smithi190.stdout:2/885: fdatasync d6/d9/d3a/d3d/d1c/d8d/dc7/f43 0 2024-03-26T18:54:39.498 INFO:tasks.workunit.client.0.smithi102.stdout:0/546: rename d6/d3e/d87/f29 to d6/de/d13/d25/d4f/d66/fb7 0 2024-03-26T18:54:39.498 INFO:tasks.workunit.client.0.smithi102.stdout:0/547: write d6/d3e/d87/f52 [906291,116288] 0 2024-03-26T18:54:39.498 INFO:tasks.workunit.client.0.smithi102.stdout:0/548: write d6/de/d13/d25/d31/d45/d82/d84/d88/fa6 [600954,57379] 0 2024-03-26T18:54:39.499 INFO:tasks.workunit.client.0.smithi102.stdout:5/747: symlink d2/d25/d75/l107 0 2024-03-26T18:54:39.501 INFO:tasks.workunit.client.1.smithi190.stdout:4/914: dread d7/d1d/f32 [0,4194304] 0 2024-03-26T18:54:39.501 INFO:tasks.workunit.client.1.smithi190.stdout:4/915: fsync d7/d1d/d42/f123 0 2024-03-26T18:54:39.501 INFO:tasks.workunit.client.1.smithi190.stdout:4/916: write d7/d8/d14/d23/d28/d9e/fe7 [2958082,130014] 0 2024-03-26T18:54:39.501 INFO:tasks.workunit.client.1.smithi190.stdout:8/919: mknod d9/da2/d73/d108/d12b/d37/db8/c137 0 2024-03-26T18:54:39.502 INFO:tasks.workunit.client.0.smithi102.stdout:8/548: creat d0/d9/fb5 x:0 0 0 2024-03-26T18:54:39.503 INFO:tasks.workunit.client.0.smithi102.stdout:0/549: mknod d6/de/d13/d25/d31/d45/d82/d84/d88/db2/cb8 0 2024-03-26T18:54:39.504 INFO:tasks.workunit.client.1.smithi190.stdout:4/917: rename d7/d8/d14/d23/d28/d38/d96/cdd to d7/d8/d14/d23/d28/d38/d96/d10f/d40/c138 0 2024-03-26T18:54:39.505 INFO:tasks.workunit.client.0.smithi102.stdout:5/748: creat d2/d25/d75/db9/d43/d9e/daf/f108 x:0 0 0 2024-03-26T18:54:39.507 INFO:tasks.workunit.client.1.smithi190.stdout:8/920: rename d9/d39/d79/f90 to d9/da2/d73/d108/d12b/d37/d3a/d61/f138 0 2024-03-26T18:54:39.507 INFO:tasks.workunit.client.1.smithi190.stdout:8/921: truncate d9/dd/d23/f131 400670 0 2024-03-26T18:54:39.508 INFO:tasks.workunit.client.0.smithi102.stdout:8/549: unlink d0/d3/d2c/f34 0 2024-03-26T18:54:39.510 INFO:tasks.workunit.client.0.smithi102.stdout:0/550: mknod d6/d2d/d3d/d80/d83/cb9 0 2024-03-26T18:54:39.511 INFO:tasks.workunit.client.0.smithi102.stdout:5/749: creat d2/d25/d75/db9/d43/f109 x:0 0 0 2024-03-26T18:54:39.512 INFO:tasks.workunit.client.1.smithi190.stdout:8/922: getdents d9/da2/d73/d108/d12b/d37/d3a/d43/d8f/dd2 0 2024-03-26T18:54:39.512 INFO:tasks.workunit.client.1.smithi190.stdout:8/923: dread d9/dd/d23/f131 [0,4194304] 0 2024-03-26T18:54:39.512 INFO:tasks.workunit.client.1.smithi190.stdout:8/924: truncate d9/da2/dc4/f96 1656625 0 2024-03-26T18:54:39.512 INFO:tasks.workunit.client.1.smithi190.stdout:8/925: write d9/da2/d73/d108/d12b/d37/d3a/d43/d8f/dd2/fda [704568,30286] 0 2024-03-26T18:54:39.513 INFO:tasks.workunit.client.1.smithi190.stdout:8/926: chown d9/da2/d73/d108/d12b/d37/d3a/d43/c7b 13 1 2024-03-26T18:54:39.513 INFO:tasks.workunit.client.0.smithi102.stdout:0/551: truncate d6/de/d13/d1f/d55/d57/f5e 1026432 0 2024-03-26T18:54:39.514 INFO:tasks.workunit.client.0.smithi102.stdout:5/750: creat d2/d25/d75/db9/db3/f10a x:0 0 0 2024-03-26T18:54:39.514 INFO:tasks.workunit.client.0.smithi102.stdout:5/751: chown d2/c19 28903794 1 2024-03-26T18:54:39.514 INFO:tasks.workunit.client.0.smithi102.stdout:5/752: creat d2/d25/db0/f10b x:0 0 0 2024-03-26T18:54:39.515 INFO:tasks.workunit.client.1.smithi190.stdout:8/927: link d9/da2/l12e d9/da2/l139 0 2024-03-26T18:54:39.515 INFO:tasks.workunit.client.1.smithi190.stdout:8/928: chown d9/dd/d15/ccb 2887460 1 2024-03-26T18:54:39.516 INFO:tasks.workunit.client.1.smithi190.stdout:8/929: read d9/dd/d23/f131 [251216,53015] 0 2024-03-26T18:54:39.516 INFO:tasks.workunit.client.0.smithi102.stdout:9/506: write df/d1a/d23/fb5 [3941946,130613] 0 2024-03-26T18:54:39.516 INFO:tasks.workunit.client.0.smithi102.stdout:9/507: readlink df/d1a/d1d/l4d 0 2024-03-26T18:54:39.516 INFO:tasks.workunit.client.0.smithi102.stdout:9/508: write df/d1a/d81/fa2 [255775,128840] 0 2024-03-26T18:54:39.516 INFO:tasks.workunit.client.0.smithi102.stdout:9/509: dread - df/d1a/d23/d3e/d75/f8d zero size 2024-03-26T18:54:39.517 INFO:tasks.workunit.client.0.smithi102.stdout:9/510: write df/d1a/f6e [4699070,127543] 0 2024-03-26T18:54:39.517 INFO:tasks.workunit.client.0.smithi102.stdout:0/552: creat d6/de/d13/d1f/d55/d57/d7d/fba x:0 0 0 2024-03-26T18:54:39.517 INFO:tasks.workunit.client.0.smithi102.stdout:0/553: fsync d6/de/f4c 0 2024-03-26T18:54:39.518 INFO:tasks.workunit.client.1.smithi190.stdout:8/930: mkdir d9/da2/d73/d108/d12b/d37/d3a/d61/daf/db3/db5/de3/d13a 0 2024-03-26T18:54:39.520 INFO:tasks.workunit.client.1.smithi190.stdout:5/723: dwrite d2/d17/d98/d3a/f52 [0,4194304] 0 2024-03-26T18:54:39.521 INFO:tasks.workunit.client.0.smithi102.stdout:0/554: symlink d6/de/d13/d9a/d85/lbb 0 2024-03-26T18:54:39.522 INFO:tasks.workunit.client.1.smithi190.stdout:0/889: dread d0/dd/d20/d1c/d8b/d8d/db7/d118/f11a [0,4194304] 0 2024-03-26T18:54:39.524 INFO:tasks.workunit.client.0.smithi102.stdout:0/555: mkdir d6/dbc 0 2024-03-26T18:54:39.526 INFO:tasks.workunit.client.0.smithi102.stdout:5/753: write d2/d25/d75/db9/d43/d9e/d82/f8f [1940038,533] 0 2024-03-26T18:54:39.527 INFO:tasks.workunit.client.1.smithi190.stdout:5/724: symlink d2/d17/d1b/d8e/leb 0 2024-03-26T18:54:39.529 INFO:tasks.workunit.client.1.smithi190.stdout:0/890: link d0/dd/d20/d1c/d4e/le2 d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/l142 0 2024-03-26T18:54:39.529 INFO:tasks.workunit.client.1.smithi190.stdout:0/891: creat d0/d105/f143 x:0 0 0 2024-03-26T18:54:39.529 INFO:tasks.workunit.client.1.smithi190.stdout:0/892: write d0/dd/d20/d1c/d46/da7/df0/d4b/f89 [1552191,6505] 0 2024-03-26T18:54:39.529 INFO:tasks.workunit.client.1.smithi190.stdout:0/893: chown d0/dd/d20/d61/f7b 6 1 2024-03-26T18:54:39.529 INFO:tasks.workunit.client.1.smithi190.stdout:5/725: symlink d2/d17/d1b/db6/d68/lec 0 2024-03-26T18:54:39.531 INFO:tasks.workunit.client.1.smithi190.stdout:8/931: dread d9/dd/d23/f27 [0,4194304] 0 2024-03-26T18:54:39.532 INFO:tasks.workunit.client.1.smithi190.stdout:5/726: rename d2/d17/d1b/db6/d24/d3b/da5/fcc to d2/d17/d1b/d60/d63/dcd/fed 0 2024-03-26T18:54:39.532 INFO:tasks.workunit.client.1.smithi190.stdout:5/727: write d2/d17/d1b/d30/f73 [1562011,109652] 0 2024-03-26T18:54:39.532 INFO:tasks.workunit.client.1.smithi190.stdout:5/728: fdatasync d2/d17/d1b/db6/f2d 0 2024-03-26T18:54:39.532 INFO:tasks.workunit.client.1.smithi190.stdout:5/729: chown d2/d17/f4f 1723 1 2024-03-26T18:54:39.533 INFO:tasks.workunit.client.1.smithi190.stdout:8/932: unlink d9/d39/d79/ce6 0 2024-03-26T18:54:39.534 INFO:tasks.workunit.client.1.smithi190.stdout:5/730: rename d2/d17/d1b/db6/f54 to d2/d17/d1b/db6/d88/fee 0 2024-03-26T18:54:39.534 INFO:tasks.workunit.client.1.smithi190.stdout:5/731: write d2/fcb [658611,90273] 0 2024-03-26T18:54:39.534 INFO:tasks.workunit.client.1.smithi190.stdout:5/732: stat d2/d17 0 2024-03-26T18:54:39.536 INFO:tasks.workunit.client.1.smithi190.stdout:5/733: symlink d2/d17/d1b/d60/d63/d85/d99/lef 0 2024-03-26T18:54:39.537 INFO:tasks.workunit.client.0.smithi102.stdout:5/754: dread d2/d25/d5b/d72/f57 [0,4194304] 0 2024-03-26T18:54:39.542 INFO:tasks.workunit.client.0.smithi102.stdout:5/755: dread d2/d25/d5b/ff0 [0,4194304] 0 2024-03-26T18:54:39.546 INFO:tasks.workunit.client.0.smithi102.stdout:0/556: write d6/de/d13/d25/d31/d45/f48 [1810826,127154] 0 2024-03-26T18:54:39.553 INFO:tasks.workunit.client.1.smithi190.stdout:5/734: dread d2/d17/d1b/db6/d24/d3b/f56 [0,4194304] 0 2024-03-26T18:54:39.553 INFO:tasks.workunit.client.1.smithi190.stdout:3/720: dwrite d1/d3/f34 [0,4194304] 0 2024-03-26T18:54:39.553 INFO:tasks.workunit.client.1.smithi190.stdout:3/721: dread - d1/d3/df/d31/f56 zero size 2024-03-26T18:54:39.556 INFO:tasks.workunit.client.1.smithi190.stdout:3/722: getdents d1/de/d81/dd7 0 2024-03-26T18:54:39.556 INFO:tasks.workunit.client.1.smithi190.stdout:3/723: readlink d1/d3/df/d18/d35/d41/dae/d86/lb8 0 2024-03-26T18:54:39.557 INFO:tasks.workunit.client.0.smithi102.stdout:6/520: dwrite d2/d3d/d53/f83 [0,4194304] 0 2024-03-26T18:54:39.557 INFO:tasks.workunit.client.0.smithi102.stdout:6/521: write d2/d7/f1d [4726128,32603] 0 2024-03-26T18:54:39.558 INFO:tasks.workunit.client.0.smithi102.stdout:6/522: chown d2/df/d35/d43/l68 317683 1 2024-03-26T18:54:39.558 INFO:tasks.workunit.client.0.smithi102.stdout:0/557: read d6/de/f1d [1285918,65733] 0 2024-03-26T18:54:39.558 INFO:tasks.workunit.client.0.smithi102.stdout:9/511: dwrite df/d1a/d23/d4e/d8e/f56 [0,4194304] 0 2024-03-26T18:54:39.559 INFO:tasks.workunit.client.1.smithi190.stdout:8/933: dwrite d9/dd/d15/f1c [0,4194304] 0 2024-03-26T18:54:39.559 INFO:tasks.workunit.client.1.smithi190.stdout:4/918: dwrite d7/d8/d14/d23/d28/d38/d96/f103 [0,4194304] 0 2024-03-26T18:54:39.562 INFO:tasks.workunit.client.0.smithi102.stdout:3/696: dwrite df/d29/d3a/d93/ff1 [0,4194304] 0 2024-03-26T18:54:39.562 INFO:tasks.workunit.client.1.smithi190.stdout:2/886: dwrite d6/d9/d3a/d3d/f11c [0,4194304] 0 2024-03-26T18:54:39.563 INFO:tasks.workunit.client.0.smithi102.stdout:0/558: symlink d6/de/d13/d1f/d55/lbd 0 2024-03-26T18:54:39.569 INFO:tasks.workunit.client.1.smithi190.stdout:2/887: stat d6/d9/d3a/d3d/d2e/l84 0 2024-03-26T18:54:39.569 INFO:tasks.workunit.client.1.smithi190.stdout:2/888: chown d6/d21/dd8/ff4 1942295 1 2024-03-26T18:54:39.571 INFO:tasks.workunit.client.1.smithi190.stdout:4/919: rename d7/dc8/dd5 to d7/d1d/d42/d4a/d139 0 2024-03-26T18:54:39.579 INFO:tasks.workunit.client.1.smithi190.stdout:2/889: creat d6/d9/d3a/d3d/d1c/d8d/dc7/d72/f135 x:0 0 0 2024-03-26T18:54:39.581 INFO:tasks.workunit.client.1.smithi190.stdout:4/920: unlink d7/d1d/d42/d4a/cae 0 2024-03-26T18:54:39.581 INFO:tasks.workunit.client.1.smithi190.stdout:4/921: write d7/d8/d1c/f80 [557059,57317] 0 2024-03-26T18:54:39.581 INFO:tasks.workunit.client.1.smithi190.stdout:4/922: write d7/d1d/d5e/d98/f120 [463675,84433] 0 2024-03-26T18:54:39.581 INFO:tasks.workunit.client.1.smithi190.stdout:4/923: write d7/d8/d14/d23/feb [4398684,16648] 0 2024-03-26T18:54:39.582 INFO:tasks.workunit.client.1.smithi190.stdout:2/890: unlink d6/d9/d3a/d3d/d2e/d10e/d9e/d10a/f125 0 2024-03-26T18:54:39.590 INFO:tasks.workunit.client.1.smithi190.stdout:5/735: dwrite d2/f4 [0,4194304] 0 2024-03-26T18:54:39.590 INFO:tasks.workunit.client.1.smithi190.stdout:5/736: readlink d2/lf 0 2024-03-26T18:54:39.590 INFO:tasks.workunit.client.0.smithi102.stdout:9/512: link df/d1a/d23/fb8 df/d1a/d23/fc3 0 2024-03-26T18:54:39.597 INFO:tasks.workunit.client.1.smithi190.stdout:3/724: rmdir d1/de 39 2024-03-26T18:54:39.597 INFO:tasks.workunit.client.1.smithi190.stdout:8/934: rmdir d9/da2 39 2024-03-26T18:54:39.597 INFO:tasks.workunit.client.1.smithi190.stdout:8/935: chown d9/dd/d23 435977061 1 2024-03-26T18:54:39.612 INFO:tasks.workunit.client.0.smithi102.stdout:8/550: rename d0/d3/d31 to d0/d3c/d96/d98/db6 0 2024-03-26T18:54:39.613 INFO:tasks.workunit.client.0.smithi102.stdout:3/697: dwrite df/d29/f64 [0,4194304] 0 2024-03-26T18:54:39.615 INFO:tasks.workunit.client.0.smithi102.stdout:9/513: creat df/d1a/d23/d29/fc4 x:0 0 0 2024-03-26T18:54:39.618 INFO:tasks.workunit.client.0.smithi102.stdout:8/551: creat d0/d3/d2c/d58/fb7 x:0 0 0 2024-03-26T18:54:39.618 INFO:tasks.workunit.client.1.smithi190.stdout:3/725: mknod d1/cdf 0 2024-03-26T18:54:39.621 INFO:tasks.workunit.client.0.smithi102.stdout:3/698: unlink df/d29/f33 0 2024-03-26T18:54:39.623 INFO:tasks.workunit.client.0.smithi102.stdout:0/559: dwrite d6/de/d13/d9a/f99 [0,4194304] 0 2024-03-26T18:54:39.624 INFO:tasks.workunit.client.0.smithi102.stdout:5/756: getdents d2/d25/db0 0 2024-03-26T18:54:39.625 INFO:tasks.workunit.client.1.smithi190.stdout:4/924: dwrite d7/d1d/d42/f50 [0,4194304] 0 2024-03-26T18:54:39.625 INFO:tasks.workunit.client.0.smithi102.stdout:3/699: mkdir df/d1c/d70/dcd/df7 0 2024-03-26T18:54:39.626 INFO:tasks.workunit.client.0.smithi102.stdout:3/700: fdatasync df/fdc 0 2024-03-26T18:54:39.627 INFO:tasks.workunit.client.1.smithi190.stdout:4/925: mknod d7/d8/d14/d4b/db2/c13a 0 2024-03-26T18:54:39.628 INFO:tasks.workunit.client.1.smithi190.stdout:4/926: fdatasync d7/d8/d14/d6e/d6f/f93 0 2024-03-26T18:54:39.628 INFO:tasks.workunit.client.1.smithi190.stdout:4/927: creat d7/d8/d14/d4b/d7d/da1/d113/f13b x:0 0 0 2024-03-26T18:54:39.628 INFO:tasks.workunit.client.1.smithi190.stdout:4/928: write d7/d8/d14/d23/fdc [975922,121267] 0 2024-03-26T18:54:39.628 INFO:tasks.workunit.client.1.smithi190.stdout:4/929: chown d7/d8/d14/d23/d28/d38/d96/d10f/f58 1576 1 2024-03-26T18:54:39.628 INFO:tasks.workunit.client.1.smithi190.stdout:4/930: write d7/d1d/fa7 [397253,98283] 0 2024-03-26T18:54:39.628 INFO:tasks.workunit.client.0.smithi102.stdout:9/514: rmdir df/d1a/d1d/d72/d7d 39 2024-03-26T18:54:39.628 INFO:tasks.workunit.client.0.smithi102.stdout:9/515: chown df/d1a/l3c 16083 1 2024-03-26T18:54:39.629 INFO:tasks.workunit.client.1.smithi190.stdout:4/931: getdents d7/d8/d1c 0 2024-03-26T18:54:39.630 INFO:tasks.workunit.client.0.smithi102.stdout:5/757: mknod d2/d25/d75/db9/d43/d9e/d82/c10c 0 2024-03-26T18:54:39.630 INFO:tasks.workunit.client.1.smithi190.stdout:4/932: unlink d7/d8/d1c/c45 0 2024-03-26T18:54:39.632 INFO:tasks.workunit.client.1.smithi190.stdout:4/933: unlink d7/d1d/d41/f91 0 2024-03-26T18:54:39.632 INFO:tasks.workunit.client.1.smithi190.stdout:4/934: creat d7/d8/d14/d23/d28/d9e/f13c x:0 0 0 2024-03-26T18:54:39.632 INFO:tasks.workunit.client.1.smithi190.stdout:4/935: write d7/d8/d14/d5c/f97 [4425675,97930] 0 2024-03-26T18:54:39.632 INFO:tasks.workunit.client.0.smithi102.stdout:3/701: creat df/d61/de5/ff8 x:0 0 0 2024-03-26T18:54:39.632 INFO:tasks.workunit.client.0.smithi102.stdout:3/702: write df/d29/d3a/f6d [799837,47080] 0 2024-03-26T18:54:39.632 INFO:tasks.workunit.client.0.smithi102.stdout:3/703: creat df/d1c/ff9 x:0 0 0 2024-03-26T18:54:39.633 INFO:tasks.workunit.client.0.smithi102.stdout:3/704: write df/d61/f5f [1949418,33436] 0 2024-03-26T18:54:39.633 INFO:tasks.workunit.client.0.smithi102.stdout:3/705: chown df/d1c/f69 6 1 2024-03-26T18:54:39.633 INFO:tasks.workunit.client.1.smithi190.stdout:2/891: dwrite d6/d9/d3a/d3d/d2e/f5c [4194304,4194304] 0 2024-03-26T18:54:39.633 INFO:tasks.workunit.client.1.smithi190.stdout:2/892: creat d6/f136 x:0 0 0 2024-03-26T18:54:39.633 INFO:tasks.workunit.client.1.smithi190.stdout:4/936: symlink d7/d1d/d42/d4a/d139/de4/def/l13d 0 2024-03-26T18:54:39.634 INFO:tasks.workunit.client.1.smithi190.stdout:0/894: rmdir d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104 39 2024-03-26T18:54:39.635 INFO:tasks.workunit.client.0.smithi102.stdout:0/560: write d6/de/d13/d1f/d55/d57/f30 [1766886,119469] 0 2024-03-26T18:54:39.637 INFO:tasks.workunit.client.1.smithi190.stdout:5/737: dwrite d2/d17/d1b/db6/d24/f84 [0,4194304] 0 2024-03-26T18:54:39.637 INFO:tasks.workunit.client.0.smithi102.stdout:5/758: dread d2/d25/d5b/f94 [0,4194304] 0 2024-03-26T18:54:39.638 INFO:tasks.workunit.client.1.smithi190.stdout:8/936: dwrite d9/dd/d15/f30 [0,4194304] 0 2024-03-26T18:54:39.639 INFO:tasks.workunit.client.0.smithi102.stdout:3/706: mknod df/d29/d31/d77/d8a/dae/dee/df2/cfa 0 2024-03-26T18:54:39.639 INFO:tasks.workunit.client.0.smithi102.stdout:3/707: truncate df/d29/d31/d36/d3c/d6f/fda 949702 0 2024-03-26T18:54:39.639 INFO:tasks.workunit.client.1.smithi190.stdout:2/893: rmdir d6/d9/d45/d6b/ddb 0 2024-03-26T18:54:39.641 INFO:tasks.workunit.client.0.smithi102.stdout:9/516: read f7 [1433070,26296] 0 2024-03-26T18:54:39.641 INFO:tasks.workunit.client.1.smithi190.stdout:4/937: link d7/d8/d14/d4b/l51 d7/d8/d14/d4b/db2/d11f/l13e 0 2024-03-26T18:54:39.644 INFO:tasks.workunit.client.1.smithi190.stdout:4/938: mknod d7/d1d/d5e/c13f 0 2024-03-26T18:54:39.646 INFO:tasks.workunit.client.1.smithi190.stdout:8/937: dread d9/fc8 [0,4194304] 0 2024-03-26T18:54:39.647 INFO:tasks.workunit.client.0.smithi102.stdout:3/708: unlink df/d29/d31/d77/f9f 0 2024-03-26T18:54:39.647 INFO:tasks.workunit.client.0.smithi102.stdout:8/552: dwrite d0/d3c/d96/d98/db6/f4a [4194304,4194304] 0 2024-03-26T18:54:39.649 INFO:tasks.workunit.client.1.smithi190.stdout:5/738: dread d2/d17/d98/f5f [0,4194304] 0 2024-03-26T18:54:39.650 INFO:tasks.workunit.client.1.smithi190.stdout:5/739: creat d2/d17/d1b/db6/d24/d3b/da7/ff0 x:0 0 0 2024-03-26T18:54:39.653 INFO:tasks.workunit.client.0.smithi102.stdout:3/709: creat df/d29/d31/d36/ffb x:0 0 0 2024-03-26T18:54:39.654 INFO:tasks.workunit.client.0.smithi102.stdout:5/759: dread d2/d14/ff6 [0,4194304] 0 2024-03-26T18:54:39.659 INFO:tasks.workunit.client.1.smithi190.stdout:8/938: write d9/da2/d73/d108/d12b/d37/d3a/d61/f72 [3249189,72975] 0 2024-03-26T18:54:39.661 INFO:tasks.workunit.client.0.smithi102.stdout:3/710: read df/d61/d55/f95 [49520,110821] 0 2024-03-26T18:54:39.661 INFO:tasks.workunit.client.1.smithi190.stdout:2/894: read d6/d9/d3a/d3d/d55/f92 [2452203,17833] 0 2024-03-26T18:54:39.662 INFO:tasks.workunit.client.1.smithi190.stdout:2/895: stat d6/d9/d10/db7/dcf/f101 0 2024-03-26T18:54:39.662 INFO:tasks.workunit.client.0.smithi102.stdout:3/711: stat df/d61/l72 0 2024-03-26T18:54:39.669 INFO:tasks.workunit.client.0.smithi102.stdout:2/433: sync 2024-03-26T18:54:39.669 INFO:tasks.workunit.client.0.smithi102.stdout:4/410: sync 2024-03-26T18:54:39.670 INFO:tasks.workunit.client.0.smithi102.stdout:7/476: sync 2024-03-26T18:54:39.670 INFO:tasks.workunit.client.0.smithi102.stdout:7/477: chown d7/f41 913 1 2024-03-26T18:54:39.670 INFO:tasks.workunit.client.0.smithi102.stdout:7/478: stat d7/d10/d17/f1b 0 2024-03-26T18:54:39.672 INFO:tasks.workunit.client.0.smithi102.stdout:4/411: link d1/d27/c67 d1/d6c/c80 0 2024-03-26T18:54:39.672 INFO:tasks.workunit.client.0.smithi102.stdout:4/412: fdatasync d1/d25/d39/d3e/d4c/f6a 0 2024-03-26T18:54:39.675 INFO:tasks.workunit.client.0.smithi102.stdout:2/434: rename d3/l51 to d3/d6/d1a/d25/l95 0 2024-03-26T18:54:39.676 INFO:tasks.workunit.client.0.smithi102.stdout:0/561: dwrite d6/d2d/f38 [0,4194304] 0 2024-03-26T18:54:39.676 INFO:tasks.workunit.client.0.smithi102.stdout:3/712: rmdir df/d61/de5 39 2024-03-26T18:54:39.676 INFO:tasks.workunit.client.0.smithi102.stdout:3/713: chown df/d29/d31/d36/d3c/d6f/lef 41185710 1 2024-03-26T18:54:39.677 INFO:tasks.workunit.client.1.smithi190.stdout:2/896: dread d6/d9/d10/fd9 [0,4194304] 0 2024-03-26T18:54:39.677 INFO:tasks.workunit.client.1.smithi190.stdout:8/939: write d9/da2/d73/d108/d12b/d37/d3a/fb7 [1742138,124866] 0 2024-03-26T18:54:39.678 INFO:tasks.workunit.client.0.smithi102.stdout:4/413: creat d1/d17/d1a/d21/d77/f81 x:0 0 0 2024-03-26T18:54:39.681 INFO:tasks.workunit.client.0.smithi102.stdout:2/435: unlink d3/d6/f1d 0 2024-03-26T18:54:39.683 INFO:tasks.workunit.client.1.smithi190.stdout:6/724: sync 2024-03-26T18:54:39.683 INFO:tasks.workunit.client.1.smithi190.stdout:6/725: stat f2 0 2024-03-26T18:54:39.683 INFO:tasks.workunit.client.1.smithi190.stdout:6/726: creat d17/d28/d74/d97/ff1 x:0 0 0 2024-03-26T18:54:39.683 INFO:tasks.workunit.client.1.smithi190.stdout:1/715: sync 2024-03-26T18:54:39.684 INFO:tasks.workunit.client.0.smithi102.stdout:3/714: mknod df/d29/d31/dc0/cfc 0 2024-03-26T18:54:39.684 INFO:tasks.workunit.client.0.smithi102.stdout:3/715: creat df/d29/d31/d36/d3c/d6f/ffd x:0 0 0 2024-03-26T18:54:39.684 INFO:tasks.workunit.client.0.smithi102.stdout:9/517: dwrite df/d1a/d1d/d72/fbf [0,4194304] 0 2024-03-26T18:54:39.685 INFO:tasks.workunit.client.1.smithi190.stdout:4/939: dwrite d7/d8/d14/d23/d28/d38/d96/d10f/f47 [0,4194304] 0 2024-03-26T18:54:39.685 INFO:tasks.workunit.client.0.smithi102.stdout:2/436: mknod d3/d11/d15/d31/c96 0 2024-03-26T18:54:39.687 INFO:tasks.workunit.client.0.smithi102.stdout:8/553: dwrite d0/d3/d26/d5c/f62 [0,4194304] 0 2024-03-26T18:54:39.687 INFO:tasks.workunit.client.0.smithi102.stdout:8/554: dread - d0/d3/d2c/d2f/f7e zero size 2024-03-26T18:54:39.687 INFO:tasks.workunit.client.0.smithi102.stdout:8/555: chown d0/d3/d2c/f92 0 1 2024-03-26T18:54:39.691 INFO:tasks.workunit.client.0.smithi102.stdout:5/760: dwrite d2/d25/d5b/ff0 [0,4194304] 0 2024-03-26T18:54:39.691 INFO:tasks.workunit.client.0.smithi102.stdout:5/761: write d2/d25/d75/db9/d43/d9e/d82/f8f [93375,103742] 0 2024-03-26T18:54:39.692 INFO:tasks.workunit.client.1.smithi190.stdout:6/727: rename d17/d28/d38/d79/ddf/leb to d17/d28/d46/dd7/lf2 0 2024-03-26T18:54:39.693 INFO:tasks.workunit.client.0.smithi102.stdout:3/716: creat df/d8b/dd5/ffe x:0 0 0 2024-03-26T18:54:39.694 INFO:tasks.workunit.client.1.smithi190.stdout:2/897: write d6/d9/d48/da0/fa7 [1153037,14640] 0 2024-03-26T18:54:39.694 INFO:tasks.workunit.client.1.smithi190.stdout:1/716: unlink d3/da/l68 0 2024-03-26T18:54:39.696 INFO:tasks.workunit.client.1.smithi190.stdout:6/728: getdents d17/d28 0 2024-03-26T18:54:39.696 INFO:tasks.workunit.client.1.smithi190.stdout:6/729: read - d17/d28/d8b/d95/fc6 zero size 2024-03-26T18:54:39.697 INFO:tasks.workunit.client.1.smithi190.stdout:6/730: dread - d17/d28/d46/fbe zero size 2024-03-26T18:54:39.697 INFO:tasks.workunit.client.0.smithi102.stdout:5/762: dread d2/d2c/f4c [0,4194304] 0 2024-03-26T18:54:39.697 INFO:tasks.workunit.client.0.smithi102.stdout:9/518: mknod df/d1a/d65/dbe/cc5 0 2024-03-26T18:54:39.698 INFO:tasks.workunit.client.0.smithi102.stdout:9/519: fdatasync df/d1a/d65/f73 0 2024-03-26T18:54:39.699 INFO:tasks.workunit.client.1.smithi190.stdout:6/731: mkdir d17/d28/d2c/db8/df3 0 2024-03-26T18:54:39.699 INFO:tasks.workunit.client.0.smithi102.stdout:2/437: mkdir d3/d11/d15/d46/d7b/d97 0 2024-03-26T18:54:39.701 INFO:tasks.workunit.client.1.smithi190.stdout:6/732: mkdir d17/d28/d80/da8/df4 0 2024-03-26T18:54:39.702 INFO:tasks.workunit.client.0.smithi102.stdout:8/556: mknod d0/d15/d1b/cb8 0 2024-03-26T18:54:39.705 INFO:tasks.workunit.client.1.smithi190.stdout:6/733: creat d17/d28/d46/dd7/ff5 x:0 0 0 2024-03-26T18:54:39.706 INFO:tasks.workunit.client.0.smithi102.stdout:7/479: dwrite d7/d10/d17/f81 [0,4194304] 0 2024-03-26T18:54:39.706 INFO:tasks.workunit.client.1.smithi190.stdout:8/940: dwrite d9/f12 [0,4194304] 0 2024-03-26T18:54:39.712 INFO:tasks.workunit.client.0.smithi102.stdout:0/562: dwrite d6/de/d13/d25/d31/f50 [0,4194304] 0 2024-03-26T18:54:39.712 INFO:tasks.workunit.client.0.smithi102.stdout:0/563: write d6/d2d/d3d/fa0 [675166,70578] 0 2024-03-26T18:54:39.712 INFO:tasks.workunit.client.0.smithi102.stdout:0/564: stat d6/de/d13/d25/d4f/d66/f90 0 2024-03-26T18:54:39.712 INFO:tasks.workunit.client.0.smithi102.stdout:3/717: rename df/d1c/d70/dcd/cf0 to df/d29/d31/d36/d45/daf/cff 0 2024-03-26T18:54:39.713 INFO:tasks.workunit.client.0.smithi102.stdout:5/763: readlink d2/d25/d75/db9/lb7 0 2024-03-26T18:54:39.715 INFO:tasks.workunit.client.1.smithi190.stdout:4/940: write d7/d8/d1c/f9f [2960657,83605] 0 2024-03-26T18:54:39.716 INFO:tasks.workunit.client.1.smithi190.stdout:4/941: stat d7/d8/d14/d4b/db2 0 2024-03-26T18:54:39.716 INFO:tasks.workunit.client.0.smithi102.stdout:2/438: creat d3/d20/d78/d84/f98 x:0 0 0 2024-03-26T18:54:39.717 INFO:tasks.workunit.client.0.smithi102.stdout:8/557: creat d0/d9/fb9 x:0 0 0 2024-03-26T18:54:39.719 INFO:tasks.workunit.client.1.smithi190.stdout:8/941: dread d9/f1a [0,4194304] 0 2024-03-26T18:54:39.719 INFO:tasks.workunit.client.0.smithi102.stdout:7/480: rmdir d7/d10/d67 39 2024-03-26T18:54:39.720 INFO:tasks.workunit.client.1.smithi190.stdout:8/942: truncate d9/d39/d79/f12c 257061 0 2024-03-26T18:54:39.720 INFO:tasks.workunit.client.1.smithi190.stdout:1/717: dwrite d3/da/dd/d2f/d33/f8a [0,4194304] 0 2024-03-26T18:54:39.721 INFO:tasks.workunit.client.1.smithi190.stdout:8/943: mkdir d9/da2/d73/d108/d12b/d37/db8/d13b 0 2024-03-26T18:54:39.728 INFO:tasks.workunit.client.1.smithi190.stdout:1/718: mknod d3/d9/db/db0/ce5 0 2024-03-26T18:54:39.731 INFO:tasks.workunit.client.0.smithi102.stdout:0/565: truncate d6/d2d/d3d/fa2 9701 0 2024-03-26T18:54:39.732 INFO:tasks.workunit.client.0.smithi102.stdout:3/718: mkdir df/d29/d31/d77/d8a/dae/dee/df2/d100 0 2024-03-26T18:54:39.732 INFO:tasks.workunit.client.0.smithi102.stdout:3/719: chown df/d29/d31/d36/d45/daf/cc8 52683 1 2024-03-26T18:54:39.733 INFO:tasks.workunit.client.0.smithi102.stdout:5/764: symlink d2/d2c/d6f/db8/l10d 0 2024-03-26T18:54:39.733 INFO:tasks.workunit.client.0.smithi102.stdout:5/765: truncate d2/d25/d75/db9/d43/d9e/daf/f108 306351 0 2024-03-26T18:54:39.734 INFO:tasks.workunit.client.0.smithi102.stdout:2/439: mkdir d3/d6/d99 0 2024-03-26T18:54:39.734 INFO:tasks.workunit.client.0.smithi102.stdout:2/440: chown d3/d11/d15/d46/f47 113 1 2024-03-26T18:54:39.735 INFO:tasks.workunit.client.0.smithi102.stdout:8/558: symlink d0/d3/d2c/d33/d87/lba 0 2024-03-26T18:54:39.736 INFO:tasks.workunit.client.0.smithi102.stdout:7/481: symlink d7/d12/d4b/d83/l89 0 2024-03-26T18:54:39.739 INFO:tasks.workunit.client.0.smithi102.stdout:5/766: mkdir d2/dea/d4f/da9/d10e 0 2024-03-26T18:54:39.739 INFO:tasks.workunit.client.0.smithi102.stdout:5/767: write d2/dea/f103 [662474,1603] 0 2024-03-26T18:54:39.739 INFO:tasks.workunit.client.0.smithi102.stdout:5/768: stat d2/d25/d5b/d72/l28 0 2024-03-26T18:54:39.739 INFO:tasks.workunit.client.1.smithi190.stdout:6/734: rename d17/d28/d80 to d17/d28/df6 0 2024-03-26T18:54:39.741 INFO:tasks.workunit.client.0.smithi102.stdout:2/441: creat d3/d6/d99/f9a x:0 0 0 2024-03-26T18:54:39.743 INFO:tasks.workunit.client.1.smithi190.stdout:1/719: dread d3/da/dd/d2f/d5b/f63 [0,4194304] 0 2024-03-26T18:54:39.747 INFO:tasks.workunit.client.1.smithi190.stdout:8/944: dwrite d9/da2/dc4/f35 [0,4194304] 0 2024-03-26T18:54:39.748 INFO:tasks.workunit.client.1.smithi190.stdout:8/945: symlink d9/dd/d23/db1/l13c 0 2024-03-26T18:54:39.749 INFO:tasks.workunit.client.1.smithi190.stdout:8/946: getdents d9/da2/d73/d108 0 2024-03-26T18:54:39.754 INFO:tasks.workunit.client.0.smithi102.stdout:8/559: mkdir d0/d3/d26/d80/dbb 0 2024-03-26T18:54:39.754 INFO:tasks.workunit.client.0.smithi102.stdout:8/560: write d0/d9/fb9 [650640,56102] 0 2024-03-26T18:54:39.754 INFO:tasks.workunit.client.0.smithi102.stdout:8/561: creat d0/d3c/fbc x:0 0 0 2024-03-26T18:54:39.754 INFO:tasks.workunit.client.0.smithi102.stdout:1/455: sync 2024-03-26T18:54:39.755 INFO:tasks.workunit.client.1.smithi190.stdout:6/735: mkdir d17/d28/d2c/d50/df7 0 2024-03-26T18:54:39.756 INFO:tasks.workunit.client.0.smithi102.stdout:7/482: unlink d7/d10/c5f 0 2024-03-26T18:54:39.758 INFO:tasks.workunit.client.1.smithi190.stdout:1/720: mknod d3/da/daf/ce6 0 2024-03-26T18:54:39.758 INFO:tasks.workunit.client.0.smithi102.stdout:3/720: mknod df/c101 0 2024-03-26T18:54:39.760 INFO:tasks.workunit.client.1.smithi190.stdout:6/736: truncate d17/d28/d2c/d50/f9c 3727697 0 2024-03-26T18:54:39.761 INFO:tasks.workunit.client.0.smithi102.stdout:2/442: link d3/d11/d15/d31/l61 d3/d82/l9b 0 2024-03-26T18:54:39.765 INFO:tasks.workunit.client.0.smithi102.stdout:1/456: symlink de/d1f/l94 0 2024-03-26T18:54:39.765 INFO:tasks.workunit.client.0.smithi102.stdout:1/457: truncate de/f15 280446 0 2024-03-26T18:54:39.765 INFO:tasks.workunit.client.0.smithi102.stdout:1/458: chown de/d16/d1d/d2a/d2c/f60 340 1 2024-03-26T18:54:39.766 INFO:tasks.workunit.client.0.smithi102.stdout:6/523: sync 2024-03-26T18:54:39.766 INFO:tasks.workunit.client.0.smithi102.stdout:6/524: write d2/d3d/d53/d63/f84 [136579,48400] 0 2024-03-26T18:54:39.766 INFO:tasks.workunit.client.1.smithi190.stdout:8/947: dwrite d9/dd/d23/dea/dfe/f120 [0,4194304] 0 2024-03-26T18:54:39.767 INFO:tasks.workunit.client.1.smithi190.stdout:6/737: rename d17/d28/d2c/db8/l52 to d17/lf8 0 2024-03-26T18:54:39.768 INFO:tasks.workunit.client.1.smithi190.stdout:8/948: creat d9/da2/d73/d108/d12b/d37/d3a/d61/daf/db3/db5/de3/d13a/f13d x:0 0 0 2024-03-26T18:54:39.769 INFO:tasks.workunit.client.1.smithi190.stdout:8/949: read d9/dd/d123/fe7 [3641370,79459] 0 2024-03-26T18:54:39.769 INFO:tasks.workunit.client.0.smithi102.stdout:2/443: mknod d3/d20/d78/c9c 0 2024-03-26T18:54:39.769 INFO:tasks.workunit.client.0.smithi102.stdout:2/444: dread - d3/d11/d15/d46/f92 zero size 2024-03-26T18:54:39.771 INFO:tasks.workunit.client.0.smithi102.stdout:1/459: rename de/d1f/f7f to de/d16/d1d/d2a/d67/f95 0 2024-03-26T18:54:39.773 INFO:tasks.workunit.client.1.smithi190.stdout:6/738: creat d17/d28/d2c/db8/ff9 x:0 0 0 2024-03-26T18:54:39.774 INFO:tasks.workunit.client.0.smithi102.stdout:6/525: mkdir d2/da4 0 2024-03-26T18:54:39.775 INFO:tasks.workunit.client.1.smithi190.stdout:1/721: dread d3/d9/f2a [4194304,4194304] 0 2024-03-26T18:54:39.775 INFO:tasks.workunit.client.1.smithi190.stdout:1/722: readlink d3/da/dd/d95/d52/lc8 0 2024-03-26T18:54:39.776 INFO:tasks.workunit.client.0.smithi102.stdout:1/460: readlink de/d16/d1d/d2a/l2b 0 2024-03-26T18:54:39.777 INFO:tasks.workunit.client.1.smithi190.stdout:1/723: rmdir d3/d9/db/dd4/dd7 39 2024-03-26T18:54:39.778 INFO:tasks.workunit.client.1.smithi190.stdout:1/724: stat d3/d73/d88/fad 0 2024-03-26T18:54:39.779 INFO:tasks.workunit.client.0.smithi102.stdout:1/461: creat de/d16/d1d/d2a/d2c/d56/d78/d80/d8e/f96 x:0 0 0 2024-03-26T18:54:39.779 INFO:tasks.workunit.client.1.smithi190.stdout:8/950: dread d9/da2/f63 [0,4194304] 0 2024-03-26T18:54:39.781 INFO:tasks.workunit.client.1.smithi190.stdout:1/725: symlink d3/da/dd/d2f/d5b/d83/le7 0 2024-03-26T18:54:39.782 INFO:tasks.workunit.client.1.smithi190.stdout:1/726: write d3/da/dd/d95/f8e [3958568,93351] 0 2024-03-26T18:54:39.784 INFO:tasks.workunit.client.0.smithi102.stdout:1/462: rename de/d16/d1d/d2a/d2c/d43/f46 to de/f97 0 2024-03-26T18:54:39.785 INFO:tasks.workunit.client.1.smithi190.stdout:1/727: truncate d3/d9/f20 2821376 0 2024-03-26T18:54:39.786 INFO:tasks.workunit.client.1.smithi190.stdout:1/728: chown d3/d9/f5c 1870558368 1 2024-03-26T18:54:39.786 INFO:tasks.workunit.client.1.smithi190.stdout:1/729: write d3/da/dd/d95/f8e [543479,31096] 0 2024-03-26T18:54:39.787 INFO:tasks.workunit.client.0.smithi102.stdout:1/463: creat de/d16/d1d/d2a/d2c/d56/d78/d80/f98 x:0 0 0 2024-03-26T18:54:39.787 INFO:tasks.workunit.client.0.smithi102.stdout:1/464: creat de/f99 x:0 0 0 2024-03-26T18:54:39.790 INFO:tasks.workunit.client.1.smithi190.stdout:1/730: mkdir d3/da/dd/d2f/d33/de8 0 2024-03-26T18:54:39.793 INFO:tasks.workunit.client.1.smithi190.stdout:8/951: write d9/dd/d15/f1c [665661,107892] 0 2024-03-26T18:54:39.793 INFO:tasks.workunit.client.0.smithi102.stdout:1/465: rename de/d16/f5e to de/d16/d1d/d2a/d2c/d56/d6e/f9a 0 2024-03-26T18:54:39.794 INFO:tasks.workunit.client.0.smithi102.stdout:1/466: chown de/f19 0 1 2024-03-26T18:54:39.794 INFO:tasks.workunit.client.1.smithi190.stdout:8/952: mknod d9/d39/d67/c13e 0 2024-03-26T18:54:39.794 INFO:tasks.workunit.client.1.smithi190.stdout:8/953: fdatasync d9/dd/d23/f27 0 2024-03-26T18:54:39.794 INFO:tasks.workunit.client.1.smithi190.stdout:8/954: dread - d9/d39/d9e/d136/f126 zero size 2024-03-26T18:54:39.796 INFO:tasks.workunit.client.1.smithi190.stdout:1/731: mknod d3/da/daf/ce9 0 2024-03-26T18:54:39.796 INFO:tasks.workunit.client.1.smithi190.stdout:8/955: creat d9/da2/d73/d108/d12b/d37/d3a/d43/f13f x:0 0 0 2024-03-26T18:54:39.798 INFO:tasks.workunit.client.1.smithi190.stdout:8/956: mkdir d9/d48/d10c/d140 0 2024-03-26T18:54:39.798 INFO:tasks.workunit.client.1.smithi190.stdout:8/957: write d9/d39/d9e/d136/fde [1460918,109749] 0 2024-03-26T18:54:39.800 INFO:tasks.workunit.client.1.smithi190.stdout:8/958: rmdir d9/da2/d73/d108/d12b/d37/db8 39 2024-03-26T18:54:39.801 INFO:tasks.workunit.client.0.smithi102.stdout:3/721: write df/d29/d31/d36/f41 [2462826,25762] 0 2024-03-26T18:54:39.807 INFO:tasks.workunit.client.1.smithi190.stdout:8/959: write d9/da2/d73/d108/d12b/d37/d3a/d61/daf/db3/db5/ff0 [268356,7726] 0 2024-03-26T18:54:39.808 INFO:tasks.workunit.client.1.smithi190.stdout:8/960: symlink d9/d39/d67/l141 0 2024-03-26T18:54:39.809 INFO:tasks.workunit.client.1.smithi190.stdout:8/961: getdents d9/d39/d9e/d114 0 2024-03-26T18:54:39.822 INFO:tasks.workunit.client.1.smithi190.stdout:4/942: dwrite d7/d1d/d42/f50 [0,4194304] 0 2024-03-26T18:54:39.823 INFO:tasks.workunit.client.0.smithi102.stdout:5/769: dwrite d2/dea/d4f/fc2 [0,4194304] 0 2024-03-26T18:54:39.823 INFO:tasks.workunit.client.0.smithi102.stdout:5/770: readlink d2/l29 0 2024-03-26T18:54:39.825 INFO:tasks.workunit.client.1.smithi190.stdout:4/943: symlink d7/d8/d14/d23/d28/d38/d96/d9a/l140 0 2024-03-26T18:54:39.826 INFO:tasks.workunit.client.1.smithi190.stdout:4/944: creat d7/d8/f141 x:0 0 0 2024-03-26T18:54:39.826 INFO:tasks.workunit.client.1.smithi190.stdout:4/945: truncate d7/d8/d14/d23/d28/d38/dd3/dfd/f10d 648000 0 2024-03-26T18:54:39.837 INFO:tasks.workunit.client.1.smithi190.stdout:8/962: fsync d9/da2/dc4/ff3 0 2024-03-26T18:54:39.839 INFO:tasks.workunit.client.1.smithi190.stdout:8/963: mknod d9/da2/d73/c142 0 2024-03-26T18:54:39.840 INFO:tasks.workunit.client.1.smithi190.stdout:8/964: rename d9/dd/d23/dea/dfe/l101 to d9/da2/d73/d108/d12b/d37/d3a/d61/daf/db3/l143 0 2024-03-26T18:54:39.841 INFO:tasks.workunit.client.1.smithi190.stdout:8/965: chown d9/d39/d79/dad/lae 236657665 1 2024-03-26T18:54:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:8/562: dwrite d0/d3/f11 [0,4194304] 0 2024-03-26T18:54:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:8/563: truncate d0/f97 700916 0 2024-03-26T18:54:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:2/445: dwrite d3/d20/d78/f7a [0,4194304] 0 2024-03-26T18:54:39.844 INFO:tasks.workunit.client.1.smithi190.stdout:1/732: dwrite d3/da/dd/d2f/d5b/db5/fc7 [0,4194304] 0 2024-03-26T18:54:39.844 INFO:tasks.workunit.client.0.smithi102.stdout:3/722: dwrite df/d29/d31/d36/d3c/d6f/fda [0,4194304] 0 2024-03-26T18:54:39.844 INFO:tasks.workunit.client.0.smithi102.stdout:1/467: dwrite de/d16/f35 [0,4194304] 0 2024-03-26T18:54:39.846 INFO:tasks.workunit.client.0.smithi102.stdout:7/483: dwrite d7/d2a/f4a [0,4194304] 0 2024-03-26T18:54:39.846 INFO:tasks.workunit.client.0.smithi102.stdout:6/526: dwrite d2/f34 [0,4194304] 0 2024-03-26T18:54:39.848 INFO:tasks.workunit.client.0.smithi102.stdout:6/527: write d2/df/f27 [383920,27261] 0 2024-03-26T18:54:39.848 INFO:tasks.workunit.client.0.smithi102.stdout:8/564: getdents d0/d15 0 2024-03-26T18:54:39.854 INFO:tasks.workunit.client.1.smithi190.stdout:1/733: mkdir d3/da/d2e/dea 0 2024-03-26T18:54:39.854 INFO:tasks.workunit.client.1.smithi190.stdout:1/734: readlink d3/lc9 0 2024-03-26T18:54:39.856 INFO:tasks.workunit.client.0.smithi102.stdout:3/723: rename df/d29/fe3 to df/d1c/d34/d53/f102 0 2024-03-26T18:54:39.860 INFO:tasks.workunit.client.0.smithi102.stdout:2/446: dread d3/d11/d15/d31/d39/f5a [0,4194304] 0 2024-03-26T18:54:39.862 INFO:tasks.workunit.client.0.smithi102.stdout:7/484: read d7/d10/d17/f81 [1779993,115832] 0 2024-03-26T18:54:39.862 INFO:tasks.workunit.client.0.smithi102.stdout:1/468: dwrite de/d16/f40 [0,4194304] 0 2024-03-26T18:54:39.868 INFO:tasks.workunit.client.0.smithi102.stdout:6/528: creat d2/df/d35/da1/fa5 x:0 0 0 2024-03-26T18:54:39.868 INFO:tasks.workunit.client.0.smithi102.stdout:6/529: readlink d2/df/d35/d43/l68 0 2024-03-26T18:54:39.868 INFO:tasks.workunit.client.0.smithi102.stdout:8/565: creat d0/d3c/d6f/fbd x:0 0 0 2024-03-26T18:54:39.869 INFO:tasks.workunit.client.0.smithi102.stdout:7/485: write d7/d12/f22 [1031267,52087] 0 2024-03-26T18:54:39.869 INFO:tasks.workunit.client.0.smithi102.stdout:2/447: creat d3/d6/d1a/d25/f9d x:0 0 0 2024-03-26T18:54:39.872 INFO:tasks.workunit.client.0.smithi102.stdout:1/469: creat de/d16/d1d/d2a/d2c/f9b x:0 0 0 2024-03-26T18:54:39.873 INFO:tasks.workunit.client.0.smithi102.stdout:6/530: creat d2/d3d/d53/fa6 x:0 0 0 2024-03-26T18:54:39.874 INFO:tasks.workunit.client.0.smithi102.stdout:6/531: read d2/d7/f20 [2068,93181] 0 2024-03-26T18:54:39.874 INFO:tasks.workunit.client.0.smithi102.stdout:8/566: symlink d0/d3c/d6f/lbe 0 2024-03-26T18:54:39.876 INFO:tasks.workunit.client.0.smithi102.stdout:7/486: symlink d7/d31/l8a 0 2024-03-26T18:54:39.876 INFO:tasks.workunit.client.0.smithi102.stdout:7/487: creat d7/d12/d4d/f8b x:0 0 0 2024-03-26T18:54:39.878 INFO:tasks.workunit.client.0.smithi102.stdout:1/470: rename de/f73 to de/f9c 0 2024-03-26T18:54:39.879 INFO:tasks.workunit.client.0.smithi102.stdout:6/532: readlink d2/d7/l59 0 2024-03-26T18:54:39.880 INFO:tasks.workunit.client.0.smithi102.stdout:3/724: dwrite df/d1c/d70/dcd/fcb [0,4194304] 0 2024-03-26T18:54:39.880 INFO:tasks.workunit.client.0.smithi102.stdout:3/725: dread - df/d29/d31/d36/ffb zero size 2024-03-26T18:54:39.884 INFO:tasks.workunit.client.0.smithi102.stdout:8/567: dread d0/d3/d2c/d2f/f8f [0,4194304] 0 2024-03-26T18:54:39.885 INFO:tasks.workunit.client.0.smithi102.stdout:1/471: mkdir de/d16/d1d/d2a/d2c/d56/d6e/d9d 0 2024-03-26T18:54:39.885 INFO:tasks.workunit.client.0.smithi102.stdout:1/472: fdatasync de/d16/f40 0 2024-03-26T18:54:39.886 INFO:tasks.workunit.client.0.smithi102.stdout:6/533: symlink d2/df/d35/d60/la7 0 2024-03-26T18:54:39.887 INFO:tasks.workunit.client.0.smithi102.stdout:8/568: dread d0/d3/d2c/d2f/f8f [0,4194304] 0 2024-03-26T18:54:39.887 INFO:tasks.workunit.client.0.smithi102.stdout:8/569: fdatasync d0/d3/d2c/d33/d87/f9e 0 2024-03-26T18:54:39.890 INFO:tasks.workunit.client.0.smithi102.stdout:3/726: symlink df/d29/d31/d36/d45/daf/l103 0 2024-03-26T18:54:39.890 INFO:tasks.workunit.client.0.smithi102.stdout:7/488: getdents d7/d10/d27/d34/d72 0 2024-03-26T18:54:39.893 INFO:tasks.workunit.client.0.smithi102.stdout:6/534: symlink d2/df/d71/d80/d91/la8 0 2024-03-26T18:54:39.894 INFO:tasks.workunit.client.1.smithi190.stdout:3/726: sync 2024-03-26T18:54:39.894 INFO:tasks.workunit.client.1.smithi190.stdout:3/727: creat d1/d3/df/d18/d35/d41/dae/d9c/fe0 x:0 0 0 2024-03-26T18:54:39.895 INFO:tasks.workunit.client.0.smithi102.stdout:1/473: rename de/d16/d1d/d2a/f65 to de/d16/d1d/d2a/f9e 0 2024-03-26T18:54:39.915 INFO:tasks.workunit.client.0.smithi102.stdout:1/474: truncate de/d16/d1d/d2a/d2c/d56/d6e/f9a 2405710 0 2024-03-26T18:54:39.915 INFO:tasks.workunit.client.0.smithi102.stdout:1/475: write de/d2e/d62/f91 [530629,106319] 0 2024-03-26T18:54:39.917 INFO:tasks.workunit.client.1.smithi190.stdout:3/728: unlink d1/de/d1c/f7b 0 2024-03-26T18:54:39.917 INFO:tasks.workunit.client.1.smithi190.stdout:3/729: read - d1/d3/df/d18/d57/f94 zero size 2024-03-26T18:54:39.917 INFO:tasks.workunit.client.0.smithi102.stdout:2/448: rename d3/d6 to d3/d82/d93/d9e 0 2024-03-26T18:54:39.917 INFO:tasks.workunit.client.0.smithi102.stdout:2/449: chown d3/d11/d15/d31 32261143 1 2024-03-26T18:54:39.917 INFO:tasks.workunit.client.0.smithi102.stdout:2/450: readlink d3/d82/d93/d9e/d1a/d25/l95 0 2024-03-26T18:54:39.920 INFO:tasks.workunit.client.0.smithi102.stdout:2/451: mknod d3/d82/d93/d9e/d1a/d25/c9f 0 2024-03-26T18:54:39.920 INFO:tasks.workunit.client.0.smithi102.stdout:2/452: readlink d3/d11/d15/l87 0 2024-03-26T18:54:39.920 INFO:tasks.workunit.client.0.smithi102.stdout:2/453: dread - d3/d2a/d5f/f90 zero size 2024-03-26T18:54:39.932 INFO:tasks.workunit.client.0.smithi102.stdout:3/727: dwrite df/d29/d31/d36/d45/fd8 [0,4194304] 0 2024-03-26T18:54:39.935 INFO:tasks.workunit.client.0.smithi102.stdout:3/728: truncate fd 532133 0 2024-03-26T18:54:39.935 INFO:tasks.workunit.client.0.smithi102.stdout:3/729: creat df/d29/d31/d77/d8a/dae/f104 x:0 0 0 2024-03-26T18:54:39.949 INFO:tasks.workunit.client.0.smithi102.stdout:3/730: dread df/d29/d3a/d93/ff1 [0,4194304] 0 2024-03-26T18:54:39.953 INFO:tasks.workunit.client.0.smithi102.stdout:8/570: dwrite d0/d3c/fa4 [0,4194304] 0 2024-03-26T18:54:39.953 INFO:tasks.workunit.client.0.smithi102.stdout:6/535: dwrite d2/d3d/d53/d63/f7c [0,4194304] 0 2024-03-26T18:54:39.954 INFO:tasks.workunit.client.0.smithi102.stdout:2/454: dwrite d3/d82/d93/d9e/f44 [0,4194304] 0 2024-03-26T18:54:39.954 INFO:tasks.workunit.client.0.smithi102.stdout:1/476: dwrite de/d2e/d62/f6b [0,4194304] 0 2024-03-26T18:54:39.954 INFO:tasks.workunit.client.0.smithi102.stdout:1/477: truncate de/d16/d1d/f82 1415939 0 2024-03-26T18:54:39.958 INFO:tasks.workunit.client.0.smithi102.stdout:8/571: link d0/d3/d2c/d33/d87/f6c d0/fbf 0 2024-03-26T18:54:39.964 INFO:tasks.workunit.client.0.smithi102.stdout:6/536: write d2/df/d35/d43/f61 [3722837,70298] 0 2024-03-26T18:54:39.966 INFO:tasks.workunit.client.0.smithi102.stdout:2/455: mkdir d3/d82/d93/d9e/da0 0 2024-03-26T18:54:39.966 INFO:tasks.workunit.client.0.smithi102.stdout:2/456: creat d3/d20/fa1 x:0 0 0 2024-03-26T18:54:39.967 INFO:tasks.workunit.client.1.smithi190.stdout:3/730: write d1/f2c [2030934,4014] 0 2024-03-26T18:54:39.971 INFO:tasks.workunit.client.0.smithi102.stdout:1/478: rmdir de/d16/d1d/d61 39 2024-03-26T18:54:39.973 INFO:tasks.workunit.client.0.smithi102.stdout:1/479: mkdir de/d1f/d9f 0 2024-03-26T18:54:39.975 INFO:tasks.workunit.client.0.smithi102.stdout:1/480: unlink de/d16/d1d/f87 0 2024-03-26T18:54:39.977 INFO:tasks.workunit.client.0.smithi102.stdout:3/731: dwrite df/d29/d31/dc0/fe8 [0,4194304] 0 2024-03-26T18:54:39.977 INFO:tasks.workunit.client.0.smithi102.stdout:3/732: dread - df/d29/d31/d36/d3c/fc2 zero size 2024-03-26T18:54:39.977 INFO:tasks.workunit.client.0.smithi102.stdout:3/733: truncate df/d1c/d34/d53/fa4 986695 0 2024-03-26T18:54:39.977 INFO:tasks.workunit.client.0.smithi102.stdout:3/734: truncate df/d29/d31/d77/d8a/fd2 255387 0 2024-03-26T18:54:39.977 INFO:tasks.workunit.client.0.smithi102.stdout:4/414: sync 2024-03-26T18:54:39.982 INFO:tasks.workunit.client.0.smithi102.stdout:2/457: write d3/d2d/f32 [186450,97431] 0 2024-03-26T18:54:39.982 INFO:tasks.workunit.client.0.smithi102.stdout:8/572: dwrite d0/d3c/d6f/fab [0,4194304] 0 2024-03-26T18:54:39.982 INFO:tasks.workunit.client.0.smithi102.stdout:8/573: dread - d0/d3c/fbc zero size 2024-03-26T18:54:39.985 INFO:tasks.workunit.client.0.smithi102.stdout:3/735: mkdir df/d29/d31/d36/d45/daf/d105 0 2024-03-26T18:54:39.986 INFO:tasks.workunit.client.0.smithi102.stdout:4/415: mknod d1/d25/d39/d3e/c82 0 2024-03-26T18:54:39.987 INFO:tasks.workunit.client.0.smithi102.stdout:8/574: write d0/d3/d2c/d2f/f8f [3748583,50894] 0 2024-03-26T18:54:39.987 INFO:tasks.workunit.client.0.smithi102.stdout:8/575: stat d0/d15/d1b 0 2024-03-26T18:54:39.989 INFO:tasks.workunit.client.0.smithi102.stdout:2/458: creat d3/d11/d8f/fa2 x:0 0 0 2024-03-26T18:54:39.989 INFO:tasks.workunit.client.0.smithi102.stdout:2/459: creat d3/d11/d15/d31/fa3 x:0 0 0 2024-03-26T18:54:39.992 INFO:tasks.workunit.client.0.smithi102.stdout:8/576: read d0/d3/d2c/f85 [4156672,23964] 0 2024-03-26T18:54:39.992 INFO:tasks.workunit.client.0.smithi102.stdout:4/416: getdents d1/d17/d1f 0 2024-03-26T18:54:39.992 INFO:tasks.workunit.client.0.smithi102.stdout:4/417: chown d1/d17/c22 7810578 1 2024-03-26T18:54:39.993 INFO:tasks.workunit.client.0.smithi102.stdout:4/418: write d1/d25/d39/d3e/d4c/f68 [525805,100370] 0 2024-03-26T18:54:39.993 INFO:tasks.workunit.client.0.smithi102.stdout:4/419: chown d1/f8 1050 1 2024-03-26T18:54:39.995 INFO:tasks.workunit.client.0.smithi102.stdout:2/460: creat d3/d82/d93/fa4 x:0 0 0 2024-03-26T18:54:39.995 INFO:tasks.workunit.client.0.smithi102.stdout:2/461: fdatasync d3/d82/d93/d9e/f1b 0 2024-03-26T18:54:39.995 INFO:tasks.workunit.client.0.smithi102.stdout:2/462: read d3/d11/f55 [671424,123850] 0 2024-03-26T18:54:39.995 INFO:tasks.workunit.client.0.smithi102.stdout:2/463: creat d3/d20/d78/d7c/fa5 x:0 0 0 2024-03-26T18:54:39.995 INFO:tasks.workunit.client.0.smithi102.stdout:2/464: getdents d3/d11/d27 0 2024-03-26T18:54:39.999 INFO:tasks.workunit.client.0.smithi102.stdout:2/465: unlink d3/d11/d15/l4a 0 2024-03-26T18:54:39.999 INFO:tasks.workunit.client.0.smithi102.stdout:2/466: fdatasync d3/d11/d15/d31/d39/f63 0 2024-03-26T18:54:39.999 INFO:tasks.workunit.client.0.smithi102.stdout:2/467: write d3/f19 [4452373,80801] 0 2024-03-26T18:54:40.001 INFO:tasks.workunit.client.0.smithi102.stdout:6/537: dwrite d2/d7/d3c/f3e [0,4194304] 0 2024-03-26T18:54:40.001 INFO:tasks.workunit.client.0.smithi102.stdout:6/538: write d2/d3d/f70 [739340,59891] 0 2024-03-26T18:54:40.004 INFO:tasks.workunit.client.0.smithi102.stdout:8/577: dread d0/d3/d2c/f8e [0,4194304] 0 2024-03-26T18:54:40.004 INFO:tasks.workunit.client.0.smithi102.stdout:8/578: fsync d0/d3/d2c/d33/d87/fa8 0 2024-03-26T18:54:40.004 INFO:tasks.workunit.client.0.smithi102.stdout:8/579: write d0/d3/faf [527965,88133] 0 2024-03-26T18:54:40.004 INFO:tasks.workunit.client.0.smithi102.stdout:8/580: fsync d0/d3c/d6f/f8b 0 2024-03-26T18:54:40.005 INFO:tasks.workunit.client.0.smithi102.stdout:6/539: creat d2/df/d71/d80/d91/fa9 x:0 0 0 2024-03-26T18:54:40.007 INFO:tasks.workunit.client.0.smithi102.stdout:8/581: unlink d0/cf 0 2024-03-26T18:54:40.008 INFO:tasks.workunit.client.0.smithi102.stdout:6/540: mknod d2/d7/d3c/caa 0 2024-03-26T18:54:40.008 INFO:tasks.workunit.client.0.smithi102.stdout:6/541: write d2/d12/f3f [4661192,42867] 0 2024-03-26T18:54:40.010 INFO:tasks.workunit.client.0.smithi102.stdout:8/582: rename d0/d15/d1b/c72 to d0/d3c/cc0 0 2024-03-26T18:54:40.011 INFO:tasks.workunit.client.0.smithi102.stdout:6/542: symlink d2/df/d71/lab 0 2024-03-26T18:54:40.013 INFO:tasks.workunit.client.0.smithi102.stdout:3/736: dwrite df/d1c/d34/da5/fd3 [0,4194304] 0 2024-03-26T18:54:40.013 INFO:tasks.workunit.client.0.smithi102.stdout:4/420: dwrite d1/d17/d1a/d21/f2c [12582912,4194304] 0 2024-03-26T18:54:40.014 INFO:tasks.workunit.client.0.smithi102.stdout:3/737: dread df/d29/fc7 [0,4194304] 0 2024-03-26T18:54:40.014 INFO:tasks.workunit.client.0.smithi102.stdout:1/481: dwrite f2 [0,4194304] 0 2024-03-26T18:54:40.019 INFO:tasks.workunit.client.0.smithi102.stdout:2/468: dwrite d3/d2a/f6d [4194304,4194304] 0 2024-03-26T18:54:40.024 INFO:tasks.workunit.client.0.smithi102.stdout:3/738: creat df/d29/d31/f106 x:0 0 0 2024-03-26T18:54:40.024 INFO:tasks.workunit.client.0.smithi102.stdout:3/739: chown df/d1c/l28 78313 1 2024-03-26T18:54:40.036 INFO:tasks.workunit.client.0.smithi102.stdout:2/469: dread d3/d82/d93/d9e/ff [0,4194304] 0 2024-03-26T18:54:40.041 INFO:tasks.workunit.client.1.smithi190.stdout:0/895: sync 2024-03-26T18:54:40.041 INFO:tasks.workunit.client.1.smithi190.stdout:5/740: sync 2024-03-26T18:54:40.041 INFO:tasks.workunit.client.1.smithi190.stdout:0/896: write d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/f129 [394093,113502] 0 2024-03-26T18:54:40.043 INFO:tasks.workunit.client.1.smithi190.stdout:5/741: truncate d2/d17/d1b/f58 1087655 0 2024-03-26T18:54:40.043 INFO:tasks.workunit.client.1.smithi190.stdout:5/742: readlink l1 0 2024-03-26T18:54:40.043 INFO:tasks.workunit.client.1.smithi190.stdout:5/743: chown d2/d17/d1b/d60/d63/d85/ddc 506131926 1 2024-03-26T18:54:40.043 INFO:tasks.workunit.client.1.smithi190.stdout:5/744: write d2/d17/d62/d64/f70 [4310587,6308] 0 2024-03-26T18:54:40.043 INFO:tasks.workunit.client.1.smithi190.stdout:5/745: chown d2/d17/d98/c31 454 1 2024-03-26T18:54:40.043 INFO:tasks.workunit.client.1.smithi190.stdout:5/746: stat d2/d17/d98/d3a/l5b 0 2024-03-26T18:54:40.043 INFO:tasks.workunit.client.1.smithi190.stdout:5/747: creat d2/d17/d62/d64/ff1 x:0 0 0 2024-03-26T18:54:40.043 INFO:tasks.workunit.client.1.smithi190.stdout:0/897: link d0/dd/c6f d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d66/dcb/c144 0 2024-03-26T18:54:40.045 INFO:tasks.workunit.client.1.smithi190.stdout:5/748: mkdir d2/d17/d62/df2 0 2024-03-26T18:54:40.046 INFO:tasks.workunit.client.1.smithi190.stdout:0/898: rename d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/d121/fe9 to d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d66/d11c/f145 0 2024-03-26T18:54:40.047 INFO:tasks.workunit.client.1.smithi190.stdout:5/749: creat d2/d17/d1b/db6/ff3 x:0 0 0 2024-03-26T18:54:40.048 INFO:tasks.workunit.client.1.smithi190.stdout:5/750: symlink d2/d17/d1b/db6/d88/dbc/de4/lf4 0 2024-03-26T18:54:40.048 INFO:tasks.workunit.client.1.smithi190.stdout:5/751: getdents d2/d17/d1b/db6/d24/d3b 0 2024-03-26T18:54:40.054 INFO:tasks.workunit.client.0.smithi102.stdout:2/470: write d3/f19 [1494721,63380] 0 2024-03-26T18:54:40.055 INFO:tasks.workunit.client.0.smithi102.stdout:2/471: creat d3/d11/fa6 x:0 0 0 2024-03-26T18:54:40.055 INFO:tasks.workunit.client.0.smithi102.stdout:2/472: creat d3/d11/d8f/fa7 x:0 0 0 2024-03-26T18:54:40.058 INFO:tasks.workunit.client.0.smithi102.stdout:0/566: sync 2024-03-26T18:54:40.058 INFO:tasks.workunit.client.0.smithi102.stdout:9/520: sync 2024-03-26T18:54:40.058 INFO:tasks.workunit.client.0.smithi102.stdout:5/771: sync 2024-03-26T18:54:40.058 INFO:tasks.workunit.client.0.smithi102.stdout:9/521: dread - df/d1a/d23/fb2 zero size 2024-03-26T18:54:40.059 INFO:tasks.workunit.client.0.smithi102.stdout:6/543: dwrite d2/df/d72/f89 [0,4194304] 0 2024-03-26T18:54:40.059 INFO:tasks.workunit.client.0.smithi102.stdout:0/567: mkdir d6/dbe 0 2024-03-26T18:54:40.060 INFO:tasks.workunit.client.0.smithi102.stdout:9/522: symlink df/d1a/d23/d3e/d75/lc6 0 2024-03-26T18:54:40.060 INFO:tasks.workunit.client.1.smithi190.stdout:0/899: write d0/dd/d20/d1c/d46/da7/df0/da2/fa5 [680028,120142] 0 2024-03-26T18:54:40.060 INFO:tasks.workunit.client.1.smithi190.stdout:0/900: getdents d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/d125 0 2024-03-26T18:54:40.062 INFO:tasks.workunit.client.0.smithi102.stdout:6/544: rename d2/d7/l8d to d2/df/d71/d80/d91/lac 0 2024-03-26T18:54:40.062 INFO:tasks.workunit.client.0.smithi102.stdout:6/545: dread - d2/df/d35/d43/d75/f9b zero size 2024-03-26T18:54:40.062 INFO:tasks.workunit.client.0.smithi102.stdout:6/546: creat d2/d12/fad x:0 0 0 2024-03-26T18:54:40.063 INFO:tasks.workunit.client.0.smithi102.stdout:0/568: symlink d6/d2d/d3d/lbf 0 2024-03-26T18:54:40.063 INFO:tasks.workunit.client.0.smithi102.stdout:0/569: truncate d6/de/d13/d25/d4f/f73 369463 0 2024-03-26T18:54:40.065 INFO:tasks.workunit.client.0.smithi102.stdout:6/547: unlink d2/d3d/d53/d63/f84 0 2024-03-26T18:54:40.066 INFO:tasks.workunit.client.1.smithi190.stdout:0/901: dread d0/dd/fe [0,4194304] 0 2024-03-26T18:54:40.067 INFO:tasks.workunit.client.1.smithi190.stdout:0/902: symlink d0/dd/d20/d1c/d46/da7/df0/d87/d93/dc4/dd8/l146 0 2024-03-26T18:54:40.068 INFO:tasks.workunit.client.1.smithi190.stdout:0/903: stat d0/dd5/dee/df5/c13e 0 2024-03-26T18:54:40.070 INFO:tasks.workunit.client.1.smithi190.stdout:0/904: write d0/dd/fe [188851,94159] 0 2024-03-26T18:54:40.070 INFO:tasks.workunit.client.1.smithi190.stdout:0/905: fdatasync d0/d105/f143 0 2024-03-26T18:54:40.071 INFO:tasks.workunit.client.1.smithi190.stdout:0/906: creat d0/dd/d20/d1c/d4e/f147 x:0 0 0 2024-03-26T18:54:40.071 INFO:tasks.workunit.client.1.smithi190.stdout:0/907: truncate d0/dd/d20/d1c/d46/da7/df0/f7a 8921672 0 2024-03-26T18:54:40.071 INFO:tasks.workunit.client.1.smithi190.stdout:0/908: write d0/dd/d20/d1c/d46/da7/df0/db4/ff4 [258532,26860] 0 2024-03-26T18:54:40.071 INFO:tasks.workunit.client.0.smithi102.stdout:8/583: dwrite d0/d3/d2c/d33/f49 [4194304,4194304] 0 2024-03-26T18:54:40.073 INFO:tasks.workunit.client.1.smithi190.stdout:0/909: creat d0/dd/d20/d1c/d46/da7/df0/d87/d93/f148 x:0 0 0 2024-03-26T18:54:40.074 INFO:tasks.workunit.client.0.smithi102.stdout:9/523: dread df/d1a/f24 [0,4194304] 0 2024-03-26T18:54:40.074 INFO:tasks.workunit.client.0.smithi102.stdout:9/524: fdatasync df/f17 0 2024-03-26T18:54:40.074 INFO:tasks.workunit.client.0.smithi102.stdout:9/525: chown df/d1a/d57/f8f 817 1 2024-03-26T18:54:40.075 INFO:tasks.workunit.client.0.smithi102.stdout:4/421: dwrite d1/d25/d39/f66 [0,4194304] 0 2024-03-26T18:54:40.075 INFO:tasks.workunit.client.0.smithi102.stdout:4/422: fdatasync d1/d25/d39/f4e 0 2024-03-26T18:54:40.077 INFO:tasks.workunit.client.1.smithi190.stdout:0/910: mkdir d0/dd/d20/d1c/d8b/d8d/db7/d149 0 2024-03-26T18:54:40.077 INFO:tasks.workunit.client.0.smithi102.stdout:9/526: creat df/d1a/d23/d4e/db0/fc7 x:0 0 0 2024-03-26T18:54:40.078 INFO:tasks.workunit.client.1.smithi190.stdout:0/911: mknod d0/dd/d20/d1c/d46/d7c/c14a 0 2024-03-26T18:54:40.079 INFO:tasks.workunit.client.0.smithi102.stdout:4/423: truncate d1/d17/d1a/d21/f56 3164138 0 2024-03-26T18:54:40.079 INFO:tasks.workunit.client.0.smithi102.stdout:9/527: mknod df/d1a/d57/cc8 0 2024-03-26T18:54:40.080 INFO:tasks.workunit.client.0.smithi102.stdout:3/740: dwrite df/d29/d31/f106 [0,4194304] 0 2024-03-26T18:54:40.080 INFO:tasks.workunit.client.0.smithi102.stdout:1/482: dwrite f3 [0,4194304] 0 2024-03-26T18:54:40.082 INFO:tasks.workunit.client.0.smithi102.stdout:4/424: rename d1/d17/d1f/d5d/c78 to d1/d27/c83 0 2024-03-26T18:54:40.085 INFO:tasks.workunit.client.0.smithi102.stdout:0/570: dread d6/de/d13/d1f/f3b [4194304,4194304] 0 2024-03-26T18:54:40.087 INFO:tasks.workunit.client.0.smithi102.stdout:5/772: dwrite d2/d25/d5b/d76/fa3 [0,4194304] 0 2024-03-26T18:54:40.087 INFO:tasks.workunit.client.0.smithi102.stdout:6/548: dwrite d2/df/f64 [0,4194304] 0 2024-03-26T18:54:40.087 INFO:tasks.workunit.client.0.smithi102.stdout:6/549: fsync d2/f34 0 2024-03-26T18:54:40.088 INFO:tasks.workunit.client.1.smithi190.stdout:5/752: dwrite d2/f45 [0,4194304] 0 2024-03-26T18:54:40.088 INFO:tasks.workunit.client.1.smithi190.stdout:5/753: write d2/d17/d62/d64/fe7 [42517,84406] 0 2024-03-26T18:54:40.088 INFO:tasks.workunit.client.1.smithi190.stdout:5/754: creat d2/d17/d62/d64/ff5 x:0 0 0 2024-03-26T18:54:40.091 INFO:tasks.workunit.client.0.smithi102.stdout:3/741: rename df/d61/d55/c6e to df/d29/d31/d36/d45/daf/d105/c107 0 2024-03-26T18:54:40.092 INFO:tasks.workunit.client.1.smithi190.stdout:0/912: read d0/dd/d20/d61/f7b [12428736,2023] 0 2024-03-26T18:54:40.096 INFO:tasks.workunit.client.1.smithi190.stdout:5/755: creat d2/d17/ddd/ff6 x:0 0 0 2024-03-26T18:54:40.096 INFO:tasks.workunit.client.1.smithi190.stdout:5/756: dread - d2/d17/ddd/ff6 zero size 2024-03-26T18:54:40.098 INFO:tasks.workunit.client.1.smithi190.stdout:0/913: creat d0/dd/d20/d1c/d8b/f14b x:0 0 0 2024-03-26T18:54:40.099 INFO:tasks.workunit.client.0.smithi102.stdout:0/571: dread d6/de/d13/d9a/d85/f61 [4194304,4194304] 0 2024-03-26T18:54:40.101 INFO:tasks.workunit.client.0.smithi102.stdout:2/473: rmdir d3/d11 39 2024-03-26T18:54:40.103 INFO:tasks.workunit.client.1.smithi190.stdout:5/757: truncate d2/d17/d1b/db6/d24/d3b/f46 726783 0 2024-03-26T18:54:40.103 INFO:tasks.workunit.client.1.smithi190.stdout:5/758: write d2/d17/fe5 [597850,77220] 0 2024-03-26T18:54:40.104 INFO:tasks.workunit.client.0.smithi102.stdout:3/742: dread df/fb1 [0,4194304] 0 2024-03-26T18:54:40.104 INFO:tasks.workunit.client.0.smithi102.stdout:3/743: creat df/d29/d31/f108 x:0 0 0 2024-03-26T18:54:40.104 INFO:tasks.workunit.client.0.smithi102.stdout:6/550: symlink d2/d3d/d53/d63/lae 0 2024-03-26T18:54:40.104 INFO:tasks.workunit.client.0.smithi102.stdout:6/551: truncate d2/f47 4581392 0 2024-03-26T18:54:40.104 INFO:tasks.workunit.client.0.smithi102.stdout:6/552: readlink d2/d3d/d53/d63/d6a/l79 0 2024-03-26T18:54:40.106 INFO:tasks.workunit.client.0.smithi102.stdout:1/483: dwrite de/d1f/f53 [4194304,4194304] 0 2024-03-26T18:54:40.107 INFO:tasks.workunit.client.1.smithi190.stdout:5/759: rename d2/d17/d1b/db6/f28 to d2/d17/ff7 0 2024-03-26T18:54:40.108 INFO:tasks.workunit.client.0.smithi102.stdout:4/425: rename d1/d27/l5b to d1/d17/d1f/d5d/d6f/l84 0 2024-03-26T18:54:40.108 INFO:tasks.workunit.client.0.smithi102.stdout:4/426: read - d1/d25/d39/f7f zero size 2024-03-26T18:54:40.111 INFO:tasks.workunit.client.1.smithi190.stdout:5/760: dread d2/d17/d98/d3a/f55 [0,4194304] 0 2024-03-26T18:54:40.112 INFO:tasks.workunit.client.0.smithi102.stdout:0/572: dread d6/d2d/f38 [0,4194304] 0 2024-03-26T18:54:40.114 INFO:tasks.workunit.client.0.smithi102.stdout:6/553: link d2/df/d71/d80/d91/fa9 d2/faf 0 2024-03-26T18:54:40.114 INFO:tasks.workunit.client.0.smithi102.stdout:6/554: readlink d2/d3/l26 0 2024-03-26T18:54:40.115 INFO:tasks.workunit.client.0.smithi102.stdout:2/474: dread d3/f1c [0,4194304] 0 2024-03-26T18:54:40.115 INFO:tasks.workunit.client.0.smithi102.stdout:2/475: readlink d3/d82/d93/d9e/d1a/d25/l95 0 2024-03-26T18:54:40.116 INFO:tasks.workunit.client.0.smithi102.stdout:2/476: chown d3/d82/l8c 608959 1 2024-03-26T18:54:40.117 INFO:tasks.workunit.client.0.smithi102.stdout:1/484: symlink de/d16/d1d/d2a/d2c/d56/d5f/la0 0 2024-03-26T18:54:40.120 INFO:tasks.workunit.client.0.smithi102.stdout:5/773: rename d2/fcd to d2/d25/de6/f10f 0 2024-03-26T18:54:40.121 INFO:tasks.workunit.client.0.smithi102.stdout:5/774: dread d2/d25/d75/db9/d43/f92 [0,4194304] 0 2024-03-26T18:54:40.121 INFO:tasks.workunit.client.0.smithi102.stdout:0/573: symlink d6/de/d13/d9a/d85/lc0 0 2024-03-26T18:54:40.121 INFO:tasks.workunit.client.0.smithi102.stdout:0/574: chown d6/de/d13/d1f/d55/d57/cb6 0 1 2024-03-26T18:54:40.125 INFO:tasks.workunit.client.0.smithi102.stdout:6/555: dread d2/d7/f1d [0,4194304] 0 2024-03-26T18:54:40.129 INFO:tasks.workunit.client.1.smithi190.stdout:5/761: dread d2/d17/d1b/db6/fc1 [0,4194304] 0 2024-03-26T18:54:40.129 INFO:tasks.workunit.client.1.smithi190.stdout:5/762: chown d2/d17/d1b/db6/d88/dbc/lb4 498 1 2024-03-26T18:54:40.130 INFO:tasks.workunit.client.1.smithi190.stdout:0/914: dwrite d0/dd/d20/d1c/d46/da7/df0/da2/d80/f124 [0,4194304] 0 2024-03-26T18:54:40.130 INFO:tasks.workunit.client.1.smithi190.stdout:0/915: write d0/dd/d20/d1c/d8b/d8d/dc8/fce [278163,4480] 0 2024-03-26T18:54:40.130 INFO:tasks.workunit.client.1.smithi190.stdout:0/916: chown d0/dd/d20/d1c/d8b/d8d/d8f/f12a 13255 1 2024-03-26T18:54:40.130 INFO:tasks.workunit.client.0.smithi102.stdout:5/775: rmdir d2/d25/d75/db9 39 2024-03-26T18:54:40.131 INFO:tasks.workunit.client.0.smithi102.stdout:5/776: write d2/d25/d5b/d76/fac [3497182,46366] 0 2024-03-26T18:54:40.131 INFO:tasks.workunit.client.0.smithi102.stdout:9/528: write df/d1a/d23/f25 [57692,26103] 0 2024-03-26T18:54:40.131 INFO:tasks.workunit.client.0.smithi102.stdout:0/575: write d6/de/d13/d25/d4f/d66/fb7 [5843400,111204] 0 2024-03-26T18:54:40.133 INFO:tasks.workunit.client.1.smithi190.stdout:5/763: creat d2/d17/d1b/db6/d96/ff8 x:0 0 0 2024-03-26T18:54:40.133 INFO:tasks.workunit.client.0.smithi102.stdout:4/427: dwrite d1/d27/f3c [0,4194304] 0 2024-03-26T18:54:40.135 INFO:tasks.workunit.client.0.smithi102.stdout:5/777: rename d2/d25/d75/db9/d43/f8a to d2/d2c/d6f/f110 0 2024-03-26T18:54:40.138 INFO:tasks.workunit.client.0.smithi102.stdout:9/529: link df/d1a/d23/d3e/f99 df/d1a/d81/d88/fc9 0 2024-03-26T18:54:40.140 INFO:tasks.workunit.client.0.smithi102.stdout:6/556: truncate d2/f29 3187299 0 2024-03-26T18:54:40.141 INFO:tasks.workunit.client.0.smithi102.stdout:7/489: sync 2024-03-26T18:54:40.149 INFO:tasks.workunit.client.0.smithi102.stdout:9/530: creat df/fca x:0 0 0 2024-03-26T18:54:40.149 INFO:tasks.workunit.client.0.smithi102.stdout:9/531: chown df/fca 1 1 2024-03-26T18:54:40.153 INFO:tasks.workunit.client.0.smithi102.stdout:6/557: link d2/df/f27 d2/d3d/fb0 0 2024-03-26T18:54:40.156 INFO:tasks.workunit.client.0.smithi102.stdout:0/576: rename d6/de/d13/d1f/d55/d57/c62 to d6/d3e/d69/d8c/cc1 0 2024-03-26T18:54:40.156 INFO:tasks.workunit.client.0.smithi102.stdout:0/577: truncate d6/de/d13/d25/d31/f8e 693213 0 2024-03-26T18:54:40.156 INFO:tasks.workunit.client.0.smithi102.stdout:7/490: dread d7/d10/d67/f80 [0,4194304] 0 2024-03-26T18:54:40.158 INFO:tasks.workunit.client.0.smithi102.stdout:5/778: rename d2/c65 to d2/d25/d75/db9/d43/d9e/d82/dc9/c111 0 2024-03-26T18:54:40.158 INFO:tasks.workunit.client.0.smithi102.stdout:5/779: fsync d2/d25/d5b/d72/f55 0 2024-03-26T18:54:40.160 INFO:tasks.workunit.client.0.smithi102.stdout:0/578: creat d6/de/d93/dac/fc2 x:0 0 0 2024-03-26T18:54:40.162 INFO:tasks.workunit.client.0.smithi102.stdout:7/491: mknod d7/d10/d27/d34/d82/c8c 0 2024-03-26T18:54:40.163 INFO:tasks.workunit.client.1.smithi190.stdout:0/917: creat d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d81/f14c x:0 0 0 2024-03-26T18:54:40.163 INFO:tasks.workunit.client.0.smithi102.stdout:5/780: truncate d2/d25/de6/f10f 317338 0 2024-03-26T18:54:40.164 INFO:tasks.workunit.client.0.smithi102.stdout:0/579: symlink d6/d3e/lc3 0 2024-03-26T18:54:40.164 INFO:tasks.workunit.client.0.smithi102.stdout:0/580: readlink d6/de/d13/d1f/d55/la5 0 2024-03-26T18:54:40.164 INFO:tasks.workunit.client.0.smithi102.stdout:0/581: rename d6/de to d6/de/d93/dac/dc4 22 2024-03-26T18:54:40.165 INFO:tasks.workunit.client.1.smithi190.stdout:5/764: creat d2/d17/d1b/db6/d68/ff9 x:0 0 0 2024-03-26T18:54:40.165 INFO:tasks.workunit.client.1.smithi190.stdout:5/765: truncate d2/d17/d1b/db6/d24/d3b/f46 1101466 0 2024-03-26T18:54:40.166 INFO:tasks.workunit.client.0.smithi102.stdout:7/492: link d7/d12/d51/d6a/f7a d7/d12/f8d 0 2024-03-26T18:54:40.166 INFO:tasks.workunit.client.0.smithi102.stdout:7/493: chown d7/d12/c64 7984182 1 2024-03-26T18:54:40.166 INFO:tasks.workunit.client.0.smithi102.stdout:7/494: chown d7/d31 74456943 1 2024-03-26T18:54:40.166 INFO:tasks.workunit.client.1.smithi190.stdout:5/766: creat d2/d17/d62/dd7/ffa x:0 0 0 2024-03-26T18:54:40.167 INFO:tasks.workunit.client.0.smithi102.stdout:5/781: stat d2/l29 0 2024-03-26T18:54:40.167 INFO:tasks.workunit.client.0.smithi102.stdout:0/582: creat d6/de/d13/d25/fc5 x:0 0 0 2024-03-26T18:54:40.168 INFO:tasks.workunit.client.1.smithi190.stdout:5/767: mkdir d2/d17/d1b/db6/d24/d3b/da7/dfb 0 2024-03-26T18:54:40.168 INFO:tasks.workunit.client.1.smithi190.stdout:5/768: creat d2/d17/d1b/db6/ffc x:0 0 0 2024-03-26T18:54:40.169 INFO:tasks.workunit.client.0.smithi102.stdout:7/495: creat d7/d12/d4b/d83/f8e x:0 0 0 2024-03-26T18:54:40.169 INFO:tasks.workunit.client.0.smithi102.stdout:7/496: chown d7/d10/d17/f1b 1 1 2024-03-26T18:54:40.171 INFO:tasks.workunit.client.0.smithi102.stdout:1/485: dwrite de/d16/d1d/d61/f7c [0,4194304] 0 2024-03-26T18:54:40.172 INFO:tasks.workunit.client.0.smithi102.stdout:0/583: creat d6/d2d/d3d/d80/fc6 x:0 0 0 2024-03-26T18:54:40.172 INFO:tasks.workunit.client.0.smithi102.stdout:1/486: dread de/f8d [0,4194304] 0 2024-03-26T18:54:40.173 INFO:tasks.workunit.client.0.smithi102.stdout:2/477: dwrite d3/d11/f55 [0,4194304] 0 2024-03-26T18:54:40.173 INFO:tasks.workunit.client.0.smithi102.stdout:7/497: mknod d7/d31/d79/c8f 0 2024-03-26T18:54:40.174 INFO:tasks.workunit.client.0.smithi102.stdout:1/487: fsync de/d16/d1d/d2a/f5d 0 2024-03-26T18:54:40.175 INFO:tasks.workunit.client.0.smithi102.stdout:7/498: mknod d7/d31/d56/c90 0 2024-03-26T18:54:40.176 INFO:tasks.workunit.client.0.smithi102.stdout:1/488: mknod de/d16/d1d/d2a/d2c/d43/ca1 0 2024-03-26T18:54:40.176 INFO:tasks.workunit.client.0.smithi102.stdout:1/489: chown de/d1f/f53 9908 1 2024-03-26T18:54:40.176 INFO:tasks.workunit.client.0.smithi102.stdout:1/490: write de/d16/d1d/d2a/d2c/d56/d78/d80/f85 [939391,12321] 0 2024-03-26T18:54:40.176 INFO:tasks.workunit.client.0.smithi102.stdout:4/428: dwrite d1/f9 [0,4194304] 0 2024-03-26T18:54:40.177 INFO:tasks.workunit.client.0.smithi102.stdout:1/491: creat de/d16/d1d/d2a/d2c/d56/d5f/fa2 x:0 0 0 2024-03-26T18:54:40.179 INFO:tasks.workunit.client.0.smithi102.stdout:5/782: rename d2/d2c/d6f to d2/d25/d75/def/dec/d112 0 2024-03-26T18:54:40.179 INFO:tasks.workunit.client.0.smithi102.stdout:5/783: creat d2/d14/f113 x:0 0 0 2024-03-26T18:54:40.180 INFO:tasks.workunit.client.1.smithi190.stdout:0/918: write d0/dd/d20/d61/f9c [3999509,72357] 0 2024-03-26T18:54:40.181 INFO:tasks.workunit.client.0.smithi102.stdout:4/429: mkdir d1/d75/d85 0 2024-03-26T18:54:40.181 INFO:tasks.workunit.client.0.smithi102.stdout:4/430: readlink d1/l2f 0 2024-03-26T18:54:40.182 INFO:tasks.workunit.client.0.smithi102.stdout:1/492: link de/d16/d1d/d2a/f5d de/d16/d1d/d61/d6a/fa3 0 2024-03-26T18:54:40.183 INFO:tasks.workunit.client.0.smithi102.stdout:1/493: write de/d16/d1d/d2a/f9e [161565,128984] 0 2024-03-26T18:54:40.186 INFO:tasks.workunit.client.1.smithi190.stdout:0/919: mkdir d0/dd5/d106/d14d 0 2024-03-26T18:54:40.186 INFO:tasks.workunit.client.1.smithi190.stdout:0/920: stat d0/dd/d20/d1c/d46/da7/df0 0 2024-03-26T18:54:40.186 INFO:tasks.workunit.client.1.smithi190.stdout:0/921: dread - d0/dd/d20/d1c/d46/da7/df0/da2/f112 zero size 2024-03-26T18:54:40.186 INFO:tasks.workunit.client.0.smithi102.stdout:2/478: dread d3/d11/d27/f6b [0,4194304] 0 2024-03-26T18:54:40.189 INFO:tasks.workunit.client.1.smithi190.stdout:0/922: unlink d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/lc9 0 2024-03-26T18:54:40.189 INFO:tasks.workunit.client.0.smithi102.stdout:5/784: truncate d2/d25/d5b/d76/fa3 3734660 0 2024-03-26T18:54:40.190 INFO:tasks.workunit.client.0.smithi102.stdout:4/431: creat d1/d25/d76/f86 x:0 0 0 2024-03-26T18:54:40.190 INFO:tasks.workunit.client.0.smithi102.stdout:4/432: readlink d1/l1e 0 2024-03-26T18:54:40.191 INFO:tasks.workunit.client.1.smithi190.stdout:6/739: sync 2024-03-26T18:54:40.191 INFO:tasks.workunit.client.1.smithi190.stdout:4/946: sync 2024-03-26T18:54:40.191 INFO:tasks.workunit.client.1.smithi190.stdout:8/966: sync 2024-03-26T18:54:40.191 INFO:tasks.workunit.client.1.smithi190.stdout:1/735: sync 2024-03-26T18:54:40.191 INFO:tasks.workunit.client.1.smithi190.stdout:2/898: sync 2024-03-26T18:54:40.191 INFO:tasks.workunit.client.1.smithi190.stdout:1/736: readlink d3/da/l43 0 2024-03-26T18:54:40.191 INFO:tasks.workunit.client.1.smithi190.stdout:8/967: chown d9/d48/d10c/d140 792 1 2024-03-26T18:54:40.191 INFO:tasks.workunit.client.1.smithi190.stdout:2/899: write d6/d9/d3a/d3d/d55/fc6 [993442,49892] 0 2024-03-26T18:54:40.192 INFO:tasks.workunit.client.0.smithi102.stdout:1/494: unlink de/d16/d1d/d2a/f4c 0 2024-03-26T18:54:40.193 INFO:tasks.workunit.client.0.smithi102.stdout:1/495: chown de/f3f 10841 1 2024-03-26T18:54:40.193 INFO:tasks.workunit.client.1.smithi190.stdout:4/947: mkdir d7/d8/d14/d4b/d7d/da1/df8/d142 0 2024-03-26T18:54:40.193 INFO:tasks.workunit.client.1.smithi190.stdout:2/900: symlink d6/d9/d10/d5f/l137 0 2024-03-26T18:54:40.194 INFO:tasks.workunit.client.0.smithi102.stdout:4/433: dread d1/d17/d1a/d21/f63 [0,4194304] 0 2024-03-26T18:54:40.194 INFO:tasks.workunit.client.1.smithi190.stdout:4/948: stat d7/d1d/d42/d4a/d139/l105 0 2024-03-26T18:54:40.195 INFO:tasks.workunit.client.1.smithi190.stdout:2/901: unlink d6/d9/d3a/le9 0 2024-03-26T18:54:40.196 INFO:tasks.workunit.client.1.smithi190.stdout:4/949: rmdir d7/d8/d14/d23/d28/d38/d96 39 2024-03-26T18:54:40.196 INFO:tasks.workunit.client.1.smithi190.stdout:6/740: creat d17/ffa x:0 0 0 2024-03-26T18:54:40.197 INFO:tasks.workunit.client.1.smithi190.stdout:2/902: truncate d6/d9/d3a/d3d/d55/f116 529009 0 2024-03-26T18:54:40.197 INFO:tasks.workunit.client.1.smithi190.stdout:2/903: chown d6/d21/dd0/cc1 7390061 1 2024-03-26T18:54:40.199 INFO:tasks.workunit.client.1.smithi190.stdout:4/950: rmdir d7/d8/d14/d6e/df0 39 2024-03-26T18:54:40.199 INFO:tasks.workunit.client.1.smithi190.stdout:4/951: chown d7/d8/d14/f52 0 1 2024-03-26T18:54:40.199 INFO:tasks.workunit.client.1.smithi190.stdout:4/952: chown d7/d8/d14/d23/feb 1 1 2024-03-26T18:54:40.200 INFO:tasks.workunit.client.0.smithi102.stdout:2/479: link d3/d82/d93/d9e/f1b d3/d2a/d5f/fa8 0 2024-03-26T18:54:40.200 INFO:tasks.workunit.client.1.smithi190.stdout:2/904: dread d6/d9/d3a/d3d/d1c/d8d/dc7/fba [0,4194304] 0 2024-03-26T18:54:40.201 INFO:tasks.workunit.client.0.smithi102.stdout:0/584: rename d6/de/d13/d1f/c2a to d6/de/d13/d9a/d85/cc7 0 2024-03-26T18:54:40.202 INFO:tasks.workunit.client.0.smithi102.stdout:5/785: creat d2/dea/d4f/f114 x:0 0 0 2024-03-26T18:54:40.203 INFO:tasks.workunit.client.0.smithi102.stdout:5/786: readlink d2/d2c/l83 0 2024-03-26T18:54:40.203 INFO:tasks.workunit.client.0.smithi102.stdout:5/787: readlink d2/l29 0 2024-03-26T18:54:40.203 INFO:tasks.workunit.client.1.smithi190.stdout:8/968: dread d9/dd/d15/f30 [0,4194304] 0 2024-03-26T18:54:40.204 INFO:tasks.workunit.client.0.smithi102.stdout:1/496: mknod de/d16/d1d/d2a/d2c/d56/d78/d80/ca4 0 2024-03-26T18:54:40.206 INFO:tasks.workunit.client.1.smithi190.stdout:8/969: mkdir d9/da2/d73/d108/d12b/d37/db8/d116/d144 0 2024-03-26T18:54:40.206 INFO:tasks.workunit.client.1.smithi190.stdout:8/970: chown d9/d39/d79/dad/c11a 1050328663 1 2024-03-26T18:54:40.206 INFO:tasks.workunit.client.1.smithi190.stdout:8/971: write d9/da2/d73/d108/d12b/d37/d3a/d61/fb6 [4442314,46407] 0 2024-03-26T18:54:40.206 INFO:tasks.workunit.client.1.smithi190.stdout:8/972: chown d9/da2/d73/d108/d12b/d37/d3a/d61/daf/db3/cd0 0 1 2024-03-26T18:54:40.207 INFO:tasks.workunit.client.0.smithi102.stdout:2/480: creat d3/d11/d15/d46/fa9 x:0 0 0 2024-03-26T18:54:40.207 INFO:tasks.workunit.client.1.smithi190.stdout:3/731: sync 2024-03-26T18:54:40.209 INFO:tasks.workunit.client.1.smithi190.stdout:8/973: link d9/d39/f82 d9/da2/d73/d108/d12b/d37/d3a/d43/d8f/dd2/f145 0 2024-03-26T18:54:40.209 INFO:tasks.workunit.client.1.smithi190.stdout:4/953: dread d7/d8/d14/d4b/f4c [0,4194304] 0 2024-03-26T18:54:40.210 INFO:tasks.workunit.client.0.smithi102.stdout:1/497: mkdir de/d16/d1d/d61/d6a/da5 0 2024-03-26T18:54:40.211 INFO:tasks.workunit.client.1.smithi190.stdout:4/954: mknod d7/d8/d14/d23/d28/d124/c143 0 2024-03-26T18:54:40.211 INFO:tasks.workunit.client.1.smithi190.stdout:4/955: chown d7/d8/d14/d23/d28/d38/ff4 963 1 2024-03-26T18:54:40.212 INFO:tasks.workunit.client.0.smithi102.stdout:2/481: dread d3/d11/d15/d31/f60 [4194304,4194304] 0 2024-03-26T18:54:40.214 INFO:tasks.workunit.client.0.smithi102.stdout:0/585: write d6/de/d13/d25/d31/f50 [3354518,66547] 0 2024-03-26T18:54:40.214 INFO:tasks.workunit.client.0.smithi102.stdout:0/586: fsync d6/de/d13/d25/d31/d45/d82/d84/d88/fab 0 2024-03-26T18:54:40.215 INFO:tasks.workunit.client.0.smithi102.stdout:2/482: mknod d3/d11/d15/d31/d7f/caa 0 2024-03-26T18:54:40.216 INFO:tasks.workunit.client.0.smithi102.stdout:2/483: creat d3/d20/fab x:0 0 0 2024-03-26T18:54:40.217 INFO:tasks.workunit.client.1.smithi190.stdout:4/956: symlink d7/l144 0 2024-03-26T18:54:40.217 INFO:tasks.workunit.client.1.smithi190.stdout:4/957: readlink d7/d1d/d42/d4a/l8f 0 2024-03-26T18:54:40.217 INFO:tasks.workunit.client.1.smithi190.stdout:4/958: dread - d7/d8/d14/d6e/df0/ffc zero size 2024-03-26T18:54:40.217 INFO:tasks.workunit.client.1.smithi190.stdout:4/959: write d7/d1d/d42/d4a/d139/de4/def/ff9 [412875,27470] 0 2024-03-26T18:54:40.217 INFO:tasks.workunit.client.1.smithi190.stdout:4/960: creat d7/d8/d14/d6e/f145 x:0 0 0 2024-03-26T18:54:40.217 INFO:tasks.workunit.client.1.smithi190.stdout:4/961: fdatasync d7/d1d/d42/d4a/fa6 0 2024-03-26T18:54:40.217 INFO:tasks.workunit.client.1.smithi190.stdout:4/962: chown d7/d1d/d41/d74/f109 5214564 1 2024-03-26T18:54:40.217 INFO:tasks.workunit.client.0.smithi102.stdout:5/788: rename d2/dea/d4f/ddc to d2/d25/d75/db9/db3/d115 0 2024-03-26T18:54:40.223 INFO:tasks.workunit.client.1.smithi190.stdout:8/974: write d9/da2/dc4/f35 [1242423,90892] 0 2024-03-26T18:54:40.224 INFO:tasks.workunit.client.1.smithi190.stdout:8/975: creat d9/da2/d73/d108/d12b/d37/d3a/d43/d8f/f146 x:0 0 0 2024-03-26T18:54:40.224 INFO:tasks.workunit.client.1.smithi190.stdout:8/976: write d9/da2/dc4/ff3 [3423785,76611] 0 2024-03-26T18:54:40.225 INFO:tasks.workunit.client.1.smithi190.stdout:8/977: getdents d9/d39/d79/dad 0 2024-03-26T18:54:40.225 INFO:tasks.workunit.client.1.smithi190.stdout:8/978: chown d9/da2/dc4/f53 187 1 2024-03-26T18:54:40.225 INFO:tasks.workunit.client.1.smithi190.stdout:8/979: chown d9/da2/d73/d108/d12b/d37/d3a/d61/daf/db3/db5/ff0 18098326 1 2024-03-26T18:54:40.228 INFO:tasks.workunit.client.1.smithi190.stdout:2/905: dwrite d6/d9/d10/f8c [0,4194304] 0 2024-03-26T18:54:40.230 INFO:tasks.workunit.client.1.smithi190.stdout:2/906: truncate d6/d9/d45/d6b/d93/dc0/fe3 1471189 0 2024-03-26T18:54:40.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/789: dread d2/f1d [0,4194304] 0 2024-03-26T18:54:40.230 INFO:tasks.workunit.client.0.smithi102.stdout:5/790: chown d2/d25/d75/def/dec/d112 53 1 2024-03-26T18:54:40.231 INFO:tasks.workunit.client.1.smithi190.stdout:2/907: truncate d6/d9/d45/f74 2897190 0 2024-03-26T18:54:40.233 INFO:tasks.workunit.client.1.smithi190.stdout:2/908: creat d6/d9/d3a/d3d/d2e/d86/f138 x:0 0 0 2024-03-26T18:54:40.250 INFO:tasks.workunit.client.1.smithi190.stdout:1/737: dread d3/d5f/f71 [0,4194304] 0 2024-03-26T18:54:40.279 INFO:tasks.workunit.client.1.smithi190.stdout:2/909: dwrite d6/d9/d3a/d3d/d1c/dfe/f12b [0,4194304] 0 2024-03-26T18:54:40.285 INFO:tasks.workunit.client.1.smithi190.stdout:6/741: dwrite d17/d28/d2c/d77/fac [0,4194304] 0 2024-03-26T18:54:40.285 INFO:tasks.workunit.client.1.smithi190.stdout:0/923: dwrite d0/dd/fe [0,4194304] 0 2024-03-26T18:54:40.286 INFO:tasks.workunit.client.1.smithi190.stdout:3/732: dwrite d1/de/f73 [0,4194304] 0 2024-03-26T18:54:40.287 INFO:tasks.workunit.client.1.smithi190.stdout:3/733: fdatasync d1/d3/df/d31/dc4/fc5 0 2024-03-26T18:54:40.287 INFO:tasks.workunit.client.1.smithi190.stdout:4/963: dwrite d7/d8/f22 [0,4194304] 0 2024-03-26T18:54:40.290 INFO:tasks.workunit.client.0.smithi102.stdout:4/434: dwrite d1/d25/d39/f66 [0,4194304] 0 2024-03-26T18:54:40.296 INFO:tasks.workunit.client.1.smithi190.stdout:6/742: write d17/d28/d2c/d63/f70 [3161653,70791] 0 2024-03-26T18:54:40.301 INFO:tasks.workunit.client.1.smithi190.stdout:4/964: dread d7/d1d/f26 [0,4194304] 0 2024-03-26T18:54:40.301 INFO:tasks.workunit.client.1.smithi190.stdout:2/910: dwrite d6/d9/d45/d6b/deb/f131 [0,4194304] 0 2024-03-26T18:54:40.301 INFO:tasks.workunit.client.1.smithi190.stdout:2/911: truncate d6/d9/d3a/d3d/f57 897066 0 2024-03-26T18:54:40.306 INFO:tasks.workunit.client.0.smithi102.stdout:0/587: mkdir d6/de/d13/d1f/d55/d57/d7d/d7f/dc8 0 2024-03-26T18:54:40.309 INFO:tasks.workunit.client.1.smithi190.stdout:1/738: creat d3/da/dd/d2f/d5b/d83/feb x:0 0 0 2024-03-26T18:54:40.310 INFO:tasks.workunit.client.0.smithi102.stdout:0/588: dread d6/de/d13/d25/d31/d45/d82/f8a [0,4194304] 0 2024-03-26T18:54:40.316 INFO:tasks.workunit.client.1.smithi190.stdout:2/912: dread d6/d9/d3a/f83 [0,4194304] 0 2024-03-26T18:54:40.317 INFO:tasks.workunit.client.0.smithi102.stdout:5/791: mkdir d2/d116 0 2024-03-26T18:54:40.317 INFO:tasks.workunit.client.0.smithi102.stdout:5/792: fdatasync d2/d25/ffe 0 2024-03-26T18:54:40.318 INFO:tasks.workunit.client.1.smithi190.stdout:2/913: mknod d6/d21/d61/d6c/c139 0 2024-03-26T18:54:40.322 INFO:tasks.workunit.client.1.smithi190.stdout:4/965: dwrite d7/d8/d14/d23/feb [8388608,4194304] 0 2024-03-26T18:54:40.324 INFO:tasks.workunit.client.1.smithi190.stdout:4/966: link d7/d8/d14/d6e/df0/c121 d7/d8/d14/d23/d28/d38/d96/d9a/c146 0 2024-03-26T18:54:40.324 INFO:tasks.workunit.client.1.smithi190.stdout:4/967: fdatasync d7/d8/d14/d23/d28/d38/d96/d10f/f58 0 2024-03-26T18:54:40.324 INFO:tasks.workunit.client.1.smithi190.stdout:4/968: fdatasync d7/d8/d14/d5c/f97 0 2024-03-26T18:54:40.324 INFO:tasks.workunit.client.1.smithi190.stdout:4/969: fdatasync d7/d1d/d5e/f76 0 2024-03-26T18:54:40.342 INFO:tasks.workunit.client.1.smithi190.stdout:0/924: mkdir d0/dd/d128/d14e 0 2024-03-26T18:54:40.357 INFO:tasks.workunit.client.0.smithi102.stdout:4/435: creat d1/d6c/f87 x:0 0 0 2024-03-26T18:54:40.375 INFO:tasks.workunit.client.1.smithi190.stdout:1/739: symlink d3/da/d2e/d59/lec 0 2024-03-26T18:54:40.377 INFO:tasks.workunit.client.1.smithi190.stdout:0/925: creat d0/dd5/dee/df5/f14f x:0 0 0 2024-03-26T18:54:40.377 INFO:tasks.workunit.client.1.smithi190.stdout:0/926: truncate d0/dd5/dee/f101 2120615 0 2024-03-26T18:54:40.379 INFO:tasks.workunit.client.1.smithi190.stdout:1/740: creat d3/da/dd/d2f/fed x:0 0 0 2024-03-26T18:54:40.379 INFO:tasks.workunit.client.1.smithi190.stdout:1/741: fsync d3/d9/f57 0 2024-03-26T18:54:40.381 INFO:tasks.workunit.client.1.smithi190.stdout:0/927: mkdir d0/dd/d20/d1c/d46/da7/df0/d87/d150 0 2024-03-26T18:54:40.383 INFO:tasks.workunit.client.1.smithi190.stdout:1/742: mkdir d3/d9/db/dd4/dee 0 2024-03-26T18:54:40.383 INFO:tasks.workunit.client.1.smithi190.stdout:1/743: write d3/da/dd/d2f/d5b/d83/feb [808483,18064] 0 2024-03-26T18:54:40.384 INFO:tasks.workunit.client.1.smithi190.stdout:0/928: mknod d0/dd/d20/d1c/d46/da7/df0/d4b/c151 0 2024-03-26T18:54:40.389 INFO:tasks.workunit.client.1.smithi190.stdout:1/744: mknod d3/da/dd/d2f/d5b/db5/cef 0 2024-03-26T18:54:40.389 INFO:tasks.workunit.client.1.smithi190.stdout:1/745: fdatasync d3/d9/f20 0 2024-03-26T18:54:40.390 INFO:tasks.workunit.client.1.smithi190.stdout:0/929: stat d0/dd/d20/d1c/d46/d58/lbc 0 2024-03-26T18:54:40.393 INFO:tasks.workunit.client.1.smithi190.stdout:1/746: creat d3/da/dd/d95/ff0 x:0 0 0 2024-03-26T18:54:40.395 INFO:tasks.workunit.client.1.smithi190.stdout:0/930: unlink d0/dd/d20/d1c/d46/da7/df0/da2/d80/cde 0 2024-03-26T18:54:40.396 INFO:tasks.workunit.client.1.smithi190.stdout:0/931: read d0/dd/d20/d1c/d46/da7/df0/f2b [2641568,129923] 0 2024-03-26T18:54:40.398 INFO:tasks.workunit.client.1.smithi190.stdout:0/932: link d0/dd/d20/d1c/d8b/c98 d0/dd/d20/d1c/d4e/c152 0 2024-03-26T18:54:40.405 INFO:tasks.workunit.client.1.smithi190.stdout:1/747: dread d3/da/dd/d2f/f5e [4194304,4194304] 0 2024-03-26T18:54:40.405 INFO:tasks.workunit.client.1.smithi190.stdout:1/748: chown d3/da/dd/fb6 0 1 2024-03-26T18:54:40.406 INFO:tasks.workunit.client.1.smithi190.stdout:1/749: write d3/da/dd/d95/d52/f5a [661272,99130] 0 2024-03-26T18:54:40.406 INFO:tasks.workunit.client.1.smithi190.stdout:1/750: dread d3/da/f94 [0,4194304] 0 2024-03-26T18:54:40.410 INFO:tasks.workunit.client.1.smithi190.stdout:1/751: rmdir d3/da/d2e/d59 39 2024-03-26T18:54:40.410 INFO:tasks.workunit.client.1.smithi190.stdout:1/752: stat d3/da/dd/d95/dda/fdc 0 2024-03-26T18:54:40.414 INFO:tasks.workunit.client.1.smithi190.stdout:1/753: truncate d3/da/f1f 422146 0 2024-03-26T18:54:40.419 INFO:tasks.workunit.client.0.smithi102.stdout:2/484: dwrite d3/d11/d15/f2e [0,4194304] 0 2024-03-26T18:54:40.419 INFO:tasks.workunit.client.0.smithi102.stdout:2/485: write d3/d20/f30 [2037177,58813] 0 2024-03-26T18:54:40.419 INFO:tasks.workunit.client.0.smithi102.stdout:2/486: creat d3/d20/fac x:0 0 0 2024-03-26T18:54:40.420 INFO:tasks.workunit.client.0.smithi102.stdout:0/589: creat d6/dbc/fc9 x:0 0 0 2024-03-26T18:54:40.422 INFO:tasks.workunit.client.1.smithi190.stdout:0/933: dwrite d0/dd/d20/d61/f7b [4194304,4194304] 0 2024-03-26T18:54:40.422 INFO:tasks.workunit.client.1.smithi190.stdout:0/934: creat d0/dd/d20/d1c/d46/da7/df0/da2/d80/f153 x:0 0 0 2024-03-26T18:54:40.428 INFO:tasks.workunit.client.0.smithi102.stdout:8/584: sync 2024-03-26T18:54:40.432 INFO:tasks.workunit.client.1.smithi190.stdout:0/935: rmdir d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/d121 39 2024-03-26T18:54:40.436 INFO:tasks.workunit.client.0.smithi102.stdout:2/487: creat d3/d11/d15/d46/d7b/d97/fad x:0 0 0 2024-03-26T18:54:40.436 INFO:tasks.workunit.client.0.smithi102.stdout:2/488: write d3/d2a/d5f/f90 [54907,41013] 0 2024-03-26T18:54:40.437 INFO:tasks.workunit.client.0.smithi102.stdout:5/793: dread d2/d14/d1a/f1b [0,4194304] 0 2024-03-26T18:54:40.438 INFO:tasks.workunit.client.0.smithi102.stdout:2/489: mkdir d3/d2a/dae 0 2024-03-26T18:54:40.439 INFO:tasks.workunit.client.0.smithi102.stdout:2/490: rmdir d3/d82/d93/d9e/d1a 39 2024-03-26T18:54:40.440 INFO:tasks.workunit.client.0.smithi102.stdout:2/491: creat d3/d2a/faf x:0 0 0 2024-03-26T18:54:40.440 INFO:tasks.workunit.client.1.smithi190.stdout:0/936: write d0/dd/d20/d1c/f6e [6051541,49590] 0 2024-03-26T18:54:40.443 INFO:tasks.workunit.client.1.smithi190.stdout:0/937: creat d0/dd/d20/d1c/d8b/d8d/db7/d149/f154 x:0 0 0 2024-03-26T18:54:40.443 INFO:tasks.workunit.client.1.smithi190.stdout:0/938: write d0/dd/d20/d1c/d46/da7/df0/d87/d93/dc4/dd8/f115 [750370,38060] 0 2024-03-26T18:54:40.444 INFO:tasks.workunit.client.1.smithi190.stdout:0/939: chown d0/dd/d20/d1c/d4e/c152 192 1 2024-03-26T18:54:40.445 INFO:tasks.workunit.client.1.smithi190.stdout:0/940: write d0/dd5/dee/f101 [659316,112110] 0 2024-03-26T18:54:40.445 INFO:tasks.workunit.client.1.smithi190.stdout:0/941: stat d0/dd/d20/d1c/d8b/d8d/db7 0 2024-03-26T18:54:40.445 INFO:tasks.workunit.client.1.smithi190.stdout:0/942: fdatasync d0/dd/d20/d1c/f29 0 2024-03-26T18:54:40.445 INFO:tasks.workunit.client.1.smithi190.stdout:0/943: fdatasync d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/f13c 0 2024-03-26T18:54:40.445 INFO:tasks.workunit.client.0.smithi102.stdout:0/590: dwrite d6/d2d/d3d/fa0 [0,4194304] 0 2024-03-26T18:54:40.446 INFO:tasks.workunit.client.0.smithi102.stdout:4/436: creat d1/d17/f88 x:0 0 0 2024-03-26T18:54:40.450 INFO:tasks.workunit.client.0.smithi102.stdout:5/794: mkdir d2/d25/d75/db9/d43/d117 0 2024-03-26T18:54:40.451 INFO:tasks.workunit.client.1.smithi190.stdout:6/743: rename d17/d28/d46 to d17/d28/df6/da8/df4/dfb 0 2024-03-26T18:54:40.454 INFO:tasks.workunit.client.1.smithi190.stdout:4/970: rename d7/d8/d14/dc2 to d7/d8/d147 0 2024-03-26T18:54:40.455 INFO:tasks.workunit.client.1.smithi190.stdout:4/971: fdatasync d7/d1d/fa7 0 2024-03-26T18:54:40.455 INFO:tasks.workunit.client.1.smithi190.stdout:4/972: creat d7/d8/d14/d4b/d7d/da1/f148 x:0 0 0 2024-03-26T18:54:40.455 INFO:tasks.workunit.client.0.smithi102.stdout:5/795: dread d2/dea/d4f/f96 [0,4194304] 0 2024-03-26T18:54:40.455 INFO:tasks.workunit.client.0.smithi102.stdout:8/585: dwrite d0/d15/fb0 [0,4194304] 0 2024-03-26T18:54:40.456 INFO:tasks.workunit.client.0.smithi102.stdout:0/591: mknod d6/d3e/cca 0 2024-03-26T18:54:40.456 INFO:tasks.workunit.client.0.smithi102.stdout:8/586: read - d0/d9/fb5 zero size 2024-03-26T18:54:40.456 INFO:tasks.workunit.client.0.smithi102.stdout:0/592: truncate d6/de/d13/d25/d31/d45/d82/f8a 970762 0 2024-03-26T18:54:40.456 INFO:tasks.workunit.client.0.smithi102.stdout:4/437: symlink d1/d17/d1f/d5d/d6f/l89 0 2024-03-26T18:54:40.458 INFO:tasks.workunit.client.1.smithi190.stdout:4/973: truncate d7/f4f 9284 0 2024-03-26T18:54:40.458 INFO:tasks.workunit.client.1.smithi190.stdout:4/974: fdatasync d7/d8/d14/d4b/d7d/da1/f148 0 2024-03-26T18:54:40.458 INFO:tasks.workunit.client.1.smithi190.stdout:1/754: rename d3/d9/d9f to d3/d73/d88/df1 0 2024-03-26T18:54:40.459 INFO:tasks.workunit.client.0.smithi102.stdout:5/796: unlink d2/d14/la5 0 2024-03-26T18:54:40.459 INFO:tasks.workunit.client.0.smithi102.stdout:5/797: write d2/d14/ff6 [3624940,112097] 0 2024-03-26T18:54:40.462 INFO:tasks.workunit.client.1.smithi190.stdout:4/975: link d7/d8/d14/d23/d28/d9e/fe7 d7/d8/d14/d4b/d7d/da1/df8/d142/f149 0 2024-03-26T18:54:40.462 INFO:tasks.workunit.client.0.smithi102.stdout:2/492: dwrite d3/d20/fa1 [0,4194304] 0 2024-03-26T18:54:40.463 INFO:tasks.workunit.client.1.smithi190.stdout:1/755: unlink d3/da/dd/d2f/d33/c44 0 2024-03-26T18:54:40.465 INFO:tasks.workunit.client.1.smithi190.stdout:4/976: dread d7/d1d/d41/d74/f109 [0,4194304] 0 2024-03-26T18:54:40.466 INFO:tasks.workunit.client.0.smithi102.stdout:0/593: mkdir d6/dcb 0 2024-03-26T18:54:40.469 INFO:tasks.workunit.client.0.smithi102.stdout:8/587: symlink d0/d3/d26/d19/d76/lc1 0 2024-03-26T18:54:40.469 INFO:tasks.workunit.client.1.smithi190.stdout:4/977: rename d7/d8/d14/f5b to d7/d8/d14/d4b/d7d/da1/d113/f14a 0 2024-03-26T18:54:40.471 INFO:tasks.workunit.client.1.smithi190.stdout:4/978: rename d7/d1d/l92 to d7/d8/d14/d6e/l14b 0 2024-03-26T18:54:40.471 INFO:tasks.workunit.client.1.smithi190.stdout:4/979: chown d7/d8/d14/f19 5 1 2024-03-26T18:54:40.472 INFO:tasks.workunit.client.0.smithi102.stdout:5/798: mknod d2/dea/dbb/dcf/c118 0 2024-03-26T18:54:40.473 INFO:tasks.workunit.client.1.smithi190.stdout:4/980: getdents d7/d8/d14/d23/d28/d9e 0 2024-03-26T18:54:40.473 INFO:tasks.workunit.client.1.smithi190.stdout:4/981: dread - d7/d1d/d41/f86 zero size 2024-03-26T18:54:40.475 INFO:tasks.workunit.client.1.smithi190.stdout:4/982: link d7/d8/d14/c16 d7/d8/d14/d23/d28/c14c 0 2024-03-26T18:54:40.476 INFO:tasks.workunit.client.0.smithi102.stdout:2/493: dread d3/d11/d15/f2e [0,4194304] 0 2024-03-26T18:54:40.476 INFO:tasks.workunit.client.0.smithi102.stdout:0/594: truncate d6/f51 71519 0 2024-03-26T18:54:40.476 INFO:tasks.workunit.client.0.smithi102.stdout:8/588: dread d0/d3/d2c/da7/f45 [0,4194304] 0 2024-03-26T18:54:40.476 INFO:tasks.workunit.client.0.smithi102.stdout:8/589: fdatasync d0/d3/d2c/f8e 0 2024-03-26T18:54:40.477 INFO:tasks.workunit.client.0.smithi102.stdout:5/799: rename d2/d25/f26 to d2/dea/dbb/dcf/f119 0 2024-03-26T18:54:40.477 INFO:tasks.workunit.client.1.smithi190.stdout:6/744: dwrite d17/d28/d8b/d95/dda/fec [0,4194304] 0 2024-03-26T18:54:40.480 INFO:tasks.workunit.client.0.smithi102.stdout:2/494: creat d3/d82/d93/d9e/da0/fb0 x:0 0 0 2024-03-26T18:54:40.481 INFO:tasks.workunit.client.0.smithi102.stdout:4/438: dwrite d1/f8 [4194304,4194304] 0 2024-03-26T18:54:40.484 INFO:tasks.workunit.client.0.smithi102.stdout:8/590: rename d0/d3/d2c/da7/f93 to d0/d15/d5d/dac/fc2 0 2024-03-26T18:54:40.489 INFO:tasks.workunit.client.0.smithi102.stdout:2/495: link d3/d11/fa6 d3/d20/d78/d84/d6e/fb1 0 2024-03-26T18:54:40.489 INFO:tasks.workunit.client.0.smithi102.stdout:2/496: readlink d3/d82/d93/d9e/ld 0 2024-03-26T18:54:40.489 INFO:tasks.workunit.client.0.smithi102.stdout:2/497: write d3/d11/d15/d31/d39/f63 [56820,39363] 0 2024-03-26T18:54:40.489 INFO:tasks.workunit.client.0.smithi102.stdout:2/498: chown d3/d20/fab 44644855 1 2024-03-26T18:54:40.494 INFO:tasks.workunit.client.0.smithi102.stdout:8/591: dread d0/d3/d2c/d2f/f78 [0,4194304] 0 2024-03-26T18:54:40.494 INFO:tasks.workunit.client.0.smithi102.stdout:8/592: read d0/d3/d26/d19/f82 [341891,72784] 0 2024-03-26T18:54:40.495 INFO:tasks.workunit.client.0.smithi102.stdout:2/499: symlink d3/d2d/d67/lb2 0 2024-03-26T18:54:40.495 INFO:tasks.workunit.client.0.smithi102.stdout:2/500: write d3/f19 [5484474,14913] 0 2024-03-26T18:54:40.497 INFO:tasks.workunit.client.0.smithi102.stdout:8/593: rename d0/d3/d2c/d58/d75/f9d to d0/d15/d5d/dac/fc3 0 2024-03-26T18:54:40.499 INFO:tasks.workunit.client.0.smithi102.stdout:2/501: creat d3/d82/fb3 x:0 0 0 2024-03-26T18:54:40.501 INFO:tasks.workunit.client.0.smithi102.stdout:2/502: unlink d3/d11/d15/d46/c71 0 2024-03-26T18:54:40.502 INFO:tasks.workunit.client.0.smithi102.stdout:2/503: mknod d3/d11/d27/cb4 0 2024-03-26T18:54:40.506 INFO:tasks.workunit.client.0.smithi102.stdout:2/504: link d3/d2d/l59 d3/d82/d93/d9e/d1a/d25/lb5 0 2024-03-26T18:54:40.506 INFO:tasks.workunit.client.0.smithi102.stdout:2/505: write d3/d82/d93/d9e/d1a/d25/f9d [745332,80892] 0 2024-03-26T18:54:40.508 INFO:tasks.workunit.client.0.smithi102.stdout:2/506: link d3/d11/d15/d31/d7f/caa d3/d11/d15/d31/cb6 0 2024-03-26T18:54:40.508 INFO:tasks.workunit.client.0.smithi102.stdout:2/507: truncate d3/d11/d15/d46/d7b/d97/fad 541461 0 2024-03-26T18:54:40.508 INFO:tasks.workunit.client.0.smithi102.stdout:2/508: fsync d3/d20/f80 0 2024-03-26T18:54:40.508 INFO:tasks.workunit.client.0.smithi102.stdout:2/509: creat d3/d2a/fb7 x:0 0 0 2024-03-26T18:54:40.510 INFO:tasks.workunit.client.0.smithi102.stdout:2/510: rename d3/d82/d93/d9e/f1b to d3/d2d/d69/fb8 0 2024-03-26T18:54:40.510 INFO:tasks.workunit.client.0.smithi102.stdout:2/511: stat d3 0 2024-03-26T18:54:40.512 INFO:tasks.workunit.client.1.smithi190.stdout:3/734: unlink d1/d3/df/fbb 0 2024-03-26T18:54:40.512 INFO:tasks.workunit.client.1.smithi190.stdout:3/735: fsync d1/de/d8a/fc2 0 2024-03-26T18:54:40.512 INFO:tasks.workunit.client.1.smithi190.stdout:3/736: fdatasync d1/f15 0 2024-03-26T18:54:40.512 INFO:tasks.workunit.client.1.smithi190.stdout:3/737: chown d1/d3/df/d18/d35/d89/c9a 9 1 2024-03-26T18:54:40.513 INFO:tasks.workunit.client.1.smithi190.stdout:3/738: write d1/de/f69 [2064625,53220] 0 2024-03-26T18:54:40.514 INFO:tasks.workunit.client.1.smithi190.stdout:3/739: mkdir d1/de/d8a/de1 0 2024-03-26T18:54:40.516 INFO:tasks.workunit.client.1.smithi190.stdout:0/944: creat d0/dd/d20/d1c/d46/da7/df0/f155 x:0 0 0 2024-03-26T18:54:40.517 INFO:tasks.workunit.client.0.smithi102.stdout:5/800: dwrite d2/d25/ffe [0,4194304] 0 2024-03-26T18:54:40.517 INFO:tasks.workunit.client.0.smithi102.stdout:0/595: dwrite d6/dbc/fc9 [0,4194304] 0 2024-03-26T18:54:40.517 INFO:tasks.workunit.client.0.smithi102.stdout:5/801: write d2/d25/d75/db9/db3/fd9 [288794,115517] 0 2024-03-26T18:54:40.527 INFO:tasks.workunit.client.0.smithi102.stdout:2/512: dread d3/f8 [0,4194304] 0 2024-03-26T18:54:40.527 INFO:tasks.workunit.client.0.smithi102.stdout:8/594: dwrite d0/d3c/d6f/f95 [0,4194304] 0 2024-03-26T18:54:40.529 INFO:tasks.workunit.client.0.smithi102.stdout:2/513: dread d3/d2d/f89 [0,4194304] 0 2024-03-26T18:54:40.533 INFO:tasks.workunit.client.1.smithi190.stdout:4/983: dwrite d7/d8/d14/d23/fdc [0,4194304] 0 2024-03-26T18:54:40.534 INFO:tasks.workunit.client.0.smithi102.stdout:5/802: read d2/f54 [615451,8032] 0 2024-03-26T18:54:40.534 INFO:tasks.workunit.client.0.smithi102.stdout:5/803: read d2/d25/f7b [992964,129313] 0 2024-03-26T18:54:40.534 INFO:tasks.workunit.client.0.smithi102.stdout:5/804: stat d2/d25/d75/db9/d43/d9e/l73 0 2024-03-26T18:54:40.534 INFO:tasks.workunit.client.0.smithi102.stdout:4/439: dwrite d1/d17/d1a/d21/f56 [0,4194304] 0 2024-03-26T18:54:40.534 INFO:tasks.workunit.client.0.smithi102.stdout:4/440: chown d1/d75 516864540 1 2024-03-26T18:54:40.534 INFO:tasks.workunit.client.0.smithi102.stdout:4/441: dread - d1/d17/d1a/d21/d77/f81 zero size 2024-03-26T18:54:40.535 INFO:tasks.workunit.client.0.smithi102.stdout:8/595: symlink d0/d3/d26/d5c/lc4 0 2024-03-26T18:54:40.535 INFO:tasks.workunit.client.0.smithi102.stdout:5/805: truncate d2/d25/d75/db9/d43/d9e/d82/f106 779655 0 2024-03-26T18:54:40.538 INFO:tasks.workunit.client.1.smithi190.stdout:4/984: rename d7/d8/d1c/l34 to d7/d1d/d5e/l14d 0 2024-03-26T18:54:40.538 INFO:tasks.workunit.client.1.smithi190.stdout:4/985: dread - d7/d1d/d5e/d98/fb0 zero size 2024-03-26T18:54:40.539 INFO:tasks.workunit.client.0.smithi102.stdout:4/442: dread d1/d17/d1a/d21/d77/f71 [0,4194304] 0 2024-03-26T18:54:40.539 INFO:tasks.workunit.client.0.smithi102.stdout:4/443: write d1/d17/f88 [46406,32868] 0 2024-03-26T18:54:40.539 INFO:tasks.workunit.client.1.smithi190.stdout:4/986: symlink d7/d8/d14/d4b/d7d/da1/d113/l14e 0 2024-03-26T18:54:40.540 INFO:tasks.workunit.client.1.smithi190.stdout:4/987: symlink d7/d8/d14/d4b/db2/l14f 0 2024-03-26T18:54:40.541 INFO:tasks.workunit.client.0.smithi102.stdout:5/806: rename d2/dea/d4f/cd0 to d2/d25/d75/db9/c11a 0 2024-03-26T18:54:40.541 INFO:tasks.workunit.client.0.smithi102.stdout:4/444: rename d1/d17/d1f/d5d/d6f to d1/d17/d1f/d5d/d6f/d8a 22 2024-03-26T18:54:40.542 INFO:tasks.workunit.client.0.smithi102.stdout:2/514: dread d3/d2a/f34 [0,4194304] 0 2024-03-26T18:54:40.543 INFO:tasks.workunit.client.0.smithi102.stdout:5/807: chown d2/d14/d1a/f50 5101289 1 2024-03-26T18:54:40.545 INFO:tasks.workunit.client.0.smithi102.stdout:2/515: creat d3/d11/d15/d31/d7f/fb9 x:0 0 0 2024-03-26T18:54:40.545 INFO:tasks.workunit.client.0.smithi102.stdout:0/596: dwrite d6/d2d/d3d/fa2 [0,4194304] 0 2024-03-26T18:54:40.545 INFO:tasks.workunit.client.0.smithi102.stdout:0/597: stat d6/d2d/d3d/d80/d83/cb9 0 2024-03-26T18:54:40.545 INFO:tasks.workunit.client.0.smithi102.stdout:0/598: chown d6/dcb 2493282 1 2024-03-26T18:54:40.548 INFO:tasks.workunit.client.1.smithi190.stdout:3/740: dwrite d1/d3/f4b [0,4194304] 0 2024-03-26T18:54:40.548 INFO:tasks.workunit.client.1.smithi190.stdout:3/741: chown d1/de/d8a/c8e 1 1 2024-03-26T18:54:40.550 INFO:tasks.workunit.client.0.smithi102.stdout:0/599: creat d6/de/d13/d9a/fcc x:0 0 0 2024-03-26T18:54:40.550 INFO:tasks.workunit.client.0.smithi102.stdout:0/600: write d6/de/d13/d25/d31/d45/d82/d84/d88/fae [4769847,129234] 0 2024-03-26T18:54:40.550 INFO:tasks.workunit.client.0.smithi102.stdout:0/601: fdatasync d6/de/d13/d25/d4f/d66/f90 0 2024-03-26T18:54:40.550 INFO:tasks.workunit.client.0.smithi102.stdout:0/602: dread - d6/de/d13/d25/d31/fb5 zero size 2024-03-26T18:54:40.552 INFO:tasks.workunit.client.1.smithi190.stdout:3/742: link d1/de/d81/ld0 d1/d3/df/d18/d35/d41/dae/d9c/le2 0 2024-03-26T18:54:40.554 INFO:tasks.workunit.client.1.smithi190.stdout:3/743: mknod d1/de/d8a/ce3 0 2024-03-26T18:54:40.555 INFO:tasks.workunit.client.1.smithi190.stdout:3/744: rmdir d1/de/d9f 39 2024-03-26T18:54:40.562 INFO:tasks.workunit.client.0.smithi102.stdout:1/498: sync 2024-03-26T18:54:40.562 INFO:tasks.workunit.client.0.smithi102.stdout:1/499: truncate de/d16/d1d/d2a/d67/f95 351765 0 2024-03-26T18:54:40.563 INFO:tasks.workunit.client.0.smithi102.stdout:9/532: sync 2024-03-26T18:54:40.563 INFO:tasks.workunit.client.0.smithi102.stdout:8/596: rmdir d0/d3/d26 39 2024-03-26T18:54:40.563 INFO:tasks.workunit.client.0.smithi102.stdout:8/597: dread - d0/d3c/d96/d98/db6/f94 zero size 2024-03-26T18:54:40.563 INFO:tasks.workunit.client.0.smithi102.stdout:6/558: sync 2024-03-26T18:54:40.563 INFO:tasks.workunit.client.0.smithi102.stdout:7/499: sync 2024-03-26T18:54:40.564 INFO:tasks.workunit.client.0.smithi102.stdout:7/500: write d7/d44/f53 [75009,68684] 0 2024-03-26T18:54:40.564 INFO:tasks.workunit.client.0.smithi102.stdout:3/744: sync 2024-03-26T18:54:40.565 INFO:tasks.workunit.client.0.smithi102.stdout:0/603: read d6/d3e/f6a [2139988,69349] 0 2024-03-26T18:54:40.565 INFO:tasks.workunit.client.1.smithi190.stdout:0/945: dwrite d0/dd/d20/d61/f7b [0,4194304] 0 2024-03-26T18:54:40.567 INFO:tasks.workunit.client.0.smithi102.stdout:8/598: unlink d0/d3/d2c/d4e/f7b 0 2024-03-26T18:54:40.567 INFO:tasks.workunit.client.0.smithi102.stdout:2/516: write d3/d20/fa1 [2639794,75252] 0 2024-03-26T18:54:40.568 INFO:tasks.workunit.client.0.smithi102.stdout:6/559: mkdir d2/d3d/d53/db1 0 2024-03-26T18:54:40.573 INFO:tasks.workunit.client.0.smithi102.stdout:3/745: mknod df/d61/d55/c109 0 2024-03-26T18:54:40.573 INFO:tasks.workunit.client.0.smithi102.stdout:3/746: truncate df/d1c/fe0 5006170 0 2024-03-26T18:54:40.574 INFO:tasks.workunit.client.1.smithi190.stdout:0/946: symlink d0/d105/l156 0 2024-03-26T18:54:40.574 INFO:tasks.workunit.client.0.smithi102.stdout:0/604: rename d6/d3e/d69/d8c/d8f/c9d to d6/d3e/d87/ccd 0 2024-03-26T18:54:40.574 INFO:tasks.workunit.client.0.smithi102.stdout:6/560: dread d2/d7/f51 [0,4194304] 0 2024-03-26T18:54:40.574 INFO:tasks.workunit.client.0.smithi102.stdout:6/561: write d2/d3/f97 [163509,60241] 0 2024-03-26T18:54:40.580 INFO:tasks.workunit.client.0.smithi102.stdout:8/599: dread d0/d3/d2c/d33/d87/fa8 [0,4194304] 0 2024-03-26T18:54:40.581 INFO:tasks.workunit.client.0.smithi102.stdout:3/747: read df/f1a [439320,64485] 0 2024-03-26T18:54:40.582 INFO:tasks.workunit.client.0.smithi102.stdout:8/600: creat d0/d3/d2c/da7/fc5 x:0 0 0 2024-03-26T18:54:40.582 INFO:tasks.workunit.client.0.smithi102.stdout:8/601: fsync d0/d3/d2c/d2f/fa6 0 2024-03-26T18:54:40.582 INFO:tasks.workunit.client.1.smithi190.stdout:6/745: write d17/f21 [1819241,54503] 0 2024-03-26T18:54:40.582 INFO:tasks.workunit.client.1.smithi190.stdout:6/746: fdatasync f8 0 2024-03-26T18:54:40.583 INFO:tasks.workunit.client.0.smithi102.stdout:3/748: creat df/d29/d3a/d93/f10a x:0 0 0 2024-03-26T18:54:40.583 INFO:tasks.workunit.client.0.smithi102.stdout:3/749: readlink df/d29/d31/d36/d3c/d6f/d75/lbc 0 2024-03-26T18:54:40.583 INFO:tasks.workunit.client.0.smithi102.stdout:3/750: chown df/d61/c83 28222689 1 2024-03-26T18:54:40.585 INFO:tasks.workunit.client.1.smithi190.stdout:6/747: unlink d17/d28/d74/d97/ff1 0 2024-03-26T18:54:40.585 INFO:tasks.workunit.client.1.smithi190.stdout:6/748: creat d17/d28/d8b/d95/dda/ffc x:0 0 0 2024-03-26T18:54:40.590 INFO:tasks.workunit.client.0.smithi102.stdout:3/751: unlink df/l22 0 2024-03-26T18:54:40.590 INFO:tasks.workunit.client.0.smithi102.stdout:3/752: write df/d29/f8c [5221827,8696] 0 2024-03-26T18:54:40.591 INFO:tasks.workunit.client.0.smithi102.stdout:9/533: write df/d1a/d23/f71 [1061647,4168] 0 2024-03-26T18:54:40.591 INFO:tasks.workunit.client.0.smithi102.stdout:9/534: fdatasync df/d1a/f66 0 2024-03-26T18:54:40.591 INFO:tasks.workunit.client.0.smithi102.stdout:9/535: readlink df/d1a/l44 0 2024-03-26T18:54:40.592 INFO:tasks.workunit.client.0.smithi102.stdout:9/536: write df/d1a/f86 [2994564,124947] 0 2024-03-26T18:54:40.592 INFO:tasks.workunit.client.0.smithi102.stdout:9/537: fsync df/fa6 0 2024-03-26T18:54:40.595 INFO:tasks.workunit.client.0.smithi102.stdout:7/501: write d7/d2a/f4a [389860,862] 0 2024-03-26T18:54:40.595 INFO:tasks.workunit.client.0.smithi102.stdout:7/502: dread - d7/d31/f45 zero size 2024-03-26T18:54:40.597 INFO:tasks.workunit.client.1.smithi190.stdout:0/947: write d0/f5b [1087768,26404] 0 2024-03-26T18:54:40.597 INFO:tasks.workunit.client.1.smithi190.stdout:0/948: write d0/dd/d20/d1c/d8b/fd9 [818657,5116] 0 2024-03-26T18:54:40.597 INFO:tasks.workunit.client.1.smithi190.stdout:0/949: fdatasync d0/dd/d20/d1c/d46/da7/df0/d87/f11d 0 2024-03-26T18:54:40.598 INFO:tasks.workunit.client.0.smithi102.stdout:5/808: dwrite d2/d25/d75/db9/fd5 [0,4194304] 0 2024-03-26T18:54:40.598 INFO:tasks.workunit.client.0.smithi102.stdout:1/500: dwrite de/d16/d1d/d2a/d2c/d56/d78/d80/f98 [0,4194304] 0 2024-03-26T18:54:40.598 INFO:tasks.workunit.client.1.smithi190.stdout:1/756: dwrite d3/da/f1f [0,4194304] 0 2024-03-26T18:54:40.600 INFO:tasks.workunit.client.0.smithi102.stdout:7/503: unlink d7/d10/d17/f87 0 2024-03-26T18:54:40.603 INFO:tasks.workunit.client.1.smithi190.stdout:6/749: write d17/d28/d2c/f61 [3324284,44275] 0 2024-03-26T18:54:40.603 INFO:tasks.workunit.client.1.smithi190.stdout:8/980: sync 2024-03-26T18:54:40.603 INFO:tasks.workunit.client.1.smithi190.stdout:2/914: sync 2024-03-26T18:54:40.604 INFO:tasks.workunit.client.1.smithi190.stdout:5/769: sync 2024-03-26T18:54:40.604 INFO:tasks.workunit.client.1.smithi190.stdout:5/770: fdatasync d2/d17/d1b/d30/f73 0 2024-03-26T18:54:40.604 INFO:tasks.workunit.client.1.smithi190.stdout:5/771: getdents d2/d17/d1b/db6/d24/d3b 0 2024-03-26T18:54:40.604 INFO:tasks.workunit.client.1.smithi190.stdout:5/772: dread - d2/d17/d98/d3a/d3c/f6a zero size 2024-03-26T18:54:40.604 INFO:tasks.workunit.client.0.smithi102.stdout:0/605: write d6/de/d13/d25/d31/d45/d82/d84/d88/fae [742079,80159] 0 2024-03-26T18:54:40.604 INFO:tasks.workunit.client.0.smithi102.stdout:0/606: dread - d6/de/d13/d1f/fa8 zero size 2024-03-26T18:54:40.605 INFO:tasks.workunit.client.1.smithi190.stdout:5/773: fsync d2/f15 0 2024-03-26T18:54:40.605 INFO:tasks.workunit.client.0.smithi102.stdout:2/517: write d3/d2a/f6d [6666696,46744] 0 2024-03-26T18:54:40.608 INFO:tasks.workunit.client.1.smithi190.stdout:0/950: creat d0/dd/f157 x:0 0 0 2024-03-26T18:54:40.608 INFO:tasks.workunit.client.1.smithi190.stdout:0/951: truncate d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/f13c 852535 0 2024-03-26T18:54:40.608 INFO:tasks.workunit.client.1.smithi190.stdout:8/981: read d9/da2/d73/d108/d12b/d37/f8d [677584,124184] 0 2024-03-26T18:54:40.611 INFO:tasks.workunit.client.1.smithi190.stdout:8/982: mknod d9/da2/d73/d108/d12b/d37/d3a/d61/daf/db3/db5/de3/c147 0 2024-03-26T18:54:40.611 INFO:tasks.workunit.client.1.smithi190.stdout:1/757: mknod d3/d9/db/cf2 0 2024-03-26T18:54:40.613 INFO:tasks.workunit.client.1.smithi190.stdout:8/983: symlink d9/da2/d73/d108/d12b/d37/d3a/d61/daf/db3/db5/de3/d13a/l148 0 2024-03-26T18:54:40.613 INFO:tasks.workunit.client.1.smithi190.stdout:8/984: getdents d9/d39/d9e/d114 0 2024-03-26T18:54:40.615 INFO:tasks.workunit.client.1.smithi190.stdout:8/985: creat d9/d48/d10c/d140/f149 x:0 0 0 2024-03-26T18:54:40.615 INFO:tasks.workunit.client.1.smithi190.stdout:8/986: write d9/dd/d23/f131 [1219461,75800] 0 2024-03-26T18:54:40.616 INFO:tasks.workunit.client.1.smithi190.stdout:8/987: rmdir d9/da2/d73/d108/d12b/d37/db8 39 2024-03-26T18:54:40.616 INFO:tasks.workunit.client.1.smithi190.stdout:8/988: stat d9/dd/d15/ccb 0 2024-03-26T18:54:40.617 INFO:tasks.workunit.client.1.smithi190.stdout:8/989: write d9/da2/d73/d108/d12b/d37/d3a/d61/fb6 [3816178,109801] 0 2024-03-26T18:54:40.617 INFO:tasks.workunit.client.0.smithi102.stdout:5/809: creat d2/dea/d4f/da9/dee/f11b x:0 0 0 2024-03-26T18:54:40.617 INFO:tasks.workunit.client.0.smithi102.stdout:6/562: dwrite d2/d7/f54 [0,4194304] 0 2024-03-26T18:54:40.619 INFO:tasks.workunit.client.1.smithi190.stdout:8/990: mkdir d9/da2/d73/d108/d12b/d37/d3a/d43/d14a 0 2024-03-26T18:54:40.619 INFO:tasks.workunit.client.1.smithi190.stdout:8/991: fsync d9/da2/d73/d108/f128 0 2024-03-26T18:54:40.621 INFO:tasks.workunit.client.0.smithi102.stdout:4/445: dwrite d1/f9 [0,4194304] 0 2024-03-26T18:54:40.621 INFO:tasks.workunit.client.0.smithi102.stdout:8/602: dwrite d0/d3/d2c/f85 [0,4194304] 0 2024-03-26T18:54:40.621 INFO:tasks.workunit.client.0.smithi102.stdout:4/446: read d1/d17/d1a/f53 [2046480,28747] 0 2024-03-26T18:54:40.621 INFO:tasks.workunit.client.0.smithi102.stdout:8/603: write d0/d9/f7d [191399,84210] 0 2024-03-26T18:54:40.621 INFO:tasks.workunit.client.1.smithi190.stdout:2/915: read f5 [3936101,27238] 0 2024-03-26T18:54:40.621 INFO:tasks.workunit.client.1.smithi190.stdout:2/916: chown d6/d9/d3a/d3d/d1c/d8d/dc7/fba 2 1 2024-03-26T18:54:40.621 INFO:tasks.workunit.client.1.smithi190.stdout:4/988: dwrite d7/d1d/d5e/d98/fb0 [0,4194304] 0 2024-03-26T18:54:40.621 INFO:tasks.workunit.client.1.smithi190.stdout:0/952: symlink d0/dd/d20/d1c/d46/da7/df0/d87/l158 0 2024-03-26T18:54:40.621 INFO:tasks.workunit.client.1.smithi190.stdout:0/953: truncate d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/f137 257158 0 2024-03-26T18:54:40.622 INFO:tasks.workunit.client.1.smithi190.stdout:8/992: creat d9/da2/d73/d108/d12b/d37/d3a/d43/d14a/f14b x:0 0 0 2024-03-26T18:54:40.622 INFO:tasks.workunit.client.0.smithi102.stdout:3/753: dwrite df/d61/de5/ff8 [0,4194304] 0 2024-03-26T18:54:40.622 INFO:tasks.workunit.client.0.smithi102.stdout:7/504: mknod d7/d10/d27/d34/c91 0 2024-03-26T18:54:40.623 INFO:tasks.workunit.client.1.smithi190.stdout:1/758: creat d3/da/ff3 x:0 0 0 2024-03-26T18:54:40.626 INFO:tasks.workunit.client.1.smithi190.stdout:0/954: truncate d0/dd/fe 3297859 0 2024-03-26T18:54:40.626 INFO:tasks.workunit.client.1.smithi190.stdout:0/955: chown d0/dd/d20/d61/f7b 42 1 2024-03-26T18:54:40.626 INFO:tasks.workunit.client.1.smithi190.stdout:0/956: write d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/f129 [864293,51030] 0 2024-03-26T18:54:40.627 INFO:tasks.workunit.client.0.smithi102.stdout:2/518: symlink d3/d11/d15/d31/d39/lba 0 2024-03-26T18:54:40.627 INFO:tasks.workunit.client.1.smithi190.stdout:2/917: getdents d6/d9/d45/d6b/d93 0 2024-03-26T18:54:40.627 INFO:tasks.workunit.client.1.smithi190.stdout:2/918: creat d6/d9/d3a/d3d/d2e/d31/f13a x:0 0 0 2024-03-26T18:54:40.627 INFO:tasks.workunit.client.1.smithi190.stdout:2/919: readlink d6/d21/dd8/l100 0 2024-03-26T18:54:40.627 INFO:tasks.workunit.client.1.smithi190.stdout:2/920: rename d6/d9/d3a/d3d/d1c to d6/d9/d3a/d3d/d1c/d8d/dc7/d13b 22 2024-03-26T18:54:40.628 INFO:tasks.workunit.client.1.smithi190.stdout:0/957: read d0/dd/d20/d1c/d46/da7/df0/f2b [4558013,17361] 0 2024-03-26T18:54:40.628 INFO:tasks.workunit.client.1.smithi190.stdout:1/759: symlink d3/d9/db/dd4/dd7/lf4 0 2024-03-26T18:54:40.628 INFO:tasks.workunit.client.1.smithi190.stdout:1/760: dread - d3/da/d2e/d59/f65 zero size 2024-03-26T18:54:40.629 INFO:tasks.workunit.client.1.smithi190.stdout:1/761: chown d3/da/dd 2191312 1 2024-03-26T18:54:40.629 INFO:tasks.workunit.client.1.smithi190.stdout:8/993: getdents d9/da2/d73/d108/d12b/d37/d3a/d43/d8f/dd2 0 2024-03-26T18:54:40.632 INFO:tasks.workunit.client.1.smithi190.stdout:2/921: mkdir d6/d9/d3a/d3d/d1c/d8d/dc7/d13c 0 2024-03-26T18:54:40.632 INFO:tasks.workunit.client.1.smithi190.stdout:2/922: chown d6/d9/d3a/d3d/d1c/d8d/de8/dfb/c107 681981 1 2024-03-26T18:54:40.632 INFO:tasks.workunit.client.1.smithi190.stdout:2/923: fsync d6/d9/d3a/d3d/d2e/d31/f12a 0 2024-03-26T18:54:40.633 INFO:tasks.workunit.client.1.smithi190.stdout:8/994: symlink d9/d39/d9e/d114/l14c 0 2024-03-26T18:54:40.633 INFO:tasks.workunit.client.1.smithi190.stdout:8/995: fsync d9/dd/f78 0 2024-03-26T18:54:40.634 INFO:tasks.workunit.client.1.smithi190.stdout:1/762: mkdir d3/da/dd/d2f/d33/de8/df5 0 2024-03-26T18:54:40.635 INFO:tasks.workunit.client.0.smithi102.stdout:6/563: symlink d2/df/d35/da1/lb2 0 2024-03-26T18:54:40.636 INFO:tasks.workunit.client.1.smithi190.stdout:8/996: creat d9/dd/d15/f14d x:0 0 0 2024-03-26T18:54:40.636 INFO:tasks.workunit.client.1.smithi190.stdout:8/997: write d9/dd/d15/f7a [953236,100195] 0 2024-03-26T18:54:40.636 INFO:tasks.workunit.client.1.smithi190.stdout:4/989: dread d7/d8/d1c/f39 [0,4194304] 0 2024-03-26T18:54:40.641 INFO:tasks.workunit.client.1.smithi190.stdout:0/958: dread d0/dd/d20/d1c/d46/da7/df0/d4b/f89 [0,4194304] 0 2024-03-26T18:54:40.642 INFO:tasks.workunit.client.0.smithi102.stdout:2/519: dread d3/d2a/f2f [0,4194304] 0 2024-03-26T18:54:40.642 INFO:tasks.workunit.client.0.smithi102.stdout:2/520: dread - d3/d11/d8f/fa2 zero size 2024-03-26T18:54:40.642 INFO:tasks.workunit.client.0.smithi102.stdout:2/521: creat d3/d11/d15/d46/d7b/d97/fbb x:0 0 0 2024-03-26T18:54:40.642 INFO:tasks.workunit.client.0.smithi102.stdout:2/522: dread - d3/f75 zero size 2024-03-26T18:54:40.643 INFO:tasks.workunit.client.0.smithi102.stdout:2/523: stat d3/d11/d15/c36 0 2024-03-26T18:54:40.643 INFO:tasks.workunit.client.0.smithi102.stdout:2/524: read - d3/d20/d78/d84/f98 zero size 2024-03-26T18:54:40.644 INFO:tasks.workunit.client.0.smithi102.stdout:8/604: creat d0/d15/d5d/d89/fc6 x:0 0 0 2024-03-26T18:54:40.646 INFO:tasks.workunit.client.1.smithi190.stdout:1/763: dread d3/da/dd/d2f/d5b/f63 [0,4194304] 0 2024-03-26T18:54:40.646 INFO:tasks.workunit.client.1.smithi190.stdout:1/764: creat d3/d8f/ff6 x:0 0 0 2024-03-26T18:54:40.647 INFO:tasks.workunit.client.1.smithi190.stdout:2/924: read d6/d9/d3a/d3d/d1c/d8d/dc7/f58 [1690499,51427] 0 2024-03-26T18:54:40.647 INFO:tasks.workunit.client.1.smithi190.stdout:2/925: write d6/d9/d3a/d3d/f11c [4328452,66774] 0 2024-03-26T18:54:40.648 INFO:tasks.workunit.client.1.smithi190.stdout:5/774: dwrite d2/d17/d1b/d60/d63/dcd/fed [0,4194304] 0 2024-03-26T18:54:40.648 INFO:tasks.workunit.client.1.smithi190.stdout:0/959: getdents d0/dd/d20/d1c/d46/d58 0 2024-03-26T18:54:40.648 INFO:tasks.workunit.client.1.smithi190.stdout:8/998: dread d9/d48/f62 [0,4194304] 0 2024-03-26T18:54:40.649 INFO:tasks.workunit.client.0.smithi102.stdout:3/754: symlink df/d29/d31/d77/d8a/dae/dee/df2/l10b 0 2024-03-26T18:54:40.650 INFO:tasks.workunit.client.1.smithi190.stdout:2/926: creat d6/d9/d48/f13d x:0 0 0 2024-03-26T18:54:40.651 INFO:tasks.workunit.client.0.smithi102.stdout:7/505: truncate d7/d12/d4d/d62/f66 1555853 0 2024-03-26T18:54:40.651 INFO:tasks.workunit.client.1.smithi190.stdout:8/999: mknod d9/d39/d9c/c14e 0 2024-03-26T18:54:40.656 INFO:tasks.workunit.client.1.smithi190.stdout:5/775: symlink d2/d17/d98/lfd 0 2024-03-26T18:54:40.658 INFO:tasks.workunit.client.1.smithi190.stdout:0/960: creat d0/dd/d20/f159 x:0 0 0 2024-03-26T18:54:40.662 INFO:tasks.workunit.client.1.smithi190.stdout:1/765: write d3/d73/faa [2422983,40339] 0 2024-03-26T18:54:40.666 INFO:tasks.workunit.client.0.smithi102.stdout:6/564: truncate d2/df/d72/f7a 281347 0 2024-03-26T18:54:40.666 INFO:tasks.workunit.client.0.smithi102.stdout:9/538: dwrite df/d1a/d23/f48 [4194304,4194304] 0 2024-03-26T18:54:40.666 INFO:tasks.workunit.client.0.smithi102.stdout:9/539: dread - df/d1a/d65/fae zero size 2024-03-26T18:54:40.667 INFO:tasks.workunit.client.0.smithi102.stdout:3/755: dread df/d29/d3a/d93/ff1 [0,4194304] 0 2024-03-26T18:54:40.668 INFO:tasks.workunit.client.0.smithi102.stdout:1/501: dwrite de/d16/d1d/f4d [0,4194304] 0 2024-03-26T18:54:40.669 INFO:tasks.workunit.client.1.smithi190.stdout:2/927: dread d6/f4d [0,4194304] 0 2024-03-26T18:54:40.669 INFO:tasks.workunit.client.1.smithi190.stdout:2/928: creat d6/d9/d3a/d3d/d2e/d86/f13e x:0 0 0 2024-03-26T18:54:40.669 INFO:tasks.workunit.client.1.smithi190.stdout:2/929: dread - d6/d9/d10/d5f/f10d zero size 2024-03-26T18:54:40.669 INFO:tasks.workunit.client.1.smithi190.stdout:2/930: stat d6/d21/d61/d6c/fc4 0 2024-03-26T18:54:40.670 INFO:tasks.workunit.client.1.smithi190.stdout:5/776: dread d2/d17/d62/f7e [0,4194304] 0 2024-03-26T18:54:40.670 INFO:tasks.workunit.client.1.smithi190.stdout:4/990: dwrite d7/d8/d14/f94 [4194304,4194304] 0 2024-03-26T18:54:40.672 INFO:tasks.workunit.client.0.smithi102.stdout:5/810: dwrite d2/f64 [0,4194304] 0 2024-03-26T18:54:40.672 INFO:tasks.workunit.client.0.smithi102.stdout:5/811: chown d2/d25/d75/db9/ded 3 1 2024-03-26T18:54:40.674 INFO:tasks.workunit.client.0.smithi102.stdout:4/447: rename d1/d25/d39 to d1/d17/d1f/d5d/d8b 0 2024-03-26T18:54:40.677 INFO:tasks.workunit.client.0.smithi102.stdout:2/525: mknod d3/d11/d15/d31/d39/cbc 0 2024-03-26T18:54:40.677 INFO:tasks.workunit.client.0.smithi102.stdout:2/526: dread - d3/d11/d15/d31/d7f/fb9 zero size 2024-03-26T18:54:40.679 INFO:tasks.workunit.client.0.smithi102.stdout:8/605: creat d0/d15/d5d/fc7 x:0 0 0 2024-03-26T18:54:40.680 INFO:tasks.workunit.client.0.smithi102.stdout:7/506: link d7/d12/c46 d7/d10/c92 0 2024-03-26T18:54:40.680 INFO:tasks.workunit.client.0.smithi102.stdout:7/507: write d7/ff [1561655,101433] 0 2024-03-26T18:54:40.681 INFO:tasks.workunit.client.0.smithi102.stdout:7/508: chown d7/d12/d51/f58 17292 1 2024-03-26T18:54:40.681 INFO:tasks.workunit.client.0.smithi102.stdout:7/509: chown d7/d12/d4d/f8b 3667 1 2024-03-26T18:54:40.681 INFO:tasks.workunit.client.0.smithi102.stdout:7/510: stat d7/d12/d2b/f47 0 2024-03-26T18:54:40.681 INFO:tasks.workunit.client.0.smithi102.stdout:7/511: fdatasync d7/d10/f14 0 2024-03-26T18:54:40.682 INFO:tasks.workunit.client.1.smithi190.stdout:4/991: unlink d7/d8/d1c/l21 0 2024-03-26T18:54:40.684 INFO:tasks.workunit.client.1.smithi190.stdout:5/777: write d2/f14 [2783724,13833] 0 2024-03-26T18:54:40.686 INFO:tasks.workunit.client.0.smithi102.stdout:1/502: mkdir de/d16/d1d/d61/d6a/da5/da6 0 2024-03-26T18:54:40.686 INFO:tasks.workunit.client.0.smithi102.stdout:1/503: read - de/f97 zero size 2024-03-26T18:54:40.687 INFO:tasks.workunit.client.1.smithi190.stdout:5/778: symlink d2/d17/d62/lfe 0 2024-03-26T18:54:40.690 INFO:tasks.workunit.client.0.smithi102.stdout:8/606: creat d0/d15/fc8 x:0 0 0 2024-03-26T18:54:40.692 INFO:tasks.workunit.client.1.smithi190.stdout:5/779: mknod d2/d17/cff 0 2024-03-26T18:54:40.692 INFO:tasks.workunit.client.1.smithi190.stdout:4/992: rename d7/d8/d14/d4b/db2/c13a to d7/d8/d14/d23/d28/d38/d96/d9a/c150 0 2024-03-26T18:54:40.692 INFO:tasks.workunit.client.1.smithi190.stdout:4/993: chown d7/d8/d14/d23/l8e 1743701395 1 2024-03-26T18:54:40.692 INFO:tasks.workunit.client.1.smithi190.stdout:4/994: chown d7/d8/d14 0 1 2024-03-26T18:54:40.694 INFO:tasks.workunit.client.0.smithi102.stdout:1/504: symlink de/d16/d1d/d2a/d2c/d56/d6e/d9d/la7 0 2024-03-26T18:54:40.696 INFO:tasks.workunit.client.1.smithi190.stdout:5/780: mknod d2/d17/d1b/d8e/c100 0 2024-03-26T18:54:40.697 INFO:tasks.workunit.client.0.smithi102.stdout:1/505: truncate f9 3261309 0 2024-03-26T18:54:40.697 INFO:tasks.workunit.client.0.smithi102.stdout:1/506: readlink de/d16/d1d/d2a/d2c/d56/d5f/la0 0 2024-03-26T18:54:40.698 INFO:tasks.workunit.client.0.smithi102.stdout:1/507: link de/d16/d1d/d2a/d2c/d56/d6e/f9a de/d2e/fa8 0 2024-03-26T18:54:40.699 INFO:tasks.workunit.client.1.smithi190.stdout:4/995: link d7/d8/d14/d4b/d7d/da1/d113/f13b d7/d8/d14/d4b/d7d/da1/d113/f151 0 2024-03-26T18:54:40.700 INFO:tasks.workunit.client.1.smithi190.stdout:4/996: symlink d7/d8/d14/d4b/d7d/da1/df8/l152 0 2024-03-26T18:54:40.704 INFO:tasks.workunit.client.1.smithi190.stdout:4/997: truncate d7/d1d/d5e/f81 5414457 0 2024-03-26T18:54:40.708 INFO:tasks.workunit.client.1.smithi190.stdout:2/931: write d6/d9/d3a/d3d/d55/f116 [602680,55087] 0 2024-03-26T18:54:40.708 INFO:tasks.workunit.client.1.smithi190.stdout:2/932: read d6/d9/d3a/d3d/d55/f92 [3444388,92562] 0 2024-03-26T18:54:40.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/933: rmdir d6/d9/d3a/d3d/d2e/d86 39 2024-03-26T18:54:40.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/934: chown d6/d9/d45/f74 316180 1 2024-03-26T18:54:40.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/935: chown d6/d21/d61/d6c/fcc 97769784 1 2024-03-26T18:54:40.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/936: read - d6/d9/d48/f13d zero size 2024-03-26T18:54:40.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/937: write d6/dbc/fe1 [467223,84361] 0 2024-03-26T18:54:40.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/938: chown d6/d9/d3a/d3d/d1c/d8d/dc7/dd1/f102 110405286 1 2024-03-26T18:54:40.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/939: write d6/d21/dd8/ff4 [23413,126214] 0 2024-03-26T18:54:40.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/940: chown d6/d9/d45/d6b/d93/dc0/ccd 19344 1 2024-03-26T18:54:40.710 INFO:tasks.workunit.client.1.smithi190.stdout:2/941: mkdir d6/d9/d45/df3/d13f 0 2024-03-26T18:54:40.712 INFO:tasks.workunit.client.1.smithi190.stdout:2/942: rename d6/d21/d61/d6c/f9c to d6/d9/d3a/d3d/f140 0 2024-03-26T18:54:40.712 INFO:tasks.workunit.client.1.smithi190.stdout:4/998: dread d7/d8/d14/d4b/f4c [4194304,4194304] 0 2024-03-26T18:54:40.714 INFO:tasks.workunit.client.1.smithi190.stdout:2/943: symlink d6/d9/d3a/d3d/d2e/d10e/d9e/dfc/l141 0 2024-03-26T18:54:40.719 INFO:tasks.workunit.client.1.smithi190.stdout:4/999: dread d7/d1d/d42/d4a/f66 [0,4194304] 0 2024-03-26T18:54:40.731 INFO:tasks.workunit.client.0.smithi102.stdout:3/756: dwrite df/d1c/d70/fb6 [4194304,4194304] 0 2024-03-26T18:54:40.732 INFO:tasks.workunit.client.0.smithi102.stdout:3/757: truncate df/d1c/f2a 1031620 0 2024-03-26T18:54:40.732 INFO:tasks.workunit.client.0.smithi102.stdout:3/758: dread - df/d29/d3a/d93/f10a zero size 2024-03-26T18:54:40.738 INFO:tasks.workunit.client.0.smithi102.stdout:8/607: dwrite d0/d15/d5d/dac/fc3 [0,4194304] 0 2024-03-26T18:54:40.738 INFO:tasks.workunit.client.0.smithi102.stdout:8/608: read d0/d3/f21 [1975663,39404] 0 2024-03-26T18:54:40.739 INFO:tasks.workunit.client.0.smithi102.stdout:9/540: dwrite df/d1a/d23/d29/d9c/fab [0,4194304] 0 2024-03-26T18:54:40.739 INFO:tasks.workunit.client.0.smithi102.stdout:9/541: getdents df/d46 0 2024-03-26T18:54:40.740 INFO:tasks.workunit.client.0.smithi102.stdout:2/527: dwrite d3/d11/f55 [0,4194304] 0 2024-03-26T18:54:40.740 INFO:tasks.workunit.client.0.smithi102.stdout:4/448: dwrite d1/d17/d1f/d5d/d8b/f7f [0,4194304] 0 2024-03-26T18:54:40.740 INFO:tasks.workunit.client.1.smithi190.stdout:0/961: dwrite d0/dd/d20/fdf [0,4194304] 0 2024-03-26T18:54:40.741 INFO:tasks.workunit.client.1.smithi190.stdout:5/781: dwrite d2/d17/d1b/db6/d68/f75 [0,4194304] 0 2024-03-26T18:54:40.741 INFO:tasks.workunit.client.1.smithi190.stdout:5/782: fdatasync d2/d17/d1b/d60/d63/dcd/fd4 0 2024-03-26T18:54:40.741 INFO:tasks.workunit.client.1.smithi190.stdout:5/783: write d2/f15 [2857464,122519] 0 2024-03-26T18:54:40.741 INFO:tasks.workunit.client.1.smithi190.stdout:5/784: chown d2/d17/d1b/f58 14 1 2024-03-26T18:54:40.741 INFO:tasks.workunit.client.1.smithi190.stdout:5/785: write d2/d17/d1b/d30/fb1 [276080,15183] 0 2024-03-26T18:54:40.741 INFO:tasks.workunit.client.0.smithi102.stdout:8/609: symlink d0/d3/d26/d80/lc9 0 2024-03-26T18:54:40.741 INFO:tasks.workunit.client.0.smithi102.stdout:8/610: dread - d0/d15/fc8 zero size 2024-03-26T18:54:40.746 INFO:tasks.workunit.client.0.smithi102.stdout:3/759: write df/f26 [11606267,19050] 0 2024-03-26T18:54:40.746 INFO:tasks.workunit.client.0.smithi102.stdout:9/542: getdents df/d1a/d23/d4e/d8e 0 2024-03-26T18:54:40.748 INFO:tasks.workunit.client.0.smithi102.stdout:4/449: read d1/d17/d1a/f4f [4009814,77309] 0 2024-03-26T18:54:40.748 INFO:tasks.workunit.client.0.smithi102.stdout:4/450: truncate d1/d27/f5f 3297132 0 2024-03-26T18:54:40.748 INFO:tasks.workunit.client.0.smithi102.stdout:4/451: fsync d1/d17/d1f/d5d/d8b/d3e/d4c/f6a 0 2024-03-26T18:54:40.754 INFO:tasks.workunit.client.0.smithi102.stdout:8/611: mkdir d0/d3/d2c/d4e/dca 0 2024-03-26T18:54:40.754 INFO:tasks.workunit.client.1.smithi190.stdout:0/962: write d0/dd/d20/d61/fac [1249536,58103] 0 2024-03-26T18:54:40.755 INFO:tasks.workunit.client.0.smithi102.stdout:2/528: read d3/d11/d15/d31/f60 [1847345,33577] 0 2024-03-26T18:54:40.755 INFO:tasks.workunit.client.0.smithi102.stdout:2/529: creat d3/d2d/d69/fbd x:0 0 0 2024-03-26T18:54:40.758 INFO:tasks.workunit.client.1.smithi190.stdout:5/786: symlink d2/d17/d1b/db6/d96/l101 0 2024-03-26T18:54:40.762 INFO:tasks.workunit.client.1.smithi190.stdout:0/963: mkdir d0/dd/d20/d1c/d46/da7/df0/d87/d93/de1/d104/d121/d15a 0 2024-03-26T18:54:40.762 INFO:tasks.workunit.client.1.smithi190.stdout:0/964: stat d0/cf6 0 2024-03-26T18:54:40.762 INFO:tasks.workunit.client.1.smithi190.stdout:0/965: stat d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d6b/de8 0 2024-03-26T18:54:40.762 INFO:tasks.workunit.client.1.smithi190.stdout:0/966: write d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/f12d [141025,50996] 0 2024-03-26T18:54:40.764 INFO:tasks.workunit.client.1.smithi190.stdout:5/787: mkdir d2/d17/d1b/d60/d63/d102 0 2024-03-26T18:54:40.767 INFO:tasks.workunit.client.1.smithi190.stdout:5/788: symlink d2/d17/d98/l103 0 2024-03-26T18:54:40.767 INFO:tasks.workunit.client.1.smithi190.stdout:5/789: chown d2/d17/d62/d64/ff1 5276 1 2024-03-26T18:54:40.770 INFO:tasks.workunit.client.0.smithi102.stdout:1/508: fsync f9 0 2024-03-26T18:54:40.770 INFO:tasks.workunit.client.0.smithi102.stdout:1/509: fdatasync f8 0 2024-03-26T18:54:40.771 INFO:tasks.workunit.client.0.smithi102.stdout:3/760: mkdir df/d29/d31/dc0/d10c 0 2024-03-26T18:54:40.771 INFO:tasks.workunit.client.0.smithi102.stdout:3/761: stat df/d61/de5 0 2024-03-26T18:54:40.771 INFO:tasks.workunit.client.0.smithi102.stdout:3/762: readlink df/d29/d31/d36/d45/daf/l103 0 2024-03-26T18:54:40.772 INFO:tasks.workunit.client.0.smithi102.stdout:9/543: unlink df/d1a/d65/ca4 0 2024-03-26T18:54:40.773 INFO:tasks.workunit.client.0.smithi102.stdout:9/544: fsync df/d1a/d23/f5d 0 2024-03-26T18:54:40.779 INFO:tasks.workunit.client.0.smithi102.stdout:6/565: rename d2/df/d72/f7a to d2/d3d/d53/db1/fb3 0 2024-03-26T18:54:40.779 INFO:tasks.workunit.client.0.smithi102.stdout:6/566: fdatasync d2/df/d72/f89 0 2024-03-26T18:54:40.779 INFO:tasks.workunit.client.0.smithi102.stdout:6/567: stat d2/d3 0 2024-03-26T18:54:40.779 INFO:tasks.workunit.client.0.smithi102.stdout:2/530: symlink d3/d11/d15/d46/d7b/lbe 0 2024-03-26T18:54:40.779 INFO:tasks.workunit.client.0.smithi102.stdout:1/510: symlink de/d16/la9 0 2024-03-26T18:54:40.779 INFO:tasks.workunit.client.0.smithi102.stdout:1/511: write de/f19 [9353311,124577] 0 2024-03-26T18:54:40.779 INFO:tasks.workunit.client.0.smithi102.stdout:1/512: write de/d16/d1d/d2a/d2c/d56/d78/d80/f85 [1163010,60088] 0 2024-03-26T18:54:40.780 INFO:tasks.workunit.client.0.smithi102.stdout:3/763: mknod df/d29/d31/d36/d3c/c10d 0 2024-03-26T18:54:40.780 INFO:tasks.workunit.client.0.smithi102.stdout:9/545: mknod df/d84/ccb 0 2024-03-26T18:54:40.780 INFO:tasks.workunit.client.0.smithi102.stdout:9/546: truncate df/d1a/d23/fb2 402851 0 2024-03-26T18:54:40.780 INFO:tasks.workunit.client.0.smithi102.stdout:9/547: chown df/d1a/d1d/d72/d7d/cb3 3509 1 2024-03-26T18:54:40.781 INFO:tasks.workunit.client.0.smithi102.stdout:4/452: symlink d1/d17/d1f/d5d/d8b/l8c 0 2024-03-26T18:54:40.781 INFO:tasks.workunit.client.0.smithi102.stdout:4/453: fdatasync d1/d17/d1f/d5d/d8b/d3e/d4c/f68 0 2024-03-26T18:54:40.784 INFO:tasks.workunit.client.0.smithi102.stdout:5/812: rename d2/d25/d5b/d76/fac to d2/dea/dbb/dcf/f11c 0 2024-03-26T18:54:40.784 INFO:tasks.workunit.client.0.smithi102.stdout:5/813: write d2/d25/d75/db9/d43/d9e/daf/f108 [350719,18307] 0 2024-03-26T18:54:40.786 INFO:tasks.workunit.client.0.smithi102.stdout:3/764: mkdir df/d1c/d70/dcd/df7/d10e 0 2024-03-26T18:54:40.787 INFO:tasks.workunit.client.0.smithi102.stdout:9/548: mkdir df/d1a/d81/dcc 0 2024-03-26T18:54:40.787 INFO:tasks.workunit.client.0.smithi102.stdout:9/549: dread - df/d84/f9e zero size 2024-03-26T18:54:40.787 INFO:tasks.workunit.client.0.smithi102.stdout:4/454: rmdir d1/d17/d1a/d21 39 2024-03-26T18:54:40.789 INFO:tasks.workunit.client.0.smithi102.stdout:9/550: dread df/d1a/d1d/f32 [0,4194304] 0 2024-03-26T18:54:40.790 INFO:tasks.workunit.client.0.smithi102.stdout:7/512: rename d7/d12/c2d to d7/d12/d4d/d62/c93 0 2024-03-26T18:54:40.791 INFO:tasks.workunit.client.0.smithi102.stdout:4/455: truncate d1/d17/d1a/f4b 1951567 0 2024-03-26T18:54:40.792 INFO:tasks.workunit.client.0.smithi102.stdout:7/513: mknod d7/d10/d84/c94 0 2024-03-26T18:54:40.792 INFO:tasks.workunit.client.0.smithi102.stdout:7/514: readlink l5 0 2024-03-26T18:54:40.793 INFO:tasks.workunit.client.0.smithi102.stdout:7/515: mkdir d7/d12/d4d/d62/d95 0 2024-03-26T18:54:40.793 INFO:tasks.workunit.client.0.smithi102.stdout:7/516: fsync d7/d44/f53 0 2024-03-26T18:54:40.794 INFO:tasks.workunit.client.0.smithi102.stdout:7/517: unlink d7/d31/d79/c8f 0 2024-03-26T18:54:40.813 INFO:tasks.workunit.client.0.smithi102.stdout:2/531: dwrite d3/d11/fa6 [0,4194304] 0 2024-03-26T18:54:40.816 INFO:tasks.workunit.client.0.smithi102.stdout:2/532: rmdir d3/d82/d93/d9e/da0 39 2024-03-26T18:54:40.817 INFO:tasks.workunit.client.0.smithi102.stdout:2/533: write d3/d82/d93/d9e/da0/fb0 [282614,42526] 0 2024-03-26T18:54:40.817 INFO:tasks.workunit.client.0.smithi102.stdout:2/534: dread - d3/d20/f80 zero size 2024-03-26T18:54:40.820 INFO:tasks.workunit.client.0.smithi102.stdout:1/513: dwrite de/d16/d1d/d2a/d2c/d56/d5f/fa2 [0,4194304] 0 2024-03-26T18:54:40.823 INFO:tasks.workunit.client.0.smithi102.stdout:9/551: dread df/d1a/f5e [0,4194304] 0 2024-03-26T18:54:40.823 INFO:tasks.workunit.client.0.smithi102.stdout:9/552: dread - df/d1a/d23/d29/f4c zero size 2024-03-26T18:54:40.823 INFO:tasks.workunit.client.0.smithi102.stdout:9/553: truncate df/d1a/d23/d3e/d75/f8d 369029 0 2024-03-26T18:54:40.826 INFO:tasks.workunit.client.0.smithi102.stdout:5/814: dwrite d2/f62 [0,4194304] 0 2024-03-26T18:54:40.828 INFO:tasks.workunit.client.0.smithi102.stdout:5/815: link d2/d25/d75/db9/d43/f92 d2/d25/d75/db9/ded/f11d 0 2024-03-26T18:54:40.829 INFO:tasks.workunit.client.0.smithi102.stdout:5/816: symlink d2/d25/d75/db9/d43/d9e/d82/dc9/l11e 0 2024-03-26T18:54:40.829 INFO:tasks.workunit.client.0.smithi102.stdout:1/514: dread de/d16/d1d/d2a/d2c/d56/d6e/f9a [0,4194304] 0 2024-03-26T18:54:40.829 INFO:tasks.workunit.client.0.smithi102.stdout:2/535: dread d3/d2a/f6d [4194304,4194304] 0 2024-03-26T18:54:40.829 INFO:tasks.workunit.client.0.smithi102.stdout:2/536: creat d3/d82/d93/fbf x:0 0 0 2024-03-26T18:54:40.830 INFO:tasks.workunit.client.0.smithi102.stdout:5/817: mkdir d2/dea/db4/d11f 0 2024-03-26T18:54:40.831 INFO:tasks.workunit.client.0.smithi102.stdout:1/515: mkdir de/d16/d1d/d2a/d2c/d56/d5f/daa 0 2024-03-26T18:54:40.831 INFO:tasks.workunit.client.0.smithi102.stdout:1/516: fsync de/d16/f35 0 2024-03-26T18:54:40.835 INFO:tasks.workunit.client.0.smithi102.stdout:2/537: mkdir d3/d11/d27/dc0 0 2024-03-26T18:54:40.836 INFO:tasks.workunit.client.0.smithi102.stdout:1/517: symlink de/d16/d1d/d2a/d2c/d56/d5f/lab 0 2024-03-26T18:54:40.837 INFO:tasks.workunit.client.0.smithi102.stdout:4/456: write d1/d17/d1a/d21/f56 [84143,86961] 0 2024-03-26T18:54:40.842 INFO:tasks.workunit.client.0.smithi102.stdout:2/538: dread d3/d82/d93/d9e/d1a/d25/f9d [0,4194304] 0 2024-03-26T18:54:40.846 INFO:tasks.workunit.client.0.smithi102.stdout:2/539: dread d3/d82/d93/d9e/fa [0,4194304] 0 2024-03-26T18:54:40.846 INFO:tasks.workunit.client.0.smithi102.stdout:2/540: read d3/d11/d15/d46/d7b/d97/fad [439280,8493] 0 2024-03-26T18:54:40.846 INFO:tasks.workunit.client.0.smithi102.stdout:2/541: fdatasync d3/d11/d8f/fa2 0 2024-03-26T18:54:40.847 INFO:tasks.workunit.client.0.smithi102.stdout:2/542: creat d3/d82/d93/d9e/d1a/fc1 x:0 0 0 2024-03-26T18:54:40.847 INFO:tasks.workunit.client.0.smithi102.stdout:2/543: write d3/d82/d93/d9e/d1a/d25/f9d [849755,93589] 0 2024-03-26T18:54:40.847 INFO:tasks.workunit.client.0.smithi102.stdout:2/544: fsync d3/d20/d78/d84/f98 0 2024-03-26T18:54:40.848 INFO:tasks.workunit.client.1.smithi190.stdout:5/790: dwrite d2/d17/d1b/db6/d88/f8b [0,4194304] 0 2024-03-26T18:54:40.848 INFO:tasks.workunit.client.0.smithi102.stdout:8/612: dwrite d0/d3/d2c/f85 [4194304,4194304] 0 2024-03-26T18:54:40.848 INFO:tasks.workunit.client.0.smithi102.stdout:3/765: dwrite df/d29/d31/d36/d45/f65 [0,4194304] 0 2024-03-26T18:54:40.848 INFO:tasks.workunit.client.0.smithi102.stdout:8/613: truncate d0/d15/f4b 2753028 0 2024-03-26T18:54:40.848 INFO:tasks.workunit.client.0.smithi102.stdout:2/545: mknod d3/d11/d15/cc2 0 2024-03-26T18:54:40.852 INFO:tasks.workunit.client.0.smithi102.stdout:3/766: dread df/d61/d55/f7c [0,4194304] 0 2024-03-26T18:54:40.853 INFO:tasks.workunit.client.0.smithi102.stdout:8/614: link d0/d3/d2c/d2f/fa5 d0/d3/d2c/d4e/fcb 0 2024-03-26T18:54:40.853 INFO:tasks.workunit.client.0.smithi102.stdout:2/546: getdents d3/d74 0 2024-03-26T18:54:40.854 INFO:tasks.workunit.client.0.smithi102.stdout:8/615: mknod d0/d15/d5d/d89/ccc 0 2024-03-26T18:54:40.855 INFO:tasks.workunit.client.0.smithi102.stdout:2/547: dread - d3/f13 zero size 2024-03-26T18:54:40.855 INFO:tasks.workunit.client.0.smithi102.stdout:2/548: fsync d3/d82/d93/d9e/ff 0 2024-03-26T18:54:40.855 INFO:tasks.workunit.client.0.smithi102.stdout:8/616: rmdir d0/d3c/d96/d98 39 2024-03-26T18:54:40.856 INFO:tasks.workunit.client.0.smithi102.stdout:8/617: creat d0/d3/d2c/d33/fcd x:0 0 0 2024-03-26T18:54:40.858 INFO:tasks.workunit.client.0.smithi102.stdout:9/554: dwrite df/d1a/d23/d4e/db0/fc7 [0,4194304] 0 2024-03-26T18:54:40.858 INFO:tasks.workunit.client.0.smithi102.stdout:9/555: write df/d1a/d23/d3e/f99 [424024,103850] 0 2024-03-26T18:54:40.859 INFO:tasks.workunit.client.1.smithi190.stdout:5/791: dread d2/d17/d98/d3a/f4d [0,4194304] 0 2024-03-26T18:54:40.863 INFO:tasks.workunit.client.0.smithi102.stdout:5/818: dwrite d2/d25/d75/db9/db3/fd9 [0,4194304] 0 2024-03-26T18:54:40.864 INFO:tasks.workunit.client.0.smithi102.stdout:5/819: truncate d2/d25/d75/db9/d43/d9e/d8b/fab 6778722 0 2024-03-26T18:54:40.864 INFO:tasks.workunit.client.0.smithi102.stdout:6/568: dwrite d2/d3d/d53/f76 [0,4194304] 0 2024-03-26T18:54:40.866 INFO:tasks.workunit.client.0.smithi102.stdout:3/767: dread df/d1c/fbf [0,4194304] 0 2024-03-26T18:54:40.866 INFO:tasks.workunit.client.0.smithi102.stdout:3/768: chown df/d29/d31/d36/d45/daf 2 1 2024-03-26T18:54:40.869 INFO:tasks.workunit.client.0.smithi102.stdout:8/618: dread d0/d3/d2c/d4e/f52 [4194304,4194304] 0 2024-03-26T18:54:40.869 INFO:tasks.workunit.client.1.smithi190.stdout:5/792: write d2/f10 [1049702,67458] 0 2024-03-26T18:54:40.875 INFO:tasks.workunit.client.0.smithi102.stdout:7/518: rename d7/d10/d84 to d7/d12/d96 0 2024-03-26T18:54:40.876 INFO:tasks.workunit.client.0.smithi102.stdout:5/820: dread d2/d25/d75/db9/f5c [0,4194304] 0 2024-03-26T18:54:40.877 INFO:tasks.workunit.client.0.smithi102.stdout:5/821: fdatasync d2/d25/d75/def/dec/d112/db8/fdf 0 2024-03-26T18:54:40.877 INFO:tasks.workunit.client.0.smithi102.stdout:5/822: dread - d2/dea/d4f/da9/dee/f11b zero size 2024-03-26T18:54:40.877 INFO:tasks.workunit.client.1.smithi190.stdout:5/793: rename d2/d17/d1b/db6/f2d to d2/d17/d98/d3a/d3c/f104 0 2024-03-26T18:54:40.878 INFO:tasks.workunit.client.0.smithi102.stdout:8/619: write d0/d3c/d6f/fab [4191985,84240] 0 2024-03-26T18:54:40.878 INFO:tasks.workunit.client.1.smithi190.stdout:5/794: write d2/d17/d1b/fbe [204718,96770] 0 2024-03-26T18:54:40.882 INFO:tasks.workunit.client.1.smithi190.stdout:5/795: mkdir d2/d105 0 2024-03-26T18:54:40.885 INFO:tasks.workunit.client.1.smithi190.stdout:5/796: rmdir d2/d17/d1b/db6/d24/d3b/da7/dfb 0 2024-03-26T18:54:40.888 INFO:tasks.workunit.client.0.smithi102.stdout:6/569: truncate d2/d7/f86 240919 0 2024-03-26T18:54:40.890 INFO:tasks.workunit.client.0.smithi102.stdout:1/518: rename de/d2e/d62/f6b to de/fac 0 2024-03-26T18:54:40.892 INFO:tasks.workunit.client.0.smithi102.stdout:7/519: write d7/d2a/f4a [3518702,123684] 0 2024-03-26T18:54:40.894 INFO:tasks.workunit.client.0.smithi102.stdout:5/823: link d2/d25/d75/db9/l46 d2/d25/d75/db9/d43/d9e/daf/l120 0 2024-03-26T18:54:40.895 INFO:tasks.workunit.client.0.smithi102.stdout:8/620: symlink d0/d3/d2c/d33/lce 0 2024-03-26T18:54:40.895 INFO:tasks.workunit.client.0.smithi102.stdout:8/621: chown d0/d3/d26/d19/d9f 1 1 2024-03-26T18:54:40.897 INFO:tasks.workunit.client.0.smithi102.stdout:6/570: symlink d2/df/d35/d60/lb4 0 2024-03-26T18:54:40.897 INFO:tasks.workunit.client.0.smithi102.stdout:6/571: truncate d2/faf 38389 0 2024-03-26T18:54:40.898 INFO:tasks.workunit.client.0.smithi102.stdout:4/457: rename d1/d27/c83 to d1/d17/d1a/d21/d77/c8d 0 2024-03-26T18:54:40.899 INFO:tasks.workunit.client.0.smithi102.stdout:1/519: mkdir de/d16/d1d/dad 0 2024-03-26T18:54:40.901 INFO:tasks.workunit.client.0.smithi102.stdout:7/520: symlink d7/d10/d27/l97 0 2024-03-26T18:54:40.902 INFO:tasks.workunit.client.0.smithi102.stdout:5/824: symlink d2/dea/d4f/da9/dee/l121 0 2024-03-26T18:54:40.904 INFO:tasks.workunit.client.0.smithi102.stdout:6/572: getdents d2/df/d35 0 2024-03-26T18:54:40.904 INFO:tasks.workunit.client.0.smithi102.stdout:6/573: truncate d2/d3d/d53/d63/d6a/f6f 783756 0 2024-03-26T18:54:40.907 INFO:tasks.workunit.client.0.smithi102.stdout:2/549: rename d3/d82/d93/d9e/f2b to d3/d82/d93/d9e/da0/fc3 0 2024-03-26T18:54:40.909 INFO:tasks.workunit.client.0.smithi102.stdout:4/458: creat d1/d17/d1f/d5d/d8b/d3e/d4c/f8e x:0 0 0 2024-03-26T18:54:40.910 INFO:tasks.workunit.client.0.smithi102.stdout:1/520: truncate de/d16/d1d/d2a/d2c/d43/f86 449525 0 2024-03-26T18:54:40.912 INFO:tasks.workunit.client.0.smithi102.stdout:7/521: mkdir d7/d10/d67/d69/d98 0 2024-03-26T18:54:40.912 INFO:tasks.workunit.client.0.smithi102.stdout:7/522: dread - d7/d31/f45 zero size 2024-03-26T18:54:40.912 INFO:tasks.workunit.client.0.smithi102.stdout:7/523: creat d7/d12/d4d/d62/f99 x:0 0 0 2024-03-26T18:54:40.912 INFO:tasks.workunit.client.0.smithi102.stdout:7/524: fsync d7/d12/d4d/f8b 0 2024-03-26T18:54:40.914 INFO:tasks.workunit.client.0.smithi102.stdout:8/622: dwrite d0/d3/d2c/da7/f45 [0,4194304] 0 2024-03-26T18:54:40.914 INFO:tasks.workunit.client.0.smithi102.stdout:8/623: write d0/fbf [1147339,61217] 0 2024-03-26T18:54:40.916 INFO:tasks.workunit.client.0.smithi102.stdout:9/556: rename df/d1a/d57/f64 to df/d1a/d1d/d72/fcd 0 2024-03-26T18:54:40.916 INFO:tasks.workunit.client.0.smithi102.stdout:9/557: readlink df/d1a/d1d/l4d 0 2024-03-26T18:54:40.916 INFO:tasks.workunit.client.0.smithi102.stdout:9/558: dread - df/d1a/f66 zero size 2024-03-26T18:54:40.916 INFO:tasks.workunit.client.0.smithi102.stdout:9/559: read - df/d1a/d23/d3e/f9f zero size 2024-03-26T18:54:40.916 INFO:tasks.workunit.client.0.smithi102.stdout:9/560: readlink df/d1a/d23/d4e/d8e/l61 0 2024-03-26T18:54:40.917 INFO:tasks.workunit.client.0.smithi102.stdout:2/550: creat d3/d2a/dae/fc4 x:0 0 0 2024-03-26T18:54:40.917 INFO:tasks.workunit.client.0.smithi102.stdout:2/551: chown d3/f75 25247181 1 2024-03-26T18:54:40.918 INFO:tasks.workunit.client.0.smithi102.stdout:1/521: truncate de/d16/d1d/d61/d6a/f7d 964968 0 2024-03-26T18:54:40.922 INFO:tasks.workunit.client.0.smithi102.stdout:6/574: rmdir d2/d3 39 2024-03-26T18:54:40.922 INFO:tasks.workunit.client.0.smithi102.stdout:6/575: dread - d2/df/d35/da1/fa5 zero size 2024-03-26T18:54:40.922 INFO:tasks.workunit.client.0.smithi102.stdout:6/576: fdatasync d2/d7/d3c/f3e 0 2024-03-26T18:54:40.922 INFO:tasks.workunit.client.0.smithi102.stdout:6/577: creat d2/df/d35/d43/d75/fb5 x:0 0 0 2024-03-26T18:54:40.926 INFO:tasks.workunit.client.0.smithi102.stdout:1/522: dread de/d1f/f53 [0,4194304] 0 2024-03-26T18:54:40.927 INFO:tasks.workunit.client.0.smithi102.stdout:1/523: chown de/d16/d1d/d2a/d2c/d56/c59 102 1 2024-03-26T18:54:40.927 INFO:tasks.workunit.client.0.smithi102.stdout:1/524: write de/d16/d1d/d2a/d2c/d56/d5f/f8b [371967,62294] 0 2024-03-26T18:54:40.927 INFO:tasks.workunit.client.0.smithi102.stdout:4/459: dwrite d1/d17/d1f/d5d/d8b/d3e/f58 [0,4194304] 0 2024-03-26T18:54:40.927 INFO:tasks.workunit.client.0.smithi102.stdout:8/624: dread d0/d15/fb0 [0,4194304] 0 2024-03-26T18:54:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:3/769: rename df/d1c/fe0 to df/d29/d31/d36/d3c/f10f 0 2024-03-26T18:54:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:2/552: mkdir d3/d2d/d67/dc5 0 2024-03-26T18:54:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:1/525: creat de/d16/d1d/d61/fae x:0 0 0 2024-03-26T18:54:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:0/607: sync 2024-03-26T18:54:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:4/460: rmdir d1/d17/d1f/d5d/d8b 39 2024-03-26T18:54:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:4/461: write d1/d27/f55 [3169509,90605] 0 2024-03-26T18:54:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:8/625: mkdir d0/d3c/d96/d98/dcf 0 2024-03-26T18:54:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:8/626: write d0/d3c/d96/d98/db6/f40 [3408382,80253] 0 2024-03-26T18:54:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:7/525: rename d7/d12/d4b/l85 to d7/d10/d67/d69/l9a 0 2024-03-26T18:54:40.939 INFO:tasks.workunit.client.0.smithi102.stdout:2/553: mkdir d3/d20/d78/d84/d6e/dc6 0 2024-03-26T18:54:40.940 INFO:tasks.workunit.client.0.smithi102.stdout:1/526: write de/d16/d1d/d2a/d2c/d56/d6e/f7a [2757348,70313] 0 2024-03-26T18:54:40.942 INFO:tasks.workunit.client.0.smithi102.stdout:0/608: unlink d6/d2d/d3d/fb4 0 2024-03-26T18:54:40.944 INFO:tasks.workunit.client.0.smithi102.stdout:4/462: symlink d1/d6b/l8f 0 2024-03-26T18:54:40.947 INFO:tasks.workunit.client.0.smithi102.stdout:9/561: rename df/d1a/d1d/f32 to df/d1a/d23/d4e/d8e/fce 0 2024-03-26T18:54:40.948 INFO:tasks.workunit.client.0.smithi102.stdout:3/770: symlink df/d29/d3a/l110 0 2024-03-26T18:54:40.948 INFO:tasks.workunit.client.0.smithi102.stdout:3/771: fdatasync df/d29/d31/d36/d45/fd8 0 2024-03-26T18:54:40.949 INFO:tasks.workunit.client.0.smithi102.stdout:2/554: stat d3/d11/d15/d31/d39/c8a 0 2024-03-26T18:54:40.950 INFO:tasks.workunit.client.0.smithi102.stdout:3/772: write df/d29/d31/d77/d8a/dae/fc5 [779617,64589] 0 2024-03-26T18:54:40.953 INFO:tasks.workunit.client.0.smithi102.stdout:0/609: link d6/de/d13/d25/c41 d6/d3e/d69/cce 0 2024-03-26T18:54:40.953 INFO:tasks.workunit.client.0.smithi102.stdout:0/610: creat d6/fcf x:0 0 0 2024-03-26T18:54:40.955 INFO:tasks.workunit.client.1.smithi190.stdout:3/745: sync 2024-03-26T18:54:40.955 INFO:tasks.workunit.client.0.smithi102.stdout:8/627: rmdir d0/d3/d2c/d58 39 2024-03-26T18:54:40.955 INFO:tasks.workunit.client.0.smithi102.stdout:8/628: stat d0/d3/d2c/d4e/f52 0 2024-03-26T18:54:40.955 INFO:tasks.workunit.client.0.smithi102.stdout:8/629: chown d0/d3/d2c/da7/f73 1656116 1 2024-03-26T18:54:40.955 INFO:tasks.workunit.client.0.smithi102.stdout:8/630: creat d0/d15/fd0 x:0 0 0 2024-03-26T18:54:40.958 INFO:tasks.workunit.client.0.smithi102.stdout:6/578: rename d2/df/f64 to d2/d3/fb6 0 2024-03-26T18:54:40.959 INFO:tasks.workunit.client.0.smithi102.stdout:2/555: mknod d3/d11/d15/d31/d39/cc7 0 2024-03-26T18:54:40.961 INFO:tasks.workunit.client.0.smithi102.stdout:3/773: mknod df/d29/d31/d36/c111 0 2024-03-26T18:54:40.966 INFO:tasks.workunit.client.0.smithi102.stdout:7/526: dwrite d7/d12/f43 [0,4194304] 0 2024-03-26T18:54:40.967 INFO:tasks.workunit.client.0.smithi102.stdout:4/463: getdents d1/d25/d76 0 2024-03-26T18:54:40.968 INFO:tasks.workunit.client.0.smithi102.stdout:9/562: dwrite df/d1a/d23/d29/f34 [0,4194304] 0 2024-03-26T18:54:40.973 INFO:tasks.workunit.client.0.smithi102.stdout:1/527: rename de/d16/d1d/d2a/d2c/f9b to de/d16/d1d/d61/d6a/da5/da6/faf 0 2024-03-26T18:54:40.973 INFO:tasks.workunit.client.1.smithi190.stdout:3/746: dread d1/de/d1c/d36/f3a [0,4194304] 0 2024-03-26T18:54:40.974 INFO:tasks.workunit.client.0.smithi102.stdout:2/556: link d3/d82/d93/d9e/d1a/d25/c52 d3/d11/d15/d31/cc8 0 2024-03-26T18:54:40.974 INFO:tasks.workunit.client.0.smithi102.stdout:2/557: truncate d3/d2d/d69/f6a 1005716 0 2024-03-26T18:54:40.974 INFO:tasks.workunit.client.0.smithi102.stdout:2/558: readlink d3/d11/d15/d31/l45 0 2024-03-26T18:54:40.975 INFO:tasks.workunit.client.0.smithi102.stdout:0/611: unlink d6/fcf 0 2024-03-26T18:54:40.977 INFO:tasks.workunit.client.0.smithi102.stdout:4/464: mkdir d1/d25/d90 0 2024-03-26T18:54:40.977 INFO:tasks.workunit.client.0.smithi102.stdout:4/465: chown d1/d27/c46 48191410 1 2024-03-26T18:54:40.978 INFO:tasks.workunit.client.1.smithi190.stdout:1/766: sync 2024-03-26T18:54:40.978 INFO:tasks.workunit.client.1.smithi190.stdout:6/750: sync 2024-03-26T18:54:40.978 INFO:tasks.workunit.client.1.smithi190.stdout:2/944: sync 2024-03-26T18:54:40.978 INFO:tasks.workunit.client.1.smithi190.stdout:3/747: dread d1/f2c [0,4194304] 0 2024-03-26T18:54:40.983 INFO:tasks.workunit.client.0.smithi102.stdout:8/631: dwrite d0/d3/d26/d5c/f62 [4194304,4194304] 0 2024-03-26T18:54:40.983 INFO:tasks.workunit.client.0.smithi102.stdout:3/774: dwrite df/d29/d31/d36/d3c/d6f/d75/f94 [0,4194304] 0 2024-03-26T18:54:40.983 INFO:tasks.workunit.client.0.smithi102.stdout:3/775: stat df/d29/d3a/d93/ff1 0 2024-03-26T18:54:40.983 INFO:tasks.workunit.client.0.smithi102.stdout:3/776: write df/d29/d31/d36/d45/f65 [4370298,54845] 0 2024-03-26T18:54:40.983 INFO:tasks.workunit.client.1.smithi190.stdout:1/767: truncate d3/d9/f16 7080120 0 2024-03-26T18:54:40.984 INFO:tasks.workunit.client.1.smithi190.stdout:2/945: rename d6/d9/d45/d6b/deb/f131 to d6/d9/d3a/d3d/d1c/d8d/dc7/d72/f142 0 2024-03-26T18:54:40.984 INFO:tasks.workunit.client.1.smithi190.stdout:2/946: readlink d6/d9/d3a/d3d/d2e/d10e/d9e/dfc/l141 0 2024-03-26T18:54:40.984 INFO:tasks.workunit.client.1.smithi190.stdout:2/947: rename d6/c3e to d6/d9/d10/d5f/c143 0 2024-03-26T18:54:40.984 INFO:tasks.workunit.client.1.smithi190.stdout:6/751: symlink d17/d28/lfd 0 2024-03-26T18:54:40.984 INFO:tasks.workunit.client.1.smithi190.stdout:6/752: fsync d17/d28/d2c/d77/f6f 0 2024-03-26T18:54:40.985 INFO:tasks.workunit.client.0.smithi102.stdout:6/579: rename d2/d3d/d53/d63/d6a to d2/df/d72/db7 0 2024-03-26T18:54:40.987 INFO:tasks.workunit.client.0.smithi102.stdout:1/528: mknod de/d16/d1d/d2a/d2c/d56/d78/cb0 0 2024-03-26T18:54:40.987 INFO:tasks.workunit.client.0.smithi102.stdout:1/529: truncate de/d16/d1d/d61/d6a/f7d 1177836 0 2024-03-26T18:54:40.989 INFO:tasks.workunit.client.0.smithi102.stdout:2/559: truncate d3/d2a/f37 3170024 0 2024-03-26T18:54:40.995 INFO:tasks.workunit.client.1.smithi190.stdout:1/768: creat d3/d73/d88/df1/ff7 x:0 0 0 2024-03-26T18:54:40.998 INFO:tasks.workunit.client.0.smithi102.stdout:4/466: dread d1/d17/d2a/f54 [0,4194304] 0 2024-03-26T18:54:40.998 INFO:tasks.workunit.client.0.smithi102.stdout:4/467: dread - d1/d17/d1f/d5d/d8b/d3e/d4c/f8e zero size 2024-03-26T18:54:40.998 INFO:tasks.workunit.client.0.smithi102.stdout:0/612: creat d6/de/fd0 x:0 0 0 2024-03-26T18:54:40.998 INFO:tasks.workunit.client.0.smithi102.stdout:7/527: rmdir d7/d12/d4b/d83 39 2024-03-26T18:54:41.004 INFO:tasks.workunit.client.1.smithi190.stdout:1/769: rename d3/d73/d88/fc3 to d3/d73/d88/dcc/ff8 0 2024-03-26T18:54:41.009 INFO:tasks.workunit.client.1.smithi190.stdout:1/770: unlink d3/da/d3a/ca1 0 2024-03-26T18:54:41.011 INFO:tasks.workunit.client.1.smithi190.stdout:1/771: rmdir d3/da/dd 39 2024-03-26T18:54:41.012 INFO:tasks.workunit.client.1.smithi190.stdout:1/772: creat d3/da/d2e/d59/dae/ff9 x:0 0 0 2024-03-26T18:54:41.015 INFO:tasks.workunit.client.0.smithi102.stdout:3/777: creat df/d1c/d34/da5/dce/f112 x:0 0 0 2024-03-26T18:54:41.015 INFO:tasks.workunit.client.0.smithi102.stdout:3/778: truncate df/d1c/d70/f87 4386538 0 2024-03-26T18:54:41.015 INFO:tasks.workunit.client.1.smithi190.stdout:6/753: dwrite d17/d28/d2b/fc4 [0,4194304] 0 2024-03-26T18:54:41.018 INFO:tasks.workunit.client.1.smithi190.stdout:1/773: creat d3/d9/ffa x:0 0 0 2024-03-26T18:54:41.019 INFO:tasks.workunit.client.1.smithi190.stdout:2/948: dwrite d6/d21/dd8/f11b [0,4194304] 0 2024-03-26T18:54:41.020 INFO:tasks.workunit.client.0.smithi102.stdout:2/560: dwrite d3/d2d/d69/fbd [0,4194304] 0 2024-03-26T18:54:41.021 INFO:tasks.workunit.client.1.smithi190.stdout:6/754: link d17/d28/d2b/f3c d17/d28/d8b/d95/ffe 0 2024-03-26T18:54:41.021 INFO:tasks.workunit.client.1.smithi190.stdout:2/949: creat d6/d9/d3a/d3d/f144 x:0 0 0 2024-03-26T18:54:41.021 INFO:tasks.workunit.client.1.smithi190.stdout:2/950: write d6/d9/d10/db7/f49 [33193,74924] 0 2024-03-26T18:54:41.023 INFO:tasks.workunit.client.1.smithi190.stdout:1/774: creat d3/da/dd/d95/d52/ffb x:0 0 0 2024-03-26T18:54:41.023 INFO:tasks.workunit.client.1.smithi190.stdout:2/951: rename d6/f7 to d6/d9/d10/db7/dcf/f145 0 2024-03-26T18:54:41.023 INFO:tasks.workunit.client.1.smithi190.stdout:2/952: chown d6/d9/d10/d5f/f10d 466414796 1 2024-03-26T18:54:41.024 INFO:tasks.workunit.client.0.smithi102.stdout:4/468: dwrite d1/d17/d1f/d5d/d8b/f4e [0,4194304] 0 2024-03-26T18:54:41.024 INFO:tasks.workunit.client.0.smithi102.stdout:4/469: read - d1/d25/d76/f86 zero size 2024-03-26T18:54:41.024 INFO:tasks.workunit.client.1.smithi190.stdout:2/953: mknod d6/d9/d3a/d3d/d1c/d8d/de8/c146 0 2024-03-26T18:54:41.027 INFO:tasks.workunit.client.1.smithi190.stdout:6/755: creat d17/d28/d38/fff x:0 0 0 2024-03-26T18:54:41.027 INFO:tasks.workunit.client.1.smithi190.stdout:6/756: readlink d17/d28/d2c/db8/l68 0 2024-03-26T18:54:41.027 INFO:tasks.workunit.client.1.smithi190.stdout:6/757: readlink d17/d28/d8b/d95/dda/lee 0 2024-03-26T18:54:41.027 INFO:tasks.workunit.client.1.smithi190.stdout:6/758: readlink d17/d28/d38/l4f 0 2024-03-26T18:54:41.029 INFO:tasks.workunit.client.1.smithi190.stdout:1/775: link d3/d73/faa d3/d73/ffc 0 2024-03-26T18:54:41.033 INFO:tasks.workunit.client.1.smithi190.stdout:6/759: mknod d17/d28/d38/c100 0 2024-03-26T18:54:41.033 INFO:tasks.workunit.client.1.smithi190.stdout:6/760: write d17/d28/df6/da8/df4/dfb/f5c [2870117,8342] 0 2024-03-26T18:54:41.035 INFO:tasks.workunit.client.0.smithi102.stdout:6/580: mkdir d2/df/d71/d80/d91/db8 0 2024-03-26T18:54:41.035 INFO:tasks.workunit.client.0.smithi102.stdout:6/581: write d2/df/f96 [178602,104243] 0 2024-03-26T18:54:41.035 INFO:tasks.workunit.client.0.smithi102.stdout:6/582: write d2/d3d/fb0 [1750825,121836] 0 2024-03-26T18:54:41.037 INFO:tasks.workunit.client.0.smithi102.stdout:1/530: getdents de/d16/d1d/d2a/d2c/d56/d78 0 2024-03-26T18:54:41.038 INFO:tasks.workunit.client.0.smithi102.stdout:1/531: chown de/d16/c50 4618 1 2024-03-26T18:54:41.038 INFO:tasks.workunit.client.0.smithi102.stdout:1/532: write de/f2d [895430,96412] 0 2024-03-26T18:54:41.041 INFO:tasks.workunit.client.0.smithi102.stdout:0/613: creat d6/d2d/d3d/d80/d83/fd1 x:0 0 0 2024-03-26T18:54:41.042 INFO:tasks.workunit.client.1.smithi190.stdout:1/776: dread d3/da/dd/d2f/d5b/f70 [0,4194304] 0 2024-03-26T18:54:41.042 INFO:tasks.workunit.client.0.smithi102.stdout:7/528: unlink d7/d10/d17/l5a 0 2024-03-26T18:54:41.043 INFO:tasks.workunit.client.1.smithi190.stdout:1/777: write d3/d9/db/dd4/dd7/fdb [745550,69183] 0 2024-03-26T18:54:41.045 INFO:tasks.workunit.client.0.smithi102.stdout:2/561: creat d3/d11/fc9 x:0 0 0 2024-03-26T18:54:41.045 INFO:tasks.workunit.client.0.smithi102.stdout:2/562: chown d3/d2d/d69/f6a 0 1 2024-03-26T18:54:41.048 INFO:tasks.workunit.client.0.smithi102.stdout:4/470: unlink d1/d17/d1a/d21/c4a 0 2024-03-26T18:54:41.052 INFO:tasks.workunit.client.0.smithi102.stdout:1/533: symlink de/d16/d1d/d61/d6a/da5/lb1 0 2024-03-26T18:54:41.055 INFO:tasks.workunit.client.0.smithi102.stdout:0/614: mkdir d6/de/d13/d25/d31/d45/d82/d84/d88/db2/dd2 0 2024-03-26T18:54:41.058 INFO:tasks.workunit.client.1.smithi190.stdout:2/954: dwrite d6/d9/d48/da0/f110 [0,4194304] 0 2024-03-26T18:54:41.058 INFO:tasks.workunit.client.1.smithi190.stdout:2/955: dread - d6/d9/d3a/d3d/d2e/d10e/d9e/da3/f11f zero size 2024-03-26T18:54:41.058 INFO:tasks.workunit.client.1.smithi190.stdout:2/956: fdatasync d6/d21/d61/fc2 0 2024-03-26T18:54:41.058 INFO:tasks.workunit.client.1.smithi190.stdout:2/957: truncate d6/d9/d45/f74 3681292 0 2024-03-26T18:54:41.058 INFO:tasks.workunit.client.1.smithi190.stdout:2/958: chown d6/d9/d3a/d3d/d1c/l25 376066 1 2024-03-26T18:54:41.059 INFO:tasks.workunit.client.0.smithi102.stdout:7/529: mkdir d7/d10/d67/d69/d9b 0 2024-03-26T18:54:41.060 INFO:tasks.workunit.client.1.smithi190.stdout:6/761: dwrite d17/d28/d2c/d77/fe3 [0,4194304] 0 2024-03-26T18:54:41.060 INFO:tasks.workunit.client.1.smithi190.stdout:6/762: readlink d17/l62 0 2024-03-26T18:54:41.060 INFO:tasks.workunit.client.0.smithi102.stdout:2/563: mkdir d3/d20/d78/d7c/dca 0 2024-03-26T18:54:41.061 INFO:tasks.workunit.client.0.smithi102.stdout:2/564: truncate d3/d20/d78/d84/f98 54482 0 2024-03-26T18:54:41.061 INFO:tasks.workunit.client.0.smithi102.stdout:2/565: readlink d3/d11/d27/l56 0 2024-03-26T18:54:41.061 INFO:tasks.workunit.client.0.smithi102.stdout:2/566: chown d3/d11/d15/d46/d7b/d97 46459404 1 2024-03-26T18:54:41.061 INFO:tasks.workunit.client.0.smithi102.stdout:2/567: creat d3/d82/d93/fcb x:0 0 0 2024-03-26T18:54:41.061 INFO:tasks.workunit.client.0.smithi102.stdout:2/568: fsync d3/d11/d15/d46/d7b/d97/fbb 0 2024-03-26T18:54:41.061 INFO:tasks.workunit.client.0.smithi102.stdout:4/471: mknod d1/d25/c91 0 2024-03-26T18:54:41.063 INFO:tasks.workunit.client.0.smithi102.stdout:0/615: mkdir d6/de/d13/d1f/d55/d57/d7d/d7f/dc8/dd3 0 2024-03-26T18:54:41.067 INFO:tasks.workunit.client.1.smithi190.stdout:6/763: mkdir d17/d28/d8b/d101 0 2024-03-26T18:54:41.069 INFO:tasks.workunit.client.0.smithi102.stdout:2/569: dread d3/f1c [0,4194304] 0 2024-03-26T18:54:41.070 INFO:tasks.workunit.client.0.smithi102.stdout:4/472: symlink d1/d25/d90/l92 0 2024-03-26T18:54:41.070 INFO:tasks.workunit.client.0.smithi102.stdout:4/473: readlink d1/d17/d1a/d21/l72 0 2024-03-26T18:54:41.071 INFO:tasks.workunit.client.0.smithi102.stdout:0/616: truncate d6/d2d/d3d/f40 1744793 0 2024-03-26T18:54:41.074 INFO:tasks.workunit.client.0.smithi102.stdout:4/474: mknod d1/d17/d1f/d5d/d8b/d3e/d74/c93 0 2024-03-26T18:54:41.076 INFO:tasks.workunit.client.0.smithi102.stdout:0/617: mknod d6/de/d13/d1f/cd4 0 2024-03-26T18:54:41.077 INFO:tasks.workunit.client.1.smithi190.stdout:1/778: dwrite d3/da/dd/d2f/d33/f8a [0,4194304] 0 2024-03-26T18:54:41.077 INFO:tasks.workunit.client.1.smithi190.stdout:1/779: truncate d3/da/dd/d95/d52/fd8 691957 0 2024-03-26T18:54:41.077 INFO:tasks.workunit.client.0.smithi102.stdout:7/530: rmdir d7/d10/d17 39 2024-03-26T18:54:41.079 INFO:tasks.workunit.client.1.smithi190.stdout:1/780: symlink d3/d73/d88/df1/lfd 0 2024-03-26T18:54:41.080 INFO:tasks.workunit.client.1.smithi190.stdout:1/781: write d3/da/d2e/d59/dae/fcb [4368118,40683] 0 2024-03-26T18:54:41.080 INFO:tasks.workunit.client.1.smithi190.stdout:1/782: dread - d3/da/dd/d76/fa6 zero size 2024-03-26T18:54:41.080 INFO:tasks.workunit.client.1.smithi190.stdout:1/783: creat d3/da/ffe x:0 0 0 2024-03-26T18:54:41.080 INFO:tasks.workunit.client.0.smithi102.stdout:7/531: creat d7/d10/d67/d69/f9c x:0 0 0 2024-03-26T18:54:41.081 INFO:tasks.workunit.client.0.smithi102.stdout:2/570: read d3/d20/d78/d84/d6e/fb1 [102361,107492] 0 2024-03-26T18:54:41.081 INFO:tasks.workunit.client.0.smithi102.stdout:2/571: stat d3/f75 0 2024-03-26T18:54:41.081 INFO:tasks.workunit.client.0.smithi102.stdout:2/572: creat d3/d82/d93/fcc x:0 0 0 2024-03-26T18:54:41.082 INFO:tasks.workunit.client.0.smithi102.stdout:7/532: mkdir d7/d2a/d9d 0 2024-03-26T18:54:41.082 INFO:tasks.workunit.client.0.smithi102.stdout:7/533: write d7/f7b [1933836,45492] 0 2024-03-26T18:54:41.082 INFO:tasks.workunit.client.0.smithi102.stdout:7/534: stat d7/d10/d17/f81 0 2024-03-26T18:54:41.083 INFO:tasks.workunit.client.1.smithi190.stdout:1/784: mknod d3/da/dd/d2f/cff 0 2024-03-26T18:54:41.083 INFO:tasks.workunit.client.1.smithi190.stdout:1/785: dread - d3/da/dd/d95/d52/ffb zero size 2024-03-26T18:54:41.084 INFO:tasks.workunit.client.0.smithi102.stdout:2/573: creat d3/d82/d93/d9e/d1a/d25/fcd x:0 0 0 2024-03-26T18:54:41.084 INFO:tasks.workunit.client.0.smithi102.stdout:2/574: truncate d3/d11/d15/d31/d39/f4e 83240 0 2024-03-26T18:54:41.085 INFO:tasks.workunit.client.1.smithi190.stdout:1/786: mknod d3/da/d2e/d59/dae/c100 0 2024-03-26T18:54:41.085 INFO:tasks.workunit.client.1.smithi190.stdout:1/787: readlink d3/da/dd/d95/d52/l6d 0 2024-03-26T18:54:41.086 INFO:tasks.workunit.client.0.smithi102.stdout:1/534: dwrite de/d16/d1d/d2a/d67/f95 [0,4194304] 0 2024-03-26T18:54:41.086 INFO:tasks.workunit.client.0.smithi102.stdout:1/535: truncate de/d16/d1d/d2a/f74 1201187 0 2024-03-26T18:54:41.087 INFO:tasks.workunit.client.1.smithi190.stdout:2/959: dwrite d6/d9/d10/db7/f49 [0,4194304] 0 2024-03-26T18:54:41.087 INFO:tasks.workunit.client.1.smithi190.stdout:1/788: rmdir d3/da/dd/d2f/d33/de8/df5 0 2024-03-26T18:54:41.088 INFO:tasks.workunit.client.1.smithi190.stdout:6/764: dwrite d17/d28/d2c/d77/f71 [0,4194304] 0 2024-03-26T18:54:41.088 INFO:tasks.workunit.client.1.smithi190.stdout:6/765: creat d17/d28/d74/d97/f102 x:0 0 0 2024-03-26T18:54:41.090 INFO:tasks.workunit.client.1.smithi190.stdout:1/789: creat d3/da/daf/f101 x:0 0 0 2024-03-26T18:54:41.091 INFO:tasks.workunit.client.0.smithi102.stdout:0/618: dread d6/fd [4194304,4194304] 0 2024-03-26T18:54:41.091 INFO:tasks.workunit.client.0.smithi102.stdout:0/619: write d6/de/d13/d1f/d55/d57/f1c [2086261,35148] 0 2024-03-26T18:54:41.091 INFO:tasks.workunit.client.0.smithi102.stdout:0/620: creat d6/de/d13/d1f/d55/d57/d7d/fd5 x:0 0 0 2024-03-26T18:54:41.092 INFO:tasks.workunit.client.0.smithi102.stdout:1/536: creat de/d16/d1d/d2a/fb2 x:0 0 0 2024-03-26T18:54:41.092 INFO:tasks.workunit.client.0.smithi102.stdout:1/537: chown de/fac 102 1 2024-03-26T18:54:41.096 INFO:tasks.workunit.client.0.smithi102.stdout:2/575: write d3/d2d/f68 [547145,11253] 0 2024-03-26T18:54:41.098 INFO:tasks.workunit.client.0.smithi102.stdout:0/621: symlink d6/ld6 0 2024-03-26T18:54:41.098 INFO:tasks.workunit.client.1.smithi190.stdout:2/960: dread d6/dbc/f105 [0,4194304] 0 2024-03-26T18:54:41.099 INFO:tasks.workunit.client.1.smithi190.stdout:6/766: creat d17/d28/d8b/d101/f103 x:0 0 0 2024-03-26T18:54:41.100 INFO:tasks.workunit.client.0.smithi102.stdout:1/538: rename fc to de/d16/d1d/d2a/d2c/d56/fb3 0 2024-03-26T18:54:41.102 INFO:tasks.workunit.client.1.smithi190.stdout:1/790: creat d3/da/d3a/f102 x:0 0 0 2024-03-26T18:54:41.104 INFO:tasks.workunit.client.0.smithi102.stdout:2/576: symlink d3/d20/d78/d84/lce 0 2024-03-26T18:54:41.104 INFO:tasks.workunit.client.0.smithi102.stdout:2/577: creat d3/d2d/d69/fcf x:0 0 0 2024-03-26T18:54:41.106 INFO:tasks.workunit.client.0.smithi102.stdout:0/622: rename d6/de/d13/d9a/f99 to d6/de/d13/d25/d31/fd7 0 2024-03-26T18:54:41.109 INFO:tasks.workunit.client.0.smithi102.stdout:1/539: creat de/d16/d1d/d2a/d2c/d56/d5f/daa/fb4 x:0 0 0 2024-03-26T18:54:41.110 INFO:tasks.workunit.client.1.smithi190.stdout:6/767: mknod d17/d28/d2c/d63/c104 0 2024-03-26T18:54:41.113 INFO:tasks.workunit.client.1.smithi190.stdout:1/791: rename d3/d9/db/fc1 to d3/da/d2e/d59/f103 0 2024-03-26T18:54:41.116 INFO:tasks.workunit.client.1.smithi190.stdout:6/768: mknod d17/d28/df6/da8/c105 0 2024-03-26T18:54:41.117 INFO:tasks.workunit.client.1.smithi190.stdout:6/769: dread - d17/d28/df6/fd8 zero size 2024-03-26T18:54:41.117 INFO:tasks.workunit.client.0.smithi102.stdout:2/578: dread d3/d2d/d69/fbd [0,4194304] 0 2024-03-26T18:54:41.117 INFO:tasks.workunit.client.0.smithi102.stdout:2/579: chown d3/d82/d93/d9e/d1a 2 1 2024-03-26T18:54:41.118 INFO:tasks.workunit.client.0.smithi102.stdout:1/540: getdents de/d16/d1d/d2a/d2c 0 2024-03-26T18:54:41.118 INFO:tasks.workunit.client.1.smithi190.stdout:2/961: dwrite d6/d9/d3a/d3d/d2e/d31/f91 [0,4194304] 0 2024-03-26T18:54:41.119 INFO:tasks.workunit.client.1.smithi190.stdout:6/770: truncate d17/d28/d2c/d50/f9c 3310290 0 2024-03-26T18:54:41.120 INFO:tasks.workunit.client.1.smithi190.stdout:2/962: rmdir d6/d9/d3a/d3d/d2e/d10e/d9e/d10a 39 2024-03-26T18:54:41.120 INFO:tasks.workunit.client.1.smithi190.stdout:2/963: write d6/d9/d48/da0/fa7 [5152636,25715] 0 2024-03-26T18:54:41.134 INFO:tasks.workunit.client.1.smithi190.stdout:0/967: sync 2024-03-26T18:54:41.134 INFO:tasks.workunit.client.1.smithi190.stdout:5/797: sync 2024-03-26T18:54:41.136 INFO:tasks.workunit.client.1.smithi190.stdout:0/968: mkdir d0/dd/d20/d1c/d46/d58/de4/d12e/d15b 0 2024-03-26T18:54:41.136 INFO:tasks.workunit.client.1.smithi190.stdout:0/969: write d0/dd/d20/d61/fac [2912320,8448] 0 2024-03-26T18:54:41.136 INFO:tasks.workunit.client.1.smithi190.stdout:5/798: dread d2/d17/d62/fad [0,4194304] 0 2024-03-26T18:54:41.136 INFO:tasks.workunit.client.1.smithi190.stdout:5/799: write d2/d17/d1b/d60/f7c [260155,41886] 0 2024-03-26T18:54:41.136 INFO:tasks.workunit.client.1.smithi190.stdout:5/800: chown d2/d17/d1b/db6/d24/d3b/da7 26 1 2024-03-26T18:54:41.136 INFO:tasks.workunit.client.1.smithi190.stdout:5/801: creat d2/d17/ddd/f106 x:0 0 0 2024-03-26T18:54:41.139 INFO:tasks.workunit.client.1.smithi190.stdout:0/970: unlink d0/dd/d20/d1c/ld6 0 2024-03-26T18:54:41.139 INFO:tasks.workunit.client.1.smithi190.stdout:5/802: mkdir d2/d17/d1b/db6/d24/d3b/da5/d107 0 2024-03-26T18:54:41.140 INFO:tasks.workunit.client.1.smithi190.stdout:5/803: creat d2/d17/d1b/db6/d24/f108 x:0 0 0 2024-03-26T18:54:41.142 INFO:tasks.workunit.client.1.smithi190.stdout:5/804: mkdir d2/d17/d62/d109 0 2024-03-26T18:54:41.147 INFO:tasks.workunit.client.1.smithi190.stdout:6/771: dread d17/d28/d2c/d50/fce [0,4194304] 0 2024-03-26T18:54:41.148 INFO:tasks.workunit.client.1.smithi190.stdout:5/805: write d2/d17/f4f [242778,73071] 0 2024-03-26T18:54:41.148 INFO:tasks.workunit.client.1.smithi190.stdout:5/806: chown d2/d17/d1b/f36 56738669 1 2024-03-26T18:54:41.153 INFO:tasks.workunit.client.0.smithi102.stdout:2/580: dwrite d3/d82/d93/d9e/fa [0,4194304] 0 2024-03-26T18:54:41.154 INFO:tasks.workunit.client.0.smithi102.stdout:0/623: dwrite d6/de/d13/d1f/f27 [4194304,4194304] 0 2024-03-26T18:54:41.154 INFO:tasks.workunit.client.0.smithi102.stdout:2/581: chown d3/d11/d15/cc2 4 1 2024-03-26T18:54:41.154 INFO:tasks.workunit.client.0.smithi102.stdout:1/541: dwrite de/f32 [0,4194304] 0 2024-03-26T18:54:41.154 INFO:tasks.workunit.client.0.smithi102.stdout:1/542: truncate de/f2d 1385735 0 2024-03-26T18:54:41.154 INFO:tasks.workunit.client.1.smithi190.stdout:0/971: dread d0/dd/d20/d1c/d46/da7/df0/f7d [0,4194304] 0 2024-03-26T18:54:41.158 INFO:tasks.workunit.client.1.smithi190.stdout:1/792: dwrite d3/da/d3a/f50 [0,4194304] 0 2024-03-26T18:54:41.158 INFO:tasks.workunit.client.1.smithi190.stdout:2/964: dread d6/d9/d48/da0/f110 [0,4194304] 0 2024-03-26T18:54:41.158 INFO:tasks.workunit.client.1.smithi190.stdout:2/965: chown d6/d9/d10/d5f/cce 10265 1 2024-03-26T18:54:41.158 INFO:tasks.workunit.client.1.smithi190.stdout:1/793: write d3/da/dd/d2f/d33/d9c/fb7 [798855,51954] 0 2024-03-26T18:54:41.158 INFO:tasks.workunit.client.1.smithi190.stdout:0/972: unlink d0/dd/fe5 0 2024-03-26T18:54:41.159 INFO:tasks.workunit.client.0.smithi102.stdout:1/543: dread de/d16/d1d/f28 [0,4194304] 0 2024-03-26T18:54:41.159 INFO:tasks.workunit.client.0.smithi102.stdout:1/544: readlink de/d1f/l71 0 2024-03-26T18:54:41.160 INFO:tasks.workunit.client.1.smithi190.stdout:2/966: getdents d6/d9/d10 0 2024-03-26T18:54:41.163 INFO:tasks.workunit.client.1.smithi190.stdout:2/967: rename d6/d9/d10/db7/dcf/f145 to d6/d9/d3a/d3d/d1c/d8d/dc7/d72/db1/f147 0 2024-03-26T18:54:41.163 INFO:tasks.workunit.client.1.smithi190.stdout:2/968: read d6/d9/d48/da0/fa7 [3548224,89396] 0 2024-03-26T18:54:41.165 INFO:tasks.workunit.client.1.smithi190.stdout:2/969: symlink d6/d9/d45/df3/l148 0 2024-03-26T18:54:41.165 INFO:tasks.workunit.client.1.smithi190.stdout:2/970: chown d6/d9/d3a/d3d/d1c/d8d/dc7/d72/d8b 117 1 2024-03-26T18:54:41.166 INFO:tasks.workunit.client.1.smithi190.stdout:2/971: write d6/d9/d3a/d3d/d55/f116 [430870,114513] 0 2024-03-26T18:54:41.166 INFO:tasks.workunit.client.1.smithi190.stdout:2/972: write d6/d9/d3a/d3d/d1c/d8d/dc7/f64 [211107,114904] 0 2024-03-26T18:54:41.166 INFO:tasks.workunit.client.1.smithi190.stdout:2/973: fsync d6/d21/d61/d6c/db4/fed 0 2024-03-26T18:54:41.168 INFO:tasks.workunit.client.1.smithi190.stdout:2/974: unlink d6/d9/d45/d6b/d93/dc0/ff2 0 2024-03-26T18:54:41.169 INFO:tasks.workunit.client.1.smithi190.stdout:2/975: chown d6/d9/d3a/cc9 1064526 1 2024-03-26T18:54:41.182 INFO:tasks.workunit.client.1.smithi190.stdout:5/807: dwrite d2/d17/d1b/db6/d24/d3b/da7/fc0 [0,4194304] 0 2024-03-26T18:54:41.183 INFO:tasks.workunit.client.1.smithi190.stdout:6/772: dwrite d17/d28/d8b/d101/f103 [0,4194304] 0 2024-03-26T18:54:41.193 INFO:tasks.workunit.client.0.smithi102.stdout:1/545: dwrite de/d16/d1d/f1e [4194304,4194304] 0 2024-03-26T18:54:41.196 INFO:tasks.workunit.client.1.smithi190.stdout:2/976: dwrite d6/d9/d3a/d3d/d2e/d31/f91 [0,4194304] 0 2024-03-26T18:54:41.196 INFO:tasks.workunit.client.1.smithi190.stdout:2/977: chown d6/d9/d10/db7/dcf 1 1 2024-03-26T18:54:41.198 INFO:tasks.workunit.client.1.smithi190.stdout:2/978: dread d6/d21/fb5 [0,4194304] 0 2024-03-26T18:54:41.198 INFO:tasks.workunit.client.1.smithi190.stdout:2/979: chown d6/d9/d10/f59 18 1 2024-03-26T18:54:41.198 INFO:tasks.workunit.client.1.smithi190.stdout:2/980: chown d6/d9/d10/db7/dcf/f101 6107 1 2024-03-26T18:54:41.201 INFO:tasks.workunit.client.1.smithi190.stdout:2/981: unlink d6/d9/d3a/d3d/d2e/d10e/f12f 0 2024-03-26T18:54:41.202 INFO:tasks.workunit.client.1.smithi190.stdout:2/982: creat d6/d9/d3a/d3d/d1c/f149 x:0 0 0 2024-03-26T18:54:41.202 INFO:tasks.workunit.client.1.smithi190.stdout:2/983: creat d6/d21/d61/f14a x:0 0 0 2024-03-26T18:54:41.209 INFO:tasks.workunit.client.0.smithi102.stdout:0/624: symlink d6/dcb/ld8 0 2024-03-26T18:54:41.211 INFO:tasks.workunit.client.0.smithi102.stdout:2/582: mknod d3/cd0 0 2024-03-26T18:54:41.213 INFO:tasks.workunit.client.1.smithi190.stdout:2/984: dread d6/d9/d3a/d3d/d2e/d31/f91 [0,4194304] 0 2024-03-26T18:54:41.213 INFO:tasks.workunit.client.1.smithi190.stdout:2/985: readlink d6/lef 0 2024-03-26T18:54:41.217 INFO:tasks.workunit.client.1.smithi190.stdout:1/794: creat d3/d73/d88/f104 x:0 0 0 2024-03-26T18:54:41.217 INFO:tasks.workunit.client.1.smithi190.stdout:1/795: fsync f2 0 2024-03-26T18:54:41.217 INFO:tasks.workunit.client.1.smithi190.stdout:2/986: mknod d6/dbc/c14b 0 2024-03-26T18:54:41.217 INFO:tasks.workunit.client.1.smithi190.stdout:0/973: mkdir d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d66/d15c 0 2024-03-26T18:54:41.218 INFO:tasks.workunit.client.0.smithi102.stdout:1/546: dwrite de/f22 [0,4194304] 0 2024-03-26T18:54:41.220 INFO:tasks.workunit.client.1.smithi190.stdout:5/808: link d2/d17/d1b/db6/d24/d3b/da7/fe1 d2/d17/d62/dd7/f10a 0 2024-03-26T18:54:41.222 INFO:tasks.workunit.client.0.smithi102.stdout:0/625: creat d6/de/d13/d1f/fd9 x:0 0 0 2024-03-26T18:54:41.225 INFO:tasks.workunit.client.0.smithi102.stdout:1/547: unlink de/d16/d1d/d61/d6a/c8f 0 2024-03-26T18:54:41.225 INFO:tasks.workunit.client.0.smithi102.stdout:1/548: chown de/d16/d1d/d2a/d2c/d56/d6e/f9a 119 1 2024-03-26T18:54:41.225 INFO:tasks.workunit.client.0.smithi102.stdout:1/549: fdatasync de/d16/d1d/f4d 0 2024-03-26T18:54:41.228 INFO:tasks.workunit.client.0.smithi102.stdout:0/626: rmdir d6/d3e/d69/d8c 39 2024-03-26T18:54:41.230 INFO:tasks.workunit.client.1.smithi190.stdout:6/773: symlink d17/d28/d38/d79/l106 0 2024-03-26T18:54:41.230 INFO:tasks.workunit.client.1.smithi190.stdout:5/809: getdents d2 0 2024-03-26T18:54:41.230 INFO:tasks.workunit.client.1.smithi190.stdout:2/987: dread d6/d21/dd8/f11b [0,4194304] 0 2024-03-26T18:54:41.230 INFO:tasks.workunit.client.1.smithi190.stdout:5/810: rmdir d2/d17/d1b/d60/d63 39 2024-03-26T18:54:41.230 INFO:tasks.workunit.client.0.smithi102.stdout:1/550: mknod de/d16/d1d/d61/d6a/da5/cb5 0 2024-03-26T18:54:41.234 INFO:tasks.workunit.client.1.smithi190.stdout:5/811: link d2/d17/d1b/d30/fb1 d2/d17/d98/f10b 0 2024-03-26T18:54:41.250 INFO:tasks.workunit.client.1.smithi190.stdout:3/748: sync 2024-03-26T18:54:41.250 INFO:tasks.workunit.client.0.smithi102.stdout:0/627: dread d6/de/d13/d9a/d85/f61 [0,4194304] 0 2024-03-26T18:54:41.250 INFO:tasks.workunit.client.0.smithi102.stdout:0/628: chown d6/de/d13/d9a/d85/c5f 678177736 1 2024-03-26T18:54:41.251 INFO:tasks.workunit.client.0.smithi102.stdout:0/629: symlink d6/d2d/d3d/d80/d83/lda 0 2024-03-26T18:54:41.251 INFO:tasks.workunit.client.0.smithi102.stdout:1/551: dread de/d1f/f33 [0,4194304] 0 2024-03-26T18:54:41.257 INFO:tasks.workunit.client.0.smithi102.stdout:1/552: dread de/d16/d1d/d2a/d2c/d56/fb3 [0,4194304] 0 2024-03-26T18:54:41.258 INFO:tasks.workunit.client.0.smithi102.stdout:1/553: stat de/d1f 0 2024-03-26T18:54:41.258 INFO:tasks.workunit.client.0.smithi102.stdout:1/554: unlink f3 0 2024-03-26T18:54:41.259 INFO:tasks.workunit.client.0.smithi102.stdout:1/555: rename de/d2e/c45 to de/d1f/cb6 0 2024-03-26T18:54:41.265 INFO:tasks.workunit.client.0.smithi102.stdout:2/583: dwrite d3/d82/d93/fcc [0,4194304] 0 2024-03-26T18:54:41.266 INFO:tasks.workunit.client.1.smithi190.stdout:1/796: dwrite d3/da/dd/d2f/d5b/d83/feb [0,4194304] 0 2024-03-26T18:54:41.269 INFO:tasks.workunit.client.0.smithi102.stdout:2/584: link d3/d20/d78/d7c/f7e d3/d20/d78/d84/d6e/dc6/fd1 0 2024-03-26T18:54:41.273 INFO:tasks.workunit.client.0.smithi102.stdout:2/585: unlink d3/d11/d15/d31/l43 0 2024-03-26T18:54:41.273 INFO:tasks.workunit.client.0.smithi102.stdout:2/586: fdatasync d3/d20/d78/d84/d6e/f94 0 2024-03-26T18:54:41.274 INFO:tasks.workunit.client.1.smithi190.stdout:1/797: mknod d3/dd9/c105 0 2024-03-26T18:54:41.275 INFO:tasks.workunit.client.0.smithi102.stdout:2/587: mknod d3/d11/d27/dc0/cd2 0 2024-03-26T18:54:41.283 INFO:tasks.workunit.client.0.smithi102.stdout:2/588: write d3/d11/d27/f6b [958524,119859] 0 2024-03-26T18:54:41.283 INFO:tasks.workunit.client.0.smithi102.stdout:0/630: dwrite d6/de/d13/d25/d4f/d66/fb7 [0,4194304] 0 2024-03-26T18:54:41.283 INFO:tasks.workunit.client.0.smithi102.stdout:1/556: dwrite de/d16/d1d/d61/d6a/fa3 [0,4194304] 0 2024-03-26T18:54:41.284 INFO:tasks.workunit.client.1.smithi190.stdout:0/974: dwrite d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d66/d11c/f145 [0,4194304] 0 2024-03-26T18:54:41.284 INFO:tasks.workunit.client.1.smithi190.stdout:6/774: dwrite d17/ffa [0,4194304] 0 2024-03-26T18:54:41.284 INFO:tasks.workunit.client.1.smithi190.stdout:0/975: fsync d0/dd/d20/d1c/d46/da7/df0/f155 0 2024-03-26T18:54:41.284 INFO:tasks.workunit.client.1.smithi190.stdout:6/775: chown d17/d28/d2c/d50/fe9 3883 1 2024-03-26T18:54:41.284 INFO:tasks.workunit.client.1.smithi190.stdout:6/776: write d17/d28/d8b/d101/f103 [4585070,58185] 0 2024-03-26T18:54:41.284 INFO:tasks.workunit.client.1.smithi190.stdout:2/988: dwrite d6/d9/d3a/d3d/d1c/d8d/de8/dfb/f134 [0,4194304] 0 2024-03-26T18:54:41.284 INFO:tasks.workunit.client.1.smithi190.stdout:5/812: dwrite d2/f3 [0,4194304] 0 2024-03-26T18:54:41.285 INFO:tasks.workunit.client.1.smithi190.stdout:2/989: creat d6/d9/d3a/f14c x:0 0 0 2024-03-26T18:54:41.285 INFO:tasks.workunit.client.1.smithi190.stdout:2/990: fdatasync d6/dbc/fe1 0 2024-03-26T18:54:41.285 INFO:tasks.workunit.client.1.smithi190.stdout:2/991: mknod d6/d9/d3a/d3d/d2e/d10e/d9e/dfc/c14d 0 2024-03-26T18:54:41.286 INFO:tasks.workunit.client.1.smithi190.stdout:2/992: fsync d6/d9/d45/f74 0 2024-03-26T18:54:41.287 INFO:tasks.workunit.client.1.smithi190.stdout:2/993: symlink d6/d9/d3a/d3d/d2e/d86/d10b/l14e 0 2024-03-26T18:54:41.287 INFO:tasks.workunit.client.1.smithi190.stdout:2/994: readlink d6/d9/d3a/d3d/d1c/d8d/dc7/d72/lde 0 2024-03-26T18:54:41.287 INFO:tasks.workunit.client.0.smithi102.stdout:0/631: dread d6/d3e/d87/f52 [0,4194304] 0 2024-03-26T18:54:41.288 INFO:tasks.workunit.client.0.smithi102.stdout:2/589: rename d3/d2d/d67/lb2 to d3/d20/d78/d7c/dca/ld3 0 2024-03-26T18:54:41.288 INFO:tasks.workunit.client.0.smithi102.stdout:2/590: chown d3/c79 67637 1 2024-03-26T18:54:41.290 INFO:tasks.workunit.client.1.smithi190.stdout:6/777: creat d17/d28/df6/f107 x:0 0 0 2024-03-26T18:54:41.292 INFO:tasks.workunit.client.1.smithi190.stdout:5/813: creat d2/d17/d1b/d60/f10c x:0 0 0 2024-03-26T18:54:41.295 INFO:tasks.workunit.client.1.smithi190.stdout:6/778: rename d17/d28/d2c/db8/dcc/le1 to d17/d28/d2c/d63/l108 0 2024-03-26T18:54:41.298 INFO:tasks.workunit.client.0.smithi102.stdout:0/632: mknod d6/de/d13/d9a/cdb 0 2024-03-26T18:54:41.298 INFO:tasks.workunit.client.0.smithi102.stdout:0/633: write d6/de/d13/d25/d31/f50 [1930210,41103] 0 2024-03-26T18:54:41.298 INFO:tasks.workunit.client.0.smithi102.stdout:0/634: write d6/de/d13/d9a/fcc [920205,92265] 0 2024-03-26T18:54:41.298 INFO:tasks.workunit.client.0.smithi102.stdout:0/635: dread - d6/de/d13/d1f/fd9 zero size 2024-03-26T18:54:41.300 INFO:tasks.workunit.client.0.smithi102.stdout:1/557: rename de/f22 to de/d16/d1d/d61/d6a/da5/da6/fb7 0 2024-03-26T18:54:41.302 INFO:tasks.workunit.client.1.smithi190.stdout:0/976: write d0/dd/d20/d61/f7b [5483343,74801] 0 2024-03-26T18:54:41.305 INFO:tasks.workunit.client.0.smithi102.stdout:0/636: mknod d6/de/d13/d1f/d55/d57/cdc 0 2024-03-26T18:54:41.306 INFO:tasks.workunit.client.0.smithi102.stdout:9/563: sync 2024-03-26T18:54:41.306 INFO:tasks.workunit.client.0.smithi102.stdout:9/564: read - df/d1a/d7f/fba zero size 2024-03-26T18:54:41.306 INFO:tasks.workunit.client.0.smithi102.stdout:9/565: read - df/d1a/d23/d4e/f77 zero size 2024-03-26T18:54:41.306 INFO:tasks.workunit.client.0.smithi102.stdout:8/632: sync 2024-03-26T18:54:41.307 INFO:tasks.workunit.client.0.smithi102.stdout:5/825: sync 2024-03-26T18:54:41.308 INFO:tasks.workunit.client.0.smithi102.stdout:0/637: symlink d6/d2d/d3d/ldd 0 2024-03-26T18:54:41.310 INFO:tasks.workunit.client.0.smithi102.stdout:5/826: readlink d2/d14/d1a/l71 0 2024-03-26T18:54:41.310 INFO:tasks.workunit.client.0.smithi102.stdout:5/827: write d2/d25/d5b/d72/fc5 [1727463,108870] 0 2024-03-26T18:54:41.310 INFO:tasks.workunit.client.0.smithi102.stdout:5/828: readlink d2/d14/l102 0 2024-03-26T18:54:41.313 INFO:tasks.workunit.client.0.smithi102.stdout:0/638: rename d6/f51 to d6/de/d13/d25/d31/d45/d82/d84/d88/fde 0 2024-03-26T18:54:41.314 INFO:tasks.workunit.client.1.smithi190.stdout:5/814: dread d2/d17/d1b/db6/fc6 [0,4194304] 0 2024-03-26T18:54:41.316 INFO:tasks.workunit.client.1.smithi190.stdout:1/798: dwrite d3/da/dd/d2f/d5b/f70 [4194304,4194304] 0 2024-03-26T18:54:41.316 INFO:tasks.workunit.client.1.smithi190.stdout:1/799: write d3/da/d3a/f102 [449812,77283] 0 2024-03-26T18:54:41.316 INFO:tasks.workunit.client.1.smithi190.stdout:1/800: chown d3/da/d3a/da5 8186 1 2024-03-26T18:54:41.317 INFO:tasks.workunit.client.1.smithi190.stdout:2/995: dwrite d6/d9/d10/fd9 [0,4194304] 0 2024-03-26T18:54:41.317 INFO:tasks.workunit.client.1.smithi190.stdout:2/996: chown d6/d9/d45/df3/c106 27329370 1 2024-03-26T18:54:41.318 INFO:tasks.workunit.client.0.smithi102.stdout:0/639: rename d6/de/d13/d25/d4f/d66/f90 to d6/d2d/d3d/d80/fdf 0 2024-03-26T18:54:41.321 INFO:tasks.workunit.client.0.smithi102.stdout:5/829: unlink d2/d14/d1a/c24 0 2024-03-26T18:54:41.322 INFO:tasks.workunit.client.1.smithi190.stdout:2/997: dread d6/d9/d3a/d3d/d2e/d31/f12a [0,4194304] 0 2024-03-26T18:54:41.322 INFO:tasks.workunit.client.1.smithi190.stdout:2/998: write d6/d21/dd0/fee [386138,87936] 0 2024-03-26T18:54:41.322 INFO:tasks.workunit.client.1.smithi190.stdout:2/999: fdatasync d6/f136 0 2024-03-26T18:54:41.323 INFO:tasks.workunit.client.0.smithi102.stdout:5/830: write d2/dea/f103 [512962,17345] 0 2024-03-26T18:54:41.325 INFO:tasks.workunit.client.1.smithi190.stdout:5/815: rename d2/d17/d62/d64/ff1 to d2/d105/f10d 0 2024-03-26T18:54:41.325 INFO:tasks.workunit.client.1.smithi190.stdout:5/816: write d2/d17/d1b/db6/ffc [129758,2992] 0 2024-03-26T18:54:41.326 INFO:tasks.workunit.client.0.smithi102.stdout:0/640: symlink d6/d3e/le0 0 2024-03-26T18:54:41.327 INFO:tasks.workunit.client.0.smithi102.stdout:0/641: dread d6/de/d13/d25/d4f/f73 [0,4194304] 0 2024-03-26T18:54:41.327 INFO:tasks.workunit.client.0.smithi102.stdout:0/642: chown d6/de/d13/d25 174924572 1 2024-03-26T18:54:41.329 INFO:tasks.workunit.client.0.smithi102.stdout:5/831: rename d2/d25/d75/def/dec/d112/f110 to d2/dea/f122 0 2024-03-26T18:54:41.332 INFO:tasks.workunit.client.0.smithi102.stdout:0/643: link d6/f1a d6/de/d13/d1f/d71/fe1 0 2024-03-26T18:54:41.332 INFO:tasks.workunit.client.0.smithi102.stdout:0/644: stat d6/de/d13/d25/d31/fd7 0 2024-03-26T18:54:41.332 INFO:tasks.workunit.client.0.smithi102.stdout:0/645: write d6/de/d13/d25/d4f/f73 [1295244,92450] 0 2024-03-26T18:54:41.333 INFO:tasks.workunit.client.0.smithi102.stdout:5/832: symlink d2/dea/d4f/da9/dee/l123 0 2024-03-26T18:54:41.333 INFO:tasks.workunit.client.0.smithi102.stdout:5/833: chown d2/dea/d4f 0 1 2024-03-26T18:54:41.335 INFO:tasks.workunit.client.0.smithi102.stdout:0/646: chown d6/de/d13/d1f/d55/d57/f46 872 1 2024-03-26T18:54:41.335 INFO:tasks.workunit.client.0.smithi102.stdout:0/647: dread - d6/de/fd0 zero size 2024-03-26T18:54:41.335 INFO:tasks.workunit.client.0.smithi102.stdout:0/648: creat d6/de/d13/d1f/d55/d57/d7d/d7f/fe2 x:0 0 0 2024-03-26T18:54:41.337 INFO:tasks.workunit.client.1.smithi190.stdout:5/817: rename d2/l18 to d2/d17/d62/df2/l10e 0 2024-03-26T18:54:41.340 INFO:tasks.workunit.client.0.smithi102.stdout:0/649: rename d6/de/d13/d1f/d55/d57/cb6 to d6/de/d13/d1f/d55/d57/d7d/ce3 0 2024-03-26T18:54:41.341 INFO:tasks.workunit.client.0.smithi102.stdout:2/591: dwrite d3/d11/d15/d31/f83 [0,4194304] 0 2024-03-26T18:54:41.345 INFO:tasks.workunit.client.0.smithi102.stdout:2/592: getdents d3/d2d/d67 0 2024-03-26T18:54:41.348 INFO:tasks.workunit.client.0.smithi102.stdout:2/593: rmdir d3/d11/d15/d31/d7f 39 2024-03-26T18:54:41.350 INFO:tasks.workunit.client.0.smithi102.stdout:1/558: dwrite de/d1f/f57 [0,4194304] 0 2024-03-26T18:54:41.353 INFO:tasks.workunit.client.0.smithi102.stdout:1/559: symlink de/d16/d1d/d61/d6a/lb8 0 2024-03-26T18:54:41.355 INFO:tasks.workunit.client.0.smithi102.stdout:1/560: unlink de/d16/d1d/d2a/f75 0 2024-03-26T18:54:41.357 INFO:tasks.workunit.client.0.smithi102.stdout:1/561: rmdir de/d16/d1d/dad 0 2024-03-26T18:54:41.357 INFO:tasks.workunit.client.0.smithi102.stdout:1/562: readlink de/d2e/l90 0 2024-03-26T18:54:41.357 INFO:tasks.workunit.client.0.smithi102.stdout:1/563: fdatasync de/d16/d1d/d2a/f5d 0 2024-03-26T18:54:41.357 INFO:tasks.workunit.client.0.smithi102.stdout:2/594: dread d3/d82/d93/d9e/f44 [0,4194304] 0 2024-03-26T18:54:41.357 INFO:tasks.workunit.client.0.smithi102.stdout:2/595: write d3/d2d/d69/fcf [731987,36539] 0 2024-03-26T18:54:41.358 INFO:tasks.workunit.client.0.smithi102.stdout:2/596: write d3/d2a/f4f [5138168,48055] 0 2024-03-26T18:54:41.365 INFO:tasks.workunit.client.0.smithi102.stdout:8/633: dwrite d0/d3/f21 [4194304,4194304] 0 2024-03-26T18:54:41.365 INFO:tasks.workunit.client.0.smithi102.stdout:8/634: stat d0/d3/d2c/d33/d87/fa8 0 2024-03-26T18:54:41.366 INFO:tasks.workunit.client.0.smithi102.stdout:8/635: creat d0/d3/fd1 x:0 0 0 2024-03-26T18:54:41.368 INFO:tasks.workunit.client.0.smithi102.stdout:9/566: dwrite df/d1a/f24 [4194304,4194304] 0 2024-03-26T18:54:41.368 INFO:tasks.workunit.client.0.smithi102.stdout:9/567: dread - df/d1a/d23/f7e zero size 2024-03-26T18:54:41.368 INFO:tasks.workunit.client.0.smithi102.stdout:9/568: write df/d46/f6a [1066850,8742] 0 2024-03-26T18:54:41.369 INFO:tasks.workunit.client.1.smithi190.stdout:1/801: dwrite d3/d9/f1c [0,4194304] 0 2024-03-26T18:54:41.369 INFO:tasks.workunit.client.1.smithi190.stdout:3/749: dwrite d1/f26 [0,4194304] 0 2024-03-26T18:54:41.369 INFO:tasks.workunit.client.1.smithi190.stdout:1/802: fsync d3/d9/f15 0 2024-03-26T18:54:41.369 INFO:tasks.workunit.client.1.smithi190.stdout:3/750: chown d1/dd8 39580 1 2024-03-26T18:54:41.370 INFO:tasks.workunit.client.0.smithi102.stdout:8/636: rmdir d0/d3/d26/d5c 39 2024-03-26T18:54:41.370 INFO:tasks.workunit.client.0.smithi102.stdout:8/637: write d0/d3c/d6f/f8b [820588,128335] 0 2024-03-26T18:54:41.373 INFO:tasks.workunit.client.0.smithi102.stdout:8/638: mknod d0/d3c/d96/d98/db6/cd2 0 2024-03-26T18:54:41.379 INFO:tasks.workunit.client.1.smithi190.stdout:3/751: mkdir d1/d3/df/d18/d1b/de4 0 2024-03-26T18:54:41.381 INFO:tasks.workunit.client.1.smithi190.stdout:1/803: read d3/da/d2e/f30 [1132506,87446] 0 2024-03-26T18:54:41.381 INFO:tasks.workunit.client.1.smithi190.stdout:1/804: truncate d3/d73/d88/f104 169274 0 2024-03-26T18:54:41.381 INFO:tasks.workunit.client.1.smithi190.stdout:1/805: chown d3/da/d2e/f4a 5619437 1 2024-03-26T18:54:41.382 INFO:tasks.workunit.client.0.smithi102.stdout:0/650: dwrite d6/de/d13/d1f/f3b [8388608,4194304] 0 2024-03-26T18:54:41.389 INFO:tasks.workunit.client.1.smithi190.stdout:1/806: creat d3/da/dd/d76/f106 x:0 0 0 2024-03-26T18:54:41.391 INFO:tasks.workunit.client.0.smithi102.stdout:1/564: dwrite de/d16/d1d/d2a/d2c/f3c [0,4194304] 0 2024-03-26T18:54:41.391 INFO:tasks.workunit.client.0.smithi102.stdout:1/565: fsync de/d16/d1d/d2a/d2c/d56/d78/f8c 0 2024-03-26T18:54:41.391 INFO:tasks.workunit.client.0.smithi102.stdout:1/566: chown de/fac 2 1 2024-03-26T18:54:41.394 INFO:tasks.workunit.client.0.smithi102.stdout:5/834: getdents d2/dea/d4f/da9/dee 0 2024-03-26T18:54:41.394 INFO:tasks.workunit.client.1.smithi190.stdout:1/807: creat d3/da/dd/d2f/d34/f107 x:0 0 0 2024-03-26T18:54:41.394 INFO:tasks.workunit.client.1.smithi190.stdout:5/818: dwrite d2/d17/d1b/d60/d63/f6e [0,4194304] 0 2024-03-26T18:54:41.397 INFO:tasks.workunit.client.0.smithi102.stdout:2/597: dwrite d3/f4 [0,4194304] 0 2024-03-26T18:54:41.398 INFO:tasks.workunit.client.0.smithi102.stdout:9/569: dwrite df/d1a/d23/d29/f31 [0,4194304] 0 2024-03-26T18:54:41.398 INFO:tasks.workunit.client.0.smithi102.stdout:9/570: readlink df/d1a/d1d/la0 0 2024-03-26T18:54:41.398 INFO:tasks.workunit.client.1.smithi190.stdout:1/808: rename d3/da/d3a/f55 to d3/d73/d88/df1/f108 0 2024-03-26T18:54:41.402 INFO:tasks.workunit.client.1.smithi190.stdout:5/819: link d2/d17/d1b/db6/d24/c87 d2/d17/d1b/db6/d24/d3b/da7/c10f 0 2024-03-26T18:54:41.402 INFO:tasks.workunit.client.1.smithi190.stdout:5/820: truncate d2/d17/d98/d3a/d3c/fa2 228088 0 2024-03-26T18:54:41.402 INFO:tasks.workunit.client.1.smithi190.stdout:3/752: dwrite d1/d3/df/d18/d35/f39 [0,4194304] 0 2024-03-26T18:54:41.404 INFO:tasks.workunit.client.1.smithi190.stdout:5/821: dread d2/d17/d1b/db6/ffc [0,4194304] 0 2024-03-26T18:54:41.404 INFO:tasks.workunit.client.0.smithi102.stdout:3/779: sync 2024-03-26T18:54:41.406 INFO:tasks.workunit.client.1.smithi190.stdout:1/809: rename d3/d9/db/l22 to d3/da/dd/d76/dc4/l109 0 2024-03-26T18:54:41.406 INFO:tasks.workunit.client.1.smithi190.stdout:1/810: write d3/da/d45/fc2 [653741,69669] 0 2024-03-26T18:54:41.407 INFO:tasks.workunit.client.1.smithi190.stdout:1/811: write d3/da/d2e/d59/dae/ff9 [363633,75626] 0 2024-03-26T18:54:41.407 INFO:tasks.workunit.client.1.smithi190.stdout:1/812: creat d3/d73/d88/dcc/f10a x:0 0 0 2024-03-26T18:54:41.407 INFO:tasks.workunit.client.1.smithi190.stdout:1/813: chown d3/da/d45/c7a 588982 1 2024-03-26T18:54:41.409 INFO:tasks.workunit.client.0.smithi102.stdout:7/535: sync 2024-03-26T18:54:41.409 INFO:tasks.workunit.client.0.smithi102.stdout:4/475: sync 2024-03-26T18:54:41.409 INFO:tasks.workunit.client.0.smithi102.stdout:6/583: sync 2024-03-26T18:54:41.410 INFO:tasks.workunit.client.0.smithi102.stdout:4/476: write d1/d27/f4d [904105,102067] 0 2024-03-26T18:54:41.410 INFO:tasks.workunit.client.0.smithi102.stdout:4/477: write d1/d17/d1a/d21/f63 [941510,116209] 0 2024-03-26T18:54:41.411 INFO:tasks.workunit.client.0.smithi102.stdout:1/567: symlink de/d16/d1d/d2a/d2c/d56/d78/d80/d8e/lb9 0 2024-03-26T18:54:41.413 INFO:tasks.workunit.client.1.smithi190.stdout:3/753: truncate d1/de/d4e/fa0 932877 0 2024-03-26T18:54:41.415 INFO:tasks.workunit.client.0.smithi102.stdout:5/835: creat d2/d25/d5b/d72/f124 x:0 0 0 2024-03-26T18:54:41.418 INFO:tasks.workunit.client.1.smithi190.stdout:5/822: symlink d2/d17/d1b/db6/d68/l110 0 2024-03-26T18:54:41.418 INFO:tasks.workunit.client.1.smithi190.stdout:5/823: fsync d2/d17/d1b/db6/d24/d3b/da7/fc0 0 2024-03-26T18:54:41.418 INFO:tasks.workunit.client.0.smithi102.stdout:2/598: rmdir d3/d11/d27/dc0 39 2024-03-26T18:54:41.418 INFO:tasks.workunit.client.0.smithi102.stdout:2/599: creat d3/d20/fd4 x:0 0 0 2024-03-26T18:54:41.419 INFO:tasks.workunit.client.0.smithi102.stdout:2/600: write d3/d74/f77 [118948,59067] 0 2024-03-26T18:54:41.422 INFO:tasks.workunit.client.0.smithi102.stdout:0/651: dwrite d6/d2d/d3d/fa0 [0,4194304] 0 2024-03-26T18:54:41.423 INFO:tasks.workunit.client.1.smithi190.stdout:5/824: link d2/d17/d98/d3a/faa d2/d17/d62/f111 0 2024-03-26T18:54:41.425 INFO:tasks.workunit.client.0.smithi102.stdout:7/536: unlink d7/d12/d4d/d62/f99 0 2024-03-26T18:54:41.427 INFO:tasks.workunit.client.0.smithi102.stdout:4/478: mknod d1/d27/d7a/c94 0 2024-03-26T18:54:41.428 INFO:tasks.workunit.client.1.smithi190.stdout:5/825: mknod d2/d17/d62/df2/c112 0 2024-03-26T18:54:41.430 INFO:tasks.workunit.client.0.smithi102.stdout:2/601: symlink d3/d82/d93/d9e/d1a/d25/ld5 0 2024-03-26T18:54:41.434 INFO:tasks.workunit.client.0.smithi102.stdout:8/639: rename d0/d3 to d0/d15/d5d/dd3 0 2024-03-26T18:54:41.434 INFO:tasks.workunit.client.0.smithi102.stdout:8/640: fdatasync d0/d15/fd0 0 2024-03-26T18:54:41.435 INFO:tasks.workunit.client.0.smithi102.stdout:7/537: mkdir d7/d12/d4d/d9e 0 2024-03-26T18:54:41.436 INFO:tasks.workunit.client.0.smithi102.stdout:4/479: getdents d1/d25 0 2024-03-26T18:54:41.436 INFO:tasks.workunit.client.0.smithi102.stdout:4/480: write d1/d17/d1a/d21/d77/f81 [884665,76929] 0 2024-03-26T18:54:41.440 INFO:tasks.workunit.client.0.smithi102.stdout:1/568: rename de/d16/d1d/d2a/f74 to de/d16/d1d/d61/d6a/da5/da6/fba 0 2024-03-26T18:54:41.440 INFO:tasks.workunit.client.0.smithi102.stdout:1/569: stat de/d16/d1d 0 2024-03-26T18:54:41.442 INFO:tasks.workunit.client.0.smithi102.stdout:7/538: mkdir d7/d10/d17/d9f 0 2024-03-26T18:54:41.445 INFO:tasks.workunit.client.0.smithi102.stdout:8/641: write d0/d15/d5d/dd3/f21 [6556319,125197] 0 2024-03-26T18:54:41.445 INFO:tasks.workunit.client.0.smithi102.stdout:9/571: dwrite df/f40 [0,4194304] 0 2024-03-26T18:54:41.445 INFO:tasks.workunit.client.0.smithi102.stdout:4/481: creat d1/d17/d1a/d21/f95 x:0 0 0 2024-03-26T18:54:41.448 INFO:tasks.workunit.client.0.smithi102.stdout:2/602: rename d3/d20/d78/c9c to d3/d20/d78/d84/cd6 0 2024-03-26T18:54:41.450 INFO:tasks.workunit.client.0.smithi102.stdout:1/570: link de/d16/d1d/d2a/d2c/l84 de/d16/d1d/d2a/d2c/lbb 0 2024-03-26T18:54:41.451 INFO:tasks.workunit.client.0.smithi102.stdout:9/572: stat df/c18 0 2024-03-26T18:54:41.452 INFO:tasks.workunit.client.0.smithi102.stdout:4/482: symlink d1/d17/d1f/d5d/d8b/d3e/l96 0 2024-03-26T18:54:41.452 INFO:tasks.workunit.client.0.smithi102.stdout:4/483: truncate d1/f9 4680363 0 2024-03-26T18:54:41.452 INFO:tasks.workunit.client.0.smithi102.stdout:4/484: truncate d1/d17/d1f/d5d/d8b/d3e/d4c/f68 1631354 0 2024-03-26T18:54:41.455 INFO:tasks.workunit.client.0.smithi102.stdout:2/603: unlink d3/d2d/d69/f91 0 2024-03-26T18:54:41.457 INFO:tasks.workunit.client.0.smithi102.stdout:1/571: rename de/d16/l1b to de/lbc 0 2024-03-26T18:54:41.457 INFO:tasks.workunit.client.0.smithi102.stdout:6/584: dwrite d2/df/d71/d80/d91/fa9 [0,4194304] 0 2024-03-26T18:54:41.458 INFO:tasks.workunit.client.0.smithi102.stdout:6/585: creat d2/d7/d3c/fb9 x:0 0 0 2024-03-26T18:54:41.458 INFO:tasks.workunit.client.0.smithi102.stdout:9/573: symlink df/d1a/d65/lcf 0 2024-03-26T18:54:41.458 INFO:tasks.workunit.client.1.smithi190.stdout:1/814: dwrite d3/f53 [4194304,4194304] 0 2024-03-26T18:54:41.459 INFO:tasks.workunit.client.0.smithi102.stdout:2/604: mknod d3/d2d/d69/cd7 0 2024-03-26T18:54:41.461 INFO:tasks.workunit.client.0.smithi102.stdout:3/780: dwrite df/d1c/d70/fbe [0,4194304] 0 2024-03-26T18:54:41.461 INFO:tasks.workunit.client.0.smithi102.stdout:3/781: stat df/d29/d3a/d93/c9a 0 2024-03-26T18:54:41.461 INFO:tasks.workunit.client.0.smithi102.stdout:1/572: creat de/d16/d1d/fbd x:0 0 0 2024-03-26T18:54:41.461 INFO:tasks.workunit.client.0.smithi102.stdout:1/573: write de/d16/d1d/d2a/d2c/d56/d78/d80/f85 [1860384,99941] 0 2024-03-26T18:54:41.463 INFO:tasks.workunit.client.0.smithi102.stdout:7/539: write d7/d12/d51/d6a/f7a [2792846,72934] 0 2024-03-26T18:54:41.464 INFO:tasks.workunit.client.0.smithi102.stdout:0/652: dwrite d6/de/d13/d1f/d71/f9b [0,4194304] 0 2024-03-26T18:54:41.465 INFO:tasks.workunit.client.1.smithi190.stdout:1/815: truncate d3/d73/d88/df1/f108 171102 0 2024-03-26T18:54:41.466 INFO:tasks.workunit.client.0.smithi102.stdout:5/836: dwrite d2/d25/d5b/ff0 [0,4194304] 0 2024-03-26T18:54:41.467 INFO:tasks.workunit.client.0.smithi102.stdout:6/586: mkdir d2/df/dba 0 2024-03-26T18:54:41.470 INFO:tasks.workunit.client.0.smithi102.stdout:9/574: creat df/d1a/fd0 x:0 0 0 2024-03-26T18:54:41.471 INFO:tasks.workunit.client.0.smithi102.stdout:2/605: truncate d3/d82/d93/d9e/ff 5548 0 2024-03-26T18:54:41.477 INFO:tasks.workunit.client.0.smithi102.stdout:3/782: mkdir df/d29/d31/d36/d3c/d113 0 2024-03-26T18:54:41.480 INFO:tasks.workunit.client.1.smithi190.stdout:5/826: dwrite d2/d17/d1b/f67 [0,4194304] 0 2024-03-26T18:54:41.480 INFO:tasks.workunit.client.1.smithi190.stdout:1/816: unlink d3/d9/db/f23 0 2024-03-26T18:54:41.480 INFO:tasks.workunit.client.1.smithi190.stdout:1/817: fsync d3/da/d45/fc2 0 2024-03-26T18:54:41.482 INFO:tasks.workunit.client.0.smithi102.stdout:7/540: symlink d7/d12/d4b/d83/la0 0 2024-03-26T18:54:41.482 INFO:tasks.workunit.client.0.smithi102.stdout:7/541: write d7/d12/d51/f58 [478365,124639] 0 2024-03-26T18:54:41.484 INFO:tasks.workunit.client.1.smithi190.stdout:5/827: truncate d2/d17/d62/f7e 1758265 0 2024-03-26T18:54:41.484 INFO:tasks.workunit.client.0.smithi102.stdout:8/642: dwrite d0/d15/d5d/dd3/d26/d19/d76/f9c [0,4194304] 0 2024-03-26T18:54:41.484 INFO:tasks.workunit.client.0.smithi102.stdout:8/643: write d0/d15/d5d/dac/fc2 [231771,66951] 0 2024-03-26T18:54:41.485 INFO:tasks.workunit.client.0.smithi102.stdout:5/837: creat d2/dea/dbb/dcf/f125 x:0 0 0 2024-03-26T18:54:41.485 INFO:tasks.workunit.client.0.smithi102.stdout:6/587: creat d2/df/d71/fbb x:0 0 0 2024-03-26T18:54:41.487 INFO:tasks.workunit.client.1.smithi190.stdout:5/828: rename d2/d17/la8 to d2/d17/d98/d3a/d3c/l113 0 2024-03-26T18:54:41.488 INFO:tasks.workunit.client.0.smithi102.stdout:2/606: unlink d3/f8 0 2024-03-26T18:54:41.488 INFO:tasks.workunit.client.0.smithi102.stdout:2/607: write d3/d82/fb3 [658032,71932] 0 2024-03-26T18:54:41.488 INFO:tasks.workunit.client.0.smithi102.stdout:2/608: write d3/d20/d78/d84/d6e/dc6/fd1 [860731,87905] 0 2024-03-26T18:54:41.488 INFO:tasks.workunit.client.0.smithi102.stdout:2/609: write d3/d20/d78/d7c/f85 [493788,20182] 0 2024-03-26T18:54:41.489 INFO:tasks.workunit.client.1.smithi190.stdout:5/829: dread d2/d17/d98/f2e [0,4194304] 0 2024-03-26T18:54:41.489 INFO:tasks.workunit.client.0.smithi102.stdout:3/783: truncate df/d1c/fbf 2959148 0 2024-03-26T18:54:41.489 INFO:tasks.workunit.client.0.smithi102.stdout:3/784: dread - df/d1c/d34/da5/dce/f112 zero size 2024-03-26T18:54:41.489 INFO:tasks.workunit.client.0.smithi102.stdout:7/542: mkdir d7/d12/d4d/d9e/da1 0 2024-03-26T18:54:41.490 INFO:tasks.workunit.client.0.smithi102.stdout:7/543: dread - d7/d10/d67/d69/f9c zero size 2024-03-26T18:54:41.495 INFO:tasks.workunit.client.0.smithi102.stdout:8/644: creat d0/d3c/d96/d98/db6/fd4 x:0 0 0 2024-03-26T18:54:41.495 INFO:tasks.workunit.client.0.smithi102.stdout:5/838: write d2/d25/d75/db9/f49 [1244738,92996] 0 2024-03-26T18:54:41.497 INFO:tasks.workunit.client.1.smithi190.stdout:5/830: rename d2/d17/ddd to d2/d17/d1b/d60/d63/d85/d99/d114 0 2024-03-26T18:54:41.497 INFO:tasks.workunit.client.1.smithi190.stdout:5/831: truncate d2/d17/d1b/db6/d68/ff9 51515 0 2024-03-26T18:54:41.497 INFO:tasks.workunit.client.1.smithi190.stdout:5/832: fdatasync d2/d17/d98/f10b 0 2024-03-26T18:54:41.497 INFO:tasks.workunit.client.0.smithi102.stdout:4/485: dwrite d1/d17/d1a/f57 [0,4194304] 0 2024-03-26T18:54:41.498 INFO:tasks.workunit.client.0.smithi102.stdout:6/588: creat d2/d3d/fbc x:0 0 0 2024-03-26T18:54:41.498 INFO:tasks.workunit.client.0.smithi102.stdout:0/653: dwrite d6/d2d/d3d/d80/fdf [0,4194304] 0 2024-03-26T18:54:41.499 INFO:tasks.workunit.client.0.smithi102.stdout:2/610: mknod d3/d11/d15/d31/d7f/cd8 0 2024-03-26T18:54:41.500 INFO:tasks.workunit.client.0.smithi102.stdout:1/574: dwrite de/d16/f29 [0,4194304] 0 2024-03-26T18:54:41.501 INFO:tasks.workunit.client.1.smithi190.stdout:5/833: creat d2/d17/d1b/d60/d63/d85/d99/d114/f115 x:0 0 0 2024-03-26T18:54:41.504 INFO:tasks.workunit.client.1.smithi190.stdout:1/818: read d3/da/d3a/f50 [3331792,85266] 0 2024-03-26T18:54:41.508 INFO:tasks.workunit.client.0.smithi102.stdout:4/486: dread d1/d17/d1a/d21/f63 [0,4194304] 0 2024-03-26T18:54:41.509 INFO:tasks.workunit.client.1.smithi190.stdout:1/819: dread d3/da/d2e/d59/dae/ff9 [0,4194304] 0 2024-03-26T18:54:41.511 INFO:tasks.workunit.client.0.smithi102.stdout:0/654: dread d6/de/d13/d1f/d55/d57/d7d/d7f/f8b [0,4194304] 0 2024-03-26T18:54:41.514 INFO:tasks.workunit.client.0.smithi102.stdout:5/839: write d2/d25/f2b [2890537,84213] 0 2024-03-26T18:54:41.514 INFO:tasks.workunit.client.0.smithi102.stdout:5/840: chown d2/f10 65558918 1 2024-03-26T18:54:41.516 INFO:tasks.workunit.client.1.smithi190.stdout:1/820: rename d3/da/d3a/d99/ld0 to d3/da/dd/d2f/d5b/db5/l10b 0 2024-03-26T18:54:41.519 INFO:tasks.workunit.client.0.smithi102.stdout:6/589: unlink d2/df/l40 0 2024-03-26T18:54:41.520 INFO:tasks.workunit.client.0.smithi102.stdout:6/590: dread d2/df/d72/db7/f95 [0,4194304] 0 2024-03-26T18:54:41.520 INFO:tasks.workunit.client.0.smithi102.stdout:6/591: chown d2/d3/fb6 210870320 1 2024-03-26T18:54:41.525 INFO:tasks.workunit.client.0.smithi102.stdout:6/592: dread d2/df/f96 [0,4194304] 0 2024-03-26T18:54:41.525 INFO:tasks.workunit.client.0.smithi102.stdout:6/593: stat d2/d3 0 2024-03-26T18:54:41.525 INFO:tasks.workunit.client.0.smithi102.stdout:6/594: write d2/d3d/d53/fa6 [90514,92576] 0 2024-03-26T18:54:41.526 INFO:tasks.workunit.client.1.smithi190.stdout:1/821: mkdir d3/da/dd/d2f/d34/d10c 0 2024-03-26T18:54:41.526 INFO:tasks.workunit.client.1.smithi190.stdout:1/822: write d3/f7f [1433266,101853] 0 2024-03-26T18:54:41.526 INFO:tasks.workunit.client.1.smithi190.stdout:1/823: write d3/da/dd/d2f/d5b/f6b [4363027,68770] 0 2024-03-26T18:54:41.531 INFO:tasks.workunit.client.0.smithi102.stdout:8/645: dwrite d0/d15/d5d/dd3/d2c/d58/d75/f86 [0,4194304] 0 2024-03-26T18:54:41.534 INFO:tasks.workunit.client.0.smithi102.stdout:1/575: truncate de/f3f 689116 0 2024-03-26T18:54:41.538 INFO:tasks.workunit.client.0.smithi102.stdout:5/841: symlink d2/d25/d75/db9/d43/d9e/d82/l126 0 2024-03-26T18:54:41.540 INFO:tasks.workunit.client.1.smithi190.stdout:5/834: dwrite d2/d17/d1b/db6/d96/fa4 [0,4194304] 0 2024-03-26T18:54:41.540 INFO:tasks.workunit.client.0.smithi102.stdout:6/595: mkdir d2/df/d72/db7/dbd 0 2024-03-26T18:54:41.542 INFO:tasks.workunit.client.0.smithi102.stdout:8/646: mkdir d0/d3c/d96/dd5 0 2024-03-26T18:54:41.544 INFO:tasks.workunit.client.1.smithi190.stdout:1/824: rmdir d3/d73/d88/df1 39 2024-03-26T18:54:41.545 INFO:tasks.workunit.client.0.smithi102.stdout:5/842: getdents d2/d25/d75/def/dec 0 2024-03-26T18:54:41.545 INFO:tasks.workunit.client.0.smithi102.stdout:5/843: readlink d2/d25/d75/db9/db3/lbc 0 2024-03-26T18:54:41.546 INFO:tasks.workunit.client.0.smithi102.stdout:2/611: dwrite d3/d82/d93/d9e/fa [0,4194304] 0 2024-03-26T18:54:41.546 INFO:tasks.workunit.client.0.smithi102.stdout:2/612: chown d3/d2a/f37 19124507 1 2024-03-26T18:54:41.549 INFO:tasks.workunit.client.0.smithi102.stdout:5/844: creat d2/d25/d75/db9/d43/d117/f127 x:0 0 0 2024-03-26T18:54:41.549 INFO:tasks.workunit.client.0.smithi102.stdout:5/845: truncate d2/d2c/f58 2443976 0 2024-03-26T18:54:41.552 INFO:tasks.workunit.client.0.smithi102.stdout:2/613: link d3/d11/d15/d31/l45 d3/d2d/d67/dc5/ld9 0 2024-03-26T18:54:41.553 INFO:tasks.workunit.client.0.smithi102.stdout:4/487: dwrite d1/d25/f69 [0,4194304] 0 2024-03-26T18:54:41.554 INFO:tasks.workunit.client.0.smithi102.stdout:8/647: getdents d0/d15/d5d/d89 0 2024-03-26T18:54:41.556 INFO:tasks.workunit.client.0.smithi102.stdout:0/655: dwrite d6/d2d/d3d/f40 [0,4194304] 0 2024-03-26T18:54:41.556 INFO:tasks.workunit.client.0.smithi102.stdout:0/656: dread - d6/de/d13/d25/d31/d45/d82/d84/d88/fab zero size 2024-03-26T18:54:41.558 INFO:tasks.workunit.client.1.smithi190.stdout:5/835: creat d2/d17/d62/f116 x:0 0 0 2024-03-26T18:54:41.558 INFO:tasks.workunit.client.1.smithi190.stdout:5/836: chown d2/d17/d62/d64/f70 399448224 1 2024-03-26T18:54:41.559 INFO:tasks.workunit.client.1.smithi190.stdout:1/825: fdatasync d3/da/dd/d2f/d5b/f6b 0 2024-03-26T18:54:41.560 INFO:tasks.workunit.client.0.smithi102.stdout:0/657: dread d6/de/d13/d1f/d55/d57/d7d/d7f/f8b [0,4194304] 0 2024-03-26T18:54:41.562 INFO:tasks.workunit.client.1.smithi190.stdout:1/826: dread d3/da/d2e/d59/f103 [0,4194304] 0 2024-03-26T18:54:41.562 INFO:tasks.workunit.client.1.smithi190.stdout:1/827: chown d3/da/dd/d2f/d5b/d5d/fc0 1 1 2024-03-26T18:54:41.562 INFO:tasks.workunit.client.1.smithi190.stdout:1/828: readlink d3/lc9 0 2024-03-26T18:54:41.562 INFO:tasks.workunit.client.0.smithi102.stdout:4/488: truncate d1/d27/f3c 3082482 0 2024-03-26T18:54:41.562 INFO:tasks.workunit.client.0.smithi102.stdout:4/489: write d1/d17/d1f/d5d/d8b/f5e [4852317,13287] 0 2024-03-26T18:54:41.564 INFO:tasks.workunit.client.0.smithi102.stdout:1/576: dwrite de/d16/d1d/d61/d6a/da5/da6/fba [0,4194304] 0 2024-03-26T18:54:41.565 INFO:tasks.workunit.client.1.smithi190.stdout:5/837: rename d2/d17/f44 to d2/d17/d62/d109/f117 0 2024-03-26T18:54:41.568 INFO:tasks.workunit.client.1.smithi190.stdout:1/829: creat d3/d8f/da7/f10d x:0 0 0 2024-03-26T18:54:41.577 INFO:tasks.workunit.client.0.smithi102.stdout:6/596: dwrite d2/d3d/d53/d77/f88 [0,4194304] 0 2024-03-26T18:54:41.577 INFO:tasks.workunit.client.0.smithi102.stdout:0/658: write d6/d2d/d3d/fa2 [3009688,70641] 0 2024-03-26T18:54:41.577 INFO:tasks.workunit.client.0.smithi102.stdout:6/597: write d2/d3d/d53/fa6 [203446,57849] 0 2024-03-26T18:54:41.587 INFO:tasks.workunit.client.0.smithi102.stdout:4/490: dread d1/d25/f69 [0,4194304] 0 2024-03-26T18:54:41.587 INFO:tasks.workunit.client.0.smithi102.stdout:4/491: fdatasync d1/d17/d1a/f4f 0 2024-03-26T18:54:41.595 INFO:tasks.workunit.client.0.smithi102.stdout:1/577: creat de/d2e/d62/fbe x:0 0 0 2024-03-26T18:54:41.598 INFO:tasks.workunit.client.0.smithi102.stdout:2/614: dwrite d3/d82/d93/d9e/da0/fb0 [0,4194304] 0 2024-03-26T18:54:41.598 INFO:tasks.workunit.client.0.smithi102.stdout:2/615: stat d3/d82/d93/d9e/da0/fc3 0 2024-03-26T18:54:41.598 INFO:tasks.workunit.client.0.smithi102.stdout:2/616: creat d3/d20/d78/d84/fda x:0 0 0 2024-03-26T18:54:41.600 INFO:tasks.workunit.client.0.smithi102.stdout:6/598: mknod d2/df/dba/cbe 0 2024-03-26T18:54:41.602 INFO:tasks.workunit.client.0.smithi102.stdout:4/492: symlink d1/d17/d1f/d5d/d8b/d3e/l97 0 2024-03-26T18:54:41.606 INFO:tasks.workunit.client.0.smithi102.stdout:5/846: dwrite d2/f17 [4194304,4194304] 0 2024-03-26T18:54:41.606 INFO:tasks.workunit.client.0.smithi102.stdout:8/648: dwrite d0/d15/fa1 [0,4194304] 0 2024-03-26T18:54:41.608 INFO:tasks.workunit.client.0.smithi102.stdout:2/617: symlink d3/d82/d93/d9e/da0/ldb 0 2024-03-26T18:54:41.608 INFO:tasks.workunit.client.0.smithi102.stdout:2/618: fdatasync d3/d11/d15/d31/d39/f63 0 2024-03-26T18:54:41.608 INFO:tasks.workunit.client.0.smithi102.stdout:2/619: creat d3/d11/fdc x:0 0 0 2024-03-26T18:54:41.611 INFO:tasks.workunit.client.0.smithi102.stdout:5/847: mkdir d2/d25/d75/d128 0 2024-03-26T18:54:41.614 INFO:tasks.workunit.client.0.smithi102.stdout:8/649: mknod d0/d15/d5d/dd3/d26/d19/d9f/cd6 0 2024-03-26T18:54:41.616 INFO:tasks.workunit.client.0.smithi102.stdout:2/620: symlink d3/d11/d27/ldd 0 2024-03-26T18:54:41.617 INFO:tasks.workunit.client.0.smithi102.stdout:5/848: truncate d2/dea/d4f/fb2 264776 0 2024-03-26T18:54:41.618 INFO:tasks.workunit.client.0.smithi102.stdout:8/650: creat d0/d3c/d96/fd7 x:0 0 0 2024-03-26T18:54:41.620 INFO:tasks.workunit.client.0.smithi102.stdout:2/621: symlink d3/d11/d15/d46/lde 0 2024-03-26T18:54:41.623 INFO:tasks.workunit.client.0.smithi102.stdout:8/651: creat d0/d15/d5d/dd3/d2c/d2f/fd8 x:0 0 0 2024-03-26T18:54:41.623 INFO:tasks.workunit.client.0.smithi102.stdout:8/652: creat d0/d15/d5d/dd3/d2c/d2f/fd9 x:0 0 0 2024-03-26T18:54:41.624 INFO:tasks.workunit.client.0.smithi102.stdout:2/622: symlink d3/d20/d78/d84/d6e/ldf 0 2024-03-26T18:54:41.624 INFO:tasks.workunit.client.0.smithi102.stdout:2/623: chown d3/d11/d15/c49 12279412 1 2024-03-26T18:54:41.624 INFO:tasks.workunit.client.0.smithi102.stdout:2/624: write d3/d20/d78/d7c/f85 [1823374,5026] 0 2024-03-26T18:54:41.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:41 smithi102 ceph-mon[29002]: pgmap v248: 65 pgs: 65 active+clean; 2.8 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 273 MiB/s rd, 343 MiB/s wr, 544 op/s 2024-03-26T18:54:41.626 INFO:tasks.workunit.client.0.smithi102.stdout:5/849: dread d2/d25/d75/db9/d43/d9e/d82/f106 [0,4194304] 0 2024-03-26T18:54:41.627 INFO:tasks.workunit.client.0.smithi102.stdout:5/850: chown d2/dea/l86 238 1 2024-03-26T18:54:41.627 INFO:tasks.workunit.client.0.smithi102.stdout:8/653: unlink d0/d3c/d96/d98/db6/f4a 0 2024-03-26T18:54:41.627 INFO:tasks.workunit.client.0.smithi102.stdout:8/654: dread - d0/d15/d5d/dd3/d2c/d2f/fd8 zero size 2024-03-26T18:54:41.627 INFO:tasks.workunit.client.0.smithi102.stdout:2/625: mkdir d3/d82/d93/d9e/d99/de0 0 2024-03-26T18:54:41.627 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:41 smithi190 ceph-mon[35043]: pgmap v248: 65 pgs: 65 active+clean; 2.8 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 273 MiB/s rd, 343 MiB/s wr, 544 op/s 2024-03-26T18:54:41.628 INFO:tasks.workunit.client.0.smithi102.stdout:5/851: symlink d2/d25/l129 0 2024-03-26T18:54:41.630 INFO:tasks.workunit.client.0.smithi102.stdout:2/626: mkdir d3/d20/d78/d84/de1 0 2024-03-26T18:54:41.630 INFO:tasks.workunit.client.0.smithi102.stdout:1/578: dwrite de/d16/d1d/d2a/d2c/d56/d78/d80/f98 [0,4194304] 0 2024-03-26T18:54:41.631 INFO:tasks.workunit.client.0.smithi102.stdout:5/852: creat d2/d25/d75/def/dec/d112/f12a x:0 0 0 2024-03-26T18:54:41.634 INFO:tasks.workunit.client.0.smithi102.stdout:1/579: rename de/d16/d1d/d61/l6c to de/d16/d1d/d2a/d2c/d56/d78/d80/d8e/lbf 0 2024-03-26T18:54:41.637 INFO:tasks.workunit.client.0.smithi102.stdout:1/580: symlink de/d16/d1d/d2a/lc0 0 2024-03-26T18:54:41.638 INFO:tasks.workunit.client.1.smithi190.stdout:0/977: sync 2024-03-26T18:54:41.638 INFO:tasks.workunit.client.1.smithi190.stdout:0/978: chown d0/dd/lbb 230843464 1 2024-03-26T18:54:41.638 INFO:tasks.workunit.client.1.smithi190.stdout:0/979: truncate d0/dd/d20/d1c/d8b/d8d/d8f/dfe/f12c 1011637 0 2024-03-26T18:54:41.638 INFO:tasks.workunit.client.1.smithi190.stdout:3/754: sync 2024-03-26T18:54:41.638 INFO:tasks.workunit.client.1.smithi190.stdout:6/779: sync 2024-03-26T18:54:41.639 INFO:tasks.workunit.client.0.smithi102.stdout:8/655: dread d0/d3c/d96/d98/db6/f59 [0,4194304] 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.0.smithi102.stdout:2/627: dread d3/d2a/d5f/f90 [0,4194304] 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.0.smithi102.stdout:8/656: readlink d0/d15/l25 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.0.smithi102.stdout:3/785: dwrite df/d1c/fbf [0,4194304] 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.0.smithi102.stdout:8/657: fdatasync d0/d15/da0/fb3 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.0.smithi102.stdout:2/628: stat d3/d20/d78/d7c/f7e 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.0.smithi102.stdout:3/786: fsync df/d1c/d34/d53/fa4 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.0.smithi102.stdout:0/659: dwrite d6/de/d93/dac/fc2 [0,4194304] 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.0.smithi102.stdout:6/599: dwrite d2/d3d/d53/f66 [0,4194304] 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.0.smithi102.stdout:6/600: truncate d2/d7/f8 9216212 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.0.smithi102.stdout:1/581: symlink de/d16/d1d/d2a/d2c/d56/d5f/daa/lc1 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.1.smithi190.stdout:5/838: dwrite d2/d17/d98/d3a/d3c/f6a [0,4194304] 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.1.smithi190.stdout:1/830: dwrite d3/da/d2e/d59/f103 [0,4194304] 0 2024-03-26T18:54:41.640 INFO:tasks.workunit.client.1.smithi190.stdout:1/831: creat d3/da/d3a/f10e x:0 0 0 2024-03-26T18:54:41.641 INFO:tasks.workunit.client.1.smithi190.stdout:1/832: read - d3/da/dd/d2f/fed zero size 2024-03-26T18:54:41.641 INFO:tasks.workunit.client.0.smithi102.stdout:4/493: dwrite d1/d17/d1f/d5d/d8b/f51 [0,4194304] 0 2024-03-26T18:54:41.641 INFO:tasks.workunit.client.1.smithi190.stdout:3/755: rmdir d1/d3/d5a/d6e 39 2024-03-26T18:54:41.641 INFO:tasks.workunit.client.1.smithi190.stdout:3/756: write d1/d3/df/d18/d57/d91/d8c/faf [470050,63989] 0 2024-03-26T18:54:41.642 INFO:tasks.workunit.client.1.smithi190.stdout:3/757: stat d1/d3/d5a/d6e 0 2024-03-26T18:54:41.642 INFO:tasks.workunit.client.0.smithi102.stdout:8/658: rmdir d0/d15/d5d/dd3/d26/d5c 39 2024-03-26T18:54:41.645 INFO:tasks.workunit.client.0.smithi102.stdout:2/629: unlink d3/d11/d15/d31/f60 0 2024-03-26T18:54:41.648 INFO:tasks.workunit.client.0.smithi102.stdout:3/787: creat df/d1c/d34/f114 x:0 0 0 2024-03-26T18:54:41.654 INFO:tasks.workunit.client.1.smithi190.stdout:6/780: write d17/d28/d2c/d50/f9c [2475763,14300] 0 2024-03-26T18:54:41.656 INFO:tasks.workunit.client.1.smithi190.stdout:1/833: creat d3/d5f/f10f x:0 0 0 2024-03-26T18:54:41.656 INFO:tasks.workunit.client.1.smithi190.stdout:1/834: stat d3/da/dd/d2f/d5b/d83/feb 0 2024-03-26T18:54:41.657 INFO:tasks.workunit.client.1.smithi190.stdout:0/980: dread d0/dd/d20/d1c/d46/f72 [0,4194304] 0 2024-03-26T18:54:41.658 INFO:tasks.workunit.client.1.smithi190.stdout:3/758: unlink d1/d3/c8 0 2024-03-26T18:54:41.660 INFO:tasks.workunit.client.0.smithi102.stdout:0/660: dread d6/de/d13/d1f/f3b [0,4194304] 0 2024-03-26T18:54:41.660 INFO:tasks.workunit.client.0.smithi102.stdout:6/601: rename d2/d3d/d53/d5c/f65 to d2/d7/d3c/fbf 0 2024-03-26T18:54:41.664 INFO:tasks.workunit.client.0.smithi102.stdout:3/788: dread df/d1c/d70/fbe [0,4194304] 0 2024-03-26T18:54:41.666 INFO:tasks.workunit.client.1.smithi190.stdout:1/835: creat d3/d9/db/f110 x:0 0 0 2024-03-26T18:54:41.668 INFO:tasks.workunit.client.1.smithi190.stdout:0/981: unlink d0/dd/d20/d1c/d8b/d8d/cb8 0 2024-03-26T18:54:41.670 INFO:tasks.workunit.client.1.smithi190.stdout:3/759: truncate d1/f26 2808504 0 2024-03-26T18:54:41.674 INFO:tasks.workunit.client.0.smithi102.stdout:2/630: truncate d3/d11/d27/f6b 2825504 0 2024-03-26T18:54:41.674 INFO:tasks.workunit.client.0.smithi102.stdout:2/631: read d3/d11/d15/d46/d7b/d97/fad [273904,58873] 0 2024-03-26T18:54:41.675 INFO:tasks.workunit.client.1.smithi190.stdout:1/836: symlink d3/da/daf/l111 0 2024-03-26T18:54:41.676 INFO:tasks.workunit.client.0.smithi102.stdout:5/853: dwrite d2/f3a [0,4194304] 0 2024-03-26T18:54:41.676 INFO:tasks.workunit.client.0.smithi102.stdout:5/854: chown d2/d25/d5b/d72/fc5 154551 1 2024-03-26T18:54:41.677 INFO:tasks.workunit.client.1.smithi190.stdout:5/839: dwrite d2/d17/d98/d3a/f6b [0,4194304] 0 2024-03-26T18:54:41.677 INFO:tasks.workunit.client.0.smithi102.stdout:0/661: mkdir d6/de/d93/de4 0 2024-03-26T18:54:41.681 INFO:tasks.workunit.client.1.smithi190.stdout:3/760: symlink d1/de/d8a/de1/le5 0 2024-03-26T18:54:41.682 INFO:tasks.workunit.client.0.smithi102.stdout:3/789: dread df/f7b [4194304,4194304] 0 2024-03-26T18:54:41.682 INFO:tasks.workunit.client.0.smithi102.stdout:0/662: read d6/de/d13/d25/d31/f8e [272778,95290] 0 2024-03-26T18:54:41.682 INFO:tasks.workunit.client.0.smithi102.stdout:3/790: write df/d29/d31/d36/f41 [3962169,96316] 0 2024-03-26T18:54:41.682 INFO:tasks.workunit.client.0.smithi102.stdout:3/791: dread - df/d8b/dd5/ffe zero size 2024-03-26T18:54:41.682 INFO:tasks.workunit.client.0.smithi102.stdout:0/663: write d6/de/d13/d1f/d55/d57/d7d/d7f/fe2 [258020,84718] 0 2024-03-26T18:54:41.682 INFO:tasks.workunit.client.0.smithi102.stdout:0/664: chown d6/d2d/d3d/fa2 953 1 2024-03-26T18:54:41.682 INFO:tasks.workunit.client.0.smithi102.stdout:6/602: mknod d2/df/d72/db7/dbd/cc0 0 2024-03-26T18:54:41.683 INFO:tasks.workunit.client.1.smithi190.stdout:6/781: rename d17/d28/d38 to d17/d28/d2c/db8/df3/d109 0 2024-03-26T18:54:41.685 INFO:tasks.workunit.client.0.smithi102.stdout:5/855: dread d2/d25/d75/db9/fd5 [0,4194304] 0 2024-03-26T18:54:41.685 INFO:tasks.workunit.client.0.smithi102.stdout:8/659: dwrite d0/d3c/d96/fd7 [0,4194304] 0 2024-03-26T18:54:41.687 INFO:tasks.workunit.client.1.smithi190.stdout:1/837: symlink d3/da/dd/d76/l112 0 2024-03-26T18:54:41.687 INFO:tasks.workunit.client.1.smithi190.stdout:1/838: fsync d3/d8f/ff6 0 2024-03-26T18:54:41.688 INFO:tasks.workunit.client.1.smithi190.stdout:1/839: read d3/da/dd/d95/d52/fd8 [86806,100188] 0 2024-03-26T18:54:41.690 INFO:tasks.workunit.client.1.smithi190.stdout:3/761: dread d1/f4 [0,4194304] 0 2024-03-26T18:54:41.690 INFO:tasks.workunit.client.1.smithi190.stdout:5/840: write d2/d17/d1b/db6/d88/f8c [2530849,3176] 0 2024-03-26T18:54:41.691 INFO:tasks.workunit.client.0.smithi102.stdout:0/665: creat d6/de/d13/d9a/d85/fe5 x:0 0 0 2024-03-26T18:54:41.692 INFO:tasks.workunit.client.1.smithi190.stdout:6/782: write f2 [97304,40417] 0 2024-03-26T18:54:41.692 INFO:tasks.workunit.client.1.smithi190.stdout:6/783: stat d17/f87 0 2024-03-26T18:54:41.693 INFO:tasks.workunit.client.1.smithi190.stdout:0/982: dwrite d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/f82 [0,4194304] 0 2024-03-26T18:54:41.698 INFO:tasks.workunit.client.0.smithi102.stdout:6/603: mkdir d2/df/d71/d80/d91/dc1 0 2024-03-26T18:54:41.698 INFO:tasks.workunit.client.0.smithi102.stdout:6/604: dread - d2/df/d35/d43/d75/f8c zero size 2024-03-26T18:54:41.700 INFO:tasks.workunit.client.1.smithi190.stdout:1/840: mknod d3/d9/db/dd4/c113 0 2024-03-26T18:54:41.700 INFO:tasks.workunit.client.1.smithi190.stdout:1/841: write d3/d73/faa [4248711,107907] 0 2024-03-26T18:54:41.700 INFO:tasks.workunit.client.1.smithi190.stdout:1/842: chown d3/da/dd/d95/d52/f5a 6910 1 2024-03-26T18:54:41.700 INFO:tasks.workunit.client.1.smithi190.stdout:1/843: fsync d3/da/dd/f8c 0 2024-03-26T18:54:41.702 INFO:tasks.workunit.client.0.smithi102.stdout:5/856: creat d2/dea/d4f/da9/d10e/f12b x:0 0 0 2024-03-26T18:54:41.705 INFO:tasks.workunit.client.1.smithi190.stdout:5/841: dread d2/d17/ff7 [0,4194304] 0 2024-03-26T18:54:41.705 INFO:tasks.workunit.client.1.smithi190.stdout:1/844: dread d3/da/d2e/f30 [0,4194304] 0 2024-03-26T18:54:41.705 INFO:tasks.workunit.client.1.smithi190.stdout:1/845: readlink d3/da/dd/d95/d52/lca 0 2024-03-26T18:54:41.705 INFO:tasks.workunit.client.1.smithi190.stdout:5/842: dread - d2/d17/d1b/db6/d24/f108 zero size 2024-03-26T18:54:41.706 INFO:tasks.workunit.client.1.smithi190.stdout:6/784: mkdir d17/d28/d8b/d95/dda/d10a 0 2024-03-26T18:54:41.711 INFO:tasks.workunit.client.0.smithi102.stdout:4/494: rename d1/d17/d1f/d5d/d8b/d3e/d4c to d1/d27/d98 0 2024-03-26T18:54:41.711 INFO:tasks.workunit.client.0.smithi102.stdout:4/495: write d1/d17/d1a/d21/f95 [484506,65314] 0 2024-03-26T18:54:41.715 INFO:tasks.workunit.client.0.smithi102.stdout:6/605: mknod d2/d7/d3c/cc2 0 2024-03-26T18:54:41.715 INFO:tasks.workunit.client.0.smithi102.stdout:6/606: getdents d2/df/d35 0 2024-03-26T18:54:41.717 INFO:tasks.workunit.client.1.smithi190.stdout:0/983: rename d0/dd5/dee/df5/f14f to d0/dd5/dee/df5/dfd/f15d 0 2024-03-26T18:54:41.718 INFO:tasks.workunit.client.1.smithi190.stdout:5/843: write d2/d17/d1b/d60/d63/dcd/fed [4087542,130354] 0 2024-03-26T18:54:41.718 INFO:tasks.workunit.client.1.smithi190.stdout:1/846: creat d3/da/d3a/da5/f114 x:0 0 0 2024-03-26T18:54:41.720 INFO:tasks.workunit.client.0.smithi102.stdout:6/607: symlink d2/d3d/d53/lc3 0 2024-03-26T18:54:41.723 INFO:tasks.workunit.client.1.smithi190.stdout:0/984: mkdir d0/dd/d20/d1c/d8b/d8d/d8f/dfe/d15e 0 2024-03-26T18:54:41.724 INFO:tasks.workunit.client.1.smithi190.stdout:5/844: mkdir d2/d17/d1b/d60/d63/dcd/d118 0 2024-03-26T18:54:41.725 INFO:tasks.workunit.client.1.smithi190.stdout:1/847: unlink d3/d8f/da7/f10d 0 2024-03-26T18:54:41.726 INFO:tasks.workunit.client.0.smithi102.stdout:4/496: dread d1/d17/d1f/f2d [0,4194304] 0 2024-03-26T18:54:41.726 INFO:tasks.workunit.client.0.smithi102.stdout:4/497: dread - d1/d17/d1a/d21/f79 zero size 2024-03-26T18:54:41.726 INFO:tasks.workunit.client.0.smithi102.stdout:4/498: truncate d1/d17/d1a/d21/d77/f81 1346155 0 2024-03-26T18:54:41.731 INFO:tasks.workunit.client.0.smithi102.stdout:4/499: dread d1/d27/f4d [0,4194304] 0 2024-03-26T18:54:41.731 INFO:tasks.workunit.client.0.smithi102.stdout:4/500: write d1/d17/d1a/d21/f56 [4941717,64122] 0 2024-03-26T18:54:41.731 INFO:tasks.workunit.client.0.smithi102.stdout:4/501: creat d1/d27/f99 x:0 0 0 2024-03-26T18:54:41.731 INFO:tasks.workunit.client.0.smithi102.stdout:4/502: stat d1/d17/d2a/l5c 0 2024-03-26T18:54:41.737 INFO:tasks.workunit.client.0.smithi102.stdout:3/792: dwrite df/d8b/dbb/fcf [0,4194304] 0 2024-03-26T18:54:41.739 INFO:tasks.workunit.client.1.smithi190.stdout:5/845: write d2/d17/d98/d3a/d3c/f6a [4066721,87028] 0 2024-03-26T18:54:41.741 INFO:tasks.workunit.client.0.smithi102.stdout:4/503: dread d1/f26 [0,4194304] 0 2024-03-26T18:54:41.741 INFO:tasks.workunit.client.0.smithi102.stdout:8/660: dwrite d0/d9/fb5 [0,4194304] 0 2024-03-26T18:54:41.741 INFO:tasks.workunit.client.0.smithi102.stdout:8/661: write d0/d15/fc8 [457114,93624] 0 2024-03-26T18:54:41.741 INFO:tasks.workunit.client.0.smithi102.stdout:8/662: chown d0/d15/d5d/dd3/cb2 26703 1 2024-03-26T18:54:41.743 INFO:tasks.workunit.client.0.smithi102.stdout:0/666: dwrite d6/d2d/d3d/f44 [4194304,4194304] 0 2024-03-26T18:54:41.743 INFO:tasks.workunit.client.0.smithi102.stdout:0/667: write d6/d2d/d3d/d80/d83/fd1 [895666,73215] 0 2024-03-26T18:54:41.743 INFO:tasks.workunit.client.0.smithi102.stdout:5/857: dwrite d2/d25/d75/db9/db3/d115/ff3 [0,4194304] 0 2024-03-26T18:54:41.744 INFO:tasks.workunit.client.1.smithi190.stdout:3/762: dwrite d1/d3/df/d18/d35/d41/dae/d9c/fa6 [0,4194304] 0 2024-03-26T18:54:41.744 INFO:tasks.workunit.client.1.smithi190.stdout:3/763: getdents d1/de/d81/dd7 0 2024-03-26T18:54:41.744 INFO:tasks.workunit.client.1.smithi190.stdout:6/785: dwrite d17/d28/df6/da8/df4/dfb/dd7/ff5 [0,4194304] 0 2024-03-26T18:54:41.796 INFO:tasks.workunit.client.1.smithi190.stdout:1/848: mkdir d3/da/dd/d2f/d33/d115 0 2024-03-26T18:54:41.796 INFO:tasks.workunit.client.1.smithi190.stdout:1/849: dread - d3/d73/d88/dcc/ff8 zero size 2024-03-26T18:54:41.797 INFO:tasks.workunit.client.1.smithi190.stdout:5/846: mknod d2/d17/d1b/d60/d63/d85/ddc/c119 0 2024-03-26T18:54:41.798 INFO:tasks.workunit.client.1.smithi190.stdout:5/847: chown d2/d17/d62/d64/ld0 1463474 1 2024-03-26T18:54:41.802 INFO:tasks.workunit.client.0.smithi102.stdout:3/793: creat df/d1c/d34/d53/f115 x:0 0 0 2024-03-26T18:54:41.807 INFO:tasks.workunit.client.0.smithi102.stdout:9/575: sync 2024-03-26T18:54:41.807 INFO:tasks.workunit.client.0.smithi102.stdout:7/544: sync 2024-03-26T18:54:41.807 INFO:tasks.workunit.client.0.smithi102.stdout:7/545: write d7/d12/d51/f58 [892322,123029] 0 2024-03-26T18:54:41.811 INFO:tasks.workunit.client.0.smithi102.stdout:4/504: mkdir d1/d17/d2a/d9a 0 2024-03-26T18:54:41.813 INFO:tasks.workunit.client.1.smithi190.stdout:3/764: creat d1/d3/df/d31/d5b/fe6 x:0 0 0 2024-03-26T18:54:41.815 INFO:tasks.workunit.client.0.smithi102.stdout:9/576: dread df/d1a/d23/d4e/d8e/fce [0,4194304] 0 2024-03-26T18:54:41.815 INFO:tasks.workunit.client.0.smithi102.stdout:9/577: creat df/d1a/d57/fd1 x:0 0 0 2024-03-26T18:54:41.816 INFO:tasks.workunit.client.0.smithi102.stdout:8/663: truncate d0/d15/d5d/dd3/d26/f1f 960010 0 2024-03-26T18:54:41.817 INFO:tasks.workunit.client.1.smithi190.stdout:6/786: symlink d17/d28/d74/d97/l10b 0 2024-03-26T18:54:41.818 INFO:tasks.workunit.client.0.smithi102.stdout:7/546: write d7/f41 [2012258,119150] 0 2024-03-26T18:54:41.821 INFO:tasks.workunit.client.1.smithi190.stdout:5/848: creat d2/d17/d1b/d60/d63/dcd/d118/f11a x:0 0 0 2024-03-26T18:54:41.834 INFO:tasks.workunit.client.1.smithi190.stdout:5/849: creat d2/d17/d1b/d60/d63/d85/d99/f11b x:0 0 0 2024-03-26T18:54:41.837 INFO:tasks.workunit.client.1.smithi190.stdout:3/765: mkdir d1/de/d8a/de1/de7 0 2024-03-26T18:54:41.837 INFO:tasks.workunit.client.1.smithi190.stdout:3/766: dread - d1/d3/d5a/fa9 zero size 2024-03-26T18:54:41.838 INFO:tasks.workunit.client.0.smithi102.stdout:3/794: dwrite df/d29/d31/d36/ffb [0,4194304] 0 2024-03-26T18:54:41.838 INFO:tasks.workunit.client.0.smithi102.stdout:8/664: dread d0/d9/f2b [0,4194304] 0 2024-03-26T18:54:41.838 INFO:tasks.workunit.client.1.smithi190.stdout:5/850: dread d2/d17/d1b/f72 [0,4194304] 0 2024-03-26T18:54:41.840 INFO:tasks.workunit.client.1.smithi190.stdout:5/851: write d2/d17/fe5 [335892,45277] 0 2024-03-26T18:54:41.840 INFO:tasks.workunit.client.1.smithi190.stdout:5/852: dread - d2/d17/d1b/d60/f10c zero size 2024-03-26T18:54:41.840 INFO:tasks.workunit.client.1.smithi190.stdout:5/853: creat d2/d17/d1b/d60/f11c x:0 0 0 2024-03-26T18:54:41.840 INFO:tasks.workunit.client.1.smithi190.stdout:5/854: write d2/d17/d1b/db6/f9a [4697010,49463] 0 2024-03-26T18:54:41.843 INFO:tasks.workunit.client.1.smithi190.stdout:3/767: creat d1/dd3/fe8 x:0 0 0 2024-03-26T18:54:41.847 INFO:tasks.workunit.client.1.smithi190.stdout:6/787: dwrite d17/d28/df6/da8/df4/dfb/fbe [0,4194304] 0 2024-03-26T18:54:41.847 INFO:tasks.workunit.client.1.smithi190.stdout:6/788: read - d17/d28/d8b/d95/dda/ffc zero size 2024-03-26T18:54:41.847 INFO:tasks.workunit.client.1.smithi190.stdout:6/789: creat d17/d28/d2b/f10c x:0 0 0 2024-03-26T18:54:41.847 INFO:tasks.workunit.client.1.smithi190.stdout:6/790: stat d17/dd3/fef 0 2024-03-26T18:54:41.850 INFO:tasks.workunit.client.1.smithi190.stdout:0/985: rename d0/dd/d20/d1c/d8b/d8d/d8f/dfe to d0/dd5/d15f 0 2024-03-26T18:54:41.850 INFO:tasks.workunit.client.1.smithi190.stdout:0/986: fsync d0/dd/d20/d1c/d8b/f14b 0 2024-03-26T18:54:41.850 INFO:tasks.workunit.client.1.smithi190.stdout:0/987: creat d0/dd/d20/d1c/d46/da7/df0/d87/f160 x:0 0 0 2024-03-26T18:54:41.850 INFO:tasks.workunit.client.0.smithi102.stdout:4/505: mknod d1/d27/c9b 0 2024-03-26T18:54:41.853 INFO:tasks.workunit.client.1.smithi190.stdout:3/768: creat d1/d3/df/d18/d35/d41/dae/fe9 x:0 0 0 2024-03-26T18:54:41.862 INFO:tasks.workunit.client.1.smithi190.stdout:6/791: dread d17/d28/d2c/d50/fce [0,4194304] 0 2024-03-26T18:54:41.863 INFO:tasks.workunit.client.0.smithi102.stdout:8/665: dwrite d0/d15/d5d/dd3/d2c/d33/d87/f9e [0,4194304] 0 2024-03-26T18:54:41.866 INFO:tasks.workunit.client.0.smithi102.stdout:4/506: dread d1/d17/d1f/d5d/d8b/f7f [0,4194304] 0 2024-03-26T18:54:41.866 INFO:tasks.workunit.client.0.smithi102.stdout:5/858: dwrite d2/d14/d1a/f61 [0,4194304] 0 2024-03-26T18:54:41.866 INFO:tasks.workunit.client.0.smithi102.stdout:9/578: dwrite df/d1a/d23/f25 [0,4194304] 0 2024-03-26T18:54:41.866 INFO:tasks.workunit.client.0.smithi102.stdout:5/859: stat d2/d14/d1a/f1b 0 2024-03-26T18:54:41.866 INFO:tasks.workunit.client.0.smithi102.stdout:9/579: chown df/d1a/d23/d3e/d75/c91 44367097 1 2024-03-26T18:54:41.874 INFO:tasks.workunit.client.0.smithi102.stdout:8/666: dread d0/d15/d5d/dd3/d2c/d4e/fcb [0,4194304] 0 2024-03-26T18:54:41.874 INFO:tasks.workunit.client.1.smithi190.stdout:3/769: dwrite d1/d3/d5a/fa9 [0,4194304] 0 2024-03-26T18:54:41.875 INFO:tasks.workunit.client.1.smithi190.stdout:0/988: dwrite d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/f95 [0,4194304] 0 2024-03-26T18:54:41.885 INFO:tasks.workunit.client.0.smithi102.stdout:5/860: dwrite d2/d2c/f8d [0,4194304] 0 2024-03-26T18:54:41.885 INFO:tasks.workunit.client.0.smithi102.stdout:5/861: chown d2/d25/d75/def/dec/d112/ld6 111748 1 2024-03-26T18:54:41.886 INFO:tasks.workunit.client.1.smithi190.stdout:3/770: write d1/d3/df/d18/d57/d91/fbc [2853668,100175] 0 2024-03-26T18:54:41.899 INFO:tasks.workunit.client.1.smithi190.stdout:1/850: rename d3/d9/c40 to d3/da/dd/d2f/d5b/d5d/c116 0 2024-03-26T18:54:41.900 INFO:tasks.workunit.client.1.smithi190.stdout:1/851: write d3/da/d2e/f30 [1313115,42004] 0 2024-03-26T18:54:41.903 INFO:tasks.workunit.client.1.smithi190.stdout:6/792: creat d17/d28/d74/d7b/f10d x:0 0 0 2024-03-26T18:54:41.910 INFO:tasks.workunit.client.1.smithi190.stdout:0/989: unlink d0/dd/d20/d1c/d46/f5d 0 2024-03-26T18:54:41.913 INFO:tasks.workunit.client.1.smithi190.stdout:6/793: write d17/d28/df6/fa9 [1162527,58928] 0 2024-03-26T18:54:41.915 INFO:tasks.workunit.client.1.smithi190.stdout:3/771: mknod d1/dd3/cea 0 2024-03-26T18:54:41.918 INFO:tasks.workunit.client.1.smithi190.stdout:5/855: rename d2/l2b to d2/d17/d1b/db6/l11d 0 2024-03-26T18:54:41.918 INFO:tasks.workunit.client.1.smithi190.stdout:0/990: read d0/dd/d20/d1c/d46/d7c/fa9 [3947868,3405] 0 2024-03-26T18:54:41.923 INFO:tasks.workunit.client.1.smithi190.stdout:6/794: mkdir d17/d28/d2c/d63/d10e 0 2024-03-26T18:54:41.923 INFO:tasks.workunit.client.1.smithi190.stdout:6/795: creat d17/d28/d2c/d77/f10f x:0 0 0 2024-03-26T18:54:41.927 INFO:tasks.workunit.client.1.smithi190.stdout:3/772: mkdir d1/d47/deb 0 2024-03-26T18:54:41.927 INFO:tasks.workunit.client.1.smithi190.stdout:3/773: readlink d1/d3/lc8 0 2024-03-26T18:54:41.929 INFO:tasks.workunit.client.1.smithi190.stdout:5/856: rename d2/d17/d1b/d30/fde to d2/d17/d1b/db6/d24/d3b/f11e 0 2024-03-26T18:54:41.932 INFO:tasks.workunit.client.1.smithi190.stdout:0/991: getdents d0/dd/d20/d1c/d46/da7/df0/d87/d93/db5 0 2024-03-26T18:54:41.937 INFO:tasks.workunit.client.1.smithi190.stdout:1/852: rmdir d3/da/dd/d95/d52 39 2024-03-26T18:54:41.937 INFO:tasks.workunit.client.0.smithi102.stdout:4/507: rename d1/f9 to d1/d27/d98/f9c 0 2024-03-26T18:54:41.941 INFO:tasks.workunit.client.0.smithi102.stdout:1/582: sync 2024-03-26T18:54:41.941 INFO:tasks.workunit.client.0.smithi102.stdout:2/632: sync 2024-03-26T18:54:41.941 INFO:tasks.workunit.client.0.smithi102.stdout:2/633: chown d3/d2a/f34 9 1 2024-03-26T18:54:41.941 INFO:tasks.workunit.client.0.smithi102.stdout:4/508: write d1/d17/d1f/d5d/d8b/f7f [2914772,69255] 0 2024-03-26T18:54:41.942 INFO:tasks.workunit.client.0.smithi102.stdout:9/580: link df/d1a/f9a df/d84/fd2 0 2024-03-26T18:54:41.942 INFO:tasks.workunit.client.0.smithi102.stdout:9/581: read - df/d1a/d57/fd1 zero size 2024-03-26T18:54:41.946 INFO:tasks.workunit.client.0.smithi102.stdout:8/667: mkdir d0/d3c/d96/d98/dda 0 2024-03-26T18:54:41.947 INFO:tasks.workunit.client.1.smithi190.stdout:6/796: symlink d17/dd3/l110 0 2024-03-26T18:54:41.947 INFO:tasks.workunit.client.1.smithi190.stdout:6/797: stat d17/d28/d8b/d95/fd2 0 2024-03-26T18:54:41.947 INFO:tasks.workunit.client.1.smithi190.stdout:6/798: write d17/d28/d8b/d95/fd2 [868736,14027] 0 2024-03-26T18:54:41.947 INFO:tasks.workunit.client.1.smithi190.stdout:6/799: write f2 [1208554,123519] 0 2024-03-26T18:54:41.949 INFO:tasks.workunit.client.0.smithi102.stdout:5/862: creat d2/dea/db4/d11f/f12c x:0 0 0 2024-03-26T18:54:41.951 INFO:tasks.workunit.client.1.smithi190.stdout:1/853: rename d3/da/d45/f49 to d3/d8f/f117 0 2024-03-26T18:54:41.951 INFO:tasks.workunit.client.1.smithi190.stdout:1/854: chown d3/da/l4d 1891910242 1 2024-03-26T18:54:41.952 INFO:tasks.workunit.client.1.smithi190.stdout:6/800: unlink d17/d28/df6/f107 0 2024-03-26T18:54:41.953 INFO:tasks.workunit.client.0.smithi102.stdout:3/795: symlink df/d29/d3a/l116 0 2024-03-26T18:54:41.954 INFO:tasks.workunit.client.1.smithi190.stdout:1/855: symlink d3/da/dd/d2f/d5b/d83/l118 0 2024-03-26T18:54:41.956 INFO:tasks.workunit.client.1.smithi190.stdout:6/801: unlink d17/d28/d2c/f47 0 2024-03-26T18:54:41.957 INFO:tasks.workunit.client.0.smithi102.stdout:9/582: write df/d1a/d23/f25 [1215855,93602] 0 2024-03-26T18:54:41.958 INFO:tasks.workunit.client.0.smithi102.stdout:1/583: link de/d16/d1d/d2a/d2c/c7e de/d1f/d9f/cc2 0 2024-03-26T18:54:41.958 INFO:tasks.workunit.client.0.smithi102.stdout:1/584: chown de/d16/d1d/d61/d6a/da5/da6 2 1 2024-03-26T18:54:41.959 INFO:tasks.workunit.client.1.smithi190.stdout:5/857: dwrite d2/d17/d62/dd7/f10a [0,4194304] 0 2024-03-26T18:54:41.959 INFO:tasks.workunit.client.0.smithi102.stdout:2/634: mkdir d3/d82/de2 0 2024-03-26T18:54:41.961 INFO:tasks.workunit.client.0.smithi102.stdout:1/585: read de/d16/d1d/d2a/d2c/d56/d78/f8c [384,60972] 0 2024-03-26T18:54:41.961 INFO:tasks.workunit.client.0.smithi102.stdout:1/586: chown de/d16/la9 175 1 2024-03-26T18:54:41.962 INFO:tasks.workunit.client.1.smithi190.stdout:6/802: creat d17/d28/d2c/d63/d10e/f111 x:0 0 0 2024-03-26T18:54:41.962 INFO:tasks.workunit.client.0.smithi102.stdout:8/668: mknod d0/d15/d5d/dd3/d2c/da7/cdb 0 2024-03-26T18:54:41.963 INFO:tasks.workunit.client.1.smithi190.stdout:0/992: dwrite d0/dd/d20/d1c/d4e/f90 [0,4194304] 0 2024-03-26T18:54:41.963 INFO:tasks.workunit.client.1.smithi190.stdout:0/993: chown d0/dd/d20/d1c/d46/da7/df0/f7d 3228 1 2024-03-26T18:54:41.963 INFO:tasks.workunit.client.0.smithi102.stdout:5/863: truncate d2/d14/d1a/f1b 2365317 0 2024-03-26T18:54:41.964 INFO:tasks.workunit.client.1.smithi190.stdout:3/774: dread d1/de/d9f/fdd [0,4194304] 0 2024-03-26T18:54:41.965 INFO:tasks.workunit.client.0.smithi102.stdout:3/796: symlink df/d8b/dd5/l117 0 2024-03-26T18:54:41.967 INFO:tasks.workunit.client.1.smithi190.stdout:0/994: mkdir d0/dd/d20/d1c/d46/da7/df0/d4b/d161 0 2024-03-26T18:54:41.968 INFO:tasks.workunit.client.0.smithi102.stdout:9/583: symlink df/d1a/ld3 0 2024-03-26T18:54:41.970 INFO:tasks.workunit.client.0.smithi102.stdout:2/635: symlink d3/d82/d93/d9e/d1a/le3 0 2024-03-26T18:54:41.971 INFO:tasks.workunit.client.0.smithi102.stdout:1/587: mkdir de/d16/d1d/d2a/d2c/d56/d78/dc3 0 2024-03-26T18:54:41.973 INFO:tasks.workunit.client.1.smithi190.stdout:3/775: symlink d1/de/d8a/lec 0 2024-03-26T18:54:41.974 INFO:tasks.workunit.client.0.smithi102.stdout:8/669: creat d0/d3c/d6f/fdc x:0 0 0 2024-03-26T18:54:41.974 INFO:tasks.workunit.client.0.smithi102.stdout:1/588: read de/d16/d1d/d2a/d2c/d56/d78/d80/f85 [676137,48085] 0 2024-03-26T18:54:41.975 INFO:tasks.workunit.client.1.smithi190.stdout:5/858: write d2/d17/d1b/db6/d96/fa4 [3605265,120751] 0 2024-03-26T18:54:41.976 INFO:tasks.workunit.client.0.smithi102.stdout:5/864: rename d2/d25/d75/db9/d43/d9e/daf/lfa to d2/dea/d4f/da9/d10e/l12d 0 2024-03-26T18:54:41.978 INFO:tasks.workunit.client.0.smithi102.stdout:8/670: mknod d0/d15/d5d/dd3/cdd 0 2024-03-26T18:54:41.978 INFO:tasks.workunit.client.0.smithi102.stdout:8/671: chown d0/d3c 839 1 2024-03-26T18:54:41.978 INFO:tasks.workunit.client.0.smithi102.stdout:4/509: dwrite d1/d17/d1a/d21/d77/f71 [0,4194304] 0 2024-03-26T18:54:41.980 INFO:tasks.workunit.client.1.smithi190.stdout:5/859: unlink d2/d17/d98/d3a/fe0 0 2024-03-26T18:54:41.980 INFO:tasks.workunit.client.0.smithi102.stdout:1/589: mknod de/d16/d1d/d61/d6a/cc4 0 2024-03-26T18:54:41.980 INFO:tasks.workunit.client.0.smithi102.stdout:1/590: chown de/d16/la9 61 1 2024-03-26T18:54:41.983 INFO:tasks.workunit.client.1.smithi190.stdout:6/803: read d17/f19 [1409805,60243] 0 2024-03-26T18:54:41.983 INFO:tasks.workunit.client.1.smithi190.stdout:6/804: stat d17/la1 0 2024-03-26T18:54:41.983 INFO:tasks.workunit.client.0.smithi102.stdout:3/797: rename df/d29/d31/d36/d45/daf/cff to df/d8b/dbb/c118 0 2024-03-26T18:54:41.984 INFO:tasks.workunit.client.0.smithi102.stdout:3/798: write df/d29/d31/d36/d3c/f10f [4623222,52473] 0 2024-03-26T18:54:41.986 INFO:tasks.workunit.client.0.smithi102.stdout:8/672: creat d0/d3c/d96/d98/dcf/fde x:0 0 0 2024-03-26T18:54:41.990 INFO:tasks.workunit.client.1.smithi190.stdout:5/860: link d2/d17/d1b/db6/d68/ff9 d2/d17/d1b/db6/d24/dd1/f11f 0 2024-03-26T18:54:41.991 INFO:tasks.workunit.client.1.smithi190.stdout:5/861: readlink d2/l1a 0 2024-03-26T18:54:41.995 INFO:tasks.workunit.client.1.smithi190.stdout:6/805: mknod d17/d28/d2c/db8/dcc/c112 0 2024-03-26T18:54:41.995 INFO:tasks.workunit.client.0.smithi102.stdout:6/608: sync 2024-03-26T18:54:41.995 INFO:tasks.workunit.client.0.smithi102.stdout:4/510: mknod d1/d25/d90/c9d 0 2024-03-26T18:54:41.997 INFO:tasks.workunit.client.1.smithi190.stdout:0/995: dwrite d0/dd/d20/d1c/d8b/d8d/d8f/f12a [0,4194304] 0 2024-03-26T18:54:41.999 INFO:tasks.workunit.client.0.smithi102.stdout:9/584: dwrite df/d1a/d65/fae [0,4194304] 0 2024-03-26T18:54:42.002 INFO:tasks.workunit.client.0.smithi102.stdout:2/636: dwrite d3/d11/d15/d31/d39/f64 [0,4194304] 0 2024-03-26T18:54:42.002 INFO:tasks.workunit.client.0.smithi102.stdout:2/637: readlink d3/d11/l1e 0 2024-03-26T18:54:42.002 INFO:tasks.workunit.client.0.smithi102.stdout:2/638: creat d3/d11/d15/d46/fe4 x:0 0 0 2024-03-26T18:54:42.003 INFO:tasks.workunit.client.0.smithi102.stdout:3/799: creat df/d29/d31/d77/d8a/dae/dee/f119 x:0 0 0 2024-03-26T18:54:42.006 INFO:tasks.workunit.client.0.smithi102.stdout:8/673: symlink d0/d15/d5d/dd3/d2c/d4e/ldf 0 2024-03-26T18:54:42.006 INFO:tasks.workunit.client.0.smithi102.stdout:8/674: chown d0/d15/d5d/f61 14 1 2024-03-26T18:54:42.010 INFO:tasks.workunit.client.0.smithi102.stdout:6/609: mkdir d2/d7/d3c/d78/dc4 0 2024-03-26T18:54:42.010 INFO:tasks.workunit.client.1.smithi190.stdout:0/996: mknod d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/d66/c162 0 2024-03-26T18:54:42.012 INFO:tasks.workunit.client.0.smithi102.stdout:5/865: rename d2/d14/d1a/f77 to d2/d25/d75/def/dec/d112/f12e 0 2024-03-26T18:54:42.016 INFO:tasks.workunit.client.1.smithi190.stdout:6/806: dwrite d17/d28/d2c/d50/f7e [0,4194304] 0 2024-03-26T18:54:42.019 INFO:tasks.workunit.client.0.smithi102.stdout:9/585: mkdir df/d1a/d23/d60/dd4 0 2024-03-26T18:54:42.019 INFO:tasks.workunit.client.0.smithi102.stdout:9/586: truncate df/d1a/d23/d4e/d8e/f56 4197851 0 2024-03-26T18:54:42.021 INFO:tasks.workunit.client.0.smithi102.stdout:8/675: write d0/d15/d5d/dd3/d2c/d4e/f52 [9178605,78934] 0 2024-03-26T18:54:42.025 INFO:tasks.workunit.client.1.smithi190.stdout:6/807: mkdir d17/dd3/d113 0 2024-03-26T18:54:42.027 INFO:tasks.workunit.client.1.smithi190.stdout:5/862: rmdir d2/d17/d98/d3a 39 2024-03-26T18:54:42.027 INFO:tasks.workunit.client.1.smithi190.stdout:5/863: chown d2/d17/l5a 303005186 1 2024-03-26T18:54:42.028 INFO:tasks.workunit.client.1.smithi190.stdout:5/864: chown d2/d17/d1b/db6/l11d 43300 1 2024-03-26T18:54:42.028 INFO:tasks.workunit.client.1.smithi190.stdout:5/865: write d2/d17/d1b/db6/fc1 [3193223,5994] 0 2024-03-26T18:54:42.029 INFO:tasks.workunit.client.0.smithi102.stdout:4/511: dwrite d1/d17/d1a/d21/f63 [0,4194304] 0 2024-03-26T18:54:42.029 INFO:tasks.workunit.client.1.smithi190.stdout:0/997: dwrite d0/dd/d20/d1c/d4e/f100 [0,4194304] 0 2024-03-26T18:54:42.030 INFO:tasks.workunit.client.0.smithi102.stdout:3/800: mkdir df/d29/d31/d77/d8a/dae/dee/df2/d100/d11a 0 2024-03-26T18:54:42.033 INFO:tasks.workunit.client.0.smithi102.stdout:0/668: sync 2024-03-26T18:54:42.036 INFO:tasks.workunit.client.0.smithi102.stdout:9/587: mkdir df/d1a/d1d/d72/d7d/db7/dd5 0 2024-03-26T18:54:42.038 INFO:tasks.workunit.client.0.smithi102.stdout:8/676: unlink d0/f97 0 2024-03-26T18:54:42.038 INFO:tasks.workunit.client.0.smithi102.stdout:8/677: dread - d0/d15/d5d/dd3/d2c/da7/fc5 zero size 2024-03-26T18:54:42.038 INFO:tasks.workunit.client.0.smithi102.stdout:8/678: write d0/d3c/d6f/f8b [1672669,35363] 0 2024-03-26T18:54:42.038 INFO:tasks.workunit.client.0.smithi102.stdout:8/679: fsync d0/d15/d5d/dd3/d26/fad 0 2024-03-26T18:54:42.040 INFO:tasks.workunit.client.1.smithi190.stdout:5/866: dread d2/d17/d1b/db6/d24/d3b/da7/fe1 [0,4194304] 0 2024-03-26T18:54:42.040 INFO:tasks.workunit.client.1.smithi190.stdout:5/867: write d2/d17/d1b/d60/f7c [1209071,124503] 0 2024-03-26T18:54:42.043 INFO:tasks.workunit.client.0.smithi102.stdout:4/512: truncate d1/d17/d1f/d5d/d8b/f4e 3129457 0 2024-03-26T18:54:42.046 INFO:tasks.workunit.client.0.smithi102.stdout:3/801: link df/d29/d31/d77/d8a/dae/f104 df/d29/d31/d77/d8a/dae/f11b 0 2024-03-26T18:54:42.047 INFO:tasks.workunit.client.0.smithi102.stdout:3/802: write df/d29/d31/d77/d8a/dae/f11b [422470,97007] 0 2024-03-26T18:54:42.047 INFO:tasks.workunit.client.0.smithi102.stdout:3/803: dread - df/d1c/f81 zero size 2024-03-26T18:54:42.047 INFO:tasks.workunit.client.0.smithi102.stdout:0/669: symlink d6/d2d/d3d/le6 0 2024-03-26T18:54:42.049 INFO:tasks.workunit.client.1.smithi190.stdout:5/868: dread d2/d17/d98/d3a/f52 [0,4194304] 0 2024-03-26T18:54:42.049 INFO:tasks.workunit.client.0.smithi102.stdout:9/588: truncate df/d1a/f86 1989066 0 2024-03-26T18:54:42.049 INFO:tasks.workunit.client.0.smithi102.stdout:9/589: write df/d46/f6a [1659218,39173] 0 2024-03-26T18:54:42.056 INFO:tasks.workunit.client.0.smithi102.stdout:4/513: link d1/d17/d1f/c3b d1/d17/d2a/c9e 0 2024-03-26T18:54:42.056 INFO:tasks.workunit.client.0.smithi102.stdout:8/680: getdents d0/d15/d5d/dd3/d2c/d2f 0 2024-03-26T18:54:42.057 INFO:tasks.workunit.client.0.smithi102.stdout:8/681: read d0/f4d [14954,95862] 0 2024-03-26T18:54:42.060 INFO:tasks.workunit.client.0.smithi102.stdout:6/610: dwrite d2/d7/f67 [0,4194304] 0 2024-03-26T18:54:42.077 INFO:tasks.workunit.client.0.smithi102.stdout:3/804: dwrite df/d29/d31/d36/d3c/f10f [4194304,4194304] 0 2024-03-26T18:54:42.077 INFO:tasks.workunit.client.0.smithi102.stdout:3/805: truncate df/d29/d31/d36/d3c/fc2 1035327 0 2024-03-26T18:54:42.078 INFO:tasks.workunit.client.0.smithi102.stdout:3/806: dread df/d61/dd6/fe6 [0,4194304] 0 2024-03-26T18:54:42.079 INFO:tasks.workunit.client.1.smithi190.stdout:5/869: dwrite d2/d17/d1b/db6/d88/fd9 [0,4194304] 0 2024-03-26T18:54:42.079 INFO:tasks.workunit.client.1.smithi190.stdout:5/870: stat d2/d17/d1b/db6/l25 0 2024-03-26T18:54:42.079 INFO:tasks.workunit.client.0.smithi102.stdout:0/670: dwrite d6/de/d13/d1f/f3b [8388608,4194304] 0 2024-03-26T18:54:42.099 INFO:tasks.workunit.client.1.smithi190.stdout:5/871: dwrite d2/d17/d1b/db6/ff3 [0,4194304] 0 2024-03-26T18:54:42.099 INFO:tasks.workunit.client.1.smithi190.stdout:5/872: chown d2/d17/d1b/db6/d88 1 1 2024-03-26T18:54:42.100 INFO:tasks.workunit.client.0.smithi102.stdout:0/671: dwrite d6/de/d13/d9a/f5b [4194304,4194304] 0 2024-03-26T18:54:42.117 INFO:tasks.workunit.client.0.smithi102.stdout:0/672: dwrite d6/de/f28 [4194304,4194304] 0 2024-03-26T18:54:42.129 INFO:tasks.workunit.client.0.smithi102.stdout:4/514: unlink d1/c1d 0 2024-03-26T18:54:42.129 INFO:tasks.workunit.client.0.smithi102.stdout:8/682: truncate d0/d15/fb0 3320253 0 2024-03-26T18:54:42.129 INFO:tasks.workunit.client.0.smithi102.stdout:8/683: write d0/d15/d5d/dd3/d2c/d2f/f60 [265740,93881] 0 2024-03-26T18:54:42.130 INFO:tasks.workunit.client.0.smithi102.stdout:7/547: sync 2024-03-26T18:54:42.131 INFO:tasks.workunit.client.0.smithi102.stdout:6/611: mkdir d2/d3d/d53/d5c/dc5 0 2024-03-26T18:54:42.138 INFO:tasks.workunit.client.0.smithi102.stdout:8/684: write d0/d3c/fa4 [239695,10574] 0 2024-03-26T18:54:42.138 INFO:tasks.workunit.client.1.smithi190.stdout:5/873: mkdir d2/d17/d1b/d60/d63/dcd/d118/d120 0 2024-03-26T18:54:42.138 INFO:tasks.workunit.client.1.smithi190.stdout:5/874: stat d2/d17/d1b/d60/d63/dcd/fed 0 2024-03-26T18:54:42.138 INFO:tasks.workunit.client.1.smithi190.stdout:5/875: chown d2/d17/d98/d3a/f52 24 1 2024-03-26T18:54:42.139 INFO:tasks.workunit.client.0.smithi102.stdout:3/807: link df/d61/dd6/ced df/d8b/dbb/c11c 0 2024-03-26T18:54:42.141 INFO:tasks.workunit.client.0.smithi102.stdout:0/673: creat d6/de/d93/de4/fe7 x:0 0 0 2024-03-26T18:54:42.141 INFO:tasks.workunit.client.0.smithi102.stdout:0/674: truncate d6/de/d13/d25/d31/d45/d82/d84/d88/fde 1094619 0 2024-03-26T18:54:42.141 INFO:tasks.workunit.client.1.smithi190.stdout:5/876: mknod d2/d17/d1b/db6/d24/d3b/da7/c121 0 2024-03-26T18:54:42.142 INFO:tasks.workunit.client.0.smithi102.stdout:4/515: mknod d1/d27/d7a/c9f 0 2024-03-26T18:54:42.145 INFO:tasks.workunit.client.0.smithi102.stdout:7/548: mknod d7/d10/d27/d34/ca2 0 2024-03-26T18:54:42.145 INFO:tasks.workunit.client.0.smithi102.stdout:7/549: readlink d7/d31/l8a 0 2024-03-26T18:54:42.147 INFO:tasks.workunit.client.0.smithi102.stdout:6/612: creat d2/df/d71/d80/d91/dc1/fc6 x:0 0 0 2024-03-26T18:54:42.147 INFO:tasks.workunit.client.0.smithi102.stdout:6/613: creat d2/d3d/d53/d63/fc7 x:0 0 0 2024-03-26T18:54:42.148 INFO:tasks.workunit.client.0.smithi102.stdout:3/808: dread df/d1c/d70/fb6 [0,4194304] 0 2024-03-26T18:54:42.148 INFO:tasks.workunit.client.0.smithi102.stdout:3/809: stat df/d29/d31/d77/d8a/dae/dee/df2/cf5 0 2024-03-26T18:54:42.152 INFO:tasks.workunit.client.0.smithi102.stdout:8/685: unlink d0/d15/d5d/d89/ccc 0 2024-03-26T18:54:42.152 INFO:tasks.workunit.client.0.smithi102.stdout:8/686: fdatasync d0/d9/f7d 0 2024-03-26T18:54:42.154 INFO:tasks.workunit.client.0.smithi102.stdout:0/675: unlink d6/de/d13/d25/d4f/l6f 0 2024-03-26T18:54:42.156 INFO:tasks.workunit.client.0.smithi102.stdout:7/550: mkdir d7/d10/d27/da3 0 2024-03-26T18:54:42.157 INFO:tasks.workunit.client.0.smithi102.stdout:6/614: write d2/d3d/d53/d63/f7c [4081791,103368] 0 2024-03-26T18:54:42.157 INFO:tasks.workunit.client.0.smithi102.stdout:7/551: write d7/d10/d27/f55 [784589,23200] 0 2024-03-26T18:54:42.157 INFO:tasks.workunit.client.0.smithi102.stdout:6/615: stat d2/d7/l59 0 2024-03-26T18:54:42.158 INFO:tasks.workunit.client.0.smithi102.stdout:7/552: readlink d7/d31/l61 0 2024-03-26T18:54:42.158 INFO:tasks.workunit.client.0.smithi102.stdout:6/616: dread - d2/df/d35/d43/d75/fb5 zero size 2024-03-26T18:54:42.158 INFO:tasks.workunit.client.0.smithi102.stdout:7/553: write d7/d10/d17/f7d [577935,4045] 0 2024-03-26T18:54:42.158 INFO:tasks.workunit.client.0.smithi102.stdout:6/617: truncate d2/d3d/d53/f74 1493944 0 2024-03-26T18:54:42.158 INFO:tasks.workunit.client.0.smithi102.stdout:7/554: write d7/d10/d17/f59 [5384201,6026] 0 2024-03-26T18:54:42.160 INFO:tasks.workunit.client.0.smithi102.stdout:3/810: write df/d29/d31/d36/d45/f65 [2143005,34186] 0 2024-03-26T18:54:42.160 INFO:tasks.workunit.client.0.smithi102.stdout:8/687: mknod d0/d15/d5d/dd3/d2c/d58/ce0 0 2024-03-26T18:54:42.164 INFO:tasks.workunit.client.0.smithi102.stdout:6/618: creat d2/d3d/d53/d5c/dc5/fc8 x:0 0 0 2024-03-26T18:54:42.167 INFO:tasks.workunit.client.1.smithi190.stdout:5/877: dwrite d2/d17/d1b/db6/d96/fa4 [0,4194304] 0 2024-03-26T18:54:42.167 INFO:tasks.workunit.client.0.smithi102.stdout:7/555: unlink d7/d10/d27/d34/c4c 0 2024-03-26T18:54:42.167 INFO:tasks.workunit.client.0.smithi102.stdout:7/556: write d7/f41 [803186,34708] 0 2024-03-26T18:54:42.169 INFO:tasks.workunit.client.0.smithi102.stdout:4/516: dwrite d1/d17/d1a/d21/f56 [0,4194304] 0 2024-03-26T18:54:42.172 INFO:tasks.workunit.client.0.smithi102.stdout:3/811: rename df/d1c/d34/d53/f115 to df/d29/d31/dc0/d10c/f11d 0 2024-03-26T18:54:42.173 INFO:tasks.workunit.client.0.smithi102.stdout:8/688: creat d0/d15/d5d/dac/fe1 x:0 0 0 2024-03-26T18:54:42.174 INFO:tasks.workunit.client.0.smithi102.stdout:0/676: dwrite d6/f1a [4194304,4194304] 0 2024-03-26T18:54:42.175 INFO:tasks.workunit.client.0.smithi102.stdout:6/619: creat d2/df/d72/db7/dbd/fc9 x:0 0 0 2024-03-26T18:54:42.176 INFO:tasks.workunit.client.0.smithi102.stdout:7/557: dread d7/d12/d51/d6a/f7a [0,4194304] 0 2024-03-26T18:54:42.176 INFO:tasks.workunit.client.0.smithi102.stdout:7/558: write d7/d12/d2b/f47 [333414,52464] 0 2024-03-26T18:54:42.178 INFO:tasks.workunit.client.0.smithi102.stdout:4/517: creat d1/fa0 x:0 0 0 2024-03-26T18:54:42.178 INFO:tasks.workunit.client.0.smithi102.stdout:4/518: chown d1/d17/d1a/f4b 22947 1 2024-03-26T18:54:42.179 INFO:tasks.workunit.client.1.smithi190.stdout:5/878: rename d2/d17/d1b/db6/d96/ff8 to d2/d17/d1b/d30/f122 0 2024-03-26T18:54:42.182 INFO:tasks.workunit.client.0.smithi102.stdout:3/812: creat df/d29/d31/d36/d45/daf/f11e x:0 0 0 2024-03-26T18:54:42.182 INFO:tasks.workunit.client.0.smithi102.stdout:3/813: write df/d61/d55/fe7 [1255952,109258] 0 2024-03-26T18:54:42.183 INFO:tasks.workunit.client.1.smithi190.stdout:5/879: rmdir d2/d17/d1b/d60/d63/dcd/d118 39 2024-03-26T18:54:42.186 INFO:tasks.workunit.client.1.smithi190.stdout:5/880: mkdir d2/d17/d62/d123 0 2024-03-26T18:54:42.186 INFO:tasks.workunit.client.1.smithi190.stdout:5/881: creat d2/d17/d1b/db6/d96/f124 x:0 0 0 2024-03-26T18:54:42.186 INFO:tasks.workunit.client.1.smithi190.stdout:5/882: chown d2/f16 0 1 2024-03-26T18:54:42.187 INFO:tasks.workunit.client.0.smithi102.stdout:8/689: mknod d0/d15/d9a/ce2 0 2024-03-26T18:54:42.187 INFO:tasks.workunit.client.0.smithi102.stdout:8/690: readlink d0/d15/d5d/dd3/d26/d5c/la3 0 2024-03-26T18:54:42.187 INFO:tasks.workunit.client.0.smithi102.stdout:8/691: write d0/d15/d5d/dd3/d2c/f92 [2192195,64774] 0 2024-03-26T18:54:42.191 INFO:tasks.workunit.client.1.smithi190.stdout:5/883: unlink d2/d17/c2c 0 2024-03-26T18:54:42.193 INFO:tasks.workunit.client.1.smithi190.stdout:5/884: creat d2/d17/d1b/db6/d88/f125 x:0 0 0 2024-03-26T18:54:42.196 INFO:tasks.workunit.client.1.smithi190.stdout:5/885: truncate d2/f15 4970288 0 2024-03-26T18:54:42.199 INFO:tasks.workunit.client.1.smithi190.stdout:5/886: mkdir d2/d17/d1b/db6/d24/d3b/d126 0 2024-03-26T18:54:42.199 INFO:tasks.workunit.client.1.smithi190.stdout:5/887: write d2/d17/d1b/d30/f3f [2770150,77175] 0 2024-03-26T18:54:42.199 INFO:tasks.workunit.client.0.smithi102.stdout:3/814: dwrite df/d29/d31/fb5 [0,4194304] 0 2024-03-26T18:54:42.203 INFO:tasks.workunit.client.1.smithi190.stdout:1/856: sync 2024-03-26T18:54:42.203 INFO:tasks.workunit.client.1.smithi190.stdout:0/998: sync 2024-03-26T18:54:42.203 INFO:tasks.workunit.client.1.smithi190.stdout:6/808: sync 2024-03-26T18:54:42.203 INFO:tasks.workunit.client.1.smithi190.stdout:3/776: sync 2024-03-26T18:54:42.204 INFO:tasks.workunit.client.1.smithi190.stdout:5/888: unlink d2/d17/d1b/fbe 0 2024-03-26T18:54:42.204 INFO:tasks.workunit.client.1.smithi190.stdout:5/889: creat d2/d17/d1b/db6/d88/f127 x:0 0 0 2024-03-26T18:54:42.204 INFO:tasks.workunit.client.1.smithi190.stdout:5/890: creat d2/d17/d1b/db6/d96/f128 x:0 0 0 2024-03-26T18:54:42.207 INFO:tasks.workunit.client.1.smithi190.stdout:0/999: link d0/dd/lbb d0/dd/d20/d1c/d46/da7/df0/d4b/d5e/l163 0 2024-03-26T18:54:42.214 INFO:tasks.workunit.client.0.smithi102.stdout:7/559: mknod d7/d10/d67/d69/d78/ca4 0 2024-03-26T18:54:42.214 INFO:tasks.workunit.client.0.smithi102.stdout:4/519: truncate d1/d17/d1a/d21/d77/f81 799098 0 2024-03-26T18:54:42.214 INFO:tasks.workunit.client.0.smithi102.stdout:4/520: write d1/d17/f88 [990944,97886] 0 2024-03-26T18:54:42.215 INFO:tasks.workunit.client.1.smithi190.stdout:3/777: symlink d1/de/d81/dad/led 0 2024-03-26T18:54:42.215 INFO:tasks.workunit.client.1.smithi190.stdout:3/778: readlink d1/l3e 0 2024-03-26T18:54:42.218 INFO:tasks.workunit.client.0.smithi102.stdout:8/692: rmdir d0/d15/d5d/dd3/d26/d19/d76 39 2024-03-26T18:54:42.220 INFO:tasks.workunit.client.1.smithi190.stdout:3/779: creat d1/d3/d5a/fee x:0 0 0 2024-03-26T18:54:42.222 INFO:tasks.workunit.client.1.smithi190.stdout:1/857: dwrite d3/d9/db/fb4 [0,4194304] 0 2024-03-26T18:54:42.223 INFO:tasks.workunit.client.1.smithi190.stdout:1/858: chown d3/da/dd/d76/l112 4 1 2024-03-26T18:54:42.223 INFO:tasks.workunit.client.1.smithi190.stdout:1/859: fdatasync d3/f53 0 2024-03-26T18:54:42.225 INFO:tasks.workunit.client.0.smithi102.stdout:1/591: sync 2024-03-26T18:54:42.227 INFO:tasks.workunit.client.0.smithi102.stdout:6/620: write d2/f2e [1259349,86342] 0 2024-03-26T18:54:42.227 INFO:tasks.workunit.client.0.smithi102.stdout:6/621: truncate d2/d3d/f70 1150904 0 2024-03-26T18:54:42.228 INFO:tasks.workunit.client.1.smithi190.stdout:1/860: rmdir d3/da/d3a/d99 39 2024-03-26T18:54:42.228 INFO:tasks.workunit.client.1.smithi190.stdout:1/861: chown d3/da/d2e/d59/dae/c100 158788313 1 2024-03-26T18:54:42.229 INFO:tasks.workunit.client.0.smithi102.stdout:3/815: creat df/d29/d31/d36/d3c/f11f x:0 0 0 2024-03-26T18:54:42.229 INFO:tasks.workunit.client.0.smithi102.stdout:3/816: write df/d29/d31/d77/d8a/fd0 [123079,69477] 0 2024-03-26T18:54:42.230 INFO:tasks.workunit.client.1.smithi190.stdout:5/891: write d2/d17/d98/d3a/d3c/fa2 [850640,70399] 0 2024-03-26T18:54:42.231 INFO:tasks.workunit.client.1.smithi190.stdout:1/862: unlink d3/da/daf/ce9 0 2024-03-26T18:54:42.231 INFO:tasks.workunit.client.1.smithi190.stdout:3/780: dread d1/de/f1d [0,4194304] 0 2024-03-26T18:54:42.232 INFO:tasks.workunit.client.0.smithi102.stdout:0/677: truncate d6/f1a 7182660 0 2024-03-26T18:54:42.233 INFO:tasks.workunit.client.1.smithi190.stdout:5/892: write d2/d17/d1b/db6/d24/d3b/f46 [2057655,60740] 0 2024-03-26T18:54:42.233 INFO:tasks.workunit.client.1.smithi190.stdout:5/893: chown d2/d17/d1b/d60/d63/d85/cce 231467974 1 2024-03-26T18:54:42.235 INFO:tasks.workunit.client.0.smithi102.stdout:7/560: creat d7/d10/d67/fa5 x:0 0 0 2024-03-26T18:54:42.238 INFO:tasks.workunit.client.0.smithi102.stdout:4/521: creat d1/d17/d1a/fa1 x:0 0 0 2024-03-26T18:54:42.238 INFO:tasks.workunit.client.0.smithi102.stdout:4/522: chown d1/cd 120468 1 2024-03-26T18:54:42.239 INFO:tasks.workunit.client.1.smithi190.stdout:3/781: mknod d1/d3/d52/cef 0 2024-03-26T18:54:42.239 INFO:tasks.workunit.client.1.smithi190.stdout:3/782: chown d1/f4 17550988 1 2024-03-26T18:54:42.239 INFO:tasks.workunit.client.1.smithi190.stdout:3/783: readlink d1/de/d8a/de1/le5 0 2024-03-26T18:54:42.239 INFO:tasks.workunit.client.1.smithi190.stdout:3/784: truncate d1/d3/d5a/fa8 1010401 0 2024-03-26T18:54:42.242 INFO:tasks.workunit.client.0.smithi102.stdout:8/693: mkdir d0/d15/d5d/dd3/d2c/d4e/de3 0 2024-03-26T18:54:42.243 INFO:tasks.workunit.client.0.smithi102.stdout:0/678: write d6/d2d/d3d/f40 [1275345,58557] 0 2024-03-26T18:54:42.243 INFO:tasks.workunit.client.1.smithi190.stdout:5/894: mkdir d2/d17/d1b/d60/d63/dcd/d118/d120/d129 0 2024-03-26T18:54:42.243 INFO:tasks.workunit.client.1.smithi190.stdout:5/895: write d2/d17/d1b/d60/d63/d85/d99/d114/f115 [653263,4130] 0 2024-03-26T18:54:42.245 INFO:tasks.workunit.client.0.smithi102.stdout:1/592: dread de/f15 [0,4194304] 0 2024-03-26T18:54:42.245 INFO:tasks.workunit.client.0.smithi102.stdout:1/593: read - de/d2e/f55 zero size 2024-03-26T18:54:42.245 INFO:tasks.workunit.client.0.smithi102.stdout:1/594: fdatasync de/d16/d1d/d2a/d2c/d43/f4f 0 2024-03-26T18:54:42.246 INFO:tasks.workunit.client.1.smithi190.stdout:3/785: unlink d1/d3/df/d18/d35/d41/ld5 0 2024-03-26T18:54:42.246 INFO:tasks.workunit.client.1.smithi190.stdout:3/786: dread - d1/d3/df/d31/f56 zero size 2024-03-26T18:54:42.246 INFO:tasks.workunit.client.1.smithi190.stdout:3/787: write d1/d3/f13 [3682214,47300] 0 2024-03-26T18:54:42.246 INFO:tasks.workunit.client.1.smithi190.stdout:3/788: write d1/d3/fbf [860583,19136] 0 2024-03-26T18:54:42.247 INFO:tasks.workunit.client.0.smithi102.stdout:6/622: truncate d2/faf 87762 0 2024-03-26T18:54:42.249 INFO:tasks.workunit.client.1.smithi190.stdout:5/896: symlink d2/d17/d1b/d60/d63/d85/d99/l12a 0 2024-03-26T18:54:42.249 INFO:tasks.workunit.client.0.smithi102.stdout:5/866: sync 2024-03-26T18:54:42.249 INFO:tasks.workunit.client.0.smithi102.stdout:2/639: sync 2024-03-26T18:54:42.249 INFO:tasks.workunit.client.0.smithi102.stdout:9/590: sync 2024-03-26T18:54:42.251 INFO:tasks.workunit.client.0.smithi102.stdout:3/817: mkdir df/d29/d3a/d93/d120 0 2024-03-26T18:54:42.257 INFO:tasks.workunit.client.0.smithi102.stdout:7/561: symlink d7/d10/d27/d34/la6 0 2024-03-26T18:54:42.259 INFO:tasks.workunit.client.1.smithi190.stdout:5/897: mkdir d2/d17/d1b/d60/d63/d102/d12b 0 2024-03-26T18:54:42.259 INFO:tasks.workunit.client.1.smithi190.stdout:3/789: dread d1/f26 [0,4194304] 0 2024-03-26T18:54:42.259 INFO:tasks.workunit.client.0.smithi102.stdout:8/694: link d0/d15/d5d/dd3/d2c/d2f/f7e d0/d3c/d96/d98/fe4 0 2024-03-26T18:54:42.259 INFO:tasks.workunit.client.0.smithi102.stdout:6/623: write d2/d7/f54 [3544878,118715] 0 2024-03-26T18:54:42.263 INFO:tasks.workunit.client.0.smithi102.stdout:1/595: rename de/d16/d1d/d2a/d2c/f8a to de/d1f/fc5 0 2024-03-26T18:54:42.265 INFO:tasks.workunit.client.1.smithi190.stdout:3/790: write d1/de/fba [872981,25943] 0 2024-03-26T18:54:42.265 INFO:tasks.workunit.client.1.smithi190.stdout:3/791: write d1/de/d8a/f95 [1049411,112678] 0 2024-03-26T18:54:42.266 INFO:tasks.workunit.client.0.smithi102.stdout:5/867: truncate d2/d25/d5b/d72/fc5 132431 0 2024-03-26T18:54:42.271 INFO:tasks.workunit.client.0.smithi102.stdout:3/818: creat df/d29/d31/d36/d45/f121 x:0 0 0 2024-03-26T18:54:42.271 INFO:tasks.workunit.client.0.smithi102.stdout:3/819: fsync df/d1c/d34/f37 0 2024-03-26T18:54:42.271 INFO:tasks.workunit.client.0.smithi102.stdout:3/820: creat df/d8b/f122 x:0 0 0 2024-03-26T18:54:42.273 INFO:tasks.workunit.client.0.smithi102.stdout:7/562: truncate d7/d10/d27/f33 106734 0 2024-03-26T18:54:42.274 INFO:tasks.workunit.client.1.smithi190.stdout:3/792: mkdir d1/de/d8a/df0 0 2024-03-26T18:54:42.275 INFO:tasks.workunit.client.0.smithi102.stdout:8/695: getdents d0/d15/d5d/dd3/d26/d19/d76 0 2024-03-26T18:54:42.277 INFO:tasks.workunit.client.0.smithi102.stdout:4/523: dwrite d1/d25/f69 [0,4194304] 0 2024-03-26T18:54:42.278 INFO:tasks.workunit.client.0.smithi102.stdout:4/524: chown d1/d17/c22 95688063 1 2024-03-26T18:54:42.278 INFO:tasks.workunit.client.0.smithi102.stdout:6/624: unlink d2/d3d/l55 0 2024-03-26T18:54:42.278 INFO:tasks.workunit.client.0.smithi102.stdout:6/625: dread - d2/d3d/f8a zero size 2024-03-26T18:54:42.278 INFO:tasks.workunit.client.0.smithi102.stdout:9/591: rename df/d1a/d23/d3e/d75/lc6 to df/d1a/d23/d29/ld6 0 2024-03-26T18:54:42.279 INFO:tasks.workunit.client.0.smithi102.stdout:5/868: symlink d2/d25/d5b/d72/l12f 0 2024-03-26T18:54:42.280 INFO:tasks.workunit.client.0.smithi102.stdout:3/821: mknod df/d29/d31/d36/d3c/c123 0 2024-03-26T18:54:42.281 INFO:tasks.workunit.client.0.smithi102.stdout:6/626: creat d2/fca x:0 0 0 2024-03-26T18:54:42.282 INFO:tasks.workunit.client.0.smithi102.stdout:9/592: link df/d1a/d1d/l37 df/d1a/d65/ld7 0 2024-03-26T18:54:42.283 INFO:tasks.workunit.client.0.smithi102.stdout:5/869: mknod d2/dea/dbb/c130 0 2024-03-26T18:54:42.283 INFO:tasks.workunit.client.0.smithi102.stdout:5/870: creat d2/d14/f131 x:0 0 0 2024-03-26T18:54:42.283 INFO:tasks.workunit.client.0.smithi102.stdout:5/871: chown d2/d25/d75/db9/db3/d115/de9/cf7 336773361 1 2024-03-26T18:54:42.284 INFO:tasks.workunit.client.0.smithi102.stdout:3/822: unlink df/f7b 0 2024-03-26T18:54:42.285 INFO:tasks.workunit.client.0.smithi102.stdout:9/593: symlink df/d1a/d23/ld8 0 2024-03-26T18:54:42.288 INFO:tasks.workunit.client.0.smithi102.stdout:3/823: mknod df/d29/d31/d77/d8a/dae/dee/c124 0 2024-03-26T18:54:42.288 INFO:tasks.workunit.client.0.smithi102.stdout:3/824: creat df/d29/d31/d36/d3c/d6f/d75/f125 x:0 0 0 2024-03-26T18:54:42.289 INFO:tasks.workunit.client.0.smithi102.stdout:3/825: write df/d61/d55/f90 [421839,115161] 0 2024-03-26T18:54:42.291 INFO:tasks.workunit.client.0.smithi102.stdout:7/563: dread d7/d10/d17/f37 [0,4194304] 0 2024-03-26T18:54:42.292 INFO:tasks.workunit.client.0.smithi102.stdout:8/696: write d0/d15/d5d/dd3/d2c/d33/d87/fa8 [3792637,131034] 0 2024-03-26T18:54:42.292 INFO:tasks.workunit.client.0.smithi102.stdout:8/697: dread - d0/d15/d5d/fc7 zero size 2024-03-26T18:54:42.292 INFO:tasks.workunit.client.0.smithi102.stdout:8/698: fdatasync d0/d15/d5d/dd3/d2c/d33/d87/f8c 0 2024-03-26T18:54:42.294 INFO:tasks.workunit.client.0.smithi102.stdout:3/826: unlink df/d1c/fe9 0 2024-03-26T18:54:42.295 INFO:tasks.workunit.client.0.smithi102.stdout:7/564: creat d7/d12/d2b/d50/fa7 x:0 0 0 2024-03-26T18:54:42.296 INFO:tasks.workunit.client.0.smithi102.stdout:0/679: dwrite d6/de/d13/d1f/d55/d57/d7d/fd5 [0,4194304] 0 2024-03-26T18:54:42.296 INFO:tasks.workunit.client.0.smithi102.stdout:1/596: dwrite de/f2d [0,4194304] 0 2024-03-26T18:54:42.296 INFO:tasks.workunit.client.0.smithi102.stdout:2/640: dwrite d3/d2a/d5f/fa8 [4194304,4194304] 0 2024-03-26T18:54:42.297 INFO:tasks.workunit.client.1.smithi190.stdout:1/863: dwrite d3/d73/d88/df1/f108 [0,4194304] 0 2024-03-26T18:54:42.297 INFO:tasks.workunit.client.1.smithi190.stdout:5/898: dwrite d2/d17/d1b/f72 [0,4194304] 0 2024-03-26T18:54:42.298 INFO:tasks.workunit.client.1.smithi190.stdout:1/864: chown d3/d9/f3c 12 1 2024-03-26T18:54:42.298 INFO:tasks.workunit.client.1.smithi190.stdout:1/865: fdatasync d3/da/dd/d95/d52/fba 0 2024-03-26T18:54:42.298 INFO:tasks.workunit.client.1.smithi190.stdout:1/866: creat d3/d9/db/f119 x:0 0 0 2024-03-26T18:54:42.299 INFO:tasks.workunit.client.0.smithi102.stdout:3/827: dread df/d29/d31/d77/d8a/dae/f11b [0,4194304] 0 2024-03-26T18:54:42.299 INFO:tasks.workunit.client.1.smithi190.stdout:1/867: truncate d3/d73/d88/dcc/ff8 847340 0 2024-03-26T18:54:42.302 INFO:tasks.workunit.client.0.smithi102.stdout:4/525: write d1/d17/d1a/d21/f63 [3930618,47722] 0 2024-03-26T18:54:42.311 INFO:tasks.workunit.client.0.smithi102.stdout:6/627: dread d2/d7/f54 [0,4194304] 0 2024-03-26T18:54:42.312 INFO:tasks.workunit.client.0.smithi102.stdout:0/680: read d6/f1a [4833215,42575] 0 2024-03-26T18:54:42.315 INFO:tasks.workunit.client.0.smithi102.stdout:1/597: dread de/d16/d1d/d2a/d2c/d56/d78/f79 [0,4194304] 0 2024-03-26T18:54:42.316 INFO:tasks.workunit.client.0.smithi102.stdout:1/598: truncate de/d16/f54 561813 0 2024-03-26T18:54:42.320 INFO:tasks.workunit.client.0.smithi102.stdout:9/594: dwrite df/d46/da8/fbb [0,4194304] 0 2024-03-26T18:54:42.320 INFO:tasks.workunit.client.0.smithi102.stdout:9/595: write df/d1a/d23/d29/f31 [5181120,76954] 0 2024-03-26T18:54:42.321 INFO:tasks.workunit.client.0.smithi102.stdout:2/641: unlink d3/d11/l1e 0 2024-03-26T18:54:42.321 INFO:tasks.workunit.client.0.smithi102.stdout:2/642: write d3/d11/d15/d46/fe4 [711634,90772] 0 2024-03-26T18:54:42.321 INFO:tasks.workunit.client.0.smithi102.stdout:2/643: dread - d3/d2a/faf zero size 2024-03-26T18:54:42.322 INFO:tasks.workunit.client.0.smithi102.stdout:3/828: truncate df/d29/d31/d36/ffb 3862831 0 2024-03-26T18:54:42.322 INFO:tasks.workunit.client.0.smithi102.stdout:3/829: creat df/d29/d31/f126 x:0 0 0 2024-03-26T18:54:42.326 INFO:tasks.workunit.client.0.smithi102.stdout:8/699: dwrite d0/d15/d5d/dd3/d2c/f7f [0,4194304] 0 2024-03-26T18:54:42.328 INFO:tasks.workunit.client.0.smithi102.stdout:5/872: dwrite d2/d14/f1c [0,4194304] 0 2024-03-26T18:54:42.329 INFO:tasks.workunit.client.1.smithi190.stdout:1/868: link d3/d9/f3c d3/d8f/da7/f11a 0 2024-03-26T18:54:42.330 INFO:tasks.workunit.client.0.smithi102.stdout:0/681: creat d6/de/d13/d25/fe8 x:0 0 0 2024-03-26T18:54:42.332 INFO:tasks.workunit.client.0.smithi102.stdout:1/599: mknod de/d16/d1d/d2a/d2c/d56/d5f/cc6 0 2024-03-26T18:54:42.332 INFO:tasks.workunit.client.0.smithi102.stdout:1/600: fdatasync de/d1f/f6d 0 2024-03-26T18:54:42.333 INFO:tasks.workunit.client.0.smithi102.stdout:6/628: dwrite d2/f33 [0,4194304] 0 2024-03-26T18:54:42.340 INFO:tasks.workunit.client.1.smithi190.stdout:5/899: getdents d2/d17/d1b/d60/db8 0 2024-03-26T18:54:42.345 INFO:tasks.workunit.client.1.smithi190.stdout:5/900: mkdir d2/d17/d1b/db6/d24/dd1/d12c 0 2024-03-26T18:54:42.346 INFO:tasks.workunit.client.0.smithi102.stdout:9/596: creat df/d1a/d23/d60/dd4/fd9 x:0 0 0 2024-03-26T18:54:42.349 INFO:tasks.workunit.client.0.smithi102.stdout:7/565: getdents d7/d12/d2b/d50 0 2024-03-26T18:54:42.352 INFO:tasks.workunit.client.1.smithi190.stdout:5/901: rename d2/d17/l29 to d2/d17/d1b/d60/l12d 0 2024-03-26T18:54:42.354 INFO:tasks.workunit.client.0.smithi102.stdout:3/830: rename df/d29/d31/d36/d3c/d6f/f7d to df/d1c/d34/f127 0 2024-03-26T18:54:42.356 INFO:tasks.workunit.client.0.smithi102.stdout:8/700: dwrite d0/d3c/d96/d98/db6/fd4 [0,4194304] 0 2024-03-26T18:54:42.356 INFO:tasks.workunit.client.1.smithi190.stdout:5/902: mknod d2/d17/d1b/db6/d88/dc9/c12e 0 2024-03-26T18:54:42.359 INFO:tasks.workunit.client.1.smithi190.stdout:1/869: dwrite d3/da/d2e/f30 [0,4194304] 0 2024-03-26T18:54:42.359 INFO:tasks.workunit.client.1.smithi190.stdout:5/903: rename d2/d17/d1b/d60/d63/dcd/d118/d120/d129 to d2/d17/d1b/d60/d63/d102/d12f 0 2024-03-26T18:54:42.359 INFO:tasks.workunit.client.0.smithi102.stdout:1/601: dwrite de/d16/d1d/d2a/f5d [0,4194304] 0 2024-03-26T18:54:42.359 INFO:tasks.workunit.client.0.smithi102.stdout:1/602: fdatasync de/d16/d1d/d2a/fb2 0 2024-03-26T18:54:42.363 INFO:tasks.workunit.client.0.smithi102.stdout:5/873: symlink d2/dea/l132 0 2024-03-26T18:54:42.364 INFO:tasks.workunit.client.1.smithi190.stdout:1/870: mknod d3/da/dd/d76/dc4/c11b 0 2024-03-26T18:54:42.366 INFO:tasks.workunit.client.0.smithi102.stdout:0/682: symlink d6/de/d13/d25/d4f/le9 0 2024-03-26T18:54:42.366 INFO:tasks.workunit.client.0.smithi102.stdout:0/683: readlink d6/de/d13/d9a/l86 0 2024-03-26T18:54:42.366 INFO:tasks.workunit.client.0.smithi102.stdout:0/684: chown d6/d3e/d69/d8c/d8f/ca7 87977 1 2024-03-26T18:54:42.366 INFO:tasks.workunit.client.0.smithi102.stdout:0/685: read d6/d2d/d3d/f40 [1487846,4877] 0 2024-03-26T18:54:42.369 INFO:tasks.workunit.client.1.smithi190.stdout:1/871: unlink d3/d9/f20 0 2024-03-26T18:54:42.371 INFO:tasks.workunit.client.1.smithi190.stdout:5/904: mknod d2/d17/d98/c130 0 2024-03-26T18:54:42.372 INFO:tasks.workunit.client.0.smithi102.stdout:1/603: write de/d16/d1d/d61/d6a/da5/da6/fb7 [2436003,60672] 0 2024-03-26T18:54:42.372 INFO:tasks.workunit.client.0.smithi102.stdout:6/629: mknod d2/df/d71/d80/d91/ccb 0 2024-03-26T18:54:42.373 INFO:tasks.workunit.client.1.smithi190.stdout:1/872: creat d3/da/dd/d2f/d34/f11c x:0 0 0 2024-03-26T18:54:42.373 INFO:tasks.workunit.client.1.smithi190.stdout:1/873: dread - d3/da/dd/d2f/d5b/f62 zero size 2024-03-26T18:54:42.373 INFO:tasks.workunit.client.1.smithi190.stdout:1/874: write d3/da/dd/d95/d52/f93 [987641,34998] 0 2024-03-26T18:54:42.373 INFO:tasks.workunit.client.1.smithi190.stdout:1/875: dread - d3/da/d3a/da5/f114 zero size 2024-03-26T18:54:42.374 INFO:tasks.workunit.client.0.smithi102.stdout:9/597: mkdir df/d1a/d1d/d72/d7d/dda 0 2024-03-26T18:54:42.375 INFO:tasks.workunit.client.0.smithi102.stdout:2/644: rmdir d3/d2a/d5f 39 2024-03-26T18:54:42.376 INFO:tasks.workunit.client.1.smithi190.stdout:1/876: symlink d3/da/dd/d76/dc4/l11d 0 2024-03-26T18:54:42.378 INFO:tasks.workunit.client.0.smithi102.stdout:7/566: truncate d7/d10/d27/f55 264835 0 2024-03-26T18:54:42.378 INFO:tasks.workunit.client.0.smithi102.stdout:7/567: chown l5 5725558 1 2024-03-26T18:54:42.381 INFO:tasks.workunit.client.0.smithi102.stdout:3/831: mkdir df/d61/de5/d128 0 2024-03-26T18:54:42.381 INFO:tasks.workunit.client.0.smithi102.stdout:3/832: creat df/d1c/d70/f129 x:0 0 0 2024-03-26T18:54:42.384 INFO:tasks.workunit.client.0.smithi102.stdout:8/701: symlink d0/d15/d5d/dd3/d26/d80/le5 0 2024-03-26T18:54:42.389 INFO:tasks.workunit.client.0.smithi102.stdout:5/874: getdents d2/d25/de6 0 2024-03-26T18:54:42.391 INFO:tasks.workunit.client.0.smithi102.stdout:0/686: rename d6/d2d/d3d/d80/fc6 to d6/d3e/d87/fea 0 2024-03-26T18:54:42.392 INFO:tasks.workunit.client.0.smithi102.stdout:1/604: symlink de/d16/d1d/d2a/d2c/d43/lc7 0 2024-03-26T18:54:42.394 INFO:tasks.workunit.client.0.smithi102.stdout:9/598: symlink df/d84/ldb 0 2024-03-26T18:54:42.394 INFO:tasks.workunit.client.0.smithi102.stdout:9/599: write df/d1a/d1d/d72/f90 [4251800,35871] 0 2024-03-26T18:54:42.394 INFO:tasks.workunit.client.0.smithi102.stdout:9/600: truncate df/d1a/d23/d4e/d8e/f5f 745808 0 2024-03-26T18:54:42.394 INFO:tasks.workunit.client.0.smithi102.stdout:9/601: chown df/d1a/d23/d3e 1858263 1 2024-03-26T18:54:42.394 INFO:tasks.workunit.client.0.smithi102.stdout:2/645: truncate d3/d2a/f34 2458480 0 2024-03-26T18:54:42.395 INFO:tasks.workunit.client.1.smithi190.stdout:1/877: dread d3/da/dd/d2f/f7b [0,4194304] 0 2024-03-26T18:54:42.395 INFO:tasks.workunit.client.1.smithi190.stdout:1/878: truncate d3/da/dd/d2f/fed 772618 0 2024-03-26T18:54:42.396 INFO:tasks.workunit.client.0.smithi102.stdout:7/568: mkdir d7/d12/d2b/da8 0 2024-03-26T18:54:42.396 INFO:tasks.workunit.client.0.smithi102.stdout:7/569: chown d7/d12/d4d/d62 1 1 2024-03-26T18:54:42.396 INFO:tasks.workunit.client.0.smithi102.stdout:7/570: creat d7/d31/d56/fa9 x:0 0 0 2024-03-26T18:54:42.396 INFO:tasks.workunit.client.0.smithi102.stdout:7/571: write d7/d2a/f4a [4877001,43762] 0 2024-03-26T18:54:42.396 INFO:tasks.workunit.client.1.smithi190.stdout:3/793: sync 2024-03-26T18:54:42.396 INFO:tasks.workunit.client.1.smithi190.stdout:6/809: sync 2024-03-26T18:54:42.397 INFO:tasks.workunit.client.0.smithi102.stdout:8/702: getdents d0/d9 0 2024-03-26T18:54:42.397 INFO:tasks.workunit.client.0.smithi102.stdout:6/630: dwrite d2/fca [0,4194304] 0 2024-03-26T18:54:42.398 INFO:tasks.workunit.client.0.smithi102.stdout:6/631: chown d2/d3d/f7d 0 1 2024-03-26T18:54:42.398 INFO:tasks.workunit.client.0.smithi102.stdout:6/632: truncate d2/d3/f97 846635 0 2024-03-26T18:54:42.398 INFO:tasks.workunit.client.1.smithi190.stdout:5/905: dwrite d2/d17/d98/f10b [0,4194304] 0 2024-03-26T18:54:42.400 INFO:tasks.workunit.client.1.smithi190.stdout:3/794: truncate d1/d3/df/f19 2012056 0 2024-03-26T18:54:42.402 INFO:tasks.workunit.client.0.smithi102.stdout:5/875: symlink d2/d25/l133 0 2024-03-26T18:54:42.405 INFO:tasks.workunit.client.0.smithi102.stdout:0/687: write d6/de/d13/d1f/d55/d57/d7d/fd5 [2587393,86529] 0 2024-03-26T18:54:42.407 INFO:tasks.workunit.client.1.smithi190.stdout:6/810: rename d17/d28/df6/da8/fd9 to d17/d28/df6/da8/df4/dfb/f114 0 2024-03-26T18:54:42.407 INFO:tasks.workunit.client.1.smithi190.stdout:6/811: chown d17/c4a 33535 1 2024-03-26T18:54:42.408 INFO:tasks.workunit.client.0.smithi102.stdout:3/833: rename df/d8b/fd1 to df/d29/d31/d36/d45/daf/f12a 0 2024-03-26T18:54:42.409 INFO:tasks.workunit.client.0.smithi102.stdout:3/834: truncate df/d29/d31/d36/d45/f121 27555 0 2024-03-26T18:54:42.409 INFO:tasks.workunit.client.0.smithi102.stdout:3/835: chown df/d29/d31/c3d 2 1 2024-03-26T18:54:42.410 INFO:tasks.workunit.client.0.smithi102.stdout:1/605: mknod de/d2e/d62/cc8 0 2024-03-26T18:54:42.410 INFO:tasks.workunit.client.0.smithi102.stdout:1/606: fdatasync de/d16/d1d/f4d 0 2024-03-26T18:54:42.411 INFO:tasks.workunit.client.0.smithi102.stdout:1/607: creat de/d16/d1d/d2a/d67/fc9 x:0 0 0 2024-03-26T18:54:42.414 INFO:tasks.workunit.client.0.smithi102.stdout:9/602: creat df/d1a/d23/d29/fdc x:0 0 0 2024-03-26T18:54:42.414 INFO:tasks.workunit.client.0.smithi102.stdout:9/603: stat df/d1a/d7f/fba 0 2024-03-26T18:54:42.415 INFO:tasks.workunit.client.1.smithi190.stdout:1/879: dwrite d3/d9/db/f32 [4194304,4194304] 0 2024-03-26T18:54:42.415 INFO:tasks.workunit.client.0.smithi102.stdout:7/572: dwrite d7/d12/d51/f58 [0,4194304] 0 2024-03-26T18:54:42.419 INFO:tasks.workunit.client.0.smithi102.stdout:2/646: link d3/d11/d15/f2e d3/fe5 0 2024-03-26T18:54:42.425 INFO:tasks.workunit.client.0.smithi102.stdout:7/573: write d7/d12/f3c [3192052,118897] 0 2024-03-26T18:54:42.427 INFO:tasks.workunit.client.1.smithi190.stdout:6/812: truncate d17/d28/d2c/d63/fa0 1074289 0 2024-03-26T18:54:42.427 INFO:tasks.workunit.client.0.smithi102.stdout:8/703: mkdir d0/d15/de6 0 2024-03-26T18:54:42.440 INFO:tasks.workunit.client.0.smithi102.stdout:2/647: dwrite d3/d2d/f89 [0,4194304] 0 2024-03-26T18:54:42.441 INFO:tasks.workunit.client.0.smithi102.stdout:5/876: creat d2/d25/d75/db9/d43/d9e/d82/dc9/f134 x:0 0 0 2024-03-26T18:54:42.443 INFO:tasks.workunit.client.0.smithi102.stdout:0/688: mkdir d6/dbe/deb 0 2024-03-26T18:54:42.446 INFO:tasks.workunit.client.1.smithi190.stdout:1/880: dread d3/da/dd/d2f/d5b/d83/feb [0,4194304] 0 2024-03-26T18:54:42.446 INFO:tasks.workunit.client.1.smithi190.stdout:1/881: stat d3/d9/db/l26 0 2024-03-26T18:54:42.446 INFO:tasks.workunit.client.1.smithi190.stdout:1/882: write d3/f7f [1985910,31773] 0 2024-03-26T18:54:42.447 INFO:tasks.workunit.client.0.smithi102.stdout:6/633: rename d2/d3/fb6 to d2/df/d71/d80/d91/fcc 0 2024-03-26T18:54:42.447 INFO:tasks.workunit.client.0.smithi102.stdout:6/634: dread - d2/d3d/f7d zero size 2024-03-26T18:54:42.448 INFO:tasks.workunit.client.0.smithi102.stdout:3/836: creat df/d29/d31/d77/d8a/dae/dee/df2/d100/f12b x:0 0 0 2024-03-26T18:54:42.449 INFO:tasks.workunit.client.1.smithi190.stdout:1/883: mknod d3/d9/da3/c11e 0 2024-03-26T18:54:42.449 INFO:tasks.workunit.client.1.smithi190.stdout:1/884: write d3/da/dd/d2f/d33/fac [5258363,96974] 0 2024-03-26T18:54:42.449 INFO:tasks.workunit.client.1.smithi190.stdout:1/885: write d3/da/dd/d95/d52/ffb [222260,116718] 0 2024-03-26T18:54:42.451 INFO:tasks.workunit.client.0.smithi102.stdout:1/608: mkdir de/d16/d1d/d61/d6a/da5/da6/dca 0 2024-03-26T18:54:42.452 INFO:tasks.workunit.client.0.smithi102.stdout:7/574: dwrite d7/d10/d27/d34/d72/f76 [0,4194304] 0 2024-03-26T18:54:42.452 INFO:tasks.workunit.client.0.smithi102.stdout:9/604: symlink df/d1a/d1d/d72/d7d/db7/ldd 0 2024-03-26T18:54:42.455 INFO:tasks.workunit.client.1.smithi190.stdout:6/813: dwrite d17/d28/d2b/f6e [0,4194304] 0 2024-03-26T18:54:42.459 INFO:tasks.workunit.client.1.smithi190.stdout:6/814: link d17/d28/d2b/c72 d17/d28/d2c/d50/df7/c115 0 2024-03-26T18:54:42.459 INFO:tasks.workunit.client.1.smithi190.stdout:6/815: chown d17/d28/df6/da8/df4/dfb/d6a/ld5 41392724 1 2024-03-26T18:54:42.460 INFO:tasks.workunit.client.0.smithi102.stdout:8/704: creat d0/d15/d5d/dd3/d2c/d4e/fe7 x:0 0 0 2024-03-26T18:54:42.461 INFO:tasks.workunit.client.0.smithi102.stdout:1/609: dread de/d16/f40 [0,4194304] 0 2024-03-26T18:54:42.464 INFO:tasks.workunit.client.1.smithi190.stdout:6/816: creat d17/d28/d8b/f116 x:0 0 0 2024-03-26T18:54:42.470 INFO:tasks.workunit.client.0.smithi102.stdout:2/648: link d3/d82/d93/fa4 d3/d11/d15/d46/d7b/fe6 0 2024-03-26T18:54:42.470 INFO:tasks.workunit.client.1.smithi190.stdout:1/886: dwrite d3/da/dd/d2f/d33/d9c/fb7 [0,4194304] 0 2024-03-26T18:54:42.471 INFO:tasks.workunit.client.1.smithi190.stdout:6/817: creat d17/d28/df6/da8/df4/f117 x:0 0 0 2024-03-26T18:54:42.472 INFO:tasks.workunit.client.0.smithi102.stdout:0/689: creat d6/d3e/d69/d8c/fec x:0 0 0 2024-03-26T18:54:42.475 INFO:tasks.workunit.client.0.smithi102.stdout:5/877: rename d2/d25/d75/db9/db3/d115/de9 to d2/d25/d75/db9/ded/d135 0 2024-03-26T18:54:42.476 INFO:tasks.workunit.client.0.smithi102.stdout:5/878: dread - d2/dea/d4f/f114 zero size 2024-03-26T18:54:42.478 INFO:tasks.workunit.client.1.smithi190.stdout:1/887: stat d3/da/d2e/f30 0 2024-03-26T18:54:42.481 INFO:tasks.workunit.client.1.smithi190.stdout:6/818: rename d17/d28/d2c/d63/d10e/f111 to d17/d28/d2c/db8/df3/f118 0 2024-03-26T18:54:42.481 INFO:tasks.workunit.client.1.smithi190.stdout:6/819: truncate d17/d28/d2b/f10c 816658 0 2024-03-26T18:54:42.482 INFO:tasks.workunit.client.1.smithi190.stdout:1/888: chown d3/da/daf/l111 0 1 2024-03-26T18:54:42.483 INFO:tasks.workunit.client.1.smithi190.stdout:1/889: fsync d3/da/dd/d2f/d34/f35 0 2024-03-26T18:54:42.483 INFO:tasks.workunit.client.0.smithi102.stdout:7/575: dwrite d7/d2a/f30 [0,4194304] 0 2024-03-26T18:54:42.485 INFO:tasks.workunit.client.1.smithi190.stdout:6/820: write d17/d28/d2c/db8/df3/d109/d79/f8e [1641647,96482] 0 2024-03-26T18:54:42.486 INFO:tasks.workunit.client.0.smithi102.stdout:0/690: dread d6/de/d93/dac/fc2 [0,4194304] 0 2024-03-26T18:54:42.488 INFO:tasks.workunit.client.0.smithi102.stdout:5/879: dread d2/d25/d75/db9/db3/fd9 [0,4194304] 0 2024-03-26T18:54:42.488 INFO:tasks.workunit.client.0.smithi102.stdout:5/880: dread - d2/d25/d75/db9/d43/d9e/f8c zero size 2024-03-26T18:54:42.488 INFO:tasks.workunit.client.1.smithi190.stdout:6/821: rename d17/d28/d2b/c6c to d17/dd3/c119 0 2024-03-26T18:54:42.488 INFO:tasks.workunit.client.1.smithi190.stdout:6/822: chown d17/d28/d2c/d50/fc2 141017588 1 2024-03-26T18:54:42.488 INFO:tasks.workunit.client.1.smithi190.stdout:6/823: truncate d17/d28/d2c/db8/df3/d109/fb1 1762707 0 2024-03-26T18:54:42.490 INFO:tasks.workunit.client.1.smithi190.stdout:1/890: write d3/d9/f5c [3372848,92070] 0 2024-03-26T18:54:42.493 INFO:tasks.workunit.client.0.smithi102.stdout:8/705: mknod d0/d15/d5d/dd3/d2c/d2f/ce8 0 2024-03-26T18:54:42.495 INFO:tasks.workunit.client.0.smithi102.stdout:2/649: symlink d3/d2a/d5f/le7 0 2024-03-26T18:54:42.495 INFO:tasks.workunit.client.0.smithi102.stdout:2/650: creat d3/d20/fe8 x:0 0 0 2024-03-26T18:54:42.498 INFO:tasks.workunit.client.1.smithi190.stdout:1/891: dread d3/da/dd/d2f/d33/f8a [0,4194304] 0 2024-03-26T18:54:42.498 INFO:tasks.workunit.client.1.smithi190.stdout:1/892: creat d3/da/dd/d95/f11f x:0 0 0 2024-03-26T18:54:42.498 INFO:tasks.workunit.client.1.smithi190.stdout:1/893: truncate d3/da/dd/d2f/d5b/f90 216358 0 2024-03-26T18:54:42.498 INFO:tasks.workunit.client.1.smithi190.stdout:1/894: dread - d3/da/dd/d76/dc4/fdd zero size 2024-03-26T18:54:42.498 INFO:tasks.workunit.client.0.smithi102.stdout:3/837: rename df/d1c/d34/d53/f80 to df/d29/d31/d36/d45/daf/d105/f12c 0 2024-03-26T18:54:42.498 INFO:tasks.workunit.client.0.smithi102.stdout:1/610: rename de/d16/d1d/d61 to de/d16/d1d/d61/d6a/dcb 22 2024-03-26T18:54:42.499 INFO:tasks.workunit.client.0.smithi102.stdout:1/611: write de/f99 [1025096,127100] 0 2024-03-26T18:54:42.509 INFO:tasks.workunit.client.0.smithi102.stdout:2/651: dwrite d3/d2a/f34 [0,4194304] 0 2024-03-26T18:54:42.509 INFO:tasks.workunit.client.0.smithi102.stdout:2/652: truncate d3/d20/fa1 4455296 0 2024-03-26T18:54:42.511 INFO:tasks.workunit.client.1.smithi190.stdout:1/895: rmdir d3/da/dd/d2f/d33/de8 0 2024-03-26T18:54:42.511 INFO:tasks.workunit.client.1.smithi190.stdout:1/896: stat d3/da/d2e/d59/lec 0 2024-03-26T18:54:42.514 INFO:tasks.workunit.client.1.smithi190.stdout:1/897: dread d3/d9/f15 [0,4194304] 0 2024-03-26T18:54:42.515 INFO:tasks.workunit.client.1.smithi190.stdout:5/906: sync 2024-03-26T18:54:42.516 INFO:tasks.workunit.client.1.smithi190.stdout:5/907: mkdir d2/d17/d1b/db6/d96/d131 0 2024-03-26T18:54:42.517 INFO:tasks.workunit.client.1.smithi190.stdout:1/898: getdents d3/d9 0 2024-03-26T18:54:42.519 INFO:tasks.workunit.client.1.smithi190.stdout:5/908: rmdir d2/d17/d1b/d8e 39 2024-03-26T18:54:42.520 INFO:tasks.workunit.client.1.smithi190.stdout:1/899: fdatasync d3/da/d3a/f10e 0 2024-03-26T18:54:42.520 INFO:tasks.workunit.client.1.smithi190.stdout:1/900: readlink d3/da/dd/d2f/d34/l64 0 2024-03-26T18:54:42.520 INFO:tasks.workunit.client.1.smithi190.stdout:1/901: readlink d3/da/d2e/d59/dae/le1 0 2024-03-26T18:54:42.521 INFO:tasks.workunit.client.0.smithi102.stdout:6/635: rmdir d2 39 2024-03-26T18:54:42.521 INFO:tasks.workunit.client.0.smithi102.stdout:4/526: sync 2024-03-26T18:54:42.523 INFO:tasks.workunit.client.0.smithi102.stdout:7/576: mkdir d7/d10/d27/d34/d82/daa 0 2024-03-26T18:54:42.525 INFO:tasks.workunit.client.0.smithi102.stdout:0/691: mknod d6/de/d13/d25/d31/daa/ced 0 2024-03-26T18:54:42.527 INFO:tasks.workunit.client.0.smithi102.stdout:5/881: unlink d2/d25/d5b/d72/l12f 0 2024-03-26T18:54:42.527 INFO:tasks.workunit.client.0.smithi102.stdout:5/882: read - d2/d25/d5b/d72/f66 zero size 2024-03-26T18:54:42.528 INFO:tasks.workunit.client.0.smithi102.stdout:8/706: symlink d0/d3c/d96/dd5/le9 0 2024-03-26T18:54:42.531 INFO:tasks.workunit.client.0.smithi102.stdout:3/838: mknod df/d1c/d70/dcd/df7/d10e/c12d 0 2024-03-26T18:54:42.532 INFO:tasks.workunit.client.0.smithi102.stdout:3/839: fdatasync df/d29/d31/f108 0 2024-03-26T18:54:42.532 INFO:tasks.workunit.client.0.smithi102.stdout:3/840: read df/f1a [1334169,5517] 0 2024-03-26T18:54:42.532 INFO:tasks.workunit.client.0.smithi102.stdout:3/841: write df/d29/d31/d36/d3c/fc2 [2035188,8756] 0 2024-03-26T18:54:42.532 INFO:tasks.workunit.client.0.smithi102.stdout:3/842: chown df/d29/d31/d36/d45/f121 2013300402 1 2024-03-26T18:54:42.532 INFO:tasks.workunit.client.0.smithi102.stdout:1/612: creat de/d16/d1d/d61/d6a/da5/da6/dca/fcc x:0 0 0 2024-03-26T18:54:42.533 INFO:tasks.workunit.client.0.smithi102.stdout:7/577: dread d7/d10/d17/f81 [0,4194304] 0 2024-03-26T18:54:42.533 INFO:tasks.workunit.client.0.smithi102.stdout:2/653: creat d3/d74/fe9 x:0 0 0 2024-03-26T18:54:42.533 INFO:tasks.workunit.client.0.smithi102.stdout:2/654: creat d3/d11/d15/d46/fea x:0 0 0 2024-03-26T18:54:42.536 INFO:tasks.workunit.client.0.smithi102.stdout:1/613: write de/d16/f40 [1336554,81468] 0 2024-03-26T18:54:42.537 INFO:tasks.workunit.client.0.smithi102.stdout:6/636: rename d2/d3d/d53/d77/f9e to d2/d3d/d53/fcd 0 2024-03-26T18:54:42.538 INFO:tasks.workunit.client.0.smithi102.stdout:4/527: symlink d1/d17/d2a/la2 0 2024-03-26T18:54:42.538 INFO:tasks.workunit.client.0.smithi102.stdout:5/883: mkdir d2/d25/d75/db9/d43/d9e/d82/d136 0 2024-03-26T18:54:42.539 INFO:tasks.workunit.client.0.smithi102.stdout:8/707: symlink d0/d15/d5d/dd3/d26/d19/lea 0 2024-03-26T18:54:42.541 INFO:tasks.workunit.client.0.smithi102.stdout:2/655: mkdir d3/d82/d93/d9e/d99/deb 0 2024-03-26T18:54:42.545 INFO:tasks.workunit.client.0.smithi102.stdout:0/692: rename d6/dbc/fc9 to d6/de/d13/d25/da4/fee 0 2024-03-26T18:54:42.545 INFO:tasks.workunit.client.1.smithi190.stdout:5/909: dwrite d2/d17/d1b/db6/fc7 [0,4194304] 0 2024-03-26T18:54:42.545 INFO:tasks.workunit.client.0.smithi102.stdout:6/637: mkdir d2/d12/dce 0 2024-03-26T18:54:42.547 INFO:tasks.workunit.client.0.smithi102.stdout:4/528: symlink d1/d27/d98/la3 0 2024-03-26T18:54:42.548 INFO:tasks.workunit.client.1.smithi190.stdout:5/910: getdents d2/d17/d1b/d60/d63/d102 0 2024-03-26T18:54:42.549 INFO:tasks.workunit.client.1.smithi190.stdout:5/911: write d2/d17/d1b/db6/d68/f75 [5228279,23552] 0 2024-03-26T18:54:42.549 INFO:tasks.workunit.client.1.smithi190.stdout:5/912: write d2/d17/d1b/db6/d24/d3b/faf [1545783,21773] 0 2024-03-26T18:54:42.552 INFO:tasks.workunit.client.1.smithi190.stdout:5/913: symlink d2/d17/d1b/db6/d24/d3b/da5/l132 0 2024-03-26T18:54:42.554 INFO:tasks.workunit.client.0.smithi102.stdout:5/884: dwrite d2/d14/f70 [0,4194304] 0 2024-03-26T18:54:42.555 INFO:tasks.workunit.client.0.smithi102.stdout:1/614: rmdir de/d16/d1d/d2a 39 2024-03-26T18:54:42.558 INFO:tasks.workunit.client.0.smithi102.stdout:3/843: rename df/d29/d31/dc0/d10c to df/d1c/d34/da5/d12e 0 2024-03-26T18:54:42.558 INFO:tasks.workunit.client.0.smithi102.stdout:3/844: creat df/d1c/d34/d53/f12f x:0 0 0 2024-03-26T18:54:42.558 INFO:tasks.workunit.client.0.smithi102.stdout:3/845: write df/d61/d55/f95 [2057289,14166] 0 2024-03-26T18:54:42.561 INFO:tasks.workunit.client.0.smithi102.stdout:6/638: symlink d2/d3d/lcf 0 2024-03-26T18:54:42.562 INFO:tasks.workunit.client.0.smithi102.stdout:4/529: creat d1/d17/d1f/d5d/d8b/d3e/fa4 x:0 0 0 2024-03-26T18:54:42.567 INFO:tasks.workunit.client.0.smithi102.stdout:5/885: unlink d2/d25/d75/def/dec/d112/db8/fc8 0 2024-03-26T18:54:42.569 INFO:tasks.workunit.client.0.smithi102.stdout:1/615: link de/d16/d1d/d2a/d2c/l5b de/d16/d1d/d61/lcd 0 2024-03-26T18:54:42.571 INFO:tasks.workunit.client.0.smithi102.stdout:8/708: rename d0/d15/d5d/dd3/d26/d19/d76/lc1 to d0/d15/d5d/dd3/d2c/d4e/leb 0 2024-03-26T18:54:42.571 INFO:tasks.workunit.client.0.smithi102.stdout:8/709: fsync d0/d15/fb0 0 2024-03-26T18:54:42.571 INFO:tasks.workunit.client.0.smithi102.stdout:8/710: write d0/d15/d5d/dd3/d2c/d2f/f78 [3986541,44831] 0 2024-03-26T18:54:42.574 INFO:tasks.workunit.client.0.smithi102.stdout:3/846: creat df/d29/d31/dc0/f130 x:0 0 0 2024-03-26T18:54:42.574 INFO:tasks.workunit.client.0.smithi102.stdout:3/847: stat df/d1c/d34/f8e 0 2024-03-26T18:54:42.576 INFO:tasks.workunit.client.0.smithi102.stdout:0/693: rmdir d6/de/d13/d1f/d55/d57/d7d 39 2024-03-26T18:54:42.577 INFO:tasks.workunit.client.0.smithi102.stdout:4/530: rmdir d1/d25 39 2024-03-26T18:54:42.578 INFO:tasks.workunit.client.0.smithi102.stdout:5/886: rmdir d2/d25/d75/db9/d43/d9e/daf 39 2024-03-26T18:54:42.580 INFO:tasks.workunit.client.0.smithi102.stdout:8/711: getdents d0/d15/d5d/dd3/d2c/d2f 0 2024-03-26T18:54:42.581 INFO:tasks.workunit.client.0.smithi102.stdout:6/639: dwrite d2/df/d35/d43/d75/f8c [0,4194304] 0 2024-03-26T18:54:42.584 INFO:tasks.workunit.client.0.smithi102.stdout:3/848: rename df/d29/d31/f79 to df/d29/d31/d77/d8a/dae/dee/f131 0 2024-03-26T18:54:42.584 INFO:tasks.workunit.client.0.smithi102.stdout:0/694: symlink d6/d2d/d3d/d80/d83/lef 0 2024-03-26T18:54:42.586 INFO:tasks.workunit.client.0.smithi102.stdout:4/531: symlink d1/d17/d1a/d21/d77/la5 0 2024-03-26T18:54:42.589 INFO:tasks.workunit.client.0.smithi102.stdout:5/887: creat d2/d25/d75/def/f137 x:0 0 0 2024-03-26T18:54:42.590 INFO:tasks.workunit.client.0.smithi102.stdout:3/849: dread df/d1c/d34/f37 [0,4194304] 0 2024-03-26T18:54:42.592 INFO:tasks.workunit.client.0.smithi102.stdout:1/616: truncate de/d16/f40 4112194 0 2024-03-26T18:54:42.594 INFO:tasks.workunit.client.0.smithi102.stdout:8/712: dwrite d0/d9/f7d [0,4194304] 0 2024-03-26T18:54:42.595 INFO:tasks.workunit.client.0.smithi102.stdout:6/640: truncate d2/f52 841304 0 2024-03-26T18:54:42.599 INFO:tasks.workunit.client.0.smithi102.stdout:5/888: symlink d2/dea/d4f/da9/dee/l138 0 2024-03-26T18:54:42.600 INFO:tasks.workunit.client.0.smithi102.stdout:3/850: rename df/d29/d31/d36/d3c/d6f/d75/cea to df/d29/d31/d77/d8a/dae/dee/df2/d100/d11a/c132 0 2024-03-26T18:54:42.603 INFO:tasks.workunit.client.1.smithi190.stdout:3/795: sync 2024-03-26T18:54:42.603 INFO:tasks.workunit.client.1.smithi190.stdout:3/796: readlink d1/d3/df/d18/d35/d41/dae/d9c/le2 0 2024-03-26T18:54:42.605 INFO:tasks.workunit.client.0.smithi102.stdout:8/713: read d0/d15/d5d/dd3/d2c/da7/f29 [5086088,9143] 0 2024-03-26T18:54:42.605 INFO:tasks.workunit.client.0.smithi102.stdout:1/617: symlink de/d1f/lce 0 2024-03-26T18:54:42.608 INFO:tasks.workunit.client.0.smithi102.stdout:6/641: write d2/d12/f3f [2764062,23464] 0 2024-03-26T18:54:42.608 INFO:tasks.workunit.client.1.smithi190.stdout:3/797: creat d1/d3/df/d31/ff1 x:0 0 0 2024-03-26T18:54:42.611 INFO:tasks.workunit.client.0.smithi102.stdout:0/695: mknod d6/de/d13/d1f/d55/d57/d7d/d7f/dc8/cf0 0 2024-03-26T18:54:42.615 INFO:tasks.workunit.client.0.smithi102.stdout:4/532: mknod d1/d17/d1f/d5d/d8b/ca6 0 2024-03-26T18:54:42.615 INFO:tasks.workunit.client.0.smithi102.stdout:6/642: dread d2/f47 [0,4194304] 0 2024-03-26T18:54:42.616 INFO:tasks.workunit.client.0.smithi102.stdout:5/889: mkdir d2/d25/db0/d139 0 2024-03-26T18:54:42.618 INFO:tasks.workunit.client.0.smithi102.stdout:3/851: creat df/d1c/d34/d9b/f133 x:0 0 0 2024-03-26T18:54:42.619 INFO:tasks.workunit.client.0.smithi102.stdout:1/618: mknod de/d16/d1d/ccf 0 2024-03-26T18:54:42.621 INFO:tasks.workunit.client.0.smithi102.stdout:0/696: write d6/de/d13/d25/d31/fd7 [458679,78956] 0 2024-03-26T18:54:42.621 INFO:tasks.workunit.client.1.smithi190.stdout:3/798: read d1/de/f73 [1577940,8006] 0 2024-03-26T18:54:42.622 INFO:tasks.workunit.client.0.smithi102.stdout:6/643: mkdir d2/d3/dd0 0 2024-03-26T18:54:42.623 INFO:tasks.workunit.client.1.smithi190.stdout:6/824: sync 2024-03-26T18:54:42.623 INFO:tasks.workunit.client.1.smithi190.stdout:3/799: truncate d1/de/f69 5231527 0 2024-03-26T18:54:42.623 INFO:tasks.workunit.client.1.smithi190.stdout:6/825: write d17/d28/d2c/d77/f6f [4178967,8369] 0 2024-03-26T18:54:42.623 INFO:tasks.workunit.client.1.smithi190.stdout:6/826: chown d17/d28/d74/d7b/fa5 3 1 2024-03-26T18:54:42.624 INFO:tasks.workunit.client.1.smithi190.stdout:6/827: chown d17/d28/d2c/d50/fc2 517937441 1 2024-03-26T18:54:42.624 INFO:tasks.workunit.client.1.smithi190.stdout:6/828: fdatasync d17/d28/d2c/d63/f70 0 2024-03-26T18:54:42.624 INFO:tasks.workunit.client.1.smithi190.stdout:3/800: readlink d1/d3/df/d18/d35/d41/dae/l55 0 2024-03-26T18:54:42.627 INFO:tasks.workunit.client.0.smithi102.stdout:0/697: truncate d6/de/d13/d25/d31/d45/f65 1192604 0 2024-03-26T18:54:42.627 INFO:tasks.workunit.client.0.smithi102.stdout:4/533: getdents d1/d17 0 2024-03-26T18:54:42.627 INFO:tasks.workunit.client.0.smithi102.stdout:5/890: dread d2/d25/d75/def/dec/d112/f7a [0,4194304] 0 2024-03-26T18:54:42.629 INFO:tasks.workunit.client.1.smithi190.stdout:3/801: unlink d1/d3/df/d18/d1b/l37 0 2024-03-26T18:54:42.631 INFO:tasks.workunit.client.0.smithi102.stdout:6/644: creat d2/df/d35/d43/fd1 x:0 0 0 2024-03-26T18:54:42.631 INFO:tasks.workunit.client.0.smithi102.stdout:6/645: chown d2/d3d/d53/d77/f88 15 1 2024-03-26T18:54:42.631 INFO:tasks.workunit.client.0.smithi102.stdout:3/852: rmdir df/d1c/d70 39 2024-03-26T18:54:42.631 INFO:tasks.workunit.client.0.smithi102.stdout:3/853: readlink df/d1c/l63 0 2024-03-26T18:54:42.632 INFO:tasks.workunit.client.0.smithi102.stdout:3/854: chown df/d29/d31/d36/d45/daf/d105/c107 8405 1 2024-03-26T18:54:42.633 INFO:tasks.workunit.client.1.smithi190.stdout:3/802: creat d1/d3/d52/d70/db3/ff2 x:0 0 0 2024-03-26T18:54:42.633 INFO:tasks.workunit.client.1.smithi190.stdout:3/803: stat d1/de/d8a/fa7 0 2024-03-26T18:54:42.634 INFO:tasks.workunit.client.0.smithi102.stdout:5/891: dread d2/d14/d1a/f1b [0,4194304] 0 2024-03-26T18:54:42.634 INFO:tasks.workunit.client.0.smithi102.stdout:5/892: read d2/dea/d4f/f96 [94002,54776] 0 2024-03-26T18:54:42.634 INFO:tasks.workunit.client.0.smithi102.stdout:5/893: stat d2/d25/d75/db9/d43/d9e/f8c 0 2024-03-26T18:54:42.634 INFO:tasks.workunit.client.0.smithi102.stdout:5/894: write d2/d14/f131 [474228,107017] 0 2024-03-26T18:54:42.634 INFO:tasks.workunit.client.0.smithi102.stdout:0/698: mknod d6/d2d/cf1 0 2024-03-26T18:54:42.635 INFO:tasks.workunit.client.0.smithi102.stdout:4/534: mknod d1/d17/d1f/d5d/ca7 0 2024-03-26T18:54:42.636 INFO:tasks.workunit.client.1.smithi190.stdout:3/804: mknod d1/d3/df/d31/dd1/cf3 0 2024-03-26T18:54:42.636 INFO:tasks.workunit.client.1.smithi190.stdout:3/805: read d1/d3/df/d18/d35/d41/dae/f96 [202097,74256] 0 2024-03-26T18:54:42.636 INFO:tasks.workunit.client.0.smithi102.stdout:6/646: symlink d2/df/d71/d80/d91/ld2 0 2024-03-26T18:54:42.638 INFO:tasks.workunit.client.0.smithi102.stdout:5/895: rename d2/d25/f81 to d2/dea/d4f/da9/f13a 0 2024-03-26T18:54:42.640 INFO:tasks.workunit.client.0.smithi102.stdout:1/619: dwrite de/d16/f29 [0,4194304] 0 2024-03-26T18:54:42.641 INFO:tasks.workunit.client.0.smithi102.stdout:0/699: unlink d6/de/f4c 0 2024-03-26T18:54:42.641 INFO:tasks.workunit.client.0.smithi102.stdout:6/647: dread d2/faf [0,4194304] 0 2024-03-26T18:54:42.641 INFO:tasks.workunit.client.0.smithi102.stdout:0/700: fdatasync d6/de/d13/d25/d31/d45/d82/d84/d88/fde 0 2024-03-26T18:54:42.641 INFO:tasks.workunit.client.0.smithi102.stdout:0/701: fsync d6/de/d13/d9a/f68 0 2024-03-26T18:54:42.642 INFO:tasks.workunit.client.0.smithi102.stdout:4/535: unlink d1/d25/d90/l92 0 2024-03-26T18:54:42.643 INFO:tasks.workunit.client.1.smithi190.stdout:3/806: getdents d1/de/d1c/d36 0 2024-03-26T18:54:42.645 INFO:tasks.workunit.client.0.smithi102.stdout:5/896: creat d2/dea/d4f/da9/dee/f13b x:0 0 0 2024-03-26T18:54:42.645 INFO:tasks.workunit.client.0.smithi102.stdout:5/897: creat d2/dea/dbb/dcf/f13c x:0 0 0 2024-03-26T18:54:42.646 INFO:tasks.workunit.client.1.smithi190.stdout:3/807: creat d1/d3/df/ff4 x:0 0 0 2024-03-26T18:54:42.647 INFO:tasks.workunit.client.0.smithi102.stdout:6/648: creat d2/d7/fd3 x:0 0 0 2024-03-26T18:54:42.647 INFO:tasks.workunit.client.0.smithi102.stdout:4/536: chown d1/d17/d1f/d5d/d8b/c52 934 1 2024-03-26T18:54:42.649 INFO:tasks.workunit.client.1.smithi190.stdout:3/808: mkdir d1/d3/d5a/d6e/df5 0 2024-03-26T18:54:42.649 INFO:tasks.workunit.client.1.smithi190.stdout:3/809: fsync d1/f11 0 2024-03-26T18:54:42.649 INFO:tasks.workunit.client.1.smithi190.stdout:3/810: truncate d1/dd3/fe8 695508 0 2024-03-26T18:54:42.650 INFO:tasks.workunit.client.0.smithi102.stdout:6/649: truncate d2/d3d/d53/f76 816644 0 2024-03-26T18:54:42.650 INFO:tasks.workunit.client.0.smithi102.stdout:6/650: stat d2/f33 0 2024-03-26T18:54:42.650 INFO:tasks.workunit.client.0.smithi102.stdout:1/620: dread de/d16/d1d/f82 [0,4194304] 0 2024-03-26T18:54:42.651 INFO:tasks.workunit.client.1.smithi190.stdout:3/811: truncate d1/de/d81/dad/fb0 1013825 0 2024-03-26T18:54:42.653 INFO:tasks.workunit.client.0.smithi102.stdout:4/537: unlink d1/d17/d2a/c7b 0 2024-03-26T18:54:42.655 INFO:tasks.workunit.client.1.smithi190.stdout:3/812: creat d1/d47/deb/ff6 x:0 0 0 2024-03-26T18:54:42.656 INFO:tasks.workunit.client.0.smithi102.stdout:4/538: creat d1/fa8 x:0 0 0 2024-03-26T18:54:42.657 INFO:tasks.workunit.client.0.smithi102.stdout:4/539: read - d1/fa8 zero size 2024-03-26T18:54:42.658 INFO:tasks.workunit.client.1.smithi190.stdout:3/813: mkdir d1/dd3/df7 0 2024-03-26T18:54:42.658 INFO:tasks.workunit.client.1.smithi190.stdout:3/814: chown d1/d3/d5a/fee 108 1 2024-03-26T18:54:42.660 INFO:tasks.workunit.client.1.smithi190.stdout:3/815: creat d1/de/d8a/de1/ff8 x:0 0 0 2024-03-26T18:54:42.660 INFO:tasks.workunit.client.1.smithi190.stdout:3/816: creat d1/dd3/ff9 x:0 0 0 2024-03-26T18:54:42.665 INFO:tasks.workunit.client.1.smithi190.stdout:3/817: rename d1/de/d8a/de1/le5 to d1/d3/df/d31/dd1/lfa 0 2024-03-26T18:54:42.667 INFO:tasks.workunit.client.0.smithi102.stdout:0/702: dwrite d6/de/d13/f7e [0,4194304] 0 2024-03-26T18:54:42.667 INFO:tasks.workunit.client.0.smithi102.stdout:3/855: dwrite df/d8b/f122 [0,4194304] 0 2024-03-26T18:54:42.667 INFO:tasks.workunit.client.0.smithi102.stdout:5/898: dread d2/dea/f59 [0,4194304] 0 2024-03-26T18:54:42.670 INFO:tasks.workunit.client.0.smithi102.stdout:6/651: dwrite d2/df/f96 [0,4194304] 0 2024-03-26T18:54:42.671 INFO:tasks.workunit.client.0.smithi102.stdout:1/621: write de/d1f/f33 [2279492,7759] 0 2024-03-26T18:54:42.671 INFO:tasks.workunit.client.0.smithi102.stdout:1/622: chown de/d2e/d62/f64 5 1 2024-03-26T18:54:42.672 INFO:tasks.workunit.client.0.smithi102.stdout:1/623: dread - de/d16/d1d/d2a/fb2 zero size 2024-03-26T18:54:42.675 INFO:tasks.workunit.client.0.smithi102.stdout:3/856: rename df/d29/d31/d36/d3c/d6f/d75/f94 to df/d29/d31/d77/d8a/dae/dee/df2/d100/d11a/f134 0 2024-03-26T18:54:42.676 INFO:tasks.workunit.client.0.smithi102.stdout:3/857: creat df/d1c/d34/da5/dce/f135 x:0 0 0 2024-03-26T18:54:42.676 INFO:tasks.workunit.client.0.smithi102.stdout:3/858: stat df/d61/d55/f90 0 2024-03-26T18:54:42.686 INFO:tasks.workunit.client.0.smithi102.stdout:6/652: unlink d2/df/d71/d80/d91/fa9 0 2024-03-26T18:54:42.686 INFO:tasks.workunit.client.0.smithi102.stdout:5/899: dwrite d2/d14/f113 [0,4194304] 0 2024-03-26T18:54:42.686 INFO:tasks.workunit.client.0.smithi102.stdout:4/540: dwrite d1/d25/d76/f86 [0,4194304] 0 2024-03-26T18:54:42.686 INFO:tasks.workunit.client.0.smithi102.stdout:4/541: readlink d1/d27/d98/la3 0 2024-03-26T18:54:42.687 INFO:tasks.workunit.client.0.smithi102.stdout:4/542: readlink d1/d17/d2a/l5c 0 2024-03-26T18:54:42.688 INFO:tasks.workunit.client.0.smithi102.stdout:1/624: creat de/d16/d1d/d2a/d2c/d56/d5f/daa/fd0 x:0 0 0 2024-03-26T18:54:42.690 INFO:tasks.workunit.client.0.smithi102.stdout:0/703: getdents d6/de 0 2024-03-26T18:54:42.690 INFO:tasks.workunit.client.0.smithi102.stdout:0/704: readlink d6/de/d13/d1f/l7a 0 2024-03-26T18:54:42.690 INFO:tasks.workunit.client.0.smithi102.stdout:0/705: readlink d6/l10 0 2024-03-26T18:54:42.690 INFO:tasks.workunit.client.0.smithi102.stdout:0/706: write d6/de/d13/d1f/d55/d57/f46 [771188,27132] 0 2024-03-26T18:54:42.694 INFO:tasks.workunit.client.0.smithi102.stdout:3/859: symlink df/d29/d31/d77/d8a/dae/dee/df2/d100/d11a/l136 0 2024-03-26T18:54:42.695 INFO:tasks.workunit.client.0.smithi102.stdout:6/653: symlink d2/d3d/d53/d5c/dc5/ld4 0 2024-03-26T18:54:42.695 INFO:tasks.workunit.client.0.smithi102.stdout:6/654: creat d2/df/d35/d43/fd5 x:0 0 0 2024-03-26T18:54:42.698 INFO:tasks.workunit.client.0.smithi102.stdout:0/707: write d6/de/d13/d25/d4f/d66/fb7 [2187039,110229] 0 2024-03-26T18:54:42.698 INFO:tasks.workunit.client.0.smithi102.stdout:0/708: truncate d6/f7 4998352 0 2024-03-26T18:54:42.699 INFO:tasks.workunit.client.0.smithi102.stdout:0/709: dread d6/de/d13/d25/d31/f8e [0,4194304] 0 2024-03-26T18:54:42.699 INFO:tasks.workunit.client.0.smithi102.stdout:0/710: read d6/de/d13/d25/d31/d45/d82/d84/d88/fde [552656,106778] 0 2024-03-26T18:54:42.700 INFO:tasks.workunit.client.0.smithi102.stdout:5/900: creat d2/d25/d75/db9/ded/d135/f13d x:0 0 0 2024-03-26T18:54:42.701 INFO:tasks.workunit.client.1.smithi190.stdout:5/914: sync 2024-03-26T18:54:42.701 INFO:tasks.workunit.client.1.smithi190.stdout:1/902: sync 2024-03-26T18:54:42.703 INFO:tasks.workunit.client.0.smithi102.stdout:1/625: creat de/d16/d1d/d61/d6a/fd1 x:0 0 0 2024-03-26T18:54:42.703 INFO:tasks.workunit.client.0.smithi102.stdout:3/860: symlink df/d29/d31/d77/d8a/dae/dee/df2/d100/l137 0 2024-03-26T18:54:42.710 INFO:tasks.workunit.client.1.smithi190.stdout:5/915: unlink d2/d17/d1b/d60/fc5 0 2024-03-26T18:54:42.711 INFO:tasks.workunit.client.0.smithi102.stdout:6/655: mknod d2/cd6 0 2024-03-26T18:54:42.712 INFO:tasks.workunit.client.0.smithi102.stdout:0/711: creat d6/de/d13/d1f/d55/ff2 x:0 0 0 2024-03-26T18:54:42.715 INFO:tasks.workunit.client.1.smithi190.stdout:5/916: rename d2/d17/d1b/db6/d24/d3b/da7/fe1 to d2/d17/d1b/db6/d88/dc9/f133 0 2024-03-26T18:54:42.717 INFO:tasks.workunit.client.0.smithi102.stdout:5/901: symlink d2/d25/d75/db9/d43/d9e/l13e 0 2024-03-26T18:54:42.718 INFO:tasks.workunit.client.0.smithi102.stdout:3/861: link df/d29/d31/d36/d3c/f11f df/d1c/d34/d9b/f138 0 2024-03-26T18:54:42.722 INFO:tasks.workunit.client.1.smithi190.stdout:5/917: creat d2/d17/d1b/db6/d24/d3b/d126/f134 x:0 0 0 2024-03-26T18:54:42.722 INFO:tasks.workunit.client.0.smithi102.stdout:6/656: mknod d2/df/d35/cd7 0 2024-03-26T18:54:42.723 INFO:tasks.workunit.client.0.smithi102.stdout:6/657: fdatasync d2/d3d/d53/f74 0 2024-03-26T18:54:42.723 INFO:tasks.workunit.client.0.smithi102.stdout:6/658: chown d2/df/d35/d43/d75/fb5 15858 1 2024-03-26T18:54:42.723 INFO:tasks.workunit.client.0.smithi102.stdout:6/659: fsync d2/d3d/d53/f5b 0 2024-03-26T18:54:42.726 INFO:tasks.workunit.client.0.smithi102.stdout:6/660: symlink d2/df/d35/ld8 0 2024-03-26T18:54:42.732 INFO:tasks.workunit.client.0.smithi102.stdout:9/605: sync 2024-03-26T18:54:42.732 INFO:tasks.workunit.client.0.smithi102.stdout:9/606: creat df/d1a/d57/fde x:0 0 0 2024-03-26T18:54:42.735 INFO:tasks.workunit.client.0.smithi102.stdout:6/661: dread d2/d3d/d53/db1/fb3 [0,4194304] 0 2024-03-26T18:54:42.736 INFO:tasks.workunit.client.0.smithi102.stdout:1/626: dwrite f8 [4194304,4194304] 0 2024-03-26T18:54:42.738 INFO:tasks.workunit.client.0.smithi102.stdout:6/662: read d2/df/f2b [478718,68152] 0 2024-03-26T18:54:42.738 INFO:tasks.workunit.client.0.smithi102.stdout:6/663: write d2/df/d71/fbb [605110,64629] 0 2024-03-26T18:54:42.738 INFO:tasks.workunit.client.0.smithi102.stdout:1/627: truncate de/d16/d1d/d2a/d2c/d56/d5f/f8b 98591 0 2024-03-26T18:54:42.738 INFO:tasks.workunit.client.0.smithi102.stdout:1/628: creat de/d16/d1d/d2a/d2c/d56/d5f/daa/fd2 x:0 0 0 2024-03-26T18:54:42.738 INFO:tasks.workunit.client.0.smithi102.stdout:1/629: chown de/d16/d1d 48143236 1 2024-03-26T18:54:42.739 INFO:tasks.workunit.client.0.smithi102.stdout:1/630: chown de/d16/d1d/d2a/d2c/d56/d78/d80/f98 48 1 2024-03-26T18:54:42.740 INFO:tasks.workunit.client.0.smithi102.stdout:1/631: symlink de/d16/d1d/d2a/d2c/d56/d78/d80/ld3 0 2024-03-26T18:54:42.741 INFO:tasks.workunit.client.0.smithi102.stdout:1/632: dread de/d16/d1d/d2a/d2c/d56/d78/f8c [0,4194304] 0 2024-03-26T18:54:42.742 INFO:tasks.workunit.client.0.smithi102.stdout:6/664: rename d2/d7/d3c/cc2 to d2/df/d71/d80/d91/cd9 0 2024-03-26T18:54:42.751 INFO:tasks.workunit.client.0.smithi102.stdout:9/607: read df/d84/f78 [2866965,15338] 0 2024-03-26T18:54:42.751 INFO:tasks.workunit.client.0.smithi102.stdout:9/608: fdatasync df/d1a/d57/f8f 0 2024-03-26T18:54:42.751 INFO:tasks.workunit.client.0.smithi102.stdout:9/609: stat df/d1a/d23/d4e/d8e/ca1 0 2024-03-26T18:54:42.751 INFO:tasks.workunit.client.0.smithi102.stdout:9/610: truncate df/d84/f9e 664540 0 2024-03-26T18:54:42.754 INFO:tasks.workunit.client.1.smithi190.stdout:1/903: dwrite d3/d73/d88/dcc/ff8 [0,4194304] 0 2024-03-26T18:54:42.755 INFO:tasks.workunit.client.0.smithi102.stdout:6/665: dread d2/f2e [0,4194304] 0 2024-03-26T18:54:42.755 INFO:tasks.workunit.client.0.smithi102.stdout:6/666: write d2/d3d/f70 [1909579,127718] 0 2024-03-26T18:54:42.767 INFO:tasks.workunit.client.1.smithi190.stdout:5/918: dwrite d2/d17/d1b/db6/d24/d3b/faf [0,4194304] 0 2024-03-26T18:54:42.769 INFO:tasks.workunit.client.0.smithi102.stdout:5/902: dwrite d2/d25/d75/def/f137 [0,4194304] 0 2024-03-26T18:54:42.769 INFO:tasks.workunit.client.0.smithi102.stdout:3/862: dwrite df/d29/d31/d77/d8a/fbd [0,4194304] 0 2024-03-26T18:54:42.769 INFO:tasks.workunit.client.0.smithi102.stdout:5/903: chown d2/d25/d5b/fc4 4 1 2024-03-26T18:54:42.771 INFO:tasks.workunit.client.0.smithi102.stdout:1/633: dwrite de/d16/d1d/d2a/fb2 [0,4194304] 0 2024-03-26T18:54:42.772 INFO:tasks.workunit.client.0.smithi102.stdout:5/904: dread d2/d25/d75/ffb [0,4194304] 0 2024-03-26T18:54:42.772 INFO:tasks.workunit.client.0.smithi102.stdout:5/905: readlink d2/dea/d4f/da9/dee/l121 0 2024-03-26T18:54:42.772 INFO:tasks.workunit.client.0.smithi102.stdout:9/611: dwrite df/d1a/d23/d3e/d75/f4a [4194304,4194304] 0 2024-03-26T18:54:42.772 INFO:tasks.workunit.client.0.smithi102.stdout:9/612: read - df/d84/fd2 zero size 2024-03-26T18:54:42.772 INFO:tasks.workunit.client.0.smithi102.stdout:9/613: write df/d1a/d23/f7e [306802,82258] 0 2024-03-26T18:54:42.775 INFO:tasks.workunit.client.1.smithi190.stdout:5/919: creat d2/d17/d62/d123/f135 x:0 0 0 2024-03-26T18:54:42.777 INFO:tasks.workunit.client.1.smithi190.stdout:1/904: dread d3/da/f24 [0,4194304] 0 2024-03-26T18:54:42.777 INFO:tasks.workunit.client.1.smithi190.stdout:1/905: readlink d3/d73/d88/df1/lb3 0 2024-03-26T18:54:42.782 INFO:tasks.workunit.client.1.smithi190.stdout:1/906: creat d3/da/dd/d2f/d33/d115/f120 x:0 0 0 2024-03-26T18:54:42.783 INFO:tasks.workunit.client.0.smithi102.stdout:5/906: symlink d2/dfc/l13f 0 2024-03-26T18:54:42.784 INFO:tasks.workunit.client.1.smithi190.stdout:6/829: sync 2024-03-26T18:54:42.784 INFO:tasks.workunit.client.1.smithi190.stdout:6/830: fsync d17/d28/d2c/f4c 0 2024-03-26T18:54:42.785 INFO:tasks.workunit.client.0.smithi102.stdout:9/614: mkdir df/d1a/d65/dbe/ddf 0 2024-03-26T18:54:42.788 INFO:tasks.workunit.client.0.smithi102.stdout:9/615: symlink df/d1a/d23/d60/le0 0 2024-03-26T18:54:42.788 INFO:tasks.workunit.client.1.smithi190.stdout:6/831: mknod d17/d28/d8b/d95/dda/c11a 0 2024-03-26T18:54:42.789 INFO:tasks.workunit.client.1.smithi190.stdout:1/907: dread d3/da/d2e/d59/dae/fcb [0,4194304] 0 2024-03-26T18:54:42.790 INFO:tasks.workunit.client.0.smithi102.stdout:9/616: mkdir df/d1a/d7f/de1 0 2024-03-26T18:54:42.791 INFO:tasks.workunit.client.1.smithi190.stdout:3/818: sync 2024-03-26T18:54:42.792 INFO:tasks.workunit.client.1.smithi190.stdout:1/908: unlink d3/d9/da3/ld5 0 2024-03-26T18:54:42.792 INFO:tasks.workunit.client.1.smithi190.stdout:1/909: chown d3/da/dd/c96 15197 1 2024-03-26T18:54:42.793 INFO:tasks.workunit.client.1.smithi190.stdout:5/920: dread d2/f14 [0,4194304] 0 2024-03-26T18:54:42.795 INFO:tasks.workunit.client.0.smithi102.stdout:9/617: unlink df/d1a/d23/ld8 0 2024-03-26T18:54:42.795 INFO:tasks.workunit.client.0.smithi102.stdout:3/863: dwrite df/d29/d31/d36/d3c/d6f/d75/f125 [0,4194304] 0 2024-03-26T18:54:42.798 INFO:tasks.workunit.client.1.smithi190.stdout:1/910: creat d3/da/d3a/d99/f121 x:0 0 0 2024-03-26T18:54:42.802 INFO:tasks.workunit.client.1.smithi190.stdout:6/832: dread d17/f87 [0,4194304] 0 2024-03-26T18:54:42.804 INFO:tasks.workunit.client.0.smithi102.stdout:9/618: truncate df/d84/f78 232523 0 2024-03-26T18:54:42.804 INFO:tasks.workunit.client.0.smithi102.stdout:1/634: dwrite de/d16/d1d/d2a/d2c/d43/f4f [0,4194304] 0 2024-03-26T18:54:42.805 INFO:tasks.workunit.client.1.smithi190.stdout:5/921: rename d2/d17/d98/lfd to d2/d17/d98/d3a/d3c/l136 0 2024-03-26T18:54:42.807 INFO:tasks.workunit.client.1.smithi190.stdout:3/819: chown d1/de/d4e/f8b 302125 1 2024-03-26T18:54:42.807 INFO:tasks.workunit.client.1.smithi190.stdout:3/820: stat d1/d3/df/d31 0 2024-03-26T18:54:42.809 INFO:tasks.workunit.client.1.smithi190.stdout:5/922: mkdir d2/d17/d1b/d60/d63/d137 0 2024-03-26T18:54:42.809 INFO:tasks.workunit.client.1.smithi190.stdout:5/923: creat d2/d17/d62/f138 x:0 0 0 2024-03-26T18:54:42.809 INFO:tasks.workunit.client.1.smithi190.stdout:5/924: fdatasync d2/d17/d1b/db6/d24/fab 0 2024-03-26T18:54:42.810 INFO:tasks.workunit.client.0.smithi102.stdout:3/864: rename df/d29/d31/d36/f50 to df/d29/d3a/d93/f139 0 2024-03-26T18:54:42.811 INFO:tasks.workunit.client.1.smithi190.stdout:1/911: dread d3/d5f/f71 [0,4194304] 0 2024-03-26T18:54:42.812 INFO:tasks.workunit.client.1.smithi190.stdout:6/833: rename d17/d28/d2b/fca to d17/d28/f11b 0 2024-03-26T18:54:42.813 INFO:tasks.workunit.client.1.smithi190.stdout:3/821: mkdir d1/d3/df/d18/d1b/de4/dfb 0 2024-03-26T18:54:42.814 INFO:tasks.workunit.client.1.smithi190.stdout:3/822: write d1/d3/df/d18/d35/d41/dae/fe9 [571381,6184] 0 2024-03-26T18:54:42.814 INFO:tasks.workunit.client.1.smithi190.stdout:3/823: dread - d1/de/d8a/fc2 zero size 2024-03-26T18:54:42.815 INFO:tasks.workunit.client.1.smithi190.stdout:5/925: unlink d2/f3 0 2024-03-26T18:54:42.817 INFO:tasks.workunit.client.0.smithi102.stdout:3/865: mkdir df/d29/d31/d36/d45/daf/d105/d13a 0 2024-03-26T18:54:42.817 INFO:tasks.workunit.client.0.smithi102.stdout:3/866: chown df/d29/d31/d77/d8a/dae 0 1 2024-03-26T18:54:42.819 INFO:tasks.workunit.client.1.smithi190.stdout:3/824: symlink d1/d3/df/d18/d35/d41/lfc 0 2024-03-26T18:54:42.820 INFO:tasks.workunit.client.0.smithi102.stdout:1/635: write de/d16/f29 [552758,100979] 0 2024-03-26T18:54:42.821 INFO:tasks.workunit.client.0.smithi102.stdout:3/867: read df/d29/fc7 [733063,130977] 0 2024-03-26T18:54:42.821 INFO:tasks.workunit.client.0.smithi102.stdout:3/868: fdatasync df/d29/d3a/f6d 0 2024-03-26T18:54:42.821 INFO:tasks.workunit.client.0.smithi102.stdout:3/869: chown df/d29/d31/dc0 56979670 1 2024-03-26T18:54:42.822 INFO:tasks.workunit.client.1.smithi190.stdout:5/926: rename d2/d17/d1b/db6/d24/dd1 to d2/d17/d1b/d60/d63/dcd/d118/d120/d139 0 2024-03-26T18:54:42.822 INFO:tasks.workunit.client.1.smithi190.stdout:5/927: chown d2/d17/d1b/d60/d63/d85/ddc/c119 6847 1 2024-03-26T18:54:42.823 INFO:tasks.workunit.client.1.smithi190.stdout:5/928: stat d2/d17/d1b/db6/d24/d3b/c78 0 2024-03-26T18:54:42.823 INFO:tasks.workunit.client.1.smithi190.stdout:1/912: mknod d3/da/c122 0 2024-03-26T18:54:42.824 INFO:tasks.workunit.client.1.smithi190.stdout:3/825: mknod d1/d3/df/cfd 0 2024-03-26T18:54:42.824 INFO:tasks.workunit.client.0.smithi102.stdout:5/907: dwrite d2/d25/d75/db9/d43/d9e/d82/f8f [4194304,4194304] 0 2024-03-26T18:54:42.824 INFO:tasks.workunit.client.0.smithi102.stdout:1/636: truncate de/f8d 240775 0 2024-03-26T18:54:42.830 INFO:tasks.workunit.client.0.smithi102.stdout:1/637: dread de/d16/d1d/d2a/d2c/d56/d5f/f8b [0,4194304] 0 2024-03-26T18:54:42.831 INFO:tasks.workunit.client.0.smithi102.stdout:3/870: unlink df/d29/d31/d36/d45/daf/f11e 0 2024-03-26T18:54:42.832 INFO:tasks.workunit.client.0.smithi102.stdout:3/871: dread df/d61/d55/fe7 [0,4194304] 0 2024-03-26T18:54:42.832 INFO:tasks.workunit.client.0.smithi102.stdout:3/872: chown df/d61/d55/ce2 3942082 1 2024-03-26T18:54:42.833 INFO:tasks.workunit.client.1.smithi190.stdout:5/929: mknod d2/d17/d1b/d60/d63/d102/d12f/c13a 0 2024-03-26T18:54:42.833 INFO:tasks.workunit.client.1.smithi190.stdout:5/930: truncate d2/d17/d62/dd7/ffa 216437 0 2024-03-26T18:54:42.833 INFO:tasks.workunit.client.1.smithi190.stdout:3/826: symlink d1/d3/d5a/d6e/dca/lfe 0 2024-03-26T18:54:42.834 INFO:tasks.workunit.client.1.smithi190.stdout:5/931: truncate d2/d17/d1b/db6/d24/d3b/f46 1465223 0 2024-03-26T18:54:42.836 INFO:tasks.workunit.client.0.smithi102.stdout:5/908: creat d2/d25/d5b/d76/f140 x:0 0 0 2024-03-26T18:54:42.836 INFO:tasks.workunit.client.0.smithi102.stdout:5/909: truncate d2/d25/f2b 9282777 0 2024-03-26T18:54:42.837 INFO:tasks.workunit.client.1.smithi190.stdout:3/827: dread - d1/de/fcc zero size 2024-03-26T18:54:42.837 INFO:tasks.workunit.client.1.smithi190.stdout:3/828: fsync d1/d3/fbf 0 2024-03-26T18:54:42.838 INFO:tasks.workunit.client.0.smithi102.stdout:1/638: creat de/d16/d1d/d2a/d2c/d43/fd4 x:0 0 0 2024-03-26T18:54:42.845 INFO:tasks.workunit.client.0.smithi102.stdout:1/639: dread de/d16/f40 [0,4194304] 0 2024-03-26T18:54:42.858 INFO:tasks.workunit.client.1.smithi190.stdout:1/913: dwrite d3/da/d3a/d99/f121 [0,4194304] 0 2024-03-26T18:54:42.858 INFO:tasks.workunit.client.1.smithi190.stdout:1/914: write d3/da/dd/fb6 [1503311,107423] 0 2024-03-26T18:54:42.858 INFO:tasks.workunit.client.1.smithi190.stdout:1/915: truncate d3/da/daf/f101 529772 0 2024-03-26T18:54:42.858 INFO:tasks.workunit.client.1.smithi190.stdout:1/916: readlink d3/da/dd/d2f/d34/l64 0 2024-03-26T18:54:42.859 INFO:tasks.workunit.client.0.smithi102.stdout:3/873: rename df/d1c/d70/fb6 to df/d61/f13b 0 2024-03-26T18:54:42.859 INFO:tasks.workunit.client.1.smithi190.stdout:3/829: readlink d1/de/lb9 0 2024-03-26T18:54:42.859 INFO:tasks.workunit.client.1.smithi190.stdout:3/830: fsync d1/d3/df/d18/f30 0 2024-03-26T18:54:42.859 INFO:tasks.workunit.client.1.smithi190.stdout:3/831: fdatasync d1/de/d8a/fa7 0 2024-03-26T18:54:42.860 INFO:tasks.workunit.client.0.smithi102.stdout:5/910: mkdir d2/d25/d75/def/dec/d112/d141 0 2024-03-26T18:54:42.866 INFO:tasks.workunit.client.0.smithi102.stdout:1/640: truncate de/d16/d1d/d61/d6a/fa3 699839 0 2024-03-26T18:54:42.866 INFO:tasks.workunit.client.0.smithi102.stdout:1/641: creat de/d16/d1d/d2a/d2c/d56/d5f/daa/fd5 x:0 0 0 2024-03-26T18:54:42.867 INFO:tasks.workunit.client.1.smithi190.stdout:6/834: link d17/d28/f11b d17/dd3/d113/f11c 0 2024-03-26T18:54:42.867 INFO:tasks.workunit.client.1.smithi190.stdout:6/835: readlink d17/d28/d8b/l9f 0 2024-03-26T18:54:42.868 INFO:tasks.workunit.client.0.smithi102.stdout:5/911: read d2/f11 [1541789,22053] 0 2024-03-26T18:54:42.870 INFO:tasks.workunit.client.1.smithi190.stdout:6/836: dread d17/d28/d74/d7b/fa5 [0,4194304] 0 2024-03-26T18:54:42.875 INFO:tasks.workunit.client.1.smithi190.stdout:1/917: fdatasync d3/da/f78 0 2024-03-26T18:54:42.876 INFO:tasks.workunit.client.1.smithi190.stdout:1/918: creat d3/da/d3a/f123 x:0 0 0 2024-03-26T18:54:42.878 INFO:tasks.workunit.client.1.smithi190.stdout:6/837: symlink d17/d28/d2c/db8/df3/l11d 0 2024-03-26T18:54:42.881 INFO:tasks.workunit.client.0.smithi102.stdout:3/874: unlink df/d61/d55/f7c 0 2024-03-26T18:54:42.883 INFO:tasks.workunit.client.1.smithi190.stdout:1/919: mknod d3/da/dd/d2f/d34/d10c/c124 0 2024-03-26T18:54:42.883 INFO:tasks.workunit.client.1.smithi190.stdout:1/920: truncate d3/d9/db/fb4 5099589 0 2024-03-26T18:54:42.885 INFO:tasks.workunit.client.1.smithi190.stdout:6/838: creat d17/d28/de8/f11e x:0 0 0 2024-03-26T18:54:42.887 INFO:tasks.workunit.client.0.smithi102.stdout:1/642: mkdir de/d16/d1d/d61/d6a/da5/dd6 0 2024-03-26T18:54:42.887 INFO:tasks.workunit.client.0.smithi102.stdout:1/643: creat de/d16/d1d/d2a/d2c/d43/fd7 x:0 0 0 2024-03-26T18:54:42.887 INFO:tasks.workunit.client.0.smithi102.stdout:1/644: truncate de/d16/d1d/d2a/d2c/d43/f5a 1046541 0 2024-03-26T18:54:42.888 INFO:tasks.workunit.client.0.smithi102.stdout:3/875: truncate df/d29/d31/d77/d8a/fd2 74288 0 2024-03-26T18:54:42.891 INFO:tasks.workunit.client.0.smithi102.stdout:1/645: creat de/d1f/d9f/fd8 x:0 0 0 2024-03-26T18:54:42.893 INFO:tasks.workunit.client.1.smithi190.stdout:1/921: dread d3/d9/f2a [4194304,4194304] 0 2024-03-26T18:54:42.893 INFO:tasks.workunit.client.1.smithi190.stdout:1/922: write d3/da/f78 [2061666,95805] 0 2024-03-26T18:54:42.893 INFO:tasks.workunit.client.1.smithi190.stdout:1/923: readlink d3/da/dd/d2f/l66 0 2024-03-26T18:54:42.893 INFO:tasks.workunit.client.1.smithi190.stdout:1/924: truncate d3/d5f/f10f 686873 0 2024-03-26T18:54:42.895 INFO:tasks.workunit.client.0.smithi102.stdout:1/646: mkdir de/d16/d1d/d2a/d2c/dd9 0 2024-03-26T18:54:42.896 INFO:tasks.workunit.client.0.smithi102.stdout:1/647: mkdir de/d16/d1d/d61/d6a/da5/da6/dca/dda 0 2024-03-26T18:54:42.898 INFO:tasks.workunit.client.0.smithi102.stdout:5/912: dwrite d2/d25/d75/db9/d43/f92 [0,4194304] 0 2024-03-26T18:54:42.899 INFO:tasks.workunit.client.1.smithi190.stdout:3/832: dwrite d1/d3/df/d18/d35/d41/dae/f7a [0,4194304] 0 2024-03-26T18:54:42.899 INFO:tasks.workunit.client.1.smithi190.stdout:3/833: readlink d1/d3/df/d18/db4/lc9 0 2024-03-26T18:54:42.899 INFO:tasks.workunit.client.1.smithi190.stdout:3/834: write d1/de/f73 [405251,36138] 0 2024-03-26T18:54:42.899 INFO:tasks.workunit.client.1.smithi190.stdout:6/839: dread d17/d28/d2c/d50/f67 [0,4194304] 0 2024-03-26T18:54:42.900 INFO:tasks.workunit.client.1.smithi190.stdout:6/840: chown d17/d28/d2b 24237409 1 2024-03-26T18:54:42.900 INFO:tasks.workunit.client.1.smithi190.stdout:6/841: write d17/d28/df6/da8/fc7 [1035656,22338] 0 2024-03-26T18:54:42.906 INFO:tasks.workunit.client.1.smithi190.stdout:3/835: creat d1/d3/df/d18/d35/d41/d61/fff x:0 0 0 2024-03-26T18:54:42.906 INFO:tasks.workunit.client.1.smithi190.stdout:3/836: fdatasync d1/d3/d5a/fa9 0 2024-03-26T18:54:42.907 INFO:tasks.workunit.client.0.smithi102.stdout:5/913: truncate d2/f64 6585775 0 2024-03-26T18:54:42.907 INFO:tasks.workunit.client.0.smithi102.stdout:5/914: write d2/d25/d75/db9/d43/d9e/daf/f108 [620048,86521] 0 2024-03-26T18:54:42.907 INFO:tasks.workunit.client.0.smithi102.stdout:5/915: readlink d2/d25/d75/db9/d43/d9e/l73 0 2024-03-26T18:54:42.908 INFO:tasks.workunit.client.1.smithi190.stdout:3/837: mkdir d1/d3/df/d18/d35/d41/d61/d100 0 2024-03-26T18:54:42.910 INFO:tasks.workunit.client.0.smithi102.stdout:1/648: read de/d16/d1d/d2a/d67/f95 [3961795,110878] 0 2024-03-26T18:54:42.911 INFO:tasks.workunit.client.0.smithi102.stdout:5/916: creat d2/d25/db0/d139/f142 x:0 0 0 2024-03-26T18:54:42.911 INFO:tasks.workunit.client.0.smithi102.stdout:5/917: readlink d2/d25/l129 0 2024-03-26T18:54:42.913 INFO:tasks.workunit.client.0.smithi102.stdout:1/649: truncate de/d16/d1d/d61/d6a/da5/da6/fb7 2279016 0 2024-03-26T18:54:42.914 INFO:tasks.workunit.client.0.smithi102.stdout:1/650: creat de/d16/d1d/d61/d6a/fdb x:0 0 0 2024-03-26T18:54:42.915 INFO:tasks.workunit.client.0.smithi102.stdout:5/918: unlink d2/dea/dbb/dcf/f13c 0 2024-03-26T18:54:42.916 INFO:tasks.workunit.client.0.smithi102.stdout:3/876: dwrite df/d61/fb0 [0,4194304] 0 2024-03-26T18:54:42.917 INFO:tasks.workunit.client.0.smithi102.stdout:3/877: write df/d29/d3a/d93/f139 [5203799,48152] 0 2024-03-26T18:54:42.917 INFO:tasks.workunit.client.0.smithi102.stdout:3/878: write df/d29/d31/d36/d45/f65 [133554,18324] 0 2024-03-26T18:54:42.917 INFO:tasks.workunit.client.0.smithi102.stdout:3/879: readlink df/d8b/dd5/l117 0 2024-03-26T18:54:42.918 INFO:tasks.workunit.client.1.smithi190.stdout:1/925: dwrite d3/da/dd/f8c [0,4194304] 0 2024-03-26T18:54:42.918 INFO:tasks.workunit.client.1.smithi190.stdout:1/926: read - d3/d73/d88/dcc/f10a zero size 2024-03-26T18:54:42.918 INFO:tasks.workunit.client.1.smithi190.stdout:1/927: creat d3/d5f/f125 x:0 0 0 2024-03-26T18:54:42.918 INFO:tasks.workunit.client.1.smithi190.stdout:1/928: dread - d3/d9/db/f119 zero size 2024-03-26T18:54:42.918 INFO:tasks.workunit.client.1.smithi190.stdout:1/929: creat d3/d8f/da7/f126 x:0 0 0 2024-03-26T18:54:42.918 INFO:tasks.workunit.client.1.smithi190.stdout:6/842: dread d17/d28/d8b/d101/f103 [0,4194304] 0 2024-03-26T18:54:42.918 INFO:tasks.workunit.client.1.smithi190.stdout:6/843: fsync d17/d28/d2c/f4c 0 2024-03-26T18:54:42.918 INFO:tasks.workunit.client.1.smithi190.stdout:6/844: creat d17/d28/d8b/f11f x:0 0 0 2024-03-26T18:54:42.924 INFO:tasks.workunit.client.1.smithi190.stdout:1/930: truncate d3/da/d3a/f50 467490 0 2024-03-26T18:54:42.924 INFO:tasks.workunit.client.0.smithi102.stdout:1/651: dread de/d16/f35 [0,4194304] 0 2024-03-26T18:54:42.926 INFO:tasks.workunit.client.0.smithi102.stdout:3/880: symlink df/d29/d3a/d93/d120/l13c 0 2024-03-26T18:54:42.928 INFO:tasks.workunit.client.1.smithi190.stdout:6/845: symlink d17/d28/d74/d7b/l120 0 2024-03-26T18:54:42.929 INFO:tasks.workunit.client.0.smithi102.stdout:5/919: link d2/d25/d75/db9/d43/d9e/d8b/lb1 d2/d25/de6/l143 0 2024-03-26T18:54:42.930 INFO:tasks.workunit.client.0.smithi102.stdout:5/920: dread d2/dea/dbb/dcf/fda [0,4194304] 0 2024-03-26T18:54:42.933 INFO:tasks.workunit.client.1.smithi190.stdout:6/846: dread d17/d28/d2c/f53 [0,4194304] 0 2024-03-26T18:54:42.934 INFO:tasks.workunit.client.0.smithi102.stdout:1/652: mkdir de/d16/d1d/d2a/d2c/d56/d78/d80/ddc 0 2024-03-26T18:54:42.936 INFO:tasks.workunit.client.0.smithi102.stdout:5/921: creat d2/d25/d75/db9/d43/d9e/d8b/f144 x:0 0 0 2024-03-26T18:54:42.938 INFO:tasks.workunit.client.0.smithi102.stdout:5/922: mknod d2/dea/db4/d11f/c145 0 2024-03-26T18:54:42.938 INFO:tasks.workunit.client.0.smithi102.stdout:5/923: read d2/d25/d75/db9/d43/d9e/daf/f108 [360981,38958] 0 2024-03-26T18:54:42.944 INFO:tasks.workunit.client.1.smithi190.stdout:6/847: dread d17/d28/df6/f88 [0,4194304] 0 2024-03-26T18:54:42.944 INFO:tasks.workunit.client.1.smithi190.stdout:6/848: chown d17/d28/d2c/db8/dcc 13509644 1 2024-03-26T18:54:42.944 INFO:tasks.workunit.client.1.smithi190.stdout:6/849: write d17/d28/df6/da8/fc7 [197387,35620] 0 2024-03-26T18:54:42.945 INFO:tasks.workunit.client.0.smithi102.stdout:5/924: creat d2/d25/d5b/f146 x:0 0 0 2024-03-26T18:54:42.946 INFO:tasks.workunit.client.1.smithi190.stdout:1/931: dwrite d3/d73/d88/df1/ff7 [0,4194304] 0 2024-03-26T18:54:42.946 INFO:tasks.workunit.client.1.smithi190.stdout:1/932: write d3/d5f/f71 [3042814,27218] 0 2024-03-26T18:54:42.946 INFO:tasks.workunit.client.1.smithi190.stdout:1/933: write d3/da/dd/d2f/d34/f107 [843363,11334] 0 2024-03-26T18:54:42.947 INFO:tasks.workunit.client.1.smithi190.stdout:1/934: read d3/d73/d88/f104 [51408,95507] 0 2024-03-26T18:54:42.947 INFO:tasks.workunit.client.1.smithi190.stdout:6/850: truncate d17/d28/d2c/db8/f54 1442007 0 2024-03-26T18:54:42.947 INFO:tasks.workunit.client.1.smithi190.stdout:6/851: fdatasync d17/f21 0 2024-03-26T18:54:42.947 INFO:tasks.workunit.client.1.smithi190.stdout:6/852: write d17/d28/d2c/f41 [6948497,91655] 0 2024-03-26T18:54:42.947 INFO:tasks.workunit.client.1.smithi190.stdout:6/853: write d17/d28/df6/fd8 [930483,51034] 0 2024-03-26T18:54:42.948 INFO:tasks.workunit.client.0.smithi102.stdout:3/881: dwrite df/d61/dd6/fe6 [0,4194304] 0 2024-03-26T18:54:42.954 INFO:tasks.workunit.client.0.smithi102.stdout:3/882: rmdir df/d29/d3a/d93/d120 39 2024-03-26T18:54:42.956 INFO:tasks.workunit.client.0.smithi102.stdout:5/925: dread d2/d14/d1a/f41 [0,4194304] 0 2024-03-26T18:54:42.957 INFO:tasks.workunit.client.0.smithi102.stdout:5/926: read d2/d25/d5b/f94 [262090,120373] 0 2024-03-26T18:54:42.957 INFO:tasks.workunit.client.0.smithi102.stdout:3/883: unlink f7 0 2024-03-26T18:54:42.957 INFO:tasks.workunit.client.0.smithi102.stdout:3/884: chown df/d29/l2b 247733 1 2024-03-26T18:54:42.959 INFO:tasks.workunit.client.1.smithi190.stdout:6/854: mkdir d17/d28/df6/da8/df4/dfb/d121 0 2024-03-26T18:54:42.960 INFO:tasks.workunit.client.1.smithi190.stdout:3/838: dwrite d1/de/d4e/fa0 [0,4194304] 0 2024-03-26T18:54:42.960 INFO:tasks.workunit.client.1.smithi190.stdout:3/839: creat d1/d3/d5a/f101 x:0 0 0 2024-03-26T18:54:42.960 INFO:tasks.workunit.client.0.smithi102.stdout:3/885: mknod df/d8b/dbb/c13d 0 2024-03-26T18:54:42.961 INFO:tasks.workunit.client.1.smithi190.stdout:1/935: getdents d3/d73/d88 0 2024-03-26T18:54:42.965 INFO:tasks.workunit.client.0.smithi102.stdout:3/886: rmdir df/d29/d31/d77/d8a 39 2024-03-26T18:54:42.965 INFO:tasks.workunit.client.0.smithi102.stdout:3/887: write df/d29/d31/d36/d45/daf/d105/f12c [546936,26149] 0 2024-03-26T18:54:42.965 INFO:tasks.workunit.client.0.smithi102.stdout:3/888: creat df/d8b/dd5/f13e x:0 0 0 2024-03-26T18:54:42.968 INFO:tasks.workunit.client.0.smithi102.stdout:1/653: rmdir de/d16/d1d 39 2024-03-26T18:54:42.976 INFO:tasks.workunit.client.0.smithi102.stdout:3/889: write df/d29/d31/d36/d3c/f3f [2231478,86123] 0 2024-03-26T18:54:42.976 INFO:tasks.workunit.client.1.smithi190.stdout:3/840: dread d1/de/f73 [0,4194304] 0 2024-03-26T18:54:42.980 INFO:tasks.workunit.client.0.smithi102.stdout:5/927: dwrite d2/d25/d75/def/dec/d112/f12a [0,4194304] 0 2024-03-26T18:54:42.980 INFO:tasks.workunit.client.0.smithi102.stdout:3/890: truncate df/d29/f64 2754853 0 2024-03-26T18:54:42.990 INFO:tasks.workunit.client.0.smithi102.stdout:5/928: truncate d2/d14/f113 2038141 0 2024-03-26T18:54:42.990 INFO:tasks.workunit.client.0.smithi102.stdout:5/929: write d2/d25/d75/db9/d43/fdb [397135,81687] 0 2024-03-26T18:54:42.991 INFO:tasks.workunit.client.0.smithi102.stdout:3/891: rename df/d29/f8c to df/d29/d31/d77/d8a/f13f 0 2024-03-26T18:54:42.991 INFO:tasks.workunit.client.0.smithi102.stdout:3/892: chown df/d29/d31/d36/f7f 598932918 1 2024-03-26T18:54:42.992 INFO:tasks.workunit.client.0.smithi102.stdout:5/930: getdents d2/dea/dbb 0 2024-03-26T18:54:42.994 INFO:tasks.workunit.client.0.smithi102.stdout:3/893: symlink df/d29/d31/d77/l140 0 2024-03-26T18:54:42.994 INFO:tasks.workunit.client.0.smithi102.stdout:3/894: fdatasync df/d61/f58 0 2024-03-26T18:54:42.995 INFO:tasks.workunit.client.0.smithi102.stdout:5/931: mkdir d2/d25/db0/d147 0 2024-03-26T18:54:42.995 INFO:tasks.workunit.client.0.smithi102.stdout:5/932: fsync d2/f17 0 2024-03-26T18:54:42.997 INFO:tasks.workunit.client.0.smithi102.stdout:5/933: unlink d2/d25/d75/db9/db3/lbc 0 2024-03-26T18:54:42.997 INFO:tasks.workunit.client.1.smithi190.stdout:5/932: sync 2024-03-26T18:54:42.997 INFO:tasks.workunit.client.1.smithi190.stdout:5/933: chown d2/f16 393750 1 2024-03-26T18:54:42.998 INFO:tasks.workunit.client.0.smithi102.stdout:3/895: getdents df/d8b/dd5 0 2024-03-26T18:54:43.000 INFO:tasks.workunit.client.1.smithi190.stdout:5/934: unlink d2/d17/d1b/db6/d88/f8b 0 2024-03-26T18:54:43.002 INFO:tasks.workunit.client.0.smithi102.stdout:3/896: unlink df/d29/d3a/d93/f10a 0 2024-03-26T18:54:43.002 INFO:tasks.workunit.client.0.smithi102.stdout:3/897: write df/d1c/f81 [134453,81939] 0 2024-03-26T18:54:43.003 INFO:tasks.workunit.client.0.smithi102.stdout:5/934: getdents d2/d25/d75/def/dec/d112 0 2024-03-26T18:54:43.004 INFO:tasks.workunit.client.0.smithi102.stdout:2/656: sync 2024-03-26T18:54:43.004 INFO:tasks.workunit.client.0.smithi102.stdout:7/578: sync 2024-03-26T18:54:43.004 INFO:tasks.workunit.client.0.smithi102.stdout:2/657: chown d3/f1c 13533842 1 2024-03-26T18:54:43.005 INFO:tasks.workunit.client.0.smithi102.stdout:3/898: mknod df/d1c/d70/dcd/df7/c141 0 2024-03-26T18:54:43.006 INFO:tasks.workunit.client.0.smithi102.stdout:3/899: rename df/d1c/d70/dcd/df7 to df/d1c/d70/dcd/df7/d142 22 2024-03-26T18:54:43.006 INFO:tasks.workunit.client.0.smithi102.stdout:3/900: getdents df/d61/de5/d128 0 2024-03-26T18:54:43.006 INFO:tasks.workunit.client.0.smithi102.stdout:3/901: write df/fdc [342409,115820] 0 2024-03-26T18:54:43.010 INFO:tasks.workunit.client.1.smithi190.stdout:3/841: dwrite d1/de/f69 [0,4194304] 0 2024-03-26T18:54:43.010 INFO:tasks.workunit.client.1.smithi190.stdout:3/842: creat d1/d3/df/f102 x:0 0 0 2024-03-26T18:54:43.010 INFO:tasks.workunit.client.1.smithi190.stdout:3/843: fsync d1/de/d1c/d36/f58 0 2024-03-26T18:54:43.011 INFO:tasks.workunit.client.1.smithi190.stdout:3/844: chown d1/de/d1c 622091 1 2024-03-26T18:54:43.011 INFO:tasks.workunit.client.0.smithi102.stdout:5/935: unlink d2/d25/d75/db9/d43/f92 0 2024-03-26T18:54:43.011 INFO:tasks.workunit.client.0.smithi102.stdout:5/936: read d2/d25/d75/db9/d43/d9e/d8b/fab [1181124,22873] 0 2024-03-26T18:54:43.011 INFO:tasks.workunit.client.0.smithi102.stdout:1/654: dwrite de/d16/d1d/d2a/d2c/d43/f5a [0,4194304] 0 2024-03-26T18:54:43.012 INFO:tasks.workunit.client.0.smithi102.stdout:1/655: write de/d1f/f57 [5154759,54070] 0 2024-03-26T18:54:43.012 INFO:tasks.workunit.client.0.smithi102.stdout:3/902: symlink df/d1c/d70/dcd/df7/l143 0 2024-03-26T18:54:43.012 INFO:tasks.workunit.client.0.smithi102.stdout:5/937: rmdir d2/d25/d75/def/dec/d112/d141 0 2024-03-26T18:54:43.015 INFO:tasks.workunit.client.1.smithi190.stdout:3/845: creat d1/d3/df/d18/d35/d41/f103 x:0 0 0 2024-03-26T18:54:43.015 INFO:tasks.workunit.client.1.smithi190.stdout:3/846: chown d1/d3/df/d18/d35/d41/d61/fff 17532000 1 2024-03-26T18:54:43.016 INFO:tasks.workunit.client.1.smithi190.stdout:3/847: write d1/d3/df/d18/d35/d41/dae/f93 [373820,118402] 0 2024-03-26T18:54:43.021 INFO:tasks.workunit.client.0.smithi102.stdout:1/656: symlink de/d16/ldd 0 2024-03-26T18:54:43.023 INFO:tasks.workunit.client.0.smithi102.stdout:5/938: dread d2/d25/d75/def/f137 [0,4194304] 0 2024-03-26T18:54:43.025 INFO:tasks.workunit.client.0.smithi102.stdout:3/903: creat df/d61/de5/d128/f144 x:0 0 0 2024-03-26T18:54:43.028 INFO:tasks.workunit.client.0.smithi102.stdout:3/904: mkdir df/d8b/dd5/d145 0 2024-03-26T18:54:43.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:43.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:42 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:43.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:42 smithi190 ceph-mon[35043]: pgmap v249: 65 pgs: 65 active+clean; 3.1 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 298 MiB/s rd, 365 MiB/s wr, 541 op/s 2024-03-26T18:54:43.030 INFO:tasks.workunit.client.0.smithi102.stdout:3/905: creat df/d61/de5/f146 x:0 0 0 2024-03-26T18:54:43.033 INFO:tasks.workunit.client.0.smithi102.stdout:3/906: getdents df/d1c/d34/d53 0 2024-03-26T18:54:43.033 INFO:tasks.workunit.client.0.smithi102.stdout:3/907: fdatasync df/d29/d31/d77/d8a/dae/dee/df2/d100/d11a/f134 0 2024-03-26T18:54:43.037 INFO:tasks.workunit.client.0.smithi102.stdout:1/657: write de/d16/d1d/d2a/fb2 [3889464,12879] 0 2024-03-26T18:54:43.037 INFO:tasks.workunit.client.0.smithi102.stdout:1/658: stat de/d16/d1d/d2a/d2c/d56/d5f/daa/lc1 0 2024-03-26T18:54:43.040 INFO:tasks.workunit.client.0.smithi102.stdout:1/659: creat de/d16/d1d/d2a/d2c/d56/d6e/fde x:0 0 0 2024-03-26T18:54:43.042 INFO:tasks.workunit.client.1.smithi190.stdout:6/855: dwrite d17/d28/d2c/d63/fa0 [0,4194304] 0 2024-03-26T18:54:43.042 INFO:tasks.workunit.client.1.smithi190.stdout:6/856: fdatasync d17/d28/df6/da8/df4/dfb/fba 0 2024-03-26T18:54:43.042 INFO:tasks.workunit.client.1.smithi190.stdout:6/857: chown d17/d28/de8/f11e 7102395 1 2024-03-26T18:54:43.051 INFO:tasks.workunit.client.1.smithi190.stdout:6/858: write d17/d28/d8b/d95/dda/fec [3482302,112649] 0 2024-03-26T18:54:43.055 INFO:tasks.workunit.client.1.smithi190.stdout:6/859: write d17/d28/d2c/d77/f66 [84428,78509] 0 2024-03-26T18:54:43.056 INFO:tasks.workunit.client.1.smithi190.stdout:6/860: stat d17/d28/d8b/d95/caf 0 2024-03-26T18:54:43.058 INFO:tasks.workunit.client.0.smithi102.stdout:2/658: dwrite d3/d2d/d69/fcf [0,4194304] 0 2024-03-26T18:54:43.061 INFO:tasks.workunit.client.0.smithi102.stdout:2/659: link d3/d82/d93/d9e/d1a/d25/l95 d3/d82/lec 0 2024-03-26T18:54:43.061 INFO:tasks.workunit.client.0.smithi102.stdout:2/660: read - d3/d11/d8f/fa2 zero size 2024-03-26T18:54:43.063 INFO:tasks.workunit.client.0.smithi102.stdout:2/661: rename d3/d20/d78/d84/de1 to d3/d11/d15/ded 0 2024-03-26T18:54:43.063 INFO:tasks.workunit.client.0.smithi102.stdout:2/662: fsync d3/d82/d93/fbf 0 2024-03-26T18:54:43.063 INFO:tasks.workunit.client.0.smithi102.stdout:2/663: rmdir d3/d82/d93/d9e/d99/deb 0 2024-03-26T18:54:43.065 INFO:tasks.workunit.client.0.smithi102.stdout:2/664: rename d3/d20/d78/c5d to d3/d2d/cee 0 2024-03-26T18:54:43.066 INFO:tasks.workunit.client.1.smithi190.stdout:6/861: dread d17/f19 [4194304,4194304] 0 2024-03-26T18:54:43.066 INFO:tasks.workunit.client.0.smithi102.stdout:2/665: rename d3/d11/d8f to d3/d82/de2/def 0 2024-03-26T18:54:43.068 INFO:tasks.workunit.client.0.smithi102.stdout:2/666: unlink d3/d82/d93/d9e/d1a/d25/c3b 0 2024-03-26T18:54:43.068 INFO:tasks.workunit.client.0.smithi102.stdout:2/667: creat d3/d20/ff0 x:0 0 0 2024-03-26T18:54:43.070 INFO:tasks.workunit.client.0.smithi102.stdout:7/579: dwrite d7/d12/d4b/d83/f8e [0,4194304] 0 2024-03-26T18:54:43.071 INFO:tasks.workunit.client.1.smithi190.stdout:3/848: dwrite d1/d3/df/f102 [0,4194304] 0 2024-03-26T18:54:43.071 INFO:tasks.workunit.client.0.smithi102.stdout:2/668: rename d3/d11/d15/d31/d39/c8a to d3/d20/cf1 0 2024-03-26T18:54:43.077 INFO:tasks.workunit.client.1.smithi190.stdout:3/849: link d1/d3/df/d18/d35/d41/f103 d1/d3/df/d18/db4/f104 0 2024-03-26T18:54:43.077 INFO:tasks.workunit.client.1.smithi190.stdout:3/850: fdatasync d1/de/d8a/f95 0 2024-03-26T18:54:43.078 INFO:tasks.workunit.client.0.smithi102.stdout:2/669: creat d3/d20/d78/d7c/dca/ff2 x:0 0 0 2024-03-26T18:54:43.078 INFO:tasks.workunit.client.0.smithi102.stdout:2/670: write d3/d20/d78/d7c/f85 [3764039,71037] 0 2024-03-26T18:54:43.080 INFO:tasks.workunit.client.0.smithi102.stdout:2/671: mknod d3/d82/d93/d9e/da0/cf3 0 2024-03-26T18:54:43.080 INFO:tasks.workunit.client.0.smithi102.stdout:2/672: write d3/d82/d93/fcb [120684,27333] 0 2024-03-26T18:54:43.081 INFO:tasks.workunit.client.0.smithi102.stdout:2/673: creat d3/d82/de2/def/ff4 x:0 0 0 2024-03-26T18:54:43.085 INFO:tasks.workunit.client.0.smithi102.stdout:3/908: dwrite df/d1c/f2a [0,4194304] 0 2024-03-26T18:54:43.087 INFO:tasks.workunit.client.0.smithi102.stdout:3/909: truncate df/d29/d31/dc0/fe8 796081 0 2024-03-26T18:54:43.091 INFO:tasks.workunit.client.0.smithi102.stdout:1/660: dwrite de/f3e [4194304,4194304] 0 2024-03-26T18:54:43.091 INFO:tasks.workunit.client.0.smithi102.stdout:2/674: dread d3/d2a/f2f [0,4194304] 0 2024-03-26T18:54:43.093 INFO:tasks.workunit.client.1.smithi190.stdout:6/862: dwrite d17/d28/d2c/db8/df3/d109/d79/fd1 [0,4194304] 0 2024-03-26T18:54:43.093 INFO:tasks.workunit.client.0.smithi102.stdout:1/661: creat de/d16/d1d/d2a/d2c/dd9/fdf x:0 0 0 2024-03-26T18:54:43.094 INFO:tasks.workunit.client.0.smithi102.stdout:1/662: stat de/d16/d1d/d2a/d67/f95 0 2024-03-26T18:54:43.095 INFO:tasks.workunit.client.0.smithi102.stdout:5/939: dwrite d2/dea/f103 [0,4194304] 0 2024-03-26T18:54:43.097 INFO:tasks.workunit.client.0.smithi102.stdout:1/663: mkdir de/d16/d1d/d2a/d2c/d56/d78/de0 0 2024-03-26T18:54:43.100 INFO:tasks.workunit.client.1.smithi190.stdout:6/863: creat d17/d28/d2c/d50/df7/f122 x:0 0 0 2024-03-26T18:54:43.103 INFO:tasks.workunit.client.1.smithi190.stdout:6/864: rename f9 to d17/d28/df6/da8/df4/dfb/d6a/f123 0 2024-03-26T18:54:43.104 INFO:tasks.workunit.client.1.smithi190.stdout:6/865: fdatasync d17/d28/d2c/db8/df3/d109/fb1 0 2024-03-26T18:54:43.104 INFO:tasks.workunit.client.0.smithi102.stdout:3/910: dread df/d1c/f2a [0,4194304] 0 2024-03-26T18:54:43.108 INFO:tasks.workunit.client.0.smithi102.stdout:1/664: write de/d16/d1d/f1e [3822434,42057] 0 2024-03-26T18:54:43.108 INFO:tasks.workunit.client.0.smithi102.stdout:1/665: write de/d1f/d9f/fd8 [575989,46452] 0 2024-03-26T18:54:43.109 INFO:tasks.workunit.client.1.smithi190.stdout:6/866: symlink d17/d28/df6/l124 0 2024-03-26T18:54:43.109 INFO:tasks.workunit.client.1.smithi190.stdout:6/867: creat d17/d28/d8b/d95/f125 x:0 0 0 2024-03-26T18:54:43.111 INFO:tasks.workunit.client.0.smithi102.stdout:7/580: dwrite d7/d2a/f30 [0,4194304] 0 2024-03-26T18:54:43.112 INFO:tasks.workunit.client.1.smithi190.stdout:3/851: dwrite d1/de/d4e/f8f [4194304,4194304] 0 2024-03-26T18:54:43.122 INFO:tasks.workunit.client.0.smithi102.stdout:2/675: rmdir d3/d11/d15/d31 39 2024-03-26T18:54:43.123 INFO:tasks.workunit.client.1.smithi190.stdout:3/852: dread d1/f15 [0,4194304] 0 2024-03-26T18:54:43.124 INFO:tasks.workunit.client.0.smithi102.stdout:1/666: read de/d16/d1d/d2a/d2c/d56/d78/f79 [2270242,28638] 0 2024-03-26T18:54:43.124 INFO:tasks.workunit.client.0.smithi102.stdout:1/667: chown de/d16/d1d/d2a/d2c/d56/d78/dc3 382 1 2024-03-26T18:54:43.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:43.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:42 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:43.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:42 smithi102 ceph-mon[29002]: pgmap v249: 65 pgs: 65 active+clean; 3.1 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 298 MiB/s rd, 365 MiB/s wr, 541 op/s 2024-03-26T18:54:43.126 INFO:tasks.workunit.client.1.smithi190.stdout:3/853: creat d1/d3/df/d18/db4/f105 x:0 0 0 2024-03-26T18:54:43.129 INFO:tasks.workunit.client.1.smithi190.stdout:3/854: rename d1/d3/df/d18/d35/d41/dae/d9c/fa6 to d1/d3/df/d31/dc4/f106 0 2024-03-26T18:54:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:5/940: truncate d2/d14/ff6 3569067 0 2024-03-26T18:54:43.131 INFO:tasks.workunit.client.0.smithi102.stdout:3/911: unlink df/d1c/f69 0 2024-03-26T18:54:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:2/676: creat d3/d82/de2/ff5 x:0 0 0 2024-03-26T18:54:43.133 INFO:tasks.workunit.client.1.smithi190.stdout:3/855: dread d1/d3/df/d18/d35/f60 [0,4194304] 0 2024-03-26T18:54:43.133 INFO:tasks.workunit.client.0.smithi102.stdout:1/668: getdents de/d16/d1d/d2a/d67 0 2024-03-26T18:54:43.133 INFO:tasks.workunit.client.0.smithi102.stdout:1/669: fsync de/d16/f40 0 2024-03-26T18:54:43.134 INFO:tasks.workunit.client.0.smithi102.stdout:5/941: dread d2/d25/d5b/d72/f55 [0,4194304] 0 2024-03-26T18:54:43.134 INFO:tasks.workunit.client.0.smithi102.stdout:3/912: symlink df/d1c/d34/da5/dce/l147 0 2024-03-26T18:54:43.134 INFO:tasks.workunit.client.0.smithi102.stdout:3/913: truncate df/d29/d31/f108 543642 0 2024-03-26T18:54:43.136 INFO:tasks.workunit.client.0.smithi102.stdout:2/677: creat d3/d82/de2/def/ff6 x:0 0 0 2024-03-26T18:54:43.140 INFO:tasks.workunit.client.0.smithi102.stdout:5/942: truncate d2/d14/d1a/f41 3516381 0 2024-03-26T18:54:43.140 INFO:tasks.workunit.client.0.smithi102.stdout:5/943: readlink d2/d25/d75/db9/d43/d9e/l13e 0 2024-03-26T18:54:43.143 INFO:tasks.workunit.client.0.smithi102.stdout:7/581: dwrite d7/d10/d17/f1b [4194304,4194304] 0 2024-03-26T18:54:43.144 INFO:tasks.workunit.client.1.smithi190.stdout:6/868: dwrite d17/d28/d8b/d95/fd2 [0,4194304] 0 2024-03-26T18:54:43.144 INFO:tasks.workunit.client.1.smithi190.stdout:6/869: creat d17/d28/d74/d7b/f126 x:0 0 0 2024-03-26T18:54:43.145 INFO:tasks.workunit.client.0.smithi102.stdout:5/944: rename d2/c8 to d2/d25/d75/db9/d43/d9e/d82/dc9/c148 0 2024-03-26T18:54:43.149 INFO:tasks.workunit.client.0.smithi102.stdout:5/945: chown d2/d2c/f8d 36 1 2024-03-26T18:54:43.149 INFO:tasks.workunit.client.0.smithi102.stdout:5/946: truncate d2/d25/d5b/d76/f9a 316971 0 2024-03-26T18:54:43.151 INFO:tasks.workunit.client.1.smithi190.stdout:6/870: mkdir d17/d28/d8b/d95/dda/d127 0 2024-03-26T18:54:43.151 INFO:tasks.workunit.client.1.smithi190.stdout:6/871: dread - d17/d28/df6/da8/df4/f117 zero size 2024-03-26T18:54:43.153 INFO:tasks.workunit.client.0.smithi102.stdout:5/947: mknod d2/d25/de6/c149 0 2024-03-26T18:54:43.153 INFO:tasks.workunit.client.0.smithi102.stdout:5/948: chown d2/d25/d75/db9/d43/d9e/d82/c10c 128784 1 2024-03-26T18:54:43.153 INFO:tasks.workunit.client.0.smithi102.stdout:9/619: sync 2024-03-26T18:54:43.154 INFO:tasks.workunit.client.0.smithi102.stdout:6/667: sync 2024-03-26T18:54:43.154 INFO:tasks.workunit.client.0.smithi102.stdout:4/543: sync 2024-03-26T18:54:43.155 INFO:tasks.workunit.client.1.smithi190.stdout:6/872: mkdir d17/d28/d8b/d101/d128 0 2024-03-26T18:54:43.155 INFO:tasks.workunit.client.1.smithi190.stdout:6/873: fsync d17/d28/df6/da8/fc7 0 2024-03-26T18:54:43.156 INFO:tasks.workunit.client.1.smithi190.stdout:1/936: sync 2024-03-26T18:54:43.156 INFO:tasks.workunit.client.1.smithi190.stdout:5/935: sync 2024-03-26T18:54:43.156 INFO:tasks.workunit.client.0.smithi102.stdout:9/620: symlink df/d1a/d23/d29/le2 0 2024-03-26T18:54:43.157 INFO:tasks.workunit.client.0.smithi102.stdout:8/714: sync 2024-03-26T18:54:43.157 INFO:tasks.workunit.client.0.smithi102.stdout:0/712: sync 2024-03-26T18:54:43.158 INFO:tasks.workunit.client.1.smithi190.stdout:6/874: symlink d17/d28/d8b/d95/l129 0 2024-03-26T18:54:43.159 INFO:tasks.workunit.client.0.smithi102.stdout:6/668: fsync d2/d12/f3f 0 2024-03-26T18:54:43.162 INFO:tasks.workunit.client.1.smithi190.stdout:5/936: mknod d2/d17/d1b/d60/d63/dcd/d118/d120/d139/d12c/c13b 0 2024-03-26T18:54:43.162 INFO:tasks.workunit.client.0.smithi102.stdout:6/669: dread d2/d7/d3c/f41 [0,4194304] 0 2024-03-26T18:54:43.162 INFO:tasks.workunit.client.0.smithi102.stdout:6/670: stat d2/d3d/lcf 0 2024-03-26T18:54:43.162 INFO:tasks.workunit.client.0.smithi102.stdout:6/671: dread - d2/df/d35/da1/fa5 zero size 2024-03-26T18:54:43.162 INFO:tasks.workunit.client.0.smithi102.stdout:6/672: write d2/df/f8e [1081453,101777] 0 2024-03-26T18:54:43.164 INFO:tasks.workunit.client.0.smithi102.stdout:9/621: creat df/d1a/d1d/d72/d7d/db7/dd5/fe3 x:0 0 0 2024-03-26T18:54:43.164 INFO:tasks.workunit.client.0.smithi102.stdout:9/622: creat df/d1a/d23/d4e/db0/fe4 x:0 0 0 2024-03-26T18:54:43.164 INFO:tasks.workunit.client.0.smithi102.stdout:9/623: chown df/d1a/d23/d60/f74 3843321 1 2024-03-26T18:54:43.167 INFO:tasks.workunit.client.0.smithi102.stdout:8/715: mknod d0/d15/cec 0 2024-03-26T18:54:43.169 INFO:tasks.workunit.client.0.smithi102.stdout:0/713: creat d6/de/d13/d25/d31/d45/d82/ff3 x:0 0 0 2024-03-26T18:54:43.170 INFO:tasks.workunit.client.0.smithi102.stdout:6/673: unlink d2/df/d35/d43/f85 0 2024-03-26T18:54:43.170 INFO:tasks.workunit.client.1.smithi190.stdout:1/937: write d3/d73/d88/df1/f108 [2620702,19882] 0 2024-03-26T18:54:43.172 INFO:tasks.workunit.client.0.smithi102.stdout:4/544: link d1/d17/d1a/d21/l72 d1/la9 0 2024-03-26T18:54:43.172 INFO:tasks.workunit.client.0.smithi102.stdout:4/545: chown d1/d17/d2a/f61 14435844 1 2024-03-26T18:54:43.174 INFO:tasks.workunit.client.0.smithi102.stdout:5/949: write d2/dea/f59 [3294675,101632] 0 2024-03-26T18:54:43.174 INFO:tasks.workunit.client.0.smithi102.stdout:2/678: dwrite d3/d11/d15/d31/d7f/fb9 [0,4194304] 0 2024-03-26T18:54:43.174 INFO:tasks.workunit.client.0.smithi102.stdout:5/950: fsync d2/d14/f104 0 2024-03-26T18:54:43.174 INFO:tasks.workunit.client.0.smithi102.stdout:2/679: chown d3/d11/d15/d31 8 1 2024-03-26T18:54:43.175 INFO:tasks.workunit.client.1.smithi190.stdout:3/856: dwrite d1/d3/df/d18/d35/d41/f72 [0,4194304] 0 2024-03-26T18:54:43.175 INFO:tasks.workunit.client.1.smithi190.stdout:3/857: read - d1/de/d8a/fc2 zero size 2024-03-26T18:54:43.175 INFO:tasks.workunit.client.1.smithi190.stdout:6/875: dread d17/dd3/fef [0,4194304] 0 2024-03-26T18:54:43.178 INFO:tasks.workunit.client.1.smithi190.stdout:1/938: rename d3/d73/d88/df1 to d3/d8f/d127 0 2024-03-26T18:54:43.179 INFO:tasks.workunit.client.0.smithi102.stdout:1/670: dwrite de/d16/d1d/d2a/d2c/d56/d6e/f9a [0,4194304] 0 2024-03-26T18:54:43.182 INFO:tasks.workunit.client.0.smithi102.stdout:0/714: rename d6/c78 to d6/de/d13/d9a/d85/cf4 0 2024-03-26T18:54:43.184 INFO:tasks.workunit.client.1.smithi190.stdout:3/858: unlink d1/de/d1c/c59 0 2024-03-26T18:54:43.185 INFO:tasks.workunit.client.1.smithi190.stdout:5/937: dwrite d2/d17/d98/d3a/faa [0,4194304] 0 2024-03-26T18:54:43.186 INFO:tasks.workunit.client.0.smithi102.stdout:6/674: creat d2/df/d72/db7/fda x:0 0 0 2024-03-26T18:54:43.187 INFO:tasks.workunit.client.1.smithi190.stdout:6/876: mknod d17/d28/d74/d97/c12a 0 2024-03-26T18:54:43.189 INFO:tasks.workunit.client.0.smithi102.stdout:4/546: truncate d1/d17/d1a/d21/f63 375965 0 2024-03-26T18:54:43.195 INFO:tasks.workunit.client.0.smithi102.stdout:2/680: dwrite d3/d20/d78/d84/fda [0,4194304] 0 2024-03-26T18:54:43.196 INFO:tasks.workunit.client.1.smithi190.stdout:3/859: dread d1/f11 [0,4194304] 0 2024-03-26T18:54:43.196 INFO:tasks.workunit.client.1.smithi190.stdout:3/860: chown d1/d3/d52/d70/db3 130262 1 2024-03-26T18:54:43.196 INFO:tasks.workunit.client.1.smithi190.stdout:5/938: creat d2/d17/d1b/db6/d96/d131/f13c x:0 0 0 2024-03-26T18:54:43.199 INFO:tasks.workunit.client.1.smithi190.stdout:6/877: unlink d17/d28/d2c/d50/fe9 0 2024-03-26T18:54:43.199 INFO:tasks.workunit.client.1.smithi190.stdout:6/878: chown d17/d28/df6/da8/df4/dfb/f56 52198879 1 2024-03-26T18:54:43.199 INFO:tasks.workunit.client.0.smithi102.stdout:8/716: dwrite d0/d3c/d96/d98/fe4 [0,4194304] 0 2024-03-26T18:54:43.201 INFO:tasks.workunit.client.0.smithi102.stdout:9/624: link df/c14 df/d1a/d23/d60/ce5 0 2024-03-26T18:54:43.203 INFO:tasks.workunit.client.1.smithi190.stdout:1/939: rename d3/d8f/ff6 to d3/da/f128 0 2024-03-26T18:54:43.205 INFO:tasks.workunit.client.0.smithi102.stdout:4/547: dread d1/d17/d1a/d21/f64 [0,4194304] 0 2024-03-26T18:54:43.205 INFO:tasks.workunit.client.1.smithi190.stdout:3/861: mknod d1/d3/df/d31/dd1/c107 0 2024-03-26T18:54:43.206 INFO:tasks.workunit.client.0.smithi102.stdout:6/675: creat d2/df/d71/fdb x:0 0 0 2024-03-26T18:54:43.217 INFO:tasks.workunit.client.1.smithi190.stdout:3/862: unlink d1/d3/df/d18/d1b/fb1 0 2024-03-26T18:54:43.217 INFO:tasks.workunit.client.1.smithi190.stdout:3/863: chown d1/d3/df/d18/d35/ca1 632861 1 2024-03-26T18:54:43.220 INFO:tasks.workunit.client.1.smithi190.stdout:5/939: rename d2/d17/cff to d2/d17/d98/c13d 0 2024-03-26T18:54:43.222 INFO:tasks.workunit.client.0.smithi102.stdout:2/681: rmdir d3/d20/d78 39 2024-03-26T18:54:43.222 INFO:tasks.workunit.client.1.smithi190.stdout:6/879: dwrite d17/d28/d8b/d95/fd2 [0,4194304] 0 2024-03-26T18:54:43.223 INFO:tasks.workunit.client.0.smithi102.stdout:9/625: dwrite df/d1a/d65/f73 [0,4194304] 0 2024-03-26T18:54:43.225 INFO:tasks.workunit.client.1.smithi190.stdout:3/864: link d1/de/d1c/c4a d1/d3/df/d18/d35/d41/dae/d9c/c108 0 2024-03-26T18:54:43.225 INFO:tasks.workunit.client.1.smithi190.stdout:3/865: readlink d1/d3/df/d18/d35/lcb 0 2024-03-26T18:54:43.227 INFO:tasks.workunit.client.1.smithi190.stdout:5/940: rename d2/d17/d62/d64/fbf to d2/d17/d1b/db6/d88/f13e 0 2024-03-26T18:54:43.229 INFO:tasks.workunit.client.1.smithi190.stdout:3/866: symlink d1/d3/df/d18/d35/d41/dae/d86/l109 0 2024-03-26T18:54:43.229 INFO:tasks.workunit.client.1.smithi190.stdout:3/867: fdatasync d1/d3/df/d18/d57/d91/d8c/faf 0 2024-03-26T18:54:43.233 INFO:tasks.workunit.client.1.smithi190.stdout:3/868: mkdir d1/d3/df/d10a 0 2024-03-26T18:54:43.233 INFO:tasks.workunit.client.1.smithi190.stdout:3/869: write d1/d3/df/d31/ff1 [164869,91432] 0 2024-03-26T18:54:43.235 INFO:tasks.workunit.client.1.smithi190.stdout:6/880: dread d17/d28/d8b/d95/ffe [4194304,4194304] 0 2024-03-26T18:54:43.236 INFO:tasks.workunit.client.1.smithi190.stdout:1/940: dwrite d3/da/f47 [0,4194304] 0 2024-03-26T18:54:43.236 INFO:tasks.workunit.client.1.smithi190.stdout:1/941: creat d3/da/d2e/d59/dae/f129 x:0 0 0 2024-03-26T18:54:43.237 INFO:tasks.workunit.client.0.smithi102.stdout:9/626: dread df/d1a/d23/d4e/d8e/f7b [0,4194304] 0 2024-03-26T18:54:43.239 INFO:tasks.workunit.client.1.smithi190.stdout:3/870: mkdir d1/dd3/d10b 0 2024-03-26T18:54:43.239 INFO:tasks.workunit.client.1.smithi190.stdout:3/871: creat d1/d3/d52/d70/db3/f10c x:0 0 0 2024-03-26T18:54:43.239 INFO:tasks.workunit.client.1.smithi190.stdout:3/872: creat d1/d3/df/d31/d5b/f10d x:0 0 0 2024-03-26T18:54:43.241 INFO:tasks.workunit.client.1.smithi190.stdout:6/881: mkdir d17/d28/df6/da8/d12b 0 2024-03-26T18:54:43.245 INFO:tasks.workunit.client.1.smithi190.stdout:5/941: dread d2/d17/d1b/d30/f3f [0,4194304] 0 2024-03-26T18:54:43.245 INFO:tasks.workunit.client.1.smithi190.stdout:5/942: write d2/f93 [1566226,130262] 0 2024-03-26T18:54:43.246 INFO:tasks.workunit.client.0.smithi102.stdout:8/717: rename d0/d15/d5d/dd3/l69 to d0/d15/de6/led 0 2024-03-26T18:54:43.247 INFO:tasks.workunit.client.0.smithi102.stdout:0/715: symlink d6/de/d13/d1f/lf5 0 2024-03-26T18:54:43.249 INFO:tasks.workunit.client.1.smithi190.stdout:1/942: link d3/d8f/d127/lfd d3/da/dd/d95/l12a 0 2024-03-26T18:54:43.249 INFO:tasks.workunit.client.1.smithi190.stdout:1/943: dread - d3/da/dd/d95/ff0 zero size 2024-03-26T18:54:43.249 INFO:tasks.workunit.client.1.smithi190.stdout:1/944: dread - d3/da/dd/d95/ff0 zero size 2024-03-26T18:54:43.249 INFO:tasks.workunit.client.1.smithi190.stdout:1/945: readlink d3/d9/l10 0 2024-03-26T18:54:43.250 INFO:tasks.workunit.client.1.smithi190.stdout:1/946: creat d3/da/dd/d2f/d33/d115/f12b x:0 0 0 2024-03-26T18:54:43.251 INFO:tasks.workunit.client.0.smithi102.stdout:4/548: symlink d1/d17/laa 0 2024-03-26T18:54:43.252 INFO:tasks.workunit.client.0.smithi102.stdout:5/951: dwrite d2/d25/d75/def/dec/d112/f7a [0,4194304] 0 2024-03-26T18:54:43.254 INFO:tasks.workunit.client.0.smithi102.stdout:6/676: creat d2/df/d35/d43/d75/fdc x:0 0 0 2024-03-26T18:54:43.260 INFO:tasks.workunit.client.1.smithi190.stdout:6/882: creat d17/f12c x:0 0 0 2024-03-26T18:54:43.260 INFO:tasks.workunit.client.1.smithi190.stdout:6/883: chown d17/d28/df6/l85 2 1 2024-03-26T18:54:43.262 INFO:tasks.workunit.client.1.smithi190.stdout:5/943: mknod d2/d105/c13f 0 2024-03-26T18:54:43.265 INFO:tasks.workunit.client.0.smithi102.stdout:9/627: rename df/d1a/c1e to df/d84/ce6 0 2024-03-26T18:54:43.265 INFO:tasks.workunit.client.0.smithi102.stdout:9/628: write df/d1a/d23/d60/f98 [4715547,107004] 0 2024-03-26T18:54:43.265 INFO:tasks.workunit.client.0.smithi102.stdout:9/629: fsync df/d1a/f3f 0 2024-03-26T18:54:43.266 INFO:tasks.workunit.client.0.smithi102.stdout:9/630: creat df/d1a/d23/d4e/fe7 x:0 0 0 2024-03-26T18:54:43.266 INFO:tasks.workunit.client.0.smithi102.stdout:9/631: truncate df/d1a/d57/fb1 406300 0 2024-03-26T18:54:43.266 INFO:tasks.workunit.client.1.smithi190.stdout:6/884: mknod d17/d28/d74/c12d 0 2024-03-26T18:54:43.267 INFO:tasks.workunit.client.1.smithi190.stdout:6/885: chown d17/d28/d2c/d77 98 1 2024-03-26T18:54:43.267 INFO:tasks.workunit.client.1.smithi190.stdout:6/886: creat d17/d28/de8/f12e x:0 0 0 2024-03-26T18:54:43.267 INFO:tasks.workunit.client.1.smithi190.stdout:6/887: write d17/d28/d74/d97/f102 [894857,55549] 0 2024-03-26T18:54:43.269 INFO:tasks.workunit.client.0.smithi102.stdout:0/716: link d6/de/d13/d1f/d55/d57/cdc d6/dbe/deb/cf6 0 2024-03-26T18:54:43.269 INFO:tasks.workunit.client.0.smithi102.stdout:0/717: chown d6/de/d13/d1f/d55/ff2 25118 1 2024-03-26T18:54:43.269 INFO:tasks.workunit.client.0.smithi102.stdout:0/718: write d6/f91 [638879,76564] 0 2024-03-26T18:54:43.269 INFO:tasks.workunit.client.0.smithi102.stdout:0/719: fsync d6/de/d13/d25/d4f/d66/f95 0 2024-03-26T18:54:43.270 INFO:tasks.workunit.client.1.smithi190.stdout:5/944: unlink d2/le 0 2024-03-26T18:54:43.270 INFO:tasks.workunit.client.1.smithi190.stdout:5/945: write d2/d17/d98/f2e [1692547,4609] 0 2024-03-26T18:54:43.274 INFO:tasks.workunit.client.1.smithi190.stdout:3/873: dwrite d1/f11 [0,4194304] 0 2024-03-26T18:54:43.274 INFO:tasks.workunit.client.1.smithi190.stdout:3/874: write d1/de/d8a/fc2 [16116,32736] 0 2024-03-26T18:54:43.274 INFO:tasks.workunit.client.1.smithi190.stdout:3/875: fsync d1/de/d81/fc1 0 2024-03-26T18:54:43.274 INFO:tasks.workunit.client.1.smithi190.stdout:1/947: dwrite d3/fab [0,4194304] 0 2024-03-26T18:54:43.274 INFO:tasks.workunit.client.1.smithi190.stdout:1/948: dread - d3/d9/ffa zero size 2024-03-26T18:54:43.275 INFO:tasks.workunit.client.0.smithi102.stdout:6/677: chown d2/f29 441907292 1 2024-03-26T18:54:43.278 INFO:tasks.workunit.client.1.smithi190.stdout:3/876: dread d1/de/f73 [0,4194304] 0 2024-03-26T18:54:43.283 INFO:tasks.workunit.client.1.smithi190.stdout:1/949: creat d3/da/dd/d2f/d34/f12c x:0 0 0 2024-03-26T18:54:43.286 INFO:tasks.workunit.client.1.smithi190.stdout:1/950: creat d3/da/dd/d95/dda/f12d x:0 0 0 2024-03-26T18:54:43.297 INFO:tasks.workunit.client.1.smithi190.stdout:6/888: dwrite d17/d28/d8b/d95/fc6 [0,4194304] 0 2024-03-26T18:54:43.298 INFO:tasks.workunit.client.1.smithi190.stdout:6/889: chown d17/d28/d2c/ccd 1 1 2024-03-26T18:54:43.300 INFO:tasks.workunit.client.0.smithi102.stdout:2/682: rmdir d3 39 2024-03-26T18:54:43.300 INFO:tasks.workunit.client.0.smithi102.stdout:2/683: chown d3/d11/d15/d46/d7b/lbe 0 1 2024-03-26T18:54:43.304 INFO:tasks.workunit.client.0.smithi102.stdout:9/632: link df/d1a/d23/d4e/d8e/f7b df/d1a/d1d/d72/d7d/fe8 0 2024-03-26T18:54:43.304 INFO:tasks.workunit.client.0.smithi102.stdout:9/633: readlink df/d1a/d1d/l96 0 2024-03-26T18:54:43.304 INFO:tasks.workunit.client.0.smithi102.stdout:9/634: write df/d1a/d23/d29/d9c/faa [423430,14482] 0 2024-03-26T18:54:43.304 INFO:tasks.workunit.client.0.smithi102.stdout:9/635: write df/d1a/f86 [650727,44905] 0 2024-03-26T18:54:43.304 INFO:tasks.workunit.client.0.smithi102.stdout:9/636: stat df/d1a/d1d/d72/d7d/db7/ldd 0 2024-03-26T18:54:43.305 INFO:tasks.workunit.client.0.smithi102.stdout:0/720: truncate d6/de/d13/d25/d31/d45/d82/d84/d88/fa6 280992 0 2024-03-26T18:54:43.308 INFO:tasks.workunit.client.0.smithi102.stdout:6/678: creat d2/df/dba/fdd x:0 0 0 2024-03-26T18:54:43.308 INFO:tasks.workunit.client.0.smithi102.stdout:6/679: truncate d2/df/d35/da1/fa5 103963 0 2024-03-26T18:54:43.308 INFO:tasks.workunit.client.0.smithi102.stdout:6/680: write d2/df/f27 [933556,56047] 0 2024-03-26T18:54:43.308 INFO:tasks.workunit.client.0.smithi102.stdout:6/681: write d2/df/f2b [769368,3363] 0 2024-03-26T18:54:43.309 INFO:tasks.workunit.client.0.smithi102.stdout:2/684: creat d3/d2a/ff7 x:0 0 0 2024-03-26T18:54:43.310 INFO:tasks.workunit.client.0.smithi102.stdout:9/637: truncate df/f40 2447843 0 2024-03-26T18:54:43.311 INFO:tasks.workunit.client.1.smithi190.stdout:6/890: dread d17/f24 [0,4194304] 0 2024-03-26T18:54:43.311 INFO:tasks.workunit.client.1.smithi190.stdout:6/891: chown d17/d28/df6/da8/df4/dfb/d121 37342498 1 2024-03-26T18:54:43.311 INFO:tasks.workunit.client.1.smithi190.stdout:6/892: write d17/d28/de8/f11e [542658,41159] 0 2024-03-26T18:54:43.311 INFO:tasks.workunit.client.1.smithi190.stdout:5/946: dwrite d2/d17/d62/f138 [0,4194304] 0 2024-03-26T18:54:43.312 INFO:tasks.workunit.client.1.smithi190.stdout:5/947: write d2/d17/d1b/db6/d88/f125 [758434,100179] 0 2024-03-26T18:54:43.312 INFO:tasks.workunit.client.1.smithi190.stdout:3/877: dwrite d1/f15 [0,4194304] 0 2024-03-26T18:54:43.312 INFO:tasks.workunit.client.1.smithi190.stdout:1/951: dwrite d3/f53 [4194304,4194304] 0 2024-03-26T18:54:43.314 INFO:tasks.workunit.client.0.smithi102.stdout:2/685: mknod d3/d11/d15/d31/d7f/cf8 0 2024-03-26T18:54:43.315 INFO:tasks.workunit.client.0.smithi102.stdout:0/721: dread d6/d2d/d3d/d80/fdf [0,4194304] 0 2024-03-26T18:54:43.315 INFO:tasks.workunit.client.0.smithi102.stdout:0/722: truncate d6/de/d13/d25/d4f/d66/f6e 723255 0 2024-03-26T18:54:43.315 INFO:tasks.workunit.client.0.smithi102.stdout:9/638: dread df/d1a/d23/fb2 [0,4194304] 0 2024-03-26T18:54:43.321 INFO:tasks.workunit.client.0.smithi102.stdout:6/682: dread d2/df/d35/d43/f94 [0,4194304] 0 2024-03-26T18:54:43.321 INFO:tasks.workunit.client.0.smithi102.stdout:2/686: rename d3/d11/d15/d31/l61 to d3/d82/d93/lf9 0 2024-03-26T18:54:43.322 INFO:tasks.workunit.client.1.smithi190.stdout:1/952: dread d3/da/dd/d2f/d5b/db5/fc7 [0,4194304] 0 2024-03-26T18:54:43.324 INFO:tasks.workunit.client.0.smithi102.stdout:0/723: link d6/de/d13/d1f/d55/d57/d7d/fd5 d6/de/d13/d9a/d85/ff7 0 2024-03-26T18:54:43.325 INFO:tasks.workunit.client.1.smithi190.stdout:5/948: symlink d2/d17/d1b/db6/d96/l140 0 2024-03-26T18:54:43.325 INFO:tasks.workunit.client.0.smithi102.stdout:6/683: mknod d2/d12/dce/cde 0 2024-03-26T18:54:43.326 INFO:tasks.workunit.client.1.smithi190.stdout:3/878: rename d1/d3/df/d18/d35/d65/l6c to d1/d3/df/d18/d35/d41/dae/d86/l10e 0 2024-03-26T18:54:43.329 INFO:tasks.workunit.client.1.smithi190.stdout:6/893: getdents d17/d28/df6 0 2024-03-26T18:54:43.329 INFO:tasks.workunit.client.1.smithi190.stdout:6/894: dread - d17/d28/d2c/db8/df3/f118 zero size 2024-03-26T18:54:43.329 INFO:tasks.workunit.client.1.smithi190.stdout:6/895: write d17/dd3/d113/f11c [829848,1257] 0 2024-03-26T18:54:43.334 INFO:tasks.workunit.client.0.smithi102.stdout:2/687: link d3/d11/d15/d46/f92 d3/d20/d78/d84/d6e/dc6/ffa 0 2024-03-26T18:54:43.336 INFO:tasks.workunit.client.1.smithi190.stdout:1/953: truncate d3/da/dd/d2f/d5b/d83/feb 1997114 0 2024-03-26T18:54:43.336 INFO:tasks.workunit.client.1.smithi190.stdout:3/879: dread d1/d3/df/d31/dc4/f106 [0,4194304] 0 2024-03-26T18:54:43.337 INFO:tasks.workunit.client.1.smithi190.stdout:5/949: mkdir d2/d17/d1b/db6/d88/dc9/d141 0 2024-03-26T18:54:43.338 INFO:tasks.workunit.client.0.smithi102.stdout:0/724: unlink d6/de/d13/d1f/d55/d57/d7d/fba 0 2024-03-26T18:54:43.339 INFO:tasks.workunit.client.0.smithi102.stdout:6/684: mknod d2/d7/d3c/cdf 0 2024-03-26T18:54:43.342 INFO:tasks.workunit.client.0.smithi102.stdout:2/688: rmdir d3 39 2024-03-26T18:54:43.342 INFO:tasks.workunit.client.1.smithi190.stdout:6/896: mkdir d17/dd3/d12f 0 2024-03-26T18:54:43.343 INFO:tasks.workunit.client.1.smithi190.stdout:1/954: mknod d3/da/dd/c12e 0 2024-03-26T18:54:43.345 INFO:tasks.workunit.client.0.smithi102.stdout:0/725: rename d6/de/d13/d1f/d55/d57/cdc to d6/dbe/cf8 0 2024-03-26T18:54:43.345 INFO:tasks.workunit.client.0.smithi102.stdout:0/726: read - d6/d3e/d87/fea zero size 2024-03-26T18:54:43.346 INFO:tasks.workunit.client.1.smithi190.stdout:3/880: truncate d1/de/d4e/f8f 3421404 0 2024-03-26T18:54:43.347 INFO:tasks.workunit.client.0.smithi102.stdout:6/685: mkdir d2/d3d/d53/db1/de0 0 2024-03-26T18:54:43.348 INFO:tasks.workunit.client.0.smithi102.stdout:6/686: read d2/d3d/d53/f74 [105729,83942] 0 2024-03-26T18:54:43.348 INFO:tasks.workunit.client.0.smithi102.stdout:6/687: fsync d2/d3d/f70 0 2024-03-26T18:54:43.349 INFO:tasks.workunit.client.0.smithi102.stdout:2/689: truncate d3/fb 4419968 0 2024-03-26T18:54:43.349 INFO:tasks.workunit.client.0.smithi102.stdout:2/690: chown d3/d82/de2/def/ff6 16635 1 2024-03-26T18:54:43.349 INFO:tasks.workunit.client.1.smithi190.stdout:1/955: dread d3/d9/f15 [0,4194304] 0 2024-03-26T18:54:43.351 INFO:tasks.workunit.client.1.smithi190.stdout:6/897: symlink d17/d28/d2c/db8/df3/l130 0 2024-03-26T18:54:43.354 INFO:tasks.workunit.client.1.smithi190.stdout:3/881: mknod d1/d3/c10f 0 2024-03-26T18:54:43.355 INFO:tasks.workunit.client.0.smithi102.stdout:0/727: symlink d6/d3e/d87/lf9 0 2024-03-26T18:54:43.355 INFO:tasks.workunit.client.0.smithi102.stdout:0/728: creat d6/de/d13/d1f/d71/ffa x:0 0 0 2024-03-26T18:54:43.355 INFO:tasks.workunit.client.0.smithi102.stdout:0/729: creat d6/ffb x:0 0 0 2024-03-26T18:54:43.356 INFO:tasks.workunit.client.0.smithi102.stdout:2/691: link d3/f75 d3/d82/d93/ffb 0 2024-03-26T18:54:43.368 INFO:tasks.workunit.client.0.smithi102.stdout:2/692: creat d3/d11/d15/ded/ffc x:0 0 0 2024-03-26T18:54:43.368 INFO:tasks.workunit.client.0.smithi102.stdout:2/693: truncate d3/d11/d15/d46/fea 319145 0 2024-03-26T18:54:43.370 INFO:tasks.workunit.client.0.smithi102.stdout:2/694: creat d3/d11/d27/ffd x:0 0 0 2024-03-26T18:54:43.372 INFO:tasks.workunit.client.1.smithi190.stdout:6/898: chown d17/d28/d2c/db8/df3/d109/d79/la3 998344 1 2024-03-26T18:54:43.374 INFO:tasks.workunit.client.0.smithi102.stdout:6/688: dwrite d2/d3d/fbc [0,4194304] 0 2024-03-26T18:54:43.377 INFO:tasks.workunit.client.1.smithi190.stdout:6/899: dread d17/d28/df6/f88 [0,4194304] 0 2024-03-26T18:54:43.377 INFO:tasks.workunit.client.1.smithi190.stdout:6/900: fsync d17/d28/d2c/d50/f9c 0 2024-03-26T18:54:43.377 INFO:tasks.workunit.client.1.smithi190.stdout:6/901: stat d17/d28 0 2024-03-26T18:54:43.378 INFO:tasks.workunit.client.1.smithi190.stdout:1/956: dwrite d3/da/dd/d2f/d34/f35 [0,4194304] 0 2024-03-26T18:54:43.378 INFO:tasks.workunit.client.1.smithi190.stdout:1/957: write d3/d8f/da7/f11a [854756,36705] 0 2024-03-26T18:54:43.379 INFO:tasks.workunit.client.1.smithi190.stdout:3/882: dread d1/d3/df/d18/d35/f39 [0,4194304] 0 2024-03-26T18:54:43.379 INFO:tasks.workunit.client.1.smithi190.stdout:3/883: dread - d1/d47/deb/ff6 zero size 2024-03-26T18:54:43.379 INFO:tasks.workunit.client.1.smithi190.stdout:3/884: readlink d1/d3/df/d18/d1b/lb7 0 2024-03-26T18:54:43.379 INFO:tasks.workunit.client.1.smithi190.stdout:3/885: fdatasync d1/de/d8a/f95 0 2024-03-26T18:54:43.379 INFO:tasks.workunit.client.1.smithi190.stdout:3/886: creat d1/d3/df/d18/d35/d41/dae/d9c/f110 x:0 0 0 2024-03-26T18:54:43.384 INFO:tasks.workunit.client.1.smithi190.stdout:1/958: mkdir d3/d8f/d12f 0 2024-03-26T18:54:43.398 INFO:tasks.workunit.client.1.smithi190.stdout:5/950: rmdir d2/d17/d1b/db6 39 2024-03-26T18:54:43.399 INFO:tasks.workunit.client.1.smithi190.stdout:1/959: dread d3/d8f/da7/f11a [0,4194304] 0 2024-03-26T18:54:43.399 INFO:tasks.workunit.client.1.smithi190.stdout:1/960: dread - d3/da/dd/d2f/d33/d115/f12b zero size 2024-03-26T18:54:43.412 INFO:tasks.workunit.client.1.smithi190.stdout:6/902: dwrite d17/d28/f2d [4194304,4194304] 0 2024-03-26T18:54:43.418 INFO:tasks.workunit.client.1.smithi190.stdout:5/951: write d2/d17/d62/f138 [2345114,29032] 0 2024-03-26T18:54:43.419 INFO:tasks.workunit.client.1.smithi190.stdout:3/887: dwrite d1/d3/df/d18/d35/f39 [0,4194304] 0 2024-03-26T18:54:43.419 INFO:tasks.workunit.client.1.smithi190.stdout:3/888: write d1/d3/df/d31/ff1 [438107,119710] 0 2024-03-26T18:54:43.419 INFO:tasks.workunit.client.1.smithi190.stdout:3/889: dread - d1/d3/d52/d70/db3/f10c zero size 2024-03-26T18:54:43.420 INFO:tasks.workunit.client.1.smithi190.stdout:5/952: write d2/d17/f4f [654462,110750] 0 2024-03-26T18:54:43.421 INFO:tasks.workunit.client.1.smithi190.stdout:5/953: creat d2/d17/d1b/db6/d24/d3b/f142 x:0 0 0 2024-03-26T18:54:43.421 INFO:tasks.workunit.client.1.smithi190.stdout:5/954: write d2/d17/d1b/db6/d96/d131/f13c [728106,125241] 0 2024-03-26T18:54:43.423 INFO:tasks.workunit.client.0.smithi102.stdout:5/952: dwrite d2/d25/de6/f10f [0,4194304] 0 2024-03-26T18:54:43.423 INFO:tasks.workunit.client.0.smithi102.stdout:5/953: truncate d2/dea/dbb/dcf/f125 665305 0 2024-03-26T18:54:43.423 INFO:tasks.workunit.client.0.smithi102.stdout:5/954: write d2/d14/fa1 [309157,23618] 0 2024-03-26T18:54:43.424 INFO:tasks.workunit.client.1.smithi190.stdout:5/955: creat d2/d17/d1b/d60/d63/d137/f143 x:0 0 0 2024-03-26T18:54:43.425 INFO:tasks.workunit.client.1.smithi190.stdout:5/956: dread d2/d17/d62/dd7/ffa [0,4194304] 0 2024-03-26T18:54:43.425 INFO:tasks.workunit.client.1.smithi190.stdout:5/957: stat d2/d17/d98/d3a/d3c/f94 0 2024-03-26T18:54:43.425 INFO:tasks.workunit.client.1.smithi190.stdout:5/958: readlink d2/d17/d1b/lea 0 2024-03-26T18:54:43.425 INFO:tasks.workunit.client.0.smithi102.stdout:5/955: rename d2/d25/d75/db9/d43/d9e/d82/dc9/f134 to d2/d25/db0/d147/f14a 0 2024-03-26T18:54:43.425 INFO:tasks.workunit.client.0.smithi102.stdout:5/956: chown d2/l18 9 1 2024-03-26T18:54:43.426 INFO:tasks.workunit.client.1.smithi190.stdout:3/890: dread d1/d3/df/d18/d35/f60 [0,4194304] 0 2024-03-26T18:54:43.426 INFO:tasks.workunit.client.0.smithi102.stdout:5/957: read d2/d25/d75/ffb [548448,40807] 0 2024-03-26T18:54:43.429 INFO:tasks.workunit.client.1.smithi190.stdout:5/959: mkdir d2/d17/d1b/d60/d63/dcd/d118/d120/d144 0 2024-03-26T18:54:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:5/958: symlink d2/d25/d75/def/dec/l14b 0 2024-03-26T18:54:43.431 INFO:tasks.workunit.client.1.smithi190.stdout:3/891: mknod d1/d3/df/d18/d1b/de4/c111 0 2024-03-26T18:54:43.432 INFO:tasks.workunit.client.1.smithi190.stdout:3/892: symlink d1/d3/l112 0 2024-03-26T18:54:43.433 INFO:tasks.workunit.client.0.smithi102.stdout:3/914: sync 2024-03-26T18:54:43.434 INFO:tasks.workunit.client.0.smithi102.stdout:3/915: dread - df/d1c/d34/d53/f12f zero size 2024-03-26T18:54:43.436 INFO:tasks.workunit.client.0.smithi102.stdout:3/916: link df/d29/d31/d36/d45/fd8 df/d1c/d34/d9b/f148 0 2024-03-26T18:54:43.438 INFO:tasks.workunit.client.0.smithi102.stdout:3/917: mkdir df/d1c/d34/da5/dce/d149 0 2024-03-26T18:54:43.439 INFO:tasks.workunit.client.0.smithi102.stdout:3/918: mknod df/d29/d31/d77/d8a/c14a 0 2024-03-26T18:54:43.441 INFO:tasks.workunit.client.0.smithi102.stdout:3/919: rmdir df/d29/d31/d36/d45/daf/d105/d13a 0 2024-03-26T18:54:43.441 INFO:tasks.workunit.client.1.smithi190.stdout:5/960: write d2/d17/d1b/d30/f73 [2313281,122817] 0 2024-03-26T18:54:43.441 INFO:tasks.workunit.client.1.smithi190.stdout:5/961: fsync d2/d17/d1b/db6/d88/fd9 0 2024-03-26T18:54:43.442 INFO:tasks.workunit.client.0.smithi102.stdout:3/920: mknod df/d8b/dd5/d145/c14b 0 2024-03-26T18:54:43.442 INFO:tasks.workunit.client.0.smithi102.stdout:0/730: dwrite d6/de/d13/d25/d31/d45/f65 [0,4194304] 0 2024-03-26T18:54:43.442 INFO:tasks.workunit.client.0.smithi102.stdout:0/731: chown d6/f1a 10 1 2024-03-26T18:54:43.442 INFO:tasks.workunit.client.0.smithi102.stdout:0/732: stat d6/d2d/l6d 0 2024-03-26T18:54:43.442 INFO:tasks.workunit.client.0.smithi102.stdout:0/733: write d6/de/d13/d25/d31/d45/d82/d84/d88/fae [3629829,94078] 0 2024-03-26T18:54:43.446 INFO:tasks.workunit.client.1.smithi190.stdout:5/962: creat d2/d17/f145 x:0 0 0 2024-03-26T18:54:43.446 INFO:tasks.workunit.client.1.smithi190.stdout:6/903: dwrite d17/d28/df6/da8/df4/dfb/f56 [0,4194304] 0 2024-03-26T18:54:43.447 INFO:tasks.workunit.client.0.smithi102.stdout:3/921: truncate df/d29/fc7 424923 0 2024-03-26T18:54:43.447 INFO:tasks.workunit.client.0.smithi102.stdout:3/922: creat df/d61/de5/d128/f14c x:0 0 0 2024-03-26T18:54:43.449 INFO:tasks.workunit.client.0.smithi102.stdout:5/959: dwrite d2/d25/d5b/fc4 [0,4194304] 0 2024-03-26T18:54:43.451 INFO:tasks.workunit.client.1.smithi190.stdout:5/963: truncate d2/d17/d1b/db6/f9a 8014304 0 2024-03-26T18:54:43.452 INFO:tasks.workunit.client.1.smithi190.stdout:3/893: dread d1/d3/df/d31/ff1 [0,4194304] 0 2024-03-26T18:54:43.452 INFO:tasks.workunit.client.1.smithi190.stdout:3/894: getdents d1/dd3/df7 0 2024-03-26T18:54:43.452 INFO:tasks.workunit.client.1.smithi190.stdout:3/895: write d1/d3/df/d18/d35/d41/dae/d9c/f110 [767484,43211] 0 2024-03-26T18:54:43.456 INFO:tasks.workunit.client.1.smithi190.stdout:6/904: mkdir d17/d28/d8b/d131 0 2024-03-26T18:54:43.462 INFO:tasks.workunit.client.0.smithi102.stdout:3/923: link df/d1c/d34/d53/l8f df/d29/d31/d77/d8a/dae/dee/l14d 0 2024-03-26T18:54:43.463 INFO:tasks.workunit.client.0.smithi102.stdout:3/924: write df/d1c/f81 [117811,77260] 0 2024-03-26T18:54:43.466 INFO:tasks.workunit.client.0.smithi102.stdout:5/960: truncate d2/f64 2956472 0 2024-03-26T18:54:43.467 INFO:tasks.workunit.client.0.smithi102.stdout:0/734: dwrite d6/de/d13/d25/d4f/d66/fb7 [0,4194304] 0 2024-03-26T18:54:43.473 INFO:tasks.workunit.client.0.smithi102.stdout:3/925: rename df/d29/d31/d77/d8a/dae/dee/df2/d100/d11a/c132 to df/d8b/dd5/d145/c14e 0 2024-03-26T18:54:43.476 INFO:tasks.workunit.client.0.smithi102.stdout:5/961: mkdir d2/d25/d5b/d72/d14c 0 2024-03-26T18:54:43.476 INFO:tasks.workunit.client.0.smithi102.stdout:5/962: creat d2/d25/d75/def/f14d x:0 0 0 2024-03-26T18:54:43.483 INFO:tasks.workunit.client.0.smithi102.stdout:3/926: rename df/f26 to df/d29/d3a/d93/d120/f14f 0 2024-03-26T18:54:43.483 INFO:tasks.workunit.client.0.smithi102.stdout:3/927: creat df/d29/d31/d77/d8a/dae/dee/df2/d100/f150 x:0 0 0 2024-03-26T18:54:43.487 INFO:tasks.workunit.client.1.smithi190.stdout:3/896: mkdir d1/d3/df/d18/d57/d91/d8c/d113 0 2024-03-26T18:54:43.488 INFO:tasks.workunit.client.1.smithi190.stdout:3/897: read - d1/d3/df/d18/db4/f105 zero size 2024-03-26T18:54:43.488 INFO:tasks.workunit.client.0.smithi102.stdout:5/963: dread d2/f54 [0,4194304] 0 2024-03-26T18:54:43.489 INFO:tasks.workunit.client.0.smithi102.stdout:0/735: symlink d6/de/d13/d1f/d55/d57/lfc 0 2024-03-26T18:54:43.492 INFO:tasks.workunit.client.1.smithi190.stdout:6/905: getdents d17/d28/d2c/db8/dcc 0 2024-03-26T18:54:43.492 INFO:tasks.workunit.client.1.smithi190.stdout:6/906: chown d17/dd3/c119 60698 1 2024-03-26T18:54:43.495 INFO:tasks.workunit.client.0.smithi102.stdout:3/928: read df/d1c/d34/d9b/f148 [3723482,71558] 0 2024-03-26T18:54:43.495 INFO:tasks.workunit.client.0.smithi102.stdout:3/929: fdatasync df/d29/d31/d36/f7f 0 2024-03-26T18:54:43.496 INFO:tasks.workunit.client.1.smithi190.stdout:3/898: link d1/d3/df/d18/d35/d41/dae/d86/l109 d1/d3/df/d18/d1b/l114 0 2024-03-26T18:54:43.499 INFO:tasks.workunit.client.0.smithi102.stdout:0/736: mkdir d6/dbe/deb/dfd 0 2024-03-26T18:54:43.499 INFO:tasks.workunit.client.0.smithi102.stdout:0/737: chown d6/de/d13/d1f/d71/ffa 48272 1 2024-03-26T18:54:43.501 INFO:tasks.workunit.client.1.smithi190.stdout:3/899: rename d1/d3/df/d18/db4/lc9 to d1/dd3/d10b/l115 0 2024-03-26T18:54:43.505 INFO:tasks.workunit.client.1.smithi190.stdout:5/964: dwrite d2/d17/f7b [0,4194304] 0 2024-03-26T18:54:43.505 INFO:tasks.workunit.client.1.smithi190.stdout:3/900: rename d1/de/d4e/f8f to d1/de/d8a/de1/f116 0 2024-03-26T18:54:43.506 INFO:tasks.workunit.client.0.smithi102.stdout:3/930: dread df/d29/f64 [0,4194304] 0 2024-03-26T18:54:43.506 INFO:tasks.workunit.client.0.smithi102.stdout:3/931: write df/d29/d31/d77/d8a/dae/dee/f119 [230274,46805] 0 2024-03-26T18:54:43.506 INFO:tasks.workunit.client.0.smithi102.stdout:0/738: mknod d6/de/d13/d1f/d55/d57/cfe 0 2024-03-26T18:54:43.508 INFO:tasks.workunit.client.0.smithi102.stdout:3/932: mknod df/d29/c151 0 2024-03-26T18:54:43.509 INFO:tasks.workunit.client.0.smithi102.stdout:3/933: write df/d1c/d34/da5/dce/f112 [737452,28791] 0 2024-03-26T18:54:43.509 INFO:tasks.workunit.client.0.smithi102.stdout:3/934: chown df/d29/d3a/l116 692072 1 2024-03-26T18:54:43.509 INFO:tasks.workunit.client.0.smithi102.stdout:3/935: fdatasync df/d29/d31/d36/d3c/fc2 0 2024-03-26T18:54:43.511 INFO:tasks.workunit.client.0.smithi102.stdout:0/739: mknod d6/d2d/d3d/cff 0 2024-03-26T18:54:43.511 INFO:tasks.workunit.client.1.smithi190.stdout:5/965: rmdir d2/d17/d1b/db6/d24/d3b/d126 39 2024-03-26T18:54:43.511 INFO:tasks.workunit.client.1.smithi190.stdout:5/966: chown d2/d17/d1b/c53 10 1 2024-03-26T18:54:43.512 INFO:tasks.workunit.client.0.smithi102.stdout:0/740: mkdir d6/de/d13/d25/d31/d45/d82/d100 0 2024-03-26T18:54:43.521 INFO:tasks.workunit.client.1.smithi190.stdout:3/901: dread d1/de/f1d [0,4194304] 0 2024-03-26T18:54:43.522 INFO:tasks.workunit.client.1.smithi190.stdout:6/907: dwrite d17/d28/df6/da8/fc7 [0,4194304] 0 2024-03-26T18:54:43.523 INFO:tasks.workunit.client.1.smithi190.stdout:5/967: read d2/f45 [1204733,42200] 0 2024-03-26T18:54:43.525 INFO:tasks.workunit.client.1.smithi190.stdout:6/908: rename d17/d28/d2c/d50/df7/f122 to d17/d28/f132 0 2024-03-26T18:54:43.530 INFO:tasks.workunit.client.1.smithi190.stdout:3/902: dread d1/d3/df/d18/d35/d41/f72 [0,4194304] 0 2024-03-26T18:54:43.537 INFO:tasks.workunit.client.1.smithi190.stdout:6/909: dread d17/d28/d2c/fb7 [0,4194304] 0 2024-03-26T18:54:43.538 INFO:tasks.workunit.client.0.smithi102.stdout:3/936: dwrite df/d29/d31/d36/d3c/d6f/d75/f125 [0,4194304] 0 2024-03-26T18:54:43.538 INFO:tasks.workunit.client.0.smithi102.stdout:3/937: dread - df/d29/d31/d36/d3c/d6f/ffd zero size 2024-03-26T18:54:43.538 INFO:tasks.workunit.client.0.smithi102.stdout:5/964: dwrite d2/dea/d4f/f52 [0,4194304] 0 2024-03-26T18:54:43.538 INFO:tasks.workunit.client.0.smithi102.stdout:5/965: chown d2/f17 43108 1 2024-03-26T18:54:43.538 INFO:tasks.workunit.client.0.smithi102.stdout:5/966: dread - d2/dea/d4f/f114 zero size 2024-03-26T18:54:43.539 INFO:tasks.workunit.client.1.smithi190.stdout:6/910: creat d17/d28/d2c/db8/df3/d109/d79/f133 x:0 0 0 2024-03-26T18:54:43.542 INFO:tasks.workunit.client.1.smithi190.stdout:6/911: symlink d17/d28/d8b/d95/dda/d10a/l134 0 2024-03-26T18:54:43.542 INFO:tasks.workunit.client.1.smithi190.stdout:6/912: chown d17/d28/d8b/d95/caf 42101413 1 2024-03-26T18:54:43.542 INFO:tasks.workunit.client.1.smithi190.stdout:3/903: dread d1/d3/df/d18/d35/d41/dae/f96 [0,4194304] 0 2024-03-26T18:54:43.543 INFO:tasks.workunit.client.1.smithi190.stdout:5/968: dwrite d2/d17/d1b/db6/d24/d3b/d126/f134 [0,4194304] 0 2024-03-26T18:54:43.543 INFO:tasks.workunit.client.0.smithi102.stdout:3/938: rename df/d29/d31/d36/d3c/f10f to df/d29/d3a/d93/d120/f152 0 2024-03-26T18:54:43.548 INFO:tasks.workunit.client.1.smithi190.stdout:3/904: truncate d1/de/d4e/f54 1889287 0 2024-03-26T18:54:43.548 INFO:tasks.workunit.client.1.smithi190.stdout:3/905: fdatasync d1/d3/f4b 0 2024-03-26T18:54:43.555 INFO:tasks.workunit.client.0.smithi102.stdout:5/967: rename d2/d25/d5b/d76/ca7 to d2/dea/dbb/c14e 0 2024-03-26T18:54:43.555 INFO:tasks.workunit.client.0.smithi102.stdout:5/968: chown d2/dea/dbb/dcf/c118 375002218 1 2024-03-26T18:54:43.556 INFO:tasks.workunit.client.1.smithi190.stdout:3/906: dread d1/f4 [0,4194304] 0 2024-03-26T18:54:43.556 INFO:tasks.workunit.client.1.smithi190.stdout:3/907: chown d1/d3/df/d18/d35/d41/dae/d86/l10e 8252546 1 2024-03-26T18:54:43.559 INFO:tasks.workunit.client.0.smithi102.stdout:5/969: mkdir d2/d25/d75/db9/d43/d9e/d14f 0 2024-03-26T18:54:43.560 INFO:tasks.workunit.client.0.smithi102.stdout:5/970: mknod d2/d25/d75/db9/d43/d9e/daf/c150 0 2024-03-26T18:54:43.560 INFO:tasks.workunit.client.0.smithi102.stdout:5/971: creat d2/dea/db4/d11f/f151 x:0 0 0 2024-03-26T18:54:43.560 INFO:tasks.workunit.client.0.smithi102.stdout:5/972: readlink d2/d25/d75/db9/l46 0 2024-03-26T18:54:43.562 INFO:tasks.workunit.client.0.smithi102.stdout:5/973: rename d2/d25/d75/def/dec/d112/db8/ceb to d2/d25/d75/db9/d43/d9e/d82/d136/c152 0 2024-03-26T18:54:43.563 INFO:tasks.workunit.client.0.smithi102.stdout:5/974: mknod d2/d25/d75/def/c153 0 2024-03-26T18:54:43.565 INFO:tasks.workunit.client.0.smithi102.stdout:5/975: creat d2/d25/d75/db9/db3/d115/f154 x:0 0 0 2024-03-26T18:54:43.570 INFO:tasks.workunit.client.1.smithi190.stdout:3/908: read d1/de/d4e/f54 [121000,112627] 0 2024-03-26T18:54:43.571 INFO:tasks.workunit.client.0.smithi102.stdout:7/582: sync 2024-03-26T18:54:43.571 INFO:tasks.workunit.client.1.smithi190.stdout:5/969: dwrite d2/d17/d1b/db6/d96/fa4 [0,4194304] 0 2024-03-26T18:54:43.572 INFO:tasks.workunit.client.0.smithi102.stdout:3/939: dwrite df/d61/dd6/fdb [0,4194304] 0 2024-03-26T18:54:43.572 INFO:tasks.workunit.client.0.smithi102.stdout:3/940: creat df/d29/d31/dc0/f153 x:0 0 0 2024-03-26T18:54:43.572 INFO:tasks.workunit.client.0.smithi102.stdout:3/941: fdatasync df/d29/d31/d36/f41 0 2024-03-26T18:54:43.572 INFO:tasks.workunit.client.1.smithi190.stdout:3/909: write d1/de/f2d [982399,53777] 0 2024-03-26T18:54:43.572 INFO:tasks.workunit.client.1.smithi190.stdout:3/910: write d1/d3/df/d18/d35/d41/f103 [314269,39852] 0 2024-03-26T18:54:43.574 INFO:tasks.workunit.client.1.smithi190.stdout:5/970: creat d2/d17/d1b/d60/d63/dcd/d118/f146 x:0 0 0 2024-03-26T18:54:43.574 INFO:tasks.workunit.client.1.smithi190.stdout:5/971: creat d2/d17/d62/f147 x:0 0 0 2024-03-26T18:54:43.578 INFO:tasks.workunit.client.0.smithi102.stdout:3/942: symlink df/d29/d31/d36/l154 0 2024-03-26T18:54:43.580 INFO:tasks.workunit.client.0.smithi102.stdout:3/943: mknod df/d1c/d34/da5/dce/c155 0 2024-03-26T18:54:43.582 INFO:tasks.workunit.client.0.smithi102.stdout:3/944: mkdir df/d29/d31/d77/d8a/dae/d156 0 2024-03-26T18:54:43.582 INFO:tasks.workunit.client.0.smithi102.stdout:3/945: write df/d29/d31/dc0/f130 [833906,1972] 0 2024-03-26T18:54:43.584 INFO:tasks.workunit.client.0.smithi102.stdout:3/946: creat df/d29/d31/d77/d8a/dae/dee/df2/d100/d11a/f157 x:0 0 0 2024-03-26T18:54:43.584 INFO:tasks.workunit.client.0.smithi102.stdout:5/976: dwrite d2/d25/d5b/fa4 [0,4194304] 0 2024-03-26T18:54:43.590 INFO:tasks.workunit.client.0.smithi102.stdout:5/977: rename d2/d25/d75/db9/c47 to d2/d25/d75/db9/d43/d9e/d82/c155 0 2024-03-26T18:54:43.591 INFO:tasks.workunit.client.0.smithi102.stdout:5/978: rmdir d2/d25/d5b/d76 39 2024-03-26T18:54:43.592 INFO:tasks.workunit.client.0.smithi102.stdout:5/979: rmdir d2/d25/d5b/d76 39 2024-03-26T18:54:43.594 INFO:tasks.workunit.client.0.smithi102.stdout:5/980: unlink d2/d2c/la0 0 2024-03-26T18:54:43.595 INFO:tasks.workunit.client.0.smithi102.stdout:5/981: symlink d2/d25/d75/db9/d43/d9e/d82/d136/l156 0 2024-03-26T18:54:43.595 INFO:tasks.workunit.client.0.smithi102.stdout:7/583: dwrite d7/d10/d17/f59 [0,4194304] 0 2024-03-26T18:54:43.596 INFO:tasks.workunit.client.0.smithi102.stdout:7/584: mkdir d7/dab 0 2024-03-26T18:54:43.597 INFO:tasks.workunit.client.0.smithi102.stdout:7/585: fsync d7/d31/f45 0 2024-03-26T18:54:43.598 INFO:tasks.workunit.client.0.smithi102.stdout:7/586: creat d7/d12/fac x:0 0 0 2024-03-26T18:54:43.598 INFO:tasks.workunit.client.0.smithi102.stdout:7/587: write d7/d12/fac [975592,100710] 0 2024-03-26T18:54:43.598 INFO:tasks.workunit.client.0.smithi102.stdout:7/588: chown d7/d12/d2b/c70 927424 1 2024-03-26T18:54:43.599 INFO:tasks.workunit.client.0.smithi102.stdout:7/589: fsync d7/d12/f43 0 2024-03-26T18:54:43.599 INFO:tasks.workunit.client.0.smithi102.stdout:7/590: fdatasync d7/d10/d17/f1b 0 2024-03-26T18:54:43.600 INFO:tasks.workunit.client.0.smithi102.stdout:7/591: rename d7/d44/c5b to d7/d10/d27/d34/d82/cad 0 2024-03-26T18:54:43.601 INFO:tasks.workunit.client.0.smithi102.stdout:7/592: write d7/d12/d4d/d62/f65 [412748,16554] 0 2024-03-26T18:54:43.601 INFO:tasks.workunit.client.0.smithi102.stdout:7/593: mkdir d7/d10/d67/d69/d9b/dae 0 2024-03-26T18:54:43.603 INFO:tasks.workunit.client.0.smithi102.stdout:7/594: rename d7/d12/d51/d6a/c7f to d7/d12/d2b/da8/caf 0 2024-03-26T18:54:43.618 INFO:tasks.workunit.client.1.smithi190.stdout:3/911: dwrite d1/de/d8a/fa7 [0,4194304] 0 2024-03-26T18:54:43.618 INFO:tasks.workunit.client.1.smithi190.stdout:3/912: stat d1/d3/df/d18/d35/d41/d61/d100 0 2024-03-26T18:54:43.618 INFO:tasks.workunit.client.1.smithi190.stdout:3/913: fdatasync d1/d3/df/d18/d35/f39 0 2024-03-26T18:54:43.618 INFO:tasks.workunit.client.1.smithi190.stdout:3/914: mknod d1/d3/df/c117 0 2024-03-26T18:54:43.621 INFO:tasks.workunit.client.1.smithi190.stdout:1/961: sync 2024-03-26T18:54:43.621 INFO:tasks.workunit.client.1.smithi190.stdout:1/962: creat d3/d9/db/f130 x:0 0 0 2024-03-26T18:54:43.622 INFO:tasks.workunit.client.0.smithi102.stdout:7/595: dwrite d7/d10/d67/fa5 [0,4194304] 0 2024-03-26T18:54:43.622 INFO:tasks.workunit.client.0.smithi102.stdout:7/596: truncate d7/d12/d4d/f5e 4489734 0 2024-03-26T18:54:43.622 INFO:tasks.workunit.client.0.smithi102.stdout:3/947: dwrite df/d61/de5/d128/f144 [0,4194304] 0 2024-03-26T18:54:43.623 INFO:tasks.workunit.client.0.smithi102.stdout:5/982: dwrite d2/d25/d75/ffb [0,4194304] 0 2024-03-26T18:54:43.623 INFO:tasks.workunit.client.0.smithi102.stdout:5/983: fsync d2/d25/d75/db9/d43/d9e/daf/fd2 0 2024-03-26T18:54:43.623 INFO:tasks.workunit.client.0.smithi102.stdout:5/984: truncate d2/d25/d5b/ff0 4554711 0 2024-03-26T18:54:43.623 INFO:tasks.workunit.client.1.smithi190.stdout:1/963: rename d3/da/dd/d95/l82 to d3/dd9/l131 0 2024-03-26T18:54:43.624 INFO:tasks.workunit.client.1.smithi190.stdout:3/915: read d1/f15 [3782307,36689] 0 2024-03-26T18:54:43.625 INFO:tasks.workunit.client.1.smithi190.stdout:1/964: unlink d3/d9/db/f119 0 2024-03-26T18:54:43.627 INFO:tasks.workunit.client.0.smithi102.stdout:7/597: rename d7/d44/c86 to d7/d12/d2b/cb0 0 2024-03-26T18:54:43.629 INFO:tasks.workunit.client.1.smithi190.stdout:3/916: symlink d1/d3/df/d18/d35/d41/d61/d100/l118 0 2024-03-26T18:54:43.631 INFO:tasks.workunit.client.1.smithi190.stdout:1/965: symlink d3/da/d3a/d99/l132 0 2024-03-26T18:54:43.631 INFO:tasks.workunit.client.1.smithi190.stdout:1/966: creat d3/da/d3a/d99/f133 x:0 0 0 2024-03-26T18:54:43.633 INFO:tasks.workunit.client.0.smithi102.stdout:3/948: creat df/d29/d31/d77/f158 x:0 0 0 2024-03-26T18:54:43.633 INFO:tasks.workunit.client.0.smithi102.stdout:3/949: chown df/d29/d31/d77/d8a/dae/f11b 26 1 2024-03-26T18:54:43.636 INFO:tasks.workunit.client.1.smithi190.stdout:3/917: dread d1/d3/d52/f62 [0,4194304] 0 2024-03-26T18:54:43.636 INFO:tasks.workunit.client.0.smithi102.stdout:5/985: truncate d2/d25/d75/db9/fd5 3530858 0 2024-03-26T18:54:43.637 INFO:tasks.workunit.client.0.smithi102.stdout:5/986: read - d2/d25/d75/db9/db3/d115/f154 zero size 2024-03-26T18:54:43.637 INFO:tasks.workunit.client.0.smithi102.stdout:7/598: symlink d7/dab/lb1 0 2024-03-26T18:54:43.638 INFO:tasks.workunit.client.1.smithi190.stdout:3/918: symlink d1/d3/df/d31/dd1/l119 0 2024-03-26T18:54:43.640 INFO:tasks.workunit.client.1.smithi190.stdout:1/967: write d3/fab [3847678,582] 0 2024-03-26T18:54:43.648 INFO:tasks.workunit.client.1.smithi190.stdout:1/968: link d3/da/f94 d3/da/daf/f134 0 2024-03-26T18:54:43.648 INFO:tasks.workunit.client.1.smithi190.stdout:1/969: write d3/d73/d88/f92 [3606752,110713] 0 2024-03-26T18:54:43.648 INFO:tasks.workunit.client.1.smithi190.stdout:1/970: chown d3/da/f78 100 1 2024-03-26T18:54:43.650 INFO:tasks.workunit.client.1.smithi190.stdout:1/971: dread d3/da/dd/d95/d52/f93 [0,4194304] 0 2024-03-26T18:54:43.651 INFO:tasks.workunit.client.0.smithi102.stdout:5/987: unlink d2/l18 0 2024-03-26T18:54:43.651 INFO:tasks.workunit.client.0.smithi102.stdout:5/988: read - d2/d25/db0/d139/f142 zero size 2024-03-26T18:54:43.663 INFO:tasks.workunit.client.0.smithi102.stdout:3/950: rmdir df/d29/d31 39 2024-03-26T18:54:43.663 INFO:tasks.workunit.client.0.smithi102.stdout:3/951: write df/d29/d31/dc0/f153 [751460,16120] 0 2024-03-26T18:54:43.664 INFO:tasks.workunit.client.0.smithi102.stdout:5/989: truncate d2/d14/f70 903610 0 2024-03-26T18:54:43.664 INFO:tasks.workunit.client.0.smithi102.stdout:3/952: mknod df/d29/d31/c159 0 2024-03-26T18:54:43.665 INFO:tasks.workunit.client.0.smithi102.stdout:5/990: creat d2/d25/d75/def/dec/f157 x:0 0 0 2024-03-26T18:54:43.666 INFO:tasks.workunit.client.0.smithi102.stdout:3/953: truncate df/d29/d31/d36/ffb 3190224 0 2024-03-26T18:54:43.667 INFO:tasks.workunit.client.0.smithi102.stdout:5/991: chown d2/d25/d75/db9/l4d 7 1 2024-03-26T18:54:43.667 INFO:tasks.workunit.client.0.smithi102.stdout:5/992: write d2/d25/d5b/d76/f140 [624363,32413] 0 2024-03-26T18:54:43.669 INFO:tasks.workunit.client.0.smithi102.stdout:3/954: truncate df/d29/d31/d36/d3c/d6f/d75/f88 998413 0 2024-03-26T18:54:43.670 INFO:tasks.workunit.client.1.smithi190.stdout:1/972: dwrite d3/da/d2e/d59/f65 [0,4194304] 0 2024-03-26T18:54:43.670 INFO:tasks.workunit.client.0.smithi102.stdout:5/993: mknod d2/dea/db4/d11f/c158 0 2024-03-26T18:54:43.670 INFO:tasks.workunit.client.0.smithi102.stdout:5/994: write d2/f10 [2566792,12416] 0 2024-03-26T18:54:43.671 INFO:tasks.workunit.client.0.smithi102.stdout:3/955: truncate df/d29/d31/d77/d8a/dae/dee/df2/d100/d11a/f134 454341 0 2024-03-26T18:54:43.674 INFO:tasks.workunit.client.0.smithi102.stdout:5/995: readlink d2/d14/d1a/lc1 0 2024-03-26T18:54:43.678 INFO:tasks.workunit.client.1.smithi190.stdout:1/973: mkdir d3/d8f/da7/d135 0 2024-03-26T18:54:43.680 INFO:tasks.workunit.client.1.smithi190.stdout:1/974: creat d3/d9/da3/f136 x:0 0 0 2024-03-26T18:54:43.682 INFO:tasks.workunit.client.1.smithi190.stdout:1/975: read d3/da/dd/d95/d52/f93 [401455,102521] 0 2024-03-26T18:54:43.684 INFO:tasks.workunit.client.1.smithi190.stdout:1/976: creat d3/f137 x:0 0 0 2024-03-26T18:54:43.684 INFO:tasks.workunit.client.1.smithi190.stdout:1/977: mkdir d3/d5f/d138 0 2024-03-26T18:54:43.685 INFO:tasks.workunit.client.0.smithi102.stdout:5/996: write d2/d25/d75/db9/ded/f11d [1277767,45939] 0 2024-03-26T18:54:43.686 INFO:tasks.workunit.client.0.smithi102.stdout:5/997: getdents d2/d25/d75/db9/d43/d9e/d82 0 2024-03-26T18:54:43.695 INFO:tasks.workunit.client.0.smithi102.stdout:3/956: dwrite df/d29/d31/d36/d3c/d6f/d75/f125 [0,4194304] 0 2024-03-26T18:54:43.696 INFO:tasks.workunit.client.0.smithi102.stdout:3/957: chown df/d61/dd6/fe6 1171 1 2024-03-26T18:54:43.698 INFO:tasks.workunit.client.0.smithi102.stdout:5/998: rmdir d2 39 2024-03-26T18:54:43.700 INFO:tasks.workunit.client.0.smithi102.stdout:3/958: mkdir df/d1c/d70/d15a 0 2024-03-26T18:54:43.701 INFO:tasks.workunit.client.0.smithi102.stdout:3/959: symlink df/d61/de5/d128/l15b 0 2024-03-26T18:54:43.724 INFO:tasks.workunit.client.0.smithi102.stdout:1/671: sync 2024-03-26T18:54:43.724 INFO:tasks.workunit.client.0.smithi102.stdout:8/718: sync 2024-03-26T18:54:43.724 INFO:tasks.workunit.client.0.smithi102.stdout:8/719: fsync d0/d15/d5d/dd3/d2c/d2f/f60 0 2024-03-26T18:54:43.724 INFO:tasks.workunit.client.0.smithi102.stdout:8/720: fdatasync d0/d15/d5d/dd3/d2c/f8e 0 2024-03-26T18:54:43.724 INFO:tasks.workunit.client.0.smithi102.stdout:1/672: creat de/d16/d1d/d2a/d2c/d56/d6e/fe1 x:0 0 0 2024-03-26T18:54:43.725 INFO:tasks.workunit.client.0.smithi102.stdout:1/673: fdatasync de/d16/d1d/d61/f7c 0 2024-03-26T18:54:43.728 INFO:tasks.workunit.client.0.smithi102.stdout:5/999: dwrite d2/d14/d1a/f50 [0,4194304] 0 2024-03-26T18:54:43.728 INFO:tasks.workunit.client.0.smithi102.stdout:8/721: unlink d0/d15/d5d/dd3/d2c/d2f/f78 0 2024-03-26T18:54:43.729 INFO:tasks.workunit.client.0.smithi102.stdout:8/722: read d0/d3c/d6f/f8b [1338830,90479] 0 2024-03-26T18:54:43.729 INFO:tasks.workunit.client.0.smithi102.stdout:3/960: dwrite df/d8b/f122 [0,4194304] 0 2024-03-26T18:54:43.729 INFO:tasks.workunit.client.0.smithi102.stdout:3/961: creat df/d1c/d34/f15c x:0 0 0 2024-03-26T18:54:43.729 INFO:tasks.workunit.client.0.smithi102.stdout:3/962: write df/fdc [660826,109440] 0 2024-03-26T18:54:43.729 INFO:tasks.workunit.client.0.smithi102.stdout:1/674: mknod de/d2e/d62/ce2 0 2024-03-26T18:54:43.734 INFO:tasks.workunit.client.0.smithi102.stdout:3/963: rename df/d1c/d34/da5/fd3 to df/d8b/dbb/f15d 0 2024-03-26T18:54:43.735 INFO:tasks.workunit.client.0.smithi102.stdout:1/675: rename de/d16/d1d/d61/d6a/da5 to de/d1f/de3 0 2024-03-26T18:54:43.735 INFO:tasks.workunit.client.0.smithi102.stdout:3/964: getdents df/d29/d31/d36/d45/daf 0 2024-03-26T18:54:43.737 INFO:tasks.workunit.client.0.smithi102.stdout:1/676: creat de/d1f/de3/fe4 x:0 0 0 2024-03-26T18:54:43.737 INFO:tasks.workunit.client.0.smithi102.stdout:1/677: readlink de/d1f/l25 0 2024-03-26T18:54:43.738 INFO:tasks.workunit.client.0.smithi102.stdout:3/965: mkdir df/d29/d31/d77/d8a/dae/d15e 0 2024-03-26T18:54:43.738 INFO:tasks.workunit.client.0.smithi102.stdout:3/966: write df/d1c/d34/d9b/f138 [1013289,5816] 0 2024-03-26T18:54:43.740 INFO:tasks.workunit.client.0.smithi102.stdout:1/678: mkdir de/d1f/de3/da6/dca/dda/de5 0 2024-03-26T18:54:43.740 INFO:tasks.workunit.client.0.smithi102.stdout:1/679: write de/d16/d1d/d61/d6a/fd1 [803038,62653] 0 2024-03-26T18:54:43.740 INFO:tasks.workunit.client.0.smithi102.stdout:1/680: write de/d1f/f33 [2412287,49954] 0 2024-03-26T18:54:43.740 INFO:tasks.workunit.client.0.smithi102.stdout:1/681: write de/d16/d1d/d2a/d2c/d56/d6e/fe1 [769582,116236] 0 2024-03-26T18:54:43.742 INFO:tasks.workunit.client.0.smithi102.stdout:1/682: mkdir de/d16/d1d/d2a/d2c/de6 0 2024-03-26T18:54:43.742 INFO:tasks.workunit.client.0.smithi102.stdout:8/723: dwrite d0/d15/d5d/dd3/d2c/da7/f29 [0,4194304] 0 2024-03-26T18:54:43.742 INFO:tasks.workunit.client.0.smithi102.stdout:8/724: creat d0/d3c/d96/d98/dcf/fee x:0 0 0 2024-03-26T18:54:43.744 INFO:tasks.workunit.client.0.smithi102.stdout:1/683: mknod de/d16/d1d/d2a/ce7 0 2024-03-26T18:54:43.744 INFO:tasks.workunit.client.0.smithi102.stdout:1/684: write de/d16/d1d/d61/fae [198440,92109] 0 2024-03-26T18:54:43.746 INFO:tasks.workunit.client.0.smithi102.stdout:1/685: symlink de/d2e/le8 0 2024-03-26T18:54:43.748 INFO:tasks.workunit.client.0.smithi102.stdout:1/686: creat de/d16/d1d/d2a/d2c/d56/d5f/fe9 x:0 0 0 2024-03-26T18:54:43.749 INFO:tasks.workunit.client.0.smithi102.stdout:1/687: creat de/fea x:0 0 0 2024-03-26T18:54:43.769 INFO:tasks.workunit.client.0.smithi102.stdout:3/967: dwrite df/d29/d31/d36/d45/f121 [0,4194304] 0 2024-03-26T18:54:43.769 INFO:tasks.workunit.client.0.smithi102.stdout:1/688: dwrite de/d16/d1d/d2a/d2c/f3c [0,4194304] 0 2024-03-26T18:54:43.770 INFO:tasks.workunit.client.0.smithi102.stdout:8/725: dwrite d0/d15/d5d/dd3/d2c/da7/fc5 [0,4194304] 0 2024-03-26T18:54:43.770 INFO:tasks.workunit.client.0.smithi102.stdout:8/726: readlink d0/d15/d5d/dd3/d26/d80/le5 0 2024-03-26T18:54:43.771 INFO:tasks.workunit.client.0.smithi102.stdout:3/968: truncate df/d61/dd6/fe6 2584821 0 2024-03-26T18:54:43.774 INFO:tasks.workunit.client.0.smithi102.stdout:1/689: creat de/d16/d1d/d2a/d2c/d56/d78/feb x:0 0 0 2024-03-26T18:54:43.781 INFO:tasks.workunit.client.0.smithi102.stdout:3/969: dread df/d29/d31/d36/ffb [0,4194304] 0 2024-03-26T18:54:43.787 INFO:tasks.workunit.client.0.smithi102.stdout:8/727: rename d0/d15/l25 to d0/d3c/d96/dd5/lef 0 2024-03-26T18:54:43.787 INFO:tasks.workunit.client.0.smithi102.stdout:8/728: fsync d0/d9/fb5 0 2024-03-26T18:54:43.787 INFO:tasks.workunit.client.0.smithi102.stdout:1/690: mkdir de/d1f/de3/dec 0 2024-03-26T18:54:43.787 INFO:tasks.workunit.client.0.smithi102.stdout:1/691: getdents de/d1f/de3/dd6 0 2024-03-26T18:54:43.789 INFO:tasks.workunit.client.0.smithi102.stdout:8/729: creat d0/d15/d5d/dd3/d26/d19/d76/ff0 x:0 0 0 2024-03-26T18:54:43.789 INFO:tasks.workunit.client.0.smithi102.stdout:1/692: mkdir de/d16/d1d/d2a/d2c/d56/d78/de0/ded 0 2024-03-26T18:54:43.790 INFO:tasks.workunit.client.0.smithi102.stdout:1/693: readlink de/d1f/l25 0 2024-03-26T18:54:43.791 INFO:tasks.workunit.client.0.smithi102.stdout:3/970: write df/d29/d31/d36/d3c/d6f/fda [579642,97293] 0 2024-03-26T18:54:43.791 INFO:tasks.workunit.client.0.smithi102.stdout:3/971: chown df/d61/de5/d128/f14c 16259 1 2024-03-26T18:54:43.792 INFO:tasks.workunit.client.0.smithi102.stdout:8/730: rename d0/d15/d5d/dd3/fd1 to d0/d15/d5d/dd3/d26/d80/ff1 0 2024-03-26T18:54:43.794 INFO:tasks.workunit.client.0.smithi102.stdout:3/972: creat df/d29/d3a/d93/d120/f15f x:0 0 0 2024-03-26T18:54:43.794 INFO:tasks.workunit.client.0.smithi102.stdout:3/973: write df/d29/d31/d36/d45/daf/d105/f12c [135853,100758] 0 2024-03-26T18:54:43.795 INFO:tasks.workunit.client.0.smithi102.stdout:8/731: mknod d0/d15/d5d/dd3/cf2 0 2024-03-26T18:54:43.795 INFO:tasks.workunit.client.0.smithi102.stdout:8/732: readlink d0/d15/d5d/dd3/d2c/da7/l30 0 2024-03-26T18:54:43.799 INFO:tasks.workunit.client.0.smithi102.stdout:8/733: getdents d0/d15/d5d/dd3/d2c/d2f 0 2024-03-26T18:54:43.799 INFO:tasks.workunit.client.0.smithi102.stdout:8/734: write d0/d15/d5d/dd3/d2c/d2f/fa6 [714775,83691] 0 2024-03-26T18:54:43.799 INFO:tasks.workunit.client.0.smithi102.stdout:8/735: stat d0/d3c/d96/d98/dcf 0 2024-03-26T18:54:43.806 INFO:tasks.workunit.client.0.smithi102.stdout:1/694: dwrite de/d2e/fa8 [4194304,4194304] 0 2024-03-26T18:54:43.806 INFO:tasks.workunit.client.0.smithi102.stdout:1/695: stat de/d16/d1d/d61/d6a/fdb 0 2024-03-26T18:54:43.807 INFO:tasks.workunit.client.0.smithi102.stdout:6/689: sync 2024-03-26T18:54:43.807 INFO:tasks.workunit.client.0.smithi102.stdout:9/639: sync 2024-03-26T18:54:43.807 INFO:tasks.workunit.client.0.smithi102.stdout:4/549: sync 2024-03-26T18:54:43.808 INFO:tasks.workunit.client.0.smithi102.stdout:2/695: sync 2024-03-26T18:54:43.808 INFO:tasks.workunit.client.0.smithi102.stdout:2/696: fdatasync d3/d82/d93/d9e/d99/f9a 0 2024-03-26T18:54:43.808 INFO:tasks.workunit.client.0.smithi102.stdout:2/697: stat d3/d11/d15/d46/d7b/d97 0 2024-03-26T18:54:43.808 INFO:tasks.workunit.client.0.smithi102.stdout:0/741: sync 2024-03-26T18:54:43.812 INFO:tasks.workunit.client.1.smithi190.stdout:6/913: sync 2024-03-26T18:54:43.814 INFO:tasks.workunit.client.0.smithi102.stdout:8/736: rmdir d0 39 2024-03-26T18:54:43.815 INFO:tasks.workunit.client.1.smithi190.stdout:6/914: creat d17/d28/d2c/db8/dcc/f135 x:0 0 0 2024-03-26T18:54:43.816 INFO:tasks.workunit.client.0.smithi102.stdout:0/742: dread d6/de/f4b [0,4194304] 0 2024-03-26T18:54:43.816 INFO:tasks.workunit.client.0.smithi102.stdout:0/743: chown d6/de/d13/d25/d31/d45/d82/d84/d88/fa6 8 1 2024-03-26T18:54:43.817 INFO:tasks.workunit.client.0.smithi102.stdout:1/696: creat de/d16/d1d/d61/fee x:0 0 0 2024-03-26T18:54:43.817 INFO:tasks.workunit.client.0.smithi102.stdout:1/697: truncate de/d16/d1d/d2a/d2c/d56/fb3 1034760 0 2024-03-26T18:54:43.817 INFO:tasks.workunit.client.0.smithi102.stdout:1/698: read - de/d16/d1d/d61/d6a/fdb zero size 2024-03-26T18:54:43.818 INFO:tasks.workunit.client.0.smithi102.stdout:6/690: chown d2/fca 0 1 2024-03-26T18:54:43.819 INFO:tasks.workunit.client.1.smithi190.stdout:6/915: creat d17/d28/d8b/d131/f136 x:0 0 0 2024-03-26T18:54:43.820 INFO:tasks.workunit.client.0.smithi102.stdout:9/640: symlink df/d1a/d23/le9 0 2024-03-26T18:54:43.820 INFO:tasks.workunit.client.0.smithi102.stdout:9/641: dread - df/d1a/d7f/fad zero size 2024-03-26T18:54:43.820 INFO:tasks.workunit.client.0.smithi102.stdout:9/642: write df/d1a/d1d/d72/fbf [4744546,30085] 0 2024-03-26T18:54:43.820 INFO:tasks.workunit.client.0.smithi102.stdout:3/974: dwrite df/d29/d31/d36/f41 [0,4194304] 0 2024-03-26T18:54:43.820 INFO:tasks.workunit.client.0.smithi102.stdout:3/975: fdatasync df/d1c/d34/f37 0 2024-03-26T18:54:43.821 INFO:tasks.workunit.client.0.smithi102.stdout:3/976: readlink df/l14 0 2024-03-26T18:54:43.821 INFO:tasks.workunit.client.0.smithi102.stdout:2/698: mkdir d3/d11/d15/d31/d7f/dfe 0 2024-03-26T18:54:43.822 INFO:tasks.workunit.client.1.smithi190.stdout:6/916: unlink d17/d28/d2c/f4c 0 2024-03-26T18:54:43.822 INFO:tasks.workunit.client.0.smithi102.stdout:6/691: rmdir d2/df/d35 39 2024-03-26T18:54:43.822 INFO:tasks.workunit.client.0.smithi102.stdout:6/692: write d2/df/d35/d43/fd5 [216771,821] 0 2024-03-26T18:54:43.823 INFO:tasks.workunit.client.0.smithi102.stdout:2/699: dread d3/d20/d78/d7c/f7e [0,4194304] 0 2024-03-26T18:54:43.824 INFO:tasks.workunit.client.1.smithi190.stdout:6/917: mknod d17/d28/d74/d7b/c137 0 2024-03-26T18:54:43.824 INFO:tasks.workunit.client.1.smithi190.stdout:6/918: read - d17/d28/d74/d97/fcb zero size 2024-03-26T18:54:43.825 INFO:tasks.workunit.client.0.smithi102.stdout:0/744: write d6/de/d13/d1f/d55/d57/f5e [396080,66501] 0 2024-03-26T18:54:43.826 INFO:tasks.workunit.client.0.smithi102.stdout:9/643: symlink df/d46/lea 0 2024-03-26T18:54:43.826 INFO:tasks.workunit.client.0.smithi102.stdout:9/644: fdatasync df/d46/f67 0 2024-03-26T18:54:43.828 INFO:tasks.workunit.client.0.smithi102.stdout:3/977: rename df/d29/d31/d36/d45/daf/f12a to df/d29/d31/d77/f160 0 2024-03-26T18:54:43.828 INFO:tasks.workunit.client.0.smithi102.stdout:7/599: sync 2024-03-26T18:54:43.828 INFO:tasks.workunit.client.0.smithi102.stdout:7/600: creat d7/d10/d67/fb2 x:0 0 0 2024-03-26T18:54:43.828 INFO:tasks.workunit.client.0.smithi102.stdout:7/601: fdatasync d7/d10/d17/f5c 0 2024-03-26T18:54:43.829 INFO:tasks.workunit.client.1.smithi190.stdout:6/919: creat d17/d28/d2c/db8/df3/d109/f138 x:0 0 0 2024-03-26T18:54:43.830 INFO:tasks.workunit.client.0.smithi102.stdout:2/700: creat d3/d11/d27/dc0/fff x:0 0 0 2024-03-26T18:54:43.832 INFO:tasks.workunit.client.1.smithi190.stdout:6/920: truncate d17/dd3/d113/f11c 733031 0 2024-03-26T18:54:43.834 INFO:tasks.workunit.client.0.smithi102.stdout:9/645: rename df/d1a/d23/d60/ce5 to df/d1a/d23/d60/ceb 0 2024-03-26T18:54:43.836 INFO:tasks.workunit.client.0.smithi102.stdout:0/745: rename d6/de/d13/d25/d31/c98 to d6/de/d13/d25/d4f/c101 0 2024-03-26T18:54:43.836 INFO:tasks.workunit.client.0.smithi102.stdout:0/746: creat d6/de/d13/d1f/d71/f102 x:0 0 0 2024-03-26T18:54:43.838 INFO:tasks.workunit.client.0.smithi102.stdout:4/550: dread d1/d27/f5f [0,4194304] 0 2024-03-26T18:54:43.839 INFO:tasks.workunit.client.0.smithi102.stdout:4/551: chown d1/d17/d1a/fa1 2 1 2024-03-26T18:54:43.840 INFO:tasks.workunit.client.0.smithi102.stdout:7/602: creat d7/d10/d27/d34/d72/fb3 x:0 0 0 2024-03-26T18:54:43.840 INFO:tasks.workunit.client.0.smithi102.stdout:7/603: write d7/d44/f53 [860307,58004] 0 2024-03-26T18:54:43.840 INFO:tasks.workunit.client.0.smithi102.stdout:7/604: write d7/ff [1949768,96950] 0 2024-03-26T18:54:43.840 INFO:tasks.workunit.client.0.smithi102.stdout:6/693: write d2/df/d72/db7/f95 [496655,81578] 0 2024-03-26T18:54:43.841 INFO:tasks.workunit.client.1.smithi190.stdout:6/921: mkdir d17/d28/d2c/db8/df3/d109/d79/ddf/d139 0 2024-03-26T18:54:43.842 INFO:tasks.workunit.client.0.smithi102.stdout:9/646: creat df/d1a/d65/fec x:0 0 0 2024-03-26T18:54:43.844 INFO:tasks.workunit.client.1.smithi190.stdout:6/922: symlink d17/d28/d8b/d101/l13a 0 2024-03-26T18:54:43.844 INFO:tasks.workunit.client.1.smithi190.stdout:6/923: chown d17/d28/d8b/d101 37422277 1 2024-03-26T18:54:43.847 INFO:tasks.workunit.client.0.smithi102.stdout:4/552: unlink d1/d27/f4d 0 2024-03-26T18:54:43.848 INFO:tasks.workunit.client.0.smithi102.stdout:8/737: dwrite d0/d15/d5d/dd3/d2c/d33/d87/f8c [0,4194304] 0 2024-03-26T18:54:43.848 INFO:tasks.workunit.client.0.smithi102.stdout:8/738: write d0/d15/d5d/d89/fc6 [655440,1061] 0 2024-03-26T18:54:43.850 INFO:tasks.workunit.client.0.smithi102.stdout:0/747: dread d6/d2d/f38 [0,4194304] 0 2024-03-26T18:54:43.850 INFO:tasks.workunit.client.0.smithi102.stdout:0/748: fsync d6/de/d13/d1f/f27 0 2024-03-26T18:54:43.850 INFO:tasks.workunit.client.0.smithi102.stdout:7/605: creat d7/d31/d79/fb4 x:0 0 0 2024-03-26T18:54:43.851 INFO:tasks.workunit.client.0.smithi102.stdout:7/606: read - d7/d31/f45 zero size 2024-03-26T18:54:43.851 INFO:tasks.workunit.client.1.smithi190.stdout:6/924: mknod d17/c13b 0 2024-03-26T18:54:43.851 INFO:tasks.workunit.client.0.smithi102.stdout:0/749: read d6/de/d13/d25/d31/d45/d82/d84/d88/fa6 [147246,85453] 0 2024-03-26T18:54:43.852 INFO:tasks.workunit.client.0.smithi102.stdout:4/553: creat d1/d25/d90/fab x:0 0 0 2024-03-26T18:54:43.853 INFO:tasks.workunit.client.0.smithi102.stdout:1/699: dwrite de/d16/d1d/d61/fee [0,4194304] 0 2024-03-26T18:54:43.853 INFO:tasks.workunit.client.0.smithi102.stdout:1/700: chown la 18527 1 2024-03-26T18:54:43.854 INFO:tasks.workunit.client.0.smithi102.stdout:8/739: getdents d0/d3c/d96 0 2024-03-26T18:54:43.856 INFO:tasks.workunit.client.0.smithi102.stdout:7/607: mknod d7/d12/d2b/da8/cb5 0 2024-03-26T18:54:43.856 INFO:tasks.workunit.client.0.smithi102.stdout:7/608: fdatasync d7/d12/d4d/f88 0 2024-03-26T18:54:43.856 INFO:tasks.workunit.client.0.smithi102.stdout:7/609: fsync d7/d10/d27/d34/d72/f76 0 2024-03-26T18:54:43.856 INFO:tasks.workunit.client.0.smithi102.stdout:4/554: stat d1/d17/d1a/d21/d77/c70 0 2024-03-26T18:54:43.859 INFO:tasks.workunit.client.0.smithi102.stdout:4/555: creat d1/d17/d1f/d5d/d6f/fac x:0 0 0 2024-03-26T18:54:43.860 INFO:tasks.workunit.client.0.smithi102.stdout:4/556: symlink d1/d27/d7a/lad 0 2024-03-26T18:54:43.863 INFO:tasks.workunit.client.0.smithi102.stdout:9/647: dwrite df/d1a/d23/fb8 [0,4194304] 0 2024-03-26T18:54:43.864 INFO:tasks.workunit.client.0.smithi102.stdout:0/750: dwrite d6/d3e/d87/fea [0,4194304] 0 2024-03-26T18:54:43.865 INFO:tasks.workunit.client.0.smithi102.stdout:0/751: chown d6/de/d13/d25/d31/d45/f65 4297558 1 2024-03-26T18:54:43.868 INFO:tasks.workunit.client.1.smithi190.stdout:6/925: dread d17/d28/d2c/d50/f7e [0,4194304] 0 2024-03-26T18:54:43.868 INFO:tasks.workunit.client.0.smithi102.stdout:3/978: rename df/d29/d31/d77/f160 to df/d1c/d34/da5/dce/d149/f161 0 2024-03-26T18:54:43.874 INFO:tasks.workunit.client.1.smithi190.stdout:6/926: fsync d17/d28/f11b 0 2024-03-26T18:54:43.874 INFO:tasks.workunit.client.1.smithi190.stdout:6/927: write d17/d28/d2c/d77/f6f [4605595,55137] 0 2024-03-26T18:54:43.874 INFO:tasks.workunit.client.1.smithi190.stdout:6/928: write d17/d28/d8b/f116 [591069,17918] 0 2024-03-26T18:54:43.874 INFO:tasks.workunit.client.1.smithi190.stdout:6/929: creat d17/d28/d2c/d63/d10e/f13c x:0 0 0 2024-03-26T18:54:43.874 INFO:tasks.workunit.client.0.smithi102.stdout:9/648: getdents df/d1a/d23/d60/dd4 0 2024-03-26T18:54:43.876 INFO:tasks.workunit.client.0.smithi102.stdout:8/740: dread d0/d15/d5d/dac/fc2 [0,4194304] 0 2024-03-26T18:54:43.877 INFO:tasks.workunit.client.0.smithi102.stdout:2/701: rename d3/d11/fdc to d3/d82/d93/d9e/d1a/d25/f100 0 2024-03-26T18:54:43.878 INFO:tasks.workunit.client.0.smithi102.stdout:0/752: getdents d6/de/d13/d25/d31 0 2024-03-26T18:54:43.878 INFO:tasks.workunit.client.0.smithi102.stdout:0/753: write d6/d2d/d3d/f44 [9090671,18021] 0 2024-03-26T18:54:43.883 INFO:tasks.workunit.client.0.smithi102.stdout:8/741: creat d0/d9/ff3 x:0 0 0 2024-03-26T18:54:43.886 INFO:tasks.workunit.client.0.smithi102.stdout:4/557: dwrite d1/f26 [4194304,4194304] 0 2024-03-26T18:54:43.886 INFO:tasks.workunit.client.0.smithi102.stdout:4/558: dread - d1/d17/d1a/fa1 zero size 2024-03-26T18:54:43.888 INFO:tasks.workunit.client.0.smithi102.stdout:6/694: rename d2/df/d35/d60/l82 to d2/df/d71/d80/d91/dc1/le1 0 2024-03-26T18:54:43.888 INFO:tasks.workunit.client.1.smithi190.stdout:6/930: dread d17/d28/df6/da8/df4/dfb/f114 [0,4194304] 0 2024-03-26T18:54:43.890 INFO:tasks.workunit.client.0.smithi102.stdout:9/649: mkdir df/ded 0 2024-03-26T18:54:43.893 INFO:tasks.workunit.client.0.smithi102.stdout:7/610: rename d7/f9 to d7/d12/d51/d6a/fb6 0 2024-03-26T18:54:43.895 INFO:tasks.workunit.client.0.smithi102.stdout:3/979: dwrite df/d1c/d34/d53/fdd [0,4194304] 0 2024-03-26T18:54:43.896 INFO:tasks.workunit.client.1.smithi190.stdout:6/931: rename d17/d28/d2c/d63/c104 to d17/d28/d2c/d63/c13d 0 2024-03-26T18:54:43.896 INFO:tasks.workunit.client.1.smithi190.stdout:6/932: fdatasync d17/d28/d2c/d50/f9c 0 2024-03-26T18:54:43.896 INFO:tasks.workunit.client.1.smithi190.stdout:6/933: fsync d17/d28/df6/da8/df4/dfb/d6a/fa4 0 2024-03-26T18:54:43.897 INFO:tasks.workunit.client.0.smithi102.stdout:2/702: getdents d3/d2a 0 2024-03-26T18:54:43.897 INFO:tasks.workunit.client.0.smithi102.stdout:2/703: dread - d3/d74/fe9 zero size 2024-03-26T18:54:43.897 INFO:tasks.workunit.client.0.smithi102.stdout:2/704: read - d3/d82/d93/fbf zero size 2024-03-26T18:54:43.898 INFO:tasks.workunit.client.0.smithi102.stdout:9/650: creat df/d1a/d65/dbe/ddf/fee x:0 0 0 2024-03-26T18:54:43.900 INFO:tasks.workunit.client.1.smithi190.stdout:6/934: mknod d17/d28/d74/c13e 0 2024-03-26T18:54:43.900 INFO:tasks.workunit.client.0.smithi102.stdout:7/611: getdents d7/d12/d2b/da8 0 2024-03-26T18:54:43.902 INFO:tasks.workunit.client.0.smithi102.stdout:4/559: dread d1/d17/d1f/d5d/d8b/f66 [0,4194304] 0 2024-03-26T18:54:43.902 INFO:tasks.workunit.client.0.smithi102.stdout:1/701: write de/d16/d1d/d2a/d2c/d43/f86 [803290,122999] 0 2024-03-26T18:54:43.902 INFO:tasks.workunit.client.0.smithi102.stdout:1/702: fsync de/d1f/d9f/fd8 0 2024-03-26T18:54:43.904 INFO:tasks.workunit.client.0.smithi102.stdout:6/695: dwrite d2/d12/f2d [0,4194304] 0 2024-03-26T18:54:43.908 INFO:tasks.workunit.client.0.smithi102.stdout:3/980: rename df/d8b/dbb/c11c to df/d29/d31/dc0/c162 0 2024-03-26T18:54:43.909 INFO:tasks.workunit.client.0.smithi102.stdout:9/651: symlink df/d1a/d65/lef 0 2024-03-26T18:54:43.910 INFO:tasks.workunit.client.0.smithi102.stdout:7/612: creat d7/d10/d27/d34/d82/fb7 x:0 0 0 2024-03-26T18:54:43.912 INFO:tasks.workunit.client.0.smithi102.stdout:4/560: creat d1/d17/fae x:0 0 0 2024-03-26T18:54:43.912 INFO:tasks.workunit.client.0.smithi102.stdout:4/561: creat d1/d17/d1f/d5d/d6f/faf x:0 0 0 2024-03-26T18:54:43.914 INFO:tasks.workunit.client.0.smithi102.stdout:1/703: link de/d16/d1d/f82 de/d1f/de3/dd6/fef 0 2024-03-26T18:54:43.915 INFO:tasks.workunit.client.0.smithi102.stdout:1/704: fsync de/d16/d1d/d61/fae 0 2024-03-26T18:54:43.915 INFO:tasks.workunit.client.0.smithi102.stdout:6/696: write d2/d7/f8 [6362967,81800] 0 2024-03-26T18:54:43.918 INFO:tasks.workunit.client.0.smithi102.stdout:9/652: symlink df/d1a/d81/lf0 0 2024-03-26T18:54:43.921 INFO:tasks.workunit.client.0.smithi102.stdout:7/613: link d7/d10/d17/l23 d7/d10/d67/d69/d78/lb8 0 2024-03-26T18:54:43.924 INFO:tasks.workunit.client.0.smithi102.stdout:2/705: dwrite d3/d2a/faf [0,4194304] 0 2024-03-26T18:54:43.925 INFO:tasks.workunit.client.0.smithi102.stdout:2/706: chown d3/d2a/fb7 47498533 1 2024-03-26T18:54:43.926 INFO:tasks.workunit.client.0.smithi102.stdout:4/562: mknod d1/d27/cb0 0 2024-03-26T18:54:43.929 INFO:tasks.workunit.client.0.smithi102.stdout:1/705: symlink de/d16/d1d/d2a/d2c/d56/d78/de0/ded/lf0 0 2024-03-26T18:54:43.929 INFO:tasks.workunit.client.0.smithi102.stdout:1/706: write de/d16/d1d/d2a/f9e [1117152,17824] 0 2024-03-26T18:54:43.935 INFO:tasks.workunit.client.0.smithi102.stdout:4/563: mknod d1/d17/d1f/d5d/d6f/cb1 0 2024-03-26T18:54:43.938 INFO:tasks.workunit.client.0.smithi102.stdout:1/707: write de/fac [6441948,61485] 0 2024-03-26T18:54:43.938 INFO:tasks.workunit.client.0.smithi102.stdout:4/564: link d1/d17/d1f/d5d/d6f/l84 d1/d17/d1f/d5d/d8b/lb2 0 2024-03-26T18:54:43.941 INFO:tasks.workunit.client.0.smithi102.stdout:1/708: rename de/d16/d1d/d2a/d2c/l84 to de/d16/d1d/d2a/d2c/d56/d5f/daa/lf1 0 2024-03-26T18:54:43.941 INFO:tasks.workunit.client.0.smithi102.stdout:1/709: chown de/d1f/f6d 15679759 1 2024-03-26T18:54:43.941 INFO:tasks.workunit.client.0.smithi102.stdout:4/565: mkdir d1/d25/db3 0 2024-03-26T18:54:43.954 INFO:tasks.workunit.client.0.smithi102.stdout:4/566: dread d1/d17/d1a/f53 [0,4194304] 0 2024-03-26T18:54:43.954 INFO:tasks.workunit.client.0.smithi102.stdout:4/567: dread - d1/fa8 zero size 2024-03-26T18:54:43.955 INFO:tasks.workunit.client.0.smithi102.stdout:4/568: mknod d1/d17/d1f/d5d/cb4 0 2024-03-26T18:54:43.955 INFO:tasks.workunit.client.0.smithi102.stdout:4/569: stat d1/d17/d1f/d5d/ca7 0 2024-03-26T18:54:43.955 INFO:tasks.workunit.client.0.smithi102.stdout:4/570: fsync d1/d25/d76/f86 0 2024-03-26T18:54:43.956 INFO:tasks.workunit.client.0.smithi102.stdout:9/653: dwrite df/d1a/d65/f93 [0,4194304] 0 2024-03-26T18:54:43.956 INFO:tasks.workunit.client.0.smithi102.stdout:7/614: dwrite d7/d2a/f74 [0,4194304] 0 2024-03-26T18:54:43.956 INFO:tasks.workunit.client.0.smithi102.stdout:7/615: write d7/d12/d4d/d62/f66 [2492628,20255] 0 2024-03-26T18:54:43.956 INFO:tasks.workunit.client.0.smithi102.stdout:7/616: write d7/d12/f3c [4144485,62884] 0 2024-03-26T18:54:43.959 INFO:tasks.workunit.client.0.smithi102.stdout:2/707: dwrite d3/d20/d78/d7c/dca/ff2 [0,4194304] 0 2024-03-26T18:54:43.959 INFO:tasks.workunit.client.0.smithi102.stdout:9/654: symlink df/d1a/d65/dbe/ddf/lf1 0 2024-03-26T18:54:43.961 INFO:tasks.workunit.client.0.smithi102.stdout:9/655: dread df/d1a/d23/d3e/f99 [0,4194304] 0 2024-03-26T18:54:43.963 INFO:tasks.workunit.client.0.smithi102.stdout:3/981: write df/d29/d31/d77/d8a/fd2 [872163,39435] 0 2024-03-26T18:54:43.965 INFO:tasks.workunit.client.0.smithi102.stdout:7/617: dread d7/d10/d17/f35 [8388608,4194304] 0 2024-03-26T18:54:43.965 INFO:tasks.workunit.client.0.smithi102.stdout:7/618: stat d7/d10/d17/f81 0 2024-03-26T18:54:43.965 INFO:tasks.workunit.client.0.smithi102.stdout:7/619: write d7/d10/d17/f81 [3153123,85085] 0 2024-03-26T18:54:43.969 INFO:tasks.workunit.client.0.smithi102.stdout:1/710: rename de/d16/d1d/d2a/d2c/d56/d78/de0/ded to de/d16/d1d/d61/d6a/df2 0 2024-03-26T18:54:43.973 INFO:tasks.workunit.client.0.smithi102.stdout:2/708: symlink d3/d2d/d67/l101 0 2024-03-26T18:54:43.975 INFO:tasks.workunit.client.0.smithi102.stdout:9/656: symlink df/d1a/d23/d4e/d8e/lf2 0 2024-03-26T18:54:43.976 INFO:tasks.workunit.client.0.smithi102.stdout:4/571: dwrite d1/d17/d1a/d21/f2c [0,4194304] 0 2024-03-26T18:54:43.978 INFO:tasks.workunit.client.0.smithi102.stdout:7/620: mkdir d7/d10/d17/db9 0 2024-03-26T18:54:43.979 INFO:tasks.workunit.client.0.smithi102.stdout:1/711: symlink de/d1f/d9f/lf3 0 2024-03-26T18:54:43.979 INFO:tasks.workunit.client.0.smithi102.stdout:1/712: write de/d16/d1d/d2a/d2c/d56/d6e/fde [297988,74384] 0 2024-03-26T18:54:43.979 INFO:tasks.workunit.client.1.smithi190.stdout:5/972: sync 2024-03-26T18:54:43.979 INFO:tasks.workunit.client.1.smithi190.stdout:3/919: sync 2024-03-26T18:54:43.979 INFO:tasks.workunit.client.1.smithi190.stdout:1/978: sync 2024-03-26T18:54:43.980 INFO:tasks.workunit.client.1.smithi190.stdout:5/973: chown d2/d17/d1b/d8e/leb 1008498 1 2024-03-26T18:54:43.980 INFO:tasks.workunit.client.0.smithi102.stdout:2/709: creat d3/f102 x:0 0 0 2024-03-26T18:54:43.981 INFO:tasks.workunit.client.0.smithi102.stdout:9/657: truncate df/d1a/d23/d60/f98 976151 0 2024-03-26T18:54:43.982 INFO:tasks.workunit.client.0.smithi102.stdout:4/572: mknod d1/d6b/cb5 0 2024-03-26T18:54:43.983 INFO:tasks.workunit.client.1.smithi190.stdout:5/974: mkdir d2/d17/d62/dd7/d148 0 2024-03-26T18:54:43.984 INFO:tasks.workunit.client.0.smithi102.stdout:1/713: unlink de/d16/d1d/d2a/d2c/d56/d6e/f89 0 2024-03-26T18:54:43.985 INFO:tasks.workunit.client.0.smithi102.stdout:1/714: creat de/d2e/d62/ff4 x:0 0 0 2024-03-26T18:54:43.985 INFO:tasks.workunit.client.0.smithi102.stdout:1/715: chown de/d16/f51 658 1 2024-03-26T18:54:43.985 INFO:tasks.workunit.client.0.smithi102.stdout:1/716: dread - de/d2e/d62/ff4 zero size 2024-03-26T18:54:43.985 INFO:tasks.workunit.client.1.smithi190.stdout:3/920: rmdir d1/d3/d52/d70/db3 39 2024-03-26T18:54:43.986 INFO:tasks.workunit.client.0.smithi102.stdout:2/710: symlink d3/d82/d93/d9e/d99/l103 0 2024-03-26T18:54:43.987 INFO:tasks.workunit.client.0.smithi102.stdout:9/658: rename ca to df/d1a/d1d/cf3 0 2024-03-26T18:54:43.990 INFO:tasks.workunit.client.0.smithi102.stdout:4/573: rmdir d1/d17/d1f/d5d 39 2024-03-26T18:54:43.992 INFO:tasks.workunit.client.0.smithi102.stdout:3/982: dwrite df/d1c/d34/da5/dce/f112 [0,4194304] 0 2024-03-26T18:54:43.993 INFO:tasks.workunit.client.1.smithi190.stdout:1/979: read d3/da/f47 [3759847,112670] 0 2024-03-26T18:54:43.994 INFO:tasks.workunit.client.0.smithi102.stdout:4/574: dread d1/d17/d1a/d21/f63 [0,4194304] 0 2024-03-26T18:54:43.996 INFO:tasks.workunit.client.0.smithi102.stdout:7/621: dread d7/d12/d51/d6a/f7a [0,4194304] 0 2024-03-26T18:54:43.997 INFO:tasks.workunit.client.0.smithi102.stdout:1/717: creat de/d16/d1d/d2a/d67/ff5 x:0 0 0 2024-03-26T18:54:43.997 INFO:tasks.workunit.client.0.smithi102.stdout:2/711: creat d3/d74/f104 x:0 0 0 2024-03-26T18:54:44.002 INFO:tasks.workunit.client.1.smithi190.stdout:1/980: write d3/da/f24 [4035005,27818] 0 2024-03-26T18:54:44.002 INFO:tasks.workunit.client.1.smithi190.stdout:1/981: creat d3/da/dd/d76/f139 x:0 0 0 2024-03-26T18:54:44.004 INFO:tasks.workunit.client.0.smithi102.stdout:4/575: unlink d1/d27/d98/f9c 0 2024-03-26T18:54:44.006 INFO:tasks.workunit.client.0.smithi102.stdout:7/622: creat d7/d10/d67/d69/d78/fba x:0 0 0 2024-03-26T18:54:44.008 INFO:tasks.workunit.client.1.smithi190.stdout:1/982: rename d3/d9/f3c to d3/da/dd/d2f/d5b/f13a 0 2024-03-26T18:54:44.009 INFO:tasks.workunit.client.0.smithi102.stdout:7/623: mkdir d7/d12/d51/dbb 0 2024-03-26T18:54:44.009 INFO:tasks.workunit.client.0.smithi102.stdout:7/624: creat d7/d12/d51/fbc x:0 0 0 2024-03-26T18:54:44.011 INFO:tasks.workunit.client.1.smithi190.stdout:1/983: rmdir d3/d9/db/db0 39 2024-03-26T18:54:44.011 INFO:tasks.workunit.client.1.smithi190.stdout:1/984: creat d3/f13b x:0 0 0 2024-03-26T18:54:44.017 INFO:tasks.workunit.client.0.smithi102.stdout:7/625: write d7/ff [1099922,27702] 0 2024-03-26T18:54:44.017 INFO:tasks.workunit.client.0.smithi102.stdout:7/626: stat d7/d10/d67/f80 0 2024-03-26T18:54:44.022 INFO:tasks.workunit.client.0.smithi102.stdout:7/627: dread d7/d12/d4d/d62/f66 [0,4194304] 0 2024-03-26T18:54:44.023 INFO:tasks.workunit.client.0.smithi102.stdout:7/628: mkdir d7/d12/d2b/dbd 0 2024-03-26T18:54:44.036 INFO:tasks.workunit.client.0.smithi102.stdout:4/576: dread d1/d17/d1a/f57 [0,4194304] 0 2024-03-26T18:54:44.036 INFO:tasks.workunit.client.0.smithi102.stdout:4/577: readlink d1/d17/d1f/d5d/d8b/l8c 0 2024-03-26T18:54:44.037 INFO:tasks.workunit.client.0.smithi102.stdout:4/578: getdents d1/d17 0 2024-03-26T18:54:44.039 INFO:tasks.workunit.client.0.smithi102.stdout:4/579: rename d1/d17/d1a/d21/l41 to d1/d17/d1a/lb6 0 2024-03-26T18:54:44.040 INFO:tasks.workunit.client.0.smithi102.stdout:2/712: dwrite d3/d11/d15/d46/fa9 [0,4194304] 0 2024-03-26T18:54:44.040 INFO:tasks.workunit.client.0.smithi102.stdout:2/713: dread - d3/d2a/ff7 zero size 2024-03-26T18:54:44.040 INFO:tasks.workunit.client.0.smithi102.stdout:2/714: write d3/d82/d93/d9e/d1a/fc1 [363273,56654] 0 2024-03-26T18:54:44.040 INFO:tasks.workunit.client.0.smithi102.stdout:4/580: mkdir d1/d27/d98/db7 0 2024-03-26T18:54:44.041 INFO:tasks.workunit.client.1.smithi190.stdout:3/921: dwrite d1/de/f69 [0,4194304] 0 2024-03-26T18:54:44.042 INFO:tasks.workunit.client.0.smithi102.stdout:2/715: symlink d3/d20/l105 0 2024-03-26T18:54:44.043 INFO:tasks.workunit.client.1.smithi190.stdout:3/922: mkdir d1/d3/d5a/d6e/d11a 0 2024-03-26T18:54:44.043 INFO:tasks.workunit.client.1.smithi190.stdout:3/923: fsync d1/d3/df/d18/d35/f60 0 2024-03-26T18:54:44.044 INFO:tasks.workunit.client.0.smithi102.stdout:2/716: unlink d3/d82/d93/d9e/da0/cf3 0 2024-03-26T18:54:44.044 INFO:tasks.workunit.client.1.smithi190.stdout:3/924: mkdir d1/d3/df/d18/d35/d41/d11b 0 2024-03-26T18:54:44.045 INFO:tasks.workunit.client.1.smithi190.stdout:3/925: creat d1/d3/df/d18/d57/f11c x:0 0 0 2024-03-26T18:54:44.047 INFO:tasks.workunit.client.0.smithi102.stdout:2/717: getdents d3/d11/d27/dc0 0 2024-03-26T18:54:44.047 INFO:tasks.workunit.client.0.smithi102.stdout:3/983: dwrite df/d29/d31/d36/d45/f60 [4194304,4194304] 0 2024-03-26T18:54:44.048 INFO:tasks.workunit.client.0.smithi102.stdout:1/718: dwrite de/d1f/fc5 [0,4194304] 0 2024-03-26T18:54:44.049 INFO:tasks.workunit.client.0.smithi102.stdout:2/718: getdents d3/d11/d15/d46/d7b/d97 0 2024-03-26T18:54:44.049 INFO:tasks.workunit.client.0.smithi102.stdout:2/719: dread - d3/f102 zero size 2024-03-26T18:54:44.049 INFO:tasks.workunit.client.0.smithi102.stdout:2/720: fdatasync d3/d11/d15/d31/fa3 0 2024-03-26T18:54:44.049 INFO:tasks.workunit.client.0.smithi102.stdout:2/721: stat d3/d82/de2/def/fa2 0 2024-03-26T18:54:44.049 INFO:tasks.workunit.client.0.smithi102.stdout:2/722: write d3/d82/d93/d9e/ff [519637,13515] 0 2024-03-26T18:54:44.053 INFO:tasks.workunit.client.1.smithi190.stdout:3/926: link d1/d3/l17 d1/d3/df/d18/d35/d41/d61/l11d 0 2024-03-26T18:54:44.055 INFO:tasks.workunit.client.0.smithi102.stdout:7/629: dwrite d7/d31/d79/fb4 [0,4194304] 0 2024-03-26T18:54:44.055 INFO:tasks.workunit.client.0.smithi102.stdout:4/581: dread d1/d17/d1a/d21/d77/f71 [0,4194304] 0 2024-03-26T18:54:44.057 INFO:tasks.workunit.client.0.smithi102.stdout:1/719: mknod de/d1f/de3/dd6/cf6 0 2024-03-26T18:54:44.061 INFO:tasks.workunit.client.0.smithi102.stdout:4/582: mknod d1/d17/d1a/d21/cb8 0 2024-03-26T18:54:44.062 INFO:tasks.workunit.client.0.smithi102.stdout:4/583: dread - d1/d17/d1a/d21/f79 zero size 2024-03-26T18:54:44.062 INFO:tasks.workunit.client.0.smithi102.stdout:4/584: truncate d1/d27/f6e 1498690 0 2024-03-26T18:54:44.063 INFO:tasks.workunit.client.0.smithi102.stdout:1/720: mkdir de/d1f/de3/da6/dca/df7 0 2024-03-26T18:54:44.063 INFO:tasks.workunit.client.0.smithi102.stdout:1/721: readlink la 0 2024-03-26T18:54:44.065 INFO:tasks.workunit.client.1.smithi190.stdout:3/927: dread d1/de/d4e/fa0 [0,4194304] 0 2024-03-26T18:54:44.065 INFO:tasks.workunit.client.1.smithi190.stdout:3/928: readlink d1/d3/l17 0 2024-03-26T18:54:44.066 INFO:tasks.workunit.client.0.smithi102.stdout:1/722: rmdir de/d1f/de3/da6/dca/dda/de5 0 2024-03-26T18:54:44.081 INFO:tasks.workunit.client.0.smithi102.stdout:4/585: dread d1/d17/d1f/d5d/d8b/d3e/f58 [0,4194304] 0 2024-03-26T18:54:44.081 INFO:tasks.workunit.client.0.smithi102.stdout:4/586: readlink d1/l1e 0 2024-03-26T18:54:44.082 INFO:tasks.workunit.client.0.smithi102.stdout:4/587: creat d1/d27/d98/fb9 x:0 0 0 2024-03-26T18:54:44.082 INFO:tasks.workunit.client.0.smithi102.stdout:4/588: truncate d1/d17/d1a/f4b 2492744 0 2024-03-26T18:54:44.084 INFO:tasks.workunit.client.0.smithi102.stdout:4/589: rename d1/d17/d1a/c59 to d1/d17/d2a/cba 0 2024-03-26T18:54:44.085 INFO:tasks.workunit.client.0.smithi102.stdout:4/590: unlink d1/d17/d2a/c9e 0 2024-03-26T18:54:44.085 INFO:tasks.workunit.client.0.smithi102.stdout:4/591: read d1/d27/d98/f68 [1047105,90760] 0 2024-03-26T18:54:44.085 INFO:tasks.workunit.client.0.smithi102.stdout:4/592: write d1/d17/d1a/f53 [2720327,35515] 0 2024-03-26T18:54:44.087 INFO:tasks.workunit.client.0.smithi102.stdout:2/723: dwrite d3/d11/d15/d46/fa9 [0,4194304] 0 2024-03-26T18:54:44.087 INFO:tasks.workunit.client.0.smithi102.stdout:2/724: dread - d3/d11/d15/d46/d7b/d97/fbb zero size 2024-03-26T18:54:44.087 INFO:tasks.workunit.client.0.smithi102.stdout:2/725: readlink d3/d11/d15/d46/lde 0 2024-03-26T18:54:44.087 INFO:tasks.workunit.client.0.smithi102.stdout:2/726: fsync d3/d2a/f37 0 2024-03-26T18:54:44.087 INFO:tasks.workunit.client.0.smithi102.stdout:2/727: chown d3/d82/d93/fcc 11902 1 2024-03-26T18:54:44.087 INFO:tasks.workunit.client.0.smithi102.stdout:4/593: mkdir d1/d17/dbb 0 2024-03-26T18:54:44.088 INFO:tasks.workunit.client.0.smithi102.stdout:4/594: mkdir d1/d17/dbc 0 2024-03-26T18:54:44.088 INFO:tasks.workunit.client.0.smithi102.stdout:2/728: rmdir d3/d82/d93/d9e/da0 39 2024-03-26T18:54:44.091 INFO:tasks.workunit.client.1.smithi190.stdout:6/935: sync 2024-03-26T18:54:44.092 INFO:tasks.workunit.client.1.smithi190.stdout:6/936: fsync d17/d28/d2c/db8/df3/d109/d79/f82 0 2024-03-26T18:54:44.093 INFO:tasks.workunit.client.0.smithi102.stdout:4/595: dread d1/d17/d1a/d21/d77/f71 [0,4194304] 0 2024-03-26T18:54:44.093 INFO:tasks.workunit.client.0.smithi102.stdout:4/596: dread - d1/d27/d98/fb9 zero size 2024-03-26T18:54:44.101 INFO:tasks.workunit.client.0.smithi102.stdout:3/984: dwrite df/d29/d31/fb5 [0,4194304] 0 2024-03-26T18:54:44.109 INFO:tasks.workunit.client.0.smithi102.stdout:4/597: dread d1/d17/d1f/d5d/d8b/f51 [0,4194304] 0 2024-03-26T18:54:44.111 INFO:tasks.workunit.client.0.smithi102.stdout:4/598: link d1/d27/c46 d1/d6c/cbd 0 2024-03-26T18:54:44.111 INFO:tasks.workunit.client.0.smithi102.stdout:3/985: write df/d29/f2c [502497,127323] 0 2024-03-26T18:54:44.112 INFO:tasks.workunit.client.0.smithi102.stdout:3/986: dread df/d1c/d34/d53/fa4 [0,4194304] 0 2024-03-26T18:54:44.114 INFO:tasks.workunit.client.0.smithi102.stdout:3/987: rename df/d1c/c35 to df/d29/d31/d36/d3c/c163 0 2024-03-26T18:54:44.114 INFO:tasks.workunit.client.0.smithi102.stdout:3/988: chown df/d61/dd6/fdb 15 1 2024-03-26T18:54:44.115 INFO:tasks.workunit.client.0.smithi102.stdout:3/989: mkdir df/d29/d31/d36/d164 0 2024-03-26T18:54:44.116 INFO:tasks.workunit.client.0.smithi102.stdout:3/990: mkdir df/d29/d31/d36/d3c/d6f/d75/d165 0 2024-03-26T18:54:44.116 INFO:tasks.workunit.client.0.smithi102.stdout:3/991: chown df/d1c/d70/dcd/fad 126315 1 2024-03-26T18:54:44.117 INFO:tasks.workunit.client.0.smithi102.stdout:3/992: mkdir df/d1c/d34/d9b/d166 0 2024-03-26T18:54:44.118 INFO:tasks.workunit.client.0.smithi102.stdout:3/993: creat df/d29/d31/f167 x:0 0 0 2024-03-26T18:54:44.121 INFO:tasks.workunit.client.1.smithi190.stdout:3/929: dwrite d1/de/d1c/d36/f58 [0,4194304] 0 2024-03-26T18:54:44.121 INFO:tasks.workunit.client.1.smithi190.stdout:3/930: readlink d1/d3/l17 0 2024-03-26T18:54:44.121 INFO:tasks.workunit.client.1.smithi190.stdout:3/931: write d1/d3/df/d31/d5b/fe6 [561119,4589] 0 2024-03-26T18:54:44.122 INFO:tasks.workunit.client.1.smithi190.stdout:3/932: symlink d1/d3/d5a/l11e 0 2024-03-26T18:54:44.122 INFO:tasks.workunit.client.1.smithi190.stdout:3/933: dread - d1/de/fcc zero size 2024-03-26T18:54:44.124 INFO:tasks.workunit.client.0.smithi102.stdout:7/630: dwrite d7/ff [0,4194304] 0 2024-03-26T18:54:44.124 INFO:tasks.workunit.client.0.smithi102.stdout:1/723: dwrite de/d16/d1d/d2a/f58 [0,4194304] 0 2024-03-26T18:54:44.124 INFO:tasks.workunit.client.0.smithi102.stdout:2/729: dwrite d3/f1c [0,4194304] 0 2024-03-26T18:54:44.125 INFO:tasks.workunit.client.0.smithi102.stdout:7/631: dread d7/d10/d27/f55 [0,4194304] 0 2024-03-26T18:54:44.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:43 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:44.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:43 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:44.129 INFO:tasks.workunit.client.0.smithi102.stdout:3/994: read df/d1c/d34/d53/fdd [2881332,9141] 0 2024-03-26T18:54:44.129 INFO:tasks.workunit.client.0.smithi102.stdout:3/995: write df/d29/d31/d36/d45/daf/d105/f12c [1467332,66326] 0 2024-03-26T18:54:44.129 INFO:tasks.workunit.client.0.smithi102.stdout:3/996: chown df/d1c/d70/dcd/df7/d10e 4163147 1 2024-03-26T18:54:44.130 INFO:tasks.workunit.client.1.smithi190.stdout:6/937: dwrite fc [0,4194304] 0 2024-03-26T18:54:44.130 INFO:tasks.workunit.client.1.smithi190.stdout:6/938: creat d17/d28/d2c/d63/d10e/f13f x:0 0 0 2024-03-26T18:54:44.131 INFO:tasks.workunit.client.0.smithi102.stdout:4/599: dwrite d1/d17/d2a/f61 [0,4194304] 0 2024-03-26T18:54:44.134 INFO:tasks.workunit.client.1.smithi190.stdout:6/939: unlink d17/d28/d8b/d101/f103 0 2024-03-26T18:54:44.134 INFO:tasks.workunit.client.1.smithi190.stdout:6/940: truncate d17/dd3/fef 5152616 0 2024-03-26T18:54:44.134 INFO:tasks.workunit.client.1.smithi190.stdout:6/941: chown d17/d28/d2c/le5 5 1 2024-03-26T18:54:44.135 INFO:tasks.workunit.client.0.smithi102.stdout:7/632: creat d7/d10/d17/db9/fbe x:0 0 0 2024-03-26T18:54:44.137 INFO:tasks.workunit.client.0.smithi102.stdout:3/997: mknod df/d29/d31/d36/d164/c168 0 2024-03-26T18:54:44.137 INFO:tasks.workunit.client.0.smithi102.stdout:3/998: readlink df/d1c/d34/l7e 0 2024-03-26T18:54:44.139 INFO:tasks.workunit.client.1.smithi190.stdout:6/942: creat d17/d28/df6/da8/d12b/f140 x:0 0 0 2024-03-26T18:54:44.139 INFO:tasks.workunit.client.1.smithi190.stdout:6/943: dread - d17/d28/de8/f12e zero size 2024-03-26T18:54:44.141 INFO:tasks.workunit.client.1.smithi190.stdout:6/944: mkdir d17/d28/d8b/d101/d128/d141 0 2024-03-26T18:54:44.142 INFO:tasks.workunit.client.0.smithi102.stdout:4/600: truncate d1/f26 6478521 0 2024-03-26T18:54:44.144 INFO:tasks.workunit.client.0.smithi102.stdout:7/633: mknod d7/d10/d17/cbf 0 2024-03-26T18:54:44.145 INFO:tasks.workunit.client.1.smithi190.stdout:6/945: fsync d17/d28/d2c/db8/df3/d109/d79/fcf 0 2024-03-26T18:54:44.145 INFO:tasks.workunit.client.0.smithi102.stdout:3/999: chown df/d29/d3a/l7a 3528 1 2024-03-26T18:54:44.146 INFO:tasks.workunit.client.1.smithi190.stdout:6/946: creat d17/d28/d2c/db8/df3/d109/d79/f142 x:0 0 0 2024-03-26T18:54:44.147 INFO:tasks.workunit.client.1.smithi190.stdout:6/947: creat d17/d28/df6/da8/df4/dfb/d121/f143 x:0 0 0 2024-03-26T18:54:44.147 INFO:tasks.workunit.client.1.smithi190.stdout:6/948: write d17/d28/d2c/f61 [3497360,113326] 0 2024-03-26T18:54:44.147 INFO:tasks.workunit.client.1.smithi190.stdout:6/949: write d17/d28/d2c/db8/df3/d109/d79/f133 [440869,60319] 0 2024-03-26T18:54:44.148 INFO:tasks.workunit.client.0.smithi102.stdout:2/730: dwrite d3/d82/d93/fcb [0,4194304] 0 2024-03-26T18:54:44.148 INFO:tasks.workunit.client.0.smithi102.stdout:2/731: truncate d3/d11/d27/f6b 3835433 0 2024-03-26T18:54:44.148 INFO:tasks.workunit.client.0.smithi102.stdout:2/732: fdatasync d3/d20/d78/d84/d6e/dc6/fd1 0 2024-03-26T18:54:44.149 INFO:tasks.workunit.client.1.smithi190.stdout:6/950: rmdir d17/d28/d8b/d95/dda/d10a 39 2024-03-26T18:54:44.151 INFO:tasks.workunit.client.1.smithi190.stdout:6/951: creat d17/d28/f144 x:0 0 0 2024-03-26T18:54:44.152 INFO:tasks.workunit.client.0.smithi102.stdout:2/733: rename d3/d82/d93/d9e/d1a/d25/f100 to d3/d82/f106 0 2024-03-26T18:54:44.152 INFO:tasks.workunit.client.0.smithi102.stdout:9/659: sync 2024-03-26T18:54:44.152 INFO:tasks.workunit.client.0.smithi102.stdout:6/697: sync 2024-03-26T18:54:44.153 INFO:tasks.workunit.client.0.smithi102.stdout:6/698: chown d2/d3d/d53/db1/de0 0 1 2024-03-26T18:54:44.153 INFO:tasks.workunit.client.0.smithi102.stdout:0/754: sync 2024-03-26T18:54:44.153 INFO:tasks.workunit.client.0.smithi102.stdout:8/742: sync 2024-03-26T18:54:44.153 INFO:tasks.workunit.client.0.smithi102.stdout:8/743: fdatasync d0/d15/d5d/dd3/d2c/d2f/fa5 0 2024-03-26T18:54:44.154 INFO:tasks.workunit.client.1.smithi190.stdout:6/952: mkdir d17/d28/d2c/db8/df3/d109/d79/ddf/d139/d145 0 2024-03-26T18:54:44.155 INFO:tasks.workunit.client.0.smithi102.stdout:2/734: mknod d3/c107 0 2024-03-26T18:54:44.156 INFO:tasks.workunit.client.0.smithi102.stdout:6/699: chown d2/d3/f97 487001084 1 2024-03-26T18:54:44.156 INFO:tasks.workunit.client.0.smithi102.stdout:6/700: chown d2/df/f27 167281 1 2024-03-26T18:54:44.156 INFO:tasks.workunit.client.0.smithi102.stdout:7/634: dread d7/d12/f8d [0,4194304] 0 2024-03-26T18:54:44.156 INFO:tasks.workunit.client.0.smithi102.stdout:7/635: readlink d7/d12/d4d/d62/l7e 0 2024-03-26T18:54:44.156 INFO:tasks.workunit.client.0.smithi102.stdout:7/636: chown d7/d10/d67/d69/d78/ca4 1131042 1 2024-03-26T18:54:44.157 INFO:tasks.workunit.client.0.smithi102.stdout:9/660: symlink df/d1a/d23/d4e/d8e/lf4 0 2024-03-26T18:54:44.157 INFO:tasks.workunit.client.0.smithi102.stdout:9/661: dread - df/d1a/d57/f6d zero size 2024-03-26T18:54:44.159 INFO:tasks.workunit.client.0.smithi102.stdout:0/755: mknod d6/de/d13/d25/d31/d45/c103 0 2024-03-26T18:54:44.159 INFO:tasks.workunit.client.0.smithi102.stdout:0/756: chown d6/d3e/d87/l6c 54 1 2024-03-26T18:54:44.160 INFO:tasks.workunit.client.0.smithi102.stdout:8/744: symlink d0/d3c/d96/d98/dda/lf4 0 2024-03-26T18:54:44.161 INFO:tasks.workunit.client.0.smithi102.stdout:2/735: truncate d3/d82/d93/d9e/ff 33197 0 2024-03-26T18:54:44.161 INFO:tasks.workunit.client.0.smithi102.stdout:2/736: write d3/d20/ff0 [722348,18078] 0 2024-03-26T18:54:44.162 INFO:tasks.workunit.client.0.smithi102.stdout:6/701: rmdir d2 39 2024-03-26T18:54:44.165 INFO:tasks.workunit.client.0.smithi102.stdout:4/601: dwrite d1/d25/d90/fab [0,4194304] 0 2024-03-26T18:54:44.165 INFO:tasks.workunit.client.0.smithi102.stdout:7/637: truncate d7/d2a/f60 571429 0 2024-03-26T18:54:44.165 INFO:tasks.workunit.client.0.smithi102.stdout:9/662: mknod df/d1a/d65/cf5 0 2024-03-26T18:54:44.167 INFO:tasks.workunit.client.0.smithi102.stdout:8/745: creat d0/d15/d5d/dd3/d2c/d33/ff5 x:0 0 0 2024-03-26T18:54:44.171 INFO:tasks.workunit.client.0.smithi102.stdout:2/737: rmdir d3/d82/d93/d9e/d99/de0 0 2024-03-26T18:54:44.173 INFO:tasks.workunit.client.1.smithi190.stdout:6/953: dread f1 [0,4194304] 0 2024-03-26T18:54:44.176 INFO:tasks.workunit.client.0.smithi102.stdout:4/602: creat d1/d75/d85/fbe x:0 0 0 2024-03-26T18:54:44.179 INFO:tasks.workunit.client.1.smithi190.stdout:6/954: unlink d17/fea 0 2024-03-26T18:54:44.179 INFO:tasks.workunit.client.1.smithi190.stdout:6/955: dread - d17/d28/df6/da8/df4/dfb/d121/f143 zero size 2024-03-26T18:54:44.181 INFO:tasks.workunit.client.0.smithi102.stdout:8/746: creat d0/d15/d5d/dd3/d2c/ff6 x:0 0 0 2024-03-26T18:54:44.182 INFO:tasks.workunit.client.0.smithi102.stdout:6/702: dread d2/df/d35/f6e [0,4194304] 0 2024-03-26T18:54:44.183 INFO:tasks.workunit.client.0.smithi102.stdout:2/738: mkdir d3/d11/d15/d31/d108 0 2024-03-26T18:54:44.183 INFO:tasks.workunit.client.0.smithi102.stdout:2/739: dread - d3/d11/d15/d46/d7b/d97/fbb zero size 2024-03-26T18:54:44.185 INFO:tasks.workunit.client.1.smithi190.stdout:6/956: link d17/d28/d2c/db8/f54 d17/d28/d2c/db8/f146 0 2024-03-26T18:54:44.188 INFO:tasks.workunit.client.0.smithi102.stdout:4/603: rename d1/d25/c91 to d1/d25/cbf 0 2024-03-26T18:54:44.188 INFO:tasks.workunit.client.0.smithi102.stdout:4/604: write d1/d17/d1a/d21/f56 [5050178,83522] 0 2024-03-26T18:54:44.189 INFO:tasks.workunit.client.0.smithi102.stdout:8/747: link d0/d15/d5d/dd3/d26/d80/lc9 d0/d15/d5d/dd3/d2c/d33/d87/lf7 0 2024-03-26T18:54:44.190 INFO:tasks.workunit.client.0.smithi102.stdout:6/703: mknod d2/d3d/d53/db1/de0/ce2 0 2024-03-26T18:54:44.191 INFO:tasks.workunit.client.0.smithi102.stdout:9/663: read df/d1a/f5e [3788911,89783] 0 2024-03-26T18:54:44.191 INFO:tasks.workunit.client.0.smithi102.stdout:4/605: mkdir d1/d17/d1f/d5d/d6f/dc0 0 2024-03-26T18:54:44.192 INFO:tasks.workunit.client.0.smithi102.stdout:8/748: read d0/d15/d5d/dd3/d2c/d2f/f60 [250125,117302] 0 2024-03-26T18:54:44.192 INFO:tasks.workunit.client.0.smithi102.stdout:8/749: truncate d0/f4d 1446447 0 2024-03-26T18:54:44.192 INFO:tasks.workunit.client.0.smithi102.stdout:6/704: creat d2/d3d/d53/fe3 x:0 0 0 2024-03-26T18:54:44.195 INFO:tasks.workunit.client.0.smithi102.stdout:4/606: unlink d1/d17/d1f/d5d/d8b/d3e/d74/c93 0 2024-03-26T18:54:44.196 INFO:tasks.workunit.client.0.smithi102.stdout:4/607: fdatasync d1/d75/d85/fbe 0 2024-03-26T18:54:44.197 INFO:tasks.workunit.client.0.smithi102.stdout:8/750: symlink d0/d3c/d96/d98/dcf/lf8 0 2024-03-26T18:54:44.197 INFO:tasks.workunit.client.0.smithi102.stdout:8/751: truncate d0/d15/f67 406593 0 2024-03-26T18:54:44.198 INFO:tasks.workunit.client.0.smithi102.stdout:6/705: mkdir d2/df/d71/d80/d91/dc1/de4 0 2024-03-26T18:54:44.198 INFO:tasks.workunit.client.0.smithi102.stdout:6/706: chown d2/d7/f46 3460685 1 2024-03-26T18:54:44.198 INFO:tasks.workunit.client.0.smithi102.stdout:6/707: stat d2/df/d71 0 2024-03-26T18:54:44.198 INFO:tasks.workunit.client.0.smithi102.stdout:6/708: chown d2/d12/l62 1586787 1 2024-03-26T18:54:44.199 INFO:tasks.workunit.client.0.smithi102.stdout:9/664: rmdir df/d1a/d1d/d72/d7d/db7/dd5 39 2024-03-26T18:54:44.199 INFO:tasks.workunit.client.0.smithi102.stdout:9/665: write df/f3b [2534625,59397] 0 2024-03-26T18:54:44.199 INFO:tasks.workunit.client.1.smithi190.stdout:6/957: write d17/d28/df6/da8/df4/dfb/f56 [3526147,39470] 0 2024-03-26T18:54:44.200 INFO:tasks.workunit.client.1.smithi190.stdout:6/958: fsync d17/d28/d2c/db8/df3/d109/d79/f8e 0 2024-03-26T18:54:44.200 INFO:tasks.workunit.client.1.smithi190.stdout:6/959: creat d17/d28/d2c/d77/f147 x:0 0 0 2024-03-26T18:54:44.200 INFO:tasks.workunit.client.1.smithi190.stdout:6/960: fdatasync fc 0 2024-03-26T18:54:44.200 INFO:tasks.workunit.client.1.smithi190.stdout:6/961: read - d17/d28/d2c/d63/d10e/f13c zero size 2024-03-26T18:54:44.200 INFO:tasks.workunit.client.1.smithi190.stdout:6/962: fdatasync d17/f23 0 2024-03-26T18:54:44.200 INFO:tasks.workunit.client.0.smithi102.stdout:4/608: creat d1/d17/d1f/d5d/d6f/dc0/fc1 x:0 0 0 2024-03-26T18:54:44.200 INFO:tasks.workunit.client.0.smithi102.stdout:4/609: read - d1/d17/d1f/d5d/d6f/fac zero size 2024-03-26T18:54:44.202 INFO:tasks.workunit.client.0.smithi102.stdout:8/752: mkdir d0/d3c/df9 0 2024-03-26T18:54:44.203 INFO:tasks.workunit.client.1.smithi190.stdout:6/963: rename d17/d28/d8b/d131/f136 to d17/d28/d8b/d95/f148 0 2024-03-26T18:54:44.203 INFO:tasks.workunit.client.0.smithi102.stdout:6/709: symlink d2/d3/dd0/le5 0 2024-03-26T18:54:44.205 INFO:tasks.workunit.client.0.smithi102.stdout:9/666: unlink df/d1a/d23/d3e/d75/f8d 0 2024-03-26T18:54:44.206 INFO:tasks.workunit.client.0.smithi102.stdout:0/757: dwrite d6/de/d13/d9a/d85/f61 [0,4194304] 0 2024-03-26T18:54:44.206 INFO:tasks.workunit.client.0.smithi102.stdout:7/638: dwrite d7/d2a/f30 [0,4194304] 0 2024-03-26T18:54:44.206 INFO:tasks.workunit.client.1.smithi190.stdout:6/964: truncate d17/d28/d74/d97/f102 852342 0 2024-03-26T18:54:44.206 INFO:tasks.workunit.client.0.smithi102.stdout:2/740: dwrite d3/d11/d27/f57 [0,4194304] 0 2024-03-26T18:54:44.207 INFO:tasks.workunit.client.0.smithi102.stdout:2/741: dread - d3/f102 zero size 2024-03-26T18:54:44.207 INFO:tasks.workunit.client.0.smithi102.stdout:4/610: mkdir d1/d17/d2a/d9a/dc2 0 2024-03-26T18:54:44.209 INFO:tasks.workunit.client.0.smithi102.stdout:8/753: creat d0/d3c/d6f/ffa x:0 0 0 2024-03-26T18:54:44.210 INFO:tasks.workunit.client.0.smithi102.stdout:9/667: symlink df/d1a/d23/d60/lf6 0 2024-03-26T18:54:44.213 INFO:tasks.workunit.client.0.smithi102.stdout:6/710: dread d2/d7/d3c/fbf [0,4194304] 0 2024-03-26T18:54:44.213 INFO:tasks.workunit.client.0.smithi102.stdout:0/758: mkdir d6/d3e/d104 0 2024-03-26T18:54:44.216 INFO:tasks.workunit.client.0.smithi102.stdout:7/639: rename d7/d12/f22 to d7/d10/d67/d69/d98/fc0 0 2024-03-26T18:54:44.219 INFO:tasks.workunit.client.0.smithi102.stdout:2/742: link d3/f102 d3/d11/d15/d31/d39/f109 0 2024-03-26T18:54:44.221 INFO:tasks.workunit.client.0.smithi102.stdout:4/611: mknod d1/d75/d85/cc3 0 2024-03-26T18:54:44.221 INFO:tasks.workunit.client.1.smithi190.stdout:5/975: sync 2024-03-26T18:54:44.221 INFO:tasks.workunit.client.1.smithi190.stdout:1/985: sync 2024-03-26T18:54:44.221 INFO:tasks.workunit.client.1.smithi190.stdout:1/986: write d3/d9/da3/f136 [796412,64985] 0 2024-03-26T18:54:44.222 INFO:tasks.workunit.client.0.smithi102.stdout:8/754: link d0/d15/d5d/dd3/d26/d5c/la3 d0/d15/d1b/lfb 0 2024-03-26T18:54:44.223 INFO:tasks.workunit.client.0.smithi102.stdout:9/668: mkdir df/d46/df7 0 2024-03-26T18:54:44.223 INFO:tasks.workunit.client.0.smithi102.stdout:9/669: chown df/d46/da8/fbb 38094 1 2024-03-26T18:54:44.223 INFO:tasks.workunit.client.0.smithi102.stdout:9/670: truncate df/fca 489753 0 2024-03-26T18:54:44.224 INFO:tasks.workunit.client.0.smithi102.stdout:9/671: chown df/d1a/f1c 836224 1 2024-03-26T18:54:44.224 INFO:tasks.workunit.client.0.smithi102.stdout:9/672: truncate df/d1a/d23/d3e/f9f 359668 0 2024-03-26T18:54:44.225 INFO:tasks.workunit.client.1.smithi190.stdout:1/987: rename d3/da/dd/d76/f139 to d3/da/dd/d2f/d34/d10c/f13c 0 2024-03-26T18:54:44.226 INFO:tasks.workunit.client.1.smithi190.stdout:3/934: sync 2024-03-26T18:54:44.226 INFO:tasks.workunit.client.1.smithi190.stdout:3/935: write d1/fb [5934691,43292] 0 2024-03-26T18:54:44.227 INFO:tasks.workunit.client.0.smithi102.stdout:0/759: mknod d6/de/d13/d25/d31/d45/d82/d84/d88/db2/c105 0 2024-03-26T18:54:44.227 INFO:tasks.workunit.client.0.smithi102.stdout:0/760: creat d6/de/d13/d25/d31/d45/d82/d84/d88/f106 x:0 0 0 2024-03-26T18:54:44.231 INFO:tasks.workunit.client.1.smithi190.stdout:3/936: rename d1/d3/d5a/d6e to d1/d3/df/d18/d35/d89/d11f 0 2024-03-26T18:54:44.233 INFO:tasks.workunit.client.0.smithi102.stdout:4/612: creat d1/d17/d2a/fc4 x:0 0 0 2024-03-26T18:54:44.234 INFO:tasks.workunit.client.1.smithi190.stdout:3/937: symlink d1/d3/df/d18/d35/d41/dae/d9c/l120 0 2024-03-26T18:54:44.235 INFO:tasks.workunit.client.0.smithi102.stdout:8/755: getdents d0/d15/d5d/dac 0 2024-03-26T18:54:44.235 INFO:tasks.workunit.client.0.smithi102.stdout:8/756: truncate d0/d3c/d96/d98/dcf/fde 507922 0 2024-03-26T18:54:44.236 INFO:tasks.workunit.client.1.smithi190.stdout:1/988: dread d3/da/dd/d2f/d5b/f6b [0,4194304] 0 2024-03-26T18:54:44.237 INFO:tasks.workunit.client.0.smithi102.stdout:9/673: mkdir df/d1a/d1d/d72/d7d/db7/df8 0 2024-03-26T18:54:44.237 INFO:tasks.workunit.client.0.smithi102.stdout:9/674: readlink df/d1a/l3c 0 2024-03-26T18:54:44.238 INFO:tasks.workunit.client.1.smithi190.stdout:3/938: link d1/d3/df/d31/d5b/fe6 d1/d3/f121 0 2024-03-26T18:54:44.240 INFO:tasks.workunit.client.0.smithi102.stdout:6/711: getdents d2/d3d/d53 0 2024-03-26T18:54:44.241 INFO:tasks.workunit.client.0.smithi102.stdout:0/761: dread d6/fd [4194304,4194304] 0 2024-03-26T18:54:44.242 INFO:tasks.workunit.client.0.smithi102.stdout:0/762: readlink d6/l10 0 2024-03-26T18:54:44.242 INFO:tasks.workunit.client.0.smithi102.stdout:0/763: creat d6/de/d13/d9a/d85/f107 x:0 0 0 2024-03-26T18:54:44.242 INFO:tasks.workunit.client.0.smithi102.stdout:0/764: chown d6/de/d13/d9a/d85 2377598 1 2024-03-26T18:54:44.242 INFO:tasks.workunit.client.1.smithi190.stdout:1/989: rename d3/da/dd/c12e to d3/d9/db/c13d 0 2024-03-26T18:54:44.242 INFO:tasks.workunit.client.1.smithi190.stdout:1/990: readlink d3/da/dd/d95/d52/l81 0 2024-03-26T18:54:44.244 INFO:tasks.workunit.client.1.smithi190.stdout:5/976: dwrite d2/d17/d98/d3a/f52 [0,4194304] 0 2024-03-26T18:54:44.244 INFO:tasks.workunit.client.1.smithi190.stdout:3/939: symlink d1/d3/df/d31/dc4/dd9/l122 0 2024-03-26T18:54:44.246 INFO:tasks.workunit.client.0.smithi102.stdout:7/640: rmdir d7/d10/d17/db9 39 2024-03-26T18:54:44.246 INFO:tasks.workunit.client.0.smithi102.stdout:7/641: readlink d7/d10/d17/l1e 0 2024-03-26T18:54:44.252 INFO:tasks.workunit.client.1.smithi190.stdout:5/977: mkdir d2/d17/d62/d64/d149 0 2024-03-26T18:54:44.253 INFO:tasks.workunit.client.0.smithi102.stdout:9/675: symlink df/d1a/d23/d60/dd4/lf9 0 2024-03-26T18:54:44.253 INFO:tasks.workunit.client.1.smithi190.stdout:1/991: write d3/da/d2e/d59/f65 [438437,70120] 0 2024-03-26T18:54:44.253 INFO:tasks.workunit.client.1.smithi190.stdout:1/992: truncate d3/da/d2e/d59/dae/f129 131289 0 2024-03-26T18:54:44.255 INFO:tasks.workunit.client.1.smithi190.stdout:3/940: creat d1/d3/df/d18/f123 x:0 0 0 2024-03-26T18:54:44.257 INFO:tasks.workunit.client.0.smithi102.stdout:6/712: mkdir d2/df/d71/de6 0 2024-03-26T18:54:44.257 INFO:tasks.workunit.client.1.smithi190.stdout:5/978: getdents d2/d17/d1b/db6/d24/d3b/da7 0 2024-03-26T18:54:44.262 INFO:tasks.workunit.client.1.smithi190.stdout:1/993: dread d3/d9/da3/f136 [0,4194304] 0 2024-03-26T18:54:44.262 INFO:tasks.workunit.client.1.smithi190.stdout:1/994: stat d3/d73/l84 0 2024-03-26T18:54:44.262 INFO:tasks.workunit.client.1.smithi190.stdout:3/941: symlink d1/de/d8a/de1/de7/l124 0 2024-03-26T18:54:44.266 INFO:tasks.workunit.client.1.smithi190.stdout:5/979: creat d2/d17/d1b/d60/d63/f14a x:0 0 0 2024-03-26T18:54:44.268 INFO:tasks.workunit.client.1.smithi190.stdout:1/995: rename f2 to d3/da/dd/d95/f13e 0 2024-03-26T18:54:44.269 INFO:tasks.workunit.client.0.smithi102.stdout:2/743: getdents d3/d11/d15/d31/d39 0 2024-03-26T18:54:44.269 INFO:tasks.workunit.client.0.smithi102.stdout:7/642: getdents d7/d10/d17/db9 0 2024-03-26T18:54:44.272 INFO:tasks.workunit.client.1.smithi190.stdout:3/942: link d1/de/d8a/f95 d1/d3/d52/d70/f125 0 2024-03-26T18:54:44.272 INFO:tasks.workunit.client.1.smithi190.stdout:3/943: fdatasync d1/de/d8a/de1/ff8 0 2024-03-26T18:54:44.273 INFO:tasks.workunit.client.1.smithi190.stdout:5/980: mkdir d2/d17/d62/d109/d14b 0 2024-03-26T18:54:44.274 INFO:tasks.workunit.client.0.smithi102.stdout:6/713: symlink d2/d7/d3c/d78/dc4/le7 0 2024-03-26T18:54:44.274 INFO:tasks.workunit.client.0.smithi102.stdout:6/714: chown d2/d3d/d53/db1 1721655536 1 2024-03-26T18:54:44.276 INFO:tasks.workunit.client.1.smithi190.stdout:1/996: mkdir d3/d9/d13f 0 2024-03-26T18:54:44.277 INFO:tasks.workunit.client.0.smithi102.stdout:2/744: mkdir d3/d11/d15/d46/d7b/d97/d10a 0 2024-03-26T18:54:44.278 INFO:tasks.workunit.client.1.smithi190.stdout:3/944: mkdir d1/d3/df/d18/d57/d126 0 2024-03-26T18:54:44.278 INFO:tasks.workunit.client.1.smithi190.stdout:3/945: chown d1/d3/df/d18/d35/d89/d11f/df5 3 1 2024-03-26T18:54:44.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:43 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:44.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:43 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:44.279 INFO:tasks.workunit.client.0.smithi102.stdout:7/643: rename d7/d10/d27/d34/ca2 to d7/d10/d27/d34/d72/cc1 0 2024-03-26T18:54:44.280 INFO:tasks.workunit.client.1.smithi190.stdout:5/981: link d2/d105/c13f d2/d17/d1b/db6/c14c 0 2024-03-26T18:54:44.281 INFO:tasks.workunit.client.0.smithi102.stdout:0/765: dwrite d6/d2d/d3d/d80/fdf [0,4194304] 0 2024-03-26T18:54:44.281 INFO:tasks.workunit.client.0.smithi102.stdout:0/766: dread - d6/de/d13/d1f/fa8 zero size 2024-03-26T18:54:44.281 INFO:tasks.workunit.client.0.smithi102.stdout:4/613: getdents d1/d17/d2a 0 2024-03-26T18:54:44.281 INFO:tasks.workunit.client.0.smithi102.stdout:9/676: dwrite df/d1a/d57/f6d [0,4194304] 0 2024-03-26T18:54:44.285 INFO:tasks.workunit.client.1.smithi190.stdout:5/982: dread d2/d17/d1b/db6/fc6 [0,4194304] 0 2024-03-26T18:54:44.287 INFO:tasks.workunit.client.1.smithi190.stdout:3/946: getdents d1/d3/df/d18/d57/d91/d8c 0 2024-03-26T18:54:44.287 INFO:tasks.workunit.client.1.smithi190.stdout:3/947: read - d1/d3/d52/d70/db3/ff2 zero size 2024-03-26T18:54:44.288 INFO:tasks.workunit.client.1.smithi190.stdout:3/948: write d1/de/d9f/fd4 [352294,29923] 0 2024-03-26T18:54:44.288 INFO:tasks.workunit.client.1.smithi190.stdout:3/949: chown d1/d3/df/d18/d35/d41/dae 77792410 1 2024-03-26T18:54:44.292 INFO:tasks.workunit.client.0.smithi102.stdout:6/715: creat d2/d3/fe8 x:0 0 0 2024-03-26T18:54:44.293 INFO:tasks.workunit.client.0.smithi102.stdout:2/745: creat d3/d11/d15/d46/d7b/d97/d10a/f10b x:0 0 0 2024-03-26T18:54:44.294 INFO:tasks.workunit.client.1.smithi190.stdout:5/983: creat d2/d17/d1b/d60/d63/dcd/d118/d120/d144/f14d x:0 0 0 2024-03-26T18:54:44.295 INFO:tasks.workunit.client.0.smithi102.stdout:7/644: link d7/d10/d67/d69/d78/fba d7/d2a/fc2 0 2024-03-26T18:54:44.295 INFO:tasks.workunit.client.0.smithi102.stdout:7/645: stat d7/d12/f8d 0 2024-03-26T18:54:44.296 INFO:tasks.workunit.client.1.smithi190.stdout:3/950: creat d1/d3/df/d10a/f127 x:0 0 0 2024-03-26T18:54:44.297 INFO:tasks.workunit.client.1.smithi190.stdout:3/951: read d1/d3/df/d31/d5b/fe6 [38380,99240] 0 2024-03-26T18:54:44.298 INFO:tasks.workunit.client.1.smithi190.stdout:5/984: creat d2/d17/d1b/db6/f14e x:0 0 0 2024-03-26T18:54:44.298 INFO:tasks.workunit.client.1.smithi190.stdout:5/985: chown d2/d17/d1b/d60/d63/cb2 151 1 2024-03-26T18:54:44.299 INFO:tasks.workunit.client.1.smithi190.stdout:3/952: symlink d1/l128 0 2024-03-26T18:54:44.299 INFO:tasks.workunit.client.1.smithi190.stdout:3/953: creat d1/de/d8a/de1/f129 x:0 0 0 2024-03-26T18:54:44.301 INFO:tasks.workunit.client.0.smithi102.stdout:0/767: dwrite d6/de/d13/d9a/d85/fe5 [0,4194304] 0 2024-03-26T18:54:44.302 INFO:tasks.workunit.client.1.smithi190.stdout:5/986: rename d2/d17/d62/fca to d2/d17/d1b/d60/d63/d85/d99/d114/f14f 0 2024-03-26T18:54:44.303 INFO:tasks.workunit.client.0.smithi102.stdout:9/677: mknod df/d1a/d23/d3e/cfa 0 2024-03-26T18:54:44.305 INFO:tasks.workunit.client.1.smithi190.stdout:1/997: rmdir d3/da/dd/d95 39 2024-03-26T18:54:44.305 INFO:tasks.workunit.client.1.smithi190.stdout:1/998: chown d3/da/d3a/d99 0 1 2024-03-26T18:54:44.307 INFO:tasks.workunit.client.1.smithi190.stdout:3/954: rename d1/d3/df/d18/d35/d41/f72 to d1/d3/df/d31/dd1/f12a 0 2024-03-26T18:54:44.308 INFO:tasks.workunit.client.1.smithi190.stdout:3/955: fsync d1/de/d8a/f95 0 2024-03-26T18:54:44.308 INFO:tasks.workunit.client.1.smithi190.stdout:3/956: chown d1/d3/df/d18/d35/d41/d61 3712357 1 2024-03-26T18:54:44.308 INFO:tasks.workunit.client.0.smithi102.stdout:6/716: symlink d2/df/d35/d43/d75/le9 0 2024-03-26T18:54:44.310 INFO:tasks.workunit.client.1.smithi190.stdout:5/987: link d2/d17/d62/f111 d2/d17/d62/d109/d14b/f150 0 2024-03-26T18:54:44.310 INFO:tasks.workunit.client.0.smithi102.stdout:2/746: mkdir d3/d11/d10c 0 2024-03-26T18:54:44.312 INFO:tasks.workunit.client.0.smithi102.stdout:7/646: mknod d7/d2a/d9d/cc3 0 2024-03-26T18:54:44.314 INFO:tasks.workunit.client.1.smithi190.stdout:3/957: rename d1/d3/df/d18/d1b/de4 to d1/d3/df/d18/d35/d41/d12b 0 2024-03-26T18:54:44.317 INFO:tasks.workunit.client.1.smithi190.stdout:1/999: symlink d3/da/l140 0 2024-03-26T18:54:44.318 INFO:tasks.workunit.client.0.smithi102.stdout:0/768: mkdir d6/de/d13/d108 0 2024-03-26T18:54:44.319 INFO:tasks.workunit.client.0.smithi102.stdout:9/678: symlink df/d1a/d1d/lfb 0 2024-03-26T18:54:44.319 INFO:tasks.workunit.client.0.smithi102.stdout:9/679: creat df/d1a/d65/dbe/ffc x:0 0 0 2024-03-26T18:54:44.320 INFO:tasks.workunit.client.0.smithi102.stdout:6/717: symlink d2/d3d/d53/d63/lea 0 2024-03-26T18:54:44.322 INFO:tasks.workunit.client.0.smithi102.stdout:2/747: unlink d3/d20/f30 0 2024-03-26T18:54:44.322 INFO:tasks.workunit.client.0.smithi102.stdout:2/748: write d3/d82/d93/d9e/ff [96277,11945] 0 2024-03-26T18:54:44.326 INFO:tasks.workunit.client.0.smithi102.stdout:0/769: mkdir d6/dbe/deb/d109 0 2024-03-26T18:54:44.328 INFO:tasks.workunit.client.1.smithi190.stdout:3/958: dread d1/d3/f13 [0,4194304] 0 2024-03-26T18:54:44.328 INFO:tasks.workunit.client.1.smithi190.stdout:3/959: stat d1/d3/f10 0 2024-03-26T18:54:44.329 INFO:tasks.workunit.client.0.smithi102.stdout:6/718: link d2/d12/la2 d2/d3d/leb 0 2024-03-26T18:54:44.330 INFO:tasks.workunit.client.0.smithi102.stdout:4/614: write d1/d27/f5f [336356,18759] 0 2024-03-26T18:54:44.330 INFO:tasks.workunit.client.0.smithi102.stdout:4/615: getdents d1/d17/d2a 0 2024-03-26T18:54:44.330 INFO:tasks.workunit.client.0.smithi102.stdout:2/749: mknod d3/d82/d93/c10d 0 2024-03-26T18:54:44.330 INFO:tasks.workunit.client.0.smithi102.stdout:2/750: fsync d3/d2d/f68 0 2024-03-26T18:54:44.331 INFO:tasks.workunit.client.0.smithi102.stdout:2/751: fsync d3/d82/de2/def/ff6 0 2024-03-26T18:54:44.332 INFO:tasks.workunit.client.0.smithi102.stdout:7/647: rmdir d7/d10/d67/d69 39 2024-03-26T18:54:44.332 INFO:tasks.workunit.client.0.smithi102.stdout:7/648: chown d7/d2a 360752 1 2024-03-26T18:54:44.332 INFO:tasks.workunit.client.0.smithi102.stdout:7/649: fdatasync d7/d2a/fc2 0 2024-03-26T18:54:44.334 INFO:tasks.workunit.client.0.smithi102.stdout:4/616: symlink d1/d25/d90/lc5 0 2024-03-26T18:54:44.335 INFO:tasks.workunit.client.0.smithi102.stdout:4/617: mknod d1/d17/dbc/cc6 0 2024-03-26T18:54:44.335 INFO:tasks.workunit.client.0.smithi102.stdout:4/618: creat d1/d17/d1f/d5d/d6f/dc0/fc7 x:0 0 0 2024-03-26T18:54:44.344 INFO:tasks.workunit.client.0.smithi102.stdout:9/680: dwrite df/d1a/f1c [4194304,4194304] 0 2024-03-26T18:54:44.349 INFO:tasks.workunit.client.0.smithi102.stdout:6/719: dread d2/f21 [0,4194304] 0 2024-03-26T18:54:44.351 INFO:tasks.workunit.client.0.smithi102.stdout:4/619: write d1/d25/d90/fab [3108255,7836] 0 2024-03-26T18:54:44.351 INFO:tasks.workunit.client.0.smithi102.stdout:6/720: creat d2/d12/fec x:0 0 0 2024-03-26T18:54:44.352 INFO:tasks.workunit.client.0.smithi102.stdout:4/620: creat d1/d17/dbc/fc8 x:0 0 0 2024-03-26T18:54:44.352 INFO:tasks.workunit.client.0.smithi102.stdout:4/621: chown d1/d17/d1f/d5d/d6f/dc0/fc7 7428 1 2024-03-26T18:54:44.352 INFO:tasks.workunit.client.0.smithi102.stdout:6/721: getdents d2/d3d/d53/d5c 0 2024-03-26T18:54:44.353 INFO:tasks.workunit.client.0.smithi102.stdout:4/622: creat d1/d17/d1f/d5d/d8b/fc9 x:0 0 0 2024-03-26T18:54:44.353 INFO:tasks.workunit.client.0.smithi102.stdout:4/623: creat d1/d27/fca x:0 0 0 2024-03-26T18:54:44.354 INFO:tasks.workunit.client.0.smithi102.stdout:6/722: symlink d2/df/d71/d80/d91/dc1/led 0 2024-03-26T18:54:44.355 INFO:tasks.workunit.client.0.smithi102.stdout:4/624: creat d1/d25/d76/fcb x:0 0 0 2024-03-26T18:54:44.355 INFO:tasks.workunit.client.0.smithi102.stdout:4/625: fsync d1/d17/d1f/f2d 0 2024-03-26T18:54:44.355 INFO:tasks.workunit.client.0.smithi102.stdout:4/626: write d1/d17/d2a/f61 [5025919,114737] 0 2024-03-26T18:54:44.355 INFO:tasks.workunit.client.0.smithi102.stdout:4/627: creat d1/d17/dbc/fcc x:0 0 0 2024-03-26T18:54:44.356 INFO:tasks.workunit.client.0.smithi102.stdout:6/723: unlink d2/df/d71/d80/d91/la8 0 2024-03-26T18:54:44.357 INFO:tasks.workunit.client.0.smithi102.stdout:6/724: write d2/df/d71/fdb [26607,44998] 0 2024-03-26T18:54:44.359 INFO:tasks.workunit.client.0.smithi102.stdout:6/725: symlink d2/df/d71/d80/d91/lee 0 2024-03-26T18:54:44.360 INFO:tasks.workunit.client.0.smithi102.stdout:6/726: mknod d2/d3/cef 0 2024-03-26T18:54:44.371 INFO:tasks.workunit.client.0.smithi102.stdout:6/727: write d2/f33 [1815215,91143] 0 2024-03-26T18:54:44.372 INFO:tasks.workunit.client.0.smithi102.stdout:6/728: creat d2/df/d71/d80/d91/dc1/ff0 x:0 0 0 2024-03-26T18:54:44.373 INFO:tasks.workunit.client.0.smithi102.stdout:6/729: fsync d2/df/f8e 0 2024-03-26T18:54:44.373 INFO:tasks.workunit.client.0.smithi102.stdout:6/730: truncate d2/df/d72/f89 2413857 0 2024-03-26T18:54:44.398 INFO:tasks.workunit.client.1.smithi190.stdout:5/988: dwrite d2/d17/d1b/db6/d88/f8c [0,4194304] 0 2024-03-26T18:54:44.398 INFO:tasks.workunit.client.1.smithi190.stdout:3/960: dwrite d1/d3/d52/fa2 [0,4194304] 0 2024-03-26T18:54:44.398 INFO:tasks.workunit.client.1.smithi190.stdout:3/961: write d1/de/f1d [1736591,124658] 0 2024-03-26T18:54:44.399 INFO:tasks.workunit.client.0.smithi102.stdout:7/650: dwrite d7/d12/d2b/f47 [0,4194304] 0 2024-03-26T18:54:44.399 INFO:tasks.workunit.client.1.smithi190.stdout:5/989: truncate d2/d17/d98/d3a/faa 4150473 0 2024-03-26T18:54:44.400 INFO:tasks.workunit.client.0.smithi102.stdout:2/752: dwrite d3/d74/f77 [0,4194304] 0 2024-03-26T18:54:44.400 INFO:tasks.workunit.client.0.smithi102.stdout:2/753: fdatasync d3/d20/d78/d84/fda 0 2024-03-26T18:54:44.401 INFO:tasks.workunit.client.0.smithi102.stdout:7/651: link d7/d12/d2b/f2f d7/d12/d4b/d83/fc4 0 2024-03-26T18:54:44.401 INFO:tasks.workunit.client.0.smithi102.stdout:9/681: dwrite df/fa6 [0,4194304] 0 2024-03-26T18:54:44.401 INFO:tasks.workunit.client.0.smithi102.stdout:4/628: dwrite d1/d17/d1a/d21/d77/f81 [0,4194304] 0 2024-03-26T18:54:44.412 INFO:tasks.workunit.client.1.smithi190.stdout:3/962: mknod d1/de/d1c/c12c 0 2024-03-26T18:54:44.412 INFO:tasks.workunit.client.1.smithi190.stdout:3/963: write d1/d3/d52/f6d [205023,23558] 0 2024-03-26T18:54:44.413 INFO:tasks.workunit.client.1.smithi190.stdout:5/990: dread d2/f10 [0,4194304] 0 2024-03-26T18:54:44.418 INFO:tasks.workunit.client.1.smithi190.stdout:5/991: unlink d2/d17/ff7 0 2024-03-26T18:54:44.420 INFO:tasks.workunit.client.0.smithi102.stdout:2/754: rmdir d3/d82/d93/d9e/d1a/d25 39 2024-03-26T18:54:44.421 INFO:tasks.workunit.client.0.smithi102.stdout:0/770: write d6/de/d13/d25/d4f/d66/f6e [1409231,118341] 0 2024-03-26T18:54:44.422 INFO:tasks.workunit.client.1.smithi190.stdout:5/992: dread d2/f45 [0,4194304] 0 2024-03-26T18:54:44.423 INFO:tasks.workunit.client.1.smithi190.stdout:3/964: write d1/de/d4e/f54 [2739598,88415] 0 2024-03-26T18:54:44.424 INFO:tasks.workunit.client.0.smithi102.stdout:4/629: truncate d1/d17/d1a/f57 2752555 0 2024-03-26T18:54:44.424 INFO:tasks.workunit.client.0.smithi102.stdout:4/630: write d1/d17/d1f/d5d/d6f/dc0/fc1 [364792,12485] 0 2024-03-26T18:54:44.424 INFO:tasks.workunit.client.0.smithi102.stdout:4/631: read - d1/d17/fae zero size 2024-03-26T18:54:44.426 INFO:tasks.workunit.client.0.smithi102.stdout:0/771: symlink d6/d3e/d69/l10a 0 2024-03-26T18:54:44.427 INFO:tasks.workunit.client.1.smithi190.stdout:3/965: mknod d1/d3/df/d18/d35/d41/d12b/dfb/c12d 0 2024-03-26T18:54:44.427 INFO:tasks.workunit.client.0.smithi102.stdout:0/772: mkdir d6/de/d13/d25/da4/d10b 0 2024-03-26T18:54:44.429 INFO:tasks.workunit.client.0.smithi102.stdout:0/773: creat d6/dbe/deb/d109/f10c x:0 0 0 2024-03-26T18:54:44.430 INFO:tasks.workunit.client.0.smithi102.stdout:0/774: mknod d6/de/d13/d25/d31/d45/d82/d84/d88/db2/dd2/c10d 0 2024-03-26T18:54:44.430 INFO:tasks.workunit.client.0.smithi102.stdout:0/775: stat d6/de/d13/d9a/fcc 0 2024-03-26T18:54:44.431 INFO:tasks.workunit.client.0.smithi102.stdout:0/776: creat d6/d2d/f10e x:0 0 0 2024-03-26T18:54:44.433 INFO:tasks.workunit.client.0.smithi102.stdout:0/777: unlink d6/de/d13/d1f/d55/la5 0 2024-03-26T18:54:44.433 INFO:tasks.workunit.client.0.smithi102.stdout:0/778: write d6/f1a [7172244,40698] 0 2024-03-26T18:54:44.433 INFO:tasks.workunit.client.0.smithi102.stdout:0/779: creat d6/d3e/d69/d8c/f10f x:0 0 0 2024-03-26T18:54:44.434 INFO:tasks.workunit.client.0.smithi102.stdout:0/780: write d6/de/d13/d1f/d55/d57/d7d/d7f/fe2 [367914,16849] 0 2024-03-26T18:54:44.434 INFO:tasks.workunit.client.0.smithi102.stdout:0/781: write d6/d3e/d87/f52 [1062112,31758] 0 2024-03-26T18:54:44.434 INFO:tasks.workunit.client.0.smithi102.stdout:0/782: chown d6/de/d13/d1f/f27 73370307 1 2024-03-26T18:54:44.434 INFO:tasks.workunit.client.0.smithi102.stdout:9/682: dwrite df/d1a/f5e [0,4194304] 0 2024-03-26T18:54:44.437 INFO:tasks.workunit.client.1.smithi190.stdout:5/993: read d2/d17/d1b/db6/d88/f8c [1810038,27843] 0 2024-03-26T18:54:44.440 INFO:tasks.workunit.client.0.smithi102.stdout:4/632: dread d1/f8 [4194304,4194304] 0 2024-03-26T18:54:44.441 INFO:tasks.workunit.client.1.smithi190.stdout:5/994: rename d2/d17/d98/d3a/f52 to d2/d17/d1b/d60/d63/d102/f151 0 2024-03-26T18:54:44.441 INFO:tasks.workunit.client.1.smithi190.stdout:5/995: stat d2/d17/d62/df2 0 2024-03-26T18:54:44.443 INFO:tasks.workunit.client.0.smithi102.stdout:4/633: rename d1/d17/d1f/d5d/d6f/dc0/fc1 to d1/d17/d1f/fcd 0 2024-03-26T18:54:44.443 INFO:tasks.workunit.client.0.smithi102.stdout:4/634: read d1/d27/f55 [843951,4002] 0 2024-03-26T18:54:44.443 INFO:tasks.workunit.client.0.smithi102.stdout:4/635: creat d1/d17/d1f/d5d/d6f/dc0/fce x:0 0 0 2024-03-26T18:54:44.445 INFO:tasks.workunit.client.1.smithi190.stdout:5/996: creat d2/d17/d1b/d60/d63/dcd/f152 x:0 0 0 2024-03-26T18:54:44.446 INFO:tasks.workunit.client.1.smithi190.stdout:5/997: write d2/d17/d62/f111 [4471916,52617] 0 2024-03-26T18:54:44.450 INFO:tasks.workunit.client.1.smithi190.stdout:5/998: link d2/d17/d1b/d30/f122 d2/d17/d1b/d60/d63/dcd/d118/d120/f153 0 2024-03-26T18:54:44.453 INFO:tasks.workunit.client.1.smithi190.stdout:5/999: creat d2/d17/d1b/d60/d63/dcd/d118/d120/f154 x:0 0 0 2024-03-26T18:54:44.455 INFO:tasks.workunit.client.0.smithi102.stdout:8/757: sync 2024-03-26T18:54:44.455 INFO:tasks.workunit.client.0.smithi102.stdout:1/724: sync 2024-03-26T18:54:44.455 INFO:tasks.workunit.client.0.smithi102.stdout:1/725: fsync de/d16/d1d/d2a/d2c/d56/d6e/f81 0 2024-03-26T18:54:44.455 INFO:tasks.workunit.client.0.smithi102.stdout:8/758: write d0/d3c/d96/d98/dcf/fee [238284,109480] 0 2024-03-26T18:54:44.456 INFO:tasks.workunit.client.0.smithi102.stdout:1/726: dread de/d16/f41 [0,4194304] 0 2024-03-26T18:54:44.456 INFO:tasks.workunit.client.0.smithi102.stdout:1/727: truncate de/d16/d1d/d2a/d2c/d56/d5f/daa/fd5 54992 0 2024-03-26T18:54:44.456 INFO:tasks.workunit.client.0.smithi102.stdout:1/728: readlink de/d16/d1d/d2a/l2b 0 2024-03-26T18:54:44.458 INFO:tasks.workunit.client.1.smithi190.stdout:6/965: sync 2024-03-26T18:54:44.458 INFO:tasks.workunit.client.0.smithi102.stdout:8/759: unlink d0/d15/d5d/dd3/f21 0 2024-03-26T18:54:44.459 INFO:tasks.workunit.client.0.smithi102.stdout:1/729: link de/d2e/fa8 de/d16/ff8 0 2024-03-26T18:54:44.459 INFO:tasks.workunit.client.0.smithi102.stdout:1/730: stat de/d16/d1d/d61/d6a 0 2024-03-26T18:54:44.460 INFO:tasks.workunit.client.0.smithi102.stdout:8/760: mknod d0/d15/d5d/dd3/d2c/cfc 0 2024-03-26T18:54:44.462 INFO:tasks.workunit.client.1.smithi190.stdout:6/966: creat d17/d28/df6/da8/df4/dfb/d6a/f149 x:0 0 0 2024-03-26T18:54:44.462 INFO:tasks.workunit.client.1.smithi190.stdout:6/967: truncate d17/f12c 651939 0 2024-03-26T18:54:44.464 INFO:tasks.workunit.client.0.smithi102.stdout:8/761: rename d0/d9/l77 to d0/d3c/df9/lfd 0 2024-03-26T18:54:44.466 INFO:tasks.workunit.client.1.smithi190.stdout:6/968: symlink d17/d28/df6/da8/df4/dfb/l14a 0 2024-03-26T18:54:44.467 INFO:tasks.workunit.client.1.smithi190.stdout:6/969: write d17/d28/d2c/db8/dcc/f135 [216223,9531] 0 2024-03-26T18:54:44.467 INFO:tasks.workunit.client.1.smithi190.stdout:6/970: chown d17/d28/d2c/d63 21440434 1 2024-03-26T18:54:44.467 INFO:tasks.workunit.client.1.smithi190.stdout:6/971: stat d17/ffa 0 2024-03-26T18:54:44.467 INFO:tasks.workunit.client.1.smithi190.stdout:6/972: truncate d17/d28/d2c/db8/df3/d109/d79/f133 1446865 0 2024-03-26T18:54:44.467 INFO:tasks.workunit.client.1.smithi190.stdout:6/973: chown c14 213738500 1 2024-03-26T18:54:44.467 INFO:tasks.workunit.client.0.smithi102.stdout:8/762: link d0/d15/d5d/dd3/d2c/da7/f73 d0/d3c/d96/dd5/ffe 0 2024-03-26T18:54:44.468 INFO:tasks.workunit.client.1.smithi190.stdout:6/974: creat d17/d28/df6/da8/f14b x:0 0 0 2024-03-26T18:54:44.469 INFO:tasks.workunit.client.0.smithi102.stdout:8/763: creat d0/d15/de6/fff x:0 0 0 2024-03-26T18:54:44.469 INFO:tasks.workunit.client.0.smithi102.stdout:8/764: readlink d0/d15/la2 0 2024-03-26T18:54:44.470 INFO:tasks.workunit.client.1.smithi190.stdout:6/975: unlink c12 0 2024-03-26T18:54:44.470 INFO:tasks.workunit.client.1.smithi190.stdout:6/976: dread - d17/d28/d2c/db8/df3/d109/fff zero size 2024-03-26T18:54:44.470 INFO:tasks.workunit.client.1.smithi190.stdout:6/977: creat d17/d28/df6/da8/df4/dfb/d6a/f14c x:0 0 0 2024-03-26T18:54:44.471 INFO:tasks.workunit.client.0.smithi102.stdout:8/765: link d0/d15/d5d/dd3/d2c/da7/l30 d0/d15/d5d/d89/l100 0 2024-03-26T18:54:44.472 INFO:tasks.workunit.client.1.smithi190.stdout:6/978: mknod d17/d28/d2c/c14d 0 2024-03-26T18:54:44.473 INFO:tasks.workunit.client.1.smithi190.stdout:6/979: mknod d17/d28/d8b/d101/c14e 0 2024-03-26T18:54:44.473 INFO:tasks.workunit.client.1.smithi190.stdout:6/980: readlink d17/l30 0 2024-03-26T18:54:44.474 INFO:tasks.workunit.client.1.smithi190.stdout:6/981: chown d17/d28/df6/fc0 60 1 2024-03-26T18:54:44.475 INFO:tasks.workunit.client.1.smithi190.stdout:6/982: creat d17/d28/df6/da8/df4/dfb/dd7/f14f x:0 0 0 2024-03-26T18:54:44.475 INFO:tasks.workunit.client.1.smithi190.stdout:6/983: dread - d17/d28/d2c/db8/df3/f118 zero size 2024-03-26T18:54:44.477 INFO:tasks.workunit.client.1.smithi190.stdout:6/984: mknod d17/d28/df6/da8/df4/dfb/c150 0 2024-03-26T18:54:44.477 INFO:tasks.workunit.client.1.smithi190.stdout:6/985: stat d17/d28/df6/da8/df4/dfb/fba 0 2024-03-26T18:54:44.479 INFO:tasks.workunit.client.1.smithi190.stdout:3/966: dwrite d1/d3/f10 [4194304,4194304] 0 2024-03-26T18:54:44.479 INFO:tasks.workunit.client.1.smithi190.stdout:3/967: write d1/d3/df/d31/f56 [367356,17229] 0 2024-03-26T18:54:44.480 INFO:tasks.workunit.client.1.smithi190.stdout:3/968: mknod d1/de/d81/dad/c12e 0 2024-03-26T18:54:44.481 INFO:tasks.workunit.client.1.smithi190.stdout:3/969: write d1/de/f2d [929422,35022] 0 2024-03-26T18:54:44.488 INFO:tasks.workunit.client.0.smithi102.stdout:2/755: write d3/d11/d15/d31/d39/f4e [375614,127046] 0 2024-03-26T18:54:44.488 INFO:tasks.workunit.client.0.smithi102.stdout:2/756: chown d3/d11/d15/d31/fa3 511416428 1 2024-03-26T18:54:44.488 INFO:tasks.workunit.client.0.smithi102.stdout:2/757: chown d3/d82/de2/def/ff4 295312 1 2024-03-26T18:54:44.494 INFO:tasks.workunit.client.1.smithi190.stdout:6/986: write d17/d28/d2c/fb7 [190447,91975] 0 2024-03-26T18:54:44.503 INFO:tasks.workunit.client.0.smithi102.stdout:2/758: write d3/d82/d93/fcc [4032699,46879] 0 2024-03-26T18:54:44.504 INFO:tasks.workunit.client.0.smithi102.stdout:2/759: dread - d3/d82/d93/ffb zero size 2024-03-26T18:54:44.504 INFO:tasks.workunit.client.0.smithi102.stdout:2/760: dread - d3/d11/d15/ded/ffc zero size 2024-03-26T18:54:44.505 INFO:tasks.workunit.client.0.smithi102.stdout:0/783: dwrite d6/d2d/d3d/d80/fdf [4194304,4194304] 0 2024-03-26T18:54:44.505 INFO:tasks.workunit.client.0.smithi102.stdout:0/784: dread - d6/de/d13/d25/d31/d45/d82/ff3 zero size 2024-03-26T18:54:44.505 INFO:tasks.workunit.client.0.smithi102.stdout:4/636: dwrite d1/d17/f88 [0,4194304] 0 2024-03-26T18:54:44.506 INFO:tasks.workunit.client.0.smithi102.stdout:2/761: link d3/d2a/d5f/fa8 d3/d20/d78/f10e 0 2024-03-26T18:54:44.506 INFO:tasks.workunit.client.0.smithi102.stdout:8/766: dwrite d0/d15/d5d/dd3/d2c/d4e/fcb [0,4194304] 0 2024-03-26T18:54:44.507 INFO:tasks.workunit.client.0.smithi102.stdout:4/637: dread d1/d17/d1f/fcd [0,4194304] 0 2024-03-26T18:54:44.508 INFO:tasks.workunit.client.0.smithi102.stdout:4/638: write d1/d75/d85/fbe [941051,76334] 0 2024-03-26T18:54:44.508 INFO:tasks.workunit.client.0.smithi102.stdout:6/731: dwrite d2/f29 [0,4194304] 0 2024-03-26T18:54:44.510 INFO:tasks.workunit.client.0.smithi102.stdout:2/762: symlink d3/d20/d78/d7c/dca/l10f 0 2024-03-26T18:54:44.519 INFO:tasks.workunit.client.0.smithi102.stdout:8/767: dread d0/d15/fb0 [0,4194304] 0 2024-03-26T18:54:44.520 INFO:tasks.workunit.client.0.smithi102.stdout:0/785: write d6/d2d/d3d/d80/fdf [664872,38690] 0 2024-03-26T18:54:44.520 INFO:tasks.workunit.client.0.smithi102.stdout:0/786: chown d6/de/f15 27 1 2024-03-26T18:54:44.520 INFO:tasks.workunit.client.0.smithi102.stdout:6/732: mkdir d2/da4/df1 0 2024-03-26T18:54:44.520 INFO:tasks.workunit.client.0.smithi102.stdout:0/787: readlink d6/de/d13/d1f/lf5 0 2024-03-26T18:54:44.520 INFO:tasks.workunit.client.0.smithi102.stdout:1/731: dwrite de/d16/d1d/fbd [0,4194304] 0 2024-03-26T18:54:44.520 INFO:tasks.workunit.client.0.smithi102.stdout:0/788: write d6/f11 [4578473,75066] 0 2024-03-26T18:54:44.520 INFO:tasks.workunit.client.1.smithi190.stdout:6/987: dwrite d17/d28/f58 [0,4194304] 0 2024-03-26T18:54:44.521 INFO:tasks.workunit.client.0.smithi102.stdout:8/768: creat d0/d15/da0/f101 x:0 0 0 2024-03-26T18:54:44.521 INFO:tasks.workunit.client.0.smithi102.stdout:8/769: chown d0/d15/d5d/dd3/c36 1380 1 2024-03-26T18:54:44.521 INFO:tasks.workunit.client.0.smithi102.stdout:8/770: dread - d0/d15/d5d/dd3/d26/fad zero size 2024-03-26T18:54:44.527 INFO:tasks.workunit.client.0.smithi102.stdout:6/733: rename d2/d7/fd3 to d2/df/d71/d80/ff2 0 2024-03-26T18:54:44.529 INFO:tasks.workunit.client.1.smithi190.stdout:6/988: mkdir d17/d28/d74/d97/d151 0 2024-03-26T18:54:44.532 INFO:tasks.workunit.client.0.smithi102.stdout:0/789: link d6/de/c33 d6/c110 0 2024-03-26T18:54:44.534 INFO:tasks.workunit.client.1.smithi190.stdout:6/989: mkdir d17/d28/df6/da8/df4/dfb/dd7/ddb/d152 0 2024-03-26T18:54:44.535 INFO:tasks.workunit.client.0.smithi102.stdout:8/771: mknod d0/d15/d5d/dd3/d2c/d4e/c102 0 2024-03-26T18:54:44.535 INFO:tasks.workunit.client.0.smithi102.stdout:8/772: dread - d0/d15/d5d/fc7 zero size 2024-03-26T18:54:44.535 INFO:tasks.workunit.client.0.smithi102.stdout:8/773: chown d0/d3c/d96 0 1 2024-03-26T18:54:44.536 INFO:tasks.workunit.client.0.smithi102.stdout:6/734: rmdir d2/d3d/d53/d77 39 2024-03-26T18:54:44.536 INFO:tasks.workunit.client.0.smithi102.stdout:6/735: truncate d2/d7/d3c/f41 429532 0 2024-03-26T18:54:44.537 INFO:tasks.workunit.client.1.smithi190.stdout:6/990: mknod d17/d28/d2c/db8/df3/c153 0 2024-03-26T18:54:44.538 INFO:tasks.workunit.client.0.smithi102.stdout:1/732: rmdir de/d2e/d62 39 2024-03-26T18:54:44.539 INFO:tasks.workunit.client.0.smithi102.stdout:1/733: stat de/d2e/f55 0 2024-03-26T18:54:44.539 INFO:tasks.workunit.client.0.smithi102.stdout:1/734: truncate de/d16/d1d/d2a/d2c/f60 225435 0 2024-03-26T18:54:44.540 INFO:tasks.workunit.client.0.smithi102.stdout:0/790: creat d6/de/d13/d25/d31/d45/d82/d84/d88/db2/dd2/f111 x:0 0 0 2024-03-26T18:54:44.542 INFO:tasks.workunit.client.1.smithi190.stdout:6/991: unlink d17/dd3/d113/f11c 0 2024-03-26T18:54:44.542 INFO:tasks.workunit.client.1.smithi190.stdout:6/992: dread - d17/d28/d2c/db8/df3/d109/fff zero size 2024-03-26T18:54:44.542 INFO:tasks.workunit.client.1.smithi190.stdout:6/993: read d17/d28/d2b/f60 [1112661,117185] 0 2024-03-26T18:54:44.542 INFO:tasks.workunit.client.1.smithi190.stdout:6/994: fdatasync d17/d28/d74/d97/fbc 0 2024-03-26T18:54:44.543 INFO:tasks.workunit.client.0.smithi102.stdout:6/736: link d2/df/d71/fbb d2/d3d/d53/d5c/dc5/ff3 0 2024-03-26T18:54:44.544 INFO:tasks.workunit.client.0.smithi102.stdout:1/735: rename de/d1f/c69 to de/d2e/cf9 0 2024-03-26T18:54:44.546 INFO:tasks.workunit.client.1.smithi190.stdout:6/995: rename d17/la1 to d17/d28/d2c/db8/df3/d109/d79/ddf/l154 0 2024-03-26T18:54:44.547 INFO:tasks.workunit.client.0.smithi102.stdout:6/737: link d2/df/d35/ld8 d2/d7/d3c/d78/dc4/lf4 0 2024-03-26T18:54:44.549 INFO:tasks.workunit.client.0.smithi102.stdout:0/791: rename d6/de/d13/d1f/d55/d57/l9f to d6/dbc/l112 0 2024-03-26T18:54:44.551 INFO:tasks.workunit.client.0.smithi102.stdout:1/736: mkdir de/d16/d1d/d2a/d2c/d56/dfa 0 2024-03-26T18:54:44.554 INFO:tasks.workunit.client.0.smithi102.stdout:6/738: rename d2/df/l32 to d2/df/d71/de6/lf5 0 2024-03-26T18:54:44.555 INFO:tasks.workunit.client.0.smithi102.stdout:6/739: creat d2/d12/ff6 x:0 0 0 2024-03-26T18:54:44.557 INFO:tasks.workunit.client.0.smithi102.stdout:0/792: stat d6/l10 0 2024-03-26T18:54:44.557 INFO:tasks.workunit.client.0.smithi102.stdout:8/774: dread d0/d15/d5d/dd3/d2c/d2f/f7e [0,4194304] 0 2024-03-26T18:54:44.557 INFO:tasks.workunit.client.0.smithi102.stdout:8/775: fsync d0/d15/d5d/dd3/d26/d19/d76/f9c 0 2024-03-26T18:54:44.560 INFO:tasks.workunit.client.0.smithi102.stdout:8/776: rename d0/d3c/d96/d98/fe4 to d0/d15/d5d/dd3/d2c/d33/d87/f103 0 2024-03-26T18:54:44.560 INFO:tasks.workunit.client.0.smithi102.stdout:2/763: dwrite d3/d20/d78/d7c/f7e [0,4194304] 0 2024-03-26T18:54:44.561 INFO:tasks.workunit.client.0.smithi102.stdout:8/777: mknod d0/d15/d5d/dd3/d2c/d4e/de3/c104 0 2024-03-26T18:54:44.564 INFO:tasks.workunit.client.0.smithi102.stdout:2/764: rename d3/d11/d27/l56 to d3/d82/d93/d9e/d99/l110 0 2024-03-26T18:54:44.565 INFO:tasks.workunit.client.0.smithi102.stdout:8/778: creat d0/d3c/d96/d98/dcf/f105 x:0 0 0 2024-03-26T18:54:44.568 INFO:tasks.workunit.client.0.smithi102.stdout:2/765: getdents d3/d20 0 2024-03-26T18:54:44.570 INFO:tasks.workunit.client.0.smithi102.stdout:2/766: symlink d3/d20/l111 0 2024-03-26T18:54:44.570 INFO:tasks.workunit.client.0.smithi102.stdout:6/740: write d2/df/d35/d43/f94 [779943,95125] 0 2024-03-26T18:54:44.570 INFO:tasks.workunit.client.0.smithi102.stdout:2/767: mknod d3/d74/c112 0 2024-03-26T18:54:44.571 INFO:tasks.workunit.client.0.smithi102.stdout:2/768: chown d3/d11/d15 492102378 1 2024-03-26T18:54:44.572 INFO:tasks.workunit.client.1.smithi190.stdout:3/970: dwrite d1/d3/d52/f6d [0,4194304] 0 2024-03-26T18:54:44.572 INFO:tasks.workunit.client.1.smithi190.stdout:3/971: creat d1/d3/df/d18/d35/f12f x:0 0 0 2024-03-26T18:54:44.573 INFO:tasks.workunit.client.0.smithi102.stdout:6/741: mknod d2/df/d71/d80/d91/db8/cf7 0 2024-03-26T18:54:44.573 INFO:tasks.workunit.client.0.smithi102.stdout:2/769: getdents d3/d2d/d69 0 2024-03-26T18:54:44.573 INFO:tasks.workunit.client.0.smithi102.stdout:4/639: truncate d1/d17/f88 1143399 0 2024-03-26T18:54:44.575 INFO:tasks.workunit.client.0.smithi102.stdout:4/640: getdents d1/d25 0 2024-03-26T18:54:44.576 INFO:tasks.workunit.client.0.smithi102.stdout:9/683: dwrite df/d1a/d23/d4e/d8e/f5f [0,4194304] 0 2024-03-26T18:54:44.576 INFO:tasks.workunit.client.0.smithi102.stdout:6/742: dread d2/df/d35/f6e [0,4194304] 0 2024-03-26T18:54:44.577 INFO:tasks.workunit.client.1.smithi190.stdout:6/996: dwrite d17/d28/df6/fd8 [0,4194304] 0 2024-03-26T18:54:44.577 INFO:tasks.workunit.client.1.smithi190.stdout:3/972: mknod d1/d3/df/d18/d57/d126/c130 0 2024-03-26T18:54:44.577 INFO:tasks.workunit.client.1.smithi190.stdout:6/997: chown d17/d28/d2c/f53 4076845 1 2024-03-26T18:54:44.577 INFO:tasks.workunit.client.1.smithi190.stdout:6/998: fdatasync d17/d28/d2c/db8/f146 0 2024-03-26T18:54:44.577 INFO:tasks.workunit.client.1.smithi190.stdout:6/999: write d17/d28/d2c/db8/df3/d109/f89 [993982,100724] 0 2024-03-26T18:54:44.581 INFO:tasks.workunit.client.0.smithi102.stdout:9/684: unlink df/d1a/d23/d4e/d8e/f5f 0 2024-03-26T18:54:44.586 INFO:tasks.workunit.client.0.smithi102.stdout:6/743: write d2/d12/f2d [1388148,112274] 0 2024-03-26T18:54:44.590 INFO:tasks.workunit.client.0.smithi102.stdout:6/744: symlink d2/d7/lf8 0 2024-03-26T18:54:44.590 INFO:tasks.workunit.client.0.smithi102.stdout:6/745: write d2/df/f96 [4375454,32859] 0 2024-03-26T18:54:44.590 INFO:tasks.workunit.client.0.smithi102.stdout:6/746: write d2/df/d35/d43/d75/fb5 [913350,11820] 0 2024-03-26T18:54:44.591 INFO:tasks.workunit.client.1.smithi190.stdout:3/973: read d1/d3/df/f102 [3378259,22333] 0 2024-03-26T18:54:44.591 INFO:tasks.workunit.client.1.smithi190.stdout:3/974: truncate d1/d3/df/d18/d57/f11c 835246 0 2024-03-26T18:54:44.591 INFO:tasks.workunit.client.0.smithi102.stdout:6/747: mkdir d2/df/d71/d80/d91/db8/df9 0 2024-03-26T18:54:44.591 INFO:tasks.workunit.client.0.smithi102.stdout:6/748: stat d2/df/d71/fbb 0 2024-03-26T18:54:44.592 INFO:tasks.workunit.client.0.smithi102.stdout:6/749: creat d2/df/d35/da1/ffa x:0 0 0 2024-03-26T18:54:44.593 INFO:tasks.workunit.client.0.smithi102.stdout:6/750: getdents d2/d7/d3c 0 2024-03-26T18:54:44.594 INFO:tasks.workunit.client.0.smithi102.stdout:6/751: truncate d2/d3d/d53/f5b 5062267 0 2024-03-26T18:54:44.595 INFO:tasks.workunit.client.0.smithi102.stdout:6/752: mkdir d2/df/d35/d43/d75/dfb 0 2024-03-26T18:54:44.595 INFO:tasks.workunit.client.0.smithi102.stdout:6/753: chown d2/df/d35/c6d 276656 1 2024-03-26T18:54:44.595 INFO:tasks.workunit.client.0.smithi102.stdout:6/754: read - d2/df/d35/d43/d75/fdc zero size 2024-03-26T18:54:44.596 INFO:tasks.workunit.client.0.smithi102.stdout:6/755: symlink d2/d7/d3c/d78/lfc 0 2024-03-26T18:54:44.596 INFO:tasks.workunit.client.0.smithi102.stdout:6/756: truncate d2/df/d35/d43/fd5 655638 0 2024-03-26T18:54:44.600 INFO:tasks.workunit.client.0.smithi102.stdout:6/757: dread d2/d7/f67 [0,4194304] 0 2024-03-26T18:54:44.618 INFO:tasks.workunit.client.0.smithi102.stdout:9/685: dwrite df/d1a/d23/d4e/f4f [0,4194304] 0 2024-03-26T18:54:44.619 INFO:tasks.workunit.client.1.smithi190.stdout:3/975: dwrite d1/d3/df/d18/d35/d41/f103 [0,4194304] 0 2024-03-26T18:54:44.619 INFO:tasks.workunit.client.1.smithi190.stdout:3/976: fdatasync d1/d3/f34 0 2024-03-26T18:54:44.620 INFO:tasks.workunit.client.0.smithi102.stdout:9/686: dread df/d1a/d81/fa2 [0,4194304] 0 2024-03-26T18:54:44.620 INFO:tasks.workunit.client.0.smithi102.stdout:9/687: dread - df/d1a/d1d/f3a zero size 2024-03-26T18:54:44.620 INFO:tasks.workunit.client.0.smithi102.stdout:9/688: fsync df/fca 0 2024-03-26T18:54:44.637 INFO:tasks.workunit.client.1.smithi190.stdout:3/977: dwrite d1/d3/df/d31/dc4/f106 [0,4194304] 0 2024-03-26T18:54:44.652 INFO:tasks.workunit.client.0.smithi102.stdout:6/758: creat d2/d7/d3c/ffd x:0 0 0 2024-03-26T18:54:44.655 INFO:tasks.workunit.client.0.smithi102.stdout:9/689: creat df/d46/df7/ffd x:0 0 0 2024-03-26T18:54:44.658 INFO:tasks.workunit.client.0.smithi102.stdout:8/779: rename d0/d15/d5d/dd3/d2c/d33/d87/f103 to d0/d3c/d96/d98/dda/f106 0 2024-03-26T18:54:44.658 INFO:tasks.workunit.client.0.smithi102.stdout:8/780: stat d0/d3c/d6f/fdc 0 2024-03-26T18:54:44.668 INFO:tasks.workunit.client.0.smithi102.stdout:6/759: truncate d2/d3d/d53/d77/f88 1904623 0 2024-03-26T18:54:44.669 INFO:tasks.workunit.client.0.smithi102.stdout:4/641: rename d1/d17/d2a/l40 to d1/d75/d85/lcf 0 2024-03-26T18:54:44.670 INFO:tasks.workunit.client.0.smithi102.stdout:4/642: dread d1/d17/d1f/fcd [0,4194304] 0 2024-03-26T18:54:44.671 INFO:tasks.workunit.client.0.smithi102.stdout:8/781: symlink d0/d15/da0/l107 0 2024-03-26T18:54:44.674 INFO:tasks.workunit.client.0.smithi102.stdout:6/760: getdents d2/d3d/d53/d77 0 2024-03-26T18:54:44.676 INFO:tasks.workunit.client.0.smithi102.stdout:9/690: rename df/c14 to df/d1a/d65/cfe 0 2024-03-26T18:54:44.682 INFO:tasks.workunit.client.0.smithi102.stdout:9/691: symlink df/d1a/d1d/d72/d7d/db7/df8/lff 0 2024-03-26T18:54:44.684 INFO:tasks.workunit.client.0.smithi102.stdout:4/643: dread d1/d17/d1f/f2d [0,4194304] 0 2024-03-26T18:54:44.685 INFO:tasks.workunit.client.0.smithi102.stdout:6/761: rename d2/d7/l1a to d2/df/d71/d80/d91/db8/lfe 0 2024-03-26T18:54:44.688 INFO:tasks.workunit.client.0.smithi102.stdout:9/692: mknod df/d1a/d1d/d72/d7d/db7/dd5/c100 0 2024-03-26T18:54:44.689 INFO:tasks.workunit.client.1.smithi190.stdout:3/978: rmdir d1/d3/df 39 2024-03-26T18:54:44.690 INFO:tasks.workunit.client.0.smithi102.stdout:6/762: creat d2/df/d71/d80/d91/db8/fff x:0 0 0 2024-03-26T18:54:44.691 INFO:tasks.workunit.client.1.smithi190.stdout:3/979: chown d1/de/c48 255286 1 2024-03-26T18:54:44.693 INFO:tasks.workunit.client.0.smithi102.stdout:9/693: creat df/d1a/d23/d60/f101 x:0 0 0 2024-03-26T18:54:44.696 INFO:tasks.workunit.client.0.smithi102.stdout:9/694: dread df/d1a/d81/d88/fc9 [0,4194304] 0 2024-03-26T18:54:44.696 INFO:tasks.workunit.client.0.smithi102.stdout:9/695: chown df/d1a/d23/d4e/f4f 3 1 2024-03-26T18:54:44.696 INFO:tasks.workunit.client.0.smithi102.stdout:9/696: creat df/d46/da8/f102 x:0 0 0 2024-03-26T18:54:44.697 INFO:tasks.workunit.client.0.smithi102.stdout:9/697: write df/d1a/d23/f7e [153095,60229] 0 2024-03-26T18:54:44.697 INFO:tasks.workunit.client.0.smithi102.stdout:9/698: read df/d1a/f86 [1096986,19359] 0 2024-03-26T18:54:44.703 INFO:tasks.workunit.client.0.smithi102.stdout:8/782: dwrite d0/d15/d5d/dac/fc3 [0,4194304] 0 2024-03-26T18:54:44.706 INFO:tasks.workunit.client.0.smithi102.stdout:8/783: symlink d0/d15/d5d/dd3/d26/d19/d9f/l108 0 2024-03-26T18:54:44.722 INFO:tasks.workunit.client.0.smithi102.stdout:4/644: dwrite d1/d17/d1f/d5d/d8b/d3e/fa4 [0,4194304] 0 2024-03-26T18:54:44.722 INFO:tasks.workunit.client.0.smithi102.stdout:4/645: chown d1/d17/d1f/d5d/d8b/d3e/f58 10 1 2024-03-26T18:54:44.722 INFO:tasks.workunit.client.0.smithi102.stdout:4/646: fsync d1/d17/d1a/d21/d77/f81 0 2024-03-26T18:54:44.722 INFO:tasks.workunit.client.0.smithi102.stdout:4/647: write d1/fa0 [689333,107857] 0 2024-03-26T18:54:44.723 INFO:tasks.workunit.client.0.smithi102.stdout:4/648: symlink d1/d75/d85/ld0 0 2024-03-26T18:54:44.724 INFO:tasks.workunit.client.0.smithi102.stdout:4/649: symlink d1/d17/d1f/d5d/d8b/d3e/d74/ld1 0 2024-03-26T18:54:44.725 INFO:tasks.workunit.client.0.smithi102.stdout:6/763: dwrite d2/d3d/d53/d5c/dc5/fc8 [0,4194304] 0 2024-03-26T18:54:44.726 INFO:tasks.workunit.client.0.smithi102.stdout:4/650: mknod d1/d75/d85/cd2 0 2024-03-26T18:54:44.727 INFO:tasks.workunit.client.0.smithi102.stdout:6/764: symlink d2/d3d/d53/d5c/dc5/l100 0 2024-03-26T18:54:44.733 INFO:tasks.workunit.client.0.smithi102.stdout:9/699: dwrite df/d1a/f9a [0,4194304] 0 2024-03-26T18:54:44.734 INFO:tasks.workunit.client.0.smithi102.stdout:8/784: dwrite d0/d15/d5d/dd3/d26/d19/d9f/fa9 [0,4194304] 0 2024-03-26T18:54:44.738 INFO:tasks.workunit.client.0.smithi102.stdout:9/700: truncate df/fa6 1007836 0 2024-03-26T18:54:44.738 INFO:tasks.workunit.client.0.smithi102.stdout:9/701: truncate df/d1a/d23/d29/fc4 837364 0 2024-03-26T18:54:44.740 INFO:tasks.workunit.client.0.smithi102.stdout:4/651: write d1/d25/d76/f86 [3651889,78958] 0 2024-03-26T18:54:44.741 INFO:tasks.workunit.client.0.smithi102.stdout:8/785: rename d0/d15/d5d/dd3/d26/d80/ff1 to d0/d3c/d6f/f109 0 2024-03-26T18:54:44.741 INFO:tasks.workunit.client.0.smithi102.stdout:8/786: chown d0/d15/d5d/dd3/d2c/da7/f73 28008 1 2024-03-26T18:54:44.742 INFO:tasks.workunit.client.0.smithi102.stdout:8/787: chown d0/d15/d5d/dd3/d26/d5c/la3 845990989 1 2024-03-26T18:54:44.742 INFO:tasks.workunit.client.0.smithi102.stdout:9/702: symlink df/d84/l103 0 2024-03-26T18:54:44.744 INFO:tasks.workunit.client.0.smithi102.stdout:6/765: dwrite d2/df/d35/d43/d75/fdc [0,4194304] 0 2024-03-26T18:54:44.744 INFO:tasks.workunit.client.0.smithi102.stdout:4/652: truncate d1/d17/d1a/d21/d77/f71 1195972 0 2024-03-26T18:54:44.747 INFO:tasks.workunit.client.0.smithi102.stdout:6/766: dread d2/df/f27 [0,4194304] 0 2024-03-26T18:54:44.748 INFO:tasks.workunit.client.0.smithi102.stdout:9/703: getdents df/d1a/d23/d60 0 2024-03-26T18:54:44.749 INFO:tasks.workunit.client.0.smithi102.stdout:9/704: creat df/d1a/d65/dbe/f104 x:0 0 0 2024-03-26T18:54:44.750 INFO:tasks.workunit.client.0.smithi102.stdout:6/767: truncate d2/df/d72/db7/f6f 291738 0 2024-03-26T18:54:44.750 INFO:tasks.workunit.client.0.smithi102.stdout:6/768: fdatasync d2/df/f96 0 2024-03-26T18:54:44.751 INFO:tasks.workunit.client.0.smithi102.stdout:6/769: chown d2/d3d/d53/d5c/dc5/fc8 7584461 1 2024-03-26T18:54:44.751 INFO:tasks.workunit.client.0.smithi102.stdout:4/653: getdents d1/d17/dbc 0 2024-03-26T18:54:44.752 INFO:tasks.workunit.client.0.smithi102.stdout:4/654: read - d1/d17/dbc/fcc zero size 2024-03-26T18:54:44.752 INFO:tasks.workunit.client.0.smithi102.stdout:4/655: creat d1/d17/d1a/d21/d77/fd3 x:0 0 0 2024-03-26T18:54:44.757 INFO:tasks.workunit.client.0.smithi102.stdout:7/652: sync 2024-03-26T18:54:44.758 INFO:tasks.workunit.client.0.smithi102.stdout:6/770: rmdir d2/df/d71 39 2024-03-26T18:54:44.758 INFO:tasks.workunit.client.0.smithi102.stdout:7/653: getdents d7/d12/d4d/d9e 0 2024-03-26T18:54:44.759 INFO:tasks.workunit.client.0.smithi102.stdout:0/793: sync 2024-03-26T18:54:44.760 INFO:tasks.workunit.client.0.smithi102.stdout:1/737: sync 2024-03-26T18:54:44.760 INFO:tasks.workunit.client.0.smithi102.stdout:2/770: sync 2024-03-26T18:54:44.760 INFO:tasks.workunit.client.0.smithi102.stdout:1/738: getdents de/d1f/de3/da6/dca/df7 0 2024-03-26T18:54:44.760 INFO:tasks.workunit.client.0.smithi102.stdout:0/794: creat d6/d2d/d3d/d80/f113 x:0 0 0 2024-03-26T18:54:44.760 INFO:tasks.workunit.client.0.smithi102.stdout:6/771: mknod d2/d7/d3c/d78/c101 0 2024-03-26T18:54:44.761 INFO:tasks.workunit.client.0.smithi102.stdout:7/654: rmdir d7/d10/d67/d69 39 2024-03-26T18:54:44.761 INFO:tasks.workunit.client.0.smithi102.stdout:7/655: chown d7/d10/d67/d69/d78/lb8 899 1 2024-03-26T18:54:44.766 INFO:tasks.workunit.client.0.smithi102.stdout:1/739: link de/d1f/de3/da6/fba de/d1f/de3/dd6/ffb 0 2024-03-26T18:54:44.766 INFO:tasks.workunit.client.0.smithi102.stdout:1/740: chown de/d16/d1d/d2a/d2c/d56/d78/d80/c93 1845641125 1 2024-03-26T18:54:44.767 INFO:tasks.workunit.client.0.smithi102.stdout:6/772: mknod d2/df/d71/d80/d91/db8/df9/c102 0 2024-03-26T18:54:44.767 INFO:tasks.workunit.client.0.smithi102.stdout:6/773: creat d2/df/d35/d43/f103 x:0 0 0 2024-03-26T18:54:44.770 INFO:tasks.workunit.client.0.smithi102.stdout:7/656: link d7/d10/d27/f55 d7/d12/d4d/d62/fc5 0 2024-03-26T18:54:44.770 INFO:tasks.workunit.client.0.smithi102.stdout:7/657: stat d7/d4e 0 2024-03-26T18:54:44.772 INFO:tasks.workunit.client.0.smithi102.stdout:0/795: dread d6/d2d/d3d/f40 [0,4194304] 0 2024-03-26T18:54:44.773 INFO:tasks.workunit.client.0.smithi102.stdout:1/741: rename de/d16/d1d/d2a/d2c/d56/d6e/d9d to de/d16/d1d/d2a/d2c/d56/d6e/dfc 0 2024-03-26T18:54:44.775 INFO:tasks.workunit.client.0.smithi102.stdout:4/656: dwrite d1/d17/d1f/d5d/d8b/f7f [0,4194304] 0 2024-03-26T18:54:44.776 INFO:tasks.workunit.client.0.smithi102.stdout:0/796: mknod d6/d2d/d3d/d80/c114 0 2024-03-26T18:54:44.780 INFO:tasks.workunit.client.0.smithi102.stdout:4/657: creat d1/d17/dbb/fd4 x:0 0 0 2024-03-26T18:54:44.786 INFO:tasks.workunit.client.0.smithi102.stdout:0/797: dread d6/de/d13/d1f/d55/d57/f5e [0,4194304] 0 2024-03-26T18:54:44.786 INFO:tasks.workunit.client.0.smithi102.stdout:0/798: getdents d6/dbe/deb/dfd 0 2024-03-26T18:54:44.787 INFO:tasks.workunit.client.0.smithi102.stdout:0/799: fsync d6/de/d13/d1f/f3b 0 2024-03-26T18:54:44.788 INFO:tasks.workunit.client.0.smithi102.stdout:0/800: link d6/d2d/f10e d6/de/d13/d1f/d55/d57/d7d/d7f/f115 0 2024-03-26T18:54:44.788 INFO:tasks.workunit.client.0.smithi102.stdout:0/801: unlink d6/c110 0 2024-03-26T18:54:44.789 INFO:tasks.workunit.client.0.smithi102.stdout:0/802: creat d6/d3e/d69/d8c/d8f/f116 x:0 0 0 2024-03-26T18:54:44.791 INFO:tasks.workunit.client.0.smithi102.stdout:0/803: truncate d6/d2d/d3d/d80/d83/fd1 207299 0 2024-03-26T18:54:44.791 INFO:tasks.workunit.client.0.smithi102.stdout:0/804: creat d6/d3e/d69/d8c/d8f/f117 x:0 0 0 2024-03-26T18:54:44.791 INFO:tasks.workunit.client.0.smithi102.stdout:0/805: write d6/de/d93/de4/fe7 [855133,10635] 0 2024-03-26T18:54:44.791 INFO:tasks.workunit.client.0.smithi102.stdout:0/806: write d6/de/d13/d9a/d85/f58 [4203645,67860] 0 2024-03-26T18:54:44.795 INFO:tasks.workunit.client.0.smithi102.stdout:4/658: read d1/d17/d1f/d5d/d8b/d3e/fa4 [959283,22066] 0 2024-03-26T18:54:44.796 INFO:tasks.workunit.client.0.smithi102.stdout:4/659: link d1/d27/fca d1/fd5 0 2024-03-26T18:54:44.796 INFO:tasks.workunit.client.0.smithi102.stdout:4/660: fdatasync d1/d17/d1f/d5d/d8b/d3e/fa4 0 2024-03-26T18:54:44.796 INFO:tasks.workunit.client.0.smithi102.stdout:4/661: write d1/f8 [8481000,48710] 0 2024-03-26T18:54:44.810 INFO:tasks.workunit.client.0.smithi102.stdout:2/771: dwrite d3/d2a/ff7 [0,4194304] 0 2024-03-26T18:54:44.811 INFO:tasks.workunit.client.0.smithi102.stdout:1/742: dwrite de/d1f/d9f/fd8 [0,4194304] 0 2024-03-26T18:54:44.813 INFO:tasks.workunit.client.0.smithi102.stdout:2/772: mknod d3/d11/d27/c113 0 2024-03-26T18:54:44.815 INFO:tasks.workunit.client.0.smithi102.stdout:1/743: stat de/d16/d1d/d2a/l42 0 2024-03-26T18:54:44.816 INFO:tasks.workunit.client.0.smithi102.stdout:2/773: rmdir d3/d82/de2 39 2024-03-26T18:54:44.819 INFO:tasks.workunit.client.0.smithi102.stdout:1/744: rmdir de/d16/d1d/d2a/d2c/d56/d78/d80/ddc 0 2024-03-26T18:54:44.820 INFO:tasks.workunit.client.0.smithi102.stdout:7/658: dwrite d7/d10/d67/fb2 [0,4194304] 0 2024-03-26T18:54:44.820 INFO:tasks.workunit.client.0.smithi102.stdout:6/774: dwrite d2/d3d/d53/d63/fc7 [0,4194304] 0 2024-03-26T18:54:44.820 INFO:tasks.workunit.client.0.smithi102.stdout:6/775: chown d2/df/d71/d80/d91/cd9 214 1 2024-03-26T18:54:44.820 INFO:tasks.workunit.client.0.smithi102.stdout:6/776: write d2/df/d35/d43/fd1 [255236,78358] 0 2024-03-26T18:54:44.820 INFO:tasks.workunit.client.0.smithi102.stdout:4/662: dwrite d1/d25/f69 [0,4194304] 0 2024-03-26T18:54:44.821 INFO:tasks.workunit.client.0.smithi102.stdout:0/807: dwrite d6/d2d/d3d/fa2 [0,4194304] 0 2024-03-26T18:54:44.821 INFO:tasks.workunit.client.0.smithi102.stdout:2/774: mknod d3/d2a/d5f/c114 0 2024-03-26T18:54:44.821 INFO:tasks.workunit.client.0.smithi102.stdout:0/808: chown d6/dbc/l112 2267 1 2024-03-26T18:54:44.821 INFO:tasks.workunit.client.0.smithi102.stdout:2/775: read - d3/f75 zero size 2024-03-26T18:54:44.821 INFO:tasks.workunit.client.0.smithi102.stdout:0/809: truncate d6/de/f28 7734664 0 2024-03-26T18:54:44.822 INFO:tasks.workunit.client.1.smithi190.stdout:3/980: sync 2024-03-26T18:54:44.822 INFO:tasks.workunit.client.1.smithi190.stdout:3/981: write d1/d3/f10 [100432,110229] 0 2024-03-26T18:54:44.826 INFO:tasks.workunit.client.1.smithi190.stdout:3/982: unlink d1/de/d8a/de1/f116 0 2024-03-26T18:54:44.827 INFO:tasks.workunit.client.1.smithi190.stdout:3/983: write d1/de/d8a/f95 [1206632,104749] 0 2024-03-26T18:54:44.827 INFO:tasks.workunit.client.1.smithi190.stdout:3/984: chown d1/d3/df/d31 73695972 1 2024-03-26T18:54:44.827 INFO:tasks.workunit.client.1.smithi190.stdout:3/985: write d1/d3/df/d18/d57/f11c [487074,56009] 0 2024-03-26T18:54:44.828 INFO:tasks.workunit.client.0.smithi102.stdout:6/777: mknod d2/d3d/c104 0 2024-03-26T18:54:44.834 INFO:tasks.workunit.client.0.smithi102.stdout:4/663: creat d1/d27/d7a/fd6 x:0 0 0 2024-03-26T18:54:44.836 INFO:tasks.workunit.client.0.smithi102.stdout:7/659: dread d7/ff [0,4194304] 0 2024-03-26T18:54:44.837 INFO:tasks.workunit.client.0.smithi102.stdout:2/776: mknod d3/d82/d93/d9e/d1a/c115 0 2024-03-26T18:54:44.837 INFO:tasks.workunit.client.0.smithi102.stdout:2/777: chown d3/d11/d15/d46/f47 17225 1 2024-03-26T18:54:44.837 INFO:tasks.workunit.client.0.smithi102.stdout:4/664: mknod d1/d17/dbc/cd7 0 2024-03-26T18:54:44.839 INFO:tasks.workunit.client.0.smithi102.stdout:2/778: rename d3/d20/f62 to d3/d2a/f116 0 2024-03-26T18:54:44.842 INFO:tasks.workunit.client.0.smithi102.stdout:4/665: rename d1/d6b/l8f to d1/d75/ld8 0 2024-03-26T18:54:44.842 INFO:tasks.workunit.client.0.smithi102.stdout:1/745: dwrite de/d16/d1d/d61/d6a/fd1 [0,4194304] 0 2024-03-26T18:54:44.842 INFO:tasks.workunit.client.0.smithi102.stdout:2/779: write d3/d20/fa1 [2172722,3319] 0 2024-03-26T18:54:44.844 INFO:tasks.workunit.client.0.smithi102.stdout:4/666: rename d1/d17/d1f/d5d/d8b/f7f to d1/d17/d1f/fd9 0 2024-03-26T18:54:44.845 INFO:tasks.workunit.client.0.smithi102.stdout:0/810: dwrite d6/de/d13/d1f/f3b [4194304,4194304] 0 2024-03-26T18:54:44.846 INFO:tasks.workunit.client.0.smithi102.stdout:0/811: chown d6/de/d13/d1f/l7a 114925 1 2024-03-26T18:54:44.849 INFO:tasks.workunit.client.0.smithi102.stdout:1/746: link de/d16/d1d/d2a/d2c/d56/d78/d80/ld3 de/d16/d1d/d2a/d2c/d56/d6e/dfc/lfd 0 2024-03-26T18:54:44.851 INFO:tasks.workunit.client.0.smithi102.stdout:2/780: link d3/f75 d3/d11/d15/d31/d39/f117 0 2024-03-26T18:54:44.852 INFO:tasks.workunit.client.0.smithi102.stdout:4/667: symlink d1/d6b/lda 0 2024-03-26T18:54:44.856 INFO:tasks.workunit.client.0.smithi102.stdout:4/668: creat d1/fdb x:0 0 0 2024-03-26T18:54:44.856 INFO:tasks.workunit.client.0.smithi102.stdout:4/669: read - d1/d27/d98/fb9 zero size 2024-03-26T18:54:44.857 INFO:tasks.workunit.client.0.smithi102.stdout:4/670: mkdir d1/d17/d2a/d9a/ddc 0 2024-03-26T18:54:44.858 INFO:tasks.workunit.client.0.smithi102.stdout:4/671: fsync d1/d17/dbc/fcc 0 2024-03-26T18:54:44.858 INFO:tasks.workunit.client.0.smithi102.stdout:1/747: dread de/d1f/de3/dd6/ffb [0,4194304] 0 2024-03-26T18:54:44.858 INFO:tasks.workunit.client.0.smithi102.stdout:4/672: creat d1/fdd x:0 0 0 2024-03-26T18:54:44.860 INFO:tasks.workunit.client.0.smithi102.stdout:1/748: truncate de/d16/f35 1615823 0 2024-03-26T18:54:44.863 INFO:tasks.workunit.client.0.smithi102.stdout:1/749: rmdir de/d1f/de3/dec 0 2024-03-26T18:54:44.863 INFO:tasks.workunit.client.0.smithi102.stdout:4/673: rmdir d1/d25/d76 39 2024-03-26T18:54:44.865 INFO:tasks.workunit.client.0.smithi102.stdout:1/750: getdents de/d16/d1d/d2a/d2c/d56/d78/de0 0 2024-03-26T18:54:44.868 INFO:tasks.workunit.client.0.smithi102.stdout:1/751: dread de/d16/d1d/d2a/d2c/d56/d6e/fde [0,4194304] 0 2024-03-26T18:54:44.898 INFO:tasks.workunit.client.0.smithi102.stdout:1/752: dwrite de/d16/d1d/d2a/d2c/d43/f86 [0,4194304] 0 2024-03-26T18:54:44.898 INFO:tasks.workunit.client.0.smithi102.stdout:1/753: write de/d16/d1d/d2a/f9e [236587,13661] 0 2024-03-26T18:54:44.898 INFO:tasks.workunit.client.0.smithi102.stdout:1/754: chown de/d16/c24 967 1 2024-03-26T18:54:44.898 INFO:tasks.workunit.client.0.smithi102.stdout:4/674: dwrite d1/d17/d1f/d5d/d8b/f4e [0,4194304] 0 2024-03-26T18:54:44.898 INFO:tasks.workunit.client.0.smithi102.stdout:4/675: write d1/d17/d1f/d5d/d8b/fc9 [843857,97357] 0 2024-03-26T18:54:44.900 INFO:tasks.workunit.client.0.smithi102.stdout:1/755: creat de/d16/d1d/d2a/d2c/ffe x:0 0 0 2024-03-26T18:54:44.901 INFO:tasks.workunit.client.0.smithi102.stdout:4/676: symlink d1/d17/d2a/d9a/lde 0 2024-03-26T18:54:44.901 INFO:tasks.workunit.client.0.smithi102.stdout:4/677: creat d1/d17/d1a/fdf x:0 0 0 2024-03-26T18:54:44.901 INFO:tasks.workunit.client.0.smithi102.stdout:4/678: fsync d1/d17/d1f/d5d/d8b/f51 0 2024-03-26T18:54:44.903 INFO:tasks.workunit.client.0.smithi102.stdout:4/679: link d1/d17/d1f/d5d/d8b/l5a d1/d27/d98/le0 0 2024-03-26T18:54:44.904 INFO:tasks.workunit.client.0.smithi102.stdout:4/680: mkdir d1/de1 0 2024-03-26T18:54:44.904 INFO:tasks.workunit.client.0.smithi102.stdout:4/681: stat d1/d17/dbc/fc8 0 2024-03-26T18:54:44.914 INFO:tasks.workunit.client.0.smithi102.stdout:4/682: write d1/d17/d1f/d5d/d8b/f66 [1313802,82197] 0 2024-03-26T18:54:44.915 INFO:tasks.workunit.client.0.smithi102.stdout:4/683: symlink d1/d17/d1f/d5d/d8b/le2 0 2024-03-26T18:54:44.916 INFO:tasks.workunit.client.0.smithi102.stdout:4/684: symlink d1/d17/d2a/d9a/ddc/le3 0 2024-03-26T18:54:44.916 INFO:tasks.workunit.client.0.smithi102.stdout:4/685: dread - d1/d17/d1a/fdf zero size 2024-03-26T18:54:44.922 INFO:tasks.workunit.client.0.smithi102.stdout:1/756: dwrite de/d2e/f55 [0,4194304] 0 2024-03-26T18:54:44.922 INFO:tasks.workunit.client.0.smithi102.stdout:1/757: write de/d16/d1d/d61/d6a/fdb [352977,38781] 0 2024-03-26T18:54:44.924 INFO:tasks.workunit.client.0.smithi102.stdout:1/758: rename de/d16/d1d/d2a/d2c/d56/d5f/cc6 to de/d1f/de3/da6/dca/dda/cff 0 2024-03-26T18:54:44.924 INFO:tasks.workunit.client.0.smithi102.stdout:1/759: readlink de/d1f/l25 0 2024-03-26T18:54:44.925 INFO:tasks.workunit.client.0.smithi102.stdout:4/686: write d1/f26 [6042457,58611] 0 2024-03-26T18:54:44.925 INFO:tasks.workunit.client.0.smithi102.stdout:4/687: read - d1/d17/d1f/d5d/d6f/dc0/fce zero size 2024-03-26T18:54:44.925 INFO:tasks.workunit.client.0.smithi102.stdout:4/688: stat d1/d17/d1a/d21/c7e 0 2024-03-26T18:54:44.925 INFO:tasks.workunit.client.0.smithi102.stdout:1/760: creat de/d16/d1d/d2a/d2c/d56/dfa/f100 x:0 0 0 2024-03-26T18:54:44.925 INFO:tasks.workunit.client.0.smithi102.stdout:1/761: stat de/f26 0 2024-03-26T18:54:44.937 INFO:tasks.workunit.client.0.smithi102.stdout:1/762: write de/d16/d1d/d2a/d2c/d43/f86 [2260885,103455] 0 2024-03-26T18:54:44.937 INFO:tasks.workunit.client.0.smithi102.stdout:1/763: dread - de/d16/d1d/d2a/d2c/d56/d78/feb zero size 2024-03-26T18:54:44.937 INFO:tasks.workunit.client.0.smithi102.stdout:1/764: read de/f32 [2232793,88187] 0 2024-03-26T18:54:44.938 INFO:tasks.workunit.client.0.smithi102.stdout:1/765: mknod de/d16/d1d/d2a/d67/c101 0 2024-03-26T18:54:44.939 INFO:tasks.workunit.client.0.smithi102.stdout:1/766: link de/d16/d1d/d2a/d2c/d56/d5f/daa/fd2 de/d16/d1d/d61/f102 0 2024-03-26T18:54:44.940 INFO:tasks.workunit.client.0.smithi102.stdout:1/767: getdents de/d1f 0 2024-03-26T18:54:44.995 INFO:tasks.workunit.client.1.smithi190.stdout:3/986: sync 2024-03-26T18:54:44.995 INFO:tasks.workunit.client.1.smithi190.stdout:3/987: chown d1/d3/df/d18/d35/d41/d12b/dfb/c12d 354739370 1 2024-03-26T18:54:45.010 INFO:tasks.workunit.client.0.smithi102.stdout:1/768: dwrite de/d16/f41 [0,4194304] 0 2024-03-26T18:54:45.012 INFO:tasks.workunit.client.0.smithi102.stdout:1/769: rmdir de/d16/d1d 39 2024-03-26T18:54:45.012 INFO:tasks.workunit.client.0.smithi102.stdout:1/770: read de/d16/d1d/d2a/d2c/d56/d5f/daa/fd5 [9590,96732] 0 2024-03-26T18:54:45.013 INFO:tasks.workunit.client.0.smithi102.stdout:1/771: symlink de/d1f/de3/da6/dca/df7/l103 0 2024-03-26T18:54:45.016 INFO:tasks.workunit.client.0.smithi102.stdout:1/772: dread de/d16/d1d/d2a/f9e [0,4194304] 0 2024-03-26T18:54:45.016 INFO:tasks.workunit.client.0.smithi102.stdout:1/773: stat de/d16/d1d/ccf 0 2024-03-26T18:54:45.016 INFO:tasks.workunit.client.0.smithi102.stdout:1/774: read - de/fea zero size 2024-03-26T18:54:45.022 INFO:tasks.workunit.client.0.smithi102.stdout:1/775: dread de/d2e/fa8 [0,4194304] 0 2024-03-26T18:54:45.025 INFO:tasks.workunit.client.0.smithi102.stdout:1/776: unlink de/fea 0 2024-03-26T18:54:45.025 INFO:tasks.workunit.client.0.smithi102.stdout:1/777: fdatasync de/d16/d1d/d2a/d2c/d56/d6e/f7a 0 2024-03-26T18:54:45.040 INFO:tasks.workunit.client.0.smithi102.stdout:1/778: dwrite de/f14 [0,4194304] 0 2024-03-26T18:54:45.058 INFO:tasks.workunit.client.0.smithi102.stdout:6/778: sync 2024-03-26T18:54:45.058 INFO:tasks.workunit.client.0.smithi102.stdout:9/705: sync 2024-03-26T18:54:45.058 INFO:tasks.workunit.client.0.smithi102.stdout:8/788: sync 2024-03-26T18:54:45.058 INFO:tasks.workunit.client.0.smithi102.stdout:9/706: fdatasync df/d1a/d81/d88/fc9 0 2024-03-26T18:54:45.058 INFO:tasks.workunit.client.0.smithi102.stdout:9/707: dread - df/d1a/d65/dbe/ffc zero size 2024-03-26T18:54:45.058 INFO:tasks.workunit.client.0.smithi102.stdout:9/708: creat df/d46/da8/f105 x:0 0 0 2024-03-26T18:54:45.059 INFO:tasks.workunit.client.0.smithi102.stdout:8/789: creat d0/f10a x:0 0 0 2024-03-26T18:54:45.060 INFO:tasks.workunit.client.0.smithi102.stdout:9/709: rename df/d1a/d65/dbe/ddf/fee to df/d1a/d1d/d72/f106 0 2024-03-26T18:54:45.061 INFO:tasks.workunit.client.0.smithi102.stdout:8/790: mkdir d0/d15/d5d/dd3/d2c/d33/d87/d10b 0 2024-03-26T18:54:45.064 INFO:tasks.workunit.client.0.smithi102.stdout:8/791: getdents d0/d15/d5d/dd3/d2c/d58 0 2024-03-26T18:54:45.065 INFO:tasks.workunit.client.0.smithi102.stdout:8/792: chown d0/d15/d5d/dd3/d2c/d33/d87/fa8 12952 1 2024-03-26T18:54:45.065 INFO:tasks.workunit.client.0.smithi102.stdout:8/793: unlink d0/d15/d1b/c41 0 2024-03-26T18:54:45.065 INFO:tasks.workunit.client.0.smithi102.stdout:8/794: readlink d0/d15/d5d/d89/lae 0 2024-03-26T18:54:45.065 INFO:tasks.workunit.client.0.smithi102.stdout:8/795: write d0/d15/d5d/dd3/d2c/d4e/f52 [3189327,48073] 0 2024-03-26T18:54:45.065 INFO:tasks.workunit.client.0.smithi102.stdout:9/710: rmdir df/d1a/d65/dbe 39 2024-03-26T18:54:45.067 INFO:tasks.workunit.client.0.smithi102.stdout:9/711: rmdir df/ded 0 2024-03-26T18:54:45.067 INFO:tasks.workunit.client.0.smithi102.stdout:1/779: dwrite de/d1f/f6d [0,4194304] 0 2024-03-26T18:54:45.069 INFO:tasks.workunit.client.0.smithi102.stdout:9/712: rename df/d1a/f5e to df/d1a/d1d/f107 0 2024-03-26T18:54:45.069 INFO:tasks.workunit.client.0.smithi102.stdout:9/713: dread - df/d1a/d65/dbe/ffc zero size 2024-03-26T18:54:45.070 INFO:tasks.workunit.client.0.smithi102.stdout:1/780: stat de/d2e/cf9 0 2024-03-26T18:54:45.071 INFO:tasks.workunit.client.0.smithi102.stdout:9/714: symlink df/d1a/d23/d4e/l108 0 2024-03-26T18:54:45.071 INFO:tasks.workunit.client.0.smithi102.stdout:9/715: chown df/d1a/d57/f8b 23 1 2024-03-26T18:54:45.071 INFO:tasks.workunit.client.0.smithi102.stdout:6/779: dread d2/fca [0,4194304] 0 2024-03-26T18:54:45.071 INFO:tasks.workunit.client.0.smithi102.stdout:6/780: truncate d2/d3d/f8a 312227 0 2024-03-26T18:54:45.072 INFO:tasks.workunit.client.0.smithi102.stdout:1/781: rename de/d1f/de3/da6/dca/fcc to de/d16/d1d/d2a/d2c/de6/f104 0 2024-03-26T18:54:45.072 INFO:tasks.workunit.client.0.smithi102.stdout:6/781: mkdir d2/df/d71/d80/d105 0 2024-03-26T18:54:45.086 INFO:tasks.workunit.client.0.smithi102.stdout:1/782: truncate de/d1f/f6d 850646 0 2024-03-26T18:54:45.087 INFO:tasks.workunit.client.0.smithi102.stdout:1/783: mknod de/d16/d1d/d61/d6a/c105 0 2024-03-26T18:54:45.090 INFO:tasks.workunit.client.0.smithi102.stdout:1/784: dread de/d16/d1d/d2a/d2c/d56/d6e/fe1 [0,4194304] 0 2024-03-26T18:54:45.091 INFO:tasks.workunit.client.0.smithi102.stdout:8/796: dwrite d0/d3c/d96/d98/db6/f4f [0,4194304] 0 2024-03-26T18:54:45.091 INFO:tasks.workunit.client.0.smithi102.stdout:8/797: chown d0/d3c/d6f/ffa 155 1 2024-03-26T18:54:45.091 INFO:tasks.workunit.client.0.smithi102.stdout:6/782: dwrite d2/d3d/f70 [0,4194304] 0 2024-03-26T18:54:45.093 INFO:tasks.workunit.client.0.smithi102.stdout:8/798: rename d0/d3c/d6f to d0/d15/da0/d10c 0 2024-03-26T18:54:45.097 INFO:tasks.workunit.client.0.smithi102.stdout:8/799: mknod d0/d15/d5d/d89/c10d 0 2024-03-26T18:54:45.097 INFO:tasks.workunit.client.0.smithi102.stdout:1/785: dread de/d1f/de3/da6/fb7 [0,4194304] 0 2024-03-26T18:54:45.100 INFO:tasks.workunit.client.0.smithi102.stdout:8/800: rename d0/d15/d5d/dd3/d2c/da7/c2a to d0/d15/d5d/dd3/d26/c10e 0 2024-03-26T18:54:45.100 INFO:tasks.workunit.client.0.smithi102.stdout:8/801: write d0/d15/d5d/dd3/d2c/d33/d87/f6c [996484,11372] 0 2024-03-26T18:54:45.100 INFO:tasks.workunit.client.0.smithi102.stdout:8/802: creat d0/d9/f10f x:0 0 0 2024-03-26T18:54:45.100 INFO:tasks.workunit.client.0.smithi102.stdout:8/803: dread - d0/d15/d5d/dd3/d2c/d2f/fd8 zero size 2024-03-26T18:54:45.100 INFO:tasks.workunit.client.0.smithi102.stdout:8/804: chown d0/d9/fb9 0 1 2024-03-26T18:54:45.107 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:44 smithi190 ceph-mon[35043]: pgmap v250: 65 pgs: 65 active+clean; 3.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 284 MiB/s rd, 358 MiB/s wr, 550 op/s 2024-03-26T18:54:45.115 INFO:tasks.workunit.client.0.smithi102.stdout:6/783: dwrite d2/d3d/d53/fcd [0,4194304] 0 2024-03-26T18:54:45.115 INFO:tasks.workunit.client.0.smithi102.stdout:6/784: dread - d2/df/dba/fdd zero size 2024-03-26T18:54:45.115 INFO:tasks.workunit.client.0.smithi102.stdout:1/786: dwrite de/d16/d1d/d2a/d67/f95 [0,4194304] 0 2024-03-26T18:54:45.116 INFO:tasks.workunit.client.0.smithi102.stdout:6/785: symlink d2/df/d35/d60/l106 0 2024-03-26T18:54:45.117 INFO:tasks.workunit.client.0.smithi102.stdout:1/787: creat de/d2e/d62/f106 x:0 0 0 2024-03-26T18:54:45.118 INFO:tasks.workunit.client.0.smithi102.stdout:7/660: sync 2024-03-26T18:54:45.119 INFO:tasks.workunit.client.0.smithi102.stdout:6/786: rmdir d2/df/d72/db7 39 2024-03-26T18:54:45.120 INFO:tasks.workunit.client.0.smithi102.stdout:7/661: unlink d7/d10/d27/f55 0 2024-03-26T18:54:45.122 INFO:tasks.workunit.client.0.smithi102.stdout:6/787: symlink d2/df/d35/d43/l107 0 2024-03-26T18:54:45.123 INFO:tasks.workunit.client.0.smithi102.stdout:7/662: mkdir d7/d10/d27/da3/dc6 0 2024-03-26T18:54:45.124 INFO:tasks.workunit.client.0.smithi102.stdout:6/788: rename d2/d12/f2d to d2/df/d72/f108 0 2024-03-26T18:54:45.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:44 smithi102 ceph-mon[29002]: pgmap v250: 65 pgs: 65 active+clean; 3.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 284 MiB/s rd, 358 MiB/s wr, 550 op/s 2024-03-26T18:54:45.125 INFO:tasks.workunit.client.0.smithi102.stdout:7/663: symlink d7/d44/lc7 0 2024-03-26T18:54:45.126 INFO:tasks.workunit.client.0.smithi102.stdout:7/664: chown d7/d10/d17/f1b 444017 1 2024-03-26T18:54:45.127 INFO:tasks.workunit.client.0.smithi102.stdout:7/665: rename d7/d10/d17/cbf to d7/d2a/d9d/cc8 0 2024-03-26T18:54:45.128 INFO:tasks.workunit.client.0.smithi102.stdout:7/666: creat d7/d12/d4d/d62/d95/fc9 x:0 0 0 2024-03-26T18:54:45.129 INFO:tasks.workunit.client.0.smithi102.stdout:7/667: mkdir d7/d12/d2b/da8/dca 0 2024-03-26T18:54:45.143 INFO:tasks.workunit.client.0.smithi102.stdout:7/668: read d7/d10/d67/fb2 [3981241,117591] 0 2024-03-26T18:54:45.143 INFO:tasks.workunit.client.0.smithi102.stdout:1/788: dwrite de/d2e/d62/f91 [4194304,4194304] 0 2024-03-26T18:54:45.144 INFO:tasks.workunit.client.0.smithi102.stdout:6/789: dwrite d2/d3d/d53/f74 [0,4194304] 0 2024-03-26T18:54:45.147 INFO:tasks.workunit.client.0.smithi102.stdout:1/789: mknod de/d1f/de3/da6/c107 0 2024-03-26T18:54:45.148 INFO:tasks.workunit.client.0.smithi102.stdout:6/790: mkdir d2/d12/dce/d109 0 2024-03-26T18:54:45.149 INFO:tasks.workunit.client.0.smithi102.stdout:1/790: creat de/d16/d1d/d2a/d2c/d43/f108 x:0 0 0 2024-03-26T18:54:45.150 INFO:tasks.workunit.client.1.smithi190.stdout:3/988: sync 2024-03-26T18:54:45.151 INFO:tasks.workunit.client.0.smithi102.stdout:6/791: rename d2/df/d71/d80/d91/dc1/led to d2/d3/l10a 0 2024-03-26T18:54:45.153 INFO:tasks.workunit.client.0.smithi102.stdout:6/792: mkdir d2/df/d35/d60/d10b 0 2024-03-26T18:54:45.155 INFO:tasks.workunit.client.0.smithi102.stdout:6/793: symlink d2/d3d/d53/d63/l10c 0 2024-03-26T18:54:45.156 INFO:tasks.workunit.client.1.smithi190.stdout:3/989: dread d1/de/d8a/f95 [0,4194304] 0 2024-03-26T18:54:45.160 INFO:tasks.workunit.client.0.smithi102.stdout:6/794: rename d2/d3d/d53/db1/fb3 to d2/df/d71/f10d 0 2024-03-26T18:54:45.162 INFO:tasks.workunit.client.1.smithi190.stdout:3/990: symlink d1/d3/d52/l131 0 2024-03-26T18:54:45.163 INFO:tasks.workunit.client.1.smithi190.stdout:3/991: creat d1/d3/df/d18/db4/f132 x:0 0 0 2024-03-26T18:54:45.165 INFO:tasks.workunit.client.0.smithi102.stdout:6/795: read d2/df/d72/db7/f6f [151719,109705] 0 2024-03-26T18:54:45.166 INFO:tasks.workunit.client.0.smithi102.stdout:1/791: dwrite de/d16/d1d/d2a/d2c/d56/d5f/fe9 [0,4194304] 0 2024-03-26T18:54:45.166 INFO:tasks.workunit.client.0.smithi102.stdout:1/792: rename de to de/d16/d1d/d61/d6a/d109 22 2024-03-26T18:54:45.166 INFO:tasks.workunit.client.0.smithi102.stdout:1/793: chown de/d1f/f3a 3 1 2024-03-26T18:54:45.166 INFO:tasks.workunit.client.0.smithi102.stdout:6/796: symlink d2/df/d71/d80/d91/dc1/l10e 0 2024-03-26T18:54:45.166 INFO:tasks.workunit.client.0.smithi102.stdout:6/797: write d2/df/d35/d43/d75/f8c [4766273,48939] 0 2024-03-26T18:54:45.167 INFO:tasks.workunit.client.0.smithi102.stdout:7/669: dwrite d7/d10/d17/f5c [0,4194304] 0 2024-03-26T18:54:45.168 INFO:tasks.workunit.client.0.smithi102.stdout:6/798: link d2/df/d35/d43/d75/f8c d2/d12/f10f 0 2024-03-26T18:54:45.168 INFO:tasks.workunit.client.0.smithi102.stdout:6/799: stat d2/d7/f67 0 2024-03-26T18:54:45.168 INFO:tasks.workunit.client.1.smithi190.stdout:3/992: dread d1/d3/df/f102 [0,4194304] 0 2024-03-26T18:54:45.168 INFO:tasks.workunit.client.1.smithi190.stdout:3/993: chown d1/d47/deb/ff6 33342060 1 2024-03-26T18:54:45.170 INFO:tasks.workunit.client.0.smithi102.stdout:7/670: unlink d7/d10/d27/d34/f49 0 2024-03-26T18:54:45.170 INFO:tasks.workunit.client.0.smithi102.stdout:7/671: fdatasync d7/d10/d67/f80 0 2024-03-26T18:54:45.171 INFO:tasks.workunit.client.1.smithi190.stdout:3/994: getdents d1/d3/df/d18/d35/d41/d12b 0 2024-03-26T18:54:45.173 INFO:tasks.workunit.client.0.smithi102.stdout:7/672: creat d7/d12/d2b/d50/fcb x:0 0 0 2024-03-26T18:54:45.173 INFO:tasks.workunit.client.0.smithi102.stdout:7/673: write d7/d12/d2b/d50/fa7 [205481,105471] 0 2024-03-26T18:54:45.174 INFO:tasks.workunit.client.1.smithi190.stdout:3/995: creat d1/dd3/df7/f133 x:0 0 0 2024-03-26T18:54:45.182 INFO:tasks.workunit.client.1.smithi190.stdout:3/996: dread d1/de/d81/dad/fb0 [0,4194304] 0 2024-03-26T18:54:45.183 INFO:tasks.workunit.client.1.smithi190.stdout:3/997: mkdir d1/d3/df/d31/dd1/d134 0 2024-03-26T18:54:45.183 INFO:tasks.workunit.client.1.smithi190.stdout:3/998: stat d1/d3/df/d18/d35/d41/dae/d9c 0 2024-03-26T18:54:45.189 INFO:tasks.workunit.client.0.smithi102.stdout:6/800: dwrite d2/df/d35/d43/d75/fb5 [0,4194304] 0 2024-03-26T18:54:45.189 INFO:tasks.workunit.client.0.smithi102.stdout:7/674: dwrite d7/d10/d67/fa5 [0,4194304] 0 2024-03-26T18:54:45.189 INFO:tasks.workunit.client.0.smithi102.stdout:7/675: chown d7/d10/d17/f81 229161299 1 2024-03-26T18:54:45.190 INFO:tasks.workunit.client.0.smithi102.stdout:7/676: fsync d7/d12/d2b/f2f 0 2024-03-26T18:54:45.192 INFO:tasks.workunit.client.0.smithi102.stdout:6/801: mknod d2/d12/dce/c110 0 2024-03-26T18:54:45.192 INFO:tasks.workunit.client.0.smithi102.stdout:6/802: dread - d2/df/d72/db7/fda zero size 2024-03-26T18:54:45.193 INFO:tasks.workunit.client.0.smithi102.stdout:7/677: symlink d7/lcc 0 2024-03-26T18:54:45.193 INFO:tasks.workunit.client.0.smithi102.stdout:6/803: chown d2/d7/l1b 776927375 1 2024-03-26T18:54:45.195 INFO:tasks.workunit.client.0.smithi102.stdout:7/678: mkdir d7/d10/d27/d34/dcd 0 2024-03-26T18:54:45.195 INFO:tasks.workunit.client.0.smithi102.stdout:0/812: sync 2024-03-26T18:54:45.195 INFO:tasks.workunit.client.0.smithi102.stdout:2/781: sync 2024-03-26T18:54:45.195 INFO:tasks.workunit.client.0.smithi102.stdout:4/689: sync 2024-03-26T18:54:45.197 INFO:tasks.workunit.client.0.smithi102.stdout:6/804: dread d2/d3/f97 [0,4194304] 0 2024-03-26T18:54:45.197 INFO:tasks.workunit.client.0.smithi102.stdout:2/782: mknod d3/d11/d10c/c118 0 2024-03-26T18:54:45.198 INFO:tasks.workunit.client.0.smithi102.stdout:4/690: symlink d1/d17/d2a/le4 0 2024-03-26T18:54:45.198 INFO:tasks.workunit.client.0.smithi102.stdout:4/691: write d1/d17/d1a/d21/f56 [5398672,71250] 0 2024-03-26T18:54:45.198 INFO:tasks.workunit.client.0.smithi102.stdout:4/692: chown d1/fa0 17637979 1 2024-03-26T18:54:45.199 INFO:tasks.workunit.client.0.smithi102.stdout:2/783: unlink d3/d82/l8c 0 2024-03-26T18:54:45.201 INFO:tasks.workunit.client.0.smithi102.stdout:2/784: creat d3/d11/d15/d46/d7b/f119 x:0 0 0 2024-03-26T18:54:45.201 INFO:tasks.workunit.client.0.smithi102.stdout:2/785: dread - d3/d11/d15/d46/f8b zero size 2024-03-26T18:54:45.202 INFO:tasks.workunit.client.0.smithi102.stdout:2/786: truncate d3/d82/d93/d9e/d1a/f7d 697003 0 2024-03-26T18:54:45.202 INFO:tasks.workunit.client.0.smithi102.stdout:6/805: write d2/d7/f67 [460610,66443] 0 2024-03-26T18:54:45.202 INFO:tasks.workunit.client.0.smithi102.stdout:4/693: mkdir d1/d17/d1f/d5d/d8b/de5 0 2024-03-26T18:54:45.204 INFO:tasks.workunit.client.0.smithi102.stdout:6/806: rename d2/d3d/d53/f76 to d2/df/dba/f111 0 2024-03-26T18:54:45.205 INFO:tasks.workunit.client.0.smithi102.stdout:4/694: mknod d1/d17/d1f/d5d/d8b/d3e/d74/ce6 0 2024-03-26T18:54:45.206 INFO:tasks.workunit.client.0.smithi102.stdout:6/807: truncate d2/df/d72/db7/f95 1594275 0 2024-03-26T18:54:45.206 INFO:tasks.workunit.client.0.smithi102.stdout:6/808: write d2/df/d71/fbb [907069,42503] 0 2024-03-26T18:54:45.206 INFO:tasks.workunit.client.0.smithi102.stdout:6/809: chown d2/df/d71/d80/d91/dc1/l10e 6 1 2024-03-26T18:54:45.206 INFO:tasks.workunit.client.0.smithi102.stdout:6/810: readlink d2/df/d35/ld8 0 2024-03-26T18:54:45.206 INFO:tasks.workunit.client.0.smithi102.stdout:6/811: chown d2/df/d71/d80/d91/lee 177225 1 2024-03-26T18:54:45.206 INFO:tasks.workunit.client.0.smithi102.stdout:6/812: chown d2/df/d72/db7/f6f 20 1 2024-03-26T18:54:45.206 INFO:tasks.workunit.client.0.smithi102.stdout:6/813: read d2/f33 [3346111,17689] 0 2024-03-26T18:54:45.207 INFO:tasks.workunit.client.1.smithi190.stdout:3/999: dwrite d1/d3/df/f19 [0,4194304] 0 2024-03-26T18:54:45.213 INFO:tasks.workunit.client.1.smithi190.stderr:+ rm -rf -- ./tmp.EA7GdPmEKz 2024-03-26T18:54:45.213 INFO:tasks.workunit.client.0.smithi102.stdout:0/813: write d6/de/d13/d1f/d71/f9b [1744268,74008] 0 2024-03-26T18:54:45.213 INFO:tasks.workunit.client.0.smithi102.stdout:0/814: write d6/de/d13/d25/d31/d45/d82/d84/d88/fa6 [1246273,37621] 0 2024-03-26T18:54:45.215 INFO:tasks.workunit.client.0.smithi102.stdout:0/815: creat d6/de/d13/d25/da4/d10b/f118 x:0 0 0 2024-03-26T18:54:45.215 INFO:tasks.workunit.client.0.smithi102.stdout:0/816: fdatasync d6/de/d13/d1f/fa8 0 2024-03-26T18:54:45.216 INFO:tasks.workunit.client.0.smithi102.stdout:6/814: dread d2/df/d35/f6e [0,4194304] 0 2024-03-26T18:54:45.216 INFO:tasks.workunit.client.0.smithi102.stdout:7/679: dwrite d7/d12/d51/fbc [0,4194304] 0 2024-03-26T18:54:45.216 INFO:tasks.workunit.client.0.smithi102.stdout:6/815: write d2/df/d35/d43/f103 [321410,24202] 0 2024-03-26T18:54:45.217 INFO:tasks.workunit.client.0.smithi102.stdout:0/817: mkdir d6/d3e/d87/d119 0 2024-03-26T18:54:45.222 INFO:tasks.workunit.client.0.smithi102.stdout:7/680: fdatasync d7/d12/d4d/d62/d95/fc9 0 2024-03-26T18:54:45.224 INFO:tasks.workunit.client.0.smithi102.stdout:6/816: creat d2/d12/dce/d109/f112 x:0 0 0 2024-03-26T18:54:45.228 INFO:tasks.workunit.client.0.smithi102.stdout:7/681: mknod d7/d12/d51/cce 0 2024-03-26T18:54:45.229 INFO:tasks.workunit.client.0.smithi102.stdout:6/817: link d2/d3/c5f d2/da4/c113 0 2024-03-26T18:54:45.230 INFO:tasks.workunit.client.0.smithi102.stdout:7/682: truncate d7/d12/d4d/d62/fc5 1017009 0 2024-03-26T18:54:45.230 INFO:tasks.workunit.client.0.smithi102.stdout:7/683: readlink d7/d10/d27/d34/l75 0 2024-03-26T18:54:45.230 INFO:tasks.workunit.client.0.smithi102.stdout:7/684: chown d7/d10/d27/d34/d82/fb7 15921970 1 2024-03-26T18:54:45.230 INFO:tasks.workunit.client.0.smithi102.stdout:7/685: creat d7/d12/d51/d6a/fcf x:0 0 0 2024-03-26T18:54:45.230 INFO:tasks.workunit.client.0.smithi102.stdout:7/686: readlink d7/d10/d67/d69/d78/lb8 0 2024-03-26T18:54:45.230 INFO:tasks.workunit.client.0.smithi102.stdout:2/787: dwrite d3/fe5 [0,4194304] 0 2024-03-26T18:54:45.231 INFO:tasks.workunit.client.0.smithi102.stdout:4/695: dwrite d1/d17/d1a/d21/d77/f71 [0,4194304] 0 2024-03-26T18:54:45.232 INFO:tasks.workunit.client.0.smithi102.stdout:0/818: rename d6/de/d13/d9a/d85 to d6/d3e/d104/d11a 0 2024-03-26T18:54:45.232 INFO:tasks.workunit.client.0.smithi102.stdout:0/819: chown d6/d2d/d3d/fa2 384254 1 2024-03-26T18:54:45.234 INFO:tasks.workunit.client.0.smithi102.stdout:7/687: dread d7/d12/f38 [0,4194304] 0 2024-03-26T18:54:45.238 INFO:tasks.workunit.client.0.smithi102.stdout:2/788: symlink d3/d20/d78/d7c/dca/l11a 0 2024-03-26T18:54:45.241 INFO:tasks.workunit.client.0.smithi102.stdout:7/688: mkdir d7/d12/d51/dd0 0 2024-03-26T18:54:45.244 INFO:tasks.workunit.client.0.smithi102.stdout:0/820: rename d6/d2d/d3d/lb1 to d6/de/d13/d1f/d55/d57/l11b 0 2024-03-26T18:54:45.244 INFO:tasks.workunit.client.0.smithi102.stdout:0/821: fdatasync d6/de/d13/d25/d31/d45/d82/d84/d88/fab 0 2024-03-26T18:54:45.244 INFO:tasks.workunit.client.0.smithi102.stdout:2/789: mknod d3/d82/d93/d9e/d99/c11b 0 2024-03-26T18:54:45.245 INFO:tasks.workunit.client.0.smithi102.stdout:7/689: creat d7/d10/d27/fd1 x:0 0 0 2024-03-26T18:54:45.245 INFO:tasks.workunit.client.0.smithi102.stdout:7/690: fsync d7/d12/f38 0 2024-03-26T18:54:45.246 INFO:tasks.workunit.client.0.smithi102.stdout:0/822: unlink d6/de/d13/d25/d31/f8e 0 2024-03-26T18:54:45.247 INFO:tasks.workunit.client.0.smithi102.stdout:2/790: link d3/d11/d15/d46/fea d3/d11/d15/f11c 0 2024-03-26T18:54:45.248 INFO:tasks.workunit.client.0.smithi102.stdout:0/823: link d6/d3e/d104/d11a/cf4 d6/de/c11c 0 2024-03-26T18:54:45.249 INFO:tasks.workunit.client.0.smithi102.stdout:2/791: link d3/d2a/f4f d3/d82/f11d 0 2024-03-26T18:54:45.250 INFO:tasks.workunit.client.0.smithi102.stdout:4/696: dwrite d1/d17/d1f/d5d/d6f/faf [0,4194304] 0 2024-03-26T18:54:45.251 INFO:tasks.workunit.client.0.smithi102.stdout:0/824: unlink d6/f91 0 2024-03-26T18:54:45.254 INFO:tasks.workunit.client.0.smithi102.stdout:4/697: mkdir d1/d6b/de7 0 2024-03-26T18:54:45.254 INFO:tasks.workunit.client.0.smithi102.stdout:4/698: write d1/fd5 [565704,23299] 0 2024-03-26T18:54:45.256 INFO:tasks.workunit.client.0.smithi102.stdout:0/825: link d6/de/d13/d25/c41 d6/d3e/d69/d8c/d8f/c11d 0 2024-03-26T18:54:45.256 INFO:tasks.workunit.client.0.smithi102.stdout:0/826: chown d6/f11 256 1 2024-03-26T18:54:45.257 INFO:tasks.workunit.client.0.smithi102.stdout:7/691: write d7/d12/d2b/f47 [2772550,1908] 0 2024-03-26T18:54:45.257 INFO:tasks.workunit.client.0.smithi102.stdout:4/699: mkdir d1/d17/d1a/d21/de8 0 2024-03-26T18:54:45.259 INFO:tasks.workunit.client.0.smithi102.stdout:0/827: truncate d6/d2d/d3d/f40 218177 0 2024-03-26T18:54:45.259 INFO:tasks.workunit.client.0.smithi102.stdout:0/828: truncate d6/de/d13/d9a/f68 5237566 0 2024-03-26T18:54:45.261 INFO:tasks.workunit.client.0.smithi102.stdout:7/692: rename d7/d10/d17/l1e to d7/d4e/ld2 0 2024-03-26T18:54:45.262 INFO:tasks.workunit.client.0.smithi102.stdout:4/700: mknod d1/d17/d2a/d9a/dc2/ce9 0 2024-03-26T18:54:45.282 INFO:tasks.workunit.client.0.smithi102.stdout:2/792: dwrite d3/d2a/f4f [0,4194304] 0 2024-03-26T18:54:45.282 INFO:tasks.workunit.client.0.smithi102.stdout:7/693: dread d7/d12/d51/fbc [0,4194304] 0 2024-03-26T18:54:45.282 INFO:tasks.workunit.client.0.smithi102.stdout:0/829: dwrite d6/d3e/d69/d8c/f10f [0,4194304] 0 2024-03-26T18:54:45.290 INFO:tasks.workunit.client.0.smithi102.stdout:7/694: dread d7/ff [0,4194304] 0 2024-03-26T18:54:45.290 INFO:tasks.workunit.client.0.smithi102.stdout:7/695: truncate d7/d12/d4d/f8b 727935 0 2024-03-26T18:54:45.290 INFO:tasks.workunit.client.0.smithi102.stdout:7/696: read d7/d12/d51/d6a/fb6 [474382,59211] 0 2024-03-26T18:54:45.290 INFO:tasks.workunit.client.0.smithi102.stdout:7/697: chown d7/f41 711 1 2024-03-26T18:54:45.292 INFO:tasks.workunit.client.0.smithi102.stdout:0/830: dread d6/d3e/d104/d11a/ff7 [0,4194304] 0 2024-03-26T18:54:45.316 INFO:tasks.workunit.client.0.smithi102.stdout:4/701: truncate d1/fd5 319001 0 2024-03-26T18:54:45.317 INFO:tasks.workunit.client.0.smithi102.stdout:4/702: chown d1/d17/d1a/d21/d77/f81 2562 1 2024-03-26T18:54:45.326 INFO:tasks.workunit.client.0.smithi102.stdout:2/793: fdatasync d3/d20/fd4 0 2024-03-26T18:54:45.326 INFO:tasks.workunit.client.0.smithi102.stdout:2/794: chown d3/d2d/d69 267 1 2024-03-26T18:54:45.367 INFO:tasks.workunit.client.0.smithi102.stdout:7/698: dread d7/fb [0,4194304] 0 2024-03-26T18:54:45.374 INFO:tasks.workunit.client.0.smithi102.stdout:7/699: dread d7/f41 [0,4194304] 0 2024-03-26T18:54:45.379 INFO:tasks.workunit.client.0.smithi102.stdout:0/831: link d6/de/d13/d25/d31/d45/d82/d84/d88/db2/dd2/f111 d6/de/d93/f11e 0 2024-03-26T18:54:45.380 INFO:tasks.workunit.client.0.smithi102.stdout:7/700: dread d7/d10/d17/f5c [0,4194304] 0 2024-03-26T18:54:45.437 INFO:tasks.workunit.client.0.smithi102.stdout:4/703: mkdir d1/d27/d98/dea 0 2024-03-26T18:54:45.438 INFO:tasks.workunit.client.0.smithi102.stdout:4/704: stat d1/d17/d1f/d5d/d8b/d3e/d74/ce6 0 2024-03-26T18:54:45.438 INFO:tasks.workunit.client.0.smithi102.stdout:4/705: fdatasync d1/d17/d1f/f2d 0 2024-03-26T18:54:45.438 INFO:tasks.workunit.client.0.smithi102.stdout:2/795: mkdir d3/d2a/d5f/d11e 0 2024-03-26T18:54:45.440 INFO:tasks.workunit.client.0.smithi102.stdout:4/706: dread d1/d27/f6e [0,4194304] 0 2024-03-26T18:54:45.441 INFO:tasks.workunit.client.0.smithi102.stdout:9/716: sync 2024-03-26T18:54:45.442 INFO:tasks.workunit.client.0.smithi102.stdout:0/832: link d6/d3e/lc3 d6/dbe/deb/dfd/l11f 0 2024-03-26T18:54:45.442 INFO:tasks.workunit.client.0.smithi102.stdout:0/833: write d6/de/d13/d25/d31/d45/d82/d84/d88/fa6 [1873143,112436] 0 2024-03-26T18:54:45.442 INFO:tasks.workunit.client.0.smithi102.stdout:0/834: dread - d6/de/fd0 zero size 2024-03-26T18:54:45.442 INFO:tasks.workunit.client.0.smithi102.stdout:0/835: creat d6/de/d13/d25/da4/d10b/f120 x:0 0 0 2024-03-26T18:54:45.443 INFO:tasks.workunit.client.0.smithi102.stdout:0/836: dread - d6/d3e/d104/d11a/f107 zero size 2024-03-26T18:54:45.445 INFO:tasks.workunit.client.0.smithi102.stdout:2/796: creat d3/d20/d78/d84/f11f x:0 0 0 2024-03-26T18:54:45.446 INFO:tasks.workunit.client.0.smithi102.stdout:4/707: rmdir d1/d17/d2a/d9a 39 2024-03-26T18:54:45.446 INFO:tasks.workunit.client.0.smithi102.stdout:4/708: write d1/d25/d90/fab [1813385,37856] 0 2024-03-26T18:54:45.451 INFO:tasks.workunit.client.0.smithi102.stdout:9/717: creat df/d1a/d23/d4e/d8e/f109 x:0 0 0 2024-03-26T18:54:45.452 INFO:tasks.workunit.client.0.smithi102.stdout:0/837: write d6/d2d/f38 [341040,3219] 0 2024-03-26T18:54:45.455 INFO:tasks.workunit.client.0.smithi102.stdout:2/797: write d3/d20/fe8 [293342,27462] 0 2024-03-26T18:54:45.458 INFO:tasks.workunit.client.0.smithi102.stdout:4/709: creat d1/d75/feb x:0 0 0 2024-03-26T18:54:45.458 INFO:tasks.workunit.client.0.smithi102.stdout:4/710: write d1/d75/d85/fbe [1354821,125816] 0 2024-03-26T18:54:45.458 INFO:tasks.workunit.client.0.smithi102.stdout:4/711: fdatasync d1/d17/d1a/d21/f95 0 2024-03-26T18:54:45.459 INFO:tasks.workunit.client.0.smithi102.stdout:9/718: symlink df/d1a/d23/d29/l10a 0 2024-03-26T18:54:45.468 INFO:tasks.workunit.client.0.smithi102.stdout:2/798: symlink d3/d20/d78/d84/d6e/dc6/l120 0 2024-03-26T18:54:45.473 INFO:tasks.workunit.client.0.smithi102.stdout:4/712: rmdir d1/d17/d1f/d5d/d6f/dc0 39 2024-03-26T18:54:45.473 INFO:tasks.workunit.client.0.smithi102.stdout:4/713: chown d1/d17/d1a/d21/d77/c73 5 1 2024-03-26T18:54:45.475 INFO:tasks.workunit.client.0.smithi102.stdout:4/714: mknod d1/d6b/cec 0 2024-03-26T18:54:45.476 INFO:tasks.workunit.client.0.smithi102.stdout:9/719: dwrite df/d1a/fd0 [0,4194304] 0 2024-03-26T18:54:45.476 INFO:tasks.workunit.client.0.smithi102.stdout:9/720: read - df/d1a/d23/d60/dd4/fd9 zero size 2024-03-26T18:54:45.483 INFO:tasks.workunit.client.0.smithi102.stdout:9/721: rename df/d46/df7/ffd to df/d1a/f10b 0 2024-03-26T18:54:45.483 INFO:tasks.workunit.client.0.smithi102.stdout:9/722: fdatasync df/d1a/d23/d29/f31 0 2024-03-26T18:54:45.487 INFO:tasks.workunit.client.0.smithi102.stdout:4/715: read d1/d17/d2a/f61 [2084363,75763] 0 2024-03-26T18:54:45.505 INFO:tasks.workunit.client.0.smithi102.stdout:7/701: mkdir d7/d10/d27/d34/dd3 0 2024-03-26T18:54:45.506 INFO:tasks.workunit.client.0.smithi102.stdout:7/702: stat d7/d2a/f74 0 2024-03-26T18:54:45.506 INFO:tasks.workunit.client.0.smithi102.stdout:7/703: chown d7/d10/c39 148024988 1 2024-03-26T18:54:45.507 INFO:tasks.workunit.client.0.smithi102.stdout:7/704: write d7/d44/f53 [438274,72418] 0 2024-03-26T18:54:45.510 INFO:tasks.workunit.client.0.smithi102.stdout:4/716: dwrite d1/d17/dbc/fcc [0,4194304] 0 2024-03-26T18:54:45.515 INFO:tasks.workunit.client.0.smithi102.stdout:1/794: sync 2024-03-26T18:54:45.515 INFO:tasks.workunit.client.0.smithi102.stdout:8/805: sync 2024-03-26T18:54:45.515 INFO:tasks.workunit.client.0.smithi102.stdout:1/795: truncate de/d16/d1d/d61/fee 5146570 0 2024-03-26T18:54:45.515 INFO:tasks.workunit.client.0.smithi102.stdout:8/806: write d0/d15/d5d/dd3/d2c/d2f/f60 [787807,50039] 0 2024-03-26T18:54:45.515 INFO:tasks.workunit.client.0.smithi102.stdout:1/796: write de/d16/d1d/d2a/f5d [1045808,120201] 0 2024-03-26T18:54:45.516 INFO:tasks.workunit.client.0.smithi102.stdout:4/717: rename d1/d17/d1f/d5d/d8b/d3e/c65 to d1/d17/d1f/d5d/d8b/de5/ced 0 2024-03-26T18:54:45.517 INFO:tasks.workunit.client.0.smithi102.stdout:6/818: sync 2024-03-26T18:54:45.517 INFO:tasks.workunit.client.0.smithi102.stdout:8/807: symlink d0/d15/d5d/dd3/d26/d19/d76/l110 0 2024-03-26T18:54:45.521 INFO:tasks.workunit.client.0.smithi102.stdout:1/797: rename de/d16/d1d/d2a/d2c/d43/f108 to de/d16/d1d/d2a/d2c/d56/dfa/f10a 0 2024-03-26T18:54:45.522 INFO:tasks.workunit.client.0.smithi102.stdout:4/718: mkdir d1/d17/dbb/dee 0 2024-03-26T18:54:45.522 INFO:tasks.workunit.client.0.smithi102.stdout:4/719: readlink d1/l14 0 2024-03-26T18:54:45.524 INFO:tasks.workunit.client.0.smithi102.stdout:6/819: symlink d2/df/d72/db7/l114 0 2024-03-26T18:54:45.524 INFO:tasks.workunit.client.0.smithi102.stdout:6/820: getdents d2/df/d71/d80/d105 0 2024-03-26T18:54:45.527 INFO:tasks.workunit.client.0.smithi102.stdout:1/798: rename de/d16/f29 to de/d16/d1d/d2a/d2c/d43/f10b 0 2024-03-26T18:54:45.529 INFO:tasks.workunit.client.0.smithi102.stdout:4/720: creat d1/d17/dbb/dee/fef x:0 0 0 2024-03-26T18:54:45.530 INFO:tasks.workunit.client.0.smithi102.stdout:8/808: getdents d0/d9 0 2024-03-26T18:54:45.530 INFO:tasks.workunit.client.0.smithi102.stdout:8/809: chown d0/d15/d5d/dd3/d2c/d33/fcd 12 1 2024-03-26T18:54:45.530 INFO:tasks.workunit.client.0.smithi102.stdout:8/810: fsync d0/d3c/d96/d98/db6/f94 0 2024-03-26T18:54:45.530 INFO:tasks.workunit.client.0.smithi102.stdout:8/811: chown d0/d15/da0 182493 1 2024-03-26T18:54:45.531 INFO:tasks.workunit.client.0.smithi102.stdout:8/812: truncate d0/f4d 2185948 0 2024-03-26T18:54:45.534 INFO:tasks.workunit.client.0.smithi102.stdout:1/799: link de/d16/la9 de/d16/d1d/d2a/d2c/d56/d5f/l10c 0 2024-03-26T18:54:45.536 INFO:tasks.workunit.client.0.smithi102.stdout:4/721: rename d1/d27/f99 to d1/d27/ff0 0 2024-03-26T18:54:45.541 INFO:tasks.workunit.client.0.smithi102.stdout:1/800: link f8 de/d16/d1d/d2a/d2c/d56/d78/d80/d8e/f10d 0 2024-03-26T18:54:45.541 INFO:tasks.workunit.client.0.smithi102.stdout:1/801: write de/d1f/de3/dd6/fef [1062559,20537] 0 2024-03-26T18:54:45.542 INFO:tasks.workunit.client.0.smithi102.stdout:1/802: fsync de/d16/d1d/d2a/d2c/d56/d6e/f7a 0 2024-03-26T18:54:45.551 INFO:tasks.workunit.client.0.smithi102.stdout:4/722: dread d1/f8 [0,4194304] 0 2024-03-26T18:54:45.555 INFO:tasks.workunit.client.0.smithi102.stdout:1/803: write de/d16/d1d/d61/fee [133729,40846] 0 2024-03-26T18:54:45.555 INFO:tasks.workunit.client.0.smithi102.stdout:1/804: truncate de/d16/d1d/d2a/f5d 2096544 0 2024-03-26T18:54:45.561 INFO:tasks.workunit.client.0.smithi102.stdout:1/805: rename de/f83 to de/d16/d1d/d2a/d67/f10e 0 2024-03-26T18:54:45.572 INFO:tasks.workunit.client.0.smithi102.stdout:2/799: dwrite d3/d82/d93/d9e/d1a/f7d [0,4194304] 0 2024-03-26T18:54:45.577 INFO:tasks.workunit.client.0.smithi102.stdout:4/723: dwrite d1/d75/d85/fbe [0,4194304] 0 2024-03-26T18:54:45.581 INFO:tasks.workunit.client.0.smithi102.stdout:4/724: mkdir d1/d75/df1 0 2024-03-26T18:54:45.581 INFO:tasks.workunit.client.0.smithi102.stdout:4/725: stat d1/d17/d1a/c36 0 2024-03-26T18:54:45.584 INFO:tasks.workunit.client.0.smithi102.stdout:4/726: mknod d1/d17/d1f/d5d/d6f/cf2 0 2024-03-26T18:54:45.584 INFO:tasks.workunit.client.0.smithi102.stdout:4/727: fsync d1/d27/d7a/fd6 0 2024-03-26T18:54:45.584 INFO:tasks.workunit.client.0.smithi102.stdout:4/728: stat d1/d6b 0 2024-03-26T18:54:45.586 INFO:tasks.workunit.client.0.smithi102.stdout:4/729: mknod d1/d17/d1a/cf3 0 2024-03-26T18:54:45.596 INFO:tasks.workunit.client.0.smithi102.stdout:1/806: dwrite de/d16/d1d/d2a/d2c/d43/f86 [0,4194304] 0 2024-03-26T18:54:45.597 INFO:tasks.workunit.client.0.smithi102.stdout:8/813: dwrite d0/d15/f67 [0,4194304] 0 2024-03-26T18:54:45.597 INFO:tasks.workunit.client.0.smithi102.stdout:2/800: dwrite d3/d82/d93/d9e/d1a/fc1 [0,4194304] 0 2024-03-26T18:54:45.597 INFO:tasks.workunit.client.0.smithi102.stdout:2/801: chown d3/d11/d15/f2e 173065 1 2024-03-26T18:54:45.599 INFO:tasks.workunit.client.0.smithi102.stdout:1/807: unlink de/d16/d1d/d2a/d2c/d43/ca1 0 2024-03-26T18:54:45.603 INFO:tasks.workunit.client.0.smithi102.stdout:8/814: creat d0/d15/d9a/f111 x:0 0 0 2024-03-26T18:54:45.603 INFO:tasks.workunit.client.0.smithi102.stdout:8/815: chown d0/d3c/d96/d98/dcf/fde 17 1 2024-03-26T18:54:45.605 INFO:tasks.workunit.client.0.smithi102.stdout:4/730: dwrite d1/d17/d1a/d21/d77/f81 [0,4194304] 0 2024-03-26T18:54:45.609 INFO:tasks.workunit.client.0.smithi102.stdout:4/731: getdents d1/d17/d1f/d5d 0 2024-03-26T18:54:45.611 INFO:tasks.workunit.client.0.smithi102.stdout:2/802: dread d3/d20/d78/f7a [0,4194304] 0 2024-03-26T18:54:45.612 INFO:tasks.workunit.client.0.smithi102.stdout:2/803: chown d3/d11/d15/d31/d39/cbc 1877088 1 2024-03-26T18:54:45.612 INFO:tasks.workunit.client.0.smithi102.stdout:8/816: read d0/d9/fb5 [689700,84869] 0 2024-03-26T18:54:45.612 INFO:tasks.workunit.client.0.smithi102.stdout:8/817: chown d0/d15/d5d/dd3/d2c/d2f/ce8 40970 1 2024-03-26T18:54:45.612 INFO:tasks.workunit.client.0.smithi102.stdout:4/732: rmdir d1/d75/df1 0 2024-03-26T18:54:45.615 INFO:tasks.workunit.client.0.smithi102.stdout:2/804: unlink d3/d20/d78/d7c/dca/l11a 0 2024-03-26T18:54:45.617 INFO:tasks.workunit.client.0.smithi102.stdout:8/818: rename d0/d3c/d96/d98/db6/f40 to d0/d15/d5d/dd3/d26/d80/f112 0 2024-03-26T18:54:45.618 INFO:tasks.workunit.client.0.smithi102.stdout:8/819: fdatasync d0/d15/d5d/dd3/d2c/d33/d87/f9e 0 2024-03-26T18:54:45.618 INFO:tasks.workunit.client.0.smithi102.stdout:8/820: chown d0/d15/da0/d10c/fb4 16 1 2024-03-26T18:54:45.618 INFO:tasks.workunit.client.0.smithi102.stdout:8/821: write d0/d15/d5d/f61 [747820,10387] 0 2024-03-26T18:54:45.618 INFO:tasks.workunit.client.0.smithi102.stdout:4/733: symlink d1/d27/d98/db7/lf4 0 2024-03-26T18:54:45.620 INFO:tasks.workunit.client.0.smithi102.stdout:2/805: creat d3/d82/d93/d9e/f121 x:0 0 0 2024-03-26T18:54:45.620 INFO:tasks.workunit.client.0.smithi102.stdout:8/822: mkdir d0/d15/d5d/dd3/d26/d19/d76/d113 0 2024-03-26T18:54:45.621 INFO:tasks.workunit.client.0.smithi102.stdout:4/734: stat d1/c13 0 2024-03-26T18:54:45.624 INFO:tasks.workunit.client.0.smithi102.stdout:2/806: unlink d3/d2a/f37 0 2024-03-26T18:54:45.627 INFO:tasks.workunit.client.0.smithi102.stdout:2/807: mkdir d3/d82/d122 0 2024-03-26T18:54:45.628 INFO:tasks.workunit.client.0.smithi102.stdout:2/808: write d3/d82/d93/fa4 [1040277,92745] 0 2024-03-26T18:54:45.630 INFO:tasks.workunit.client.0.smithi102.stdout:2/809: mkdir d3/d11/d15/d46/d7b/d97/d10a/d123 0 2024-03-26T18:54:45.630 INFO:tasks.workunit.client.0.smithi102.stdout:2/810: chown d3/d82/d93/fcc 67536 1 2024-03-26T18:54:45.630 INFO:tasks.workunit.client.0.smithi102.stdout:8/823: dread d0/d3c/d96/d98/db6/f3a [0,4194304] 0 2024-03-26T18:54:45.634 INFO:tasks.workunit.client.0.smithi102.stdout:8/824: mknod d0/d15/d5d/dd3/d26/c114 0 2024-03-26T18:54:45.637 INFO:tasks.workunit.client.0.smithi102.stdout:8/825: creat d0/d15/f115 x:0 0 0 2024-03-26T18:54:45.638 INFO:tasks.workunit.client.0.smithi102.stdout:8/826: fdatasync d0/d15/d5d/dd3/d2c/d4e/f52 0 2024-03-26T18:54:45.638 INFO:tasks.workunit.client.0.smithi102.stdout:8/827: creat d0/d15/d5d/dd3/d2c/d4e/f116 x:0 0 0 2024-03-26T18:54:45.638 INFO:tasks.workunit.client.0.smithi102.stdout:8/828: chown d0/d15/d5d/dd3/d2c/d33/fcd 17 1 2024-03-26T18:54:45.640 INFO:tasks.workunit.client.0.smithi102.stdout:4/735: dwrite d1/d27/f6e [0,4194304] 0 2024-03-26T18:54:45.642 INFO:tasks.workunit.client.0.smithi102.stdout:4/736: mkdir d1/de1/df5 0 2024-03-26T18:54:45.649 INFO:tasks.workunit.client.0.smithi102.stdout:8/829: dread d0/d15/f16 [0,4194304] 0 2024-03-26T18:54:45.652 INFO:tasks.workunit.client.0.smithi102.stdout:8/830: creat d0/d15/d5d/d89/f117 x:0 0 0 2024-03-26T18:54:45.657 INFO:tasks.workunit.client.0.smithi102.stdout:4/737: dwrite d1/d17/d1a/f53 [8388608,4194304] 0 2024-03-26T18:54:45.674 INFO:tasks.workunit.client.0.smithi102.stdout:4/738: dwrite d1/d17/d1f/fcd [0,4194304] 0 2024-03-26T18:54:45.675 INFO:tasks.workunit.client.0.smithi102.stdout:4/739: readlink d1/d17/d1f/d5d/d8b/le2 0 2024-03-26T18:54:45.675 INFO:tasks.workunit.client.0.smithi102.stdout:8/831: dwrite d0/d15/da0/d10c/ffa [0,4194304] 0 2024-03-26T18:54:45.677 INFO:tasks.workunit.client.0.smithi102.stdout:4/740: mknod d1/d25/cf6 0 2024-03-26T18:54:45.680 INFO:tasks.workunit.client.0.smithi102.stdout:8/832: symlink d0/d15/d5d/dd3/d26/d19/d9f/l118 0 2024-03-26T18:54:45.680 INFO:tasks.workunit.client.0.smithi102.stdout:8/833: fsync d0/d15/d5d/dd3/d2c/d33/fcd 0 2024-03-26T18:54:45.681 INFO:tasks.workunit.client.0.smithi102.stdout:4/741: symlink d1/d17/d1f/d5d/d6f/dc0/lf7 0 2024-03-26T18:54:45.691 INFO:tasks.workunit.client.0.smithi102.stdout:8/834: dread d0/d15/da0/d10c/ffa [0,4194304] 0 2024-03-26T18:54:45.691 INFO:tasks.workunit.client.0.smithi102.stdout:8/835: creat d0/d15/da0/f119 x:0 0 0 2024-03-26T18:54:45.692 INFO:tasks.workunit.client.0.smithi102.stdout:4/742: dread d1/f8 [4194304,4194304] 0 2024-03-26T18:54:45.693 INFO:tasks.workunit.client.0.smithi102.stdout:8/836: rename d0/d9/f7d to d0/d15/d5d/dd3/d26/d19/d76/f11a 0 2024-03-26T18:54:45.699 INFO:tasks.workunit.client.0.smithi102.stdout:8/837: rename d0/d3c/d96/d98/dda to d0/d15/d5d/dd3/d2c/d11b 0 2024-03-26T18:54:45.699 INFO:tasks.workunit.client.0.smithi102.stdout:8/838: rename d0/d15/d5d/dd3/d2c/d58 to d0/d3c/d11c 0 2024-03-26T18:54:45.699 INFO:tasks.workunit.client.0.smithi102.stdout:8/839: truncate d0/f4d 2417183 0 2024-03-26T18:54:45.705 INFO:tasks.workunit.client.0.smithi102.stdout:8/840: dread d0/d15/d5d/dd3/d2c/d33/d87/f9e [0,4194304] 0 2024-03-26T18:54:45.705 INFO:tasks.workunit.client.0.smithi102.stdout:8/841: write d0/d15/da0/f101 [161758,26556] 0 2024-03-26T18:54:45.718 INFO:tasks.workunit.client.0.smithi102.stdout:4/743: dwrite d1/d17/d1f/d5d/d8b/d3e/fa4 [0,4194304] 0 2024-03-26T18:54:45.718 INFO:tasks.workunit.client.0.smithi102.stdout:4/744: write d1/fd5 [1086508,35842] 0 2024-03-26T18:54:45.725 INFO:tasks.workunit.client.0.smithi102.stdout:4/745: symlink d1/d17/d1f/lf8 0 2024-03-26T18:54:45.733 INFO:tasks.workunit.client.0.smithi102.stdout:8/842: dwrite d0/d15/d5d/dd3/d2c/d2f/f7e [0,4194304] 0 2024-03-26T18:54:45.736 INFO:tasks.workunit.client.0.smithi102.stdout:8/843: truncate d0/d15/f6b 2150764 0 2024-03-26T18:54:45.737 INFO:tasks.workunit.client.0.smithi102.stdout:8/844: dread d0/d15/da0/d10c/fb4 [0,4194304] 0 2024-03-26T18:54:45.737 INFO:tasks.workunit.client.0.smithi102.stdout:8/845: dread - d0/d3c/d11c/fb7 zero size 2024-03-26T18:54:45.738 INFO:tasks.workunit.client.0.smithi102.stdout:8/846: symlink d0/d9/l11d 0 2024-03-26T18:54:45.740 INFO:tasks.workunit.client.0.smithi102.stdout:8/847: mknod d0/d15/d5d/dd3/d26/d80/dbb/c11e 0 2024-03-26T18:54:45.740 INFO:tasks.workunit.client.0.smithi102.stdout:8/848: dread - d0/d9/f10f zero size 2024-03-26T18:54:45.740 INFO:tasks.workunit.client.0.smithi102.stdout:8/849: chown d0/d15/d5d/d89/lae 693 1 2024-03-26T18:54:45.741 INFO:tasks.workunit.client.0.smithi102.stdout:8/850: unlink d0/d15/d5d/dd3/d26/d19/d76/ff0 0 2024-03-26T18:54:45.742 INFO:tasks.workunit.client.0.smithi102.stdout:8/851: write d0/d15/d5d/fc7 [457260,8462] 0 2024-03-26T18:54:45.743 INFO:tasks.workunit.client.0.smithi102.stdout:4/746: dwrite d1/d25/d76/fcb [0,4194304] 0 2024-03-26T18:54:45.743 INFO:tasks.workunit.client.0.smithi102.stdout:4/747: chown d1/d17/d1f/d5d/d6f/faf 1362616731 1 2024-03-26T18:54:45.747 INFO:tasks.workunit.client.0.smithi102.stdout:4/748: getdents d1/d25/d76 0 2024-03-26T18:54:45.747 INFO:tasks.workunit.client.0.smithi102.stdout:4/749: chown d1/d17/d1f/d5d/d6f/dc0/lf7 0 1 2024-03-26T18:54:45.747 INFO:tasks.workunit.client.0.smithi102.stdout:4/750: creat d1/d17/d1a/d21/d77/ff9 x:0 0 0 2024-03-26T18:54:45.760 INFO:tasks.workunit.client.0.smithi102.stdout:0/838: sync 2024-03-26T18:54:45.760 INFO:tasks.workunit.client.0.smithi102.stdout:9/723: sync 2024-03-26T18:54:45.762 INFO:tasks.workunit.client.0.smithi102.stdout:9/724: creat df/d1a/d7f/de1/f10c x:0 0 0 2024-03-26T18:54:45.762 INFO:tasks.workunit.client.0.smithi102.stdout:0/839: creat d6/de/d13/d108/f121 x:0 0 0 2024-03-26T18:54:45.763 INFO:tasks.workunit.client.0.smithi102.stdout:0/840: write d6/d3e/d104/d11a/f107 [900104,108374] 0 2024-03-26T18:54:45.764 INFO:tasks.workunit.client.0.smithi102.stdout:9/725: creat df/d1a/d23/d3e/f10d x:0 0 0 2024-03-26T18:54:45.765 INFO:tasks.workunit.client.0.smithi102.stdout:9/726: unlink df/d1a/d23/le9 0 2024-03-26T18:54:45.769 INFO:tasks.workunit.client.0.smithi102.stdout:9/727: rename df/d1a/d57/f6d to df/d1a/d1d/d72/f10e 0 2024-03-26T18:54:45.771 INFO:tasks.workunit.client.0.smithi102.stdout:9/728: rmdir df/d1a/d65/dbe 39 2024-03-26T18:54:45.771 INFO:tasks.workunit.client.0.smithi102.stdout:9/729: fdatasync df/d1a/d23/d29/d9c/faa 0 2024-03-26T18:54:45.774 INFO:tasks.workunit.client.0.smithi102.stdout:9/730: mkdir df/d10f 0 2024-03-26T18:54:45.774 INFO:tasks.workunit.client.0.smithi102.stdout:9/731: dread - df/d1a/d57/fd1 zero size 2024-03-26T18:54:45.774 INFO:tasks.workunit.client.0.smithi102.stdout:9/732: write df/d1a/d81/d88/fc9 [1506918,109314] 0 2024-03-26T18:54:45.774 INFO:tasks.workunit.client.0.smithi102.stdout:9/733: write df/d1a/d1d/d72/d7d/fb9 [551578,18447] 0 2024-03-26T18:54:45.777 INFO:tasks.workunit.client.0.smithi102.stdout:8/852: dwrite d0/d9/fb5 [0,4194304] 0 2024-03-26T18:54:45.777 INFO:tasks.workunit.client.0.smithi102.stdout:9/734: truncate df/d1a/d1d/d72/fbf 3614964 0 2024-03-26T18:54:45.779 INFO:tasks.workunit.client.0.smithi102.stdout:8/853: truncate d0/d15/d5d/dac/fc3 1169464 0 2024-03-26T18:54:45.780 INFO:tasks.workunit.client.0.smithi102.stdout:4/751: dwrite d1/d25/d90/fab [0,4194304] 0 2024-03-26T18:54:45.781 INFO:tasks.workunit.client.0.smithi102.stdout:9/735: mkdir df/d1a/d23/d3e/d75/d110 0 2024-03-26T18:54:45.781 INFO:tasks.workunit.client.0.smithi102.stdout:9/736: readlink df/d84/ldb 0 2024-03-26T18:54:45.781 INFO:tasks.workunit.client.0.smithi102.stdout:9/737: write df/d1a/d7f/fad [727082,76246] 0 2024-03-26T18:54:45.781 INFO:tasks.workunit.client.0.smithi102.stdout:0/841: dwrite d6/de/d13/d25/da4/d10b/f120 [0,4194304] 0 2024-03-26T18:54:45.781 INFO:tasks.workunit.client.0.smithi102.stdout:0/842: chown d6/de/d13/d25/d31/d45/d82/d84/d88/c9c 470006975 1 2024-03-26T18:54:45.783 INFO:tasks.workunit.client.0.smithi102.stdout:8/854: rename d0/d15/d5d/dd3/d2c/da7/f45 to d0/d15/d9a/f11f 0 2024-03-26T18:54:45.794 INFO:tasks.workunit.client.0.smithi102.stdout:8/855: dread d0/d15/d5d/dd3/f11 [0,4194304] 0 2024-03-26T18:54:45.794 INFO:tasks.workunit.client.0.smithi102.stdout:8/856: dread - d0/d15/d5d/dd3/d2c/d33/fcd zero size 2024-03-26T18:54:45.797 INFO:tasks.workunit.client.0.smithi102.stdout:4/752: dwrite d1/d27/d7a/fd6 [0,4194304] 0 2024-03-26T18:54:45.800 INFO:tasks.workunit.client.0.smithi102.stdout:4/753: dread d1/d17/d1f/d5d/d8b/fc9 [0,4194304] 0 2024-03-26T18:54:45.802 INFO:tasks.workunit.client.0.smithi102.stdout:9/738: mknod df/d1a/d23/d4e/dc1/c111 0 2024-03-26T18:54:45.802 INFO:tasks.workunit.client.0.smithi102.stdout:1/808: sync 2024-03-26T18:54:45.803 INFO:tasks.workunit.client.0.smithi102.stdout:7/705: sync 2024-03-26T18:54:45.803 INFO:tasks.workunit.client.0.smithi102.stdout:6/821: sync 2024-03-26T18:54:45.803 INFO:tasks.workunit.client.0.smithi102.stdout:6/822: fdatasync d2/f21 0 2024-03-26T18:54:45.803 INFO:tasks.workunit.client.0.smithi102.stdout:6/823: write d2/d7/f8 [7292805,56588] 0 2024-03-26T18:54:45.803 INFO:tasks.workunit.client.0.smithi102.stdout:7/706: creat d7/d31/d56/fd4 x:0 0 0 2024-03-26T18:54:45.805 INFO:tasks.workunit.client.0.smithi102.stdout:0/843: symlink d6/dbe/l122 0 2024-03-26T18:54:45.805 INFO:tasks.workunit.client.0.smithi102.stdout:0/844: chown d6/d3e/d104/d11a/lbb 3035827 1 2024-03-26T18:54:45.808 INFO:tasks.workunit.client.0.smithi102.stdout:4/754: creat d1/d25/ffa x:0 0 0 2024-03-26T18:54:45.808 INFO:tasks.workunit.client.0.smithi102.stdout:4/755: truncate d1/d17/d1a/d21/d77/f71 4894543 0 2024-03-26T18:54:45.809 INFO:tasks.workunit.client.0.smithi102.stdout:1/809: rename de/d1f/de3/da6/dca/df7 to de/d1f/de3/d10f 0 2024-03-26T18:54:45.812 INFO:tasks.workunit.client.0.smithi102.stdout:8/857: dwrite d0/d15/d5d/dd3/d2c/d2f/f60 [0,4194304] 0 2024-03-26T18:54:45.815 INFO:tasks.workunit.client.0.smithi102.stdout:0/845: dread d6/de/d13/d25/d4f/d66/fb7 [0,4194304] 0 2024-03-26T18:54:45.815 INFO:tasks.workunit.client.0.smithi102.stdout:7/707: mkdir d7/d4e/dd5 0 2024-03-26T18:54:45.815 INFO:tasks.workunit.client.0.smithi102.stdout:7/708: fsync d7/d2a/f74 0 2024-03-26T18:54:45.818 INFO:tasks.workunit.client.0.smithi102.stdout:6/824: symlink d2/df/d71/de6/l115 0 2024-03-26T18:54:45.818 INFO:tasks.workunit.client.0.smithi102.stdout:6/825: readlink d2/d3d/d53/d63/lae 0 2024-03-26T18:54:45.818 INFO:tasks.workunit.client.0.smithi102.stdout:6/826: readlink d2/df/l19 0 2024-03-26T18:54:45.818 INFO:tasks.workunit.client.0.smithi102.stdout:6/827: fsync d2/d3d/d53/d77/f88 0 2024-03-26T18:54:45.826 INFO:tasks.workunit.client.0.smithi102.stdout:8/858: mkdir d0/d3c/d96/dd5/d120 0 2024-03-26T18:54:45.827 INFO:tasks.workunit.client.0.smithi102.stdout:7/709: mkdir d7/d12/d2b/da8/dd6 0 2024-03-26T18:54:45.827 INFO:tasks.workunit.client.0.smithi102.stdout:7/710: truncate d7/d10/d67/fb2 4626634 0 2024-03-26T18:54:45.827 INFO:tasks.workunit.client.0.smithi102.stdout:7/711: creat d7/d12/d51/d6a/fd7 x:0 0 0 2024-03-26T18:54:45.828 INFO:tasks.workunit.client.0.smithi102.stdout:6/828: write d2/faf [111954,126346] 0 2024-03-26T18:54:45.830 INFO:tasks.workunit.client.0.smithi102.stdout:8/859: unlink d0/d15/d5d/dd3/d26/d19/f82 0 2024-03-26T18:54:45.835 INFO:tasks.workunit.client.0.smithi102.stdout:7/712: creat d7/d10/d67/d69/d9b/fd8 x:0 0 0 2024-03-26T18:54:45.835 INFO:tasks.workunit.client.0.smithi102.stdout:7/713: dread - d7/d10/d27/d34/d72/fb3 zero size 2024-03-26T18:54:45.836 INFO:tasks.workunit.client.0.smithi102.stdout:6/829: creat d2/df/d71/d80/d91/f116 x:0 0 0 2024-03-26T18:54:45.837 INFO:tasks.workunit.client.0.smithi102.stdout:7/714: mknod d7/d12/d4d/cd9 0 2024-03-26T18:54:45.841 INFO:tasks.workunit.client.0.smithi102.stdout:7/715: creat d7/d10/d67/d69/d98/fda x:0 0 0 2024-03-26T18:54:45.842 INFO:tasks.workunit.client.0.smithi102.stdout:9/739: dwrite df/d1a/d7f/fba [0,4194304] 0 2024-03-26T18:54:45.843 INFO:tasks.workunit.client.0.smithi102.stdout:9/740: chown df/d1a/d65/ld7 818 1 2024-03-26T18:54:45.843 INFO:tasks.workunit.client.0.smithi102.stdout:9/741: creat df/d1a/d23/d29/d9c/f112 x:0 0 0 2024-03-26T18:54:45.844 INFO:tasks.workunit.client.0.smithi102.stdout:6/830: truncate d2/f47 2168651 0 2024-03-26T18:54:45.844 INFO:tasks.workunit.client.0.smithi102.stdout:6/831: chown d2/df/f2b 1709 1 2024-03-26T18:54:45.846 INFO:tasks.workunit.client.0.smithi102.stdout:1/810: dwrite de/d16/d1d/f1e [8388608,4194304] 0 2024-03-26T18:54:45.847 INFO:tasks.workunit.client.0.smithi102.stdout:0/846: dwrite d6/f11 [4194304,4194304] 0 2024-03-26T18:54:45.847 INFO:tasks.workunit.client.0.smithi102.stdout:7/716: creat d7/d10/d27/da3/fdb x:0 0 0 2024-03-26T18:54:45.848 INFO:tasks.workunit.client.0.smithi102.stdout:2/811: sync 2024-03-26T18:54:45.848 INFO:tasks.workunit.client.0.smithi102.stdout:4/756: dwrite d1/d17/d1a/d21/d77/fd3 [0,4194304] 0 2024-03-26T18:54:45.857 INFO:tasks.workunit.client.0.smithi102.stdout:6/832: symlink d2/d7/d3c/d78/dc4/l117 0 2024-03-26T18:54:45.857 INFO:tasks.workunit.client.0.smithi102.stdout:9/742: dwrite df/d84/f9e [0,4194304] 0 2024-03-26T18:54:45.857 INFO:tasks.workunit.client.0.smithi102.stdout:9/743: write df/d1a/d1d/d72/fcd [2011326,112047] 0 2024-03-26T18:54:45.860 INFO:tasks.workunit.client.0.smithi102.stdout:1/811: rename de/d1f/de3/dd6 to de/d16/d1d/d2a/d2c/d56/d6e/dfc/d110 0 2024-03-26T18:54:45.860 INFO:tasks.workunit.client.0.smithi102.stdout:1/812: readlink de/d16/d1d/d2a/d2c/d56/d78/d80/d8e/lb9 0 2024-03-26T18:54:45.860 INFO:tasks.workunit.client.0.smithi102.stdout:1/813: chown f8 210441202 1 2024-03-26T18:54:45.861 INFO:tasks.workunit.client.0.smithi102.stdout:0/847: creat d6/d2d/d3d/f123 x:0 0 0 2024-03-26T18:54:45.862 INFO:tasks.workunit.client.0.smithi102.stdout:2/812: creat d3/d20/d78/d84/d6e/dc6/f124 x:0 0 0 2024-03-26T18:54:45.866 INFO:tasks.workunit.client.0.smithi102.stdout:4/757: link d1/d17/d1f/d5d/d6f/l84 d1/de1/df5/lfb 0 2024-03-26T18:54:45.868 INFO:tasks.workunit.client.0.smithi102.stdout:6/833: dread d2/d3d/fbc [0,4194304] 0 2024-03-26T18:54:45.868 INFO:tasks.workunit.client.0.smithi102.stdout:6/834: readlink d2/d12/la2 0 2024-03-26T18:54:45.869 INFO:tasks.workunit.client.0.smithi102.stdout:0/848: dread d6/de/d13/d1f/d71/f9b [0,4194304] 0 2024-03-26T18:54:45.874 INFO:tasks.workunit.client.0.smithi102.stdout:1/814: write de/f3f [3372543,8956] 0 2024-03-26T18:54:45.878 INFO:tasks.workunit.client.0.smithi102.stdout:9/744: mknod df/d1a/d23/d3e/d75/d110/c113 0 2024-03-26T18:54:45.878 INFO:tasks.workunit.client.0.smithi102.stdout:9/745: write df/d1a/d1d/f26 [4946504,112017] 0 2024-03-26T18:54:45.882 INFO:tasks.workunit.client.0.smithi102.stdout:7/717: rename d7/d2a/fc2 to d7/d10/d67/d69/d9b/fdc 0 2024-03-26T18:54:45.883 INFO:tasks.workunit.client.0.smithi102.stdout:6/835: dread d2/d7/f8 [0,4194304] 0 2024-03-26T18:54:45.883 INFO:tasks.workunit.client.0.smithi102.stdout:6/836: creat d2/d12/f118 x:0 0 0 2024-03-26T18:54:45.886 INFO:tasks.workunit.client.0.smithi102.stdout:1/815: write de/d2e/f55 [2816878,118438] 0 2024-03-26T18:54:45.887 INFO:tasks.workunit.client.0.smithi102.stdout:0/849: mkdir d6/de/d13/d25/d31/d45/d82/d84/d88/d124 0 2024-03-26T18:54:45.891 INFO:tasks.workunit.client.0.smithi102.stdout:4/758: dwrite d1/d27/ff0 [0,4194304] 0 2024-03-26T18:54:45.891 INFO:tasks.workunit.client.0.smithi102.stdout:4/759: dread - d1/d17/d1a/d21/d77/ff9 zero size 2024-03-26T18:54:45.893 INFO:tasks.workunit.client.0.smithi102.stdout:9/746: dread df/d1a/d23/f48 [0,4194304] 0 2024-03-26T18:54:45.896 INFO:tasks.workunit.client.0.smithi102.stdout:0/850: rename d6/de/d13/d25/d31/d45/d82/d100 to d6/de/d13/d1f/d71/d125 0 2024-03-26T18:54:45.898 INFO:tasks.workunit.client.0.smithi102.stdout:9/747: symlink df/d1a/d23/l114 0 2024-03-26T18:54:45.902 INFO:tasks.workunit.client.0.smithi102.stdout:0/851: mknod d6/de/d13/d1f/d71/c126 0 2024-03-26T18:54:45.902 INFO:tasks.workunit.client.0.smithi102.stdout:0/852: readlink d6/d3e/d69/l10a 0 2024-03-26T18:54:45.902 INFO:tasks.workunit.client.0.smithi102.stdout:9/748: symlink df/d46/da8/l115 0 2024-03-26T18:54:45.904 INFO:tasks.workunit.client.0.smithi102.stdout:4/760: dread d1/d17/d1f/fd9 [0,4194304] 0 2024-03-26T18:54:45.905 INFO:tasks.workunit.client.0.smithi102.stdout:9/749: creat df/d1a/d1d/d72/d7d/f116 x:0 0 0 2024-03-26T18:54:45.905 INFO:tasks.workunit.client.0.smithi102.stdout:1/816: dread de/d16/d1d/d2a/d67/f95 [0,4194304] 0 2024-03-26T18:54:45.907 INFO:tasks.workunit.client.0.smithi102.stdout:9/750: truncate df/f17 1611179 0 2024-03-26T18:54:45.908 INFO:tasks.workunit.client.0.smithi102.stdout:1/817: mkdir de/d16/d1d/d2a/d2c/d43/d111 0 2024-03-26T18:54:45.910 INFO:tasks.workunit.client.0.smithi102.stdout:9/751: mkdir df/d1a/d23/d3e/d75/d117 0 2024-03-26T18:54:45.911 INFO:tasks.workunit.client.0.smithi102.stdout:9/752: mknod df/d1a/d1d/d72/d7d/c118 0 2024-03-26T18:54:45.918 INFO:tasks.workunit.client.0.smithi102.stdout:6/837: dwrite d2/d12/f10f [4194304,4194304] 0 2024-03-26T18:54:45.918 INFO:tasks.workunit.client.0.smithi102.stdout:6/838: fdatasync d2/d7/f1d 0 2024-03-26T18:54:45.919 INFO:tasks.workunit.client.0.smithi102.stdout:7/718: dwrite d7/d12/d4d/f8b [0,4194304] 0 2024-03-26T18:54:45.919 INFO:tasks.workunit.client.0.smithi102.stdout:2/813: dwrite d3/d20/ff0 [0,4194304] 0 2024-03-26T18:54:45.919 INFO:tasks.workunit.client.0.smithi102.stdout:4/761: dread d1/d17/d1f/d5d/d6f/faf [0,4194304] 0 2024-03-26T18:54:45.921 INFO:tasks.workunit.client.0.smithi102.stdout:7/719: dread d7/d12/f38 [0,4194304] 0 2024-03-26T18:54:45.923 INFO:tasks.workunit.client.0.smithi102.stdout:1/818: dread de/f3f [0,4194304] 0 2024-03-26T18:54:45.925 INFO:tasks.workunit.client.0.smithi102.stdout:2/814: symlink d3/d11/d15/d46/d7b/d97/d10a/d123/l125 0 2024-03-26T18:54:45.925 INFO:tasks.workunit.client.0.smithi102.stdout:2/815: readlink d3/d2a/d5f/l70 0 2024-03-26T18:54:45.926 INFO:tasks.workunit.client.0.smithi102.stdout:4/762: creat d1/d25/d76/ffc x:0 0 0 2024-03-26T18:54:45.926 INFO:tasks.workunit.client.0.smithi102.stdout:4/763: chown d1/d27/d98/db7/lf4 798417478 1 2024-03-26T18:54:45.927 INFO:tasks.workunit.client.0.smithi102.stdout:6/839: write d2/df/d35/f6e [1700740,56973] 0 2024-03-26T18:54:45.928 INFO:tasks.workunit.client.0.smithi102.stdout:7/720: symlink d7/d10/d67/d69/ldd 0 2024-03-26T18:54:45.928 INFO:tasks.workunit.client.0.smithi102.stdout:7/721: fdatasync d7/d10/d27/d34/d72/f76 0 2024-03-26T18:54:45.928 INFO:tasks.workunit.client.0.smithi102.stdout:6/840: dread d2/d7/f20 [0,4194304] 0 2024-03-26T18:54:45.928 INFO:tasks.workunit.client.0.smithi102.stdout:6/841: write d2/df/d35/d43/d75/f9b [270643,66302] 0 2024-03-26T18:54:45.929 INFO:tasks.workunit.client.0.smithi102.stdout:6/842: truncate d2/d3d/f8a 471800 0 2024-03-26T18:54:45.929 INFO:tasks.workunit.client.0.smithi102.stdout:4/764: creat d1/d17/d2a/d9a/ffd x:0 0 0 2024-03-26T18:54:45.929 INFO:tasks.workunit.client.0.smithi102.stdout:4/765: fsync d1/d27/f6e 0 2024-03-26T18:54:45.932 INFO:tasks.workunit.client.0.smithi102.stdout:4/766: rename d1/d17/d1f/d5d/d8b/lb2 to d1/de1/df5/lfe 0 2024-03-26T18:54:45.934 INFO:tasks.workunit.client.0.smithi102.stdout:2/816: dread d3/fe5 [0,4194304] 0 2024-03-26T18:54:45.934 INFO:tasks.workunit.client.0.smithi102.stdout:1/819: dread de/d16/d1d/d2a/d2c/d56/d78/d80/f98 [0,4194304] 0 2024-03-26T18:54:45.936 INFO:tasks.workunit.client.0.smithi102.stdout:6/843: dread d2/f33 [0,4194304] 0 2024-03-26T18:54:45.936 INFO:tasks.workunit.client.0.smithi102.stdout:4/767: unlink d1/d25/f69 0 2024-03-26T18:54:45.938 INFO:tasks.workunit.client.0.smithi102.stdout:2/817: mknod d3/d11/d15/d31/c126 0 2024-03-26T18:54:45.938 INFO:tasks.workunit.client.0.smithi102.stdout:2/818: fdatasync d3/d11/d15/d46/d7b/f119 0 2024-03-26T18:54:45.939 INFO:tasks.workunit.client.0.smithi102.stdout:1/820: creat de/d16/d1d/d61/f112 x:0 0 0 2024-03-26T18:54:45.941 INFO:tasks.workunit.client.0.smithi102.stdout:6/844: rename d2/df/d71/d80/d91/db8/lfe to d2/d7/d3c/d78/l119 0 2024-03-26T18:54:45.943 INFO:tasks.workunit.client.0.smithi102.stdout:4/768: creat d1/d17/d1f/d5d/d8b/d3e/fff x:0 0 0 2024-03-26T18:54:45.943 INFO:tasks.workunit.client.0.smithi102.stdout:4/769: truncate d1/d17/dbb/dee/fef 907505 0 2024-03-26T18:54:45.944 INFO:tasks.workunit.client.0.smithi102.stdout:1/821: mkdir de/d16/d1d/d2a/d2c/d56/dfa/d113 0 2024-03-26T18:54:45.944 INFO:tasks.workunit.client.0.smithi102.stdout:1/822: read de/d16/d1d/d2a/fb2 [235939,26463] 0 2024-03-26T18:54:45.947 INFO:tasks.workunit.client.0.smithi102.stdout:4/770: mkdir d1/d17/d1f/d5d/d6f/dc0/d100 0 2024-03-26T18:54:45.948 INFO:tasks.workunit.client.0.smithi102.stdout:1/823: mknod de/d16/d1d/c114 0 2024-03-26T18:54:45.949 INFO:tasks.workunit.client.0.smithi102.stdout:7/722: dwrite d7/d10/d27/fd1 [0,4194304] 0 2024-03-26T18:54:45.952 INFO:tasks.workunit.client.0.smithi102.stdout:2/819: write d3/d11/d15/d31/d39/f5a [3079677,56384] 0 2024-03-26T18:54:45.953 INFO:tasks.workunit.client.0.smithi102.stdout:2/820: dread - d3/d74/f104 zero size 2024-03-26T18:54:45.953 INFO:tasks.workunit.client.0.smithi102.stdout:2/821: readlink d3/d11/d15/d46/d7b/d97/d10a/d123/l125 0 2024-03-26T18:54:45.953 INFO:tasks.workunit.client.0.smithi102.stdout:2/822: write d3/d2d/d69/fb8 [727102,41704] 0 2024-03-26T18:54:45.966 INFO:tasks.workunit.client.0.smithi102.stdout:1/824: dwrite de/d1f/de3/fe4 [0,4194304] 0 2024-03-26T18:54:45.977 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:45 smithi190 ceph-mon[35043]: Standby manager daemon smithi190.zrixos restarted 2024-03-26T18:54:45.977 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:45 smithi190 ceph-mon[35043]: Standby manager daemon smithi190.zrixos started 2024-03-26T18:54:45.977 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:45 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/crt"}]: dispatch 2024-03-26T18:54:45.977 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:45 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:54:45.978 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:45 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/key"}]: dispatch 2024-03-26T18:54:45.978 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:45 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:54:46.092 INFO:tasks.workunit.client.0.smithi102.stdout:6/845: mknod d2/c11a 0 2024-03-26T18:54:46.096 INFO:tasks.workunit.client.0.smithi102.stdout:7/723: truncate d7/d2a/f30 1798062 0 2024-03-26T18:54:46.096 INFO:tasks.workunit.client.0.smithi102.stdout:7/724: read - d7/d10/d67/d69/d9b/fd8 zero size 2024-03-26T18:54:46.106 INFO:tasks.workunit.client.0.smithi102.stdout:1/825: mkdir de/d16/d1d/d2a/d2c/d56/dfa/d113/d115 0 2024-03-26T18:54:46.109 INFO:tasks.workunit.client.0.smithi102.stdout:6/846: symlink d2/df/d35/d60/d10b/l11b 0 2024-03-26T18:54:46.110 INFO:tasks.workunit.client.0.smithi102.stdout:6/847: chown d2/d12/dce/cde 66960956 1 2024-03-26T18:54:46.111 INFO:tasks.workunit.client.0.smithi102.stdout:7/725: unlink d7/d12/l2e 0 2024-03-26T18:54:46.113 INFO:tasks.workunit.client.0.smithi102.stdout:2/823: rmdir d3/d11/d15/d46/d7b 39 2024-03-26T18:54:46.114 INFO:tasks.workunit.client.0.smithi102.stdout:1/826: mknod de/d16/d1d/d2a/d67/c116 0 2024-03-26T18:54:46.115 INFO:tasks.workunit.client.0.smithi102.stdout:2/824: creat d3/d11/d15/d46/d7b/d97/d10a/f127 x:0 0 0 2024-03-26T18:54:46.115 INFO:tasks.workunit.client.0.smithi102.stdout:2/825: creat d3/d74/f128 x:0 0 0 2024-03-26T18:54:46.116 INFO:tasks.workunit.client.0.smithi102.stdout:1/827: mkdir de/d16/d1d/d2a/d2c/d56/d6e/dfc/d117 0 2024-03-26T18:54:46.119 INFO:tasks.workunit.client.0.smithi102.stdout:2/826: mkdir d3/d2a/d129 0 2024-03-26T18:54:46.120 INFO:tasks.workunit.client.0.smithi102.stdout:7/726: write d7/d10/d17/f1b [7469649,49986] 0 2024-03-26T18:54:46.120 INFO:tasks.workunit.client.0.smithi102.stdout:1/828: creat de/d16/d1d/d61/d6a/df2/f118 x:0 0 0 2024-03-26T18:54:46.120 INFO:tasks.workunit.client.0.smithi102.stdout:1/829: fsync de/d16/f51 0 2024-03-26T18:54:46.122 INFO:tasks.workunit.client.0.smithi102.stdout:2/827: mkdir d3/d11/d27/dc0/d12a 0 2024-03-26T18:54:46.122 INFO:tasks.workunit.client.0.smithi102.stdout:1/830: write de/d16/d1d/d2a/d2c/d56/d6e/fe1 [78147,63026] 0 2024-03-26T18:54:46.123 INFO:tasks.workunit.client.0.smithi102.stdout:7/727: readlink d7/d12/d4d/d62/l7e 0 2024-03-26T18:54:46.123 INFO:tasks.workunit.client.0.smithi102.stdout:7/728: dread - d7/d10/d67/d69/d78/fba zero size 2024-03-26T18:54:46.124 INFO:tasks.workunit.client.0.smithi102.stdout:7/729: symlink d7/d10/d17/db9/lde 0 2024-03-26T18:54:46.124 INFO:tasks.workunit.client.0.smithi102.stdout:7/730: creat d7/d10/d67/d69/d98/fdf x:0 0 0 2024-03-26T18:54:46.124 INFO:tasks.workunit.client.0.smithi102.stdout:7/731: fsync d7/d12/f38 0 2024-03-26T18:54:46.124 INFO:tasks.workunit.client.0.smithi102.stdout:7/732: creat d7/d10/d67/d69/d9b/fe0 x:0 0 0 2024-03-26T18:54:46.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:45 smithi102 ceph-mon[29002]: Standby manager daemon smithi190.zrixos restarted 2024-03-26T18:54:46.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:45 smithi102 ceph-mon[29002]: Standby manager daemon smithi190.zrixos started 2024-03-26T18:54:46.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:45 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/crt"}]: dispatch 2024-03-26T18:54:46.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:45 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:54:46.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:45 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/key"}]: dispatch 2024-03-26T18:54:46.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:45 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:54:46.128 INFO:tasks.workunit.client.0.smithi102.stdout:7/733: rmdir d7/d10/d17/db9 39 2024-03-26T18:54:46.128 INFO:tasks.workunit.client.0.smithi102.stdout:6/848: dwrite d2/d3/fe8 [0,4194304] 0 2024-03-26T18:54:46.128 INFO:tasks.workunit.client.0.smithi102.stdout:6/849: truncate d2/df/dba/fdd 56463 0 2024-03-26T18:54:46.130 INFO:tasks.workunit.client.0.smithi102.stdout:9/753: sync 2024-03-26T18:54:46.130 INFO:tasks.workunit.client.0.smithi102.stdout:0/853: sync 2024-03-26T18:54:46.130 INFO:tasks.workunit.client.0.smithi102.stdout:4/771: sync 2024-03-26T18:54:46.130 INFO:tasks.workunit.client.0.smithi102.stdout:9/754: fsync df/d1a/d23/d60/f98 0 2024-03-26T18:54:46.130 INFO:tasks.workunit.client.0.smithi102.stdout:8/860: sync 2024-03-26T18:54:46.133 INFO:tasks.workunit.client.0.smithi102.stdout:4/772: symlink d1/d75/l101 0 2024-03-26T18:54:46.133 INFO:tasks.workunit.client.0.smithi102.stdout:0/854: symlink d6/d3e/d69/l127 0 2024-03-26T18:54:46.133 INFO:tasks.workunit.client.0.smithi102.stdout:0/855: chown d6/de/d93/dac/fc2 0 1 2024-03-26T18:54:46.134 INFO:tasks.workunit.client.0.smithi102.stdout:4/773: mkdir d1/d17/d1a/d21/d77/d102 0 2024-03-26T18:54:46.135 INFO:tasks.workunit.client.0.smithi102.stdout:4/774: symlink d1/d25/db3/l103 0 2024-03-26T18:54:46.138 INFO:tasks.workunit.client.0.smithi102.stdout:4/775: symlink d1/d27/l104 0 2024-03-26T18:54:46.138 INFO:tasks.workunit.client.0.smithi102.stdout:4/776: write d1/d17/d1a/d21/f64 [4212080,72127] 0 2024-03-26T18:54:46.139 INFO:tasks.workunit.client.0.smithi102.stdout:1/831: dwrite de/d16/d1d/d2a/d67/ff5 [0,4194304] 0 2024-03-26T18:54:46.139 INFO:tasks.workunit.client.0.smithi102.stdout:6/850: dread d2/fca [0,4194304] 0 2024-03-26T18:54:46.140 INFO:tasks.workunit.client.0.smithi102.stdout:6/851: write d2/df/d35/d43/f61 [1491289,67629] 0 2024-03-26T18:54:46.142 INFO:tasks.workunit.client.0.smithi102.stdout:4/777: creat d1/d17/d2a/d9a/dc2/f105 x:0 0 0 2024-03-26T18:54:46.144 INFO:tasks.workunit.client.0.smithi102.stdout:9/755: write df/d1a/d23/d29/f31 [3200116,88628] 0 2024-03-26T18:54:46.144 INFO:tasks.workunit.client.0.smithi102.stdout:9/756: fsync df/d1a/d23/d29/d9c/faa 0 2024-03-26T18:54:46.145 INFO:tasks.workunit.client.0.smithi102.stdout:9/757: stat df/d1a/d23/d4e/dc1 0 2024-03-26T18:54:46.145 INFO:tasks.workunit.client.0.smithi102.stdout:9/758: fsync df/d1a/d57/f8b 0 2024-03-26T18:54:46.145 INFO:tasks.workunit.client.0.smithi102.stdout:9/759: chown df/d1a/d57/fde 6 1 2024-03-26T18:54:46.147 INFO:tasks.workunit.client.0.smithi102.stdout:6/852: symlink d2/d3d/d53/db1/l11c 0 2024-03-26T18:54:46.147 INFO:tasks.workunit.client.0.smithi102.stdout:6/853: chown d2/d3d/d53/d77/c93 17368927 1 2024-03-26T18:54:46.159 INFO:tasks.workunit.client.0.smithi102.stdout:4/778: dread d1/d27/ff0 [0,4194304] 0 2024-03-26T18:54:46.160 INFO:tasks.workunit.client.0.smithi102.stdout:7/734: dwrite d7/d10/d67/f80 [0,4194304] 0 2024-03-26T18:54:46.160 INFO:tasks.workunit.client.0.smithi102.stdout:7/735: stat d7/d10/d67/d69/d9b/fd8 0 2024-03-26T18:54:46.161 INFO:tasks.workunit.client.0.smithi102.stdout:7/736: fsync d7/d31/d79/fb4 0 2024-03-26T18:54:46.162 INFO:tasks.workunit.client.0.smithi102.stdout:7/737: mkdir d7/d10/d67/de1 0 2024-03-26T18:54:46.162 INFO:tasks.workunit.client.0.smithi102.stdout:8/861: dwrite d0/d15/d5d/dd3/d26/d19/d76/f11a [0,4194304] 0 2024-03-26T18:54:46.163 INFO:tasks.workunit.client.0.smithi102.stdout:1/832: dwrite de/d2e/f55 [0,4194304] 0 2024-03-26T18:54:46.163 INFO:tasks.workunit.client.0.smithi102.stdout:1/833: getdents de/d16/d1d/d2a/d2c/d56/d78/dc3 0 2024-03-26T18:54:46.163 INFO:tasks.workunit.client.0.smithi102.stdout:7/738: symlink d7/d10/d27/d34/d82/le2 0 2024-03-26T18:54:46.163 INFO:tasks.workunit.client.0.smithi102.stdout:7/739: stat d7/d10/d67/f80 0 2024-03-26T18:54:46.165 INFO:tasks.workunit.client.0.smithi102.stdout:4/779: dread d1/d17/d1f/d5d/d8b/d3e/f58 [0,4194304] 0 2024-03-26T18:54:46.165 INFO:tasks.workunit.client.0.smithi102.stdout:4/780: chown d1/d17/d1f/d5d/d8b/d3e/l96 353 1 2024-03-26T18:54:46.165 INFO:tasks.workunit.client.0.smithi102.stdout:1/834: creat de/d1f/de3/f119 x:0 0 0 2024-03-26T18:54:46.166 INFO:tasks.workunit.client.0.smithi102.stdout:1/835: write de/d16/f54 [312854,79833] 0 2024-03-26T18:54:46.169 INFO:tasks.workunit.client.0.smithi102.stdout:7/740: stat d7/d12/d4d/d62/c93 0 2024-03-26T18:54:46.169 INFO:tasks.workunit.client.0.smithi102.stdout:7/741: chown d7/d12/d4b 48032 1 2024-03-26T18:54:46.172 INFO:tasks.workunit.client.0.smithi102.stdout:1/836: rename de/d16/d1d/d61/f7c to de/d16/d1d/d2a/d2c/d56/d6e/dfc/d110/f11a 0 2024-03-26T18:54:46.173 INFO:tasks.workunit.client.0.smithi102.stdout:7/742: creat d7/d10/d17/d9f/fe3 x:0 0 0 2024-03-26T18:54:46.174 INFO:tasks.workunit.client.0.smithi102.stdout:7/743: chown d7/d12/d2b/c70 247299 1 2024-03-26T18:54:46.174 INFO:tasks.workunit.client.0.smithi102.stdout:7/744: write d7/d10/d67/fb2 [1877134,65228] 0 2024-03-26T18:54:46.174 INFO:tasks.workunit.client.0.smithi102.stdout:7/745: chown d7/d10/d27/d34 201928796 1 2024-03-26T18:54:46.174 INFO:tasks.workunit.client.0.smithi102.stdout:1/837: symlink de/d16/d1d/d2a/d2c/de6/l11b 0 2024-03-26T18:54:46.182 INFO:tasks.workunit.client.0.smithi102.stdout:7/746: write d7/d10/d17/f5c [433231,14855] 0 2024-03-26T18:54:46.186 INFO:tasks.workunit.client.0.smithi102.stdout:4/781: dwrite d1/d17/d1f/d5d/d8b/d3e/fa4 [4194304,4194304] 0 2024-03-26T18:54:46.189 INFO:tasks.workunit.client.0.smithi102.stdout:8/862: dwrite d0/d9/f2b [0,4194304] 0 2024-03-26T18:54:46.190 INFO:tasks.workunit.client.0.smithi102.stdout:7/747: dread d7/d12/f8d [0,4194304] 0 2024-03-26T18:54:46.190 INFO:tasks.workunit.client.0.smithi102.stdout:7/748: creat d7/d10/d27/d34/d82/fe4 x:0 0 0 2024-03-26T18:54:46.191 INFO:tasks.workunit.client.0.smithi102.stdout:8/863: creat d0/d3c/d11c/f121 x:0 0 0 2024-03-26T18:54:46.192 INFO:tasks.workunit.client.0.smithi102.stdout:7/749: rmdir d7/d12/d4d/d62 39 2024-03-26T18:54:46.194 INFO:tasks.workunit.client.0.smithi102.stdout:8/864: mknod d0/d15/d9a/c122 0 2024-03-26T18:54:46.195 INFO:tasks.workunit.client.0.smithi102.stdout:7/750: truncate d7/d10/f14 339565 0 2024-03-26T18:54:46.200 INFO:tasks.workunit.client.0.smithi102.stdout:7/751: rmdir d7/d10/d17/d9f 39 2024-03-26T18:54:46.200 INFO:tasks.workunit.client.0.smithi102.stdout:7/752: getdents d7/d10/d67/de1 0 2024-03-26T18:54:46.200 INFO:tasks.workunit.client.0.smithi102.stdout:7/753: readlink d7/l25 0 2024-03-26T18:54:46.200 INFO:tasks.workunit.client.0.smithi102.stdout:7/754: fsync d7/d31/f45 0 2024-03-26T18:54:46.200 INFO:tasks.workunit.client.0.smithi102.stdout:7/755: truncate d7/d12/d4d/f8b 4957731 0 2024-03-26T18:54:46.203 INFO:tasks.workunit.client.0.smithi102.stdout:7/756: unlink d7/f7b 0 2024-03-26T18:54:46.204 INFO:tasks.workunit.client.0.smithi102.stdout:7/757: mkdir d7/d10/d67/d69/d9b/de5 0 2024-03-26T18:54:46.204 INFO:tasks.workunit.client.0.smithi102.stdout:7/758: stat d7/d10/d27/d34/d82/fb7 0 2024-03-26T18:54:46.205 INFO:tasks.workunit.client.0.smithi102.stdout:7/759: creat d7/d10/d67/d69/d9b/dae/fe6 x:0 0 0 2024-03-26T18:54:46.205 INFO:tasks.workunit.client.0.smithi102.stdout:7/760: truncate d7/d10/d17/d9f/fe3 658502 0 2024-03-26T18:54:46.205 INFO:tasks.workunit.client.0.smithi102.stdout:7/761: write d7/d12/f3c [4172816,79621] 0 2024-03-26T18:54:46.207 INFO:tasks.workunit.client.0.smithi102.stdout:7/762: unlink d7/d12/l32 0 2024-03-26T18:54:46.207 INFO:tasks.workunit.client.0.smithi102.stdout:7/763: readlink d7/d31/l61 0 2024-03-26T18:54:46.208 INFO:tasks.workunit.client.0.smithi102.stdout:7/764: rename d7/d12/d2b/f2f to d7/d10/fe7 0 2024-03-26T18:54:46.208 INFO:tasks.workunit.client.0.smithi102.stdout:7/765: stat d7/d12/d4d/f5e 0 2024-03-26T18:54:46.209 INFO:tasks.workunit.client.0.smithi102.stdout:7/766: mkdir d7/d2a/de8 0 2024-03-26T18:54:46.209 INFO:tasks.workunit.client.0.smithi102.stdout:7/767: fsync d7/d12/d51/fbc 0 2024-03-26T18:54:46.209 INFO:tasks.workunit.client.0.smithi102.stdout:7/768: chown d7/d10/f14 6954291 1 2024-03-26T18:54:46.210 INFO:tasks.workunit.client.0.smithi102.stdout:7/769: creat d7/d10/d27/d34/dd3/fe9 x:0 0 0 2024-03-26T18:54:46.210 INFO:tasks.workunit.client.0.smithi102.stdout:7/770: chown d7/d10/d67/d69/d78/fba 1 1 2024-03-26T18:54:46.210 INFO:tasks.workunit.client.0.smithi102.stdout:7/771: chown d7/d10/d27/d34/d72 231207 1 2024-03-26T18:54:46.210 INFO:tasks.workunit.client.0.smithi102.stdout:7/772: read - d7/d10/fe7 zero size 2024-03-26T18:54:46.221 INFO:tasks.workunit.client.0.smithi102.stdout:8/865: dwrite d0/d15/da0/d10c/f95 [4194304,4194304] 0 2024-03-26T18:54:46.222 INFO:tasks.workunit.client.0.smithi102.stdout:4/782: dwrite d1/d17/fae [0,4194304] 0 2024-03-26T18:54:46.223 INFO:tasks.workunit.client.0.smithi102.stdout:8/866: rename d0/d15/d5d/dd3/d26/d19/d9f/cd6 to d0/d15/d5d/dd3/d2c/da7/c123 0 2024-03-26T18:54:46.225 INFO:tasks.workunit.client.0.smithi102.stdout:8/867: mknod d0/d3c/d96/d98/db6/c124 0 2024-03-26T18:54:46.228 INFO:tasks.workunit.client.0.smithi102.stdout:8/868: creat d0/d15/d5d/dd3/d26/d19/d76/f125 x:0 0 0 2024-03-26T18:54:46.231 INFO:tasks.workunit.client.0.smithi102.stdout:7/773: dwrite d7/fb [0,4194304] 0 2024-03-26T18:54:46.233 INFO:tasks.workunit.client.0.smithi102.stdout:7/774: truncate d7/d10/d27/d34/d72/f76 2331112 0 2024-03-26T18:54:46.247 INFO:tasks.workunit.client.0.smithi102.stdout:7/775: write d7/d10/d67/f80 [1350914,25485] 0 2024-03-26T18:54:46.254 INFO:tasks.workunit.client.0.smithi102.stdout:4/783: dwrite d1/f26 [0,4194304] 0 2024-03-26T18:54:46.255 INFO:tasks.workunit.client.0.smithi102.stdout:8/869: dwrite d0/d15/da0/d10c/fdc [0,4194304] 0 2024-03-26T18:54:46.257 INFO:tasks.workunit.client.0.smithi102.stdout:4/784: unlink d1/d17/d1a/fa1 0 2024-03-26T18:54:46.257 INFO:tasks.workunit.client.0.smithi102.stdout:4/785: readlink d1/d27/d7a/lad 0 2024-03-26T18:54:46.257 INFO:tasks.workunit.client.0.smithi102.stdout:4/786: write d1/d25/d76/ffc [849594,83977] 0 2024-03-26T18:54:46.257 INFO:tasks.workunit.client.0.smithi102.stdout:4/787: stat d1/d17/d1f/fcd 0 2024-03-26T18:54:46.258 INFO:tasks.workunit.client.0.smithi102.stdout:4/788: read - d1/d17/d2a/d9a/dc2/f105 zero size 2024-03-26T18:54:46.258 INFO:tasks.workunit.client.0.smithi102.stdout:4/789: read d1/d25/d76/f86 [3569865,78826] 0 2024-03-26T18:54:46.271 INFO:tasks.workunit.client.0.smithi102.stdout:8/870: dread d0/d15/d5d/dd3/d2c/da7/f29 [4194304,4194304] 0 2024-03-26T18:54:46.272 INFO:tasks.workunit.client.0.smithi102.stdout:8/871: creat d0/d3c/d96/dd5/d120/f126 x:0 0 0 2024-03-26T18:54:46.274 INFO:tasks.workunit.client.0.smithi102.stdout:8/872: rename d0/d15/de6/fff to d0/d15/d5d/dd3/d2c/d11b/f127 0 2024-03-26T18:54:46.275 INFO:tasks.workunit.client.0.smithi102.stdout:8/873: stat d0/d15/d5d/dd3/cb2 0 2024-03-26T18:54:46.275 INFO:tasks.workunit.client.0.smithi102.stdout:8/874: dread - d0/d15/d5d/dd3/d2c/d33/ff5 zero size 2024-03-26T18:54:46.295 INFO:tasks.workunit.client.0.smithi102.stdout:4/790: dwrite d1/d17/d1a/d21/d77/fd3 [0,4194304] 0 2024-03-26T18:54:46.296 INFO:tasks.workunit.client.0.smithi102.stdout:8/875: dwrite d0/f10a [0,4194304] 0 2024-03-26T18:54:46.299 INFO:tasks.workunit.client.0.smithi102.stdout:8/876: mknod d0/d15/d5d/dd3/d26/d19/d76/c128 0 2024-03-26T18:54:46.299 INFO:tasks.workunit.client.0.smithi102.stdout:8/877: write d0/d15/d5d/f61 [2413204,65731] 0 2024-03-26T18:54:46.299 INFO:tasks.workunit.client.0.smithi102.stdout:8/878: rmdir d0/d15/d5d/dd3/d2c/d2f 39 2024-03-26T18:54:46.309 INFO:tasks.workunit.client.0.smithi102.stdout:8/879: dread d0/d3c/f43 [0,4194304] 0 2024-03-26T18:54:46.309 INFO:tasks.workunit.client.0.smithi102.stdout:8/880: chown d0/d15/f4b 33215 1 2024-03-26T18:54:46.310 INFO:tasks.workunit.client.0.smithi102.stdout:8/881: truncate d0/d15/d5d/dd3/d26/d5c/f62 7227428 0 2024-03-26T18:54:46.311 INFO:tasks.workunit.client.0.smithi102.stdout:8/882: unlink d0/d15/d5d/dd3/d2c/da7/f5b 0 2024-03-26T18:54:46.312 INFO:tasks.workunit.client.0.smithi102.stdout:8/883: fsync d0/d15/d5d/dd3/d2c/d33/f49 0 2024-03-26T18:54:46.316 INFO:tasks.workunit.client.0.smithi102.stdout:4/791: dwrite d1/d17/f88 [0,4194304] 0 2024-03-26T18:54:46.316 INFO:tasks.workunit.client.0.smithi102.stdout:4/792: chown d1/d27/d7a/c9f 5868 1 2024-03-26T18:54:46.316 INFO:tasks.workunit.client.0.smithi102.stdout:4/793: dread - d1/d17/d2a/d9a/ffd zero size 2024-03-26T18:54:46.317 INFO:tasks.workunit.client.0.smithi102.stdout:4/794: write d1/d17/d1f/d5d/d6f/fac [574907,129580] 0 2024-03-26T18:54:46.318 INFO:tasks.workunit.client.0.smithi102.stdout:4/795: creat d1/d17/d2a/f106 x:0 0 0 2024-03-26T18:54:46.318 INFO:tasks.workunit.client.0.smithi102.stdout:4/796: stat d1/d17/d1f/d5d/d8b/l62 0 2024-03-26T18:54:46.318 INFO:tasks.workunit.client.0.smithi102.stdout:4/797: rmdir d1/d17 39 2024-03-26T18:54:46.325 INFO:tasks.workunit.client.0.smithi102.stdout:4/798: dread d1/f26 [0,4194304] 0 2024-03-26T18:54:46.326 INFO:tasks.workunit.client.0.smithi102.stdout:4/799: symlink d1/d17/dbb/dee/l107 0 2024-03-26T18:54:46.327 INFO:tasks.workunit.client.0.smithi102.stdout:4/800: creat d1/d17/d1a/d21/d77/d102/f108 x:0 0 0 2024-03-26T18:54:46.328 INFO:tasks.workunit.client.0.smithi102.stdout:4/801: symlink d1/d75/l109 0 2024-03-26T18:54:46.330 INFO:tasks.workunit.client.0.smithi102.stdout:8/884: dwrite d0/d15/d5d/dd3/d2c/f8e [0,4194304] 0 2024-03-26T18:54:46.338 INFO:tasks.workunit.client.0.smithi102.stdout:4/802: dread d1/d27/d98/f6a [0,4194304] 0 2024-03-26T18:54:46.339 INFO:tasks.workunit.client.0.smithi102.stdout:4/803: symlink d1/d75/l10a 0 2024-03-26T18:54:46.341 INFO:tasks.workunit.client.0.smithi102.stdout:4/804: rename d1/d17/d1a/d21/cb8 to d1/d17/d2a/d9a/dc2/c10b 0 2024-03-26T18:54:46.341 INFO:tasks.workunit.client.0.smithi102.stdout:4/805: fsync d1/fd5 0 2024-03-26T18:54:46.342 INFO:tasks.workunit.client.0.smithi102.stdout:8/885: dread d0/d15/d5d/dd3/d2c/f8e [0,4194304] 0 2024-03-26T18:54:46.342 INFO:tasks.workunit.client.0.smithi102.stdout:8/886: stat d0/d15/da0/d10c 0 2024-03-26T18:54:46.344 INFO:tasks.workunit.client.0.smithi102.stdout:8/887: link d0/d15/f6b d0/d15/d5d/dd3/d26/f129 0 2024-03-26T18:54:46.344 INFO:tasks.workunit.client.0.smithi102.stdout:8/888: chown d0/d15/d5d/dd3/d26/d19/d76/d113 6 1 2024-03-26T18:54:46.345 INFO:tasks.workunit.client.0.smithi102.stdout:4/806: rename d1/d6c to d1/d17/dbb/dee/d10c 0 2024-03-26T18:54:46.354 INFO:tasks.workunit.client.0.smithi102.stdout:8/889: dread d0/d3c/d96/d98/db6/f3a [0,4194304] 0 2024-03-26T18:54:46.354 INFO:tasks.workunit.client.0.smithi102.stdout:8/890: write d0/d9/f10f [291727,38833] 0 2024-03-26T18:54:46.355 INFO:tasks.workunit.client.0.smithi102.stdout:8/891: mkdir d0/d3c/d96/d98/d12a 0 2024-03-26T18:54:46.356 INFO:tasks.workunit.client.0.smithi102.stdout:8/892: symlink d0/d3c/df9/l12b 0 2024-03-26T18:54:46.356 INFO:tasks.workunit.client.0.smithi102.stdout:8/893: chown d0/d15/f115 1036930 1 2024-03-26T18:54:46.358 INFO:tasks.workunit.client.0.smithi102.stdout:8/894: link d0/d15/f4b d0/d15/de6/f12c 0 2024-03-26T18:54:46.364 INFO:tasks.workunit.client.0.smithi102.stdout:8/895: dread d0/d15/d5d/dd3/d2c/d33/d87/f9e [0,4194304] 0 2024-03-26T18:54:46.366 INFO:tasks.workunit.client.0.smithi102.stdout:8/896: unlink d0/d15/d5d/dd3/d2c/d33/f49 0 2024-03-26T18:54:46.367 INFO:tasks.workunit.client.0.smithi102.stdout:8/897: creat d0/d3c/d96/d98/dcf/f12d x:0 0 0 2024-03-26T18:54:46.368 INFO:tasks.workunit.client.0.smithi102.stdout:4/807: dwrite d1/d27/f6e [0,4194304] 0 2024-03-26T18:54:46.368 INFO:tasks.workunit.client.0.smithi102.stdout:4/808: chown d1/d25/d90 34522 1 2024-03-26T18:54:46.368 INFO:tasks.workunit.client.0.smithi102.stdout:4/809: chown d1/d17/d1f/d5d/d6f/l84 1017 1 2024-03-26T18:54:46.369 INFO:tasks.workunit.client.0.smithi102.stdout:4/810: chown d1/d75/d85/cd2 24 1 2024-03-26T18:54:46.374 INFO:tasks.workunit.client.0.smithi102.stdout:4/811: symlink d1/d75/l10d 0 2024-03-26T18:54:46.374 INFO:tasks.workunit.client.0.smithi102.stdout:4/812: dread - d1/fa8 zero size 2024-03-26T18:54:46.374 INFO:tasks.workunit.client.0.smithi102.stdout:8/898: truncate d0/d15/d5d/dd3/d2c/d2f/fa5 2765723 0 2024-03-26T18:54:46.376 INFO:tasks.workunit.client.0.smithi102.stdout:4/813: mkdir d1/d27/d98/dea/d10e 0 2024-03-26T18:54:46.381 INFO:tasks.workunit.client.0.smithi102.stdout:8/899: write d0/d15/d5d/dd3/d26/f1f [498607,55876] 0 2024-03-26T18:54:46.381 INFO:tasks.workunit.client.0.smithi102.stdout:8/900: readlink d0/d15/d5d/dd3/d26/d19/d76/l110 0 2024-03-26T18:54:46.382 INFO:tasks.workunit.client.0.smithi102.stdout:8/901: truncate d0/d15/da0/d10c/fab 2110652 0 2024-03-26T18:54:46.386 INFO:tasks.workunit.client.0.smithi102.stdout:4/814: dread d1/f8 [0,4194304] 0 2024-03-26T18:54:46.386 INFO:tasks.workunit.client.0.smithi102.stdout:4/815: creat d1/d75/d85/f10f x:0 0 0 2024-03-26T18:54:46.390 INFO:tasks.workunit.client.0.smithi102.stdout:8/902: dread d0/d15/da0/d10c/f95 [0,4194304] 0 2024-03-26T18:54:46.392 INFO:tasks.workunit.client.0.smithi102.stdout:8/903: creat d0/d15/d5d/dd3/d26/d5c/f12e x:0 0 0 2024-03-26T18:54:46.392 INFO:tasks.workunit.client.0.smithi102.stdout:8/904: fdatasync d0/d3c/d96/dd5/d120/f126 0 2024-03-26T18:54:46.394 INFO:tasks.workunit.client.0.smithi102.stdout:4/816: write d1/d75/d85/fbe [2774382,126858] 0 2024-03-26T18:54:46.395 INFO:tasks.workunit.client.0.smithi102.stdout:4/817: dread - d1/d17/d2a/d9a/dc2/f105 zero size 2024-03-26T18:54:46.395 INFO:tasks.workunit.client.0.smithi102.stdout:4/818: chown d1/d17/d1a/d21/c7e 3 1 2024-03-26T18:54:46.397 INFO:tasks.workunit.client.0.smithi102.stdout:4/819: creat d1/d17/f110 x:0 0 0 2024-03-26T18:54:46.399 INFO:tasks.workunit.client.0.smithi102.stdout:4/820: rename d1/d17/d1a/f4f to d1/d17/dbb/f111 0 2024-03-26T18:54:46.399 INFO:tasks.workunit.client.0.smithi102.stdout:8/905: dread d0/d15/d5d/dd3/d26/f129 [0,4194304] 0 2024-03-26T18:54:46.408 INFO:tasks.workunit.client.0.smithi102.stdout:2/828: sync 2024-03-26T18:54:46.408 INFO:tasks.workunit.client.0.smithi102.stdout:0/856: sync 2024-03-26T18:54:46.410 INFO:tasks.workunit.client.0.smithi102.stdout:0/857: rename d6/de/d13/d25/d31/fd7 to d6/d3e/d104/f128 0 2024-03-26T18:54:46.410 INFO:tasks.workunit.client.0.smithi102.stdout:0/858: fdatasync d6/de/d13/d25/d4f/d66/f6e 0 2024-03-26T18:54:46.413 INFO:tasks.workunit.client.0.smithi102.stdout:8/906: write d0/d15/d5d/dd3/d2c/f85 [1262167,3221] 0 2024-03-26T18:54:46.414 INFO:tasks.workunit.client.0.smithi102.stdout:8/907: creat d0/d15/d5d/dd3/d2c/f12f x:0 0 0 2024-03-26T18:54:46.414 INFO:tasks.workunit.client.0.smithi102.stdout:8/908: fdatasync d0/d15/da0/d10c/fbd 0 2024-03-26T18:54:46.415 INFO:tasks.workunit.client.0.smithi102.stdout:8/909: dread - d0/d15/da0/d10c/fbd zero size 2024-03-26T18:54:46.416 INFO:tasks.workunit.client.0.smithi102.stdout:8/910: rename d0/d15/d5d/dd3/d2c/f42 to d0/d15/d1b/f130 0 2024-03-26T18:54:46.418 INFO:tasks.workunit.client.0.smithi102.stdout:0/859: dread d6/de/d13/d1f/d55/d57/f30 [0,4194304] 0 2024-03-26T18:54:46.419 INFO:tasks.workunit.client.0.smithi102.stdout:4/821: dwrite d1/d17/d1a/fdf [0,4194304] 0 2024-03-26T18:54:46.420 INFO:tasks.workunit.client.0.smithi102.stdout:0/860: creat d6/de/d93/f129 x:0 0 0 2024-03-26T18:54:46.420 INFO:tasks.workunit.client.0.smithi102.stdout:0/861: readlink d6/de/d13/d25/da4/lb3 0 2024-03-26T18:54:46.423 INFO:tasks.workunit.client.0.smithi102.stdout:4/822: mkdir d1/d17/d2a/d9a/d112 0 2024-03-26T18:54:46.423 INFO:tasks.workunit.client.0.smithi102.stdout:4/823: stat d1/l1e 0 2024-03-26T18:54:46.423 INFO:tasks.workunit.client.0.smithi102.stdout:2/829: dwrite d3/d20/d78/d7c/f85 [4194304,4194304] 0 2024-03-26T18:54:46.424 INFO:tasks.workunit.client.0.smithi102.stdout:0/862: mknod d6/de/d13/d25/d31/d45/d82/c12a 0 2024-03-26T18:54:46.424 INFO:tasks.workunit.client.0.smithi102.stdout:0/863: chown d6/d2d/d3d/d80/d83 0 1 2024-03-26T18:54:46.426 INFO:tasks.workunit.client.0.smithi102.stdout:2/830: symlink d3/d74/l12b 0 2024-03-26T18:54:46.427 INFO:tasks.workunit.client.0.smithi102.stdout:2/831: fsync d3/d11/d15/ded/ffc 0 2024-03-26T18:54:46.427 INFO:tasks.workunit.client.0.smithi102.stdout:2/832: truncate d3/d2a/ff7 2459217 0 2024-03-26T18:54:46.427 INFO:tasks.workunit.client.0.smithi102.stdout:2/833: chown d3/d2a/f6d 8773 1 2024-03-26T18:54:46.427 INFO:tasks.workunit.client.0.smithi102.stdout:2/834: truncate d3/d11/d15/d31/d39/f63 865225 0 2024-03-26T18:54:46.428 INFO:tasks.workunit.client.0.smithi102.stdout:7/776: sync 2024-03-26T18:54:46.428 INFO:tasks.workunit.client.0.smithi102.stdout:9/760: sync 2024-03-26T18:54:46.428 INFO:tasks.workunit.client.0.smithi102.stdout:6/854: sync 2024-03-26T18:54:46.429 INFO:tasks.workunit.client.0.smithi102.stdout:7/777: write d7/d10/d67/d69/d9b/dae/fe6 [574849,99619] 0 2024-03-26T18:54:46.429 INFO:tasks.workunit.client.0.smithi102.stdout:1/838: sync 2024-03-26T18:54:46.431 INFO:tasks.workunit.client.0.smithi102.stdout:6/855: dread d2/d7/d3c/f41 [0,4194304] 0 2024-03-26T18:54:46.431 INFO:tasks.workunit.client.0.smithi102.stdout:6/856: write d2/df/d35/d43/fd1 [587762,39475] 0 2024-03-26T18:54:46.434 INFO:tasks.workunit.client.0.smithi102.stdout:6/857: mknod d2/df/d35/d43/c11d 0 2024-03-26T18:54:46.434 INFO:tasks.workunit.client.0.smithi102.stdout:6/858: chown d2/df/d71/de6 130 1 2024-03-26T18:54:46.434 INFO:tasks.workunit.client.0.smithi102.stdout:6/859: stat d2/df/d72/f108 0 2024-03-26T18:54:46.435 INFO:tasks.workunit.client.0.smithi102.stdout:4/824: dread d1/d17/d1f/fd9 [0,4194304] 0 2024-03-26T18:54:46.437 INFO:tasks.workunit.client.0.smithi102.stdout:6/860: mknod d2/df/d71/d80/d91/c11e 0 2024-03-26T18:54:46.437 INFO:tasks.workunit.client.0.smithi102.stdout:6/861: fdatasync d2/df/d35/f6e 0 2024-03-26T18:54:46.459 INFO:tasks.workunit.client.0.smithi102.stdout:0/864: dwrite d6/de/d13/d25/da4/fee [0,4194304] 0 2024-03-26T18:54:46.459 INFO:tasks.workunit.client.0.smithi102.stdout:0/865: chown d6/dbe/deb/cf6 0 1 2024-03-26T18:54:46.459 INFO:tasks.workunit.client.0.smithi102.stdout:0/866: chown d6/d3e/d69/d8c/cc1 0 1 2024-03-26T18:54:46.459 INFO:tasks.workunit.client.0.smithi102.stdout:0/867: fdatasync d6/d3e/d104/d11a/ff7 0 2024-03-26T18:54:46.459 INFO:tasks.workunit.client.0.smithi102.stdout:0/868: fdatasync d6/d3e/d69/d8c/fec 0 2024-03-26T18:54:46.461 INFO:tasks.workunit.client.0.smithi102.stdout:0/869: creat d6/d3e/d69/f12b x:0 0 0 2024-03-26T18:54:46.461 INFO:tasks.workunit.client.0.smithi102.stdout:0/870: fdatasync d6/d2d/d3d/fa0 0 2024-03-26T18:54:46.461 INFO:tasks.workunit.client.0.smithi102.stdout:0/871: truncate d6/de/d13/d9a/fcc 1865957 0 2024-03-26T18:54:46.462 INFO:tasks.workunit.client.0.smithi102.stdout:0/872: mknod d6/d2d/d3d/d80/c12c 0 2024-03-26T18:54:46.491 INFO:tasks.workunit.client.0.smithi102.stdout:7/778: dwrite d7/d10/d27/d34/d72/f76 [0,4194304] 0 2024-03-26T18:54:46.491 INFO:tasks.workunit.client.0.smithi102.stdout:7/779: chown d7/d10/d67/d69/d98 6457 1 2024-03-26T18:54:46.493 INFO:tasks.workunit.client.0.smithi102.stdout:7/780: write d7/d10/d17/f7d [84439,20735] 0 2024-03-26T18:54:46.493 INFO:tasks.workunit.client.0.smithi102.stdout:2/835: dwrite d3/d2d/d69/fbd [0,4194304] 0 2024-03-26T18:54:46.493 INFO:tasks.workunit.client.0.smithi102.stdout:6/862: dwrite d2/d3/f38 [0,4194304] 0 2024-03-26T18:54:46.494 INFO:tasks.workunit.client.0.smithi102.stdout:2/836: mkdir d3/d82/d122/d12c 0 2024-03-26T18:54:46.494 INFO:tasks.workunit.client.0.smithi102.stdout:9/761: dwrite df/d1a/d23/d4e/d8e/fce [0,4194304] 0 2024-03-26T18:54:46.495 INFO:tasks.workunit.client.0.smithi102.stdout:9/762: chown df/d1a/d23/d60/f101 39 1 2024-03-26T18:54:46.499 INFO:tasks.workunit.client.0.smithi102.stdout:4/825: dwrite d1/d17/d1a/d21/f2c [4194304,4194304] 0 2024-03-26T18:54:46.500 INFO:tasks.workunit.client.0.smithi102.stdout:0/873: dwrite d6/d2d/f10e [0,4194304] 0 2024-03-26T18:54:46.500 INFO:tasks.workunit.client.0.smithi102.stdout:2/837: rename d3/d2d/d67/l101 to d3/d2d/d67/l12d 0 2024-03-26T18:54:46.503 INFO:tasks.workunit.client.0.smithi102.stdout:2/838: getdents d3/d20/d78/d7c/dca 0 2024-03-26T18:54:46.503 INFO:tasks.workunit.client.0.smithi102.stdout:1/839: dwrite de/d16/f54 [0,4194304] 0 2024-03-26T18:54:46.503 INFO:tasks.workunit.client.0.smithi102.stdout:1/840: readlink de/d16/la9 0 2024-03-26T18:54:46.507 INFO:tasks.workunit.client.0.smithi102.stdout:2/839: creat d3/d82/f12e x:0 0 0 2024-03-26T18:54:46.507 INFO:tasks.workunit.client.0.smithi102.stdout:1/841: stat de/d16/d1d/d2a/d2c/d56/d6e/c72 0 2024-03-26T18:54:46.507 INFO:tasks.workunit.client.0.smithi102.stdout:9/763: dread df/d1a/d23/f25 [0,4194304] 0 2024-03-26T18:54:46.507 INFO:tasks.workunit.client.0.smithi102.stdout:9/764: readlink df/d1a/d23/d4e/l108 0 2024-03-26T18:54:46.507 INFO:tasks.workunit.client.0.smithi102.stdout:9/765: fsync df/d1a/d23/d29/d9c/fab 0 2024-03-26T18:54:46.508 INFO:tasks.workunit.client.0.smithi102.stdout:2/840: symlink d3/d82/d93/d9e/d99/l12f 0 2024-03-26T18:54:46.508 INFO:tasks.workunit.client.0.smithi102.stdout:2/841: write d3/d2d/d69/fbd [4609196,30176] 0 2024-03-26T18:54:46.508 INFO:tasks.workunit.client.0.smithi102.stdout:2/842: chown d3/d82/d93/d9e/d1a/c115 3 1 2024-03-26T18:54:46.509 INFO:tasks.workunit.client.0.smithi102.stdout:1/842: truncate de/f14 211356 0 2024-03-26T18:54:46.510 INFO:tasks.workunit.client.0.smithi102.stdout:9/766: creat df/d1a/d23/d3e/d75/d110/f119 x:0 0 0 2024-03-26T18:54:46.511 INFO:tasks.workunit.client.0.smithi102.stdout:2/843: rmdir d3/d20/d78 39 2024-03-26T18:54:46.513 INFO:tasks.workunit.client.0.smithi102.stdout:1/843: rename de/d16/d1d/d2a/d2c/d56/d5f/daa/lf1 to de/d16/d1d/l11c 0 2024-03-26T18:54:46.516 INFO:tasks.workunit.client.0.smithi102.stdout:0/874: dread d6/de/f15 [0,4194304] 0 2024-03-26T18:54:46.516 INFO:tasks.workunit.client.0.smithi102.stdout:9/767: rename df/d1a/d23/d3e/f9f to df/d1a/d23/d4e/dc1/f11a 0 2024-03-26T18:54:46.516 INFO:tasks.workunit.client.0.smithi102.stdout:9/768: creat df/d1a/d65/f11b x:0 0 0 2024-03-26T18:54:46.517 INFO:tasks.workunit.client.0.smithi102.stdout:1/844: symlink de/d16/d1d/l11d 0 2024-03-26T18:54:46.522 INFO:tasks.workunit.client.0.smithi102.stdout:2/844: rename d3/d20/d78/d7c/dca/l10f to d3/d20/d78/d84/d6e/dc6/l130 0 2024-03-26T18:54:46.523 INFO:tasks.workunit.client.0.smithi102.stdout:9/769: rename df/d1a/d81/l89 to df/d1a/l11c 0 2024-03-26T18:54:46.523 INFO:tasks.workunit.client.0.smithi102.stdout:2/845: rmdir d3/d20/d78/d7c/dca 39 2024-03-26T18:54:46.523 INFO:tasks.workunit.client.0.smithi102.stdout:2/846: creat d3/d11/d15/d46/d7b/d97/d10a/f131 x:0 0 0 2024-03-26T18:54:46.524 INFO:tasks.workunit.client.0.smithi102.stdout:9/770: creat df/d1a/d23/d3e/d75/d117/f11d x:0 0 0 2024-03-26T18:54:46.525 INFO:tasks.workunit.client.0.smithi102.stdout:9/771: chown df/d1a/d23/d29/d9c/f112 727290 1 2024-03-26T18:54:46.526 INFO:tasks.workunit.client.0.smithi102.stdout:2/847: mknod d3/d11/c132 0 2024-03-26T18:54:46.526 INFO:tasks.workunit.client.0.smithi102.stdout:2/848: write d3/d11/d15/f2e [4514276,73113] 0 2024-03-26T18:54:46.537 INFO:tasks.workunit.client.0.smithi102.stdout:2/849: write d3/d82/d93/d9e/fa [3175312,60385] 0 2024-03-26T18:54:46.549 INFO:tasks.workunit.client.0.smithi102.stdout:7/781: dwrite d7/f41 [4194304,4194304] 0 2024-03-26T18:54:46.549 INFO:tasks.workunit.client.0.smithi102.stdout:7/782: creat d7/d10/d67/d69/d98/fea x:0 0 0 2024-03-26T18:54:46.550 INFO:tasks.workunit.client.0.smithi102.stdout:6/863: dwrite d2/d3d/d53/f83 [0,4194304] 0 2024-03-26T18:54:46.551 INFO:tasks.workunit.client.0.smithi102.stdout:7/783: mknod d7/d10/d27/d34/d72/ceb 0 2024-03-26T18:54:46.555 INFO:tasks.workunit.client.0.smithi102.stdout:7/784: dread d7/d10/d67/d69/d9b/dae/fe6 [0,4194304] 0 2024-03-26T18:54:46.556 INFO:tasks.workunit.client.0.smithi102.stdout:7/785: unlink d7/d12/f43 0 2024-03-26T18:54:46.563 INFO:tasks.workunit.client.0.smithi102.stdout:4/826: dwrite d1/d17/d1a/d21/d77/f81 [0,4194304] 0 2024-03-26T18:54:46.563 INFO:tasks.workunit.client.0.smithi102.stdout:4/827: fdatasync d1/d27/d98/f8e 0 2024-03-26T18:54:46.565 INFO:tasks.workunit.client.0.smithi102.stdout:2/850: dwrite d3/d2d/f68 [0,4194304] 0 2024-03-26T18:54:46.565 INFO:tasks.workunit.client.0.smithi102.stdout:0/875: dwrite d6/d3e/d87/f52 [0,4194304] 0 2024-03-26T18:54:46.565 INFO:tasks.workunit.client.0.smithi102.stdout:6/864: write d2/d3d/d53/f83 [2494947,1338] 0 2024-03-26T18:54:46.565 INFO:tasks.workunit.client.0.smithi102.stdout:1/845: dwrite de/d1f/f53 [4194304,4194304] 0 2024-03-26T18:54:46.565 INFO:tasks.workunit.client.0.smithi102.stdout:9/772: dwrite df/d46/f6a [0,4194304] 0 2024-03-26T18:54:46.565 INFO:tasks.workunit.client.0.smithi102.stdout:2/851: dread - d3/d20/fac zero size 2024-03-26T18:54:46.565 INFO:tasks.workunit.client.0.smithi102.stdout:2/852: chown d3/d11/fa6 114 1 2024-03-26T18:54:46.567 INFO:tasks.workunit.client.0.smithi102.stdout:0/876: symlink d6/d3e/d69/d8c/d8f/l12d 0 2024-03-26T18:54:46.567 INFO:tasks.workunit.client.0.smithi102.stdout:2/853: write d3/d82/d93/d9e/d1a/d25/f9d [685580,82750] 0 2024-03-26T18:54:46.570 INFO:tasks.workunit.client.0.smithi102.stdout:6/865: mknod d2/df/d71/d80/d91/c11f 0 2024-03-26T18:54:46.570 INFO:tasks.workunit.client.0.smithi102.stdout:6/866: write d2/df/dba/fdd [195921,29199] 0 2024-03-26T18:54:46.570 INFO:tasks.workunit.client.0.smithi102.stdout:6/867: write d2/d3d/d53/f83 [4219173,28336] 0 2024-03-26T18:54:46.570 INFO:tasks.workunit.client.0.smithi102.stdout:7/786: dread d7/fb [0,4194304] 0 2024-03-26T18:54:46.571 INFO:tasks.workunit.client.0.smithi102.stdout:1/846: mkdir de/d2e/d11e 0 2024-03-26T18:54:46.573 INFO:tasks.workunit.client.0.smithi102.stdout:4/828: mkdir d1/d113 0 2024-03-26T18:54:46.574 INFO:tasks.workunit.client.0.smithi102.stdout:0/877: dread d6/d3e/d104/d11a/f58 [0,4194304] 0 2024-03-26T18:54:46.575 INFO:tasks.workunit.client.0.smithi102.stdout:2/854: write d3/d20/fab [649029,123604] 0 2024-03-26T18:54:46.578 INFO:tasks.workunit.client.0.smithi102.stdout:6/868: rename d2/d7/d3c/cdf to d2/df/d72/c120 0 2024-03-26T18:54:46.578 INFO:tasks.workunit.client.0.smithi102.stdout:6/869: fsync d2/d7/f92 0 2024-03-26T18:54:46.579 INFO:tasks.workunit.client.0.smithi102.stdout:7/787: mkdir d7/d31/d56/dec 0 2024-03-26T18:54:46.580 INFO:tasks.workunit.client.0.smithi102.stdout:1/847: mknod de/c11f 0 2024-03-26T18:54:46.580 INFO:tasks.workunit.client.0.smithi102.stdout:1/848: chown de/d1f/l94 1068 1 2024-03-26T18:54:46.580 INFO:tasks.workunit.client.0.smithi102.stdout:1/849: fsync de/f2d 0 2024-03-26T18:54:46.581 INFO:tasks.workunit.client.0.smithi102.stdout:9/773: getdents df/d1a/d57 0 2024-03-26T18:54:46.582 INFO:tasks.workunit.client.0.smithi102.stdout:9/774: creat df/d46/da8/f11e x:0 0 0 2024-03-26T18:54:46.582 INFO:tasks.workunit.client.0.smithi102.stdout:9/775: readlink df/d1a/d1d/la0 0 2024-03-26T18:54:46.583 INFO:tasks.workunit.client.0.smithi102.stdout:4/829: creat d1/d17/dbb/dee/d10c/f114 x:0 0 0 2024-03-26T18:54:46.584 INFO:tasks.workunit.client.0.smithi102.stdout:2/855: read d3/d20/ff0 [2075045,9808] 0 2024-03-26T18:54:46.584 INFO:tasks.workunit.client.0.smithi102.stdout:2/856: chown d3/d11/d15/d46/d7b/d97/d10a/d123 51 1 2024-03-26T18:54:46.585 INFO:tasks.workunit.client.0.smithi102.stdout:2/857: dread d3/d82/d93/d9e/ff [0,4194304] 0 2024-03-26T18:54:46.585 INFO:tasks.workunit.client.0.smithi102.stdout:2/858: creat d3/d82/d93/d9e/f133 x:0 0 0 2024-03-26T18:54:46.591 INFO:tasks.workunit.client.0.smithi102.stdout:0/878: rename d6/de/d13/d25/d4f/c101 to d6/de/d13/d25/d4f/d66/c12e 0 2024-03-26T18:54:46.592 INFO:tasks.workunit.client.0.smithi102.stdout:6/870: stat d2/f34 0 2024-03-26T18:54:46.592 INFO:tasks.workunit.client.0.smithi102.stdout:7/788: write d7/d10/d17/f35 [457783,103624] 0 2024-03-26T18:54:46.594 INFO:tasks.workunit.client.0.smithi102.stdout:9/776: mkdir df/d1a/d23/d4e/d11f 0 2024-03-26T18:54:46.598 INFO:tasks.workunit.client.0.smithi102.stdout:4/830: mknod d1/d17/d2a/d9a/dc2/c115 0 2024-03-26T18:54:46.598 INFO:tasks.workunit.client.0.smithi102.stdout:4/831: write d1/d17/dbc/fc8 [408748,88763] 0 2024-03-26T18:54:46.599 INFO:tasks.workunit.client.0.smithi102.stdout:4/832: chown d1/d17/dbb/dee/d10c/f114 5104 1 2024-03-26T18:54:46.600 INFO:tasks.workunit.client.0.smithi102.stdout:2/859: symlink d3/d11/d15/d31/d7f/dfe/l134 0 2024-03-26T18:54:46.600 INFO:tasks.workunit.client.0.smithi102.stdout:2/860: fdatasync d3/f4 0 2024-03-26T18:54:46.601 INFO:tasks.workunit.client.0.smithi102.stdout:6/871: mknod d2/df/d71/d80/d91/db8/c121 0 2024-03-26T18:54:46.605 INFO:tasks.workunit.client.0.smithi102.stdout:9/777: creat df/d1a/d81/dcc/f120 x:0 0 0 2024-03-26T18:54:46.606 INFO:tasks.workunit.client.0.smithi102.stdout:9/778: write df/d46/da8/f105 [418994,90585] 0 2024-03-26T18:54:46.606 INFO:tasks.workunit.client.0.smithi102.stdout:1/850: dwrite de/d16/d1d/d2a/d67/f92 [0,4194304] 0 2024-03-26T18:54:46.606 INFO:tasks.workunit.client.0.smithi102.stdout:4/833: symlink d1/d25/d90/l116 0 2024-03-26T18:54:46.608 INFO:tasks.workunit.client.0.smithi102.stdout:9/779: mkdir df/d1a/d1d/d72/d121 0 2024-03-26T18:54:46.613 INFO:tasks.workunit.client.0.smithi102.stdout:1/851: truncate de/d16/d1d/f4d 2696211 0 2024-03-26T18:54:46.615 INFO:tasks.workunit.client.0.smithi102.stdout:9/780: unlink df/d1a/d23/d29/f4c 0 2024-03-26T18:54:46.615 INFO:tasks.workunit.client.0.smithi102.stdout:9/781: write df/f30 [2434857,63607] 0 2024-03-26T18:54:46.615 INFO:tasks.workunit.client.0.smithi102.stdout:9/782: fdatasync df/d1a/d65/f73 0 2024-03-26T18:54:46.616 INFO:tasks.workunit.client.0.smithi102.stdout:9/783: mknod df/d1a/d23/d29/d9c/c122 0 2024-03-26T18:54:46.622 INFO:tasks.workunit.client.0.smithi102.stdout:7/789: dwrite d7/d10/d67/d69/d78/fba [0,4194304] 0 2024-03-26T18:54:46.624 INFO:tasks.workunit.client.0.smithi102.stdout:9/784: dread df/d1a/d1d/f26 [0,4194304] 0 2024-03-26T18:54:46.628 INFO:tasks.workunit.client.0.smithi102.stdout:0/879: dwrite d6/de/d13/d25/d31/f50 [0,4194304] 0 2024-03-26T18:54:46.630 INFO:tasks.workunit.client.0.smithi102.stdout:0/880: creat d6/de/d13/d25/d31/d45/d82/d84/f12f x:0 0 0 2024-03-26T18:54:46.630 INFO:tasks.workunit.client.0.smithi102.stdout:0/881: creat d6/de/d13/d108/f130 x:0 0 0 2024-03-26T18:54:46.634 INFO:tasks.workunit.client.0.smithi102.stdout:0/882: link d6/de/d13/d1f/fd9 d6/de/d13/d25/d4f/f131 0 2024-03-26T18:54:46.634 INFO:tasks.workunit.client.0.smithi102.stdout:0/883: read - d6/d3e/d69/d8c/fec zero size 2024-03-26T18:54:46.636 INFO:tasks.workunit.client.0.smithi102.stdout:1/852: write de/d16/d1d/d2a/d2c/d56/d6e/dfc/d110/ffb [1308619,26515] 0 2024-03-26T18:54:46.645 INFO:tasks.workunit.client.0.smithi102.stdout:9/785: write df/d1a/d23/f5d [2608873,102202] 0 2024-03-26T18:54:46.649 INFO:tasks.workunit.client.0.smithi102.stdout:9/786: dread df/d1a/d23/d3e/f99 [0,4194304] 0 2024-03-26T18:54:46.649 INFO:tasks.workunit.client.0.smithi102.stdout:9/787: readlink df/d1a/d23/d29/lc2 0 2024-03-26T18:54:46.650 INFO:tasks.workunit.client.0.smithi102.stdout:9/788: mknod df/d1a/d23/d29/c123 0 2024-03-26T18:54:46.650 INFO:tasks.workunit.client.0.smithi102.stdout:9/789: fdatasync df/d1a/d81/d88/fc9 0 2024-03-26T18:54:46.650 INFO:tasks.workunit.client.0.smithi102.stdout:9/790: symlink df/d1a/d65/l124 0 2024-03-26T18:54:46.652 INFO:tasks.workunit.client.0.smithi102.stdout:9/791: getdents df/d1a 0 2024-03-26T18:54:46.653 INFO:tasks.workunit.client.0.smithi102.stdout:9/792: creat df/d1a/d1d/d72/d7d/db7/df8/f125 x:0 0 0 2024-03-26T18:54:46.654 INFO:tasks.workunit.client.0.smithi102.stdout:2/861: dwrite d3/d11/d15/f11c [0,4194304] 0 2024-03-26T18:54:46.656 INFO:tasks.workunit.client.0.smithi102.stdout:2/862: link d3/d2a/f4f d3/f135 0 2024-03-26T18:54:46.659 INFO:tasks.workunit.client.0.smithi102.stdout:2/863: rmdir d3/d11/d27/dc0/d12a 0 2024-03-26T18:54:46.659 INFO:tasks.workunit.client.0.smithi102.stdout:2/864: fsync d3/d11/d15/d31/d39/f117 0 2024-03-26T18:54:46.660 INFO:tasks.workunit.client.0.smithi102.stdout:2/865: truncate d3/d2a/ff7 1588842 0 2024-03-26T18:54:46.660 INFO:tasks.workunit.client.0.smithi102.stdout:2/866: truncate d3/d11/d15/d46/d7b/fe6 574039 0 2024-03-26T18:54:46.662 INFO:tasks.workunit.client.0.smithi102.stdout:2/867: rmdir d3/d11/d15/d31/d7f 39 2024-03-26T18:54:46.662 INFO:tasks.workunit.client.0.smithi102.stdout:2/868: fsync d3/d82/d93/d9e/d1a/d25/fcd 0 2024-03-26T18:54:46.663 INFO:tasks.workunit.client.0.smithi102.stdout:2/869: creat d3/d82/d122/d12c/f136 x:0 0 0 2024-03-26T18:54:46.666 INFO:tasks.workunit.client.0.smithi102.stdout:2/870: symlink d3/d20/d78/d84/l137 0 2024-03-26T18:54:46.668 INFO:tasks.workunit.client.0.smithi102.stdout:2/871: unlink d3/d74/c112 0 2024-03-26T18:54:46.678 INFO:tasks.workunit.client.0.smithi102.stdout:8/911: sync 2024-03-26T18:54:46.678 INFO:tasks.workunit.client.0.smithi102.stdout:4/834: dwrite d1/f26 [4194304,4194304] 0 2024-03-26T18:54:46.678 INFO:tasks.workunit.client.0.smithi102.stdout:4/835: stat d1/d27/d98 0 2024-03-26T18:54:46.679 INFO:tasks.workunit.client.0.smithi102.stdout:4/836: creat d1/d75/f117 x:0 0 0 2024-03-26T18:54:46.681 INFO:tasks.workunit.client.0.smithi102.stdout:0/884: dwrite d6/de/d93/f129 [0,4194304] 0 2024-03-26T18:54:46.681 INFO:tasks.workunit.client.0.smithi102.stdout:6/872: dwrite d2/d12/f3f [0,4194304] 0 2024-03-26T18:54:46.684 INFO:tasks.workunit.client.0.smithi102.stdout:2/872: dread d3/d2a/d5f/fa8 [0,4194304] 0 2024-03-26T18:54:46.684 INFO:tasks.workunit.client.0.smithi102.stdout:7/790: dwrite d7/d10/d67/fa5 [0,4194304] 0 2024-03-26T18:54:46.685 INFO:tasks.workunit.client.0.smithi102.stdout:7/791: fsync d7/d31/d79/fb4 0 2024-03-26T18:54:46.685 INFO:tasks.workunit.client.0.smithi102.stdout:7/792: creat d7/d44/fed x:0 0 0 2024-03-26T18:54:46.685 INFO:tasks.workunit.client.0.smithi102.stdout:7/793: fsync d7/d10/d67/d69/d9b/dae/fe6 0 2024-03-26T18:54:46.688 INFO:tasks.workunit.client.0.smithi102.stdout:1/853: dwrite de/d16/d1d/d2a/d2c/d56/d78/d80/f98 [0,4194304] 0 2024-03-26T18:54:46.705 INFO:tasks.workunit.client.0.smithi102.stdout:8/912: dwrite d0/fbf [0,4194304] 0 2024-03-26T18:54:46.706 INFO:tasks.workunit.client.0.smithi102.stdout:0/885: dwrite d6/de/d13/d25/d31/d45/d82/d84/d88/fa6 [0,4194304] 0 2024-03-26T18:54:46.708 INFO:tasks.workunit.client.0.smithi102.stdout:8/913: dread d0/d15/f6b [0,4194304] 0 2024-03-26T18:54:47.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:46 smithi190 ceph-mon[35043]: mgrmap e20: smithi102.rsfpnv(active, since 7m), standbys: smithi190.zrixos 2024-03-26T18:54:47.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:46 smithi190 ceph-mon[35043]: pgmap v251: 65 pgs: 65 active+clean; 3.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 319 MiB/s rd, 406 MiB/s wr, 518 op/s 2024-03-26T18:54:47.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:46 smithi102 ceph-mon[29002]: mgrmap e20: smithi102.rsfpnv(active, since 7m), standbys: smithi190.zrixos 2024-03-26T18:54:47.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:46 smithi102 ceph-mon[29002]: pgmap v251: 65 pgs: 65 active+clean; 3.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 319 MiB/s rd, 406 MiB/s wr, 518 op/s 2024-03-26T18:54:47.460 INFO:tasks.workunit.client.0.smithi102.stdout:4/837: creat d1/d17/d1f/d5d/f118 x:0 0 0 2024-03-26T18:54:47.461 INFO:tasks.workunit.client.0.smithi102.stdout:4/838: chown d1/d27/d98/fb9 12 1 2024-03-26T18:54:47.461 INFO:tasks.workunit.client.0.smithi102.stdout:4/839: readlink d1/d17/d1f/d5d/d6f/dc0/lf7 0 2024-03-26T18:54:47.463 INFO:tasks.workunit.client.0.smithi102.stdout:6/873: rmdir d2/d3d/d53/d63 39 2024-03-26T18:54:47.463 INFO:tasks.workunit.client.0.smithi102.stdout:6/874: chown d2/d3d/d53/d77/f88 3306 1 2024-03-26T18:54:47.466 INFO:tasks.workunit.client.0.smithi102.stdout:2/873: write d3/d20/f80 [830815,113268] 0 2024-03-26T18:54:47.469 INFO:tasks.workunit.client.0.smithi102.stdout:7/794: unlink d7/d31/l8a 0 2024-03-26T18:54:47.469 INFO:tasks.workunit.client.0.smithi102.stdout:7/795: fsync d7/d10/d67/d69/d98/fc0 0 2024-03-26T18:54:47.469 INFO:tasks.workunit.client.0.smithi102.stdout:7/796: write d7/d10/d67/d69/d98/fdf [399566,81620] 0 2024-03-26T18:54:47.471 INFO:tasks.workunit.client.0.smithi102.stdout:1/854: mkdir de/d1f/de3/da6/dca/d120 0 2024-03-26T18:54:47.477 INFO:tasks.workunit.client.0.smithi102.stdout:0/886: symlink d6/de/d13/d25/d4f/l132 0 2024-03-26T18:54:47.479 INFO:tasks.workunit.client.0.smithi102.stdout:4/840: creat d1/d17/d1f/d5d/d8b/d3e/d74/f119 x:0 0 0 2024-03-26T18:54:47.480 INFO:tasks.workunit.client.0.smithi102.stdout:6/875: link d2/df/d72/db7/dbd/cc0 d2/df/d35/d43/d75/c122 0 2024-03-26T18:54:47.483 INFO:tasks.workunit.client.0.smithi102.stdout:7/797: mkdir d7/d10/d27/d34/d72/dee 0 2024-03-26T18:54:47.485 INFO:tasks.workunit.client.0.smithi102.stdout:1/855: rename de/d16/d1d/d2a/d2c/d56/d78/d80/ca4 to de/d16/d1d/d2a/d2c/d43/c121 0 2024-03-26T18:54:47.488 INFO:tasks.workunit.client.0.smithi102.stdout:0/887: getdents d6/de/d13/d25/d31/d45/d82/d84/d88/db2 0 2024-03-26T18:54:47.490 INFO:tasks.workunit.client.0.smithi102.stdout:4/841: symlink d1/d17/dbc/l11a 0 2024-03-26T18:54:47.490 INFO:tasks.workunit.client.0.smithi102.stdout:4/842: chown d1/d17/d1f/d5d/d8b/d3e/fff 236 1 2024-03-26T18:54:47.490 INFO:tasks.workunit.client.0.smithi102.stdout:4/843: chown d1/d17/d1f/d5d/d6f/dc0/lf7 63356768 1 2024-03-26T18:54:47.491 INFO:tasks.workunit.client.0.smithi102.stdout:4/844: dread - d1/d17/dbb/fd4 zero size 2024-03-26T18:54:47.491 INFO:tasks.workunit.client.0.smithi102.stdout:4/845: write d1/d17/d1a/d21/d77/d102/f108 [952606,104960] 0 2024-03-26T18:54:47.491 INFO:tasks.workunit.client.0.smithi102.stdout:8/914: sync 2024-03-26T18:54:47.491 INFO:tasks.workunit.client.0.smithi102.stdout:9/793: sync 2024-03-26T18:54:47.491 INFO:tasks.workunit.client.0.smithi102.stdout:7/798: symlink d7/d12/d4b/lef 0 2024-03-26T18:54:47.491 INFO:tasks.workunit.client.0.smithi102.stdout:7/799: write d7/d12/f3c [1294620,20652] 0 2024-03-26T18:54:47.492 INFO:tasks.workunit.client.0.smithi102.stdout:7/800: truncate d7/d10/d67/d69/d9b/fe0 1797 0 2024-03-26T18:54:47.494 INFO:tasks.workunit.client.0.smithi102.stdout:1/856: rmdir de/d16/d1d/d2a/d2c/d56/d6e/dfc/d117 0 2024-03-26T18:54:47.495 INFO:tasks.workunit.client.0.smithi102.stdout:0/888: mknod d6/de/d93/de4/c133 0 2024-03-26T18:54:47.496 INFO:tasks.workunit.client.0.smithi102.stdout:4/846: mkdir d1/d25/d90/d11b 0 2024-03-26T18:54:47.496 INFO:tasks.workunit.client.0.smithi102.stdout:8/915: truncate d0/d15/d5d/dd3/d2c/da7/f73 4343371 0 2024-03-26T18:54:47.498 INFO:tasks.workunit.client.0.smithi102.stdout:0/889: symlink d6/de/d13/d1f/d71/d125/l134 0 2024-03-26T18:54:47.498 INFO:tasks.workunit.client.0.smithi102.stdout:0/890: fdatasync d6/d3e/d69/d8c/fec 0 2024-03-26T18:54:47.498 INFO:tasks.workunit.client.0.smithi102.stdout:0/891: read - d6/de/d13/d25/d31/d45/d82/d84/d88/db2/dd2/f111 zero size 2024-03-26T18:54:47.498 INFO:tasks.workunit.client.0.smithi102.stdout:0/892: readlink d6/d2d/l6d 0 2024-03-26T18:54:47.498 INFO:tasks.workunit.client.0.smithi102.stdout:0/893: write d6/de/d13/d1f/d71/ffa [632153,90291] 0 2024-03-26T18:54:47.498 INFO:tasks.workunit.client.0.smithi102.stdout:0/894: chown d6/de/d13/d1f/d55/d57/d7d/d7f/fe2 39588730 1 2024-03-26T18:54:47.499 INFO:tasks.workunit.client.0.smithi102.stdout:4/847: mknod d1/d27/d98/c11c 0 2024-03-26T18:54:47.499 INFO:tasks.workunit.client.0.smithi102.stdout:8/916: truncate d0/d15/d5d/dd3/d2c/d2f/f60 1238323 0 2024-03-26T18:54:47.500 INFO:tasks.workunit.client.0.smithi102.stdout:7/801: getdents d7/d31/d79 0 2024-03-26T18:54:47.501 INFO:tasks.workunit.client.0.smithi102.stdout:0/895: symlink d6/de/d13/d25/d31/daa/l135 0 2024-03-26T18:54:47.502 INFO:tasks.workunit.client.0.smithi102.stdout:6/876: dwrite d2/df/d72/db7/dbd/fc9 [0,4194304] 0 2024-03-26T18:54:47.504 INFO:tasks.workunit.client.0.smithi102.stdout:7/802: creat d7/d12/d51/dd0/ff0 x:0 0 0 2024-03-26T18:54:47.505 INFO:tasks.workunit.client.0.smithi102.stdout:4/848: mknod d1/d17/d1f/d5d/d8b/c11d 0 2024-03-26T18:54:47.507 INFO:tasks.workunit.client.0.smithi102.stdout:6/877: creat d2/da4/df1/f123 x:0 0 0 2024-03-26T18:54:47.509 INFO:tasks.workunit.client.0.smithi102.stdout:4/849: unlink d1/d75/feb 0 2024-03-26T18:54:47.513 INFO:tasks.workunit.client.0.smithi102.stdout:4/850: getdents d1/d17/d1f/d5d/d8b/d3e 0 2024-03-26T18:54:47.513 INFO:tasks.workunit.client.0.smithi102.stdout:4/851: stat d1/d17/d2a/c2e 0 2024-03-26T18:54:47.514 INFO:tasks.workunit.client.0.smithi102.stdout:9/794: dwrite df/d1a/d65/f93 [0,4194304] 0 2024-03-26T18:54:47.514 INFO:tasks.workunit.client.0.smithi102.stdout:7/803: dread d7/d10/d17/f7d [0,4194304] 0 2024-03-26T18:54:47.514 INFO:tasks.workunit.client.0.smithi102.stdout:9/795: truncate df/d46/da8/fbb 5163411 0 2024-03-26T18:54:47.514 INFO:tasks.workunit.client.0.smithi102.stdout:1/857: write de/d16/d1d/d2a/d67/f92 [1459555,120777] 0 2024-03-26T18:54:47.515 INFO:tasks.workunit.client.0.smithi102.stdout:1/858: write de/d16/d1d/d2a/d2c/d43/f4f [4772967,94814] 0 2024-03-26T18:54:47.517 INFO:tasks.workunit.client.0.smithi102.stdout:6/878: write d2/df/d72/db7/dbd/fc9 [70826,108301] 0 2024-03-26T18:54:47.520 INFO:tasks.workunit.client.0.smithi102.stdout:9/796: creat df/d1a/d1d/d72/f126 x:0 0 0 2024-03-26T18:54:47.521 INFO:tasks.workunit.client.0.smithi102.stdout:9/797: write df/d1a/d1d/d72/d7d/fb9 [719919,101590] 0 2024-03-26T18:54:47.523 INFO:tasks.workunit.client.0.smithi102.stdout:7/804: dread d7/d2a/f60 [0,4194304] 0 2024-03-26T18:54:47.524 INFO:tasks.workunit.client.0.smithi102.stdout:1/859: symlink de/d16/d1d/d2a/d2c/d56/d78/dc3/l122 0 2024-03-26T18:54:47.524 INFO:tasks.workunit.client.0.smithi102.stdout:6/879: symlink d2/d3d/d53/d5c/dc5/l124 0 2024-03-26T18:54:47.524 INFO:tasks.workunit.client.0.smithi102.stdout:4/852: dread d1/d17/d1a/f57 [0,4194304] 0 2024-03-26T18:54:47.525 INFO:tasks.workunit.client.0.smithi102.stdout:9/798: symlink df/d1a/l127 0 2024-03-26T18:54:47.528 INFO:tasks.workunit.client.0.smithi102.stdout:1/860: unlink de/d16/d1d/d61/fee 0 2024-03-26T18:54:47.530 INFO:tasks.workunit.client.0.smithi102.stdout:1/861: mkdir de/d2e/d123 0 2024-03-26T18:54:47.530 INFO:tasks.workunit.client.0.smithi102.stdout:8/917: dwrite d0/d15/d5d/dd3/d26/d19/d76/f125 [0,4194304] 0 2024-03-26T18:54:47.530 INFO:tasks.workunit.client.0.smithi102.stdout:8/918: chown d0/d15/d5d/dd3/d2c/da7 61629821 1 2024-03-26T18:54:47.536 INFO:tasks.workunit.client.0.smithi102.stdout:8/919: truncate d0/f10a 3806777 0 2024-03-26T18:54:47.540 INFO:tasks.workunit.client.0.smithi102.stdout:8/920: dread d0/d3c/d96/d98/db6/f59 [4194304,4194304] 0 2024-03-26T18:54:47.540 INFO:tasks.workunit.client.0.smithi102.stdout:4/853: dread d1/d17/d1f/d5d/d8b/d3e/fa4 [4194304,4194304] 0 2024-03-26T18:54:47.540 INFO:tasks.workunit.client.0.smithi102.stdout:4/854: write d1/d17/d1f/d5d/f118 [260754,79696] 0 2024-03-26T18:54:47.541 INFO:tasks.workunit.client.0.smithi102.stdout:9/799: read df/d1a/d1d/d72/f90 [971778,22508] 0 2024-03-26T18:54:47.541 INFO:tasks.workunit.client.0.smithi102.stdout:0/896: dwrite d6/de/d13/d25/da4/d10b/f120 [0,4194304] 0 2024-03-26T18:54:47.543 INFO:tasks.workunit.client.0.smithi102.stdout:4/855: mkdir d1/d27/d98/db7/d11e 0 2024-03-26T18:54:47.545 INFO:tasks.workunit.client.0.smithi102.stdout:0/897: unlink d6/de/d13/d25/d31/fb5 0 2024-03-26T18:54:47.547 INFO:tasks.workunit.client.0.smithi102.stdout:8/921: write d0/d15/d5d/dd3/f11 [2355217,75018] 0 2024-03-26T18:54:47.548 INFO:tasks.workunit.client.0.smithi102.stdout:4/856: creat d1/d17/dbc/f11f x:0 0 0 2024-03-26T18:54:47.552 INFO:tasks.workunit.client.0.smithi102.stdout:0/898: unlink d6/de/d13/d9a/cdb 0 2024-03-26T18:54:47.552 INFO:tasks.workunit.client.0.smithi102.stdout:8/922: mknod d0/d15/d5d/dd3/d2c/d2f/c131 0 2024-03-26T18:54:47.553 INFO:tasks.workunit.client.0.smithi102.stdout:8/923: fsync d0/d15/d5d/dd3/d26/d5c/f62 0 2024-03-26T18:54:47.553 INFO:tasks.workunit.client.0.smithi102.stdout:8/924: write d0/d15/da0/f119 [571937,109329] 0 2024-03-26T18:54:47.554 INFO:tasks.workunit.client.0.smithi102.stdout:4/857: mkdir d1/de1/d120 0 2024-03-26T18:54:47.554 INFO:tasks.workunit.client.0.smithi102.stdout:4/858: chown d1/d17/d1a/d21/f56 0 1 2024-03-26T18:54:47.555 INFO:tasks.workunit.client.0.smithi102.stdout:0/899: dread d6/d3e/d104/d11a/f58 [0,4194304] 0 2024-03-26T18:54:47.556 INFO:tasks.workunit.client.0.smithi102.stdout:8/925: symlink d0/d15/d5d/dd3/d2c/d33/d87/d10b/l132 0 2024-03-26T18:54:47.559 INFO:tasks.workunit.client.0.smithi102.stdout:8/926: creat d0/d3c/f133 x:0 0 0 2024-03-26T18:54:47.560 INFO:tasks.workunit.client.0.smithi102.stdout:8/927: write d0/d15/d9a/f111 [182923,28410] 0 2024-03-26T18:54:47.560 INFO:tasks.workunit.client.0.smithi102.stdout:4/859: getdents d1/d17/d1f/d5d/d6f 0 2024-03-26T18:54:47.562 INFO:tasks.workunit.client.0.smithi102.stdout:8/928: unlink d0/d15/da0/d10c/fb4 0 2024-03-26T18:54:47.562 INFO:tasks.workunit.client.0.smithi102.stdout:4/860: stat d1/la9 0 2024-03-26T18:54:47.562 INFO:tasks.workunit.client.0.smithi102.stdout:4/861: dread - d1/d17/d1a/d21/d77/ff9 zero size 2024-03-26T18:54:47.562 INFO:tasks.workunit.client.0.smithi102.stdout:4/862: readlink d1/d17/d2a/d9a/lde 0 2024-03-26T18:54:47.564 INFO:tasks.workunit.client.0.smithi102.stdout:6/880: dwrite d2/df/d35/d43/fd1 [0,4194304] 0 2024-03-26T18:54:47.564 INFO:tasks.workunit.client.0.smithi102.stdout:8/929: symlink d0/d3c/df9/l134 0 2024-03-26T18:54:47.564 INFO:tasks.workunit.client.0.smithi102.stdout:8/930: chown d0/d15/d5d/dd3/d2c/d33/l3f 1329820 1 2024-03-26T18:54:47.564 INFO:tasks.workunit.client.0.smithi102.stdout:8/931: readlink d0/d3c/df9/lfd 0 2024-03-26T18:54:47.564 INFO:tasks.workunit.client.0.smithi102.stdout:8/932: chown d0/d3c/d96/d98/db6/c37 252 1 2024-03-26T18:54:47.564 INFO:tasks.workunit.client.0.smithi102.stdout:8/933: write d0/d15/fc8 [979862,116235] 0 2024-03-26T18:54:47.564 INFO:tasks.workunit.client.0.smithi102.stdout:8/934: chown d0/d15/d1b/f130 120999 1 2024-03-26T18:54:47.564 INFO:tasks.workunit.client.0.smithi102.stdout:8/935: write d0/d15/d5d/fc7 [696818,57547] 0 2024-03-26T18:54:47.565 INFO:tasks.workunit.client.0.smithi102.stdout:4/863: mknod d1/d17/dbb/dee/d10c/c121 0 2024-03-26T18:54:47.565 INFO:tasks.workunit.client.0.smithi102.stdout:4/864: read - d1/d17/d1f/d5d/d8b/d3e/d74/f119 zero size 2024-03-26T18:54:47.565 INFO:tasks.workunit.client.0.smithi102.stdout:1/862: dwrite de/d16/d1d/d2a/d2c/dd9/fdf [0,4194304] 0 2024-03-26T18:54:47.571 INFO:tasks.workunit.client.0.smithi102.stdout:6/881: creat d2/df/d35/d60/f125 x:0 0 0 2024-03-26T18:54:47.573 INFO:tasks.workunit.client.0.smithi102.stdout:8/936: truncate d0/d15/d5d/dd3/d26/d80/f112 957883 0 2024-03-26T18:54:47.577 INFO:tasks.workunit.client.0.smithi102.stdout:7/805: dwrite d7/d12/d4d/f5e [0,4194304] 0 2024-03-26T18:54:47.577 INFO:tasks.workunit.client.0.smithi102.stdout:7/806: stat d7/d12/d2b/d50/fa7 0 2024-03-26T18:54:47.577 INFO:tasks.workunit.client.0.smithi102.stdout:9/800: dwrite df/f27 [0,4194304] 0 2024-03-26T18:54:47.580 INFO:tasks.workunit.client.0.smithi102.stdout:1/863: dread de/d16/d1d/d2a/d2c/d43/f5a [0,4194304] 0 2024-03-26T18:54:47.582 INFO:tasks.workunit.client.0.smithi102.stdout:6/882: creat d2/d3d/d53/d5c/f126 x:0 0 0 2024-03-26T18:54:47.585 INFO:tasks.workunit.client.0.smithi102.stdout:4/865: dwrite d1/d27/f6e [0,4194304] 0 2024-03-26T18:54:47.585 INFO:tasks.workunit.client.0.smithi102.stdout:8/937: unlink d0/d15/la2 0 2024-03-26T18:54:47.588 INFO:tasks.workunit.client.0.smithi102.stdout:7/807: mknod d7/d10/d27/d34/d82/cf1 0 2024-03-26T18:54:47.588 INFO:tasks.workunit.client.0.smithi102.stdout:7/808: fdatasync d7/d10/d67/d69/f9c 0 2024-03-26T18:54:47.588 INFO:tasks.workunit.client.0.smithi102.stdout:9/801: symlink df/d1a/d81/d88/l128 0 2024-03-26T18:54:47.590 INFO:tasks.workunit.client.0.smithi102.stdout:1/864: truncate de/d16/d1d/d2a/d67/ff5 1055479 0 2024-03-26T18:54:47.590 INFO:tasks.workunit.client.0.smithi102.stdout:1/865: write f8 [2051685,88328] 0 2024-03-26T18:54:47.590 INFO:tasks.workunit.client.0.smithi102.stdout:1/866: getdents de/d16/d1d/d2a/d2c/d56/d78/de0 0 2024-03-26T18:54:47.590 INFO:tasks.workunit.client.0.smithi102.stdout:1/867: dread - de/d1f/de3/f119 zero size 2024-03-26T18:54:47.591 INFO:tasks.workunit.client.0.smithi102.stdout:6/883: symlink d2/d12/dce/l127 0 2024-03-26T18:54:47.591 INFO:tasks.workunit.client.0.smithi102.stdout:6/884: chown d2/d3/l26 9732 1 2024-03-26T18:54:47.591 INFO:tasks.workunit.client.0.smithi102.stdout:6/885: creat d2/df/d35/da1/f128 x:0 0 0 2024-03-26T18:54:47.591 INFO:tasks.workunit.client.0.smithi102.stdout:8/938: dread - d0/d15/d5d/dd3/d2c/d4e/f116 zero size 2024-03-26T18:54:47.594 INFO:tasks.workunit.client.0.smithi102.stdout:7/809: rename d7/d10/d27/d34/c91 to d7/d10/d27/d34/dcd/cf2 0 2024-03-26T18:54:47.594 INFO:tasks.workunit.client.0.smithi102.stdout:7/810: chown d7/d12/d4b/d83 514211 1 2024-03-26T18:54:47.596 INFO:tasks.workunit.client.0.smithi102.stdout:9/802: link df/d1a/d1d/l1f df/d46/l129 0 2024-03-26T18:54:47.596 INFO:tasks.workunit.client.0.smithi102.stdout:4/866: dread d1/d27/f6e [0,4194304] 0 2024-03-26T18:54:47.598 INFO:tasks.workunit.client.0.smithi102.stdout:1/868: link de/d1f/l5c de/d16/d1d/d61/l124 0 2024-03-26T18:54:47.603 INFO:tasks.workunit.client.0.smithi102.stdout:8/939: mkdir d0/d15/d5d/dd3/d26/d19/d76/d135 0 2024-03-26T18:54:47.603 INFO:tasks.workunit.client.0.smithi102.stdout:8/940: write d0/d15/da0/d10c/f8b [2199372,7942] 0 2024-03-26T18:54:47.603 INFO:tasks.workunit.client.0.smithi102.stdout:9/803: dread df/d1a/d23/d3e/d75/f4a [0,4194304] 0 2024-03-26T18:54:47.605 INFO:tasks.workunit.client.0.smithi102.stdout:7/811: mknod d7/d10/d67/cf3 0 2024-03-26T18:54:47.605 INFO:tasks.workunit.client.0.smithi102.stdout:7/812: chown d7/d2a/d9d/cc8 0 1 2024-03-26T18:54:47.607 INFO:tasks.workunit.client.0.smithi102.stdout:4/867: mknod d1/d17/d2a/d9a/c122 0 2024-03-26T18:54:47.607 INFO:tasks.workunit.client.0.smithi102.stdout:4/868: fsync d1/d75/f117 0 2024-03-26T18:54:47.607 INFO:tasks.workunit.client.0.smithi102.stdout:4/869: chown d1/d17/dbb/dee/d10c/c121 784272658 1 2024-03-26T18:54:47.609 INFO:tasks.workunit.client.0.smithi102.stdout:1/869: creat de/d16/d1d/d2a/d2c/d56/d78/dc3/f125 x:0 0 0 2024-03-26T18:54:47.611 INFO:tasks.workunit.client.0.smithi102.stdout:6/886: getdents d2/da4/df1 0 2024-03-26T18:54:47.611 INFO:tasks.workunit.client.0.smithi102.stdout:6/887: chown d2/df/d71/d80/d91/db8/df9 232713613 1 2024-03-26T18:54:47.613 INFO:tasks.workunit.client.0.smithi102.stdout:8/941: symlink d0/d15/de6/l136 0 2024-03-26T18:54:47.613 INFO:tasks.workunit.client.0.smithi102.stdout:8/942: dread - d0/d9/f24 zero size 2024-03-26T18:54:47.613 INFO:tasks.workunit.client.0.smithi102.stdout:8/943: chown d0/d15/da0/f119 16103 1 2024-03-26T18:54:47.613 INFO:tasks.workunit.client.0.smithi102.stdout:8/944: chown d0/d15/d5d/dd3/d2c/d2f/c74 23957485 1 2024-03-26T18:54:47.618 INFO:tasks.workunit.client.0.smithi102.stdout:7/813: fsync d7/d12/d4d/d62/f66 0 2024-03-26T18:54:47.618 INFO:tasks.workunit.client.0.smithi102.stdout:7/814: write d7/f41 [2630962,22762] 0 2024-03-26T18:54:47.619 INFO:tasks.workunit.client.0.smithi102.stdout:7/815: getdents d7/d12/d2b/da8/dca 0 2024-03-26T18:54:47.621 INFO:tasks.workunit.client.0.smithi102.stdout:4/870: rmdir d1/d27/d98/db7/d11e 0 2024-03-26T18:54:47.622 INFO:tasks.workunit.client.0.smithi102.stdout:1/870: mknod de/d16/d1d/d2a/d2c/d56/d78/dc3/c126 0 2024-03-26T18:54:47.622 INFO:tasks.workunit.client.0.smithi102.stdout:1/871: chown de/c13 1135750 1 2024-03-26T18:54:47.623 INFO:tasks.workunit.client.0.smithi102.stdout:6/888: rmdir d2/da4/df1 39 2024-03-26T18:54:47.623 INFO:tasks.workunit.client.0.smithi102.stdout:6/889: readlink d2/d3/l10a 0 2024-03-26T18:54:47.627 INFO:tasks.workunit.client.0.smithi102.stdout:1/872: mknod de/d16/d1d/c127 0 2024-03-26T18:54:47.627 INFO:tasks.workunit.client.0.smithi102.stdout:1/873: chown de/d2e/cf9 12 1 2024-03-26T18:54:47.631 INFO:tasks.workunit.client.0.smithi102.stdout:1/874: link f8 de/d16/d1d/d2a/d2c/d56/d78/dc3/f128 0 2024-03-26T18:54:47.632 INFO:tasks.workunit.client.0.smithi102.stdout:1/875: rename de/d16/d1d/d61/l70 to de/d2e/d62/l129 0 2024-03-26T18:54:47.634 INFO:tasks.workunit.client.0.smithi102.stdout:8/945: dwrite d0/d15/da0/d10c/fdc [0,4194304] 0 2024-03-26T18:54:47.634 INFO:tasks.workunit.client.0.smithi102.stdout:8/946: stat d0/d9/l11d 0 2024-03-26T18:54:47.636 INFO:tasks.workunit.client.0.smithi102.stdout:1/876: write de/d16/d1d/d2a/d67/ff5 [199079,93571] 0 2024-03-26T18:54:47.636 INFO:tasks.workunit.client.0.smithi102.stdout:9/804: truncate df/d46/da8/fbb 2053068 0 2024-03-26T18:54:47.639 INFO:tasks.workunit.client.0.smithi102.stdout:8/947: dread d0/d15/d5d/dd3/d26/f129 [0,4194304] 0 2024-03-26T18:54:47.641 INFO:tasks.workunit.client.0.smithi102.stdout:1/877: read de/d16/d1d/d61/fae [159544,116588] 0 2024-03-26T18:54:47.641 INFO:tasks.workunit.client.0.smithi102.stdout:1/878: chown de/d1f/d9f 199 1 2024-03-26T18:54:47.642 INFO:tasks.workunit.client.0.smithi102.stdout:8/948: creat d0/d15/d5d/dd3/d2c/d33/f137 x:0 0 0 2024-03-26T18:54:47.642 INFO:tasks.workunit.client.0.smithi102.stdout:8/949: creat d0/d15/d1b/f138 x:0 0 0 2024-03-26T18:54:47.643 INFO:tasks.workunit.client.0.smithi102.stdout:8/950: truncate d0/d15/d5d/dd3/d26/f129 2383799 0 2024-03-26T18:54:47.644 INFO:tasks.workunit.client.0.smithi102.stdout:8/951: mknod d0/d15/d5d/dd3/d2c/da7/c139 0 2024-03-26T18:54:47.644 INFO:tasks.workunit.client.0.smithi102.stdout:8/952: write d0/d15/da0/fb3 [108760,117279] 0 2024-03-26T18:54:47.644 INFO:tasks.workunit.client.0.smithi102.stdout:8/953: creat d0/d15/d9a/f13a x:0 0 0 2024-03-26T18:54:47.646 INFO:tasks.workunit.client.0.smithi102.stdout:8/954: rename d0/d15/d9a/f11f to d0/d3c/d96/d98/dcf/f13b 0 2024-03-26T18:54:47.650 INFO:tasks.workunit.client.0.smithi102.stdout:1/879: dread de/fac [8388608,4194304] 0 2024-03-26T18:54:47.652 INFO:tasks.workunit.client.0.smithi102.stdout:4/871: dwrite d1/d17/d1f/d5d/d8b/f51 [0,4194304] 0 2024-03-26T18:54:47.654 INFO:tasks.workunit.client.0.smithi102.stdout:4/872: mkdir d1/d27/d98/db7/d123 0 2024-03-26T18:54:47.655 INFO:tasks.workunit.client.0.smithi102.stdout:4/873: link d1/d27/d7a/c94 d1/d17/dbb/dee/c124 0 2024-03-26T18:54:47.657 INFO:tasks.workunit.client.0.smithi102.stdout:1/880: dread de/d1f/de3/da6/fb7 [0,4194304] 0 2024-03-26T18:54:47.657 INFO:tasks.workunit.client.0.smithi102.stdout:1/881: chown de/d16/d1d/d2a/d2c/d56/d78/de0 1109896 1 2024-03-26T18:54:47.657 INFO:tasks.workunit.client.0.smithi102.stdout:1/882: stat de/f2d 0 2024-03-26T18:54:47.657 INFO:tasks.workunit.client.0.smithi102.stdout:1/883: fdatasync de/d16/f40 0 2024-03-26T18:54:47.659 INFO:tasks.workunit.client.0.smithi102.stdout:4/874: write d1/f26 [1307440,20294] 0 2024-03-26T18:54:47.660 INFO:tasks.workunit.client.0.smithi102.stdout:8/955: dread d0/d15/da0/d10c/ffa [0,4194304] 0 2024-03-26T18:54:47.660 INFO:tasks.workunit.client.0.smithi102.stdout:6/890: dwrite d2/d7/f8 [4194304,4194304] 0 2024-03-26T18:54:47.660 INFO:tasks.workunit.client.0.smithi102.stdout:9/805: dwrite df/d1a/d23/d4e/db0/fe4 [0,4194304] 0 2024-03-26T18:54:47.662 INFO:tasks.workunit.client.0.smithi102.stdout:1/884: rename de/d16/d1d/d2a/f88 to de/d16/f12a 0 2024-03-26T18:54:47.662 INFO:tasks.workunit.client.0.smithi102.stdout:1/885: chown de/d1f/d9f/cc2 4 1 2024-03-26T18:54:47.662 INFO:tasks.workunit.client.0.smithi102.stdout:1/886: dread - de/d16/d1d/d2a/d67/f10e zero size 2024-03-26T18:54:47.664 INFO:tasks.workunit.client.0.smithi102.stdout:4/875: mknod d1/d17/d1f/d5d/d6f/dc0/d100/c125 0 2024-03-26T18:54:47.666 INFO:tasks.workunit.client.0.smithi102.stdout:6/891: mknod d2/d7/d3c/d78/c129 0 2024-03-26T18:54:47.671 INFO:tasks.workunit.client.0.smithi102.stdout:9/806: link df/d1a/d65/f73 df/d1a/d65/dbe/ddf/f12a 0 2024-03-26T18:54:47.674 INFO:tasks.workunit.client.0.smithi102.stdout:1/887: unlink de/d16/d1d/d61/f102 0 2024-03-26T18:54:47.679 INFO:tasks.workunit.client.0.smithi102.stdout:4/876: rename d1/d17/d1a/d21/d77/d102 to d1/d17/dbb/d126 0 2024-03-26T18:54:47.680 INFO:tasks.workunit.client.0.smithi102.stdout:4/877: readlink d1/d17/laa 0 2024-03-26T18:54:47.680 INFO:tasks.workunit.client.0.smithi102.stdout:4/878: creat d1/d25/f127 x:0 0 0 2024-03-26T18:54:47.681 INFO:tasks.workunit.client.0.smithi102.stdout:1/888: dread de/d16/d1d/d2a/d2c/d56/d6e/f81 [0,4194304] 0 2024-03-26T18:54:47.681 INFO:tasks.workunit.client.0.smithi102.stdout:1/889: truncate de/d16/d1d/d2a/d2c/d43/fd4 14750 0 2024-03-26T18:54:47.681 INFO:tasks.workunit.client.0.smithi102.stdout:6/892: rename d2/d7/d3c/d78/dc4/le7 to d2/d3/dd0/l12a 0 2024-03-26T18:54:47.681 INFO:tasks.workunit.client.0.smithi102.stdout:9/807: rename df/d1a/d1d/d72/d121 to df/d1a/d1d/d72/d121/d12b 22 2024-03-26T18:54:47.681 INFO:tasks.workunit.client.0.smithi102.stdout:9/808: write df/d1a/d1d/d72/d7d/db7/df8/f125 [939687,119193] 0 2024-03-26T18:54:47.682 INFO:tasks.workunit.client.0.smithi102.stdout:8/956: dwrite d0/d3c/d11c/fb7 [0,4194304] 0 2024-03-26T18:54:47.682 INFO:tasks.workunit.client.0.smithi102.stdout:4/879: truncate d1/d27/f55 4130059 0 2024-03-26T18:54:47.684 INFO:tasks.workunit.client.0.smithi102.stdout:1/890: mknod de/d2e/d11e/c12b 0 2024-03-26T18:54:47.685 INFO:tasks.workunit.client.0.smithi102.stdout:6/893: mkdir d2/d12/dce/d109/d12b 0 2024-03-26T18:54:47.685 INFO:tasks.workunit.client.0.smithi102.stdout:6/894: readlink d2/df/d71/d80/d91/lac 0 2024-03-26T18:54:47.687 INFO:tasks.workunit.client.0.smithi102.stdout:9/809: rename df/d1a/d57/cc8 to df/d1a/d1d/d72/d121/c12c 0 2024-03-26T18:54:47.687 INFO:tasks.workunit.client.0.smithi102.stdout:9/810: truncate df/d1a/d23/fb5 1056231 0 2024-03-26T18:54:47.689 INFO:tasks.workunit.client.0.smithi102.stdout:8/957: mknod d0/d3c/d96/dd5/c13c 0 2024-03-26T18:54:47.689 INFO:tasks.workunit.client.0.smithi102.stdout:8/958: chown d0/d3c/d96/dd5/c13c 59 1 2024-03-26T18:54:47.691 INFO:tasks.workunit.client.0.smithi102.stdout:4/880: link d1/d75/d85/f10f d1/d17/d2a/d9a/ddc/f128 0 2024-03-26T18:54:47.692 INFO:tasks.workunit.client.0.smithi102.stdout:1/891: symlink de/d16/d1d/d61/d6a/l12c 0 2024-03-26T18:54:47.692 INFO:tasks.workunit.client.0.smithi102.stdout:1/892: readlink de/d16/d1d/d2a/d2c/d56/d78/dc3/l122 0 2024-03-26T18:54:47.694 INFO:tasks.workunit.client.0.smithi102.stdout:6/895: creat d2/df/d71/d80/d91/db8/f12c x:0 0 0 2024-03-26T18:54:47.694 INFO:tasks.workunit.client.0.smithi102.stdout:6/896: write d2/d3d/f7d [837150,82385] 0 2024-03-26T18:54:47.695 INFO:tasks.workunit.client.0.smithi102.stdout:9/811: symlink df/d1a/d81/dcc/l12d 0 2024-03-26T18:54:47.697 INFO:tasks.workunit.client.0.smithi102.stdout:8/959: dread - d0/d15/d5d/dd3/d2c/ff6 zero size 2024-03-26T18:54:47.698 INFO:tasks.workunit.client.0.smithi102.stdout:6/897: dread d2/f2e [0,4194304] 0 2024-03-26T18:54:47.699 INFO:tasks.workunit.client.0.smithi102.stdout:1/893: getdents de/d16 0 2024-03-26T18:54:47.701 INFO:tasks.workunit.client.0.smithi102.stdout:9/812: unlink df/d1a/d23/d29/d9c/fab 0 2024-03-26T18:54:47.702 INFO:tasks.workunit.client.0.smithi102.stdout:4/881: mknod d1/d17/d1f/d5d/d8b/c129 0 2024-03-26T18:54:47.704 INFO:tasks.workunit.client.0.smithi102.stdout:6/898: creat d2/d3d/d53/f12d x:0 0 0 2024-03-26T18:54:47.706 INFO:tasks.workunit.client.0.smithi102.stdout:1/894: mkdir de/d16/d1d/d2a/d2c/d43/d111/d12d 0 2024-03-26T18:54:47.706 INFO:tasks.workunit.client.0.smithi102.stdout:1/895: fdatasync de/f26 0 2024-03-26T18:54:47.706 INFO:tasks.workunit.client.0.smithi102.stdout:1/896: read - de/d16/d1d/d2a/d2c/d56/d78/feb zero size 2024-03-26T18:54:47.707 INFO:tasks.workunit.client.0.smithi102.stdout:6/899: mkdir d2/d3d/d53/d5c/d12e 0 2024-03-26T18:54:47.707 INFO:tasks.workunit.client.0.smithi102.stdout:8/960: dread d0/d15/d5d/dd3/d2c/d2f/f8f [0,4194304] 0 2024-03-26T18:54:47.707 INFO:tasks.workunit.client.0.smithi102.stdout:8/961: readlink d0/d9/l11d 0 2024-03-26T18:54:47.709 INFO:tasks.workunit.client.0.smithi102.stdout:1/897: link de/d16/d1d/d2a/d2c/d43/lc7 de/d1f/de3/d10f/l12e 0 2024-03-26T18:54:47.710 INFO:tasks.workunit.client.0.smithi102.stdout:4/882: dread d1/d17/d1f/d5d/d8b/f66 [0,4194304] 0 2024-03-26T18:54:47.713 INFO:tasks.workunit.client.0.smithi102.stdout:6/900: mkdir d2/d3d/d12f 0 2024-03-26T18:54:47.714 INFO:tasks.workunit.client.0.smithi102.stdout:1/898: symlink de/d1f/d9f/l12f 0 2024-03-26T18:54:47.714 INFO:tasks.workunit.client.0.smithi102.stdout:1/899: read - de/d16/d1d/d2a/d67/fc9 zero size 2024-03-26T18:54:47.716 INFO:tasks.workunit.client.0.smithi102.stdout:6/901: rmdir d2/d3d/d53/d5c/d12e 0 2024-03-26T18:54:47.716 INFO:tasks.workunit.client.0.smithi102.stdout:6/902: write d2/da4/df1/f123 [679045,22823] 0 2024-03-26T18:54:47.718 INFO:tasks.workunit.client.0.smithi102.stdout:1/900: mkdir de/d16/d1d/d2a/d2c/d56/d6e/d130 0 2024-03-26T18:54:47.720 INFO:tasks.workunit.client.0.smithi102.stdout:6/903: symlink d2/d7/d3c/d78/dc4/l130 0 2024-03-26T18:54:47.721 INFO:tasks.workunit.client.0.smithi102.stdout:1/901: getdents de/d16/d1d/d2a/d2c/d56/d6e/dfc/d110 0 2024-03-26T18:54:47.728 INFO:tasks.workunit.client.0.smithi102.stdout:6/904: dread d2/df/d35/d43/f94 [0,4194304] 0 2024-03-26T18:54:47.729 INFO:tasks.workunit.client.0.smithi102.stdout:6/905: chown d2/d3d/d53/lc3 7643 1 2024-03-26T18:54:47.731 INFO:tasks.workunit.client.0.smithi102.stdout:6/906: rmdir d2/df/d71/d80/d105 0 2024-03-26T18:54:47.731 INFO:tasks.workunit.client.0.smithi102.stdout:6/907: fdatasync d2/d3d/d53/d5c/dc5/fc8 0 2024-03-26T18:54:47.732 INFO:tasks.workunit.client.0.smithi102.stdout:1/902: dread f9 [0,4194304] 0 2024-03-26T18:54:47.732 INFO:tasks.workunit.client.0.smithi102.stdout:1/903: truncate de/d16/d1d/d61/d6a/fa3 2489185 0 2024-03-26T18:54:47.734 INFO:tasks.workunit.client.0.smithi102.stdout:1/904: stat de/d1f/de3/d10f/l12e 0 2024-03-26T18:54:47.734 INFO:tasks.workunit.client.0.smithi102.stdout:9/813: dwrite df/d1a/d1d/d72/d7d/db7/df8/f125 [0,4194304] 0 2024-03-26T18:54:47.734 INFO:tasks.workunit.client.0.smithi102.stdout:9/814: write df/d1a/d23/d4e/f53 [1433494,109305] 0 2024-03-26T18:54:47.735 INFO:tasks.workunit.client.0.smithi102.stdout:8/962: dwrite d0/d15/d5d/dd3/f32 [0,4194304] 0 2024-03-26T18:54:47.735 INFO:tasks.workunit.client.0.smithi102.stdout:8/963: chown d0/d15/d5d/dd3/d26/d19/d76/f9c 9983637 1 2024-03-26T18:54:47.735 INFO:tasks.workunit.client.0.smithi102.stdout:8/964: stat d0/d3c/d96/d98/dcf/f105 0 2024-03-26T18:54:47.739 INFO:tasks.workunit.client.0.smithi102.stdout:9/815: creat df/d1a/d23/f12e x:0 0 0 2024-03-26T18:54:47.743 INFO:tasks.workunit.client.0.smithi102.stdout:1/905: write de/d16/d1d/d2a/d2c/d56/d78/dc3/f128 [5546614,13098] 0 2024-03-26T18:54:47.745 INFO:tasks.workunit.client.0.smithi102.stdout:9/816: mkdir df/d1a/d81/d12f 0 2024-03-26T18:54:47.745 INFO:tasks.workunit.client.0.smithi102.stdout:8/965: link d0/d15/d5d/dd3/d26/d80/f112 d0/d15/d5d/dd3/d26/f13d 0 2024-03-26T18:54:47.749 INFO:tasks.workunit.client.0.smithi102.stdout:1/906: mkdir de/d2e/d131 0 2024-03-26T18:54:47.750 INFO:tasks.workunit.client.0.smithi102.stdout:1/907: chown de/d16/d1d/d2a/d2c/d56/d78/f8c 138 1 2024-03-26T18:54:47.750 INFO:tasks.workunit.client.0.smithi102.stdout:9/817: symlink df/d1a/d23/d3e/d75/d110/l130 0 2024-03-26T18:54:47.750 INFO:tasks.workunit.client.0.smithi102.stdout:8/966: creat d0/d3c/d96/d98/f13e x:0 0 0 2024-03-26T18:54:47.751 INFO:tasks.workunit.client.0.smithi102.stdout:1/908: symlink de/d1f/d9f/l132 0 2024-03-26T18:54:47.755 INFO:tasks.workunit.client.0.smithi102.stdout:6/908: dwrite d2/df/f2b [0,4194304] 0 2024-03-26T18:54:47.755 INFO:tasks.workunit.client.0.smithi102.stdout:6/909: creat d2/df/d71/d80/d91/db8/f131 x:0 0 0 2024-03-26T18:54:47.758 INFO:tasks.workunit.client.0.smithi102.stdout:9/818: read df/d1a/f3f [3212141,101085] 0 2024-03-26T18:54:47.765 INFO:tasks.workunit.client.0.smithi102.stdout:6/910: creat d2/d3d/d53/db1/f132 x:0 0 0 2024-03-26T18:54:47.767 INFO:tasks.workunit.client.0.smithi102.stdout:6/911: mknod d2/d3d/d53/d63/c133 0 2024-03-26T18:54:47.769 INFO:tasks.workunit.client.0.smithi102.stdout:9/819: dread df/d1a/d23/d29/f34 [0,4194304] 0 2024-03-26T18:54:47.771 INFO:tasks.workunit.client.0.smithi102.stdout:9/820: rename df/d1a/d23/fc3 to df/d1a/d23/d29/d9c/f131 0 2024-03-26T18:54:47.771 INFO:tasks.workunit.client.0.smithi102.stdout:9/821: write df/d1a/d23/f71 [411040,110613] 0 2024-03-26T18:54:47.772 INFO:tasks.workunit.client.0.smithi102.stdout:9/822: symlink df/d1a/d1d/d72/d7d/db7/l132 0 2024-03-26T18:54:47.772 INFO:tasks.workunit.client.0.smithi102.stdout:9/823: creat df/d1a/d81/dcc/f133 x:0 0 0 2024-03-26T18:54:47.772 INFO:tasks.workunit.client.0.smithi102.stdout:9/824: dread - df/d1a/d81/dcc/f133 zero size 2024-03-26T18:54:47.777 INFO:tasks.workunit.client.0.smithi102.stdout:6/912: dread d2/df/d72/f99 [0,4194304] 0 2024-03-26T18:54:47.777 INFO:tasks.workunit.client.0.smithi102.stdout:6/913: chown d2/df/d35/d43/fd5 4085 1 2024-03-26T18:54:47.778 INFO:tasks.workunit.client.0.smithi102.stdout:1/909: dwrite de/d16/d1d/d2a/d67/f10e [0,4194304] 0 2024-03-26T18:54:47.779 INFO:tasks.workunit.client.0.smithi102.stdout:8/967: dwrite d0/d15/d5d/dd3/d26/d19/d9f/fa9 [4194304,4194304] 0 2024-03-26T18:54:47.780 INFO:tasks.workunit.client.0.smithi102.stdout:6/914: unlink d2/d3d/d53/d63/c133 0 2024-03-26T18:54:47.780 INFO:tasks.workunit.client.0.smithi102.stdout:6/915: dread - d2/df/d35/d60/f125 zero size 2024-03-26T18:54:47.782 INFO:tasks.workunit.client.0.smithi102.stdout:9/825: write df/d1a/d7f/fba [1704326,50289] 0 2024-03-26T18:54:47.782 INFO:tasks.workunit.client.0.smithi102.stdout:9/826: dread - df/d1a/d23/d3e/d75/d110/f119 zero size 2024-03-26T18:54:47.784 INFO:tasks.workunit.client.0.smithi102.stdout:9/827: creat df/d1a/d1d/d72/d7d/db7/f134 x:0 0 0 2024-03-26T18:54:47.784 INFO:tasks.workunit.client.0.smithi102.stdout:9/828: creat df/d1a/d1d/d72/d7d/db7/df8/f135 x:0 0 0 2024-03-26T18:54:47.789 INFO:tasks.workunit.client.0.smithi102.stdout:9/829: unlink df/d1a/d81/d88/l128 0 2024-03-26T18:54:47.791 INFO:tasks.workunit.client.0.smithi102.stdout:6/916: write d2/df/d35/d43/fd1 [2687427,74898] 0 2024-03-26T18:54:47.799 INFO:tasks.workunit.client.0.smithi102.stdout:9/830: unlink df/d1a/d1d/d72/c83 0 2024-03-26T18:54:47.800 INFO:tasks.workunit.client.0.smithi102.stdout:6/917: mknod d2/df/dba/c134 0 2024-03-26T18:54:47.801 INFO:tasks.workunit.client.0.smithi102.stdout:9/831: stat df/f17 0 2024-03-26T18:54:47.802 INFO:tasks.workunit.client.0.smithi102.stdout:9/832: dread df/fca [0,4194304] 0 2024-03-26T18:54:47.802 INFO:tasks.workunit.client.0.smithi102.stdout:9/833: write df/d1a/d23/d3e/f10d [610921,51247] 0 2024-03-26T18:54:47.802 INFO:tasks.workunit.client.0.smithi102.stdout:6/918: link d2/df/d72/db7/l114 d2/d3d/d12f/l135 0 2024-03-26T18:54:47.803 INFO:tasks.workunit.client.0.smithi102.stdout:6/919: fsync d2/d12/dce/d109/f112 0 2024-03-26T18:54:47.803 INFO:tasks.workunit.client.0.smithi102.stdout:9/834: mknod df/d1a/d23/d29/d9c/c136 0 2024-03-26T18:54:47.805 INFO:tasks.workunit.client.0.smithi102.stdout:8/968: dwrite d0/d15/d5d/dd3/d2c/d2f/f7e [0,4194304] 0 2024-03-26T18:54:47.805 INFO:tasks.workunit.client.0.smithi102.stdout:9/835: mkdir df/d1a/d23/d3e/d75/d137 0 2024-03-26T18:54:47.806 INFO:tasks.workunit.client.0.smithi102.stdout:8/969: getdents d0/d15/d5d/dd3/d2c/d33/d87/d10b 0 2024-03-26T18:54:47.807 INFO:tasks.workunit.client.0.smithi102.stdout:9/836: creat df/d1a/d1d/f138 x:0 0 0 2024-03-26T18:54:47.807 INFO:tasks.workunit.client.0.smithi102.stdout:9/837: chown df/d1a/d23/d29/l92 104762 1 2024-03-26T18:54:47.810 INFO:tasks.workunit.client.0.smithi102.stdout:6/920: link d2/d7/c25 d2/c136 0 2024-03-26T18:54:47.810 INFO:tasks.workunit.client.0.smithi102.stdout:6/921: chown d2/df/d35/d43/l90 3508 1 2024-03-26T18:54:47.810 INFO:tasks.workunit.client.0.smithi102.stdout:8/970: getdents d0/d3c/d96/d98 0 2024-03-26T18:54:47.832 INFO:tasks.workunit.client.0.smithi102.stdout:8/971: dread d0/d15/d5d/dd3/d2c/f8e [0,4194304] 0 2024-03-26T18:54:47.838 INFO:tasks.workunit.client.0.smithi102.stdout:9/838: dwrite df/d46/da8/f105 [0,4194304] 0 2024-03-26T18:54:47.839 INFO:tasks.workunit.client.0.smithi102.stdout:6/922: dwrite d2/d7/d3c/d78/f7b [0,4194304] 0 2024-03-26T18:54:47.840 INFO:tasks.workunit.client.0.smithi102.stdout:9/839: creat df/d1a/d81/d88/f139 x:0 0 0 2024-03-26T18:54:47.843 INFO:tasks.workunit.client.0.smithi102.stdout:6/923: creat d2/da4/f137 x:0 0 0 2024-03-26T18:54:47.843 INFO:tasks.workunit.client.0.smithi102.stdout:8/972: write d0/d3c/d96/d98/db6/f3a [3372737,26483] 0 2024-03-26T18:54:47.843 INFO:tasks.workunit.client.0.smithi102.stdout:8/973: dread - d0/d3c/d96/d98/f13e zero size 2024-03-26T18:54:47.843 INFO:tasks.workunit.client.0.smithi102.stdout:9/840: mknod df/d1a/d1d/d72/d7d/db7/df8/c13a 0 2024-03-26T18:54:47.844 INFO:tasks.workunit.client.0.smithi102.stdout:6/924: creat d2/df/d71/d80/d91/db8/df9/f138 x:0 0 0 2024-03-26T18:54:47.845 INFO:tasks.workunit.client.0.smithi102.stdout:8/974: mkdir d0/d3c/d96/d98/db6/d13f 0 2024-03-26T18:54:47.846 INFO:tasks.workunit.client.0.smithi102.stdout:6/925: rmdir d2/d3/dd0 39 2024-03-26T18:54:47.847 INFO:tasks.workunit.client.0.smithi102.stdout:9/841: dread df/d1a/d23/d29/fc4 [0,4194304] 0 2024-03-26T18:54:47.847 INFO:tasks.workunit.client.0.smithi102.stdout:8/975: symlink d0/d3c/l140 0 2024-03-26T18:54:47.847 INFO:tasks.workunit.client.0.smithi102.stdout:8/976: truncate d0/d9/fb9 1129543 0 2024-03-26T18:54:47.848 INFO:tasks.workunit.client.0.smithi102.stdout:6/926: rename d2/d3d/d53/d63/l10c to d2/df/d71/d80/d91/db8/l139 0 2024-03-26T18:54:47.850 INFO:tasks.workunit.client.0.smithi102.stdout:9/842: mknod df/d1a/d1d/d72/d7d/db7/dd5/c13b 0 2024-03-26T18:54:47.850 INFO:tasks.workunit.client.0.smithi102.stdout:9/843: write df/d84/f78 [12654,54008] 0 2024-03-26T18:54:47.850 INFO:tasks.workunit.client.0.smithi102.stdout:8/977: mknod d0/d15/d5d/dd3/d26/d80/c141 0 2024-03-26T18:54:47.851 INFO:tasks.workunit.client.0.smithi102.stdout:6/927: creat d2/df/f13a x:0 0 0 2024-03-26T18:54:47.852 INFO:tasks.workunit.client.0.smithi102.stdout:8/978: symlink d0/d15/d5d/dd3/d2c/d4e/de3/l142 0 2024-03-26T18:54:47.852 INFO:tasks.workunit.client.0.smithi102.stdout:8/979: chown d0/d15/d5d/dd3/d26/d5c/lc4 4 1 2024-03-26T18:54:47.855 INFO:tasks.workunit.client.0.smithi102.stdout:6/928: rename d2/d3d/fb0 to d2/df/d72/db7/dbd/f13b 0 2024-03-26T18:54:47.856 INFO:tasks.workunit.client.0.smithi102.stdout:6/929: creat d2/d12/dce/d109/f13c x:0 0 0 2024-03-26T18:54:47.857 INFO:tasks.workunit.client.0.smithi102.stdout:8/980: rmdir d0/d15/d5d/dd3/d26/d19/d76/d135 0 2024-03-26T18:54:47.860 INFO:tasks.workunit.client.0.smithi102.stdout:8/981: rename d0/d15/fb0 to d0/f143 0 2024-03-26T18:54:47.861 INFO:tasks.workunit.client.0.smithi102.stdout:8/982: mkdir d0/d15/d5d/dd3/d2c/d4e/dca/d144 0 2024-03-26T18:54:47.863 INFO:tasks.workunit.client.0.smithi102.stdout:8/983: creat d0/d9/f145 x:0 0 0 2024-03-26T18:54:47.863 INFO:tasks.workunit.client.0.smithi102.stdout:8/984: creat d0/d15/d5d/dd3/d2c/d4e/f146 x:0 0 0 2024-03-26T18:54:47.872 INFO:tasks.workunit.client.0.smithi102.stdout:9/844: dwrite df/d1a/d81/dcc/f120 [0,4194304] 0 2024-03-26T18:54:47.877 INFO:tasks.workunit.client.0.smithi102.stdout:9/845: dread df/d1a/d1d/d72/f90 [0,4194304] 0 2024-03-26T18:54:47.877 INFO:tasks.workunit.client.0.smithi102.stdout:9/846: chown df/d1a/d81/d88/fc9 1 1 2024-03-26T18:54:47.877 INFO:tasks.workunit.client.0.smithi102.stdout:9/847: fsync df/d1a/d23/d29/fdc 0 2024-03-26T18:54:47.877 INFO:tasks.workunit.client.0.smithi102.stdout:9/848: stat df/d46/da8 0 2024-03-26T18:54:47.878 INFO:tasks.workunit.client.0.smithi102.stdout:9/849: mkdir df/d1a/d1d/d72/d7d/db7/d13c 0 2024-03-26T18:54:47.878 INFO:tasks.workunit.client.0.smithi102.stdout:9/850: truncate df/d1a/f10b 395558 0 2024-03-26T18:54:47.878 INFO:tasks.workunit.client.0.smithi102.stdout:6/930: dwrite d2/df/d71/fdb [0,4194304] 0 2024-03-26T18:54:47.879 INFO:tasks.workunit.client.0.smithi102.stdout:9/851: symlink df/d1a/d1d/d72/d7d/db7/df8/l13d 0 2024-03-26T18:54:47.882 INFO:tasks.workunit.client.0.smithi102.stdout:6/931: rmdir d2/df/d71/d80/d91/dc1 39 2024-03-26T18:54:47.883 INFO:tasks.workunit.client.0.smithi102.stdout:9/852: creat df/d1a/f13e x:0 0 0 2024-03-26T18:54:47.884 INFO:tasks.workunit.client.0.smithi102.stdout:9/853: symlink df/d1a/d1d/d72/d7d/db7/l13f 0 2024-03-26T18:54:47.891 INFO:tasks.workunit.client.0.smithi102.stdout:8/985: dread d0/d15/d5d/dd3/d2c/f8e [0,4194304] 0 2024-03-26T18:54:47.891 INFO:tasks.workunit.client.0.smithi102.stdout:8/986: rename d0/d3c/df9 to d0/d3c/df9/d147 22 2024-03-26T18:54:47.891 INFO:tasks.workunit.client.0.smithi102.stdout:8/987: chown d0/d15/d5d/f61 33614903 1 2024-03-26T18:54:47.892 INFO:tasks.workunit.client.0.smithi102.stdout:9/854: dread df/d46/da8/fbb [0,4194304] 0 2024-03-26T18:54:47.893 INFO:tasks.workunit.client.0.smithi102.stdout:8/988: link d0/d15/d1b/cb8 d0/d15/d5d/dd3/d2c/da7/c148 0 2024-03-26T18:54:47.894 INFO:tasks.workunit.client.0.smithi102.stdout:9/855: mknod df/d1a/d81/d12f/c140 0 2024-03-26T18:54:47.898 INFO:tasks.workunit.client.0.smithi102.stdout:8/989: creat d0/d15/d5d/dd3/d2c/d33/d87/d10b/f149 x:0 0 0 2024-03-26T18:54:47.898 INFO:tasks.workunit.client.0.smithi102.stdout:8/990: chown d0/d15/d5d/dd3/d2c/d2f/f7e 4950826 1 2024-03-26T18:54:47.898 INFO:tasks.workunit.client.0.smithi102.stdout:8/991: chown d0/d15/d1b/f130 13 1 2024-03-26T18:54:47.898 INFO:tasks.workunit.client.0.smithi102.stdout:8/992: readlink d0/d15/d5d/dd3/d2c/d33/l3f 0 2024-03-26T18:54:47.898 INFO:tasks.workunit.client.0.smithi102.stdout:8/993: write d0/d15/d5d/dd3/d26/fad [667596,105698] 0 2024-03-26T18:54:47.900 INFO:tasks.workunit.client.0.smithi102.stdout:8/994: link d0/d15/d5d/dd3/d2c/d2f/fd8 d0/d3c/d11c/f14a 0 2024-03-26T18:54:47.904 INFO:tasks.workunit.client.0.smithi102.stdout:6/932: read d2/d3d/d53/d63/fc7 [2876798,29285] 0 2024-03-26T18:54:47.904 INFO:tasks.workunit.client.0.smithi102.stdout:8/995: truncate d0/d15/d5d/dd3/d2c/da7/f29 6787178 0 2024-03-26T18:54:47.908 INFO:tasks.workunit.client.0.smithi102.stdout:8/996: mknod d0/d15/da0/c14b 0 2024-03-26T18:54:47.909 INFO:tasks.workunit.client.0.smithi102.stdout:8/997: rmdir d0/d15/d5d/dac 39 2024-03-26T18:54:47.915 INFO:tasks.workunit.client.0.smithi102.stdout:6/933: dread d2/df/d35/d43/f61 [0,4194304] 0 2024-03-26T18:54:47.920 INFO:tasks.workunit.client.0.smithi102.stdout:6/934: dread d2/df/d72/f99 [0,4194304] 0 2024-03-26T18:54:47.931 INFO:tasks.workunit.client.0.smithi102.stdout:6/935: write d2/d3/fe8 [984972,54271] 0 2024-03-26T18:54:47.931 INFO:tasks.workunit.client.0.smithi102.stdout:6/936: write d2/df/d71/d80/ff2 [364758,29187] 0 2024-03-26T18:54:47.935 INFO:tasks.workunit.client.0.smithi102.stdout:8/998: dwrite d0/d15/d5d/dd3/d2c/d4e/f146 [0,4194304] 0 2024-03-26T18:54:47.935 INFO:tasks.workunit.client.0.smithi102.stdout:6/937: dread d2/d7/f51 [0,4194304] 0 2024-03-26T18:54:47.955 INFO:tasks.workunit.client.0.smithi102.stdout:8/999: write d0/d15/d5d/dd3/d26/d19/d76/f9c [4043558,108058] 0 2024-03-26T18:54:47.955 INFO:tasks.workunit.client.0.smithi102.stdout:6/938: dread d2/df/d71/fdb [0,4194304] 0 2024-03-26T18:54:47.955 INFO:tasks.workunit.client.0.smithi102.stdout:6/939: link d2/d12/dce/c110 d2/d3d/d12f/c13d 0 2024-03-26T18:54:47.964 INFO:tasks.workunit.client.0.smithi102.stdout:6/940: read d2/d7/f46 [2876509,48477] 0 2024-03-26T18:54:47.964 INFO:tasks.workunit.client.0.smithi102.stdout:6/941: fsync d2/d3d/d53/d5c/f126 0 2024-03-26T18:54:47.965 INFO:tasks.workunit.client.0.smithi102.stdout:6/942: rmdir d2/d3d/d53/d77 39 2024-03-26T18:54:47.966 INFO:tasks.workunit.client.0.smithi102.stdout:6/943: creat d2/df/d35/d43/d75/f13e x:0 0 0 2024-03-26T18:54:47.966 INFO:tasks.workunit.client.0.smithi102.stdout:6/944: dread - d2/d3d/d53/fe3 zero size 2024-03-26T18:54:47.966 INFO:tasks.workunit.client.0.smithi102.stdout:6/945: truncate d2/d3/f97 1842203 0 2024-03-26T18:54:48.280 INFO:tasks.workunit.client.0.smithi102.stdout:2/874: sync 2024-03-26T18:54:48.283 INFO:tasks.workunit.client.0.smithi102.stdout:2/875: rename d3/d82/d93/d9e/da0/fc3 to d3/d11/d15/d31/d7f/dfe/f138 0 2024-03-26T18:54:48.309 INFO:tasks.workunit.client.0.smithi102.stdout:2/876: dwrite d3/d11/d15/f11c [0,4194304] 0 2024-03-26T18:54:48.311 INFO:tasks.workunit.client.0.smithi102.stdout:2/877: dread d3/d20/f80 [0,4194304] 0 2024-03-26T18:54:48.313 INFO:tasks.workunit.client.0.smithi102.stdout:2/878: unlink d3/d20/d78/c6f 0 2024-03-26T18:54:48.314 INFO:tasks.workunit.client.0.smithi102.stdout:2/879: unlink d3/d11/d15/d46/fa9 0 2024-03-26T18:54:48.315 INFO:tasks.workunit.client.0.smithi102.stdout:2/880: write d3/d20/ff0 [67954,30415] 0 2024-03-26T18:54:48.315 INFO:tasks.workunit.client.0.smithi102.stdout:2/881: write d3/d82/d93/d9e/d1a/d25/fcd [269669,87744] 0 2024-03-26T18:54:48.315 INFO:tasks.workunit.client.0.smithi102.stdout:2/882: symlink d3/d2a/dae/l139 0 2024-03-26T18:54:48.333 INFO:tasks.workunit.client.0.smithi102.stdout:2/883: dwrite d3/d2a/f2f [0,4194304] 0 2024-03-26T18:54:48.348 INFO:tasks.workunit.client.0.smithi102.stdout:2/884: dwrite d3/d11/fa6 [0,4194304] 0 2024-03-26T18:54:48.350 INFO:tasks.workunit.client.0.smithi102.stdout:2/885: symlink d3/d20/d78/d84/d6e/l13a 0 2024-03-26T18:54:48.350 INFO:tasks.workunit.client.0.smithi102.stdout:2/886: readlink d3/d11/d27/ldd 0 2024-03-26T18:54:48.350 INFO:tasks.workunit.client.0.smithi102.stdout:2/887: chown d3/d20/d78/d84/c8d 2 1 2024-03-26T18:54:48.350 INFO:tasks.workunit.client.0.smithi102.stdout:2/888: creat d3/d11/d15/d31/f13b x:0 0 0 2024-03-26T18:54:48.351 INFO:tasks.workunit.client.0.smithi102.stdout:2/889: mknod d3/d11/d15/d46/d7b/d97/d10a/c13c 0 2024-03-26T18:54:48.372 INFO:tasks.workunit.client.0.smithi102.stdout:1/910: sync 2024-03-26T18:54:48.373 INFO:tasks.workunit.client.0.smithi102.stdout:4/883: sync 2024-03-26T18:54:48.373 INFO:tasks.workunit.client.0.smithi102.stdout:9/856: sync 2024-03-26T18:54:48.373 INFO:tasks.workunit.client.0.smithi102.stdout:0/900: sync 2024-03-26T18:54:48.373 INFO:tasks.workunit.client.0.smithi102.stdout:6/946: sync 2024-03-26T18:54:48.373 INFO:tasks.workunit.client.0.smithi102.stdout:6/947: stat d2/df/d35/c6d 0 2024-03-26T18:54:48.373 INFO:tasks.workunit.client.0.smithi102.stdout:0/901: write d6/de/d13/d25/d4f/d66/f95 [784648,19509] 0 2024-03-26T18:54:48.373 INFO:tasks.workunit.client.0.smithi102.stdout:0/902: chown d6/d3e/l89 3303 1 2024-03-26T18:54:48.373 INFO:tasks.workunit.client.0.smithi102.stdout:0/903: readlink d6/d3e/d69/l127 0 2024-03-26T18:54:48.373 INFO:tasks.workunit.client.0.smithi102.stdout:0/904: chown d6/de/d13/d9a/f5b 1185878 1 2024-03-26T18:54:48.373 INFO:tasks.workunit.client.0.smithi102.stdout:7/816: sync 2024-03-26T18:54:48.379 INFO:tasks.workunit.client.0.smithi102.stdout:0/905: link d6/d3e/d69/cce d6/de/d13/d25/d31/c136 0 2024-03-26T18:54:48.379 INFO:tasks.workunit.client.0.smithi102.stdout:0/906: readlink d6/de/d13/d25/d31/daa/l135 0 2024-03-26T18:54:48.379 INFO:tasks.workunit.client.0.smithi102.stdout:0/907: creat d6/de/d13/d25/d4f/d66/f137 x:0 0 0 2024-03-26T18:54:48.381 INFO:tasks.workunit.client.0.smithi102.stdout:6/948: rename d2/d12/dce to d2/df/d35/da1/d13f 0 2024-03-26T18:54:48.383 INFO:tasks.workunit.client.0.smithi102.stdout:0/908: mknod d6/d3e/d87/c138 0 2024-03-26T18:54:48.384 INFO:tasks.workunit.client.0.smithi102.stdout:7/817: getdents d7/d12/d51/d6a 0 2024-03-26T18:54:48.387 INFO:tasks.workunit.client.0.smithi102.stdout:0/909: unlink d6/de/d13/d25/d31/f50 0 2024-03-26T18:54:48.387 INFO:tasks.workunit.client.0.smithi102.stdout:7/818: mkdir d7/d10/d67/de1/df4 0 2024-03-26T18:54:48.388 INFO:tasks.workunit.client.0.smithi102.stdout:0/910: mknod d6/de/d13/d25/d31/d45/d82/c139 0 2024-03-26T18:54:48.389 INFO:tasks.workunit.client.0.smithi102.stdout:7/819: truncate d7/d10/d17/f1b 6208985 0 2024-03-26T18:54:48.391 INFO:tasks.workunit.client.0.smithi102.stdout:0/911: creat d6/de/d13/d1f/d55/d57/d7d/d7f/dc8/f13a x:0 0 0 2024-03-26T18:54:48.393 INFO:tasks.workunit.client.0.smithi102.stdout:0/912: mkdir d6/d2d/d3d/d80/d13b 0 2024-03-26T18:54:48.393 INFO:tasks.workunit.client.0.smithi102.stdout:0/913: write d6/de/d13/d25/fe8 [796205,129227] 0 2024-03-26T18:54:48.400 INFO:tasks.workunit.client.0.smithi102.stdout:7/820: dread d7/d12/d51/fbc [0,4194304] 0 2024-03-26T18:54:48.401 INFO:tasks.workunit.client.0.smithi102.stdout:7/821: creat d7/d10/d67/d69/d9b/de5/ff5 x:0 0 0 2024-03-26T18:54:48.410 INFO:tasks.workunit.client.0.smithi102.stdout:6/949: dwrite d2/d7/f20 [0,4194304] 0 2024-03-26T18:54:48.411 INFO:tasks.workunit.client.0.smithi102.stdout:1/911: dwrite de/d16/d1d/f82 [0,4194304] 0 2024-03-26T18:54:48.413 INFO:tasks.workunit.client.0.smithi102.stdout:9/857: dwrite df/d1a/d65/dbe/ddf/f12a [0,4194304] 0 2024-03-26T18:54:48.413 INFO:tasks.workunit.client.0.smithi102.stdout:9/858: fsync df/d1a/d23/d4e/d8e/f56 0 2024-03-26T18:54:48.421 INFO:tasks.workunit.client.0.smithi102.stdout:1/912: rename de/d1f/de3/d10f/l103 to de/d16/d1d/d61/d6a/df2/l133 0 2024-03-26T18:54:48.426 INFO:tasks.workunit.client.0.smithi102.stdout:1/913: creat de/d1f/de3/da6/dca/dda/f134 x:0 0 0 2024-03-26T18:54:48.426 INFO:tasks.workunit.client.0.smithi102.stdout:1/914: chown de/d2e/d62/l129 31 1 2024-03-26T18:54:48.427 INFO:tasks.workunit.client.0.smithi102.stdout:7/822: dwrite d7/d10/d17/f37 [0,4194304] 0 2024-03-26T18:54:48.427 INFO:tasks.workunit.client.0.smithi102.stdout:7/823: read - d7/d12/d4b/d83/fc4 zero size 2024-03-26T18:54:48.429 INFO:tasks.workunit.client.0.smithi102.stdout:1/915: unlink de/d1f/f57 0 2024-03-26T18:54:48.429 INFO:tasks.workunit.client.0.smithi102.stdout:1/916: rename de/d16/d1d/d2a/d2c/d56/d6e/d130 to de/d16/d1d/d2a/d2c/d56/d6e/d130/d135 22 2024-03-26T18:54:48.430 INFO:tasks.workunit.client.0.smithi102.stdout:7/824: symlink d7/d10/d67/d69/d9b/dae/lf6 0 2024-03-26T18:54:48.432 INFO:tasks.workunit.client.0.smithi102.stdout:1/917: getdents de/d16/d1d/d2a/d2c/d56/dfa/d113 0 2024-03-26T18:54:48.432 INFO:tasks.workunit.client.0.smithi102.stdout:1/918: creat de/d16/d1d/d2a/d2c/d56/dfa/f136 x:0 0 0 2024-03-26T18:54:48.434 INFO:tasks.workunit.client.0.smithi102.stdout:7/825: rename d7/d10/d67/d69/l9a to d7/d10/d67/lf7 0 2024-03-26T18:54:48.436 INFO:tasks.workunit.client.0.smithi102.stdout:1/919: mkdir de/d1f/de3/da6/d137 0 2024-03-26T18:54:48.437 INFO:tasks.workunit.client.0.smithi102.stdout:7/826: creat d7/d12/d2b/da8/dca/ff8 x:0 0 0 2024-03-26T18:54:48.439 INFO:tasks.workunit.client.0.smithi102.stdout:1/920: rename de/d16/d1d/d2a/d2c/d56/dfa/f136 to de/d16/d1d/d2a/d2c/f138 0 2024-03-26T18:54:48.439 INFO:tasks.workunit.client.0.smithi102.stdout:1/921: fdatasync de/d16/d1d/d2a/d2c/d56/d6e/fe1 0 2024-03-26T18:54:48.439 INFO:tasks.workunit.client.0.smithi102.stdout:1/922: chown de/d16/d1d/d2a/d2c/d56/d6e/dfc/d110/ffb 62885 1 2024-03-26T18:54:48.440 INFO:tasks.workunit.client.0.smithi102.stdout:7/827: creat d7/d2a/d9d/ff9 x:0 0 0 2024-03-26T18:54:48.441 INFO:tasks.workunit.client.0.smithi102.stdout:1/923: mkdir de/d16/d1d/d2a/d2c/d56/dfa/d113/d115/d139 0 2024-03-26T18:54:48.443 INFO:tasks.workunit.client.0.smithi102.stdout:6/950: dwrite d2/d7/d3c/fb9 [0,4194304] 0 2024-03-26T18:54:48.443 INFO:tasks.workunit.client.0.smithi102.stdout:6/951: write d2/df/d35/da1/fa5 [213443,47429] 0 2024-03-26T18:54:48.443 INFO:tasks.workunit.client.0.smithi102.stdout:6/952: write d2/d3/fe8 [179601,7157] 0 2024-03-26T18:54:48.443 INFO:tasks.workunit.client.0.smithi102.stdout:9/859: dwrite df/d1a/d23/f5d [0,4194304] 0 2024-03-26T18:54:48.444 INFO:tasks.workunit.client.0.smithi102.stdout:7/828: write d7/d10/d17/f59 [5079997,23087] 0 2024-03-26T18:54:48.447 INFO:tasks.workunit.client.0.smithi102.stdout:6/953: truncate d2/df/f27 1793094 0 2024-03-26T18:54:48.447 INFO:tasks.workunit.client.0.smithi102.stdout:6/954: fsync d2/d7/f54 0 2024-03-26T18:54:48.450 INFO:tasks.workunit.client.0.smithi102.stdout:7/829: mkdir d7/d10/d67/d69/d78/dfa 0 2024-03-26T18:54:48.453 INFO:tasks.workunit.client.0.smithi102.stdout:7/830: mknod d7/d10/d67/d69/d9b/cfb 0 2024-03-26T18:54:48.456 INFO:tasks.workunit.client.0.smithi102.stdout:7/831: rename d7/d10/d67/d69/d98/fea to d7/d10/d67/de1/df4/ffc 0 2024-03-26T18:54:48.459 INFO:tasks.workunit.client.0.smithi102.stdout:7/832: symlink d7/d12/d2b/da8/dca/lfd 0 2024-03-26T18:54:48.460 INFO:tasks.workunit.client.0.smithi102.stdout:1/924: rmdir de/d1f/de3 39 2024-03-26T18:54:48.460 INFO:tasks.workunit.client.0.smithi102.stdout:1/925: write de/d16/d1d/f28 [796571,130919] 0 2024-03-26T18:54:48.460 INFO:tasks.workunit.client.0.smithi102.stdout:1/926: chown de/f14 1 1 2024-03-26T18:54:48.464 INFO:tasks.workunit.client.0.smithi102.stdout:7/833: mknod d7/d10/d27/d34/dcd/cfe 0 2024-03-26T18:54:48.464 INFO:tasks.workunit.client.0.smithi102.stdout:7/834: write d7/d10/d27/d34/d82/fb7 [206899,40095] 0 2024-03-26T18:54:48.465 INFO:tasks.workunit.client.0.smithi102.stdout:9/860: dwrite df/d1a/d23/f71 [4194304,4194304] 0 2024-03-26T18:54:48.466 INFO:tasks.workunit.client.0.smithi102.stdout:1/927: truncate de/f3e 392807 0 2024-03-26T18:54:48.470 INFO:tasks.workunit.client.0.smithi102.stdout:1/928: mknod de/d16/d1d/d61/d6a/df2/c13a 0 2024-03-26T18:54:48.471 INFO:tasks.workunit.client.0.smithi102.stdout:1/929: symlink de/d16/d1d/d2a/d2c/d56/d78/d80/l13b 0 2024-03-26T18:54:48.471 INFO:tasks.workunit.client.0.smithi102.stdout:1/930: creat de/d16/d1d/d2a/d2c/d56/d5f/f13c x:0 0 0 2024-03-26T18:54:48.477 INFO:tasks.workunit.client.0.smithi102.stdout:6/955: dwrite d2/df/d35/d43/fd1 [0,4194304] 0 2024-03-26T18:54:48.478 INFO:tasks.workunit.client.0.smithi102.stdout:7/835: dread d7/d10/d67/fb2 [0,4194304] 0 2024-03-26T18:54:48.485 INFO:tasks.workunit.client.0.smithi102.stdout:9/861: dread df/d46/fb4 [0,4194304] 0 2024-03-26T18:54:48.485 INFO:tasks.workunit.client.0.smithi102.stdout:7/836: rename d7/d12/d2b/da8/dd6 to d7/d10/d67/dff 0 2024-03-26T18:54:48.485 INFO:tasks.workunit.client.0.smithi102.stdout:9/862: write df/d1a/d81/d88/fc9 [1810717,128541] 0 2024-03-26T18:54:48.485 INFO:tasks.workunit.client.0.smithi102.stdout:9/863: creat df/d46/da8/f141 x:0 0 0 2024-03-26T18:54:48.485 INFO:tasks.workunit.client.0.smithi102.stdout:9/864: dread - df/d1a/d65/f11b zero size 2024-03-26T18:54:48.485 INFO:tasks.workunit.client.0.smithi102.stdout:9/865: write df/d46/da8/f141 [461149,122161] 0 2024-03-26T18:54:48.488 INFO:tasks.workunit.client.0.smithi102.stdout:7/837: creat d7/d10/d27/d34/d82/daa/f100 x:0 0 0 2024-03-26T18:54:48.491 INFO:tasks.workunit.client.0.smithi102.stdout:7/838: mkdir d7/d10/d67/de1/df4/d101 0 2024-03-26T18:54:48.491 INFO:tasks.workunit.client.0.smithi102.stdout:7/839: readlink d7/d10/d67/d69/d78/lb8 0 2024-03-26T18:54:48.497 INFO:tasks.workunit.client.0.smithi102.stdout:7/840: rename d7/d12/d4d/f8b to d7/d2a/f102 0 2024-03-26T18:54:48.510 INFO:tasks.workunit.client.0.smithi102.stdout:6/956: dwrite d2/df/d35/d43/d75/f13e [0,4194304] 0 2024-03-26T18:54:48.510 INFO:tasks.workunit.client.0.smithi102.stdout:1/931: dwrite de/d2e/d62/f64 [0,4194304] 0 2024-03-26T18:54:48.522 INFO:tasks.workunit.client.0.smithi102.stdout:6/957: mkdir d2/df/d72/db7/dbd/d140 0 2024-03-26T18:54:48.522 INFO:tasks.workunit.client.0.smithi102.stdout:1/932: dread de/d16/d1d/d2a/d2c/d56/d78/f79 [0,4194304] 0 2024-03-26T18:54:48.526 INFO:tasks.workunit.client.0.smithi102.stdout:1/933: rename de/d16/d1d/d2a/d2c/d56/d6e/f81 to de/d2e/f13d 0 2024-03-26T18:54:48.528 INFO:tasks.workunit.client.0.smithi102.stdout:6/958: dread d2/f52 [0,4194304] 0 2024-03-26T18:54:48.529 INFO:tasks.workunit.client.0.smithi102.stdout:6/959: dread d2/d7/d3c/f41 [0,4194304] 0 2024-03-26T18:54:48.539 INFO:tasks.workunit.client.0.smithi102.stdout:1/934: write de/d16/d1d/d2a/f58 [2764415,32916] 0 2024-03-26T18:54:48.549 INFO:tasks.workunit.client.0.smithi102.stdout:6/960: dwrite d2/d3/f38 [4194304,4194304] 0 2024-03-26T18:54:48.551 INFO:tasks.workunit.client.0.smithi102.stdout:1/935: dread de/d16/d1d/f82 [0,4194304] 0 2024-03-26T18:54:48.551 INFO:tasks.workunit.client.0.smithi102.stdout:1/936: truncate de/d16/d1d/d2a/d2c/d56/dfa/f10a 764185 0 2024-03-26T18:54:48.553 INFO:tasks.workunit.client.0.smithi102.stdout:1/937: mkdir de/d2e/d11e/d13e 0 2024-03-26T18:54:48.553 INFO:tasks.workunit.client.0.smithi102.stdout:6/961: rmdir d2/d3d/d53/d5c 39 2024-03-26T18:54:48.554 INFO:tasks.workunit.client.0.smithi102.stdout:1/938: rmdir de/d16/d1d/d2a/d2c/d56/d6e/dfc/d110 39 2024-03-26T18:54:48.554 INFO:tasks.workunit.client.0.smithi102.stdout:1/939: fdatasync de/d16/d1d/d2a/d2c/d43/f10b 0 2024-03-26T18:54:48.556 INFO:tasks.workunit.client.0.smithi102.stdout:6/962: creat d2/d3d/d53/d77/f141 x:0 0 0 2024-03-26T18:54:48.557 INFO:tasks.workunit.client.0.smithi102.stdout:1/940: symlink de/d16/d1d/d2a/d67/l13f 0 2024-03-26T18:54:48.557 INFO:tasks.workunit.client.0.smithi102.stdout:1/941: chown de/d16/d1d/d2a/d2c/d56/d5f 5384 1 2024-03-26T18:54:48.559 INFO:tasks.workunit.client.0.smithi102.stdout:1/942: link de/d16/d1d/d2a/d2c/d56/d6e/fde de/d16/d1d/d2a/d2c/f140 0 2024-03-26T18:54:48.563 INFO:tasks.workunit.client.0.smithi102.stdout:1/943: link de/d2e/c3d de/d2e/d11e/c141 0 2024-03-26T18:54:48.563 INFO:tasks.workunit.client.0.smithi102.stdout:1/944: chown de/d16/d1d/d61/f112 115 1 2024-03-26T18:54:48.563 INFO:tasks.workunit.client.0.smithi102.stdout:1/945: creat de/d16/d1d/d2a/d2c/d56/d78/dc3/f142 x:0 0 0 2024-03-26T18:54:48.563 INFO:tasks.workunit.client.0.smithi102.stdout:1/946: creat de/d16/d1d/d2a/d2c/dd9/f143 x:0 0 0 2024-03-26T18:54:48.564 INFO:tasks.workunit.client.0.smithi102.stdout:1/947: fsync de/d16/d1d/f4d 0 2024-03-26T18:54:48.564 INFO:tasks.workunit.client.0.smithi102.stdout:1/948: chown de/d16/d1d/d2a/d2c/d56/d78/d80/d8e/f10d 346029 1 2024-03-26T18:54:48.565 INFO:tasks.workunit.client.0.smithi102.stdout:1/949: creat de/d16/d1d/d2a/f144 x:0 0 0 2024-03-26T18:54:48.570 INFO:tasks.workunit.client.0.smithi102.stdout:6/963: dread d2/d3d/fbc [0,4194304] 0 2024-03-26T18:54:48.570 INFO:tasks.workunit.client.0.smithi102.stdout:6/964: truncate d2/df/d35/da1/d13f/d109/f13c 762900 0 2024-03-26T18:54:48.570 INFO:tasks.workunit.client.0.smithi102.stdout:6/965: creat d2/d3d/d53/f142 x:0 0 0 2024-03-26T18:54:48.571 INFO:tasks.workunit.client.0.smithi102.stdout:6/966: rename d2/df/d35/d43/d75/f13e to d2/d3d/d53/db1/de0/f143 0 2024-03-26T18:54:48.572 INFO:tasks.workunit.client.0.smithi102.stdout:6/967: creat d2/d3/dd0/f144 x:0 0 0 2024-03-26T18:54:48.573 INFO:tasks.workunit.client.0.smithi102.stdout:6/968: mkdir d2/df/d35/da1/d13f/d109/d12b/d145 0 2024-03-26T18:54:48.575 INFO:tasks.workunit.client.0.smithi102.stdout:1/950: dread de/d16/d1d/d2a/d2c/d56/d6e/f7a [0,4194304] 0 2024-03-26T18:54:48.576 INFO:tasks.workunit.client.0.smithi102.stdout:1/951: mknod de/d1f/d9f/c145 0 2024-03-26T18:54:48.581 INFO:tasks.workunit.client.0.smithi102.stdout:6/969: write d2/df/f96 [555744,125344] 0 2024-03-26T18:54:48.584 INFO:tasks.workunit.client.0.smithi102.stdout:6/970: creat d2/df/d72/f146 x:0 0 0 2024-03-26T18:54:48.584 INFO:tasks.workunit.client.0.smithi102.stdout:6/971: truncate d2/df/d35/da1/fa5 1162539 0 2024-03-26T18:54:48.593 INFO:tasks.workunit.client.0.smithi102.stdout:1/952: dwrite de/d16/d1d/d2a/d67/f10e [0,4194304] 0 2024-03-26T18:54:48.594 INFO:tasks.workunit.client.0.smithi102.stdout:6/972: read d2/f29 [1981555,119756] 0 2024-03-26T18:54:48.594 INFO:tasks.workunit.client.0.smithi102.stdout:6/973: chown d2/df/d71/d80/d91/f116 558 1 2024-03-26T18:54:48.594 INFO:tasks.workunit.client.0.smithi102.stdout:1/953: write de/d1f/de3/f119 [63724,60719] 0 2024-03-26T18:54:48.602 INFO:tasks.workunit.client.0.smithi102.stdout:1/954: dread de/d16/d1d/d2a/d2c/d56/d5f/fe9 [0,4194304] 0 2024-03-26T18:54:48.602 INFO:tasks.workunit.client.0.smithi102.stdout:1/955: write de/d16/d1d/d2a/d2c/d56/d78/dc3/f125 [216010,40443] 0 2024-03-26T18:54:48.604 INFO:tasks.workunit.client.0.smithi102.stdout:6/974: dread d2/d7/f36 [0,4194304] 0 2024-03-26T18:54:48.606 INFO:tasks.workunit.client.0.smithi102.stdout:1/956: mknod de/d16/c146 0 2024-03-26T18:54:48.606 INFO:tasks.workunit.client.0.smithi102.stdout:1/957: dread - de/d16/d1d/d2a/d2c/d56/d5f/daa/fb4 zero size 2024-03-26T18:54:48.606 INFO:tasks.workunit.client.0.smithi102.stdout:6/975: symlink d2/d3d/d53/d77/l147 0 2024-03-26T18:54:48.607 INFO:tasks.workunit.client.0.smithi102.stdout:1/958: rmdir de/d1f/de3/da6/dca 39 2024-03-26T18:54:48.609 INFO:tasks.workunit.client.0.smithi102.stdout:1/959: creat de/d1f/de3/da6/f147 x:0 0 0 2024-03-26T18:54:48.617 INFO:tasks.workunit.client.0.smithi102.stdout:6/976: getdents d2/d3/dd0 0 2024-03-26T18:54:48.654 INFO:tasks.workunit.client.0.smithi102.stdout:6/977: creat d2/d3d/d53/db1/de0/f148 x:0 0 0 2024-03-26T18:54:48.654 INFO:tasks.workunit.client.0.smithi102.stdout:6/978: unlink d2/c136 0 2024-03-26T18:54:48.654 INFO:tasks.workunit.client.0.smithi102.stdout:6/979: truncate d2/df/d71/d80/d91/db8/f131 863090 0 2024-03-26T18:54:48.654 INFO:tasks.workunit.client.0.smithi102.stdout:6/980: fsync d2/df/dba/fdd 0 2024-03-26T18:54:48.655 INFO:tasks.workunit.client.0.smithi102.stdout:1/960: dwrite de/d16/d1d/d61/d6a/df2/f118 [0,4194304] 0 2024-03-26T18:54:48.655 INFO:tasks.workunit.client.0.smithi102.stdout:1/961: write de/d16/d1d/d2a/d2c/d43/f4f [2929989,115745] 0 2024-03-26T18:54:48.655 INFO:tasks.workunit.client.0.smithi102.stdout:1/962: getdents de/d2e/d123 0 2024-03-26T18:54:48.655 INFO:tasks.workunit.client.0.smithi102.stdout:1/963: read - de/d16/d1d/d2a/d2c/d56/d5f/f13c zero size 2024-03-26T18:54:48.655 INFO:tasks.workunit.client.0.smithi102.stdout:6/981: dread d2/f29 [0,4194304] 0 2024-03-26T18:54:48.655 INFO:tasks.workunit.client.0.smithi102.stdout:1/964: mknod de/d16/d1d/d2a/d2c/d56/d5f/c148 0 2024-03-26T18:54:48.655 INFO:tasks.workunit.client.0.smithi102.stdout:1/965: unlink de/d16/d1d/d2a/d67/f10e 0 2024-03-26T18:54:48.655 INFO:tasks.workunit.client.0.smithi102.stdout:1/966: creat de/d1f/de3/da6/dca/f149 x:0 0 0 2024-03-26T18:54:48.655 INFO:tasks.workunit.client.0.smithi102.stdout:1/967: mknod de/d1f/de3/da6/dca/c14a 0 2024-03-26T18:54:48.657 INFO:tasks.workunit.client.0.smithi102.stdout:6/982: read d2/d3d/d53/f69 [3099096,52151] 0 2024-03-26T18:54:48.657 INFO:tasks.workunit.client.0.smithi102.stdout:1/968: write de/f3e [1287235,70671] 0 2024-03-26T18:54:48.657 INFO:tasks.workunit.client.0.smithi102.stdout:1/969: rename de/d16/d1d/d2a/d2c/d56/d78/d80/l13b to de/d16/l14b 0 2024-03-26T18:54:48.657 INFO:tasks.workunit.client.0.smithi102.stdout:2/890: sync 2024-03-26T18:54:48.658 INFO:tasks.workunit.client.0.smithi102.stdout:6/983: dread d2/df/d71/fbb [0,4194304] 0 2024-03-26T18:54:48.658 INFO:tasks.workunit.client.0.smithi102.stdout:1/970: symlink de/d16/d1d/d2a/d2c/d56/d6e/l14c 0 2024-03-26T18:54:48.658 INFO:tasks.workunit.client.0.smithi102.stdout:2/891: mkdir d3/d2d/d67/d13d 0 2024-03-26T18:54:48.658 INFO:tasks.workunit.client.0.smithi102.stdout:1/971: unlink de/d2e/le8 0 2024-03-26T18:54:48.658 INFO:tasks.workunit.client.0.smithi102.stdout:2/892: mkdir d3/d11/d15/d13e 0 2024-03-26T18:54:48.658 INFO:tasks.workunit.client.0.smithi102.stdout:2/893: fdatasync d3/d20/ff0 0 2024-03-26T18:54:48.661 INFO:tasks.workunit.client.0.smithi102.stdout:2/894: dwrite d3/d20/d78/d7c/fa5 [0,4194304] 0 2024-03-26T18:54:48.663 INFO:tasks.workunit.client.0.smithi102.stdout:2/895: creat d3/d20/d78/d7c/dca/f13f x:0 0 0 2024-03-26T18:54:48.664 INFO:tasks.workunit.client.0.smithi102.stdout:2/896: creat d3/d11/d15/d46/d7b/d97/f140 x:0 0 0 2024-03-26T18:54:48.667 INFO:tasks.workunit.client.0.smithi102.stdout:1/972: dwrite de/d16/d1d/d2a/d2c/d56/d6e/dfc/d110/fef [0,4194304] 0 2024-03-26T18:54:48.668 INFO:tasks.workunit.client.0.smithi102.stdout:1/973: chown de/d16/d1d/d2a/d2c/d56/d5f/daa 490691 1 2024-03-26T18:54:48.668 INFO:tasks.workunit.client.0.smithi102.stdout:1/974: dread - de/d1f/de3/da6/faf zero size 2024-03-26T18:54:48.669 INFO:tasks.workunit.client.0.smithi102.stdout:1/975: mkdir de/d16/d1d/d2a/d2c/d43/d111/d14d 0 2024-03-26T18:54:48.670 INFO:tasks.workunit.client.0.smithi102.stdout:1/976: symlink de/d16/d1d/d2a/l14e 0 2024-03-26T18:54:48.674 INFO:tasks.workunit.client.0.smithi102.stdout:2/897: dread d3/d11/d27/f57 [0,4194304] 0 2024-03-26T18:54:48.674 INFO:tasks.workunit.client.0.smithi102.stdout:2/898: write d3/d82/de2/def/ff4 [214370,94763] 0 2024-03-26T18:54:48.675 INFO:tasks.workunit.client.0.smithi102.stdout:2/899: mkdir d3/d11/d15/d31/d108/d141 0 2024-03-26T18:54:48.676 INFO:tasks.workunit.client.0.smithi102.stdout:2/900: mkdir d3/d11/d15/ded/d142 0 2024-03-26T18:54:48.677 INFO:tasks.workunit.client.0.smithi102.stdout:2/901: stat d3/d20/l111 0 2024-03-26T18:54:48.684 INFO:tasks.workunit.client.0.smithi102.stdout:2/902: dread d3/d11/d15/d31/d39/f64 [0,4194304] 0 2024-03-26T18:54:48.690 INFO:tasks.workunit.client.0.smithi102.stdout:1/977: dwrite de/d1f/f53 [4194304,4194304] 0 2024-03-26T18:54:48.693 INFO:tasks.workunit.client.0.smithi102.stdout:1/978: mkdir de/d16/d1d/d2a/d2c/d56/d6e/d14f 0 2024-03-26T18:54:48.693 INFO:tasks.workunit.client.0.smithi102.stdout:1/979: fsync de/d2e/f55 0 2024-03-26T18:54:48.693 INFO:tasks.workunit.client.0.smithi102.stdout:1/980: stat de/d16/d1d/d2a/d2c/d56/d5f/c76 0 2024-03-26T18:54:48.706 INFO:tasks.workunit.client.0.smithi102.stdout:2/903: dwrite d3/d82/de2/def/ff4 [0,4194304] 0 2024-03-26T18:54:48.706 INFO:tasks.workunit.client.0.smithi102.stdout:2/904: fdatasync d3/d82/d93/d9e/f133 0 2024-03-26T18:54:48.708 INFO:tasks.workunit.client.0.smithi102.stdout:2/905: mknod d3/d20/d78/d84/d6e/c143 0 2024-03-26T18:54:48.708 INFO:tasks.workunit.client.0.smithi102.stdout:2/906: dread - d3/d11/d15/d46/d7b/d97/d10a/f127 zero size 2024-03-26T18:54:48.709 INFO:tasks.workunit.client.0.smithi102.stdout:2/907: mkdir d3/d20/d78/d7c/d144 0 2024-03-26T18:54:48.710 INFO:tasks.workunit.client.0.smithi102.stdout:2/908: symlink d3/d2a/d5f/l145 0 2024-03-26T18:54:48.711 INFO:tasks.workunit.client.0.smithi102.stdout:2/909: symlink d3/d2a/dae/l146 0 2024-03-26T18:54:48.714 INFO:tasks.workunit.client.0.smithi102.stdout:1/981: dwrite de/d2e/f55 [0,4194304] 0 2024-03-26T18:54:48.717 INFO:tasks.workunit.client.0.smithi102.stdout:2/910: rmdir d3/d11/d15/ded 39 2024-03-26T18:54:48.717 INFO:tasks.workunit.client.0.smithi102.stdout:2/911: dread - d3/d11/d15/d31/f13b zero size 2024-03-26T18:54:48.717 INFO:tasks.workunit.client.0.smithi102.stdout:2/912: symlink d3/d2d/d69/l147 0 2024-03-26T18:54:48.720 INFO:tasks.workunit.client.0.smithi102.stdout:2/913: rename d3/d11/d15/d31/d39/cbc to d3/d11/d15/ded/d142/c148 0 2024-03-26T18:54:48.723 INFO:tasks.workunit.client.0.smithi102.stdout:2/914: unlink d3/d20/d78/d84/f11f 0 2024-03-26T18:54:48.723 INFO:tasks.workunit.client.0.smithi102.stdout:1/982: dread de/f2d [0,4194304] 0 2024-03-26T18:54:48.723 INFO:tasks.workunit.client.0.smithi102.stdout:1/983: write de/d16/d1d/d2a/d2c/d43/f86 [5060974,108194] 0 2024-03-26T18:54:48.723 INFO:tasks.workunit.client.0.smithi102.stdout:1/984: write de/d16/d1d/d2a/f144 [598830,76524] 0 2024-03-26T18:54:48.726 INFO:tasks.workunit.client.0.smithi102.stdout:1/985: rename de/d16/d1d/d2a/d67/c116 to de/d16/d1d/d2a/d2c/dd9/c150 0 2024-03-26T18:54:48.739 INFO:tasks.workunit.client.0.smithi102.stdout:2/915: dwrite d3/d82/d93/d9e/d1a/d25/fcd [0,4194304] 0 2024-03-26T18:54:48.739 INFO:tasks.workunit.client.0.smithi102.stdout:2/916: write d3/d20/d78/d84/d6e/dc6/ffa [709767,75653] 0 2024-03-26T18:54:48.744 INFO:tasks.workunit.client.0.smithi102.stdout:4/884: sync 2024-03-26T18:54:48.744 INFO:tasks.workunit.client.0.smithi102.stdout:0/914: sync 2024-03-26T18:54:48.744 INFO:tasks.workunit.client.0.smithi102.stdout:4/885: chown d1/d27/f3c 4 1 2024-03-26T18:54:48.744 INFO:tasks.workunit.client.0.smithi102.stdout:4/886: creat d1/d17/d2a/f12a x:0 0 0 2024-03-26T18:54:48.746 INFO:tasks.workunit.client.0.smithi102.stdout:0/915: creat d6/f13c x:0 0 0 2024-03-26T18:54:48.750 INFO:tasks.workunit.client.0.smithi102.stdout:2/917: dread d3/f1c [0,4194304] 0 2024-03-26T18:54:48.751 INFO:tasks.workunit.client.0.smithi102.stdout:0/916: mkdir d6/d3e/d104/d11a/d13d 0 2024-03-26T18:54:48.754 INFO:tasks.workunit.client.0.smithi102.stdout:2/918: unlink d3/d2d/c5b 0 2024-03-26T18:54:48.756 INFO:tasks.workunit.client.0.smithi102.stdout:0/917: mkdir d6/d3e/d104/d13e 0 2024-03-26T18:54:48.756 INFO:tasks.workunit.client.0.smithi102.stdout:0/918: readlink d6/de/d13/d9a/l86 0 2024-03-26T18:54:48.757 INFO:tasks.workunit.client.0.smithi102.stdout:2/919: symlink d3/d20/d78/d7c/dca/l149 0 2024-03-26T18:54:48.757 INFO:tasks.workunit.client.0.smithi102.stdout:0/919: mknod d6/de/d13/d108/c13f 0 2024-03-26T18:54:48.760 INFO:tasks.workunit.client.0.smithi102.stdout:2/920: mknod d3/d2d/c14a 0 2024-03-26T18:54:48.761 INFO:tasks.workunit.client.0.smithi102.stdout:2/921: symlink d3/d82/d93/d9e/l14b 0 2024-03-26T18:54:48.761 INFO:tasks.workunit.client.0.smithi102.stdout:2/922: chown d3/d2a/d5f/f90 18969 1 2024-03-26T18:54:48.761 INFO:tasks.workunit.client.0.smithi102.stdout:2/923: read - d3/d11/d27/dc0/fff zero size 2024-03-26T18:54:48.761 INFO:tasks.workunit.client.0.smithi102.stdout:2/924: stat d3/d2d/l42 0 2024-03-26T18:54:48.761 INFO:tasks.workunit.client.0.smithi102.stdout:2/925: fsync d3/d82/d93/d9e/f121 0 2024-03-26T18:54:48.762 INFO:tasks.workunit.client.0.smithi102.stdout:2/926: rename d3/d2a/f2f to d3/d82/d93/d9e/f14c 0 2024-03-26T18:54:48.763 INFO:tasks.workunit.client.0.smithi102.stdout:2/927: creat d3/d11/f14d x:0 0 0 2024-03-26T18:54:48.769 INFO:tasks.workunit.client.0.smithi102.stdout:4/887: dwrite d1/d27/d98/f6a [0,4194304] 0 2024-03-26T18:54:48.772 INFO:tasks.workunit.client.0.smithi102.stdout:2/928: dread d3/f4 [0,4194304] 0 2024-03-26T18:54:48.772 INFO:tasks.workunit.client.0.smithi102.stdout:4/888: creat d1/d113/f12b x:0 0 0 2024-03-26T18:54:48.778 INFO:tasks.workunit.client.0.smithi102.stdout:0/920: dwrite d6/de/d13/d25/fc5 [0,4194304] 0 2024-03-26T18:54:48.785 INFO:tasks.workunit.client.0.smithi102.stdout:0/921: dread d6/de/d13/d25/d4f/d66/fb7 [0,4194304] 0 2024-03-26T18:54:48.787 INFO:tasks.workunit.client.0.smithi102.stdout:0/922: dread d6/de/d13/d9a/f5b [8388608,4194304] 0 2024-03-26T18:54:48.793 INFO:tasks.workunit.client.0.smithi102.stdout:2/929: dwrite d3/d11/d15/ded/ffc [0,4194304] 0 2024-03-26T18:54:48.797 INFO:tasks.workunit.client.0.smithi102.stdout:4/889: dwrite d1/d75/d85/fbe [0,4194304] 0 2024-03-26T18:54:48.797 INFO:tasks.workunit.client.0.smithi102.stdout:4/890: readlink d1/d17/d1f/d5d/d6f/dc0/lf7 0 2024-03-26T18:54:48.798 INFO:tasks.workunit.client.0.smithi102.stdout:2/930: rename d3/f75 to d3/d11/d15/d31/d7f/dfe/f14e 0 2024-03-26T18:54:48.800 INFO:tasks.workunit.client.0.smithi102.stdout:4/891: symlink d1/d17/dbb/dee/d10c/l12c 0 2024-03-26T18:54:48.800 INFO:tasks.workunit.client.0.smithi102.stdout:4/892: read - d1/d17/d2a/d9a/dc2/f105 zero size 2024-03-26T18:54:48.800 INFO:tasks.workunit.client.0.smithi102.stdout:4/893: dread - d1/d17/d2a/d9a/ffd zero size 2024-03-26T18:54:48.800 INFO:tasks.workunit.client.0.smithi102.stdout:4/894: write d1/d17/d1f/d5d/d6f/fac [1429571,55130] 0 2024-03-26T18:54:48.800 INFO:tasks.workunit.client.0.smithi102.stdout:4/895: write d1/d17/d2a/d9a/ddc/f128 [621048,71130] 0 2024-03-26T18:54:48.801 INFO:tasks.workunit.client.0.smithi102.stdout:4/896: readlink d1/d17/d1a/d21/d77/la5 0 2024-03-26T18:54:48.801 INFO:tasks.workunit.client.0.smithi102.stdout:0/923: write d6/d3e/d104/d11a/f61 [4009583,114310] 0 2024-03-26T18:54:48.801 INFO:tasks.workunit.client.0.smithi102.stdout:2/931: creat d3/d11/d15/d31/d7f/f14f x:0 0 0 2024-03-26T18:54:48.803 INFO:tasks.workunit.client.0.smithi102.stdout:2/932: rename d3/d11/c18 to d3/d82/d93/c150 0 2024-03-26T18:54:48.809 INFO:tasks.workunit.client.0.smithi102.stdout:0/924: dread d6/de/d13/d25/d31/d45/f65 [0,4194304] 0 2024-03-26T18:54:48.809 INFO:tasks.workunit.client.0.smithi102.stdout:0/925: readlink d6/de/l2e 0 2024-03-26T18:54:48.809 INFO:tasks.workunit.client.0.smithi102.stdout:0/926: chown d6/de/d13/d1f/d71/c126 1683003681 1 2024-03-26T18:54:48.817 INFO:tasks.workunit.client.0.smithi102.stdout:0/927: write d6/de/d13/d25/d4f/d66/fb7 [8169799,90836] 0 2024-03-26T18:54:48.817 INFO:tasks.workunit.client.0.smithi102.stdout:2/933: read d3/d2a/f4f [190201,121274] 0 2024-03-26T18:54:48.826 INFO:tasks.workunit.client.0.smithi102.stdout:4/897: dwrite d1/d17/dbb/d126/f108 [0,4194304] 0 2024-03-26T18:54:48.828 INFO:tasks.workunit.client.0.smithi102.stdout:2/934: write d3/d20/d78/d7c/f85 [7877496,74733] 0 2024-03-26T18:54:48.836 INFO:tasks.workunit.client.0.smithi102.stdout:0/928: dwrite d6/de/d13/d108/f121 [0,4194304] 0 2024-03-26T18:54:48.843 INFO:tasks.workunit.client.0.smithi102.stdout:0/929: symlink d6/de/d13/d25/l140 0 2024-03-26T18:54:48.847 INFO:tasks.workunit.client.0.smithi102.stdout:2/935: dwrite d3/d2a/f34 [0,4194304] 0 2024-03-26T18:54:48.849 INFO:tasks.workunit.client.0.smithi102.stdout:4/898: dwrite d1/d17/d1a/d21/f95 [0,4194304] 0 2024-03-26T18:54:48.849 INFO:tasks.workunit.client.0.smithi102.stdout:4/899: fsync d1/d17/d1a/d21/f95 0 2024-03-26T18:54:48.850 INFO:tasks.workunit.client.0.smithi102.stdout:4/900: dread - d1/d17/d1a/d21/d77/ff9 zero size 2024-03-26T18:54:48.850 INFO:tasks.workunit.client.0.smithi102.stdout:2/936: unlink d3/d82/d93/d9e/d99/c11b 0 2024-03-26T18:54:48.850 INFO:tasks.workunit.client.0.smithi102.stdout:2/937: truncate d3/d82/d93/fa4 990430 0 2024-03-26T18:54:48.854 INFO:tasks.workunit.client.0.smithi102.stdout:2/938: mknod d3/d11/d15/d31/d7f/c151 0 2024-03-26T18:54:48.857 INFO:tasks.workunit.client.0.smithi102.stdout:7/841: sync 2024-03-26T18:54:48.857 INFO:tasks.workunit.client.0.smithi102.stdout:7/842: getdents d7/d12/d2b/dbd 0 2024-03-26T18:54:48.857 INFO:tasks.workunit.client.0.smithi102.stdout:9/866: sync 2024-03-26T18:54:48.857 INFO:tasks.workunit.client.0.smithi102.stdout:6/984: sync 2024-03-26T18:54:48.857 INFO:tasks.workunit.client.0.smithi102.stdout:6/985: truncate d2/d3d/d53/db1/de0/f148 761091 0 2024-03-26T18:54:48.857 INFO:tasks.workunit.client.0.smithi102.stdout:6/986: chown d2/da4/c113 26412 1 2024-03-26T18:54:48.858 INFO:tasks.workunit.client.0.smithi102.stdout:9/867: write df/d1a/d7f/de1/f10c [306700,33393] 0 2024-03-26T18:54:48.858 INFO:tasks.workunit.client.0.smithi102.stdout:1/986: sync 2024-03-26T18:54:48.858 INFO:tasks.workunit.client.0.smithi102.stdout:1/987: fsync de/d16/d1d/d2a/d2c/d56/d5f/fe9 0 2024-03-26T18:54:48.858 INFO:tasks.workunit.client.0.smithi102.stdout:1/988: write de/d16/d1d/d2a/d2c/d56/d5f/daa/fb4 [640023,12317] 0 2024-03-26T18:54:48.858 INFO:tasks.workunit.client.0.smithi102.stdout:1/989: chown de/d16/d1d/d2a/d2c/d56/d78/d80/d8e/f96 15 1 2024-03-26T18:54:48.859 INFO:tasks.workunit.client.0.smithi102.stdout:1/990: mkdir de/d1f/de3/da6/dca/d151 0 2024-03-26T18:54:48.859 INFO:tasks.workunit.client.0.smithi102.stdout:1/991: fdatasync de/d16/d1d/d61/d6a/f7d 0 2024-03-26T18:54:48.860 INFO:tasks.workunit.client.0.smithi102.stdout:7/843: getdents d7/d10/d67 0 2024-03-26T18:54:48.860 INFO:tasks.workunit.client.0.smithi102.stdout:7/844: chown d7/d10/d17 111 1 2024-03-26T18:54:48.860 INFO:tasks.workunit.client.0.smithi102.stdout:7/845: fsync d7/d10/d17/f35 0 2024-03-26T18:54:48.860 INFO:tasks.workunit.client.0.smithi102.stdout:7/846: creat d7/d12/f103 x:0 0 0 2024-03-26T18:54:48.862 INFO:tasks.workunit.client.0.smithi102.stdout:6/987: getdents d2/df/d71/d80/d91/db8/df9 0 2024-03-26T18:54:48.862 INFO:tasks.workunit.client.0.smithi102.stdout:9/868: getdents df/d1a/d23/d3e/d75/d117 0 2024-03-26T18:54:48.867 INFO:tasks.workunit.client.0.smithi102.stdout:9/869: symlink df/d1a/d81/dcc/l142 0 2024-03-26T18:54:48.867 INFO:tasks.workunit.client.0.smithi102.stdout:1/992: dread de/d16/d1d/d2a/d2c/d56/d5f/daa/fb4 [0,4194304] 0 2024-03-26T18:54:48.870 INFO:tasks.workunit.client.0.smithi102.stdout:9/870: symlink df/d1a/d23/d3e/d75/l143 0 2024-03-26T18:54:48.871 INFO:tasks.workunit.client.0.smithi102.stdout:9/871: truncate df/d1a/d65/dbe/fc0 179854 0 2024-03-26T18:54:48.871 INFO:tasks.workunit.client.0.smithi102.stdout:9/872: creat df/d46/df7/f144 x:0 0 0 2024-03-26T18:54:48.871 INFO:tasks.workunit.client.0.smithi102.stdout:9/873: chown df/d1a/d23/d3e/f99 1552 1 2024-03-26T18:54:48.871 INFO:tasks.workunit.client.0.smithi102.stdout:9/874: chown df/f40 6412 1 2024-03-26T18:54:48.871 INFO:tasks.workunit.client.0.smithi102.stdout:1/993: truncate de/d16/d1d/d2a/d2c/d56/d6e/dfc/d110/ffb 7422777 0 2024-03-26T18:54:48.872 INFO:tasks.workunit.client.0.smithi102.stdout:1/994: chown de/d16/d1d/d2a/d2c/d56/d78/d80/d8e/lbf 0 1 2024-03-26T18:54:48.875 INFO:tasks.workunit.client.0.smithi102.stdout:1/995: unlink de/d16/d1d/f28 0 2024-03-26T18:54:48.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:48.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:48.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:54:48.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:54:48.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:48.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:54:48.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr fail", "who": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:54:48.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: osdmap e48: 6 total, 6 up, 6 in 2024-03-26T18:54:48.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "mgr fail", "who": "smithi102.rsfpnv"}]': finished 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: mgrmap e21: smithi190.zrixos(active, starting, since 0.0162971s) 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mgr metadata", "who": "smithi190.zrixos", "id": "smithi190.zrixos"}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:54:48.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:54:48.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:54:48.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:54:48.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: Manager daemon smithi190.zrixos is now available 2024-03-26T18:54:48.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:54:48.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:54:48.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.zrixos/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:54:48.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.zrixos/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:54:48.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.zrixos/trash_purge_schedule"}]: dispatch 2024-03-26T18:54:48.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:48 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.zrixos/trash_purge_schedule"}]: dispatch 2024-03-26T18:54:48.890 INFO:tasks.workunit.client.0.smithi102.stdout:6/988: dwrite d2/df/d71/d80/d91/fcc [0,4194304] 0 2024-03-26T18:54:48.892 INFO:tasks.workunit.client.0.smithi102.stdout:9/875: dread df/d1a/d23/d29/f31 [0,4194304] 0 2024-03-26T18:54:48.892 INFO:tasks.workunit.client.0.smithi102.stdout:2/939: dwrite d3/d20/d78/d7c/f7e [4194304,4194304] 0 2024-03-26T18:54:48.892 INFO:tasks.workunit.client.0.smithi102.stdout:9/876: fdatasync df/d1a/d7f/fad 0 2024-03-26T18:54:48.892 INFO:tasks.workunit.client.0.smithi102.stdout:2/940: fsync d3/d11/d15/d46/f47 0 2024-03-26T18:54:48.892 INFO:tasks.workunit.client.0.smithi102.stdout:9/877: chown df/d1a/d81/d12f/c140 18 1 2024-03-26T18:54:48.892 INFO:tasks.workunit.client.0.smithi102.stdout:7/847: dwrite d7/ff [0,4194304] 0 2024-03-26T18:54:48.893 INFO:tasks.workunit.client.0.smithi102.stdout:9/878: dread df/d1a/d23/d3e/d75/fbd [0,4194304] 0 2024-03-26T18:54:48.893 INFO:tasks.workunit.client.0.smithi102.stdout:6/989: link d2/d3d/d53/d63/lae d2/df/d35/d60/l149 0 2024-03-26T18:54:48.896 INFO:tasks.workunit.client.0.smithi102.stdout:2/941: creat d3/d20/d78/d7c/d144/f152 x:0 0 0 2024-03-26T18:54:48.897 INFO:tasks.workunit.client.0.smithi102.stdout:7/848: mkdir d7/d104 0 2024-03-26T18:54:48.900 INFO:tasks.workunit.client.0.smithi102.stdout:6/990: truncate d2/df/d71/d80/ff2 244402 0 2024-03-26T18:54:48.900 INFO:tasks.workunit.client.0.smithi102.stdout:6/991: getdents d2/df/d35/d43/d75/dfb 0 2024-03-26T18:54:48.901 INFO:tasks.workunit.client.0.smithi102.stdout:1/996: dwrite de/d1f/f6d [0,4194304] 0 2024-03-26T18:54:48.901 INFO:tasks.workunit.client.0.smithi102.stdout:7/849: creat d7/d10/d27/da3/dc6/f105 x:0 0 0 2024-03-26T18:54:48.903 INFO:tasks.workunit.client.0.smithi102.stdout:6/992: dread d2/df/d35/d43/f103 [0,4194304] 0 2024-03-26T18:54:48.907 INFO:tasks.workunit.client.0.smithi102.stdout:2/942: write d3/d82/d93/fcb [90437,38083] 0 2024-03-26T18:54:48.907 INFO:tasks.workunit.client.0.smithi102.stdout:2/943: fsync d3/d82/de2/def/ff6 0 2024-03-26T18:54:48.909 INFO:tasks.workunit.client.0.smithi102.stdout:7/850: dread d7/d10/d17/f5c [0,4194304] 0 2024-03-26T18:54:48.911 INFO:tasks.workunit.client.0.smithi102.stdout:9/879: rmdir df/d1a/d1d 39 2024-03-26T18:54:48.911 INFO:tasks.workunit.client.0.smithi102.stdout:9/880: chown df/d1a/l11c 0 1 2024-03-26T18:54:48.912 INFO:tasks.workunit.client.0.smithi102.stdout:1/997: write de/d16/d1d/d61/d6a/df2/f118 [10004,17234] 0 2024-03-26T18:54:48.913 INFO:tasks.workunit.client.0.smithi102.stdout:1/998: chown de/c23 0 1 2024-03-26T18:54:48.913 INFO:tasks.workunit.client.0.smithi102.stdout:1/999: chown de/f15 183 1 2024-03-26T18:54:48.913 INFO:tasks.workunit.client.0.smithi102.stdout:6/993: dread d2/df/d71/d80/d91/db8/f131 [0,4194304] 0 2024-03-26T18:54:48.913 INFO:tasks.workunit.client.0.smithi102.stdout:6/994: fdatasync d2/df/d35/da1/ffa 0 2024-03-26T18:54:48.918 INFO:tasks.workunit.client.0.smithi102.stdout:7/851: unlink d7/d31/d56/fa9 0 2024-03-26T18:54:48.918 INFO:tasks.workunit.client.0.smithi102.stdout:7/852: creat d7/d10/d67/d69/d98/f106 x:0 0 0 2024-03-26T18:54:48.920 INFO:tasks.workunit.client.0.smithi102.stdout:7/853: mknod d7/d10/d27/da3/dc6/c107 0 2024-03-26T18:54:48.920 INFO:tasks.workunit.client.0.smithi102.stdout:7/854: truncate d7/d10/d17/db9/fbe 898217 0 2024-03-26T18:54:48.920 INFO:tasks.workunit.client.0.smithi102.stdout:7/855: chown d7/d44/l73 4848 1 2024-03-26T18:54:48.924 INFO:tasks.workunit.client.0.smithi102.stdout:9/881: write df/d1a/d65/fae [1712438,92375] 0 2024-03-26T18:54:48.927 INFO:tasks.workunit.client.0.smithi102.stdout:7/856: dread d7/d10/d67/fb2 [0,4194304] 0 2024-03-26T18:54:48.928 INFO:tasks.workunit.client.0.smithi102.stdout:9/882: rename df/d1a/f6f to df/d1a/d1d/d72/d7d/db7/d13c/f145 0 2024-03-26T18:54:48.930 INFO:tasks.workunit.client.0.smithi102.stdout:6/995: dwrite d2/d3d/f8a [0,4194304] 0 2024-03-26T18:54:48.931 INFO:tasks.workunit.client.0.smithi102.stdout:2/944: dwrite d3/d2d/d69/fbd [0,4194304] 0 2024-03-26T18:54:48.934 INFO:tasks.workunit.client.0.smithi102.stdout:9/883: creat df/d1a/d23/d60/dd4/f146 x:0 0 0 2024-03-26T18:54:48.934 INFO:tasks.workunit.client.0.smithi102.stdout:9/884: write df/f82 [764081,53697] 0 2024-03-26T18:54:48.938 INFO:tasks.workunit.client.0.smithi102.stdout:2/945: unlink d3/d20/d78/d84/fda 0 2024-03-26T18:54:48.939 INFO:tasks.workunit.client.0.smithi102.stdout:9/885: creat df/d1a/d81/d12f/f147 x:0 0 0 2024-03-26T18:54:48.939 INFO:tasks.workunit.client.0.smithi102.stdout:9/886: readlink df/d84/l103 0 2024-03-26T18:54:48.941 INFO:tasks.workunit.client.0.smithi102.stdout:2/946: symlink d3/l153 0 2024-03-26T18:54:48.943 INFO:tasks.workunit.client.0.smithi102.stdout:2/947: link d3/d82/d93/d9e/f24 d3/d11/d15/ded/f154 0 2024-03-26T18:54:48.943 INFO:tasks.workunit.client.0.smithi102.stdout:2/948: chown d3/d11/d15/d46/d7b/d97/f140 16663958 1 2024-03-26T18:54:48.945 INFO:tasks.workunit.client.0.smithi102.stdout:2/949: mkdir d3/d11/d27/d155 0 2024-03-26T18:54:48.945 INFO:tasks.workunit.client.0.smithi102.stdout:2/950: read - d3/d11/d15/d31/d39/f109 zero size 2024-03-26T18:54:48.945 INFO:tasks.workunit.client.0.smithi102.stdout:2/951: read - d3/d11/d15/d46/d7b/d97/d10a/f127 zero size 2024-03-26T18:54:48.945 INFO:tasks.workunit.client.0.smithi102.stdout:9/887: write df/d46/fb4 [2433139,44305] 0 2024-03-26T18:54:48.945 INFO:tasks.workunit.client.0.smithi102.stdout:9/888: write df/d1a/d1d/d72/d7d/db7/dd5/fe3 [532616,80532] 0 2024-03-26T18:54:48.945 INFO:tasks.workunit.client.0.smithi102.stdout:9/889: chown df/d1a/d81/d12f/c140 3514 1 2024-03-26T18:54:48.950 INFO:tasks.workunit.client.0.smithi102.stdout:7/857: dwrite d7/d2a/f4a [4194304,4194304] 0 2024-03-26T18:54:48.952 INFO:tasks.workunit.client.0.smithi102.stdout:6/996: dwrite d2/df/d71/d80/d91/f116 [0,4194304] 0 2024-03-26T18:54:48.952 INFO:tasks.workunit.client.0.smithi102.stdout:6/997: readlink d2/d3d/leb 0 2024-03-26T18:54:48.952 INFO:tasks.workunit.client.0.smithi102.stdout:7/858: symlink d7/d12/d4b/d83/l108 0 2024-03-26T18:54:48.956 INFO:tasks.workunit.client.0.smithi102.stdout:7/859: truncate d7/d12/f8d 866241 0 2024-03-26T18:54:48.960 INFO:tasks.workunit.client.0.smithi102.stdout:2/952: rename d3/d20/d78 to d3/d2a/d5f/d156 0 2024-03-26T18:54:48.962 INFO:tasks.workunit.client.0.smithi102.stdout:6/998: dread d2/df/d35/d43/fd1 [0,4194304] 0 2024-03-26T18:54:48.962 INFO:tasks.workunit.client.0.smithi102.stdout:6/999: write d2/d3d/d53/d77/fa3 [636025,39796] 0 2024-03-26T18:54:48.962 INFO:tasks.workunit.client.0.smithi102.stdout:9/890: dwrite df/d1a/d7f/fad [0,4194304] 0 2024-03-26T18:54:48.963 INFO:tasks.workunit.client.0.smithi102.stdout:7/860: rename d7/fb to d7/d10/d67/d69/d9b/dae/f109 0 2024-03-26T18:54:48.963 INFO:tasks.workunit.client.0.smithi102.stdout:7/861: chown d7/d10/d27/d34/d82/cf1 8029704 1 2024-03-26T18:54:48.965 INFO:tasks.workunit.client.0.smithi102.stdout:2/953: mkdir d3/d11/d15/d31/d39/d157 0 2024-03-26T18:54:48.972 INFO:tasks.workunit.client.0.smithi102.stdout:7/862: dread d7/d10/d67/d69/d9b/fdc [0,4194304] 0 2024-03-26T18:54:48.972 INFO:tasks.workunit.client.0.smithi102.stdout:7/863: chown d7/d12/d4b/d83 48919264 1 2024-03-26T18:54:48.972 INFO:tasks.workunit.client.0.smithi102.stdout:7/864: readlink d7/d10/d67/d69/ldd 0 2024-03-26T18:54:48.977 INFO:tasks.workunit.client.0.smithi102.stdout:9/891: mkdir df/d1a/d7f/d148 0 2024-03-26T18:54:48.982 INFO:tasks.workunit.client.0.smithi102.stdout:2/954: symlink d3/d82/d93/l158 0 2024-03-26T18:54:48.982 INFO:tasks.workunit.client.0.smithi102.stdout:2/955: creat d3/d2a/d5f/d156/d84/d6e/dc6/f159 x:0 0 0 2024-03-26T18:54:48.985 INFO:tasks.workunit.client.0.smithi102.stdout:9/892: rmdir df/d1a/d81/d88 39 2024-03-26T18:54:48.986 INFO:tasks.workunit.client.0.smithi102.stdout:9/893: mknod df/d1a/d65/dbe/ddf/c149 0 2024-03-26T18:54:48.990 INFO:tasks.workunit.client.0.smithi102.stdout:7/865: dwrite d7/d12/d51/d6a/fcf [0,4194304] 0 2024-03-26T18:54:48.992 INFO:tasks.workunit.client.0.smithi102.stdout:0/930: sync 2024-03-26T18:54:48.992 INFO:tasks.workunit.client.0.smithi102.stdout:4/901: sync 2024-03-26T18:54:48.992 INFO:tasks.workunit.client.0.smithi102.stdout:0/931: fsync d6/de/d13/d1f/d55/d57/f30 0 2024-03-26T18:54:48.993 INFO:tasks.workunit.client.0.smithi102.stdout:7/866: creat d7/d12/d2b/da8/f10a x:0 0 0 2024-03-26T18:54:48.994 INFO:tasks.workunit.client.0.smithi102.stdout:7/867: symlink d7/d12/d4d/d62/d95/l10b 0 2024-03-26T18:54:49.000 INFO:tasks.workunit.client.0.smithi102.stdout:0/932: read d6/d3e/d87/fea [3198156,59353] 0 2024-03-26T18:54:49.001 INFO:tasks.workunit.client.0.smithi102.stdout:0/933: symlink d6/d3e/d69/d8c/l141 0 2024-03-26T18:54:49.001 INFO:tasks.workunit.client.0.smithi102.stdout:0/934: fdatasync d6/dbe/deb/d109/f10c 0 2024-03-26T18:54:49.002 INFO:tasks.workunit.client.0.smithi102.stdout:0/935: rename d6/de/d13/d25/d31/d45/d82/d84/d88/d124 to d6/d3e/d142 0 2024-03-26T18:54:49.003 INFO:tasks.workunit.client.0.smithi102.stdout:0/936: mknod d6/d3e/d142/c143 0 2024-03-26T18:54:49.004 INFO:tasks.workunit.client.0.smithi102.stdout:0/937: symlink d6/de/d13/d1f/d55/d57/d7d/d7f/dc8/dd3/l144 0 2024-03-26T18:54:49.010 INFO:tasks.workunit.client.0.smithi102.stdout:7/868: dread d7/d2a/f74 [0,4194304] 0 2024-03-26T18:54:49.010 INFO:tasks.workunit.client.0.smithi102.stdout:7/869: write d7/d2a/f30 [2289094,37009] 0 2024-03-26T18:54:49.011 INFO:tasks.workunit.client.0.smithi102.stdout:7/870: getdents d7/d10/d67/de1/df4 0 2024-03-26T18:54:49.012 INFO:tasks.workunit.client.0.smithi102.stdout:7/871: mkdir d7/d2a/d10c 0 2024-03-26T18:54:49.012 INFO:tasks.workunit.client.0.smithi102.stdout:7/872: creat d7/d10/d27/d34/dd3/f10d x:0 0 0 2024-03-26T18:54:49.013 INFO:tasks.workunit.client.0.smithi102.stdout:4/902: dwrite d1/d17/d1a/d21/f64 [0,4194304] 0 2024-03-26T18:54:49.014 INFO:tasks.workunit.client.0.smithi102.stdout:4/903: mkdir d1/d6b/de7/d12d 0 2024-03-26T18:54:49.015 INFO:tasks.workunit.client.0.smithi102.stdout:4/904: getdents d1/d17/dbb/dee 0 2024-03-26T18:54:49.020 INFO:tasks.workunit.client.0.smithi102.stdout:9/894: dwrite df/d1a/d1d/d72/f126 [0,4194304] 0 2024-03-26T18:54:49.020 INFO:tasks.workunit.client.0.smithi102.stdout:9/895: write df/d1a/d65/f11b [554677,1382] 0 2024-03-26T18:54:49.020 INFO:tasks.workunit.client.0.smithi102.stdout:9/896: creat df/d1a/d23/d3e/d75/f14a x:0 0 0 2024-03-26T18:54:49.020 INFO:tasks.workunit.client.0.smithi102.stdout:9/897: creat df/d1a/d81/dcc/f14b x:0 0 0 2024-03-26T18:54:49.023 INFO:tasks.workunit.client.0.smithi102.stdout:7/873: write d7/d10/d67/d69/d9b/dae/f109 [1004186,42150] 0 2024-03-26T18:54:49.023 INFO:tasks.workunit.client.0.smithi102.stdout:7/874: write d7/d31/d56/fd4 [278865,46910] 0 2024-03-26T18:54:49.024 INFO:tasks.workunit.client.0.smithi102.stdout:9/898: truncate df/d1a/d1d/d72/d7d/db7/df8/f125 3320183 0 2024-03-26T18:54:49.024 INFO:tasks.workunit.client.0.smithi102.stdout:9/899: readlink df/d1a/d23/d4e/d8e/l95 0 2024-03-26T18:54:49.026 INFO:tasks.workunit.client.0.smithi102.stdout:9/900: mkdir df/d1a/d14c 0 2024-03-26T18:54:49.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:49.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:49.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:54:49.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:54:49.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:54:49.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:54:49.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr fail", "who": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:54:49.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: osdmap e48: 6 total, 6 up, 6 in 2024-03-26T18:54:49.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.14211 172.21.15.102:0/709643198' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "mgr fail", "who": "smithi102.rsfpnv"}]': finished 2024-03-26T18:54:49.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: mgrmap e21: smithi190.zrixos(active, starting, since 0.0162971s) 2024-03-26T18:54:49.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:54:49.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:54:49.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T18:54:49.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T18:54:49.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T18:54:49.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T18:54:49.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mgr metadata", "who": "smithi190.zrixos", "id": "smithi190.zrixos"}]: dispatch 2024-03-26T18:54:49.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T18:54:49.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T18:54:49.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T18:54:49.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:54:49.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:54:49.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:54:49.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:54:49.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:54:49.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:54:49.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: Manager daemon smithi190.zrixos is now available 2024-03-26T18:54:49.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:54:49.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:54:49.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.zrixos/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:54:49.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.zrixos/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:54:49.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.zrixos/trash_purge_schedule"}]: dispatch 2024-03-26T18:54:49.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:48 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.zrixos/trash_purge_schedule"}]: dispatch 2024-03-26T18:54:49.037 INFO:tasks.workunit.client.0.smithi102.stdout:9/901: dread df/d1a/d23/f5d [0,4194304] 0 2024-03-26T18:54:49.044 INFO:tasks.workunit.client.0.smithi102.stdout:7/875: dwrite d7/d10/d67/d69/f9c [0,4194304] 0 2024-03-26T18:54:49.047 INFO:tasks.workunit.client.0.smithi102.stdout:7/876: symlink d7/d12/d4d/l10e 0 2024-03-26T18:54:49.047 INFO:tasks.workunit.client.0.smithi102.stdout:7/877: fdatasync d7/d31/f45 0 2024-03-26T18:54:49.049 INFO:tasks.workunit.client.0.smithi102.stdout:9/902: dread df/d1a/d23/d60/f74 [0,4194304] 0 2024-03-26T18:54:49.049 INFO:tasks.workunit.client.0.smithi102.stdout:9/903: stat df/d1a 0 2024-03-26T18:54:49.049 INFO:tasks.workunit.client.0.smithi102.stdout:9/904: chown df/d1a/d65/dbe 2589993 1 2024-03-26T18:54:49.070 INFO:tasks.workunit.client.0.smithi102.stdout:7/878: dwrite d7/d10/d67/d69/d98/f106 [0,4194304] 0 2024-03-26T18:54:49.071 INFO:tasks.workunit.client.0.smithi102.stdout:9/905: dwrite df/d46/da8/fbb [0,4194304] 0 2024-03-26T18:54:49.072 INFO:tasks.workunit.client.0.smithi102.stdout:7/879: creat d7/d10/d17/f10f x:0 0 0 2024-03-26T18:54:49.073 INFO:tasks.workunit.client.0.smithi102.stdout:7/880: read - d7/d12/d51/d6a/fd7 zero size 2024-03-26T18:54:49.074 INFO:tasks.workunit.client.0.smithi102.stdout:9/906: creat df/d1a/d1d/d72/d7d/dda/f14d x:0 0 0 2024-03-26T18:54:49.094 INFO:tasks.workunit.client.0.smithi102.stdout:7/881: dwrite d7/d10/d67/de1/df4/ffc [0,4194304] 0 2024-03-26T18:54:49.094 INFO:tasks.workunit.client.0.smithi102.stdout:7/882: chown d7/d10/d17/db9/fbe 13 1 2024-03-26T18:54:49.094 INFO:tasks.workunit.client.0.smithi102.stdout:9/907: dwrite df/d1a/d23/fb2 [0,4194304] 0 2024-03-26T18:54:49.095 INFO:tasks.workunit.client.0.smithi102.stdout:7/883: mkdir d7/d10/d27/d34/dcd/d110 0 2024-03-26T18:54:49.095 INFO:tasks.workunit.client.0.smithi102.stdout:7/884: write d7/d44/f53 [374767,27475] 0 2024-03-26T18:54:49.097 INFO:tasks.workunit.client.0.smithi102.stdout:9/908: mknod df/d1a/d23/d4e/dc1/c14e 0 2024-03-26T18:54:49.097 INFO:tasks.workunit.client.0.smithi102.stdout:9/909: write df/d1a/d1d/f138 [956073,76391] 0 2024-03-26T18:54:49.097 INFO:tasks.workunit.client.0.smithi102.stdout:7/885: symlink d7/d12/d2b/dbd/l111 0 2024-03-26T18:54:49.098 INFO:tasks.workunit.client.0.smithi102.stdout:9/910: mknod df/d46/df7/c14f 0 2024-03-26T18:54:49.099 INFO:tasks.workunit.client.0.smithi102.stdout:7/886: link d7/d10/d27/d34/d82/daa/f100 d7/d12/d4b/f112 0 2024-03-26T18:54:49.101 INFO:tasks.workunit.client.0.smithi102.stdout:7/887: mknod d7/d12/d51/c113 0 2024-03-26T18:54:49.102 INFO:tasks.workunit.client.0.smithi102.stdout:7/888: creat d7/d10/f114 x:0 0 0 2024-03-26T18:54:49.104 INFO:tasks.workunit.client.0.smithi102.stdout:7/889: mknod d7/d10/d67/d69/d9b/dae/c115 0 2024-03-26T18:54:49.104 INFO:tasks.workunit.client.0.smithi102.stdout:7/890: getdents d7/d10/d27/d34/d72/dee 0 2024-03-26T18:54:49.105 INFO:tasks.workunit.client.0.smithi102.stdout:9/911: write df/d1a/d7f/fad [3875989,76646] 0 2024-03-26T18:54:49.105 INFO:tasks.workunit.client.0.smithi102.stdout:9/912: chown df/d1a/d23/d4e/d8e/lf2 321 1 2024-03-26T18:54:49.108 INFO:tasks.workunit.client.0.smithi102.stdout:7/891: symlink d7/d12/d4d/l116 0 2024-03-26T18:54:49.110 INFO:tasks.workunit.client.0.smithi102.stdout:9/913: truncate df/d1a/d23/f25 2916856 0 2024-03-26T18:54:49.117 INFO:tasks.workunit.client.0.smithi102.stdout:7/892: dread d7/f20 [0,4194304] 0 2024-03-26T18:54:49.117 INFO:tasks.workunit.client.0.smithi102.stdout:7/893: rename d7/d12/d2b/d50 to d7/d12/d2b/d50/d117 22 2024-03-26T18:54:49.121 INFO:tasks.workunit.client.0.smithi102.stdout:4/905: sync 2024-03-26T18:54:49.121 INFO:tasks.workunit.client.0.smithi102.stdout:0/938: sync 2024-03-26T18:54:49.121 INFO:tasks.workunit.client.0.smithi102.stdout:4/906: chown d1/d25/d90/d11b 0 1 2024-03-26T18:54:49.121 INFO:tasks.workunit.client.0.smithi102.stdout:4/907: read - d1/d27/d98/f8e zero size 2024-03-26T18:54:49.121 INFO:tasks.workunit.client.0.smithi102.stdout:2/956: sync 2024-03-26T18:54:49.122 INFO:tasks.workunit.client.0.smithi102.stdout:0/939: creat d6/de/d13/d25/d31/d45/d82/f145 x:0 0 0 2024-03-26T18:54:49.124 INFO:tasks.workunit.client.0.smithi102.stdout:2/957: creat d3/f15a x:0 0 0 2024-03-26T18:54:49.126 INFO:tasks.workunit.client.0.smithi102.stdout:0/940: mknod d6/dbc/c146 0 2024-03-26T18:54:49.128 INFO:tasks.workunit.client.0.smithi102.stdout:2/958: mkdir d3/d2a/d5f/d156/d7c/dca/d15b 0 2024-03-26T18:54:49.129 INFO:tasks.workunit.client.0.smithi102.stdout:0/941: symlink d6/de/d13/d25/d4f/d66/l147 0 2024-03-26T18:54:49.131 INFO:tasks.workunit.client.0.smithi102.stdout:2/959: creat d3/d82/d93/d9e/da0/f15c x:0 0 0 2024-03-26T18:54:49.131 INFO:tasks.workunit.client.0.smithi102.stdout:2/960: chown d3/d11/d15/c36 84 1 2024-03-26T18:54:49.131 INFO:tasks.workunit.client.0.smithi102.stdout:9/914: dwrite df/fa6 [0,4194304] 0 2024-03-26T18:54:49.135 INFO:tasks.workunit.client.0.smithi102.stdout:2/961: unlink d3/d20/c2c 0 2024-03-26T18:54:49.136 INFO:tasks.workunit.client.0.smithi102.stdout:9/915: dread df/d1a/d1d/d72/d7d/fb9 [0,4194304] 0 2024-03-26T18:54:49.136 INFO:tasks.workunit.client.0.smithi102.stdout:9/916: fsync df/d1a/d1d/d72/f126 0 2024-03-26T18:54:49.136 INFO:tasks.workunit.client.0.smithi102.stdout:2/962: creat d3/d11/d15/d31/d108/f15d x:0 0 0 2024-03-26T18:54:49.137 INFO:tasks.workunit.client.0.smithi102.stdout:7/894: dwrite d7/d12/d4b/d83/fc4 [0,4194304] 0 2024-03-26T18:54:49.137 INFO:tasks.workunit.client.0.smithi102.stdout:7/895: truncate d7/d10/d27/d34/d82/daa/f100 594881 0 2024-03-26T18:54:49.137 INFO:tasks.workunit.client.0.smithi102.stdout:7/896: truncate d7/d10/d17/d9f/fe3 788342 0 2024-03-26T18:54:49.137 INFO:tasks.workunit.client.0.smithi102.stdout:7/897: creat d7/d12/d51/dd0/f118 x:0 0 0 2024-03-26T18:54:49.137 INFO:tasks.workunit.client.0.smithi102.stdout:9/917: mkdir df/d1a/d81/d12f/d150 0 2024-03-26T18:54:49.138 INFO:tasks.workunit.client.0.smithi102.stdout:2/963: mkdir d3/d82/d93/d9e/da0/d15e 0 2024-03-26T18:54:49.138 INFO:tasks.workunit.client.0.smithi102.stdout:2/964: chown d3/d11/d15/d46/d7b/fe6 18079 1 2024-03-26T18:54:49.138 INFO:tasks.workunit.client.0.smithi102.stdout:2/965: chown d3/d2a/d5f/d156 102010 1 2024-03-26T18:54:49.139 INFO:tasks.workunit.client.0.smithi102.stdout:2/966: fdatasync d3/d2d/f68 0 2024-03-26T18:54:49.141 INFO:tasks.workunit.client.0.smithi102.stdout:7/898: mknod d7/d10/d27/c119 0 2024-03-26T18:54:49.142 INFO:tasks.workunit.client.0.smithi102.stdout:9/918: symlink df/d46/da8/l151 0 2024-03-26T18:54:49.152 INFO:tasks.workunit.client.0.smithi102.stdout:4/908: dwrite d1/d17/d1f/d5d/d8b/f4e [0,4194304] 0 2024-03-26T18:54:49.152 INFO:tasks.workunit.client.0.smithi102.stdout:7/899: dread d7/d10/d67/fa5 [0,4194304] 0 2024-03-26T18:54:49.153 INFO:tasks.workunit.client.0.smithi102.stdout:0/942: dwrite d6/d2d/d3d/f123 [0,4194304] 0 2024-03-26T18:54:49.154 INFO:tasks.workunit.client.0.smithi102.stdout:2/967: dread d3/d2a/d5f/d156/d84/d6e/dc6/fd1 [4194304,4194304] 0 2024-03-26T18:54:49.155 INFO:tasks.workunit.client.0.smithi102.stdout:7/900: symlink d7/d10/d27/d34/l11a 0 2024-03-26T18:54:49.158 INFO:tasks.workunit.client.0.smithi102.stdout:0/943: mkdir d6/de/d13/d9a/d148 0 2024-03-26T18:54:49.160 INFO:tasks.workunit.client.0.smithi102.stdout:2/968: dread d3/f1c [0,4194304] 0 2024-03-26T18:54:49.160 INFO:tasks.workunit.client.0.smithi102.stdout:2/969: creat d3/d2a/d5f/d156/d84/d6e/dc6/f15f x:0 0 0 2024-03-26T18:54:49.162 INFO:tasks.workunit.client.0.smithi102.stdout:0/944: creat d6/dbe/deb/dfd/f149 x:0 0 0 2024-03-26T18:54:49.164 INFO:tasks.workunit.client.0.smithi102.stdout:9/919: dwrite df/d1a/d23/d29/f34 [0,4194304] 0 2024-03-26T18:54:49.167 INFO:tasks.workunit.client.0.smithi102.stdout:0/945: dread d6/de/d13/d1f/d55/d57/f30 [0,4194304] 0 2024-03-26T18:54:49.169 INFO:tasks.workunit.client.0.smithi102.stdout:4/909: write d1/f26 [2542619,31570] 0 2024-03-26T18:54:49.175 INFO:tasks.workunit.client.0.smithi102.stdout:0/946: creat d6/de/d13/d25/d31/d45/d82/d84/d88/db2/f14a x:0 0 0 2024-03-26T18:54:49.176 INFO:tasks.workunit.client.0.smithi102.stdout:4/910: mknod d1/d17/d1f/d5d/d8b/d3e/d74/c12e 0 2024-03-26T18:54:49.177 INFO:tasks.workunit.client.0.smithi102.stdout:0/947: truncate d6/de/d13/d1f/d55/d57/d7d/fd5 3361613 0 2024-03-26T18:54:49.177 INFO:tasks.workunit.client.0.smithi102.stdout:0/948: chown d6/de/d13/d25/d31/d45/d82/d84/d88/f106 752487 1 2024-03-26T18:54:49.194 INFO:tasks.workunit.client.0.smithi102.stdout:7/901: dwrite d7/d10/d17/f37 [0,4194304] 0 2024-03-26T18:54:49.195 INFO:tasks.workunit.client.0.smithi102.stdout:7/902: creat d7/d2a/de8/f11b x:0 0 0 2024-03-26T18:54:49.196 INFO:tasks.workunit.client.0.smithi102.stdout:7/903: symlink d7/d10/d17/db9/l11c 0 2024-03-26T18:54:49.196 INFO:tasks.workunit.client.0.smithi102.stdout:7/904: dread - d7/d12/d2b/d50/fcb zero size 2024-03-26T18:54:49.197 INFO:tasks.workunit.client.0.smithi102.stdout:7/905: symlink d7/d10/d67/l11d 0 2024-03-26T18:54:49.197 INFO:tasks.workunit.client.0.smithi102.stdout:7/906: read - d7/d10/d27/d34/d72/fb3 zero size 2024-03-26T18:54:49.198 INFO:tasks.workunit.client.0.smithi102.stdout:2/970: dwrite d3/d82/d93/fbf [0,4194304] 0 2024-03-26T18:54:49.198 INFO:tasks.workunit.client.0.smithi102.stdout:9/920: dwrite df/d1a/d1d/f107 [4194304,4194304] 0 2024-03-26T18:54:49.198 INFO:tasks.workunit.client.0.smithi102.stdout:9/921: readlink df/d1a/d1d/d72/d7d/db7/df8/l13d 0 2024-03-26T18:54:49.198 INFO:tasks.workunit.client.0.smithi102.stdout:7/907: creat d7/d4e/f11e x:0 0 0 2024-03-26T18:54:49.200 INFO:tasks.workunit.client.0.smithi102.stdout:4/911: dwrite d1/d75/f117 [0,4194304] 0 2024-03-26T18:54:49.201 INFO:tasks.workunit.client.0.smithi102.stdout:0/949: dwrite d6/f13c [0,4194304] 0 2024-03-26T18:54:49.201 INFO:tasks.workunit.client.0.smithi102.stdout:2/971: symlink d3/d11/d15/d31/d7f/dfe/l160 0 2024-03-26T18:54:49.201 INFO:tasks.workunit.client.0.smithi102.stdout:2/972: dread - d3/d2a/d5f/d156/d84/d6e/dc6/f124 zero size 2024-03-26T18:54:49.206 INFO:tasks.workunit.client.0.smithi102.stdout:0/950: mkdir d6/d3e/d104/d11a/d13d/d14b 0 2024-03-26T18:54:49.207 INFO:tasks.workunit.client.0.smithi102.stdout:2/973: unlink d3/d82/d93/lf9 0 2024-03-26T18:54:49.207 INFO:tasks.workunit.client.0.smithi102.stdout:2/974: fsync d3/d82/fb3 0 2024-03-26T18:54:49.208 INFO:tasks.workunit.client.0.smithi102.stdout:2/975: getdents d3/d82/de2 0 2024-03-26T18:54:49.211 INFO:tasks.workunit.client.0.smithi102.stdout:2/976: rename d3/d11/d15/d31/d7f/dfe/f138 to d3/d2d/d67/d13d/f161 0 2024-03-26T18:54:49.211 INFO:tasks.workunit.client.0.smithi102.stdout:2/977: mknod d3/d74/c162 0 2024-03-26T18:54:49.212 INFO:tasks.workunit.client.0.smithi102.stdout:4/912: dread d1/d75/d85/fbe [0,4194304] 0 2024-03-26T18:54:49.213 INFO:tasks.workunit.client.0.smithi102.stdout:4/913: symlink d1/d6b/de7/d12d/l12f 0 2024-03-26T18:54:49.214 INFO:tasks.workunit.client.0.smithi102.stdout:4/914: truncate d1/d27/f5f 1897258 0 2024-03-26T18:54:49.216 INFO:tasks.workunit.client.0.smithi102.stdout:4/915: rename d1/d25/d90/l116 to d1/d17/d1f/d5d/l130 0 2024-03-26T18:54:49.218 INFO:tasks.workunit.client.0.smithi102.stdout:4/916: mkdir d1/d17/d1f/d5d/d6f/dc0/d131 0 2024-03-26T18:54:49.218 INFO:tasks.workunit.client.0.smithi102.stdout:4/917: stat d1/cd 0 2024-03-26T18:54:49.218 INFO:tasks.workunit.client.0.smithi102.stdout:4/918: fdatasync d1/d17/d1f/d5d/f118 0 2024-03-26T18:54:49.218 INFO:tasks.workunit.client.0.smithi102.stdout:4/919: truncate d1/fdb 776924 0 2024-03-26T18:54:49.219 INFO:tasks.workunit.client.0.smithi102.stdout:4/920: rmdir d1/d17/d1f/d5d/d6f/dc0 39 2024-03-26T18:54:49.219 INFO:tasks.workunit.client.0.smithi102.stdout:4/921: chown d1/d17/d1f/d5d/d8b/c129 0 1 2024-03-26T18:54:49.222 INFO:tasks.workunit.client.0.smithi102.stdout:0/951: dread d6/de/d13/d25/da4/fee [0,4194304] 0 2024-03-26T18:54:49.222 INFO:tasks.workunit.client.0.smithi102.stdout:0/952: dread - d6/d3e/d69/d8c/d8f/f117 zero size 2024-03-26T18:54:49.223 INFO:tasks.workunit.client.0.smithi102.stdout:0/953: dread d6/de/d13/d1f/d55/d57/f30 [0,4194304] 0 2024-03-26T18:54:49.225 INFO:tasks.workunit.client.0.smithi102.stdout:0/954: truncate d6/de/d13/d1f/d55/d57/f1c 4177147 0 2024-03-26T18:54:49.225 INFO:tasks.workunit.client.0.smithi102.stdout:0/955: creat d6/dbe/deb/d109/f14c x:0 0 0 2024-03-26T18:54:49.230 INFO:tasks.workunit.client.0.smithi102.stdout:4/922: dread d1/d17/d1a/f57 [0,4194304] 0 2024-03-26T18:54:49.230 INFO:tasks.workunit.client.0.smithi102.stdout:0/956: creat d6/dbe/deb/f14d x:0 0 0 2024-03-26T18:54:49.230 INFO:tasks.workunit.client.0.smithi102.stdout:0/957: creat d6/d3e/d69/f14e x:0 0 0 2024-03-26T18:54:49.233 INFO:tasks.workunit.client.0.smithi102.stdout:9/922: dwrite df/d1a/d65/f8a [0,4194304] 0 2024-03-26T18:54:49.237 INFO:tasks.workunit.client.0.smithi102.stdout:9/923: rename df/d1a/d23/d4e/f4f to df/d1a/d65/dbe/f152 0 2024-03-26T18:54:49.238 INFO:tasks.workunit.client.0.smithi102.stdout:2/978: dwrite d3/d2d/d67/d13d/f161 [0,4194304] 0 2024-03-26T18:54:49.239 INFO:tasks.workunit.client.0.smithi102.stdout:9/924: truncate df/fca 448451 0 2024-03-26T18:54:49.239 INFO:tasks.workunit.client.0.smithi102.stdout:9/925: creat df/d1a/d65/f153 x:0 0 0 2024-03-26T18:54:49.242 INFO:tasks.workunit.client.0.smithi102.stdout:2/979: symlink d3/d11/d15/l163 0 2024-03-26T18:54:49.243 INFO:tasks.workunit.client.0.smithi102.stdout:0/958: dread d6/de/d13/d25/d31/f4e [0,4194304] 0 2024-03-26T18:54:49.243 INFO:tasks.workunit.client.0.smithi102.stdout:0/959: write d6/de/d13/d25/d4f/d66/f95 [1662815,99755] 0 2024-03-26T18:54:49.243 INFO:tasks.workunit.client.0.smithi102.stdout:0/960: readlink d6/de/d13/d1f/d55/d57/d7d/d7f/dc8/dd3/l144 0 2024-03-26T18:54:49.246 INFO:tasks.workunit.client.0.smithi102.stdout:9/926: creat df/d1a/d1d/f154 x:0 0 0 2024-03-26T18:54:49.246 INFO:tasks.workunit.client.0.smithi102.stdout:9/927: dread - df/d46/da8/f11e zero size 2024-03-26T18:54:49.248 INFO:tasks.workunit.client.0.smithi102.stdout:2/980: mknod d3/d11/d15/ded/c164 0 2024-03-26T18:54:49.248 INFO:tasks.workunit.client.0.smithi102.stdout:2/981: readlink d3/d11/d15/l87 0 2024-03-26T18:54:49.248 INFO:tasks.workunit.client.0.smithi102.stdout:2/982: dread - d3/d2a/d5f/d156/d84/d6e/dc6/f15f zero size 2024-03-26T18:54:49.248 INFO:tasks.workunit.client.0.smithi102.stdout:2/983: stat d3/d82/f106 0 2024-03-26T18:54:49.248 INFO:tasks.workunit.client.0.smithi102.stdout:2/984: write d3/d11/d15/d46/d7b/d97/fad [1319819,70847] 0 2024-03-26T18:54:49.250 INFO:tasks.workunit.client.0.smithi102.stdout:0/961: symlink d6/de/d13/d1f/d71/l14f 0 2024-03-26T18:54:49.250 INFO:tasks.workunit.client.0.smithi102.stdout:0/962: creat d6/de/d13/d1f/d55/d57/d7d/d7f/dc8/f150 x:0 0 0 2024-03-26T18:54:49.250 INFO:tasks.workunit.client.0.smithi102.stdout:0/963: stat d6/d2d/d3d/le6 0 2024-03-26T18:54:49.255 INFO:tasks.workunit.client.0.smithi102.stdout:2/985: write d3/d11/d27/f6b [4344157,56478] 0 2024-03-26T18:54:49.255 INFO:tasks.workunit.client.0.smithi102.stdout:2/986: fdatasync d3/d11/d15/d31/fa3 0 2024-03-26T18:54:49.269 INFO:tasks.workunit.client.0.smithi102.stdout:0/964: dwrite d6/de/d13/d25/d31/d45/d82/d84/d88/fab [0,4194304] 0 2024-03-26T18:54:49.280 INFO:tasks.workunit.client.0.smithi102.stdout:2/987: dwrite d3/d82/d93/d9e/fa [4194304,4194304] 0 2024-03-26T18:54:49.287 INFO:tasks.workunit.client.0.smithi102.stdout:0/965: dwrite d6/d2d/d3d/fa0 [4194304,4194304] 0 2024-03-26T18:54:49.287 INFO:tasks.workunit.client.0.smithi102.stdout:2/988: dread d3/d20/fe8 [0,4194304] 0 2024-03-26T18:54:49.311 INFO:tasks.workunit.client.0.smithi102.stdout:2/989: dwrite d3/d11/d15/d46/f47 [0,4194304] 0 2024-03-26T18:54:49.314 INFO:tasks.workunit.client.0.smithi102.stdout:2/990: unlink d3/d2d/d69/f6a 0 2024-03-26T18:54:49.314 INFO:tasks.workunit.client.0.smithi102.stdout:2/991: write d3/d82/f11d [1423155,28406] 0 2024-03-26T18:54:49.343 INFO:tasks.workunit.client.0.smithi102.stdout:2/992: dwrite d3/d11/d27/dc0/fff [0,4194304] 0 2024-03-26T18:54:49.344 INFO:tasks.workunit.client.0.smithi102.stdout:2/993: dread d3/d2a/d5f/d156/d84/f98 [0,4194304] 0 2024-03-26T18:54:49.344 INFO:tasks.workunit.client.0.smithi102.stdout:2/994: dread - d3/d82/de2/ff5 zero size 2024-03-26T18:54:49.347 INFO:tasks.workunit.client.0.smithi102.stdout:2/995: mknod d3/d11/d10c/c165 0 2024-03-26T18:54:49.369 INFO:tasks.workunit.client.0.smithi102.stdout:2/996: dwrite d3/d2a/f6d [4194304,4194304] 0 2024-03-26T18:54:49.371 INFO:tasks.workunit.client.0.smithi102.stdout:2/997: unlink d3/d20/fe8 0 2024-03-26T18:54:49.372 INFO:tasks.workunit.client.0.smithi102.stdout:2/998: symlink d3/d11/d27/l166 0 2024-03-26T18:54:49.393 INFO:tasks.workunit.client.0.smithi102.stdout:2/999: dwrite d3/d20/f4c [0,4194304] 0 2024-03-26T18:54:49.411 INFO:tasks.workunit.client.0.smithi102.stdout:7/908: sync 2024-03-26T18:54:49.413 INFO:tasks.workunit.client.0.smithi102.stdout:7/909: dread d7/d12/d2b/d50/fa7 [0,4194304] 0 2024-03-26T18:54:49.433 INFO:tasks.workunit.client.0.smithi102.stdout:7/910: dwrite d7/d10/d27/d34/d82/daa/f100 [0,4194304] 0 2024-03-26T18:54:49.451 INFO:tasks.workunit.client.0.smithi102.stdout:7/911: dwrite d7/d10/d27/d34/d72/fb3 [0,4194304] 0 2024-03-26T18:54:49.451 INFO:tasks.workunit.client.0.smithi102.stdout:7/912: write d7/d31/f45 [860469,26471] 0 2024-03-26T18:54:49.459 INFO:tasks.workunit.client.0.smithi102.stdout:7/913: dread d7/d10/fe7 [0,4194304] 0 2024-03-26T18:54:49.460 INFO:tasks.workunit.client.0.smithi102.stdout:7/914: mkdir d7/d12/d51/dd0/d11f 0 2024-03-26T18:54:49.460 INFO:tasks.workunit.client.0.smithi102.stdout:7/915: creat d7/d10/d27/d34/dd3/f120 x:0 0 0 2024-03-26T18:54:49.463 INFO:tasks.workunit.client.0.smithi102.stdout:9/928: sync 2024-03-26T18:54:49.463 INFO:tasks.workunit.client.0.smithi102.stdout:0/966: sync 2024-03-26T18:54:49.463 INFO:tasks.workunit.client.0.smithi102.stdout:4/923: sync 2024-03-26T18:54:49.463 INFO:tasks.workunit.client.0.smithi102.stdout:9/929: readlink df/d1a/d23/d29/l41 0 2024-03-26T18:54:49.464 INFO:tasks.workunit.client.0.smithi102.stdout:0/967: creat d6/d3e/f151 x:0 0 0 2024-03-26T18:54:49.466 INFO:tasks.workunit.client.0.smithi102.stdout:4/924: symlink d1/d17/d1a/d21/de8/l132 0 2024-03-26T18:54:49.468 INFO:tasks.workunit.client.0.smithi102.stdout:0/968: rename d6/l10 to d6/dbe/deb/d109/l152 0 2024-03-26T18:54:49.469 INFO:tasks.workunit.client.0.smithi102.stdout:4/925: mkdir d1/d75/d85/d133 0 2024-03-26T18:54:49.471 INFO:tasks.workunit.client.0.smithi102.stdout:4/926: unlink d1/d17/d1f/d5d/d8b/de5/ced 0 2024-03-26T18:54:49.480 INFO:tasks.workunit.client.0.smithi102.stdout:0/969: read d6/de/d13/d1f/d55/d57/d7d/d7f/f115 [221629,108352] 0 2024-03-26T18:54:49.480 INFO:tasks.workunit.client.0.smithi102.stdout:0/970: write d6/dbe/deb/dfd/f149 [609410,32135] 0 2024-03-26T18:54:49.481 INFO:tasks.workunit.client.0.smithi102.stdout:0/971: mknod d6/de/d13/d25/da4/c153 0 2024-03-26T18:54:49.482 INFO:tasks.workunit.client.0.smithi102.stdout:0/972: getdents d6/de/d13/d1f/d55/d57 0 2024-03-26T18:54:49.482 INFO:tasks.workunit.client.0.smithi102.stdout:0/973: chown d6/de/d13/d1f/fa8 1 1 2024-03-26T18:54:49.485 INFO:tasks.workunit.client.0.smithi102.stdout:0/974: dread d6/de/d13/d25/d31/d45/d82/f8a [0,4194304] 0 2024-03-26T18:54:49.485 INFO:tasks.workunit.client.0.smithi102.stdout:0/975: creat d6/d2d/d3d/d80/d83/f154 x:0 0 0 2024-03-26T18:54:49.485 INFO:tasks.workunit.client.0.smithi102.stdout:0/976: dread - d6/de/d13/d1f/fd9 zero size 2024-03-26T18:54:49.487 INFO:tasks.workunit.client.0.smithi102.stdout:0/977: mknod d6/de/d13/d1f/d55/c155 0 2024-03-26T18:54:49.487 INFO:tasks.workunit.client.0.smithi102.stdout:0/978: fsync d6/de/d13/d25/d4f/d66/f95 0 2024-03-26T18:54:49.487 INFO:tasks.workunit.client.0.smithi102.stdout:0/979: creat d6/f156 x:0 0 0 2024-03-26T18:54:49.487 INFO:tasks.workunit.client.0.smithi102.stdout:0/980: stat d6/de/f28 0 2024-03-26T18:54:49.487 INFO:tasks.workunit.client.0.smithi102.stdout:0/981: dread - d6/de/d13/d25/d31/d45/d82/d84/f12f zero size 2024-03-26T18:54:49.487 INFO:tasks.workunit.client.0.smithi102.stdout:0/982: chown d6/dbe/cf8 1205 1 2024-03-26T18:54:49.498 INFO:tasks.workunit.client.0.smithi102.stdout:0/983: write d6/de/d13/d25/da4/d10b/f120 [2409309,41199] 0 2024-03-26T18:54:49.499 INFO:tasks.workunit.client.0.smithi102.stdout:7/916: dwrite d7/d12/d4d/f88 [0,4194304] 0 2024-03-26T18:54:49.499 INFO:tasks.workunit.client.0.smithi102.stdout:7/917: dread - d7/d2a/d9d/ff9 zero size 2024-03-26T18:54:49.499 INFO:tasks.workunit.client.0.smithi102.stdout:7/918: readlink d7/d12/d2b/dbd/l111 0 2024-03-26T18:54:49.499 INFO:tasks.workunit.client.0.smithi102.stdout:4/927: dwrite d1/d17/d2a/f54 [4194304,4194304] 0 2024-03-26T18:54:49.500 INFO:tasks.workunit.client.0.smithi102.stdout:9/930: dwrite df/d1a/d7f/fba [0,4194304] 0 2024-03-26T18:54:49.500 INFO:tasks.workunit.client.0.smithi102.stdout:9/931: truncate df/d1a/d1d/d72/d7d/db7/df8/f125 4181007 0 2024-03-26T18:54:49.500 INFO:tasks.workunit.client.0.smithi102.stdout:9/932: truncate df/d1a/f10b 1237051 0 2024-03-26T18:54:49.500 INFO:tasks.workunit.client.0.smithi102.stdout:9/933: write df/d46/da8/f141 [999851,78026] 0 2024-03-26T18:54:49.501 INFO:tasks.workunit.client.0.smithi102.stdout:7/919: rename d7/d12/d4b/d83/l89 to d7/d10/d27/d34/dd3/l121 0 2024-03-26T18:54:49.501 INFO:tasks.workunit.client.0.smithi102.stdout:7/920: chown d7/d2a/f30 56 1 2024-03-26T18:54:49.502 INFO:tasks.workunit.client.0.smithi102.stdout:4/928: mkdir d1/d17/d1a/d134 0 2024-03-26T18:54:49.503 INFO:tasks.workunit.client.0.smithi102.stdout:9/934: symlink df/d1a/d23/d60/l155 0 2024-03-26T18:54:49.504 INFO:tasks.workunit.client.0.smithi102.stdout:7/921: stat d7/d12/d2b/c70 0 2024-03-26T18:54:49.507 INFO:tasks.workunit.client.0.smithi102.stdout:4/929: mkdir d1/d17/d1f/d5d/d8b/d3e/d74/d135 0 2024-03-26T18:54:49.507 INFO:tasks.workunit.client.0.smithi102.stdout:4/930: dread - d1/d17/d1a/d21/f79 zero size 2024-03-26T18:54:49.509 INFO:tasks.workunit.client.0.smithi102.stdout:9/935: rename df/f17 to df/d1a/d7f/f156 0 2024-03-26T18:54:49.510 INFO:tasks.workunit.client.0.smithi102.stdout:0/984: write d6/de/d13/d1f/d55/d57/d7d/d7f/f8b [3372104,91921] 0 2024-03-26T18:54:49.510 INFO:tasks.workunit.client.0.smithi102.stdout:4/931: creat d1/d6b/de7/d12d/f136 x:0 0 0 2024-03-26T18:54:49.511 INFO:tasks.workunit.client.0.smithi102.stdout:4/932: creat d1/d17/d1a/f137 x:0 0 0 2024-03-26T18:54:49.512 INFO:tasks.workunit.client.0.smithi102.stdout:9/936: rmdir df/d1a/d1d/d72 39 2024-03-26T18:54:49.513 INFO:tasks.workunit.client.0.smithi102.stdout:4/933: dread d1/d17/d1a/d21/f63 [0,4194304] 0 2024-03-26T18:54:49.516 INFO:tasks.workunit.client.0.smithi102.stdout:0/985: creat d6/de/d13/f157 x:0 0 0 2024-03-26T18:54:49.516 INFO:tasks.workunit.client.0.smithi102.stdout:0/986: write d6/de/fd0 [295087,22660] 0 2024-03-26T18:54:49.520 INFO:tasks.workunit.client.0.smithi102.stdout:9/937: unlink df/d1a/d1d/cf3 0 2024-03-26T18:54:49.521 INFO:tasks.workunit.client.0.smithi102.stdout:0/987: mknod d6/d3e/d104/d11a/d13d/c158 0 2024-03-26T18:54:49.522 INFO:tasks.workunit.client.0.smithi102.stdout:9/938: creat df/d1a/d81/d88/f157 x:0 0 0 2024-03-26T18:54:49.523 INFO:tasks.workunit.client.0.smithi102.stdout:0/988: mknod d6/de/d13/d1f/d71/d125/c159 0 2024-03-26T18:54:49.533 INFO:tasks.workunit.client.0.smithi102.stdout:7/922: dwrite d7/d10/d17/db9/fbe [0,4194304] 0 2024-03-26T18:54:49.533 INFO:tasks.workunit.client.0.smithi102.stdout:7/923: fsync d7/d10/d67/d69/d98/fda 0 2024-03-26T18:54:49.534 INFO:tasks.workunit.client.0.smithi102.stdout:4/934: dwrite d1/d17/d1f/d5d/d8b/f51 [4194304,4194304] 0 2024-03-26T18:54:49.537 INFO:tasks.workunit.client.0.smithi102.stdout:0/989: read d6/d3e/d104/d11a/fe5 [1278131,87458] 0 2024-03-26T18:54:49.537 INFO:tasks.workunit.client.0.smithi102.stdout:0/990: dread - d6/de/d13/d25/d31/d45/d82/f145 zero size 2024-03-26T18:54:49.539 INFO:tasks.workunit.client.0.smithi102.stdout:7/924: getdents d7/d10/d27/d34/d72 0 2024-03-26T18:54:49.539 INFO:tasks.workunit.client.0.smithi102.stdout:7/925: write d7/d10/d67/d69/d98/fdf [803368,126609] 0 2024-03-26T18:54:49.540 INFO:tasks.workunit.client.0.smithi102.stdout:4/935: getdents d1/d17/d2a/d9a 0 2024-03-26T18:54:49.542 INFO:tasks.workunit.client.0.smithi102.stdout:4/936: creat d1/de1/f138 x:0 0 0 2024-03-26T18:54:49.543 INFO:tasks.workunit.client.0.smithi102.stdout:4/937: write d1/d27/ff0 [4657023,21959] 0 2024-03-26T18:54:49.543 INFO:tasks.workunit.client.0.smithi102.stdout:0/991: getdents d6/d2d/d3d/d80 0 2024-03-26T18:54:49.545 INFO:tasks.workunit.client.0.smithi102.stdout:7/926: rmdir d7/d12/d2b/da8 39 2024-03-26T18:54:49.549 INFO:tasks.workunit.client.0.smithi102.stdout:4/938: mknod d1/d17/d1f/d5d/d8b/d3e/c139 0 2024-03-26T18:54:49.552 INFO:tasks.workunit.client.0.smithi102.stdout:0/992: symlink d6/d2d/l15a 0 2024-03-26T18:54:49.562 INFO:tasks.workunit.client.0.smithi102.stdout:7/927: dwrite d7/d2a/f74 [4194304,4194304] 0 2024-03-26T18:54:49.566 INFO:tasks.workunit.client.0.smithi102.stdout:7/928: creat d7/d10/d67/d69/d78/dfa/f122 x:0 0 0 2024-03-26T18:54:49.567 INFO:tasks.workunit.client.0.smithi102.stdout:7/929: chown d7/d10/d67/de1 7 1 2024-03-26T18:54:49.567 INFO:tasks.workunit.client.0.smithi102.stdout:7/930: mknod d7/d12/d4b/c123 0 2024-03-26T18:54:49.567 INFO:tasks.workunit.client.0.smithi102.stdout:7/931: chown d7/d12/d4d 1012423801 1 2024-03-26T18:54:49.568 INFO:tasks.workunit.client.0.smithi102.stdout:7/932: unlink d7/d2a/de8/f11b 0 2024-03-26T18:54:49.568 INFO:tasks.workunit.client.0.smithi102.stdout:7/933: fsync d7/d10/f114 0 2024-03-26T18:54:49.569 INFO:tasks.workunit.client.0.smithi102.stdout:7/934: symlink d7/d12/d4b/d83/l124 0 2024-03-26T18:54:49.569 INFO:tasks.workunit.client.0.smithi102.stdout:7/935: fdatasync d7/d12/d4b/f112 0 2024-03-26T18:54:49.569 INFO:tasks.workunit.client.0.smithi102.stdout:7/936: creat d7/d12/d4d/f125 x:0 0 0 2024-03-26T18:54:49.580 INFO:tasks.workunit.client.0.smithi102.stdout:4/939: dwrite d1/d17/d1f/d5d/d8b/d3e/f58 [0,4194304] 0 2024-03-26T18:54:49.581 INFO:tasks.workunit.client.0.smithi102.stdout:0/993: dwrite d6/d2d/d3d/d80/d83/f154 [0,4194304] 0 2024-03-26T18:54:49.583 INFO:tasks.workunit.client.0.smithi102.stdout:7/937: truncate d7/d10/d67/d69/d98/fdf 539775 0 2024-03-26T18:54:49.584 INFO:tasks.workunit.client.0.smithi102.stdout:7/938: mknod d7/d12/d4b/d83/c126 0 2024-03-26T18:54:49.584 INFO:tasks.workunit.client.0.smithi102.stdout:7/939: creat d7/d44/f127 x:0 0 0 2024-03-26T18:54:49.584 INFO:tasks.workunit.client.0.smithi102.stdout:7/940: creat d7/d10/d27/da3/f128 x:0 0 0 2024-03-26T18:54:49.587 INFO:tasks.workunit.client.0.smithi102.stdout:7/941: mknod d7/d12/d2b/da8/c129 0 2024-03-26T18:54:49.587 INFO:tasks.workunit.client.0.smithi102.stdout:7/942: stat d7/d12/d4b/d83 0 2024-03-26T18:54:49.587 INFO:tasks.workunit.client.0.smithi102.stdout:7/943: chown d7/d10/d67/fb2 98336 1 2024-03-26T18:54:49.589 INFO:tasks.workunit.client.0.smithi102.stdout:7/944: getdents d7/d10/d17 0 2024-03-26T18:54:49.589 INFO:tasks.workunit.client.0.smithi102.stdout:0/994: dread d6/de/d13/d1f/d71/fe1 [0,4194304] 0 2024-03-26T18:54:49.591 INFO:tasks.workunit.client.0.smithi102.stdout:7/945: symlink d7/d31/l12a 0 2024-03-26T18:54:49.592 INFO:tasks.workunit.client.0.smithi102.stdout:4/940: dread d1/d27/ff0 [0,4194304] 0 2024-03-26T18:54:49.592 INFO:tasks.workunit.client.0.smithi102.stdout:4/941: chown d1/d17/d1f/d5d/d8b/d3e/d74/c12e 398407 1 2024-03-26T18:54:49.593 INFO:tasks.workunit.client.0.smithi102.stdout:7/946: symlink d7/d12/d51/dd0/d11f/l12b 0 2024-03-26T18:54:49.595 INFO:tasks.workunit.client.0.smithi102.stdout:4/942: link d1/d17/d1f/d5d/d8b/c52 d1/d75/d85/d133/c13a 0 2024-03-26T18:54:49.599 INFO:tasks.workunit.client.0.smithi102.stdout:4/943: mknod d1/d17/d1f/d5d/d8b/de5/c13b 0 2024-03-26T18:54:49.599 INFO:tasks.workunit.client.0.smithi102.stdout:4/944: chown d1/d17/d1f/d5d/d8b/d3e/d74/ld1 1110940055 1 2024-03-26T18:54:49.599 INFO:tasks.workunit.client.0.smithi102.stdout:4/945: creat d1/d27/d7a/f13c x:0 0 0 2024-03-26T18:54:49.600 INFO:tasks.workunit.client.0.smithi102.stdout:7/947: getdents d7/d44 0 2024-03-26T18:54:49.603 INFO:tasks.workunit.client.0.smithi102.stdout:4/946: mknod d1/d17/d1f/d5d/d6f/c13d 0 2024-03-26T18:54:49.616 INFO:tasks.workunit.client.0.smithi102.stdout:4/947: dwrite d1/fdd [0,4194304] 0 2024-03-26T18:54:49.618 INFO:tasks.workunit.client.0.smithi102.stdout:4/948: rename d1/d17/d1f/fd9 to d1/de1/df5/f13e 0 2024-03-26T18:54:49.619 INFO:tasks.workunit.client.0.smithi102.stdout:4/949: write d1/d75/d85/f10f [155309,96711] 0 2024-03-26T18:54:49.621 INFO:tasks.workunit.client.0.smithi102.stdout:4/950: mkdir d1/d17/dbb/d126/d13f 0 2024-03-26T18:54:49.626 INFO:tasks.workunit.client.0.smithi102.stdout:4/951: rmdir d1/d17/dbb 39 2024-03-26T18:54:49.626 INFO:tasks.workunit.client.0.smithi102.stdout:4/952: chown d1/d17/d1a/d21/f64 31072 1 2024-03-26T18:54:49.627 INFO:tasks.workunit.client.0.smithi102.stdout:4/953: getdents d1/d17/d1f/d5d/d8b/de5 0 2024-03-26T18:54:49.635 INFO:tasks.workunit.client.0.smithi102.stdout:4/954: dread d1/d75/f117 [0,4194304] 0 2024-03-26T18:54:49.669 INFO:tasks.workunit.client.0.smithi102.stdout:9/939: sync 2024-03-26T18:54:49.670 INFO:tasks.workunit.client.0.smithi102.stdout:9/940: fdatasync df/d1a/d65/dbe/ffc 0 2024-03-26T18:54:49.670 INFO:tasks.workunit.client.0.smithi102.stdout:9/941: chown df/d1a/d1d/d72/d7d/db7/ldd 0 1 2024-03-26T18:54:49.671 INFO:tasks.workunit.client.0.smithi102.stdout:9/942: rename df/d1a/d81/fa2 to df/d1a/d23/d3e/d75/d117/f158 0 2024-03-26T18:54:49.672 INFO:tasks.workunit.client.0.smithi102.stdout:9/943: truncate df/d1a/d23/d3e/d75/fbd 149540 0 2024-03-26T18:54:49.672 INFO:tasks.workunit.client.0.smithi102.stdout:9/944: chown df/d46/da8/f141 10219 1 2024-03-26T18:54:49.673 INFO:tasks.workunit.client.0.smithi102.stdout:9/945: symlink df/d1a/d1d/d72/d7d/db7/d13c/l159 0 2024-03-26T18:54:49.673 INFO:tasks.workunit.client.0.smithi102.stdout:9/946: creat df/d1a/d23/d3e/d75/d110/f15a x:0 0 0 2024-03-26T18:54:49.673 INFO:tasks.workunit.client.0.smithi102.stdout:9/947: creat df/d1a/f15b x:0 0 0 2024-03-26T18:54:49.677 INFO:tasks.workunit.client.0.smithi102.stdout:9/948: rename df/d1a/d23/d3e/d75/fbd to df/d46/da8/f15c 0 2024-03-26T18:54:49.678 INFO:tasks.workunit.client.0.smithi102.stdout:9/949: symlink df/d1a/d23/d3e/d75/d137/l15d 0 2024-03-26T18:54:49.678 INFO:tasks.workunit.client.0.smithi102.stdout:9/950: creat df/d1a/d23/d4e/db0/f15e x:0 0 0 2024-03-26T18:54:49.680 INFO:tasks.workunit.client.0.smithi102.stdout:9/951: symlink df/d1a/d1d/d72/d7d/l15f 0 2024-03-26T18:54:49.681 INFO:tasks.workunit.client.0.smithi102.stdout:9/952: symlink df/l160 0 2024-03-26T18:54:49.681 INFO:tasks.workunit.client.0.smithi102.stdout:9/953: fdatasync df/d1a/d7f/f156 0 2024-03-26T18:54:49.697 INFO:tasks.workunit.client.0.smithi102.stdout:9/954: dwrite df/d1a/d1d/f138 [0,4194304] 0 2024-03-26T18:54:49.850 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:49 smithi102 ceph-mon[29002]: Deploying cephadm binary to smithi102 2024-03-26T18:54:49.850 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:49 smithi102 ceph-mon[29002]: Deploying cephadm binary to smithi190 2024-03-26T18:54:49.850 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:49 smithi102 ceph-mon[29002]: [26/Mar/2024:18:54:48] ENGINE Bus STARTING 2024-03-26T18:54:49.850 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:49 smithi102 ceph-mon[29002]: [26/Mar/2024:18:54:48] ENGINE Serving on http://172.21.15.190:8765 2024-03-26T18:54:49.850 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:49 smithi102 ceph-mon[29002]: [26/Mar/2024:18:54:48] ENGINE Serving on https://172.21.15.190:7150 2024-03-26T18:54:49.850 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:49 smithi102 ceph-mon[29002]: [26/Mar/2024:18:54:48] ENGINE Bus STARTED 2024-03-26T18:54:49.850 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:49 smithi102 ceph-mon[29002]: mgrmap e22: smithi190.zrixos(active, since 1.01979s) 2024-03-26T18:54:49.850 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:49 smithi102 ceph-mon[29002]: pgmap v3: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail 2024-03-26T18:54:50.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:49 smithi190 ceph-mon[35043]: Deploying cephadm binary to smithi102 2024-03-26T18:54:50.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:49 smithi190 ceph-mon[35043]: Deploying cephadm binary to smithi190 2024-03-26T18:54:50.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:49 smithi190 ceph-mon[35043]: [26/Mar/2024:18:54:48] ENGINE Bus STARTING 2024-03-26T18:54:50.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:49 smithi190 ceph-mon[35043]: [26/Mar/2024:18:54:48] ENGINE Serving on http://172.21.15.190:8765 2024-03-26T18:54:50.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:49 smithi190 ceph-mon[35043]: [26/Mar/2024:18:54:48] ENGINE Serving on https://172.21.15.190:7150 2024-03-26T18:54:50.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:49 smithi190 ceph-mon[35043]: [26/Mar/2024:18:54:48] ENGINE Bus STARTED 2024-03-26T18:54:50.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:49 smithi190 ceph-mon[35043]: mgrmap e22: smithi190.zrixos(active, since 1.01979s) 2024-03-26T18:54:50.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:49 smithi190 ceph-mon[35043]: pgmap v3: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail 2024-03-26T18:54:50.329 INFO:tasks.workunit.client.0.smithi102.stdout:7/948: sync 2024-03-26T18:54:50.329 INFO:tasks.workunit.client.0.smithi102.stdout:4/955: sync 2024-03-26T18:54:50.329 INFO:tasks.workunit.client.0.smithi102.stdout:9/955: sync 2024-03-26T18:54:50.329 INFO:tasks.workunit.client.0.smithi102.stdout:0/995: sync 2024-03-26T18:54:50.330 INFO:tasks.workunit.client.0.smithi102.stdout:7/949: fdatasync d7/d44/f53 0 2024-03-26T18:54:50.330 INFO:tasks.workunit.client.0.smithi102.stdout:0/996: stat d6/de/d13/d25/d31/d45/d82/ff3 0 2024-03-26T18:54:50.330 INFO:tasks.workunit.client.0.smithi102.stdout:0/997: fsync d6/de/d13/d25/d4f/f73 0 2024-03-26T18:54:50.330 INFO:tasks.workunit.client.0.smithi102.stdout:4/956: truncate d1/d17/d1f/d5d/d6f/dc0/fc7 544617 0 2024-03-26T18:54:50.330 INFO:tasks.workunit.client.0.smithi102.stdout:0/998: truncate d6/de/d13/d25/d31/d45/d82/ff3 133787 0 2024-03-26T18:54:50.330 INFO:tasks.workunit.client.0.smithi102.stdout:4/957: readlink d1/d17/d1f/d5d/d8b/l5a 0 2024-03-26T18:54:50.348 INFO:tasks.workunit.client.0.smithi102.stdout:4/958: dwrite d1/d25/d90/fab [0,4194304] 0 2024-03-26T18:54:50.442 INFO:tasks.workunit.client.0.smithi102.stdout:0/999: mkdir d6/de/d13/d25/d31/d45/d82/d84/d88/d15b 0 2024-03-26T18:54:50.443 INFO:tasks.workunit.client.0.smithi102.stdout:7/950: unlink d7/d2a/d9d/cc8 0 2024-03-26T18:54:50.444 INFO:tasks.workunit.client.0.smithi102.stdout:4/959: getdents d1/d27/d98/db7 0 2024-03-26T18:54:50.453 INFO:tasks.workunit.client.0.smithi102.stdout:7/951: dread d7/d12/d51/d6a/fb6 [0,4194304] 0 2024-03-26T18:54:50.457 INFO:tasks.workunit.client.0.smithi102.stdout:7/952: mknod d7/d10/d67/de1/c12c 0 2024-03-26T18:54:50.458 INFO:tasks.workunit.client.0.smithi102.stdout:9/956: rename df/d1a/d23/d3e to df/d1a/d14c/d161 0 2024-03-26T18:54:50.460 INFO:tasks.workunit.client.0.smithi102.stdout:7/953: mknod d7/d10/d67/d69/d78/dfa/c12d 0 2024-03-26T18:54:50.460 INFO:tasks.workunit.client.0.smithi102.stdout:4/960: dwrite d1/d27/d98/f6a [4194304,4194304] 0 2024-03-26T18:54:50.461 INFO:tasks.workunit.client.0.smithi102.stdout:9/957: link df/d1a/d23/d29/d9c/c122 df/d10f/c162 0 2024-03-26T18:54:50.462 INFO:tasks.workunit.client.0.smithi102.stdout:7/954: truncate d7/d12/f3c 2446016 0 2024-03-26T18:54:50.476 INFO:tasks.workunit.client.0.smithi102.stdout:9/958: read df/d1a/d7f/fba [228442,25157] 0 2024-03-26T18:54:50.476 INFO:tasks.workunit.client.0.smithi102.stdout:4/961: write d1/d17/fae [225424,94692] 0 2024-03-26T18:54:50.508 INFO:tasks.workunit.client.0.smithi102.stdout:7/955: dwrite d7/d2a/f60 [0,4194304] 0 2024-03-26T18:54:50.509 INFO:tasks.workunit.client.0.smithi102.stdout:9/959: dwrite df/d1a/d23/d60/f74 [0,4194304] 0 2024-03-26T18:54:50.509 INFO:tasks.workunit.client.0.smithi102.stdout:4/962: dwrite d1/d17/d1a/f4b [0,4194304] 0 2024-03-26T18:54:50.509 INFO:tasks.workunit.client.0.smithi102.stdout:4/963: dread - d1/d17/dbb/fd4 zero size 2024-03-26T18:54:50.513 INFO:tasks.workunit.client.0.smithi102.stdout:9/960: rename df/d1a/d23/d29/f31 to df/d46/da8/f163 0 2024-03-26T18:54:50.516 INFO:tasks.workunit.client.0.smithi102.stdout:9/961: link df/d46/da8/f105 df/d1a/d65/dbe/f164 0 2024-03-26T18:54:50.518 INFO:tasks.workunit.client.0.smithi102.stdout:9/962: mknod df/d1a/d23/d4e/c165 0 2024-03-26T18:54:50.522 INFO:tasks.workunit.client.0.smithi102.stdout:4/964: rename d1/d113 to d1/d27/d140 0 2024-03-26T18:54:50.522 INFO:tasks.workunit.client.0.smithi102.stdout:4/965: readlink d1/d17/d1f/d5d/d6f/dc0/lf7 0 2024-03-26T18:54:50.523 INFO:tasks.workunit.client.0.smithi102.stdout:7/956: dwrite d7/d10/d67/de1/df4/ffc [0,4194304] 0 2024-03-26T18:54:50.523 INFO:tasks.workunit.client.0.smithi102.stdout:7/957: fdatasync d7/d12/f8d 0 2024-03-26T18:54:50.524 INFO:tasks.workunit.client.0.smithi102.stdout:7/958: readlink d7/d12/d51/dd0/d11f/l12b 0 2024-03-26T18:54:50.524 INFO:tasks.workunit.client.0.smithi102.stdout:7/959: chown d7/d12/c77 413063 1 2024-03-26T18:54:50.525 INFO:tasks.workunit.client.0.smithi102.stdout:4/966: symlink d1/d25/d90/l141 0 2024-03-26T18:54:50.527 INFO:tasks.workunit.client.0.smithi102.stdout:7/960: unlink d7/d10/d27/d34/l75 0 2024-03-26T18:54:50.529 INFO:tasks.workunit.client.0.smithi102.stdout:4/967: getdents d1/d17/d1f/d5d/d8b/d3e/d74 0 2024-03-26T18:54:50.531 INFO:tasks.workunit.client.0.smithi102.stdout:4/968: unlink d1/fdd 0 2024-03-26T18:54:50.531 INFO:tasks.workunit.client.0.smithi102.stdout:4/969: stat d1/d17/d1f/d5d/c7d 0 2024-03-26T18:54:50.531 INFO:tasks.workunit.client.0.smithi102.stdout:4/970: write d1/d17/d1a/d21/f63 [658631,120993] 0 2024-03-26T18:54:50.531 INFO:tasks.workunit.client.0.smithi102.stdout:4/971: getdents d1/d17/d1f/d5d/d8b/de5 0 2024-03-26T18:54:50.531 INFO:tasks.workunit.client.0.smithi102.stdout:4/972: chown d1/d27/d98/db7 0 1 2024-03-26T18:54:50.531 INFO:tasks.workunit.client.0.smithi102.stdout:4/973: readlink d1/d25/d90/lc5 0 2024-03-26T18:54:50.532 INFO:tasks.workunit.client.0.smithi102.stdout:4/974: symlink d1/d25/l142 0 2024-03-26T18:54:50.532 INFO:tasks.workunit.client.0.smithi102.stdout:4/975: chown d1/d17/d1f/d5d/d8b 5 1 2024-03-26T18:54:50.534 INFO:tasks.workunit.client.0.smithi102.stdout:4/976: rename d1/d27/d98/le0 to d1/d25/l143 0 2024-03-26T18:54:50.534 INFO:tasks.workunit.client.0.smithi102.stdout:4/977: fsync d1/fa0 0 2024-03-26T18:54:50.535 INFO:tasks.workunit.client.0.smithi102.stdout:7/961: dread d7/d10/d67/fa5 [0,4194304] 0 2024-03-26T18:54:50.535 INFO:tasks.workunit.client.0.smithi102.stdout:4/978: creat d1/d17/d1f/d5d/d6f/f144 x:0 0 0 2024-03-26T18:54:50.536 INFO:tasks.workunit.client.0.smithi102.stdout:4/979: rmdir d1/d17/dbb 39 2024-03-26T18:54:50.536 INFO:tasks.workunit.client.0.smithi102.stdout:4/980: creat d1/d17/d1f/d5d/d6f/f145 x:0 0 0 2024-03-26T18:54:50.537 INFO:tasks.workunit.client.0.smithi102.stdout:9/963: dwrite df/d1a/f52 [0,4194304] 0 2024-03-26T18:54:50.542 INFO:tasks.workunit.client.0.smithi102.stdout:9/964: dread df/d1a/d14c/d161/f99 [0,4194304] 0 2024-03-26T18:54:50.543 INFO:tasks.workunit.client.0.smithi102.stdout:9/965: truncate df/d46/da8/f11e 375209 0 2024-03-26T18:54:50.545 INFO:tasks.workunit.client.0.smithi102.stdout:4/981: rename d1/d75/d85/f10f to d1/d17/d2a/d9a/ddc/f146 0 2024-03-26T18:54:50.546 INFO:tasks.workunit.client.0.smithi102.stdout:9/966: symlink df/d1a/d1d/l166 0 2024-03-26T18:54:50.548 INFO:tasks.workunit.client.0.smithi102.stdout:9/967: creat df/d1a/d1d/d72/d7d/db7/df8/f167 x:0 0 0 2024-03-26T18:54:50.548 INFO:tasks.workunit.client.0.smithi102.stdout:9/968: write df/d1a/d81/dcc/f14b [100570,76570] 0 2024-03-26T18:54:50.549 INFO:tasks.workunit.client.0.smithi102.stdout:9/969: getdents df/d1a/d14c/d161/d75/d137 0 2024-03-26T18:54:50.549 INFO:tasks.workunit.client.0.smithi102.stdout:9/970: write df/d84/f9e [4823181,2287] 0 2024-03-26T18:54:50.551 INFO:tasks.workunit.client.0.smithi102.stdout:9/971: symlink df/d1a/d65/dbe/ddf/l168 0 2024-03-26T18:54:50.552 INFO:tasks.workunit.client.0.smithi102.stdout:7/962: dwrite d7/d12/d2b/da8/dca/ff8 [0,4194304] 0 2024-03-26T18:54:50.552 INFO:tasks.workunit.client.0.smithi102.stdout:7/963: dread - d7/d10/d27/da3/dc6/f105 zero size 2024-03-26T18:54:50.552 INFO:tasks.workunit.client.0.smithi102.stdout:7/964: chown d7/d12/d4d/l6f 116 1 2024-03-26T18:54:50.554 INFO:tasks.workunit.client.0.smithi102.stdout:4/982: dread d1/d25/d90/fab [0,4194304] 0 2024-03-26T18:54:50.554 INFO:tasks.workunit.client.0.smithi102.stdout:4/983: chown d1/d75/d85/fbe 112712848 1 2024-03-26T18:54:50.556 INFO:tasks.workunit.client.0.smithi102.stdout:7/965: truncate d7/d10/d67/d69/d9b/dae/fe6 379675 0 2024-03-26T18:54:50.556 INFO:tasks.workunit.client.0.smithi102.stdout:7/966: truncate d7/d4e/f11e 314876 0 2024-03-26T18:54:50.560 INFO:tasks.workunit.client.0.smithi102.stdout:9/972: rename df/d1a/d23/d29 to df/d1a/d1d/d72/d7d/db7/df8/d169 0 2024-03-26T18:54:50.562 INFO:tasks.workunit.client.0.smithi102.stdout:4/984: symlink d1/d17/d1f/d5d/d8b/d3e/d74/d135/l147 0 2024-03-26T18:54:50.562 INFO:tasks.workunit.client.0.smithi102.stdout:4/985: creat d1/d6b/de7/d12d/f148 x:0 0 0 2024-03-26T18:54:50.562 INFO:tasks.workunit.client.0.smithi102.stdout:4/986: creat d1/d17/dbc/f149 x:0 0 0 2024-03-26T18:54:50.562 INFO:tasks.workunit.client.0.smithi102.stdout:4/987: stat d1/d75/l109 0 2024-03-26T18:54:50.562 INFO:tasks.workunit.client.0.smithi102.stdout:4/988: chown d1/d17/dbc/f11f 12 1 2024-03-26T18:54:50.563 INFO:tasks.workunit.client.0.smithi102.stdout:4/989: write d1/d17/f110 [596211,17555] 0 2024-03-26T18:54:50.564 INFO:tasks.workunit.client.0.smithi102.stdout:7/967: mknod d7/d12/d51/dbb/c12e 0 2024-03-26T18:54:50.564 INFO:tasks.workunit.client.0.smithi102.stdout:7/968: truncate d7/d10/d27/d34/d82/fe4 635818 0 2024-03-26T18:54:50.564 INFO:tasks.workunit.client.0.smithi102.stdout:7/969: write d7/d10/d67/d69/d9b/fd8 [144426,67657] 0 2024-03-26T18:54:50.566 INFO:tasks.workunit.client.0.smithi102.stdout:9/973: unlink df/d1a/d1d/d72/d7d/db7/f134 0 2024-03-26T18:54:50.578 INFO:tasks.workunit.client.0.smithi102.stdout:4/990: creat d1/d17/d1a/d21/f14a x:0 0 0 2024-03-26T18:54:50.578 INFO:tasks.workunit.client.0.smithi102.stdout:4/991: stat d1/d25/cf6 0 2024-03-26T18:54:50.599 INFO:tasks.workunit.client.0.smithi102.stdout:7/970: dwrite d7/d10/d27/da3/fdb [0,4194304] 0 2024-03-26T18:54:50.599 INFO:tasks.workunit.client.0.smithi102.stdout:7/971: chown d7/d10/d17/f7d 6230 1 2024-03-26T18:54:50.599 INFO:tasks.workunit.client.0.smithi102.stdout:9/974: dwrite df/d1a/d23/fb2 [4194304,4194304] 0 2024-03-26T18:54:50.599 INFO:tasks.workunit.client.0.smithi102.stdout:9/975: write df/d1a/d14c/d161/d75/d117/f11d [478918,100953] 0 2024-03-26T18:54:50.599 INFO:tasks.workunit.client.0.smithi102.stdout:4/992: dwrite d1/d17/d1f/d5d/d6f/dc0/fce [0,4194304] 0 2024-03-26T18:54:50.599 INFO:tasks.workunit.client.0.smithi102.stdout:4/993: write d1/d17/fae [2022576,82169] 0 2024-03-26T18:54:50.602 INFO:tasks.workunit.client.0.smithi102.stdout:7/972: mknod d7/d31/d79/c12f 0 2024-03-26T18:54:50.602 INFO:tasks.workunit.client.0.smithi102.stdout:7/973: read d7/d12/d51/d6a/fb6 [1346108,111180] 0 2024-03-26T18:54:50.606 INFO:tasks.workunit.client.0.smithi102.stdout:9/976: link df/d1a/d1d/d72/d7d/cb3 df/d1a/d23/d4e/dc1/c16a 0 2024-03-26T18:54:50.606 INFO:tasks.workunit.client.0.smithi102.stdout:9/977: write df/d84/f78 [890461,47832] 0 2024-03-26T18:54:50.606 INFO:tasks.workunit.client.0.smithi102.stdout:9/978: write df/d1a/d81/dcc/f120 [4469798,57803] 0 2024-03-26T18:54:50.610 INFO:tasks.workunit.client.0.smithi102.stdout:9/979: dread df/d46/da8/f141 [0,4194304] 0 2024-03-26T18:54:50.611 INFO:tasks.workunit.client.0.smithi102.stdout:4/994: mknod d1/d17/d1a/d21/de8/c14b 0 2024-03-26T18:54:50.611 INFO:tasks.workunit.client.0.smithi102.stdout:7/974: dread d7/d12/d51/d6a/fb6 [0,4194304] 0 2024-03-26T18:54:50.611 INFO:tasks.workunit.client.0.smithi102.stdout:7/975: write d7/d12/d51/d6a/fd7 [299084,31598] 0 2024-03-26T18:54:50.612 INFO:tasks.workunit.client.0.smithi102.stdout:4/995: write d1/d17/d1f/d5d/d8b/fc9 [351812,127743] 0 2024-03-26T18:54:50.613 INFO:tasks.workunit.client.0.smithi102.stdout:4/996: dread - d1/d17/d1a/d21/f14a zero size 2024-03-26T18:54:50.615 INFO:tasks.workunit.client.0.smithi102.stdout:9/980: creat df/d1a/d23/d4e/dc1/f16b x:0 0 0 2024-03-26T18:54:50.618 INFO:tasks.workunit.client.0.smithi102.stdout:4/997: rename d1/d75/l101 to d1/d27/d98/dea/d10e/l14c 0 2024-03-26T18:54:50.621 INFO:tasks.workunit.client.0.smithi102.stdout:9/981: mknod df/d46/da8/c16c 0 2024-03-26T18:54:50.622 INFO:tasks.workunit.client.0.smithi102.stdout:9/982: dread df/d1a/d1d/d72/d7d/db7/df8/d169/d9c/faa [0,4194304] 0 2024-03-26T18:54:50.623 INFO:tasks.workunit.client.0.smithi102.stdout:4/998: rename d1/d17/d1a/d21/f64 to d1/d17/d1f/d5d/d6f/dc0/d131/f14d 0 2024-03-26T18:54:50.625 INFO:tasks.workunit.client.0.smithi102.stdout:9/983: getdents df/d1a/d1d/d72/d7d 0 2024-03-26T18:54:50.637 INFO:tasks.workunit.client.0.smithi102.stdout:7/976: dwrite d7/d12/d51/d6a/fcf [0,4194304] 0 2024-03-26T18:54:50.637 INFO:tasks.workunit.client.0.smithi102.stdout:7/977: chown d7/d12/f8d 893383 1 2024-03-26T18:54:50.637 INFO:tasks.workunit.client.0.smithi102.stdout:7/978: chown d7/d10/d17 4 1 2024-03-26T18:54:50.639 INFO:tasks.workunit.client.0.smithi102.stdout:9/984: write df/d1a/d23/d4e/d8e/f56 [2206610,55799] 0 2024-03-26T18:54:50.639 INFO:tasks.workunit.client.0.smithi102.stdout:9/985: fsync df/d1a/d1d/d72/d7d/db7/df8/d169/fc4 0 2024-03-26T18:54:50.639 INFO:tasks.workunit.client.0.smithi102.stdout:9/986: dread - df/d1a/d81/d88/f157 zero size 2024-03-26T18:54:50.639 INFO:tasks.workunit.client.0.smithi102.stdout:9/987: write df/d1a/f3f [3305230,60284] 0 2024-03-26T18:54:50.639 INFO:tasks.workunit.client.0.smithi102.stdout:9/988: chown df/d1a/d81/dcc/f14b 26152165 1 2024-03-26T18:54:50.639 INFO:tasks.workunit.client.0.smithi102.stdout:9/989: getdents df/dbc 0 2024-03-26T18:54:50.640 INFO:tasks.workunit.client.0.smithi102.stdout:4/999: dwrite d1/d6b/de7/d12d/f148 [0,4194304] 0 2024-03-26T18:54:50.660 INFO:tasks.workunit.client.0.smithi102.stdout:7/979: dwrite d7/d10/d67/d69/d9b/fe0 [0,4194304] 0 2024-03-26T18:54:50.660 INFO:tasks.workunit.client.0.smithi102.stdout:9/990: dwrite df/d1a/d1d/f154 [0,4194304] 0 2024-03-26T18:54:50.660 INFO:tasks.workunit.client.0.smithi102.stdout:9/991: chown df/d1a/d1d/lfb 1651 1 2024-03-26T18:54:50.663 INFO:tasks.workunit.client.0.smithi102.stdout:7/980: truncate d7/d10/d17/db9/fbe 1281672 0 2024-03-26T18:54:50.664 INFO:tasks.workunit.client.0.smithi102.stdout:7/981: write d7/d12/d2b/f47 [5102551,39275] 0 2024-03-26T18:54:50.668 INFO:tasks.workunit.client.0.smithi102.stdout:9/992: rename df/d1a/l44 to df/d1a/d57/l16d 0 2024-03-26T18:54:50.691 INFO:tasks.workunit.client.0.smithi102.stdout:7/982: dwrite d7/d10/d27/d34/d82/daa/f100 [0,4194304] 0 2024-03-26T18:54:50.691 INFO:tasks.workunit.client.0.smithi102.stdout:7/983: chown d7/d31/d79/c12f 498748882 1 2024-03-26T18:54:50.691 INFO:tasks.workunit.client.0.smithi102.stdout:9/993: dwrite df/d1a/d23/fb5 [0,4194304] 0 2024-03-26T18:54:50.693 INFO:tasks.workunit.client.0.smithi102.stdout:7/984: rename d7/d10/d17/f37 to d7/d12/d51/dd0/d11f/f130 0 2024-03-26T18:54:50.719 INFO:tasks.workunit.client.0.smithi102.stdout:7/985: dwrite d7/d44/fed [0,4194304] 0 2024-03-26T18:54:50.722 INFO:tasks.workunit.client.0.smithi102.stdout:7/986: unlink d7/d44/fed 0 2024-03-26T18:54:50.724 INFO:tasks.workunit.client.0.smithi102.stdout:7/987: creat d7/d10/f131 x:0 0 0 2024-03-26T18:54:50.726 INFO:tasks.workunit.client.0.smithi102.stdout:7/988: creat d7/d2a/d10c/f132 x:0 0 0 2024-03-26T18:54:50.726 INFO:tasks.workunit.client.0.smithi102.stdout:7/989: chown d7/d4e/ld2 11145 1 2024-03-26T18:54:50.744 INFO:tasks.workunit.client.0.smithi102.stdout:7/990: dwrite d7/d10/d17/db9/fbe [0,4194304] 0 2024-03-26T18:54:50.744 INFO:tasks.workunit.client.0.smithi102.stdout:7/991: readlink d7/d10/d17/l23 0 2024-03-26T18:54:50.768 INFO:tasks.workunit.client.0.smithi102.stdout:7/992: dwrite d7/d10/d67/fb2 [4194304,4194304] 0 2024-03-26T18:54:50.768 INFO:tasks.workunit.client.0.smithi102.stdout:7/993: readlink d7/d44/lc7 0 2024-03-26T18:54:50.768 INFO:tasks.workunit.client.0.smithi102.stdout:7/994: fsync d7/d12/d2b/da8/f10a 0 2024-03-26T18:54:50.770 INFO:tasks.workunit.client.0.smithi102.stdout:7/995: rename d7/d4e/dd5 to d7/d12/d4d/d62/d133 0 2024-03-26T18:54:50.770 INFO:tasks.workunit.client.0.smithi102.stdout:7/996: chown d7/d4e/ld2 12 1 2024-03-26T18:54:50.771 INFO:tasks.workunit.client.0.smithi102.stdout:7/997: symlink d7/d10/d67/de1/df4/l134 0 2024-03-26T18:54:50.773 INFO:tasks.workunit.client.0.smithi102.stdout:7/998: symlink d7/d10/l135 0 2024-03-26T18:54:50.774 INFO:tasks.workunit.client.0.smithi102.stdout:7/999: mknod d7/dab/c136 0 2024-03-26T18:54:50.780 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:50 smithi102 ceph-mon[29002]: pgmap v4: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail 2024-03-26T18:54:50.780 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:50 smithi102 ceph-mon[29002]: mgrmap e23: smithi190.zrixos(active, since 2s) 2024-03-26T18:54:50.959 INFO:tasks.workunit.client.0.smithi102.stdout:9/994: sync 2024-03-26T18:54:50.959 INFO:tasks.workunit.client.0.smithi102.stdout:9/995: fsync df/d1a/d1d/f138 0 2024-03-26T18:54:50.960 INFO:tasks.workunit.client.0.smithi102.stdout:9/996: fdatasync df/f3b 0 2024-03-26T18:54:50.985 INFO:tasks.workunit.client.0.smithi102.stdout:9/997: dwrite df/d1a/d1d/d72/f126 [0,4194304] 0 2024-03-26T18:54:50.987 INFO:tasks.workunit.client.0.smithi102.stdout:9/998: creat df/d1a/d23/f16e x:0 0 0 2024-03-26T18:54:50.987 INFO:tasks.workunit.client.0.smithi102.stdout:9/999: chown df/f30 684 1 2024-03-26T18:54:50.989 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm -rf -- ./tmp.yZZUd38di3 2024-03-26T18:54:51.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:50 smithi190 ceph-mon[35043]: pgmap v4: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail 2024-03-26T18:54:51.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:50 smithi190 ceph-mon[35043]: mgrmap e23: smithi190.zrixos(active, since 2s) 2024-03-26T18:54:53.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:52 smithi190 ceph-mon[35043]: pgmap v5: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail 2024-03-26T18:54:53.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:52 smithi190 ceph-mon[35043]: mgrmap e24: smithi190.zrixos(active, since 4s) 2024-03-26T18:54:53.056 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:52 smithi102 ceph-mon[29002]: pgmap v5: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail 2024-03-26T18:54:53.056 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:52 smithi102 ceph-mon[29002]: mgrmap e24: smithi190.zrixos(active, since 4s) 2024-03-26T18:54:54.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.853+0000 7ff58ad3c700 1 -- 172.21.15.102:0/880935285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5840ff070 msgr2=0x7ff5840ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:54.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.853+0000 7ff58ad3c700 1 --2- 172.21.15.102:0/880935285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5840ff070 0x7ff5840ff480 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff578009910 tx=0x7ff578009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:54.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.853+0000 7ff58ad3c700 1 -- 172.21.15.102:0/880935285 shutdown_connections 2024-03-26T18:54:54.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.853+0000 7ff58ad3c700 1 --2- 172.21.15.102:0/880935285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff584100270 0x7ff5841006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:54.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.853+0000 7ff58ad3c700 1 --2- 172.21.15.102:0/880935285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5840ff070 0x7ff5840ff480 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:54.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.853+0000 7ff58ad3c700 1 -- 172.21.15.102:0/880935285 >> 172.21.15.102:0/880935285 conn(0x7ff5840fa600 msgr2=0x7ff5840fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:54.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.854+0000 7ff58ad3c700 1 -- 172.21.15.102:0/880935285 shutdown_connections 2024-03-26T18:54:54.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.854+0000 7ff58ad3c700 1 -- 172.21.15.102:0/880935285 wait complete. 2024-03-26T18:54:54.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.855+0000 7ff58ad3c700 1 Processor -- start 2024-03-26T18:54:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.856+0000 7ff58ad3c700 1 -- start start 2024-03-26T18:54:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.856+0000 7ff58ad3c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5840ff070 0x7ff584193770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.856+0000 7ff58ad3c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff584100270 0x7ff584193cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.856+0000 7ff58ad3c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5841942d0 con 0x7ff584100270 2024-03-26T18:54:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.856+0000 7ff58ad3c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff584194410 con 0x7ff5840ff070 2024-03-26T18:54:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.856+0000 7ff583fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff584100270 0x7ff584193cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.856+0000 7ff583fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff584100270 0x7ff584193cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37870/0 (socket says 172.21.15.102:37870) 2024-03-26T18:54:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.856+0000 7ff583fff700 1 -- 172.21.15.102:0/2790322579 learned_addr learned my addr 172.21.15.102:0/2790322579 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.856+0000 7ff588ad8700 1 --2- 172.21.15.102:0/2790322579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5840ff070 0x7ff584193770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:54.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.857+0000 7ff583fff700 1 -- 172.21.15.102:0/2790322579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5840ff070 msgr2=0x7ff584193770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:54.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.857+0000 7ff583fff700 1 --2- 172.21.15.102:0/2790322579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5840ff070 0x7ff584193770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:54.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.857+0000 7ff583fff700 1 -- 172.21.15.102:0/2790322579 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57000d590 con 0x7ff584100270 2024-03-26T18:54:54.850 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:54 smithi102 ceph-mon[29002]: pgmap v6: 65 pgs: 65 active+clean; 3.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 12 MiB/s rd, 22 MiB/s wr, 72 op/s 2024-03-26T18:54:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.857+0000 7ff583fff700 1 --2- 172.21.15.102:0/2790322579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff584100270 0x7ff584193cb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff570009af0 tx=0x7ff570009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.857+0000 7ff581ffb700 1 -- 172.21.15.102:0/2790322579 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff570007660 con 0x7ff584100270 2024-03-26T18:54:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.857+0000 7ff581ffb700 1 -- 172.21.15.102:0/2790322579 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff570007c70 con 0x7ff584100270 2024-03-26T18:54:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.858+0000 7ff581ffb700 1 -- 172.21.15.102:0/2790322579 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff57000c420 con 0x7ff584100270 2024-03-26T18:54:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.859+0000 7ff58ad3c700 1 -- 172.21.15.102:0/2790322579 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5780095f0 con 0x7ff584100270 2024-03-26T18:54:54.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.859+0000 7ff58ad3c700 1 -- 172.21.15.102:0/2790322579 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff584199250 con 0x7ff584100270 2024-03-26T18:54:54.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.862+0000 7ff56f7fe700 1 -- 172.21.15.102:0/2790322579 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff584106ab0 con 0x7ff584100270 2024-03-26T18:54:54.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.862+0000 7ff581ffb700 1 -- 172.21.15.102:0/2790322579 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 24) v1 ==== 46435+0+0 (secure 0 0 0) 0x7ff5700077c0 con 0x7ff584100270 2024-03-26T18:54:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.863+0000 7ff581ffb700 1 --2- 172.21.15.102:0/2790322579 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7ff5740394e0 0x7ff57403b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.863+0000 7ff581ffb700 1 -- 172.21.15.102:0/2790322579 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7ff570051140 con 0x7ff584100270 2024-03-26T18:54:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.863+0000 7ff588ad8700 1 --2- 172.21.15.102:0/2790322579 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7ff5740394e0 0x7ff57403b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:54.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.870+0000 7ff588ad8700 1 --2- 172.21.15.102:0/2790322579 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7ff5740394e0 0x7ff57403b990 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff578009fd0 tx=0x7ff578005fd0 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:54.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:54.872+0000 7ff581ffb700 1 -- 172.21.15.102:0/2790322579 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7ff570007de0 con 0x7ff584100270 2024-03-26T18:54:55.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:54 smithi190 ceph-mon[35043]: pgmap v6: 65 pgs: 65 active+clean; 3.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 12 MiB/s rd, 22 MiB/s wr, 72 op/s 2024-03-26T18:54:55.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.221+0000 7ff56f7fe700 1 -- 172.21.15.102:0/2790322579 --> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff584106c40 con 0x7ff5740394e0 2024-03-26T18:54:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.222+0000 7ff581ffb700 1 -- 172.21.15.102:0/2790322579 <== mgr.24423 v2:172.21.15.190:6828/2858265260 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7ff584106c40 con 0x7ff5740394e0 2024-03-26T18:54:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.225+0000 7ff56f7fe700 1 -- 172.21.15.102:0/2790322579 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7ff5740394e0 msgr2=0x7ff57403b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.225+0000 7ff56f7fe700 1 --2- 172.21.15.102:0/2790322579 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7ff5740394e0 0x7ff57403b990 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff578009fd0 tx=0x7ff578005fd0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.225+0000 7ff56f7fe700 1 -- 172.21.15.102:0/2790322579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff584100270 msgr2=0x7ff584193cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.225+0000 7ff56f7fe700 1 --2- 172.21.15.102:0/2790322579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff584100270 0x7ff584193cb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff570009af0 tx=0x7ff570009eb0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.225+0000 7ff56f7fe700 1 -- 172.21.15.102:0/2790322579 shutdown_connections 2024-03-26T18:54:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.226+0000 7ff56f7fe700 1 --2- 172.21.15.102:0/2790322579 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7ff5740394e0 0x7ff57403b990 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.226+0000 7ff56f7fe700 1 --2- 172.21.15.102:0/2790322579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5840ff070 0x7ff584193770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.226+0000 7ff56f7fe700 1 --2- 172.21.15.102:0/2790322579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff584100270 0x7ff584193cb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.226+0000 7ff56f7fe700 1 -- 172.21.15.102:0/2790322579 >> 172.21.15.102:0/2790322579 conn(0x7ff5840fa600 msgr2=0x7ff5841034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.226+0000 7ff56f7fe700 1 -- 172.21.15.102:0/2790322579 shutdown_connections 2024-03-26T18:54:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.226+0000 7ff56f7fe700 1 -- 172.21.15.102:0/2790322579 wait complete. 2024-03-26T18:54:55.231 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:54:55.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.403+0000 7f67b1af5700 1 -- 172.21.15.102:0/481944717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67ac108990 msgr2=0x7f67ac10ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:55.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.403+0000 7f67b1af5700 1 --2- 172.21.15.102:0/481944717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67ac108990 0x7f67ac10ce50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6794009910 tx=0x7f6794009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:55.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.403+0000 7f67b1af5700 1 -- 172.21.15.102:0/481944717 shutdown_connections 2024-03-26T18:54:55.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.403+0000 7f67b1af5700 1 --2- 172.21.15.102:0/481944717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67ac108990 0x7f67ac10ce50 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6794009910 tx=0x7f6794009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:55.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.403+0000 7f67b1af5700 1 --2- 172.21.15.102:0/481944717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ac107fb0 0x7f67ac1083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.403+0000 7f67b1af5700 1 -- 172.21.15.102:0/481944717 >> 172.21.15.102:0/481944717 conn(0x7f67ac06e020 msgr2=0x7f67ac070470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:55.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.410+0000 7f67b1af5700 1 -- 172.21.15.102:0/481944717 shutdown_connections 2024-03-26T18:54:55.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.410+0000 7f67b1af5700 1 -- 172.21.15.102:0/481944717 wait complete. 2024-03-26T18:54:55.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.411+0000 7f67b1af5700 1 Processor -- start 2024-03-26T18:54:55.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.412+0000 7f67b1af5700 1 -- start start 2024-03-26T18:54:55.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.412+0000 7f67b1af5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ac107fb0 0x7f67ac19d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:55.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.412+0000 7f67b1af5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67ac19d840 0x7f67ac1a28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:55.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.413+0000 7f67b1af5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac19dd40 con 0x7f67ac107fb0 2024-03-26T18:54:55.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.413+0000 7f67b1af5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac19deb0 con 0x7f67ac19d840 2024-03-26T18:54:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.413+0000 7f67a2ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67ac19d840 0x7f67ac1a28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:55.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.413+0000 7f67a2ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67ac19d840 0x7f67ac1a28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:39182/0 (socket says 172.21.15.102:39182) 2024-03-26T18:54:55.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.413+0000 7f67a2ffd700 1 -- 172.21.15.102:0/1937380598 learned_addr learned my addr 172.21.15.102:0/1937380598 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:55.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.413+0000 7f67a2ffd700 1 -- 172.21.15.102:0/1937380598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ac107fb0 msgr2=0x7f67ac19d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:55.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.413+0000 7f67ab7fe700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ac107fb0 0x7f67ac19d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:55.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.413+0000 7f67a2ffd700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ac107fb0 0x7f67ac19d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.413+0000 7f67a2ffd700 1 -- 172.21.15.102:0/1937380598 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c009c40 con 0x7f67ac19d840 2024-03-26T18:54:55.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.414+0000 7f67ab7fe700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ac107fb0 0x7f67ac19d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T18:54:55.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.414+0000 7f67a2ffd700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67ac19d840 0x7f67ac1a28b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6794000c00 tx=0x7f679400fab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:55.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.419+0000 7f67a97fa700 1 -- 172.21.15.102:0/1937380598 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f679401d070 con 0x7f67ac19d840 2024-03-26T18:54:55.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.419+0000 7f67a97fa700 1 -- 172.21.15.102:0/1937380598 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f679400fe60 con 0x7f67ac19d840 2024-03-26T18:54:55.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.419+0000 7f67a97fa700 1 -- 172.21.15.102:0/1937380598 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6794017400 con 0x7f67ac19d840 2024-03-26T18:54:55.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.420+0000 7f67b1af5700 1 -- 172.21.15.102:0/1937380598 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67940095f0 con 0x7f67ac19d840 2024-03-26T18:54:55.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.420+0000 7f67b1af5700 1 -- 172.21.15.102:0/1937380598 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ac1a3120 con 0x7f67ac19d840 2024-03-26T18:54:55.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.422+0000 7f67a97fa700 1 -- 172.21.15.102:0/1937380598 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 24) v1 ==== 46435+0+0 (secure 0 0 0) 0x7f6794017580 con 0x7f67ac19d840 2024-03-26T18:54:55.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.422+0000 7f67a97fa700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6798039570 0x7f679803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:55.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.422+0000 7f67a97fa700 1 -- 172.21.15.102:0/1937380598 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f679404fc00 con 0x7f67ac19d840 2024-03-26T18:54:55.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.422+0000 7f67b1af5700 1 -- 172.21.15.102:0/1937380598 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ac197290 con 0x7f67ac19d840 2024-03-26T18:54:55.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.423+0000 7f67ab7fe700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6798039570 0x7f679803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:55.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.424+0000 7f67ab7fe700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6798039570 0x7f679803ba20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f679c009fd0 tx=0x7f679c00b040 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:55.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.433+0000 7f67a97fa700 1 -- 172.21.15.102:0/1937380598 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f6794022420 con 0x7f67ac19d840 2024-03-26T18:54:55.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.807+0000 7f67b1af5700 1 -- 172.21.15.102:0/1937380598 --> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f67ac061190 con 0x7f6798039570 2024-03-26T18:54:55.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.808+0000 7f67a97fa700 1 -- 172.21.15.102:0/1937380598 <== mgr.24423 v2:172.21.15.190:6828/2858265260 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f67ac061190 con 0x7f6798039570 2024-03-26T18:54:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.811+0000 7f67b1af5700 1 -- 172.21.15.102:0/1937380598 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6798039570 msgr2=0x7f679803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.811+0000 7f67b1af5700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6798039570 0x7f679803ba20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f679c009fd0 tx=0x7f679c00b040 comp rx=0 tx=0).stop 2024-03-26T18:54:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.811+0000 7f67b1af5700 1 -- 172.21.15.102:0/1937380598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67ac19d840 msgr2=0x7f67ac1a28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.811+0000 7f67b1af5700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67ac19d840 0x7f67ac1a28b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6794000c00 tx=0x7f679400fab0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.811+0000 7f67b1af5700 1 -- 172.21.15.102:0/1937380598 shutdown_connections 2024-03-26T18:54:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.811+0000 7f67b1af5700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6798039570 0x7f679803ba20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.811+0000 7f67b1af5700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67ac107fb0 0x7f67ac19d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.811+0000 7f67b1af5700 1 --2- 172.21.15.102:0/1937380598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67ac19d840 0x7f67ac1a28b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.811+0000 7f67b1af5700 1 -- 172.21.15.102:0/1937380598 >> 172.21.15.102:0/1937380598 conn(0x7f67ac06e020 msgr2=0x7f67ac10b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.811+0000 7f67b1af5700 1 -- 172.21.15.102:0/1937380598 shutdown_connections 2024-03-26T18:54:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.811+0000 7f67b1af5700 1 -- 172.21.15.102:0/1937380598 wait complete. 2024-03-26T18:54:55.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.960+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/125949520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9f80737b0 msgr2=0x7fe9f8073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:55.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.960+0000 7fe9fcc3b700 1 --2- 172.21.15.102:0/125949520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9f80737b0 0x7fe9f8073c20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009940 tx=0x7fe9ec009c50 comp rx=0 tx=0).stop 2024-03-26T18:54:55.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.960+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/125949520 shutdown_connections 2024-03-26T18:54:55.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.960+0000 7fe9fcc3b700 1 --2- 172.21.15.102:0/125949520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9f80737b0 0x7fe9f8073c20 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.960+0000 7fe9fcc3b700 1 --2- 172.21.15.102:0/125949520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9f8074d80 0x7fe9f80731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.960+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/125949520 >> 172.21.15.102:0/125949520 conn(0x7fe9f80fb720 msgr2=0x7fe9f80fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:55.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.960+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/125949520 shutdown_connections 2024-03-26T18:54:55.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.960+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/125949520 wait complete. 2024-03-26T18:54:55.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.961+0000 7fe9fcc3b700 1 Processor -- start 2024-03-26T18:54:55.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.961+0000 7fe9fcc3b700 1 -- start start 2024-03-26T18:54:55.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.961+0000 7fe9fcc3b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9f80737b0 0x7fe9f8197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:55.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.961+0000 7fe9fcc3b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9f8074d80 0x7fe9f8198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:55.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.961+0000 7fe9fcc3b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f81986b0 con 0x7fe9f8074d80 2024-03-26T18:54:55.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.961+0000 7fe9fcc3b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f81987f0 con 0x7fe9f80737b0 2024-03-26T18:54:55.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.961+0000 7fe9f5d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9f8074d80 0x7fe9f8198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.961+0000 7fe9f5d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9f8074d80 0x7fe9f8198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37882/0 (socket says 172.21.15.102:37882) 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.961+0000 7fe9f5d9b700 1 -- 172.21.15.102:0/2804248475 learned_addr learned my addr 172.21.15.102:0/2804248475 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.961+0000 7fe9f659c700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9f80737b0 0x7fe9f8197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.962+0000 7fe9f659c700 1 -- 172.21.15.102:0/2804248475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9f8074d80 msgr2=0x7fe9f8198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.962+0000 7fe9f659c700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9f8074d80 0x7fe9f8198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.962+0000 7fe9f659c700 1 -- 172.21.15.102:0/2804248475 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec0095f0 con 0x7fe9f80737b0 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.962+0000 7fe9f5d9b700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9f8074d80 0x7fe9f8198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.962+0000 7fe9f659c700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9f80737b0 0x7fe9f8197b50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe9e800e970 tx=0x7fe9e800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.962+0000 7fe9e77fe700 1 -- 172.21.15.102:0/2804248475 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e800c9d0 con 0x7fe9f80737b0 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.962+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/2804248475 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f819d2a0 con 0x7fe9f80737b0 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.962+0000 7fe9e77fe700 1 -- 172.21.15.102:0/2804248475 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe9e8010430 con 0x7fe9f80737b0 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.963+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/2804248475 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f819d7f0 con 0x7fe9f80737b0 2024-03-26T18:54:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.963+0000 7fe9e77fe700 1 -- 172.21.15.102:0/2804248475 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e8018650 con 0x7fe9f80737b0 2024-03-26T18:54:55.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.963+0000 7fe9e77fe700 1 -- 172.21.15.102:0/2804248475 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 24) v1 ==== 46435+0+0 (secure 0 0 0) 0x7fe9e80188c0 con 0x7fe9f80737b0 2024-03-26T18:54:55.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.963+0000 7fe9e77fe700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fe9e0039490 0x7fe9e003b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:55.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.964+0000 7fe9e77fe700 1 -- 172.21.15.102:0/2804248475 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fe9e8051070 con 0x7fe9f80737b0 2024-03-26T18:54:55.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.964+0000 7fe9f5d9b700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fe9e0039490 0x7fe9e003b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:55.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.964+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/2804248475 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f810aed0 con 0x7fe9f80737b0 2024-03-26T18:54:55.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.964+0000 7fe9f5d9b700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fe9e0039490 0x7fe9e003b940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0054a0 tx=0x7fe9ec00b5a0 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:55.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:55.969+0000 7fe9e77fe700 1 -- 172.21.15.102:0/2804248475 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7fe9e80156e0 con 0x7fe9f80737b0 2024-03-26T18:54:56.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.309+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/2804248475 --> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9f8107690 con 0x7fe9e0039490 2024-03-26T18:54:56.322 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:54:56.322 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (6m) 2m ago 7m 25.5M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:54:56.322 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (8m) 2m ago 8m 7457k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:54:56.322 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (7m) 14s ago 7m 7512k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:54:56.322 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (8m) 2m ago 8m 6665k - 18.2.0 dc2bc1663786 bfac00217c68 2024-03-26T18:54:56.322 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (7m) 14s ago 7m 6656k - 18.2.0 dc2bc1663786 7379f1f6b746 2024-03-26T18:54:56.322 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (6m) 2m ago 7m 85.5M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:54:56.322 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (2m) 2m ago 2m 15.9M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:54:56.322 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (2m) 2m ago 2m 11.5M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:54:56.322 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (2m) 14s ago 2m 13.1M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:54:56.322 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (2m) 14s ago 2m 113M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:9283,8765,8443 running (9m) 2m ago 9m 487M - 18.2.0 dc2bc1663786 61d2bd7101b6 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (23s) 14s ago 6m 66.5M - 19.0.0-2311-g36051606 497bb87bc5da 888321bf02fa 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (9m) 2m ago 9m 45.1M 2048M 18.2.0 dc2bc1663786 b5abd46c3973 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (6m) 14s ago 6m 35.7M 2048M 18.2.0 dc2bc1663786 6bd913d483f0 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (8m) 2m ago 8m 15.9M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (7m) 14s ago 7m 15.6M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (6m) 2m ago 6m 47.9M 3946M 18.2.0 dc2bc1663786 3e2065fb5fbc 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (5m) 2m ago 5m 48.5M 3946M 18.2.0 dc2bc1663786 0bf773515cc1 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (5m) 2m ago 5m 46.3M 3946M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (4m) 14s ago 4m 272M 4970M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (4m) 14s ago 4m 236M 4970M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (3m) 14s ago 3m 264M 4970M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:54:56.323 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (6m) 2m ago 7m 61.7M - 2.43.0 a07b618ecd1d d0dc2ca7f41f 2024-03-26T18:54:56.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.328+0000 7fe9e77fe700 1 -- 172.21.15.102:0/2804248475 <== mgr.24423 v2:172.21.15.190:6828/2858265260 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe9f8107690 con 0x7fe9e0039490 2024-03-26T18:54:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.332+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/2804248475 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fe9e0039490 msgr2=0x7fe9e003b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.332+0000 7fe9fcc3b700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fe9e0039490 0x7fe9e003b940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0054a0 tx=0x7fe9ec00b5a0 comp rx=0 tx=0).stop 2024-03-26T18:54:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.332+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/2804248475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9f80737b0 msgr2=0x7fe9f8197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.332+0000 7fe9fcc3b700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9f80737b0 0x7fe9f8197b50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe9e800e970 tx=0x7fe9e800ec80 comp rx=0 tx=0).stop 2024-03-26T18:54:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.332+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/2804248475 shutdown_connections 2024-03-26T18:54:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.332+0000 7fe9fcc3b700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fe9e0039490 0x7fe9e003b940 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.332+0000 7fe9fcc3b700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9f80737b0 0x7fe9f8197b50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:56.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.332+0000 7fe9fcc3b700 1 --2- 172.21.15.102:0/2804248475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9f8074d80 0x7fe9f8198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:56.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.332+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/2804248475 >> 172.21.15.102:0/2804248475 conn(0x7fe9f80fb720 msgr2=0x7fe9f8105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:56.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.333+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/2804248475 shutdown_connections 2024-03-26T18:54:56.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.333+0000 7fe9fcc3b700 1 -- 172.21.15.102:0/2804248475 wait complete. 2024-03-26T18:54:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.474+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/326406555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce0100250 msgr2=0x7f6ce01006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.474+0000 7f6ce4eb5700 1 --2- 172.21.15.102:0/326406555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce0100250 0x7f6ce01006a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009910 tx=0x7f6cd4009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.474+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/326406555 shutdown_connections 2024-03-26T18:54:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.474+0000 7f6ce4eb5700 1 --2- 172.21.15.102:0/326406555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce0100250 0x7f6ce01006a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.474+0000 7f6ce4eb5700 1 --2- 172.21.15.102:0/326406555 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ce00ff050 0x7f6ce00ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.474+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/326406555 >> 172.21.15.102:0/326406555 conn(0x7f6ce00fa5e0 msgr2=0x7f6ce00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:56.467 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:56 smithi102 ceph-mon[29002]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:56.468 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:56 smithi102 ceph-mon[29002]: pgmap v7: 65 pgs: 65 active+clean; 3.0 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 27 MiB/s rd, 46 MiB/s wr, 230 op/s 2024-03-26T18:54:56.468 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:56 smithi102 ceph-mon[29002]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.474+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/326406555 shutdown_connections 2024-03-26T18:54:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.474+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/326406555 wait complete. 2024-03-26T18:54:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.475+0000 7f6ce4eb5700 1 Processor -- start 2024-03-26T18:54:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.475+0000 7f6ce4eb5700 1 -- start start 2024-03-26T18:54:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.475+0000 7f6ce4eb5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ce00ff050 0x7f6ce0195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.475+0000 7f6ce4eb5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce0100250 0x7f6ce0195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.475+0000 7f6ce4eb5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01964e0 con 0x7f6ce0100250 2024-03-26T18:54:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.475+0000 7f6ce4eb5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce0196620 con 0x7f6ce00ff050 2024-03-26T18:54:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.476+0000 7f6cde59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ce00ff050 0x7f6ce0195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.476+0000 7f6cddd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce0100250 0x7f6ce0195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.476+0000 7f6cde59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ce00ff050 0x7f6ce0195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:39222/0 (socket says 172.21.15.102:39222) 2024-03-26T18:54:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.476+0000 7f6cddd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce0100250 0x7f6ce0195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37902/0 (socket says 172.21.15.102:37902) 2024-03-26T18:54:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.476+0000 7f6cde59c700 1 -- 172.21.15.102:0/718283128 learned_addr learned my addr 172.21.15.102:0/718283128 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.476+0000 7f6cde59c700 1 -- 172.21.15.102:0/718283128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce0100250 msgr2=0x7f6ce0195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.476+0000 7f6cde59c700 1 --2- 172.21.15.102:0/718283128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce0100250 0x7f6ce0195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.476+0000 7f6cde59c700 1 -- 172.21.15.102:0/718283128 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd40095f0 con 0x7f6ce00ff050 2024-03-26T18:54:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.476+0000 7f6cddd9b700 1 --2- 172.21.15.102:0/718283128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce0100250 0x7f6ce0195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:54:56.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.477+0000 7f6cde59c700 1 --2- 172.21.15.102:0/718283128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ce00ff050 0x7f6ce0195980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6cd000e8f0 tx=0x7f6cd000ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:56.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.477+0000 7f6ccf7fe700 1 -- 172.21.15.102:0/718283128 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cd0009790 con 0x7f6ce00ff050 2024-03-26T18:54:56.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.477+0000 7f6ccf7fe700 1 -- 172.21.15.102:0/718283128 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6cd000c810 con 0x7f6ce00ff050 2024-03-26T18:54:56.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.477+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/718283128 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce01014d0 con 0x7f6ce00ff050 2024-03-26T18:54:56.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.477+0000 7f6ccf7fe700 1 -- 172.21.15.102:0/718283128 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cd0003d20 con 0x7f6ce00ff050 2024-03-26T18:54:56.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.477+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/718283128 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce0101a20 con 0x7f6ce00ff050 2024-03-26T18:54:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.478+0000 7f6ccf7fe700 1 -- 172.21.15.102:0/718283128 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 24) v1 ==== 46435+0+0 (secure 0 0 0) 0x7f6cd0018430 con 0x7f6ce00ff050 2024-03-26T18:54:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.478+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/718283128 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce0107ba0 con 0x7f6ce00ff050 2024-03-26T18:54:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.478+0000 7f6ccf7fe700 1 --2- 172.21.15.102:0/718283128 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6cc8039490 0x7f6cc803b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.479+0000 7f6ccf7fe700 1 -- 172.21.15.102:0/718283128 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f6cd0014070 con 0x7f6ce00ff050 2024-03-26T18:54:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.479+0000 7f6cddd9b700 1 --2- 172.21.15.102:0/718283128 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6cc8039490 0x7f6cc803b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.479+0000 7f6cddd9b700 1 --2- 172.21.15.102:0/718283128 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6cc8039490 0x7f6cc803b940 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009fd0 tx=0x7f6cd4005940 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:56.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.487+0000 7f6ccf7fe700 1 -- 172.21.15.102:0/718283128 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f6cd00217b0 con 0x7f6ce00ff050 2024-03-26T18:54:56.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:56 smithi190 ceph-mon[35043]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:56.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:56 smithi190 ceph-mon[35043]: pgmap v7: 65 pgs: 65 active+clean; 3.0 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 27 MiB/s rd, 46 MiB/s wr, 230 op/s 2024-03-26T18:54:56.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:56 smithi190 ceph-mon[35043]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:56.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.955+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/718283128 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6ce01020c0 con 0x7f6ce00ff050 2024-03-26T18:54:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.956+0000 7f6ccf7fe700 1 -- 172.21.15.102:0/718283128 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f6cd000f3c0 con 0x7f6ce00ff050 2024-03-26T18:54:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:54:56.949 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:54:56.949 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-03-26T18:54:56.949 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:54:56.949 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 1 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 12, 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 1 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:54:56.950 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:54:56.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.959+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/718283128 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6cc8039490 msgr2=0x7f6cc803b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:56.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.959+0000 7f6ce4eb5700 1 --2- 172.21.15.102:0/718283128 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6cc8039490 0x7f6cc803b940 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009fd0 tx=0x7f6cd4005940 comp rx=0 tx=0).stop 2024-03-26T18:54:56.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.959+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/718283128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ce00ff050 msgr2=0x7f6ce0195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:56.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.959+0000 7f6ce4eb5700 1 --2- 172.21.15.102:0/718283128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ce00ff050 0x7f6ce0195980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6cd000e8f0 tx=0x7f6cd000ec00 comp rx=0 tx=0).stop 2024-03-26T18:54:56.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.959+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/718283128 shutdown_connections 2024-03-26T18:54:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.959+0000 7f6ce4eb5700 1 --2- 172.21.15.102:0/718283128 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f6cc8039490 0x7f6cc803b940 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.959+0000 7f6ce4eb5700 1 --2- 172.21.15.102:0/718283128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ce00ff050 0x7f6ce0195980 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.959+0000 7f6ce4eb5700 1 --2- 172.21.15.102:0/718283128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce0100250 0x7f6ce0195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.959+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/718283128 >> 172.21.15.102:0/718283128 conn(0x7f6ce00fa5e0 msgr2=0x7f6ce00683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.960+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/718283128 shutdown_connections 2024-03-26T18:54:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:56.960+0000 7f6ce4eb5700 1 -- 172.21.15.102:0/718283128 wait complete. 2024-03-26T18:54:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.079+0000 7f61bc25d700 1 -- 172.21.15.102:0/3852132243 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b4100150 msgr2=0x7f61b4100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.079+0000 7f61bc25d700 1 --2- 172.21.15.102:0/3852132243 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b4100150 0x7f61b4100560 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009910 tx=0x7f61b0009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.079+0000 7f61bc25d700 1 -- 172.21.15.102:0/3852132243 shutdown_connections 2024-03-26T18:54:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.079+0000 7f61bc25d700 1 --2- 172.21.15.102:0/3852132243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b4101350 0x7f61b41017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.079+0000 7f61bc25d700 1 --2- 172.21.15.102:0/3852132243 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b4100150 0x7f61b4100560 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.079+0000 7f61bc25d700 1 -- 172.21.15.102:0/3852132243 >> 172.21.15.102:0/3852132243 conn(0x7f61b40fb6e0 msgr2=0x7f61b40fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.079+0000 7f61bc25d700 1 -- 172.21.15.102:0/3852132243 shutdown_connections 2024-03-26T18:54:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.079+0000 7f61bc25d700 1 -- 172.21.15.102:0/3852132243 wait complete. 2024-03-26T18:54:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61bc25d700 1 Processor -- start 2024-03-26T18:54:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61bc25d700 1 -- start start 2024-03-26T18:54:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61bc25d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b4100150 0x7f61b419bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61bc25d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b4101350 0x7f61b419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61bc25d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b419cae0 con 0x7f61b4100150 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61bc25d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b419cc20 con 0x7f61b4101350 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61b9ff9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b4100150 0x7f61b419bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61b97f8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b4101350 0x7f61b419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61b9ff9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b4100150 0x7f61b419bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37920/0 (socket says 172.21.15.102:37920) 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61b97f8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b4101350 0x7f61b419c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:39234/0 (socket says 172.21.15.102:39234) 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61b9ff9700 1 -- 172.21.15.102:0/3409065323 learned_addr learned my addr 172.21.15.102:0/3409065323 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61b97f8700 1 -- 172.21.15.102:0/3409065323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b4100150 msgr2=0x7f61b419bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61b97f8700 1 --2- 172.21.15.102:0/3409065323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b4100150 0x7f61b419bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.080+0000 7f61b97f8700 1 -- 172.21.15.102:0/3409065323 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b00095f0 con 0x7f61b4101350 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.081+0000 7f61b9ff9700 1 --2- 172.21.15.102:0/3409065323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b4100150 0x7f61b419bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.081+0000 7f61b97f8700 1 --2- 172.21.15.102:0/3409065323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b4101350 0x7f61b419c4c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f61a400e970 tx=0x7f61a400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.081+0000 7f61aaffd700 1 -- 172.21.15.102:0/3409065323 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61a400c9d0 con 0x7f61b4101350 2024-03-26T18:54:57.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.081+0000 7f61bc25d700 1 -- 172.21.15.102:0/3409065323 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b41a16d0 con 0x7f61b4101350 2024-03-26T18:54:57.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.081+0000 7f61aaffd700 1 -- 172.21.15.102:0/3409065323 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f61a4010430 con 0x7f61b4101350 2024-03-26T18:54:57.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.081+0000 7f61aaffd700 1 -- 172.21.15.102:0/3409065323 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61a4018650 con 0x7f61b4101350 2024-03-26T18:54:57.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.081+0000 7f61bc25d700 1 -- 172.21.15.102:0/3409065323 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b41a1c20 con 0x7f61b4101350 2024-03-26T18:54:57.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.082+0000 7f61aaffd700 1 -- 172.21.15.102:0/3409065323 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 24) v1 ==== 46435+0+0 (secure 0 0 0) 0x7f61a40188c0 con 0x7f61b4101350 2024-03-26T18:54:57.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.082+0000 7f61bc25d700 1 -- 172.21.15.102:0/3409065323 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b4105430 con 0x7f61b4101350 2024-03-26T18:54:57.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.082+0000 7f61aaffd700 1 --2- 172.21.15.102:0/3409065323 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f61a0039490 0x7f61a003b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:57.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.082+0000 7f61aaffd700 1 -- 172.21.15.102:0/3409065323 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f61a4014070 con 0x7f61b4101350 2024-03-26T18:54:57.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.083+0000 7f61b9ff9700 1 --2- 172.21.15.102:0/3409065323 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f61a0039490 0x7f61a003b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:57.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.083+0000 7f61b9ff9700 1 --2- 172.21.15.102:0/3409065323 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f61a0039490 0x7f61a003b940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009fd0 tx=0x7f61b0005940 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:57.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.090+0000 7f61aaffd700 1 -- 172.21.15.102:0/3409065323 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f61a401fbe0 con 0x7f61b4101350 2024-03-26T18:54:57.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.290+0000 7f61aaffd700 1 -- 172.21.15.102:0/3409065323 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mgrmap(e 25) v1 ==== 91096+0+0 (secure 0 0 0) 0x7f61a4010610 con 0x7f61b4101350 2024-03-26T18:54:57.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.434+0000 7f61bc25d700 1 -- 172.21.15.102:0/3409065323 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f61b402cd20 con 0x7f61b4101350 2024-03-26T18:54:57.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.435+0000 7f61aaffd700 1 -- 172.21.15.102:0/3409065323 <== mon.1 v2:172.21.15.190:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1854 (secure 0 0 0) 0x7f61a405d4f0 con 0x7f61b4101350 2024-03-26T18:54:57.428 INFO:teuthology.orchestra.run.smithi102.stdout:e18 2024-03-26T18:54:57.428 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:54:57.428 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:54:57.428 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:54:57.428 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:54:57.428 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:54:57.428 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:54:57.428 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 18 2024-03-26T18:54:57.428 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:54:57.428 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:53:56.228104+0000 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:54:57.429 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:54:57.430 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:54:57.431 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:54:57.431 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:54:57.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.438+0000 7f61bc25d700 1 -- 172.21.15.102:0/3409065323 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f61a0039490 msgr2=0x7f61a003b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.438+0000 7f61bc25d700 1 --2- 172.21.15.102:0/3409065323 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f61a0039490 0x7f61a003b940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009fd0 tx=0x7f61b0005940 comp rx=0 tx=0).stop 2024-03-26T18:54:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.438+0000 7f61bc25d700 1 -- 172.21.15.102:0/3409065323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b4101350 msgr2=0x7f61b419c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.438+0000 7f61bc25d700 1 --2- 172.21.15.102:0/3409065323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b4101350 0x7f61b419c4c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f61a400e970 tx=0x7f61a400ec80 comp rx=0 tx=0).stop 2024-03-26T18:54:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.438+0000 7f61bc25d700 1 -- 172.21.15.102:0/3409065323 shutdown_connections 2024-03-26T18:54:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.438+0000 7f61bc25d700 1 --2- 172.21.15.102:0/3409065323 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f61a0039490 0x7f61a003b940 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.438+0000 7f61bc25d700 1 --2- 172.21.15.102:0/3409065323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b4100150 0x7f61b419bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.438+0000 7f61bc25d700 1 --2- 172.21.15.102:0/3409065323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b4101350 0x7f61b419c4c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.438+0000 7f61bc25d700 1 -- 172.21.15.102:0/3409065323 >> 172.21.15.102:0/3409065323 conn(0x7f61b40fb6e0 msgr2=0x7f61b4102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.438+0000 7f61bc25d700 1 -- 172.21.15.102:0/3409065323 shutdown_connections 2024-03-26T18:54:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.438+0000 7f61bc25d700 1 -- 172.21.15.102:0/3409065323 wait complete. 2024-03-26T18:54:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 18 2024-03-26T18:54:57.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:57 smithi190 ceph-mon[35043]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:57.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:57 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/718283128' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:54:57.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:57 smithi190 ceph-mon[35043]: Standby manager daemon smithi102.rsfpnv started 2024-03-26T18:54:57.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:57 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.102:0/2727088503' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.rsfpnv/crt"}]: dispatch 2024-03-26T18:54:57.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:57 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.102:0/2727088503' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:54:57.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:57 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.102:0/2727088503' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.rsfpnv/key"}]: dispatch 2024-03-26T18:54:57.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:57 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.102:0/2727088503' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:54:57.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.559+0000 7fddde1ad700 1 -- 172.21.15.102:0/2941248921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd80fe430 msgr2=0x7fddd80fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:57.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.559+0000 7fddde1ad700 1 --2- 172.21.15.102:0/2941248921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd80fe430 0x7fddd80fe840 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fddc8009910 tx=0x7fddc8009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:57.552 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:57 smithi102 ceph-mon[29002]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:57.552 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:57 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/718283128' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:54:57.552 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:57 smithi102 ceph-mon[29002]: Standby manager daemon smithi102.rsfpnv started 2024-03-26T18:54:57.553 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:57 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.102:0/2727088503' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.rsfpnv/crt"}]: dispatch 2024-03-26T18:54:57.553 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:57 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.102:0/2727088503' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:54:57.553 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:57 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.102:0/2727088503' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.rsfpnv/key"}]: dispatch 2024-03-26T18:54:57.553 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:57 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.102:0/2727088503' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:54:57.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.560+0000 7fddde1ad700 1 -- 172.21.15.102:0/2941248921 shutdown_connections 2024-03-26T18:54:57.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.560+0000 7fddde1ad700 1 --2- 172.21.15.102:0/2941248921 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddd81014b0 0x7fddd8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.560+0000 7fddde1ad700 1 --2- 172.21.15.102:0/2941248921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd80fe430 0x7fddd80fe840 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.560+0000 7fddde1ad700 1 -- 172.21.15.102:0/2941248921 >> 172.21.15.102:0/2941248921 conn(0x7fddd80f9d80 msgr2=0x7fddd80fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:57.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.560+0000 7fddde1ad700 1 -- 172.21.15.102:0/2941248921 shutdown_connections 2024-03-26T18:54:57.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.561+0000 7fddde1ad700 1 -- 172.21.15.102:0/2941248921 wait complete. 2024-03-26T18:54:57.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.561+0000 7fddde1ad700 1 Processor -- start 2024-03-26T18:54:57.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.562+0000 7fddde1ad700 1 -- start start 2024-03-26T18:54:57.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.562+0000 7fddde1ad700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddd80fe430 0x7fddd8195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:57.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.562+0000 7fddde1ad700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd81014b0 0x7fddd8195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:57.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.562+0000 7fddde1ad700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd81964e0 con 0x7fddd81014b0 2024-03-26T18:54:57.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.562+0000 7fddde1ad700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd8196620 con 0x7fddd80fe430 2024-03-26T18:54:57.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.562+0000 7fddd6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd81014b0 0x7fddd8195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:57.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.562+0000 7fddd6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd81014b0 0x7fddd8195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37942/0 (socket says 172.21.15.102:37942) 2024-03-26T18:54:57.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.562+0000 7fddd6ffd700 1 -- 172.21.15.102:0/588236576 learned_addr learned my addr 172.21.15.102:0/588236576 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:57.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.562+0000 7fddd77fe700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddd80fe430 0x7fddd8195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:57.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.563+0000 7fddd77fe700 1 -- 172.21.15.102:0/588236576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd81014b0 msgr2=0x7fddd8195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:57.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.563+0000 7fddd77fe700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd81014b0 0x7fddd8195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.563+0000 7fddd77fe700 1 -- 172.21.15.102:0/588236576 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc80095f0 con 0x7fddd80fe430 2024-03-26T18:54:57.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.563+0000 7fddd6ffd700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd81014b0 0x7fddd8195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:54:57.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.563+0000 7fddd77fe700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddd80fe430 0x7fddd8195980 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fddc8000c00 tx=0x7fddc800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:57.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.563+0000 7fddd4ff9700 1 -- 172.21.15.102:0/588236576 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddc801d070 con 0x7fddd80fe430 2024-03-26T18:54:57.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.563+0000 7fddd4ff9700 1 -- 172.21.15.102:0/588236576 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fddc8005dc0 con 0x7fddd80fe430 2024-03-26T18:54:57.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.563+0000 7fddde1ad700 1 -- 172.21.15.102:0/588236576 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd80ff2a0 con 0x7fddd80fe430 2024-03-26T18:54:57.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.564+0000 7fddd4ff9700 1 -- 172.21.15.102:0/588236576 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddc8022470 con 0x7fddd80fe430 2024-03-26T18:54:57.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.564+0000 7fddde1ad700 1 -- 172.21.15.102:0/588236576 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd80ff790 con 0x7fddd80fe430 2024-03-26T18:54:57.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.565+0000 7fddde1ad700 1 -- 172.21.15.102:0/588236576 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd8108c90 con 0x7fddd80fe430 2024-03-26T18:54:57.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.565+0000 7fddd4ff9700 1 -- 172.21.15.102:0/588236576 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 25) v1 ==== 91096+0+0 (secure 0 0 0) 0x7fddc8003810 con 0x7fddd80fe430 2024-03-26T18:54:57.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.566+0000 7fddd4ff9700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fddc006d540 0x7fddc006f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:57.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.566+0000 7fddd4ff9700 1 -- 172.21.15.102:0/588236576 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fddc808e200 con 0x7fddd80fe430 2024-03-26T18:54:57.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.566+0000 7fddd6ffd700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fddc006d540 0x7fddc006f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:57.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.567+0000 7fddd6ffd700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fddc006d540 0x7fddc006f9f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fddcc011550 tx=0x7fddcc009260 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:57.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.571+0000 7fddd4ff9700 1 -- 172.21.15.102:0/588236576 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7fddc805b320 con 0x7fddd80fe430 2024-03-26T18:54:57.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.948+0000 7fddde1ad700 1 -- 172.21.15.102:0/588236576 --> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fddd8105e00 con 0x7fddc006d540 2024-03-26T18:54:57.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.949+0000 7fddd4ff9700 1 -- 172.21.15.102:0/588236576 <== mgr.24423 v2:172.21.15.190:6828/2858265260 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fddd8105e00 con 0x7fddc006d540 2024-03-26T18:54:57.942 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:54:57.942 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:54:57.942 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:54:57.942 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:54:57.942 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-03-26T18:54:57.942 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "1/23 daemons upgraded", 2024-03-26T18:54:57.942 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "", 2024-03-26T18:54:57.942 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:54:57.942 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:54:57.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.951+0000 7fddde1ad700 1 -- 172.21.15.102:0/588236576 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fddc006d540 msgr2=0x7fddc006f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:57.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.951+0000 7fddde1ad700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fddc006d540 0x7fddc006f9f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fddcc011550 tx=0x7fddcc009260 comp rx=0 tx=0).stop 2024-03-26T18:54:57.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.951+0000 7fddde1ad700 1 -- 172.21.15.102:0/588236576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddd80fe430 msgr2=0x7fddd8195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:57.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.951+0000 7fddde1ad700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddd80fe430 0x7fddd8195980 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fddc8000c00 tx=0x7fddc800f690 comp rx=0 tx=0).stop 2024-03-26T18:54:57.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.951+0000 7fddde1ad700 1 -- 172.21.15.102:0/588236576 shutdown_connections 2024-03-26T18:54:57.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.951+0000 7fddde1ad700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fddc006d540 0x7fddc006f9f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.951+0000 7fddde1ad700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddd80fe430 0x7fddd8195980 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.951+0000 7fddde1ad700 1 --2- 172.21.15.102:0/588236576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd81014b0 0x7fddd8195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.951+0000 7fddde1ad700 1 -- 172.21.15.102:0/588236576 >> 172.21.15.102:0/588236576 conn(0x7fddd80f9d80 msgr2=0x7fddd81046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.951+0000 7fddde1ad700 1 -- 172.21.15.102:0/588236576 shutdown_connections 2024-03-26T18:54:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:57.951+0000 7fddde1ad700 1 -- 172.21.15.102:0/588236576 wait complete. 2024-03-26T18:54:58.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.083+0000 7f1bceb42700 1 -- 172.21.15.102:0/71428737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bc8100c40 msgr2=0x7f1bc8103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.083+0000 7f1bceb42700 1 --2- 172.21.15.102:0/71428737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bc8100c40 0x7f1bc8103020 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009910 tx=0x7f1bbc009c20 comp rx=0 tx=0).stop 2024-03-26T18:54:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.083+0000 7f1bceb42700 1 -- 172.21.15.102:0/71428737 shutdown_connections 2024-03-26T18:54:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.083+0000 7f1bceb42700 1 --2- 172.21.15.102:0/71428737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc8103560 0x7f1bc8105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.083+0000 7f1bceb42700 1 --2- 172.21.15.102:0/71428737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bc8100c40 0x7f1bc8103020 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.083+0000 7f1bceb42700 1 -- 172.21.15.102:0/71428737 >> 172.21.15.102:0/71428737 conn(0x7f1bc80fa5e0 msgr2=0x7f1bc80fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.083+0000 7f1bceb42700 1 -- 172.21.15.102:0/71428737 shutdown_connections 2024-03-26T18:54:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.083+0000 7f1bceb42700 1 -- 172.21.15.102:0/71428737 wait complete. 2024-03-26T18:54:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.084+0000 7f1bceb42700 1 Processor -- start 2024-03-26T18:54:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.084+0000 7f1bceb42700 1 -- start start 2024-03-26T18:54:58.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.084+0000 7f1bceb42700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc8100c40 0x7f1bc81005a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.084+0000 7f1bceb42700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bc8103560 0x7f1bc80febf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.084+0000 7f1bceb42700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc80ff130 con 0x7f1bc8103560 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.084+0000 7f1bceb42700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc80ff270 con 0x7f1bc8100c40 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.084+0000 7f1bc7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bc8103560 0x7f1bc80febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.084+0000 7f1bc7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bc8103560 0x7f1bc80febf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37952/0 (socket says 172.21.15.102:37952) 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.084+0000 7f1bcc8de700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc8100c40 0x7f1bc81005a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.084+0000 7f1bc7fff700 1 -- 172.21.15.102:0/122068740 learned_addr learned my addr 172.21.15.102:0/122068740 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.085+0000 7f1bcc8de700 1 -- 172.21.15.102:0/122068740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bc8103560 msgr2=0x7f1bc80febf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.085+0000 7f1bcc8de700 1 --2- 172.21.15.102:0/122068740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bc8103560 0x7f1bc80febf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.085+0000 7f1bcc8de700 1 -- 172.21.15.102:0/122068740 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc0095f0 con 0x7f1bc8100c40 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.085+0000 7f1bc7fff700 1 --2- 172.21.15.102:0/122068740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bc8103560 0x7f1bc80febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.085+0000 7f1bcc8de700 1 --2- 172.21.15.102:0/122068740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc8100c40 0x7f1bc81005a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc000c00 tx=0x7f1bbc00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.085+0000 7f1bc5ffb700 1 -- 172.21.15.102:0/122068740 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bbc01d070 con 0x7f1bc8100c40 2024-03-26T18:54:58.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.085+0000 7f1bceb42700 1 -- 172.21.15.102:0/122068740 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc80ff4f0 con 0x7f1bc8100c40 2024-03-26T18:54:58.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.085+0000 7f1bc5ffb700 1 -- 172.21.15.102:0/122068740 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1bbc005dc0 con 0x7f1bc8100c40 2024-03-26T18:54:58.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.085+0000 7f1bc5ffb700 1 -- 172.21.15.102:0/122068740 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bbc017400 con 0x7f1bc8100c40 2024-03-26T18:54:58.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.086+0000 7f1bceb42700 1 -- 172.21.15.102:0/122068740 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc80ff9e0 con 0x7f1bc8100c40 2024-03-26T18:54:58.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.087+0000 7f1bceb42700 1 -- 172.21.15.102:0/122068740 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc818fb70 con 0x7f1bc8100c40 2024-03-26T18:54:58.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.087+0000 7f1bc5ffb700 1 -- 172.21.15.102:0/122068740 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 25) v1 ==== 91096+0+0 (secure 0 0 0) 0x7f1bbc003810 con 0x7f1bc8100c40 2024-03-26T18:54:58.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.088+0000 7f1bc5ffb700 1 --2- 172.21.15.102:0/122068740 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f1bb806d4f0 0x7f1bb806f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:54:58.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.088+0000 7f1bc5ffb700 1 -- 172.21.15.102:0/122068740 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f1bbc08e170 con 0x7f1bc8100c40 2024-03-26T18:54:58.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.088+0000 7f1bc7fff700 1 --2- 172.21.15.102:0/122068740 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f1bb806d4f0 0x7f1bb806f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:54:58.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.089+0000 7f1bc7fff700 1 --2- 172.21.15.102:0/122068740 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f1bb806d4f0 0x7f1bb806f9a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4011550 tx=0x7f1bb4009260 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:54:58.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.093+0000 7f1bc5ffb700 1 -- 172.21.15.102:0/122068740 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f1bbc05b330 con 0x7f1bc8100c40 2024-03-26T18:54:58.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.533+0000 7f1bceb42700 1 -- 172.21.15.102:0/122068740 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1bc802cea0 con 0x7f1bc8100c40 2024-03-26T18:54:58.526 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:58 smithi102 ceph-mon[29002]: mgrmap e25: smithi190.zrixos(active, since 9s), standbys: smithi102.rsfpnv 2024-03-26T18:54:58.526 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:58 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mgr metadata", "who": "smithi102.rsfpnv", "id": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:54:58.526 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:58 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3409065323' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:54:58.526 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:58 smithi102 ceph-mon[29002]: pgmap v8: 65 pgs: 65 active+clean; 3.0 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 21 MiB/s rd, 36 MiB/s wr, 179 op/s 2024-03-26T18:54:58.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.533+0000 7f1bc5ffb700 1 -- 172.21.15.102:0/122068740 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f1bbc020370 con 0x7f1bc8100c40 2024-03-26T18:54:58.527 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T18:54:58.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.536+0000 7f1bceb42700 1 -- 172.21.15.102:0/122068740 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f1bb806d4f0 msgr2=0x7f1bb806f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:58.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.536+0000 7f1bceb42700 1 --2- 172.21.15.102:0/122068740 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f1bb806d4f0 0x7f1bb806f9a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4011550 tx=0x7f1bb4009260 comp rx=0 tx=0).stop 2024-03-26T18:54:58.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.536+0000 7f1bceb42700 1 -- 172.21.15.102:0/122068740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc8100c40 msgr2=0x7f1bc81005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:54:58.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.536+0000 7f1bceb42700 1 --2- 172.21.15.102:0/122068740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc8100c40 0x7f1bc81005a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc000c00 tx=0x7f1bbc00f690 comp rx=0 tx=0).stop 2024-03-26T18:54:58.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.536+0000 7f1bceb42700 1 -- 172.21.15.102:0/122068740 shutdown_connections 2024-03-26T18:54:58.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.536+0000 7f1bceb42700 1 --2- 172.21.15.102:0/122068740 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f1bb806d4f0 0x7f1bb806f9a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:58.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.536+0000 7f1bceb42700 1 --2- 172.21.15.102:0/122068740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc8100c40 0x7f1bc81005a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:58.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.536+0000 7f1bceb42700 1 --2- 172.21.15.102:0/122068740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bc8103560 0x7f1bc80febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:54:58.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.536+0000 7f1bceb42700 1 -- 172.21.15.102:0/122068740 >> 172.21.15.102:0/122068740 conn(0x7f1bc80fa5e0 msgr2=0x7f1bc80fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:54:58.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.537+0000 7f1bceb42700 1 -- 172.21.15.102:0/122068740 shutdown_connections 2024-03-26T18:54:58.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:54:58.537+0000 7f1bceb42700 1 -- 172.21.15.102:0/122068740 wait complete. 2024-03-26T18:54:58.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:58 smithi190 ceph-mon[35043]: mgrmap e25: smithi190.zrixos(active, since 9s), standbys: smithi102.rsfpnv 2024-03-26T18:54:58.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:58 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mgr metadata", "who": "smithi102.rsfpnv", "id": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:54:58.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:58 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3409065323' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:54:58.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:58 smithi190 ceph-mon[35043]: pgmap v8: 65 pgs: 65 active+clean; 3.0 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 21 MiB/s rd, 36 MiB/s wr, 179 op/s 2024-03-26T18:54:59.563 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T18:54:59.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-26T18:54:59.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:59 smithi102 ceph-mon[29002]: from='client.24471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:59.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:59 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/122068740' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:54:59.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:59 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:54:59.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:54:59 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:54:59.637 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:59 smithi190 ceph-mon[35043]: from='client.24471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:54:59.637 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:59 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/122068740' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:54:59.637 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:59 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:54:59.637 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:54:59 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:00.620 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:00 smithi190 ceph-mon[35043]: pgmap v9: 65 pgs: 65 active+clean; 1.3 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 17 MiB/s rd, 30 MiB/s wr, 272 op/s 2024-03-26T18:55:00.620 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:00 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:00.620 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:00 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:00.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:00 smithi102 ceph-mon[29002]: pgmap v9: 65 pgs: 65 active+clean; 1.3 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 17 MiB/s rd, 30 MiB/s wr, 272 op/s 2024-03-26T18:55:00.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:00 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:00.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:00 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:01.712 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T18:55:01.712 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T18:55:01.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:01 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:01.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:01 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:01.786 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:01 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:01.787 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:01 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:02.832 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:02 smithi190 ceph-mon[35043]: pgmap v10: 65 pgs: 65 active+clean; 1.3 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 16 MiB/s rd, 28 MiB/s wr, 249 op/s 2024-03-26T18:55:02.832 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:02 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:02.832 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:02 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:02.833 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:02 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.833 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:02 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.833 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:02 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.833 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:02 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.833 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:02 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.833 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:02 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.833 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:02 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:02.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:02 smithi102 ceph-mon[29002]: pgmap v10: 65 pgs: 65 active+clean; 1.3 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 16 MiB/s rd, 28 MiB/s wr, 249 op/s 2024-03-26T18:55:02.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:02 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:02.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:02 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:02.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:02 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:02 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:02 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:02 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:02 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:02 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:02.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:02 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:03.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:03 smithi102 ceph-mon[29002]: Adjusting osd_memory_target on smithi102 to 1216M 2024-03-26T18:55:03.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:03 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:55:04.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:03 smithi190 ceph-mon[35043]: Adjusting osd_memory_target on smithi102 to 1216M 2024-03-26T18:55:04.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:03 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:55:04.840 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:04 smithi190 ceph-mon[35043]: pgmap v11: 65 pgs: 65 active+clean; 1009 MiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 16 MiB/s rd, 28 MiB/s wr, 301 op/s 2024-03-26T18:55:04.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:04 smithi102 ceph-mon[29002]: pgmap v11: 65 pgs: 65 active+clean; 1009 MiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 16 MiB/s rd, 28 MiB/s wr, 301 op/s 2024-03-26T18:55:06.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:06 smithi102 ceph-mon[29002]: pgmap v12: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 19 MiB/s wr, 349 op/s 2024-03-26T18:55:06.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:06 smithi190 ceph-mon[35043]: pgmap v12: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 19 MiB/s wr, 349 op/s 2024-03-26T18:55:09.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:08 smithi102 ceph-mon[29002]: pgmap v13: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 36 KiB/s rd, 1.7 MiB/s wr, 244 op/s 2024-03-26T18:55:09.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:08 smithi190 ceph-mon[35043]: pgmap v13: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 36 KiB/s rd, 1.7 MiB/s wr, 244 op/s 2024-03-26T18:55:11.064 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:10 smithi190 ceph-mon[35043]: pgmap v14: 65 pgs: 65 active+clean; 303 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 45 KiB/s rd, 2.6 MiB/s wr, 330 op/s 2024-03-26T18:55:11.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:10 smithi102 ceph-mon[29002]: pgmap v14: 65 pgs: 65 active+clean; 303 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 45 KiB/s rd, 2.6 MiB/s wr, 330 op/s 2024-03-26T18:55:13.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:12 smithi190 ceph-mon[35043]: pgmap v15: 65 pgs: 65 active+clean; 303 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 25 KiB/s rd, 1.7 MiB/s wr, 215 op/s 2024-03-26T18:55:13.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:12 smithi102 ceph-mon[29002]: pgmap v15: 65 pgs: 65 active+clean; 303 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 25 KiB/s rd, 1.7 MiB/s wr, 215 op/s 2024-03-26T18:55:14.953 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:14 smithi190 ceph-mon[35043]: pgmap v16: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 39 KiB/s rd, 1.7 MiB/s wr, 251 op/s 2024-03-26T18:55:15.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:14 smithi102 ceph-mon[29002]: pgmap v16: 65 pgs: 65 active+clean; 303 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 39 KiB/s rd, 1.7 MiB/s wr, 251 op/s 2024-03-26T18:55:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: Adjusting osd_memory_target on smithi190 to 2240M 2024-03-26T18:55:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:16.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:55:16.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:55:16.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-26T18:55:16.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-26T18:55:16.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:55:16.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:55:16.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:55:16.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:16 smithi102 ceph-mon[29002]: pgmap v17: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 2.0 MiB/s wr, 283 op/s 2024-03-26T18:55:16.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:16.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:16.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: Adjusting osd_memory_target on smithi190 to 2240M 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:55:16.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:16 smithi190 ceph-mon[35043]: pgmap v17: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 2.0 MiB/s wr, 283 op/s 2024-03-26T18:55:17.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:17 smithi102 ceph-mon[29002]: Reconfiguring prometheus.smithi102 (dependencies changed)... 2024-03-26T18:55:17.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:55:17.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:55:17.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:55:17.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:17.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:17 smithi190 ceph-mon[35043]: Reconfiguring prometheus.smithi102 (dependencies changed)... 2024-03-26T18:55:18.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:18 smithi102 ceph-mon[29002]: Reconfiguring daemon prometheus.smithi102 on smithi102 2024-03-26T18:55:18.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:18 smithi102 ceph-mon[29002]: pgmap v18: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-03-26T18:55:18.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:18 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:55:18.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:18 smithi190 ceph-mon[35043]: Reconfiguring daemon prometheus.smithi102 on smithi102 2024-03-26T18:55:18.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:18 smithi190 ceph-mon[35043]: pgmap v18: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-03-26T18:55:18.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:18 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:55:20.901 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:20 smithi102 ceph-mon[29002]: pgmap v19: 65 pgs: 65 active+clean; 313 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 38 KiB/s rd, 2.1 MiB/s wr, 338 op/s 2024-03-26T18:55:21.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:20 smithi190 ceph-mon[35043]: pgmap v19: 65 pgs: 65 active+clean; 313 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 38 KiB/s rd, 2.1 MiB/s wr, 338 op/s 2024-03-26T18:55:23.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:22 smithi102 ceph-mon[29002]: pgmap v20: 65 pgs: 65 active+clean; 313 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 253 op/s 2024-03-26T18:55:23.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:22 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:23.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:22 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:23.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:22 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:55:23.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:22 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:55:23.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:22 smithi190 ceph-mon[35043]: pgmap v20: 65 pgs: 65 active+clean; 313 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 253 op/s 2024-03-26T18:55:23.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:22 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:23.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:22 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:23.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:22 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:55:23.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:22 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:55:24.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:23 smithi190 ceph-mon[35043]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:55:24.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:23 smithi190 ceph-mon[35043]: Upgrade: Need to upgrade myself (mgr.smithi190.zrixos) 2024-03-26T18:55:24.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:23 smithi190 ceph-mon[35043]: Upgrade: Need to upgrade myself (mgr.smithi190.zrixos) 2024-03-26T18:55:24.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:23 smithi102 ceph-mon[29002]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:55:24.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:23 smithi102 ceph-mon[29002]: Upgrade: Need to upgrade myself (mgr.smithi190.zrixos) 2024-03-26T18:55:24.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:23 smithi102 ceph-mon[29002]: Upgrade: Need to upgrade myself (mgr.smithi190.zrixos) 2024-03-26T18:55:25.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:24 smithi102 ceph-mon[29002]: pgmap v21: 65 pgs: 65 active+clean; 313 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 47 KiB/s rd, 1.4 MiB/s wr, 325 op/s 2024-03-26T18:55:25.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:24 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:25.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:24 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.rsfpnv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:55:25.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:24 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.rsfpnv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:55:25.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:24 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:55:25.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:24 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:55:25.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:24 smithi190 ceph-mon[35043]: pgmap v21: 65 pgs: 65 active+clean; 313 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 47 KiB/s rd, 1.4 MiB/s wr, 325 op/s 2024-03-26T18:55:25.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:24 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:25.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:24 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.rsfpnv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:55:25.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:24 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.rsfpnv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:55:25.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:24 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:55:25.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:24 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:55:25.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:25 smithi102 ceph-mon[29002]: Upgrade: Updating mgr.smithi102.rsfpnv 2024-03-26T18:55:25.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:25 smithi102 ceph-mon[29002]: Deploying daemon mgr.smithi102.rsfpnv on smithi102 2024-03-26T18:55:26.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:25 smithi190 ceph-mon[35043]: Upgrade: Updating mgr.smithi102.rsfpnv 2024-03-26T18:55:26.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:25 smithi190 ceph-mon[35043]: Deploying daemon mgr.smithi102.rsfpnv on smithi102 2024-03-26T18:55:27.065 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:26 smithi102 ceph-mon[29002]: pgmap v22: 65 pgs: 65 active+clean; 317 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 32 KiB/s rd, 2.0 MiB/s wr, 424 op/s 2024-03-26T18:55:27.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:26 smithi190 ceph-mon[35043]: pgmap v22: 65 pgs: 65 active+clean; 317 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 32 KiB/s rd, 2.0 MiB/s wr, 424 op/s 2024-03-26T18:55:28.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.658+0000 7f97a9bed700 1 -- 172.21.15.102:0/855757886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a40ff260 msgr2=0x7f97a40ff6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:28.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.658+0000 7f97a9bed700 1 --2- 172.21.15.102:0/855757886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a40ff260 0x7f97a40ff6d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9798009910 tx=0x7f9798009c20 comp rx=0 tx=0).stop 2024-03-26T18:55:28.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.658+0000 7f97a9bed700 1 -- 172.21.15.102:0/855757886 shutdown_connections 2024-03-26T18:55:28.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.658+0000 7f97a9bed700 1 --2- 172.21.15.102:0/855757886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a40ff260 0x7f97a40ff6d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:28.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.658+0000 7f97a9bed700 1 --2- 172.21.15.102:0/855757886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a4100600 0x7f97a4100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:28.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.658+0000 7f97a9bed700 1 -- 172.21.15.102:0/855757886 >> 172.21.15.102:0/855757886 conn(0x7f97a40f9d80 msgr2=0x7f97a40fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:28.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.658+0000 7f97a9bed700 1 -- 172.21.15.102:0/855757886 shutdown_connections 2024-03-26T18:55:28.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.658+0000 7f97a9bed700 1 -- 172.21.15.102:0/855757886 wait complete. 2024-03-26T18:55:28.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.659+0000 7f97a9bed700 1 Processor -- start 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.659+0000 7f97a9bed700 1 -- start start 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.659+0000 7f97a9bed700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a40ff260 0x7f97a40fecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.659+0000 7f97a9bed700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a4100600 0x7f97a419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.659+0000 7f97a9bed700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a419ac80 con 0x7f97a4100600 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.659+0000 7f97a9bed700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a419adc0 con 0x7f97a40ff260 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.659+0000 7f97a2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a4100600 0x7f97a419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.660+0000 7f97a37fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a40ff260 0x7f97a40fecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.660+0000 7f97a2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a4100600 0x7f97a419a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48160/0 (socket says 172.21.15.102:48160) 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.660+0000 7f97a2ffd700 1 -- 172.21.15.102:0/4043138897 learned_addr learned my addr 172.21.15.102:0/4043138897 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.660+0000 7f97a37fe700 1 -- 172.21.15.102:0/4043138897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a4100600 msgr2=0x7f97a419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.660+0000 7f97a37fe700 1 --2- 172.21.15.102:0/4043138897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a4100600 0x7f97a419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.660+0000 7f97a37fe700 1 -- 172.21.15.102:0/4043138897 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97980095f0 con 0x7f97a40ff260 2024-03-26T18:55:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.660+0000 7f97a37fe700 1 --2- 172.21.15.102:0/4043138897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a40ff260 0x7f97a40fecb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f979400e9a0 tx=0x7f979400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:28.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.660+0000 7f97a0ff9700 1 -- 172.21.15.102:0/4043138897 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f979400cba0 con 0x7f97a40ff260 2024-03-26T18:55:28.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.660+0000 7f97a9bed700 1 -- 172.21.15.102:0/4043138897 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a419b0a0 con 0x7f97a40ff260 2024-03-26T18:55:28.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.660+0000 7f97a0ff9700 1 -- 172.21.15.102:0/4043138897 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f979400ee90 con 0x7f97a40ff260 2024-03-26T18:55:28.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.660+0000 7f97a0ff9700 1 -- 172.21.15.102:0/4043138897 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9794018650 con 0x7f97a40ff260 2024-03-26T18:55:28.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.661+0000 7f97a9bed700 1 -- 172.21.15.102:0/4043138897 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a419b5f0 con 0x7f97a40ff260 2024-03-26T18:55:28.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.662+0000 7f97a9bed700 1 -- 172.21.15.102:0/4043138897 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a4107b90 con 0x7f97a40ff260 2024-03-26T18:55:28.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.662+0000 7f97a0ff9700 1 -- 172.21.15.102:0/4043138897 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 25) v1 ==== 91096+0+0 (secure 0 0 0) 0x7f9794010430 con 0x7f97a40ff260 2024-03-26T18:55:28.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.662+0000 7f97a0ff9700 1 --2- 172.21.15.102:0/4043138897 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f978c06d4f0 0x7f978c06f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:28.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.662+0000 7f97a0ff9700 1 -- 172.21.15.102:0/4043138897 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f9794014070 con 0x7f97a40ff260 2024-03-26T18:55:28.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.663+0000 7f97a2ffd700 1 --2- 172.21.15.102:0/4043138897 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f978c06d4f0 0x7f978c06f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:28.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.663+0000 7f97a2ffd700 1 --2- 172.21.15.102:0/4043138897 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f978c06d4f0 0x7f978c06f9a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97a40ff230 tx=0x7f9798005940 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:28.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.668+0000 7f97a0ff9700 1 -- 172.21.15.102:0/4043138897 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f9794053470 con 0x7f97a40ff260 2024-03-26T18:55:28.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:28 smithi102 ceph-mon[29002]: pgmap v23: 65 pgs: 65 active+clean; 317 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 32 KiB/s rd, 1.4 MiB/s wr, 341 op/s 2024-03-26T18:55:28.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:28 smithi102 ceph-mon[29002]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:55:28.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.974+0000 7f97a9bed700 1 -- 172.21.15.102:0/4043138897 --> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97a4104bc0 con 0x7f978c06d4f0 2024-03-26T18:55:28.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.975+0000 7f97a0ff9700 1 -- 172.21.15.102:0/4043138897 <== mgr.24423 v2:172.21.15.190:6828/2858265260 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7f97a4104bc0 con 0x7f978c06d4f0 2024-03-26T18:55:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.978+0000 7f97a9bed700 1 -- 172.21.15.102:0/4043138897 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f978c06d4f0 msgr2=0x7f978c06f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.978+0000 7f97a9bed700 1 --2- 172.21.15.102:0/4043138897 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f978c06d4f0 0x7f978c06f9a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97a40ff230 tx=0x7f9798005940 comp rx=0 tx=0).stop 2024-03-26T18:55:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.978+0000 7f97a9bed700 1 -- 172.21.15.102:0/4043138897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a40ff260 msgr2=0x7f97a40fecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.978+0000 7f97a9bed700 1 --2- 172.21.15.102:0/4043138897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a40ff260 0x7f97a40fecb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f979400e9a0 tx=0x7f979400ecb0 comp rx=0 tx=0).stop 2024-03-26T18:55:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.978+0000 7f97a9bed700 1 -- 172.21.15.102:0/4043138897 shutdown_connections 2024-03-26T18:55:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.978+0000 7f97a9bed700 1 --2- 172.21.15.102:0/4043138897 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f978c06d4f0 0x7f978c06f9a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.978+0000 7f97a9bed700 1 --2- 172.21.15.102:0/4043138897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a40ff260 0x7f97a40fecb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.978+0000 7f97a9bed700 1 --2- 172.21.15.102:0/4043138897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97a4100600 0x7f97a419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.978+0000 7f97a9bed700 1 -- 172.21.15.102:0/4043138897 >> 172.21.15.102:0/4043138897 conn(0x7f97a40f9d80 msgr2=0x7f97a41034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.978+0000 7f97a9bed700 1 -- 172.21.15.102:0/4043138897 shutdown_connections 2024-03-26T18:55:28.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:28.978+0000 7f97a9bed700 1 -- 172.21.15.102:0/4043138897 wait complete. 2024-03-26T18:55:28.983 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:55:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.145+0000 7f403cb97700 1 -- 172.21.15.102:0/3986804365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40380ff810 msgr2=0x7f40380ffc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:29.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.145+0000 7f403cb97700 1 --2- 172.21.15.102:0/3986804365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40380ff810 0x7f40380ffc60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4028009910 tx=0x7f4028009c20 comp rx=0 tx=0).stop 2024-03-26T18:55:29.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.145+0000 7f403cb97700 1 -- 172.21.15.102:0/3986804365 shutdown_connections 2024-03-26T18:55:29.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.145+0000 7f403cb97700 1 --2- 172.21.15.102:0/3986804365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40380ff810 0x7f40380ffc60 secure :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4028009910 tx=0x7f4028009c20 comp rx=0 tx=0).stop 2024-03-26T18:55:29.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.145+0000 7f403cb97700 1 --2- 172.21.15.102:0/3986804365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40380fe610 0x7f40380fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:29.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.145+0000 7f403cb97700 1 -- 172.21.15.102:0/3986804365 >> 172.21.15.102:0/3986804365 conn(0x7f40380f9bc0 msgr2=0x7f40380fbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:29.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.147+0000 7f403cb97700 1 -- 172.21.15.102:0/3986804365 shutdown_connections 2024-03-26T18:55:29.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.148+0000 7f403cb97700 1 -- 172.21.15.102:0/3986804365 wait complete. 2024-03-26T18:55:29.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.149+0000 7f403cb97700 1 Processor -- start 2024-03-26T18:55:29.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.149+0000 7f403cb97700 1 -- start start 2024-03-26T18:55:29.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.149+0000 7f403cb97700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40380fe610 0x7f4038072960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:29.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.149+0000 7f403cb97700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4038109240 0x7f403810b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:29.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.149+0000 7f403cb97700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403810bc00 con 0x7f4038109240 2024-03-26T18:55:29.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.149+0000 7f403cb97700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403810bd70 con 0x7f40380fe610 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.149+0000 7f402ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4038109240 0x7f403810b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.149+0000 7f402ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4038109240 0x7f403810b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48194/0 (socket says 172.21.15.102:48194) 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.149+0000 7f402ffff700 1 -- 172.21.15.102:0/1025582118 learned_addr learned my addr 172.21.15.102:0/1025582118 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.149+0000 7f40377fe700 1 --2- 172.21.15.102:0/1025582118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40380fe610 0x7f4038072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.150+0000 7f402ffff700 1 -- 172.21.15.102:0/1025582118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40380fe610 msgr2=0x7f4038072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.150+0000 7f402ffff700 1 --2- 172.21.15.102:0/1025582118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40380fe610 0x7f4038072960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.150+0000 7f402ffff700 1 -- 172.21.15.102:0/1025582118 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40280095f0 con 0x7f4038109240 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.150+0000 7f402ffff700 1 --2- 172.21.15.102:0/1025582118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4038109240 0x7f403810b6c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4028009fd0 tx=0x7f402800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.151+0000 7f40357fa700 1 -- 172.21.15.102:0/1025582118 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f402801d070 con 0x7f4038109240 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.151+0000 7f403cb97700 1 -- 172.21.15.102:0/1025582118 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403810bf90 con 0x7f4038109240 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.151+0000 7f403cb97700 1 -- 172.21.15.102:0/1025582118 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403810c480 con 0x7f4038109240 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.151+0000 7f40357fa700 1 -- 172.21.15.102:0/1025582118 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f402800f460 con 0x7f4038109240 2024-03-26T18:55:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.151+0000 7f40357fa700 1 -- 172.21.15.102:0/1025582118 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4028021640 con 0x7f4038109240 2024-03-26T18:55:29.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.152+0000 7f403cb97700 1 -- 172.21.15.102:0/1025582118 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403806bf10 con 0x7f4038109240 2024-03-26T18:55:29.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.153+0000 7f40357fa700 1 -- 172.21.15.102:0/1025582118 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 25) v1 ==== 91096+0+0 (secure 0 0 0) 0x7f402802b440 con 0x7f4038109240 2024-03-26T18:55:29.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.154+0000 7f40357fa700 1 --2- 172.21.15.102:0/1025582118 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f402406d530 0x7f402406f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:29.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.154+0000 7f40357fa700 1 -- 172.21.15.102:0/1025582118 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f4028003a60 con 0x7f4038109240 2024-03-26T18:55:29.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.154+0000 7f40377fe700 1 --2- 172.21.15.102:0/1025582118 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f402406d530 0x7f402406f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:29.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.155+0000 7f40377fe700 1 --2- 172.21.15.102:0/1025582118 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f402406d530 0x7f402406f9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f40380ff670 tx=0x7f402000a380 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:29.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.164+0000 7f40357fa700 1 -- 172.21.15.102:0/1025582118 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f402805c130 con 0x7f4038109240 2024-03-26T18:55:29.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:28 smithi190 ceph-mon[35043]: pgmap v23: 65 pgs: 65 active+clean; 317 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 32 KiB/s rd, 1.4 MiB/s wr, 341 op/s 2024-03-26T18:55:29.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:28 smithi190 ceph-mon[35043]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:55:29.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.486+0000 7f403cb97700 1 -- 172.21.15.102:0/1025582118 --> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4038061190 con 0x7f402406d530 2024-03-26T18:55:29.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.488+0000 7f40357fa700 1 -- 172.21.15.102:0/1025582118 <== mgr.24423 v2:172.21.15.190:6828/2858265260 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+328 (secure 0 0 0) 0x7f4038061190 con 0x7f402406d530 2024-03-26T18:55:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.490+0000 7f403cb97700 1 -- 172.21.15.102:0/1025582118 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f402406d530 msgr2=0x7f402406f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.490+0000 7f403cb97700 1 --2- 172.21.15.102:0/1025582118 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f402406d530 0x7f402406f9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f40380ff670 tx=0x7f402000a380 comp rx=0 tx=0).stop 2024-03-26T18:55:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.490+0000 7f403cb97700 1 -- 172.21.15.102:0/1025582118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4038109240 msgr2=0x7f403810b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.490+0000 7f403cb97700 1 --2- 172.21.15.102:0/1025582118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4038109240 0x7f403810b6c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4028009fd0 tx=0x7f402800bd70 comp rx=0 tx=0).stop 2024-03-26T18:55:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.490+0000 7f403cb97700 1 -- 172.21.15.102:0/1025582118 shutdown_connections 2024-03-26T18:55:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.490+0000 7f403cb97700 1 --2- 172.21.15.102:0/1025582118 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f402406d530 0x7f402406f9e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.490+0000 7f403cb97700 1 --2- 172.21.15.102:0/1025582118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40380fe610 0x7f4038072960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.490+0000 7f403cb97700 1 --2- 172.21.15.102:0/1025582118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4038109240 0x7f403810b6c0 secure :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4028009fd0 tx=0x7f402800bd70 comp rx=0 tx=0).stop 2024-03-26T18:55:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.490+0000 7f403cb97700 1 -- 172.21.15.102:0/1025582118 >> 172.21.15.102:0/1025582118 conn(0x7f40380f9bc0 msgr2=0x7f40380fbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.490+0000 7f403cb97700 1 -- 172.21.15.102:0/1025582118 shutdown_connections 2024-03-26T18:55:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.490+0000 7f403cb97700 1 -- 172.21.15.102:0/1025582118 wait complete. 2024-03-26T18:55:29.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.636+0000 7f0f640c0700 1 -- 172.21.15.102:0/3998009623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f5c103500 msgr2=0x7f0f5c1058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:29.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.636+0000 7f0f640c0700 1 --2- 172.21.15.102:0/3998009623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f5c103500 0x7f0f5c1058e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009910 tx=0x7f0f4c009c20 comp rx=0 tx=0).stop 2024-03-26T18:55:29.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.636+0000 7f0f640c0700 1 -- 172.21.15.102:0/3998009623 shutdown_connections 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.636+0000 7f0f640c0700 1 --2- 172.21.15.102:0/3998009623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f5c103500 0x7f0f5c1058e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.636+0000 7f0f640c0700 1 --2- 172.21.15.102:0/3998009623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f5c100be0 0x7f0f5c102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.636+0000 7f0f640c0700 1 -- 172.21.15.102:0/3998009623 >> 172.21.15.102:0/3998009623 conn(0x7f0f5c0fa5e0 msgr2=0x7f0f5c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.636+0000 7f0f640c0700 1 -- 172.21.15.102:0/3998009623 shutdown_connections 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.636+0000 7f0f640c0700 1 -- 172.21.15.102:0/3998009623 wait complete. 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.637+0000 7f0f640c0700 1 Processor -- start 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.637+0000 7f0f640c0700 1 -- start start 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.637+0000 7f0f640c0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f5c100be0 0x7f0f5c195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.637+0000 7f0f61e5c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f5c100be0 0x7f0f5c195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.637+0000 7f0f61e5c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f5c100be0 0x7f0f5c195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48200/0 (socket says 172.21.15.102:48200) 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.637+0000 7f0f640c0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f5c103500 0x7f0f5c195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.637+0000 7f0f61e5c700 1 -- 172.21.15.102:0/1701078487 learned_addr learned my addr 172.21.15.102:0/1701078487 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.637+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c196490 con 0x7f0f5c100be0 2024-03-26T18:55:29.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.637+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1965d0 con 0x7f0f5c103500 2024-03-26T18:55:29.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.638+0000 7f0f6165b700 1 --2- 172.21.15.102:0/1701078487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f5c103500 0x7f0f5c195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:29.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.638+0000 7f0f61e5c700 1 -- 172.21.15.102:0/1701078487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f5c103500 msgr2=0x7f0f5c195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:29.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.638+0000 7f0f61e5c700 1 --2- 172.21.15.102:0/1701078487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f5c103500 0x7f0f5c195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:29.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.638+0000 7f0f61e5c700 1 -- 172.21.15.102:0/1701078487 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c0095f0 con 0x7f0f5c100be0 2024-03-26T18:55:29.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.638+0000 7f0f61e5c700 1 --2- 172.21.15.102:0/1701078487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f5c100be0 0x7f0f5c195930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0f5800e920 tx=0x7f0f5800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:29.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.638+0000 7f0f52ffd700 1 -- 172.21.15.102:0/1701078487 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f5800c9d0 con 0x7f0f5c100be0 2024-03-26T18:55:29.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.638+0000 7f0f52ffd700 1 -- 172.21.15.102:0/1701078487 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0f58010430 con 0x7f0f5c100be0 2024-03-26T18:55:29.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.638+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c19b080 con 0x7f0f5c100be0 2024-03-26T18:55:29.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.638+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c19b510 con 0x7f0f5c100be0 2024-03-26T18:55:29.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.639+0000 7f0f52ffd700 1 -- 172.21.15.102:0/1701078487 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f58018650 con 0x7f0f5c100be0 2024-03-26T18:55:29.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.640+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5c18fb70 con 0x7f0f5c100be0 2024-03-26T18:55:29.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.641+0000 7f0f52ffd700 1 -- 172.21.15.102:0/1701078487 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 25) v1 ==== 91096+0+0 (secure 0 0 0) 0x7f0f580105a0 con 0x7f0f5c100be0 2024-03-26T18:55:29.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.641+0000 7f0f52ffd700 1 --2- 172.21.15.102:0/1701078487 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f0f4806d4f0 0x7f0f4806f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:29.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.642+0000 7f0f6165b700 1 --2- 172.21.15.102:0/1701078487 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f0f4806d4f0 0x7f0f4806f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:29.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.642+0000 7f0f6165b700 1 --2- 172.21.15.102:0/1701078487 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f0f4806d4f0 0x7f0f4806f9a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009fd0 tx=0x7f0f4c005940 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:29.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.642+0000 7f0f52ffd700 1 -- 172.21.15.102:0/1701078487 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f0f58014070 con 0x7f0f5c100be0 2024-03-26T18:55:29.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:29.650+0000 7f0f52ffd700 1 -- 172.21.15.102:0/1701078487 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f0f58053470 con 0x7f0f5c100be0 2024-03-26T18:55:30.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.011+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 --> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f5c061190 con 0x7f0f4806d4f0 2024-03-26T18:55:30.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.022+0000 7f0f52ffd700 1 -- 172.21.15.102:0/1701078487 <== mgr.24423 v2:172.21.15.190:6828/2858265260 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0f5c061190 con 0x7f0f4806d4f0 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (7m) 29s ago 8m 25.0M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (8m) 29s ago 8m 7638k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (7m) 31s ago 7m 7537k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (8m) 29s ago 8m 6665k - 18.2.0 dc2bc1663786 bfac00217c68 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (7m) 31s ago 7m 6656k - 18.2.0 dc2bc1663786 7379f1f6b746 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (6m) 29s ago 8m 86.0M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (3m) 29s ago 2m 164M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (3m) 29s ago 3m 13.0M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (2m) 31s ago 2m 13.7M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (3m) 31s ago 3m 186M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 starting - - - - 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (57s) 31s ago 7m 491M - 19.0.0-2311-g36051606 497bb87bc5da 888321bf02fa 2024-03-26T18:55:30.016 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (10m) 29s ago 10m 48.9M 2048M 18.2.0 dc2bc1663786 b5abd46c3973 2024-03-26T18:55:30.017 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (7m) 31s ago 7m 39.6M 2048M 18.2.0 dc2bc1663786 6bd913d483f0 2024-03-26T18:55:30.017 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (8m) 29s ago 8m 15.8M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:55:30.017 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (7m) 31s ago 7m 15.4M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:55:30.017 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (6m) 29s ago 6m 215M 1216M 18.2.0 dc2bc1663786 3e2065fb5fbc 2024-03-26T18:55:30.017 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (6m) 29s ago 6m 230M 1216M 18.2.0 dc2bc1663786 0bf773515cc1 2024-03-26T18:55:30.017 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (5m) 29s ago 5m 179M 1216M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:55:30.017 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (5m) 31s ago 5m 257M 2240M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:55:30.017 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (4m) 31s ago 4m 229M 2240M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:55:30.017 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (4m) 31s ago 4m 244M 2240M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:55:30.017 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 starting - - - - 2024-03-26T18:55:30.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.024+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f0f4806d4f0 msgr2=0x7f0f4806f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:30.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.024+0000 7f0f640c0700 1 --2- 172.21.15.102:0/1701078487 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f0f4806d4f0 0x7f0f4806f9a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009fd0 tx=0x7f0f4c005940 comp rx=0 tx=0).stop 2024-03-26T18:55:30.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.024+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f5c100be0 msgr2=0x7f0f5c195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:30.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.024+0000 7f0f640c0700 1 --2- 172.21.15.102:0/1701078487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f5c100be0 0x7f0f5c195930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0f5800e920 tx=0x7f0f5800ec30 comp rx=0 tx=0).stop 2024-03-26T18:55:30.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.025+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 shutdown_connections 2024-03-26T18:55:30.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.025+0000 7f0f640c0700 1 --2- 172.21.15.102:0/1701078487 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f0f4806d4f0 0x7f0f4806f9a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.025+0000 7f0f640c0700 1 --2- 172.21.15.102:0/1701078487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f5c100be0 0x7f0f5c195930 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.025+0000 7f0f640c0700 1 --2- 172.21.15.102:0/1701078487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f5c103500 0x7f0f5c195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.025+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 >> 172.21.15.102:0/1701078487 conn(0x7f0f5c0fa5e0 msgr2=0x7f0f5c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:30.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.025+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 shutdown_connections 2024-03-26T18:55:30.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.025+0000 7f0f640c0700 1 -- 172.21.15.102:0/1701078487 wait complete. 2024-03-26T18:55:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.149+0000 7fd9fbd97700 1 -- 172.21.15.102:0/1247735707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f4100be0 msgr2=0x7fd9f4102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.149+0000 7fd9fbd97700 1 --2- 172.21.15.102:0/1247735707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f4100be0 0x7fd9f4102fc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009910 tx=0x7fd9f0009c20 comp rx=0 tx=0).stop 2024-03-26T18:55:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.149+0000 7fd9fbd97700 1 -- 172.21.15.102:0/1247735707 shutdown_connections 2024-03-26T18:55:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.149+0000 7fd9fbd97700 1 --2- 172.21.15.102:0/1247735707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9f4103500 0x7fd9f41058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.149+0000 7fd9fbd97700 1 --2- 172.21.15.102:0/1247735707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f4100be0 0x7fd9f4102fc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.149+0000 7fd9fbd97700 1 -- 172.21.15.102:0/1247735707 >> 172.21.15.102:0/1247735707 conn(0x7fd9f40fa5e0 msgr2=0x7fd9f40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.149+0000 7fd9fbd97700 1 -- 172.21.15.102:0/1247735707 shutdown_connections 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.149+0000 7fd9fbd97700 1 -- 172.21.15.102:0/1247735707 wait complete. 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.150+0000 7fd9fbd97700 1 Processor -- start 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.150+0000 7fd9fbd97700 1 -- start start 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.150+0000 7fd9fbd97700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f4100be0 0x7fd9f4195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.150+0000 7fd9fbd97700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9f4103500 0x7fd9f4195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.150+0000 7fd9fbd97700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f4196480 con 0x7fd9f4100be0 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.150+0000 7fd9fbd97700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f41965c0 con 0x7fd9f4103500 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9f9b33700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f4100be0 0x7fd9f4195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9f9b33700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f4100be0 0x7fd9f4195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48218/0 (socket says 172.21.15.102:48218) 2024-03-26T18:55:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9f9b33700 1 -- 172.21.15.102:0/314342772 learned_addr learned my addr 172.21.15.102:0/314342772 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9f9332700 1 --2- 172.21.15.102:0/314342772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9f4103500 0x7fd9f4195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9f9b33700 1 -- 172.21.15.102:0/314342772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9f4103500 msgr2=0x7fd9f4195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9f9b33700 1 --2- 172.21.15.102:0/314342772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9f4103500 0x7fd9f4195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9f9b33700 1 -- 172.21.15.102:0/314342772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f00095f0 con 0x7fd9f4100be0 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9f9b33700 1 --2- 172.21.15.102:0/314342772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f4100be0 0x7fd9f4195920 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0000c00 tx=0x7fd9f000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9eaffd700 1 -- 172.21.15.102:0/314342772 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9f001d070 con 0x7fd9f4100be0 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9fbd97700 1 -- 172.21.15.102:0/314342772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f419b010 con 0x7fd9f4100be0 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9eaffd700 1 -- 172.21.15.102:0/314342772 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd9f0005dc0 con 0x7fd9f4100be0 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.151+0000 7fd9eaffd700 1 -- 172.21.15.102:0/314342772 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9f00226a0 con 0x7fd9f4100be0 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.152+0000 7fd9fbd97700 1 -- 172.21.15.102:0/314342772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f419b440 con 0x7fd9f4100be0 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.153+0000 7fd9fbd97700 1 -- 172.21.15.102:0/314342772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f418fb70 con 0x7fd9f4100be0 2024-03-26T18:55:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.154+0000 7fd9eaffd700 1 -- 172.21.15.102:0/314342772 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 25) v1 ==== 91096+0+0 (secure 0 0 0) 0x7fd9f00038f0 con 0x7fd9f4100be0 2024-03-26T18:55:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.154+0000 7fd9eaffd700 1 --2- 172.21.15.102:0/314342772 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd9e006d4f0 0x7fd9e006f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.154+0000 7fd9eaffd700 1 -- 172.21.15.102:0/314342772 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fd9f008e230 con 0x7fd9f4100be0 2024-03-26T18:55:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.154+0000 7fd9f9332700 1 --2- 172.21.15.102:0/314342772 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd9e006d4f0 0x7fd9e006f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.155+0000 7fd9f9332700 1 --2- 172.21.15.102:0/314342772 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd9e006d4f0 0x7fd9e006f9a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009c60 tx=0x7fd9e4009260 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.161+0000 7fd9eaffd700 1 -- 172.21.15.102:0/314342772 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7fd9f005b3f0 con 0x7fd9f4100be0 2024-03-26T18:55:30.354 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:30 smithi102 ceph-mon[29002]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:55:30.354 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:30 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:30.354 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:30 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:30.354 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:30 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:55:30.354 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:30 smithi102 ceph-mon[29002]: from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:55:30.355 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:30 smithi102 ceph-mon[29002]: pgmap v24: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 57 KiB/s rd, 2.6 MiB/s wr, 580 op/s 2024-03-26T18:55:30.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:30 smithi190 ceph-mon[35043]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:55:30.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:30 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:30.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:30 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:30.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:30 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:55:30.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:30 smithi190 ceph-mon[35043]: from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:55:30.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:30 smithi190 ceph-mon[35043]: pgmap v24: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 57 KiB/s rd, 2.6 MiB/s wr, 580 op/s 2024-03-26T18:55:30.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.598+0000 7fd9fbd97700 1 -- 172.21.15.102:0/314342772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd9f402cea0 con 0x7fd9f4100be0 2024-03-26T18:55:30.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.599+0000 7fd9eaffd700 1 -- 172.21.15.102:0/314342772 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7fd9f0003680 con 0x7fd9f4100be0 2024-03-26T18:55:30.592 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:55:30.592 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 1 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:55:30.593 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 13, 2024-03-26T18:55:30.594 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 1 2024-03-26T18:55:30.594 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:55:30.594 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:55:30.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.602+0000 7fd9fbd97700 1 -- 172.21.15.102:0/314342772 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd9e006d4f0 msgr2=0x7fd9e006f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:30.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.602+0000 7fd9fbd97700 1 --2- 172.21.15.102:0/314342772 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd9e006d4f0 0x7fd9e006f9a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009c60 tx=0x7fd9e4009260 comp rx=0 tx=0).stop 2024-03-26T18:55:30.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.602+0000 7fd9fbd97700 1 -- 172.21.15.102:0/314342772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f4100be0 msgr2=0x7fd9f4195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:30.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.602+0000 7fd9fbd97700 1 --2- 172.21.15.102:0/314342772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f4100be0 0x7fd9f4195920 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0000c00 tx=0x7fd9f000f690 comp rx=0 tx=0).stop 2024-03-26T18:55:30.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.602+0000 7fd9fbd97700 1 -- 172.21.15.102:0/314342772 shutdown_connections 2024-03-26T18:55:30.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.602+0000 7fd9fbd97700 1 --2- 172.21.15.102:0/314342772 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd9e006d4f0 0x7fd9e006f9a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.602+0000 7fd9fbd97700 1 --2- 172.21.15.102:0/314342772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f4100be0 0x7fd9f4195920 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.602+0000 7fd9fbd97700 1 --2- 172.21.15.102:0/314342772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9f4103500 0x7fd9f4195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.602+0000 7fd9fbd97700 1 -- 172.21.15.102:0/314342772 >> 172.21.15.102:0/314342772 conn(0x7fd9f40fa5e0 msgr2=0x7fd9f40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:30.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.603+0000 7fd9fbd97700 1 -- 172.21.15.102:0/314342772 shutdown_connections 2024-03-26T18:55:30.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.603+0000 7fd9fbd97700 1 -- 172.21.15.102:0/314342772 wait complete. 2024-03-26T18:55:30.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.727+0000 7fd8cde59700 1 -- 172.21.15.102:0/3498880445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8c81013b0 msgr2=0x7fd8c8101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:30.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.727+0000 7fd8cde59700 1 --2- 172.21.15.102:0/3498880445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8c81013b0 0x7fd8c8101800 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009910 tx=0x7fd8bc009c20 comp rx=0 tx=0).stop 2024-03-26T18:55:30.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.727+0000 7fd8cde59700 1 -- 172.21.15.102:0/3498880445 shutdown_connections 2024-03-26T18:55:30.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.727+0000 7fd8cde59700 1 --2- 172.21.15.102:0/3498880445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8c81013b0 0x7fd8c8101800 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.727+0000 7fd8cde59700 1 --2- 172.21.15.102:0/3498880445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c81001b0 0x7fd8c81005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.727+0000 7fd8cde59700 1 -- 172.21.15.102:0/3498880445 >> 172.21.15.102:0/3498880445 conn(0x7fd8c80fb780 msgr2=0x7fd8c80fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:30.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.727+0000 7fd8cde59700 1 -- 172.21.15.102:0/3498880445 shutdown_connections 2024-03-26T18:55:30.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.727+0000 7fd8cde59700 1 -- 172.21.15.102:0/3498880445 wait complete. 2024-03-26T18:55:30.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.728+0000 7fd8cde59700 1 Processor -- start 2024-03-26T18:55:30.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.728+0000 7fd8cde59700 1 -- start start 2024-03-26T18:55:30.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.728+0000 7fd8cde59700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8c81001b0 0x7fd8c8074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.728+0000 7fd8cde59700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c81013b0 0x7fd8c8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.728+0000 7fd8cde59700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c8073680 con 0x7fd8c81001b0 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.728+0000 7fd8cde59700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c80737c0 con 0x7fd8c81013b0 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.728+0000 7fd8c6ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c81013b0 0x7fd8c8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.728+0000 7fd8c6ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c81013b0 0x7fd8c8073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.102:60080/0 (socket says 172.21.15.102:60080) 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.728+0000 7fd8c6ffd700 1 -- 172.21.15.102:0/3504792966 learned_addr learned my addr 172.21.15.102:0/3504792966 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.728+0000 7fd8c77fe700 1 --2- 172.21.15.102:0/3504792966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8c81001b0 0x7fd8c8074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.729+0000 7fd8c6ffd700 1 -- 172.21.15.102:0/3504792966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8c81001b0 msgr2=0x7fd8c8074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.729+0000 7fd8c6ffd700 1 --2- 172.21.15.102:0/3504792966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8c81001b0 0x7fd8c8074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.729+0000 7fd8c6ffd700 1 -- 172.21.15.102:0/3504792966 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc0095f0 con 0x7fd8c81013b0 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.729+0000 7fd8c6ffd700 1 --2- 172.21.15.102:0/3504792966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c81013b0 0x7fd8c8073140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc005d10 tx=0x7fd8bc00fa20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.729+0000 7fd8c4ff9700 1 -- 172.21.15.102:0/3504792966 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8bc01d070 con 0x7fd8c81013b0 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.729+0000 7fd8c4ff9700 1 -- 172.21.15.102:0/3504792966 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd8bc00fe90 con 0x7fd8c81013b0 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.729+0000 7fd8c4ff9700 1 -- 172.21.15.102:0/3504792966 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8bc022680 con 0x7fd8c81013b0 2024-03-26T18:55:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.729+0000 7fd8cde59700 1 -- 172.21.15.102:0/3504792966 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c8073a40 con 0x7fd8c81013b0 2024-03-26T18:55:30.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.729+0000 7fd8cde59700 1 -- 172.21.15.102:0/3504792966 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c8073ee0 con 0x7fd8c81013b0 2024-03-26T18:55:30.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.731+0000 7fd8cde59700 1 -- 172.21.15.102:0/3504792966 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c8108d00 con 0x7fd8c81013b0 2024-03-26T18:55:30.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.731+0000 7fd8c4ff9700 1 -- 172.21.15.102:0/3504792966 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 25) v1 ==== 91096+0+0 (secure 0 0 0) 0x7fd8bc003680 con 0x7fd8c81013b0 2024-03-26T18:55:30.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.732+0000 7fd8c4ff9700 1 --2- 172.21.15.102:0/3504792966 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd8b006d4f0 0x7fd8b006f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:30.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.732+0000 7fd8c4ff9700 1 -- 172.21.15.102:0/3504792966 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fd8bc08e4e0 con 0x7fd8c81013b0 2024-03-26T18:55:30.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.732+0000 7fd8c77fe700 1 --2- 172.21.15.102:0/3504792966 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd8b006d4f0 0x7fd8b006f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:30.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.733+0000 7fd8c77fe700 1 --2- 172.21.15.102:0/3504792966 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd8b006d4f0 0x7fd8b006f9a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8101210 tx=0x7fd8b8009260 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:30.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:30.737+0000 7fd8c4ff9700 1 -- 172.21.15.102:0/3504792966 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7fd8bc05b640 con 0x7fd8c81013b0 2024-03-26T18:55:31.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.129+0000 7fd8cde59700 1 -- 172.21.15.102:0/3504792966 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd8c819dc70 con 0x7fd8c81013b0 2024-03-26T18:55:31.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.130+0000 7fd8c4ff9700 1 -- 172.21.15.102:0/3504792966 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fd8bc05b1d0 con 0x7fd8c81013b0 2024-03-26T18:55:31.123 INFO:teuthology.orchestra.run.smithi102.stdout:e19 2024-03-26T18:55:31.123 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:55:31.123 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:55:31.123 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 19 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:55:27.824831+0000 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:55:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:55:31.125 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:55:31.126 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:55:31.126 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:55:31.126 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:55:31.126 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:55:31.126 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:55:31.126 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:55:31.126 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:55:31.126 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:55:31.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.134+0000 7fd8cde59700 1 -- 172.21.15.102:0/3504792966 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd8b006d4f0 msgr2=0x7fd8b006f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:31.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.134+0000 7fd8cde59700 1 --2- 172.21.15.102:0/3504792966 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd8b006d4f0 0x7fd8b006f9a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8101210 tx=0x7fd8b8009260 comp rx=0 tx=0).stop 2024-03-26T18:55:31.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.134+0000 7fd8cde59700 1 -- 172.21.15.102:0/3504792966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c81013b0 msgr2=0x7fd8c8073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:31.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.134+0000 7fd8cde59700 1 --2- 172.21.15.102:0/3504792966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c81013b0 0x7fd8c8073140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc005d10 tx=0x7fd8bc00fa20 comp rx=0 tx=0).stop 2024-03-26T18:55:31.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.134+0000 7fd8cde59700 1 -- 172.21.15.102:0/3504792966 shutdown_connections 2024-03-26T18:55:31.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.134+0000 7fd8cde59700 1 --2- 172.21.15.102:0/3504792966 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7fd8b006d4f0 0x7fd8b006f9a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.134+0000 7fd8cde59700 1 --2- 172.21.15.102:0/3504792966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8c81001b0 0x7fd8c8074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.134+0000 7fd8cde59700 1 --2- 172.21.15.102:0/3504792966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c81013b0 0x7fd8c8073140 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.134+0000 7fd8cde59700 1 -- 172.21.15.102:0/3504792966 >> 172.21.15.102:0/3504792966 conn(0x7fd8c80fb780 msgr2=0x7fd8c80fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:31.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.134+0000 7fd8cde59700 1 -- 172.21.15.102:0/3504792966 shutdown_connections 2024-03-26T18:55:31.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.134+0000 7fd8cde59700 1 -- 172.21.15.102:0/3504792966 wait complete. 2024-03-26T18:55:31.128 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 19 2024-03-26T18:55:31.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.307+0000 7f8b43e78700 1 -- 172.21.15.102:0/4138405830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c101350 msgr2=0x7f8b3c1017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.307+0000 7f8b43e78700 1 --2- 172.21.15.102:0/4138405830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c101350 0x7f8b3c1017a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009910 tx=0x7f8b2c009c20 comp rx=0 tx=0).stop 2024-03-26T18:55:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.307+0000 7f8b43e78700 1 -- 172.21.15.102:0/4138405830 shutdown_connections 2024-03-26T18:55:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.307+0000 7f8b43e78700 1 --2- 172.21.15.102:0/4138405830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c101350 0x7f8b3c1017a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.307+0000 7f8b43e78700 1 --2- 172.21.15.102:0/4138405830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b3c100150 0x7f8b3c100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.307+0000 7f8b43e78700 1 -- 172.21.15.102:0/4138405830 >> 172.21.15.102:0/4138405830 conn(0x7f8b3c0fb6e0 msgr2=0x7f8b3c0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.307+0000 7f8b43e78700 1 -- 172.21.15.102:0/4138405830 shutdown_connections 2024-03-26T18:55:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.307+0000 7f8b43e78700 1 -- 172.21.15.102:0/4138405830 wait complete. 2024-03-26T18:55:31.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.308+0000 7f8b43e78700 1 Processor -- start 2024-03-26T18:55:31.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.308+0000 7f8b43e78700 1 -- start start 2024-03-26T18:55:31.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.308+0000 7f8b43e78700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c100150 0x7f8b3c195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:31.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.308+0000 7f8b43e78700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b3c101350 0x7f8b3c195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:31.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.308+0000 7f8b43e78700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c1964c0 con 0x7f8b3c100150 2024-03-26T18:55:31.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.308+0000 7f8b43e78700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c196600 con 0x7f8b3c101350 2024-03-26T18:55:31.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.309+0000 7f8b41c14700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c100150 0x7f8b3c195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:31.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.309+0000 7f8b41c14700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c100150 0x7f8b3c195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48240/0 (socket says 172.21.15.102:48240) 2024-03-26T18:55:31.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.309+0000 7f8b41c14700 1 -- 172.21.15.102:0/1365337864 learned_addr learned my addr 172.21.15.102:0/1365337864 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:55:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.309+0000 7f8b41c14700 1 -- 172.21.15.102:0/1365337864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b3c101350 msgr2=0x7f8b3c195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.309+0000 7f8b41c14700 1 --2- 172.21.15.102:0/1365337864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b3c101350 0x7f8b3c195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.309+0000 7f8b41c14700 1 -- 172.21.15.102:0/1365337864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c0095f0 con 0x7f8b3c100150 2024-03-26T18:55:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.309+0000 7f8b41c14700 1 --2- 172.21.15.102:0/1365337864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c100150 0x7f8b3c195960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8b3800e8f0 tx=0x7f8b3800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.309+0000 7f8b32ffd700 1 -- 172.21.15.102:0/1365337864 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b38009790 con 0x7f8b3c100150 2024-03-26T18:55:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.309+0000 7f8b32ffd700 1 -- 172.21.15.102:0/1365337864 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8b380044d0 con 0x7f8b3c100150 2024-03-26T18:55:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.309+0000 7f8b32ffd700 1 -- 172.21.15.102:0/1365337864 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b38003da0 con 0x7f8b3c100150 2024-03-26T18:55:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.309+0000 7f8b43e78700 1 -- 172.21.15.102:0/1365337864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c19b0b0 con 0x7f8b3c100150 2024-03-26T18:55:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.310+0000 7f8b43e78700 1 -- 172.21.15.102:0/1365337864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c19b600 con 0x7f8b3c100150 2024-03-26T18:55:31.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.311+0000 7f8b43e78700 1 -- 172.21.15.102:0/1365337864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c108ca0 con 0x7f8b3c100150 2024-03-26T18:55:31.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.312+0000 7f8b32ffd700 1 -- 172.21.15.102:0/1365337864 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 25) v1 ==== 91096+0+0 (secure 0 0 0) 0x7f8b3800cba0 con 0x7f8b3c100150 2024-03-26T18:55:31.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.312+0000 7f8b32ffd700 1 --2- 172.21.15.102:0/1365337864 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f8b2806d4e0 0x7f8b2806f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:31.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.312+0000 7f8b32ffd700 1 -- 172.21.15.102:0/1365337864 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f8b38014070 con 0x7f8b3c100150 2024-03-26T18:55:31.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.313+0000 7f8b41413700 1 --2- 172.21.15.102:0/1365337864 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f8b2806d4e0 0x7f8b2806f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:31.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.313+0000 7f8b41413700 1 --2- 172.21.15.102:0/1365337864 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f8b2806d4e0 0x7f8b2806f990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009fd0 tx=0x7f8b2c000bc0 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:31.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.321+0000 7f8b32ffd700 1 -- 172.21.15.102:0/1365337864 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f8b3805a430 con 0x7f8b3c100150 2024-03-26T18:55:31.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:31 smithi190 ceph-mon[35043]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:55:31.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:31 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/314342772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:55:31.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:31 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/3504792966' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:55:31.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:31 smithi102 ceph-mon[29002]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:55:31.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:31 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/314342772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:55:31.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:31 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/3504792966' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:55:31.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.747+0000 7f8b43e78700 1 -- 172.21.15.102:0/1365337864 --> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b3c105ca0 con 0x7f8b2806d4e0 2024-03-26T18:55:31.743 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:55:31.744 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:55:31.744 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:55:31.744 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:55:31.744 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-03-26T18:55:31.744 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "", 2024-03-26T18:55:31.744 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading mgr daemons", 2024-03-26T18:55:31.744 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:55:31.744 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:55:31.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.749+0000 7f8b32ffd700 1 -- 172.21.15.102:0/1365337864 <== mgr.24423 v2:172.21.15.190:6828/2858265260 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+328 (secure 0 0 0) 0x7f8b3c105ca0 con 0x7f8b2806d4e0 2024-03-26T18:55:31.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.753+0000 7f8b43e78700 1 -- 172.21.15.102:0/1365337864 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f8b2806d4e0 msgr2=0x7f8b2806f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:31.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.753+0000 7f8b43e78700 1 --2- 172.21.15.102:0/1365337864 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f8b2806d4e0 0x7f8b2806f990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009fd0 tx=0x7f8b2c000bc0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.753+0000 7f8b43e78700 1 -- 172.21.15.102:0/1365337864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c100150 msgr2=0x7f8b3c195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:31.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.753+0000 7f8b43e78700 1 --2- 172.21.15.102:0/1365337864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c100150 0x7f8b3c195960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8b3800e8f0 tx=0x7f8b3800ecb0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.755+0000 7f8b43e78700 1 -- 172.21.15.102:0/1365337864 shutdown_connections 2024-03-26T18:55:31.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.755+0000 7f8b43e78700 1 --2- 172.21.15.102:0/1365337864 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7f8b2806d4e0 0x7f8b2806f990 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.755+0000 7f8b43e78700 1 --2- 172.21.15.102:0/1365337864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c100150 0x7f8b3c195960 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.755+0000 7f8b43e78700 1 --2- 172.21.15.102:0/1365337864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b3c101350 0x7f8b3c195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.755+0000 7f8b43e78700 1 -- 172.21.15.102:0/1365337864 >> 172.21.15.102:0/1365337864 conn(0x7f8b3c0fb6e0 msgr2=0x7f8b3c104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:31.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.755+0000 7f8b43e78700 1 -- 172.21.15.102:0/1365337864 shutdown_connections 2024-03-26T18:55:31.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.755+0000 7f8b43e78700 1 -- 172.21.15.102:0/1365337864 wait complete. 2024-03-26T18:55:31.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.939+0000 7fecbaba9700 1 -- 172.21.15.102:0/1286683783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb4100250 msgr2=0x7fecb41006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:31.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.939+0000 7fecbaba9700 1 --2- 172.21.15.102:0/1286683783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb4100250 0x7fecb41006a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feca0009870 tx=0x7feca0009b80 comp rx=0 tx=0).stop 2024-03-26T18:55:31.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.939+0000 7fecbaba9700 1 -- 172.21.15.102:0/1286683783 shutdown_connections 2024-03-26T18:55:31.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.939+0000 7fecbaba9700 1 --2- 172.21.15.102:0/1286683783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb4100250 0x7fecb41006a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.939+0000 7fecbaba9700 1 --2- 172.21.15.102:0/1286683783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecb40ff050 0x7fecb40ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.939+0000 7fecbaba9700 1 -- 172.21.15.102:0/1286683783 >> 172.21.15.102:0/1286683783 conn(0x7fecb40fa5e0 msgr2=0x7fecb40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:31.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.939+0000 7fecbaba9700 1 -- 172.21.15.102:0/1286683783 shutdown_connections 2024-03-26T18:55:31.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.939+0000 7fecbaba9700 1 -- 172.21.15.102:0/1286683783 wait complete. 2024-03-26T18:55:31.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.940+0000 7fecbaba9700 1 Processor -- start 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.940+0000 7fecbaba9700 1 -- start start 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.940+0000 7fecbaba9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecb40ff050 0x7fecb4193750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.940+0000 7fecbaba9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb4100250 0x7fecb4193c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.940+0000 7fecbaba9700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb41942b0 con 0x7fecb40ff050 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.940+0000 7fecbaba9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb41943f0 con 0x7fecb4100250 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.940+0000 7fecb8945700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecb40ff050 0x7fecb4193750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.940+0000 7fecb8945700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecb40ff050 0x7fecb4193750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48254/0 (socket says 172.21.15.102:48254) 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.940+0000 7fecb3fff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb4100250 0x7fecb4193c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.940+0000 7fecb8945700 1 -- 172.21.15.102:0/2373357424 learned_addr learned my addr 172.21.15.102:0/2373357424 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.941+0000 7fecb3fff700 1 -- 172.21.15.102:0/2373357424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecb40ff050 msgr2=0x7fecb4193750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.941+0000 7fecb3fff700 1 --2- 172.21.15.102:0/2373357424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecb40ff050 0x7fecb4193750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.941+0000 7fecb3fff700 1 -- 172.21.15.102:0/2373357424 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca80095f0 con 0x7fecb4100250 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.941+0000 7fecb3fff700 1 --2- 172.21.15.102:0/2373357424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb4100250 0x7fecb4193c90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feca0000c00 tx=0x7feca000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:31.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.941+0000 7fecb1ffb700 1 -- 172.21.15.102:0/2373357424 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feca001c070 con 0x7fecb4100250 2024-03-26T18:55:31.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.941+0000 7fecb1ffb700 1 -- 172.21.15.102:0/2373357424 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feca000fd60 con 0x7fecb4100250 2024-03-26T18:55:31.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.942+0000 7fecb1ffb700 1 -- 172.21.15.102:0/2373357424 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feca0022da0 con 0x7fecb4100250 2024-03-26T18:55:31.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.942+0000 7fecbaba9700 1 -- 172.21.15.102:0/2373357424 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca0009520 con 0x7fecb4100250 2024-03-26T18:55:31.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.942+0000 7fecbaba9700 1 -- 172.21.15.102:0/2373357424 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb4199220 con 0x7fecb4100250 2024-03-26T18:55:31.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.944+0000 7fecbaba9700 1 -- 172.21.15.102:0/2373357424 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb4106a90 con 0x7fecb4100250 2024-03-26T18:55:31.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.944+0000 7fecb1ffb700 1 -- 172.21.15.102:0/2373357424 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 25) v1 ==== 91096+0+0 (secure 0 0 0) 0x7feca0020070 con 0x7fecb4100250 2024-03-26T18:55:31.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.944+0000 7fecb1ffb700 1 --2- 172.21.15.102:0/2373357424 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7feca406f6e0 0x7feca4071b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:55:31.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.944+0000 7fecb1ffb700 1 -- 172.21.15.102:0/2373357424 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7feca008dc10 con 0x7fecb4100250 2024-03-26T18:55:31.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.945+0000 7fecb8945700 1 --2- 172.21.15.102:0/2373357424 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7feca406f6e0 0x7feca4071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:55:31.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.945+0000 7fecb8945700 1 --2- 172.21.15.102:0/2373357424 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7feca406f6e0 0x7feca4071b90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feca8005950 tx=0x7feca8009310 comp rx=0 tx=0).ready entity=mgr.24423 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:55:31.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:31.953+0000 7fecb1ffb700 1 -- 172.21.15.102:0/2373357424 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7feca005ae80 con 0x7fecb4100250 2024-03-26T18:55:32.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.395+0000 7fecbaba9700 1 -- 172.21.15.102:0/2373357424 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fecb4106c80 con 0x7fecb4100250 2024-03-26T18:55:32.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.395+0000 7fecb1ffb700 1 -- 172.21.15.102:0/2373357424 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7feca0027090 con 0x7fecb4100250 2024-03-26T18:55:32.389 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T18:55:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.399+0000 7fecbaba9700 1 -- 172.21.15.102:0/2373357424 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7feca406f6e0 msgr2=0x7feca4071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.399+0000 7fecbaba9700 1 --2- 172.21.15.102:0/2373357424 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7feca406f6e0 0x7feca4071b90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feca8005950 tx=0x7feca8009310 comp rx=0 tx=0).stop 2024-03-26T18:55:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.399+0000 7fecbaba9700 1 -- 172.21.15.102:0/2373357424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb4100250 msgr2=0x7fecb4193c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:55:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.399+0000 7fecbaba9700 1 --2- 172.21.15.102:0/2373357424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb4100250 0x7fecb4193c90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feca0000c00 tx=0x7feca000f690 comp rx=0 tx=0).stop 2024-03-26T18:55:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.399+0000 7fecbaba9700 1 -- 172.21.15.102:0/2373357424 shutdown_connections 2024-03-26T18:55:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.399+0000 7fecbaba9700 1 --2- 172.21.15.102:0/2373357424 >> [v2:172.21.15.190:6828/2858265260,v1:172.21.15.190:6829/2858265260] conn(0x7feca406f6e0 0x7feca4071b90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.399+0000 7fecbaba9700 1 --2- 172.21.15.102:0/2373357424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecb40ff050 0x7fecb4193750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.399+0000 7fecbaba9700 1 --2- 172.21.15.102:0/2373357424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb4100250 0x7fecb4193c90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:55:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.399+0000 7fecbaba9700 1 -- 172.21.15.102:0/2373357424 >> 172.21.15.102:0/2373357424 conn(0x7fecb40fa5e0 msgr2=0x7fecb4103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:55:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.399+0000 7fecbaba9700 1 -- 172.21.15.102:0/2373357424 shutdown_connections 2024-03-26T18:55:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:55:32.399+0000 7fecbaba9700 1 -- 172.21.15.102:0/2373357424 wait complete. 2024-03-26T18:55:32.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:32 smithi190 ceph-mon[35043]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:55:32.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:32 smithi190 ceph-mon[35043]: pgmap v25: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 42 KiB/s rd, 2.0 MiB/s wr, 447 op/s 2024-03-26T18:55:32.561 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:32 smithi102 ceph-mon[29002]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:55:32.561 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:32 smithi102 ceph-mon[29002]: pgmap v25: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 42 KiB/s rd, 2.0 MiB/s wr, 447 op/s 2024-03-26T18:55:33.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:33 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2373357424' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:55:33.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:33 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:55:33.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:33 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/2373357424' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:55:33.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:33 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:55:34.272 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:34 smithi102 ceph-mon[29002]: pgmap v26: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67 KiB/s rd, 2.4 MiB/s wr, 586 op/s 2024-03-26T18:55:34.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:34 smithi190 ceph-mon[35043]: pgmap v26: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67 KiB/s rd, 2.4 MiB/s wr, 586 op/s 2024-03-26T18:55:36.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:36 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:36.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:36 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:36.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:36 smithi102 ceph-mon[29002]: pgmap v27: 65 pgs: 65 active+clean; 304 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 49 KiB/s rd, 2.9 MiB/s wr, 589 op/s 2024-03-26T18:55:36.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:36 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:36.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:36 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:36.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:36 smithi190 ceph-mon[35043]: pgmap v27: 65 pgs: 65 active+clean; 304 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 49 KiB/s rd, 2.9 MiB/s wr, 589 op/s 2024-03-26T18:55:37.836 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:37 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:37.836 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:37 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:38.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:37 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:38.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:37 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:39.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:38 smithi190 ceph-mon[35043]: pgmap v28: 65 pgs: 65 active+clean; 304 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 49 KiB/s rd, 2.3 MiB/s wr, 454 op/s 2024-03-26T18:55:39.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:38 smithi190 ceph-mon[35043]: Standby manager daemon smithi102.rsfpnv restarted 2024-03-26T18:55:39.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:38 smithi190 ceph-mon[35043]: Standby manager daemon smithi102.rsfpnv started 2024-03-26T18:55:39.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:38 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.rsfpnv/crt"}]: dispatch 2024-03-26T18:55:39.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:38 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:55:39.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:38 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.rsfpnv/key"}]: dispatch 2024-03-26T18:55:39.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:38 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:55:39.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:38 smithi102 ceph-mon[29002]: pgmap v28: 65 pgs: 65 active+clean; 304 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 49 KiB/s rd, 2.3 MiB/s wr, 454 op/s 2024-03-26T18:55:39.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:38 smithi102 ceph-mon[29002]: Standby manager daemon smithi102.rsfpnv restarted 2024-03-26T18:55:39.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:38 smithi102 ceph-mon[29002]: Standby manager daemon smithi102.rsfpnv started 2024-03-26T18:55:39.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:38 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.rsfpnv/crt"}]: dispatch 2024-03-26T18:55:39.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:38 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:55:39.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:38 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.rsfpnv/key"}]: dispatch 2024-03-26T18:55:39.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:38 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:55:40.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:39 smithi190 ceph-mon[35043]: mgrmap e26: smithi190.zrixos(active, since 51s), standbys: smithi102.rsfpnv 2024-03-26T18:55:40.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:39 smithi102 ceph-mon[29002]: mgrmap e26: smithi190.zrixos(active, since 51s), standbys: smithi102.rsfpnv 2024-03-26T18:55:40.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:40 smithi102 ceph-mon[29002]: pgmap v29: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 70 KiB/s rd, 3.3 MiB/s wr, 698 op/s 2024-03-26T18:55:41.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:40 smithi190 ceph-mon[35043]: pgmap v29: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 70 KiB/s rd, 3.3 MiB/s wr, 698 op/s 2024-03-26T18:55:43.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: pgmap v30: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 46 KiB/s rd, 2.1 MiB/s wr, 458 op/s 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mgr fail", "who": "smithi190.zrixos"}]: dispatch 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mgr fail", "who": "smithi190.zrixos"}]: dispatch 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: osdmap e49: 6 total, 6 up, 6 in 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd='[{"prefix": "mgr fail", "who": "smithi190.zrixos"}]': finished 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: mgrmap e27: smithi102.rsfpnv(active, starting, since 0.00736836s) 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:55:43.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi102.rsfpnv", "id": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T18:55:43.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T18:55:43.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: Manager daemon smithi102.rsfpnv is now available 2024-03-26T18:55:43.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:55:43.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:55:43.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:55:43.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:42 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/trash_purge_schedule"}]: dispatch 2024-03-26T18:55:43.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: pgmap v30: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 46 KiB/s rd, 2.1 MiB/s wr, 458 op/s 2024-03-26T18:55:43.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:43.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.24423 172.21.15.190:0/3157462934' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mgr fail", "who": "smithi190.zrixos"}]: dispatch 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "mgr fail", "who": "smithi190.zrixos"}]: dispatch 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: osdmap e49: 6 total, 6 up, 6 in 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.24423 ' entity='mgr.smithi190.zrixos' cmd='[{"prefix": "mgr fail", "who": "smithi190.zrixos"}]': finished 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: mgrmap e27: smithi102.rsfpnv(active, starting, since 0.00736836s) 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T18:55:43.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi102.rsfpnv", "id": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T18:55:43.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: Manager daemon smithi102.rsfpnv is now available 2024-03-26T18:55:43.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:55:43.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:55:43.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:55:43.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:42 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/trash_purge_schedule"}]: dispatch 2024-03-26T18:55:44.523 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:44 smithi190 ceph-mon[35043]: mgrmap e28: smithi102.rsfpnv(active, since 1.01096s) 2024-03-26T18:55:44.523 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:44 smithi190 ceph-mon[35043]: pgmap v3: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:55:44.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:44 smithi102 ceph-mon[29002]: mgrmap e28: smithi102.rsfpnv(active, since 1.01096s) 2024-03-26T18:55:44.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:44 smithi102 ceph-mon[29002]: pgmap v3: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:55:45.612 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:45 smithi190 ceph-mon[35043]: [26/Mar/2024:18:55:44] ENGINE Bus STARTING 2024-03-26T18:55:45.612 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:45 smithi190 ceph-mon[35043]: pgmap v4: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:55:45.612 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:45 smithi190 ceph-mon[35043]: [26/Mar/2024:18:55:44] ENGINE Serving on https://172.21.15.102:7150 2024-03-26T18:55:45.612 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:45 smithi190 ceph-mon[35043]: [26/Mar/2024:18:55:44] ENGINE Serving on http://172.21.15.102:8765 2024-03-26T18:55:45.612 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:45 smithi190 ceph-mon[35043]: [26/Mar/2024:18:55:44] ENGINE Bus STARTED 2024-03-26T18:55:45.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:45 smithi102 ceph-mon[29002]: [26/Mar/2024:18:55:44] ENGINE Bus STARTING 2024-03-26T18:55:45.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:45 smithi102 ceph-mon[29002]: pgmap v4: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:55:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:45 smithi102 ceph-mon[29002]: [26/Mar/2024:18:55:44] ENGINE Serving on https://172.21.15.102:7150 2024-03-26T18:55:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:45 smithi102 ceph-mon[29002]: [26/Mar/2024:18:55:44] ENGINE Serving on http://172.21.15.102:8765 2024-03-26T18:55:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:45 smithi102 ceph-mon[29002]: [26/Mar/2024:18:55:44] ENGINE Bus STARTED 2024-03-26T18:55:46.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:46 smithi190 ceph-mon[35043]: mgrmap e29: smithi102.rsfpnv(active, since 3s) 2024-03-26T18:55:46.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:46 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:55:46.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:46 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:55:46.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:46 smithi102 ceph-mon[29002]: mgrmap e29: smithi102.rsfpnv(active, since 3s) 2024-03-26T18:55:46.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:46 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:55:46.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:46 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:55:47.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:47 smithi190 ceph-mon[35043]: pgmap v5: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:55:47.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:47 smithi102 ceph-mon[29002]: pgmap v5: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:55:49.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:48 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:55:49.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:48 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:55:49.164 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:48 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:55:49.164 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:48 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:55:50.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:49 smithi102 ceph-mon[29002]: pgmap v6: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:55:50.279 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:49 smithi190 ceph-mon[35043]: pgmap v6: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:55:51.418 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:51 smithi190 ceph-mon[35043]: pgmap v7: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 61 KiB/s rd, 1.6 MiB/s wr, 363 op/s 2024-03-26T18:55:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:51 smithi102 ceph-mon[29002]: pgmap v7: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 61 KiB/s rd, 1.6 MiB/s wr, 363 op/s 2024-03-26T18:55:52.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:52 smithi190 ceph-mon[35043]: Standby manager daemon smithi190.zrixos started 2024-03-26T18:55:52.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:52 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3387321022' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/crt"}]: dispatch 2024-03-26T18:55:52.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:52 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3387321022' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:55:52.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:52 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3387321022' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/key"}]: dispatch 2024-03-26T18:55:52.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:52 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3387321022' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:55:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:52 smithi102 ceph-mon[29002]: Standby manager daemon smithi190.zrixos started 2024-03-26T18:55:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:52 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/3387321022' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/crt"}]: dispatch 2024-03-26T18:55:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:52 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/3387321022' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:55:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:52 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/3387321022' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/key"}]: dispatch 2024-03-26T18:55:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:52 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/3387321022' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:55:53.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:53 smithi102 ceph-mon[29002]: mgrmap e30: smithi102.rsfpnv(active, since 9s), standbys: smithi190.zrixos 2024-03-26T18:55:53.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:53 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi190.zrixos", "id": "smithi190.zrixos"}]: dispatch 2024-03-26T18:55:53.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:53 smithi102 ceph-mon[29002]: pgmap v8: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 48 KiB/s rd, 1.2 MiB/s wr, 282 op/s 2024-03-26T18:55:53.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:53 smithi190 ceph-mon[35043]: mgrmap e30: smithi102.rsfpnv(active, since 9s), standbys: smithi190.zrixos 2024-03-26T18:55:53.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:53 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi190.zrixos", "id": "smithi190.zrixos"}]: dispatch 2024-03-26T18:55:53.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:53 smithi190 ceph-mon[35043]: pgmap v8: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 48 KiB/s rd, 1.2 MiB/s wr, 282 op/s 2024-03-26T18:55:55.511 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-26T18:55:55.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-26T18:55:55.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:55 smithi190 ceph-mon[35043]: pgmap v9: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 61 KiB/s rd, 2.1 MiB/s wr, 448 op/s 2024-03-26T18:55:55.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:55 smithi102 ceph-mon[29002]: pgmap v9: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 61 KiB/s rd, 2.1 MiB/s wr, 448 op/s 2024-03-26T18:55:55.944 DEBUG:teuthology.parallel:result is None 2024-03-26T18:55:56.859 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-26T18:55:56.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-03-26T18:55:57.256 DEBUG:teuthology.parallel:result is None 2024-03-26T18:55:57.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T18:55:57.296 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T18:55:57.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-03-26T18:55:57.335 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-03-26T18:55:57.336 DEBUG:teuthology.parallel:result is None 2024-03-26T18:55:57.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:57 smithi190 ceph-mon[35043]: pgmap v10: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56 KiB/s rd, 1.9 MiB/s wr, 410 op/s 2024-03-26T18:55:57.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:57 smithi102 ceph-mon[29002]: pgmap v10: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56 KiB/s rd, 1.9 MiB/s wr, 410 op/s 2024-03-26T18:55:58.770 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:58 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:55:58.779 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:58 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:55:59.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:59 smithi190 ceph-mon[35043]: pgmap v11: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56 KiB/s rd, 1.9 MiB/s wr, 410 op/s 2024-03-26T18:55:59.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:59 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:55:59.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:55:59 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:55:59.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:59 smithi102 ceph-mon[29002]: pgmap v11: 65 pgs: 65 active+clean; 308 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56 KiB/s rd, 1.9 MiB/s wr, 410 op/s 2024-03-26T18:55:59.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:59 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:55:59.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:55:59 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:01.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:01 smithi102 ceph-mon[29002]: pgmap v12: 65 pgs: 65 active+clean; 301 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 74 KiB/s rd, 2.5 MiB/s wr, 573 op/s 2024-03-26T18:56:01.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:01 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:01.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:01 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:01.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:01 smithi190 ceph-mon[35043]: pgmap v12: 65 pgs: 65 active+clean; 301 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 74 KiB/s rd, 2.5 MiB/s wr, 573 op/s 2024-03-26T18:56:01.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:01 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:01.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:01 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.519+0000 7f0e5e1de700 1 -- 172.21.15.102:0/29943783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e580ff320 msgr2=0x7f0e580ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.519+0000 7f0e5e1de700 1 --2- 172.21.15.102:0/29943783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e580ff320 0x7f0e580ff790 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009910 tx=0x7f0e4c009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.519+0000 7f0e5e1de700 1 -- 172.21.15.102:0/29943783 shutdown_connections 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.519+0000 7f0e5e1de700 1 --2- 172.21.15.102:0/29943783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e580ff320 0x7f0e580ff790 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.519+0000 7f0e5e1de700 1 --2- 172.21.15.102:0/29943783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e580fe9d0 0x7f0e580fede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.519+0000 7f0e5e1de700 1 -- 172.21.15.102:0/29943783 >> 172.21.15.102:0/29943783 conn(0x7f0e580fa5c0 msgr2=0x7f0e580fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.519+0000 7f0e5e1de700 1 -- 172.21.15.102:0/29943783 shutdown_connections 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.519+0000 7f0e5e1de700 1 -- 172.21.15.102:0/29943783 wait complete. 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.520+0000 7f0e5e1de700 1 Processor -- start 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.520+0000 7f0e5e1de700 1 -- start start 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.520+0000 7f0e5e1de700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e580fe9d0 0x7f0e58197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.520+0000 7f0e5e1de700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e580ff320 0x7f0e58198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.520+0000 7f0e5e1de700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e581986a0 con 0x7f0e580fe9d0 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.520+0000 7f0e5e1de700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e581987e0 con 0x7f0e580ff320 2024-03-26T18:56:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.520+0000 7f0e577fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e580fe9d0 0x7f0e58197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.520+0000 7f0e577fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e580fe9d0 0x7f0e58197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41174/0 (socket says 172.21.15.102:41174) 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.520+0000 7f0e577fe700 1 -- 172.21.15.102:0/50308130 learned_addr learned my addr 172.21.15.102:0/50308130 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.520+0000 7f0e56ffd700 1 --2- 172.21.15.102:0/50308130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e580ff320 0x7f0e58198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.521+0000 7f0e577fe700 1 -- 172.21.15.102:0/50308130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e580ff320 msgr2=0x7f0e58198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.521+0000 7f0e577fe700 1 --2- 172.21.15.102:0/50308130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e580ff320 0x7f0e58198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.521+0000 7f0e577fe700 1 -- 172.21.15.102:0/50308130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c0095f0 con 0x7f0e580fe9d0 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.521+0000 7f0e577fe700 1 --2- 172.21.15.102:0/50308130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e580fe9d0 0x7f0e58197b40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800d8d0 tx=0x7f0e4800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.521+0000 7f0e54ff9700 1 -- 172.21.15.102:0/50308130 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e480098a0 con 0x7f0e580fe9d0 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.521+0000 7f0e5e1de700 1 -- 172.21.15.102:0/50308130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5819d290 con 0x7f0e580fe9d0 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.521+0000 7f0e54ff9700 1 -- 172.21.15.102:0/50308130 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0e48010460 con 0x7f0e580fe9d0 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.521+0000 7f0e54ff9700 1 -- 172.21.15.102:0/50308130 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e4800f5f0 con 0x7f0e580fe9d0 2024-03-26T18:56:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.521+0000 7f0e5e1de700 1 -- 172.21.15.102:0/50308130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5819d7e0 con 0x7f0e580fe9d0 2024-03-26T18:56:02.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.522+0000 7f0e5e1de700 1 -- 172.21.15.102:0/50308130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5810ae80 con 0x7f0e580fe9d0 2024-03-26T18:56:02.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.523+0000 7f0e54ff9700 1 -- 172.21.15.102:0/50308130 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 30) v1 ==== 92174+0+0 (secure 0 0 0) 0x7f0e48009a00 con 0x7f0e580fe9d0 2024-03-26T18:56:02.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.523+0000 7f0e54ff9700 1 --2- 172.21.15.102:0/50308130 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f0e4006e210 0x7f0e400706c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:02.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.523+0000 7f0e54ff9700 1 -- 172.21.15.102:0/50308130 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f0e4808df70 con 0x7f0e580fe9d0 2024-03-26T18:56:02.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.523+0000 7f0e56ffd700 1 --2- 172.21.15.102:0/50308130 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f0e4006e210 0x7f0e400706c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:02.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.524+0000 7f0e56ffd700 1 --2- 172.21.15.102:0/50308130 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f0e4006e210 0x7f0e400706c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c006010 tx=0x7f0e4c00b560 comp rx=0 tx=0).ready entity=mgr.14704 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:02.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.528+0000 7f0e54ff9700 1 -- 172.21.15.102:0/50308130 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f0e480c2470 con 0x7f0e580fe9d0 2024-03-26T18:56:02.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.757+0000 7f0e5e1de700 1 -- 172.21.15.102:0/50308130 --> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e58108720 con 0x7f0e4006e210 2024-03-26T18:56:02.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.758+0000 7f0e54ff9700 1 -- 172.21.15.102:0/50308130 <== mgr.14704 v2:172.21.15.102:6800/2831156761 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f0e58108720 con 0x7f0e4006e210 2024-03-26T18:56:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.761+0000 7f0e5e1de700 1 -- 172.21.15.102:0/50308130 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f0e4006e210 msgr2=0x7f0e400706c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.761+0000 7f0e5e1de700 1 --2- 172.21.15.102:0/50308130 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f0e4006e210 0x7f0e400706c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c006010 tx=0x7f0e4c00b560 comp rx=0 tx=0).stop 2024-03-26T18:56:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.761+0000 7f0e5e1de700 1 -- 172.21.15.102:0/50308130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e580fe9d0 msgr2=0x7f0e58197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.761+0000 7f0e5e1de700 1 --2- 172.21.15.102:0/50308130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e580fe9d0 0x7f0e58197b40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800d8d0 tx=0x7f0e4800dbe0 comp rx=0 tx=0).stop 2024-03-26T18:56:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.761+0000 7f0e5e1de700 1 -- 172.21.15.102:0/50308130 shutdown_connections 2024-03-26T18:56:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.761+0000 7f0e5e1de700 1 --2- 172.21.15.102:0/50308130 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f0e4006e210 0x7f0e400706c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.761+0000 7f0e5e1de700 1 --2- 172.21.15.102:0/50308130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e580fe9d0 0x7f0e58197b40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.761+0000 7f0e5e1de700 1 --2- 172.21.15.102:0/50308130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e580ff320 0x7f0e58198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.761+0000 7f0e5e1de700 1 -- 172.21.15.102:0/50308130 >> 172.21.15.102:0/50308130 conn(0x7f0e580fa5c0 msgr2=0x7f0e58107000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.761+0000 7f0e5e1de700 1 -- 172.21.15.102:0/50308130 shutdown_connections 2024-03-26T18:56:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.761+0000 7f0e5e1de700 1 -- 172.21.15.102:0/50308130 wait complete. 2024-03-26T18:56:02.764 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:56:02.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.868+0000 7f877a478700 1 -- 172.21.15.102:0/2034518908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8774103500 msgr2=0x7f87741058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.868+0000 7f877a478700 1 --2- 172.21.15.102:0/2034518908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8774103500 0x7f87741058e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f875c009910 tx=0x7f875c009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.868+0000 7f877a478700 1 -- 172.21.15.102:0/2034518908 shutdown_connections 2024-03-26T18:56:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.868+0000 7f877a478700 1 --2- 172.21.15.102:0/2034518908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8774103500 0x7f87741058e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.868+0000 7f877a478700 1 --2- 172.21.15.102:0/2034518908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8774100be0 0x7f8774102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.868+0000 7f877a478700 1 -- 172.21.15.102:0/2034518908 >> 172.21.15.102:0/2034518908 conn(0x7f87740fa5c0 msgr2=0x7f87740fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.869+0000 7f877a478700 1 -- 172.21.15.102:0/2034518908 shutdown_connections 2024-03-26T18:56:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.869+0000 7f877a478700 1 -- 172.21.15.102:0/2034518908 wait complete. 2024-03-26T18:56:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.869+0000 7f877a478700 1 Processor -- start 2024-03-26T18:56:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.869+0000 7f877a478700 1 -- start start 2024-03-26T18:56:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.869+0000 7f877a478700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8774100be0 0x7f8774195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.869+0000 7f877a478700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8774103500 0x7f8774195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.869+0000 7f877a478700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87741964b0 con 0x7f8774103500 2024-03-26T18:56:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.869+0000 7f877a478700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87741965f0 con 0x7f8774100be0 2024-03-26T18:56:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.869+0000 7f876bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8774103500 0x7f8774195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.869+0000 7f876bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8774103500 0x7f8774195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41184/0 (socket says 172.21.15.102:41184) 2024-03-26T18:56:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.869+0000 7f876bfff700 1 -- 172.21.15.102:0/886379584 learned_addr learned my addr 172.21.15.102:0/886379584 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.870+0000 7f8773fff700 1 --2- 172.21.15.102:0/886379584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8774100be0 0x7f8774195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.870+0000 7f876bfff700 1 -- 172.21.15.102:0/886379584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8774100be0 msgr2=0x7f8774195950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.870+0000 7f876bfff700 1 --2- 172.21.15.102:0/886379584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8774100be0 0x7f8774195950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.870+0000 7f876bfff700 1 -- 172.21.15.102:0/886379584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c0095f0 con 0x7f8774103500 2024-03-26T18:56:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.870+0000 7f876bfff700 1 --2- 172.21.15.102:0/886379584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8774103500 0x7f8774195e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f875c006010 tx=0x7f875c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.870+0000 7f8771ffb700 1 -- 172.21.15.102:0/886379584 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f875c01d070 con 0x7f8774103500 2024-03-26T18:56:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.870+0000 7f8771ffb700 1 -- 172.21.15.102:0/886379584 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f875c00f460 con 0x7f8774103500 2024-03-26T18:56:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.870+0000 7f877a478700 1 -- 172.21.15.102:0/886379584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877419b040 con 0x7f8774103500 2024-03-26T18:56:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.870+0000 7f8771ffb700 1 -- 172.21.15.102:0/886379584 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f875c021640 con 0x7f8774103500 2024-03-26T18:56:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.870+0000 7f877a478700 1 -- 172.21.15.102:0/886379584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877419b530 con 0x7f8774103500 2024-03-26T18:56:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.871+0000 7f877a478700 1 -- 172.21.15.102:0/886379584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877418fb70 con 0x7f8774103500 2024-03-26T18:56:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.872+0000 7f8771ffb700 1 -- 172.21.15.102:0/886379584 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 30) v1 ==== 92174+0+0 (secure 0 0 0) 0x7f875c00fab0 con 0x7f8774103500 2024-03-26T18:56:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.873+0000 7f8771ffb700 1 --2- 172.21.15.102:0/886379584 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f876006e220 0x7f87600706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.873+0000 7f8771ffb700 1 -- 172.21.15.102:0/886379584 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f875c08f500 con 0x7f8774103500 2024-03-26T18:56:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.873+0000 7f8773fff700 1 --2- 172.21.15.102:0/886379584 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f876006e220 0x7f87600706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.873+0000 7f8773fff700 1 --2- 172.21.15.102:0/886379584 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f876006e220 0x7f87600706d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8764005950 tx=0x7f876400a300 comp rx=0 tx=0).ready entity=mgr.14704 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:02.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:02.880+0000 7f8771ffb700 1 -- 172.21.15.102:0/886379584 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f875c05c4f0 con 0x7f8774103500 2024-03-26T18:56:03.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.112+0000 7f877a478700 1 -- 172.21.15.102:0/886379584 --> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8774061190 con 0x7f876006e220 2024-03-26T18:56:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.114+0000 7f8771ffb700 1 -- 172.21.15.102:0/886379584 <== mgr.14704 v2:172.21.15.102:6800/2831156761 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f8774061190 con 0x7f876006e220 2024-03-26T18:56:03.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.116+0000 7f877a478700 1 -- 172.21.15.102:0/886379584 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f876006e220 msgr2=0x7f87600706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:03.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.116+0000 7f877a478700 1 --2- 172.21.15.102:0/886379584 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f876006e220 0x7f87600706d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8764005950 tx=0x7f876400a300 comp rx=0 tx=0).stop 2024-03-26T18:56:03.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.116+0000 7f877a478700 1 -- 172.21.15.102:0/886379584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8774103500 msgr2=0x7f8774195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:03.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.116+0000 7f877a478700 1 --2- 172.21.15.102:0/886379584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8774103500 0x7f8774195e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f875c006010 tx=0x7f875c00bd70 comp rx=0 tx=0).stop 2024-03-26T18:56:03.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.116+0000 7f877a478700 1 -- 172.21.15.102:0/886379584 shutdown_connections 2024-03-26T18:56:03.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.116+0000 7f877a478700 1 --2- 172.21.15.102:0/886379584 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f876006e220 0x7f87600706d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.116+0000 7f877a478700 1 --2- 172.21.15.102:0/886379584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8774100be0 0x7f8774195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.116+0000 7f877a478700 1 --2- 172.21.15.102:0/886379584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8774103500 0x7f8774195e90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.116+0000 7f877a478700 1 -- 172.21.15.102:0/886379584 >> 172.21.15.102:0/886379584 conn(0x7f87740fa5c0 msgr2=0x7f87740fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.116+0000 7f877a478700 1 -- 172.21.15.102:0/886379584 shutdown_connections 2024-03-26T18:56:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.116+0000 7f877a478700 1 -- 172.21.15.102:0/886379584 wait complete. 2024-03-26T18:56:03.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.224+0000 7f2da5cc3700 1 -- 172.21.15.102:0/1829120211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da0103500 msgr2=0x7f2da01058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:03.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.224+0000 7f2da5cc3700 1 --2- 172.21.15.102:0/1829120211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da0103500 0x7f2da01058e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009910 tx=0x7f2d90009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:03.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.225+0000 7f2da5cc3700 1 -- 172.21.15.102:0/1829120211 shutdown_connections 2024-03-26T18:56:03.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.225+0000 7f2da5cc3700 1 --2- 172.21.15.102:0/1829120211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da0103500 0x7f2da01058e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.225+0000 7f2da5cc3700 1 --2- 172.21.15.102:0/1829120211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da0100be0 0x7f2da0102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.225+0000 7f2da5cc3700 1 -- 172.21.15.102:0/1829120211 >> 172.21.15.102:0/1829120211 conn(0x7f2da00fa600 msgr2=0x7f2da00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:03.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.225+0000 7f2da5cc3700 1 -- 172.21.15.102:0/1829120211 shutdown_connections 2024-03-26T18:56:03.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.225+0000 7f2da5cc3700 1 -- 172.21.15.102:0/1829120211 wait complete. 2024-03-26T18:56:03.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.225+0000 7f2da5cc3700 1 Processor -- start 2024-03-26T18:56:03.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.225+0000 7f2da5cc3700 1 -- start start 2024-03-26T18:56:03.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2da5cc3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da0100be0 0x7f2da0195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:03.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2da5cc3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da0103500 0x7f2da0195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:03.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2da5cc3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da01964b0 con 0x7f2da0103500 2024-03-26T18:56:03.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2da5cc3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da01965f0 con 0x7f2da0100be0 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2d9effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da0103500 0x7f2da0195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2d9effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da0103500 0x7f2da0195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41206/0 (socket says 172.21.15.102:41206) 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2d9effd700 1 -- 172.21.15.102:0/3970255845 learned_addr learned my addr 172.21.15.102:0/3970255845 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2d9f7fe700 1 --2- 172.21.15.102:0/3970255845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da0100be0 0x7f2da0195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2d9effd700 1 -- 172.21.15.102:0/3970255845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da0100be0 msgr2=0x7f2da0195950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2d9effd700 1 --2- 172.21.15.102:0/3970255845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da0100be0 0x7f2da0195950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2d9effd700 1 -- 172.21.15.102:0/3970255845 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d900095f0 con 0x7f2da0103500 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.226+0000 7f2d9effd700 1 --2- 172.21.15.102:0/3970255845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da0103500 0x7f2da0195e90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2d90006010 tx=0x7f2d9000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.227+0000 7f2d9cff9700 1 -- 172.21.15.102:0/3970255845 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d9001d070 con 0x7f2da0103500 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.227+0000 7f2d9cff9700 1 -- 172.21.15.102:0/3970255845 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2d9000f460 con 0x7f2da0103500 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.227+0000 7f2d9cff9700 1 -- 172.21.15.102:0/3970255845 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d90021640 con 0x7f2da0103500 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.227+0000 7f2da5cc3700 1 -- 172.21.15.102:0/3970255845 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da019b040 con 0x7f2da0103500 2024-03-26T18:56:03.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.227+0000 7f2da5cc3700 1 -- 172.21.15.102:0/3970255845 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da019b530 con 0x7f2da0103500 2024-03-26T18:56:03.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.228+0000 7f2da5cc3700 1 -- 172.21.15.102:0/3970255845 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da018fb70 con 0x7f2da0103500 2024-03-26T18:56:03.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.228+0000 7f2d9cff9700 1 -- 172.21.15.102:0/3970255845 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 30) v1 ==== 92174+0+0 (secure 0 0 0) 0x7f2d9000fab0 con 0x7f2da0103500 2024-03-26T18:56:03.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.229+0000 7f2d9cff9700 1 --2- 172.21.15.102:0/3970255845 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2d8c06e220 0x7f2d8c0706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:03.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.229+0000 7f2d9cff9700 1 -- 172.21.15.102:0/3970255845 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f2d9008f410 con 0x7f2da0103500 2024-03-26T18:56:03.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.229+0000 7f2d9f7fe700 1 --2- 172.21.15.102:0/3970255845 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2d8c06e220 0x7f2d8c0706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:03.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.229+0000 7f2d9f7fe700 1 --2- 172.21.15.102:0/3970255845 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2d8c06e220 0x7f2d8c0706d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2d88005950 tx=0x7f2d8800b410 comp rx=0 tx=0).ready entity=mgr.14704 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:03.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.234+0000 7f2d9cff9700 1 -- 172.21.15.102:0/3970255845 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f2d9005c3a0 con 0x7f2da0103500 2024-03-26T18:56:03.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.455+0000 7f2da5cc3700 1 -- 172.21.15.102:0/3970255845 --> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2da0061190 con 0x7f2d8c06e220 2024-03-26T18:56:03.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.466+0000 7f2d9cff9700 1 -- 172.21.15.102:0/3970255845 <== mgr.14704 v2:172.21.15.102:6800/2831156761 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2da0061190 con 0x7f2d8c06e220 2024-03-26T18:56:03.460 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:56:03.460 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (7m) 4s ago 9m 25.1M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:56:03.460 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (9m) 4s ago 9m 7700k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:56:03.460 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (8m) 17s ago 8m 7596k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:56:03.460 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (9m) 4s ago 9m 6665k - 18.2.0 dc2bc1663786 bfac00217c68 2024-03-26T18:56:03.460 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (8m) 17s ago 8m 6656k - 18.2.0 dc2bc1663786 7379f1f6b746 2024-03-26T18:56:03.460 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (7m) 4s ago 8m 84.2M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:56:03.460 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (3m) 4s ago 3m 150M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:56:03.460 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (3m) 4s ago 3m 13.6M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:56:03.460 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (3m) 17s ago 3m 14.3M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (3m) 17s ago 3m 143M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (34s) 4s ago 10m 478M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (90s) 17s ago 8m 75.0M - 19.0.0-2311-g36051606 497bb87bc5da 888321bf02fa 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (10m) 4s ago 10m 54.4M 2048M 18.2.0 dc2bc1663786 b5abd46c3973 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (8m) 17s ago 8m 41.2M 2048M 18.2.0 dc2bc1663786 6bd913d483f0 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (9m) 4s ago 9m 15.8M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (8m) 17s ago 8m 15.1M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (7m) 4s ago 7m 217M 1216M 18.2.0 dc2bc1663786 3e2065fb5fbc 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (6m) 4s ago 6m 240M 1216M 18.2.0 dc2bc1663786 0bf773515cc1 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (6m) 4s ago 6m 182M 1216M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (5m) 17s ago 5m 264M 2240M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (5m) 17s ago 5m 246M 2240M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (5m) 17s ago 5m 218M 2240M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:56:03.461 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (41s) 4s ago 8m 54.9M - 2.43.0 a07b618ecd1d 21b452cc4ead 2024-03-26T18:56:03.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.469+0000 7f2da5cc3700 1 -- 172.21.15.102:0/3970255845 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2d8c06e220 msgr2=0x7f2d8c0706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:03.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.469+0000 7f2da5cc3700 1 --2- 172.21.15.102:0/3970255845 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2d8c06e220 0x7f2d8c0706d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2d88005950 tx=0x7f2d8800b410 comp rx=0 tx=0).stop 2024-03-26T18:56:03.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.469+0000 7f2da5cc3700 1 -- 172.21.15.102:0/3970255845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da0103500 msgr2=0x7f2da0195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.469+0000 7f2da5cc3700 1 --2- 172.21.15.102:0/3970255845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da0103500 0x7f2da0195e90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2d90006010 tx=0x7f2d9000bd70 comp rx=0 tx=0).stop 2024-03-26T18:56:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.469+0000 7f2da5cc3700 1 -- 172.21.15.102:0/3970255845 shutdown_connections 2024-03-26T18:56:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.469+0000 7f2da5cc3700 1 --2- 172.21.15.102:0/3970255845 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2d8c06e220 0x7f2d8c0706d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.469+0000 7f2da5cc3700 1 --2- 172.21.15.102:0/3970255845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da0100be0 0x7f2da0195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.469+0000 7f2da5cc3700 1 --2- 172.21.15.102:0/3970255845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da0103500 0x7f2da0195e90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.469+0000 7f2da5cc3700 1 -- 172.21.15.102:0/3970255845 >> 172.21.15.102:0/3970255845 conn(0x7f2da00fa600 msgr2=0x7f2da00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.469+0000 7f2da5cc3700 1 -- 172.21.15.102:0/3970255845 shutdown_connections 2024-03-26T18:56:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.469+0000 7f2da5cc3700 1 -- 172.21.15.102:0/3970255845 wait complete. 2024-03-26T18:56:03.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:03 smithi102 ceph-mon[29002]: pgmap v13: 65 pgs: 65 active+clean; 301 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 38 KiB/s rd, 1.6 MiB/s wr, 361 op/s 2024-03-26T18:56:03.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:03 smithi102 ceph-mon[29002]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:03.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.577+0000 7fa2b3317700 1 -- 172.21.15.102:0/2714417612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2ac101370 msgr2=0x7fa2ac1017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:03.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.577+0000 7fa2b3317700 1 --2- 172.21.15.102:0/2714417612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2ac101370 0x7fa2ac1017c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa2a000b0a0 tx=0x7fa2a000b3b0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.577+0000 7fa2b3317700 1 -- 172.21.15.102:0/2714417612 shutdown_connections 2024-03-26T18:56:03.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.577+0000 7fa2b3317700 1 --2- 172.21.15.102:0/2714417612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2ac101370 0x7fa2ac1017c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.577+0000 7fa2b3317700 1 --2- 172.21.15.102:0/2714417612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2ac100170 0x7fa2ac100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.577+0000 7fa2b3317700 1 -- 172.21.15.102:0/2714417612 >> 172.21.15.102:0/2714417612 conn(0x7fa2ac0fb6e0 msgr2=0x7fa2ac0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:03.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.577+0000 7fa2b3317700 1 -- 172.21.15.102:0/2714417612 shutdown_connections 2024-03-26T18:56:03.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.577+0000 7fa2b3317700 1 -- 172.21.15.102:0/2714417612 wait complete. 2024-03-26T18:56:03.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b3317700 1 Processor -- start 2024-03-26T18:56:03.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b3317700 1 -- start start 2024-03-26T18:56:03.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b3317700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2ac100170 0x7fa2ac195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:03.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b3317700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2ac101370 0x7fa2ac195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:03.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b3317700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac1964e0 con 0x7fa2ac100170 2024-03-26T18:56:03.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b3317700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac196620 con 0x7fa2ac101370 2024-03-26T18:56:03.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b10b3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2ac100170 0x7fa2ac195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:03.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b10b3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2ac100170 0x7fa2ac195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41224/0 (socket says 172.21.15.102:41224) 2024-03-26T18:56:03.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b10b3700 1 -- 172.21.15.102:0/1521831864 learned_addr learned my addr 172.21.15.102:0/1521831864 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:03.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b08b2700 1 --2- 172.21.15.102:0/1521831864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2ac101370 0x7fa2ac195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:03.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b10b3700 1 -- 172.21.15.102:0/1521831864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2ac101370 msgr2=0x7fa2ac195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:03.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b10b3700 1 --2- 172.21.15.102:0/1521831864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2ac101370 0x7fa2ac195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.578+0000 7fa2b10b3700 1 -- 172.21.15.102:0/1521831864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a0009d00 con 0x7fa2ac100170 2024-03-26T18:56:03.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.579+0000 7fa2b10b3700 1 --2- 172.21.15.102:0/1521831864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2ac100170 0x7fa2ac195980 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400d900 tx=0x7fa2a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:03.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.579+0000 7fa2aa7fc700 1 -- 172.21.15.102:0/1521831864 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2a40041f0 con 0x7fa2ac100170 2024-03-26T18:56:03.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.579+0000 7fa2aa7fc700 1 -- 172.21.15.102:0/1521831864 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa2a4004350 con 0x7fa2ac100170 2024-03-26T18:56:03.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.579+0000 7fa2b3317700 1 -- 172.21.15.102:0/1521831864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac19b0d0 con 0x7fa2ac100170 2024-03-26T18:56:03.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.579+0000 7fa2b3317700 1 -- 172.21.15.102:0/1521831864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ac19b6a0 con 0x7fa2ac100170 2024-03-26T18:56:03.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.579+0000 7fa2aa7fc700 1 -- 172.21.15.102:0/1521831864 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2a4003da0 con 0x7fa2ac100170 2024-03-26T18:56:03.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.580+0000 7fa2b3317700 1 -- 172.21.15.102:0/1521831864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ac108cc0 con 0x7fa2ac100170 2024-03-26T18:56:03.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.580+0000 7fa2aa7fc700 1 -- 172.21.15.102:0/1521831864 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 30) v1 ==== 92174+0+0 (secure 0 0 0) 0x7fa2a4009730 con 0x7fa2ac100170 2024-03-26T18:56:03.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.581+0000 7fa2aa7fc700 1 --2- 172.21.15.102:0/1521831864 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7fa29406e220 0x7fa2940706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:03.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.581+0000 7fa2aa7fc700 1 -- 172.21.15.102:0/1521831864 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fa2a408e210 con 0x7fa2ac100170 2024-03-26T18:56:03.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.581+0000 7fa2b08b2700 1 --2- 172.21.15.102:0/1521831864 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7fa29406e220 0x7fa2940706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:03.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.581+0000 7fa2b08b2700 1 --2- 172.21.15.102:0/1521831864 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7fa29406e220 0x7fa2940706d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009480 tx=0x7fa2a0000f40 comp rx=0 tx=0).ready entity=mgr.14704 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:03.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.586+0000 7fa2aa7fc700 1 -- 172.21.15.102:0/1521831864 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7fa2a405b200 con 0x7fa2ac100170 2024-03-26T18:56:03.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:03 smithi190 ceph-mon[35043]: pgmap v13: 65 pgs: 65 active+clean; 301 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 38 KiB/s rd, 1.6 MiB/s wr, 361 op/s 2024-03-26T18:56:03.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:03 smithi190 ceph-mon[35043]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:03.963 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 12, 2024-03-26T18:56:03.964 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:56:03.965 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:56:03.965 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:56:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.969+0000 7fa2b3317700 1 -- 172.21.15.102:0/1521831864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa2ac19bd90 con 0x7fa2ac100170 2024-03-26T18:56:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.970+0000 7fa2aa7fc700 1 -- 172.21.15.102:0/1521831864 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fa2a405ad90 con 0x7fa2ac100170 2024-03-26T18:56:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.972+0000 7fa2b3317700 1 -- 172.21.15.102:0/1521831864 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7fa29406e220 msgr2=0x7fa2940706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.972+0000 7fa2b3317700 1 --2- 172.21.15.102:0/1521831864 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7fa29406e220 0x7fa2940706d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009480 tx=0x7fa2a0000f40 comp rx=0 tx=0).stop 2024-03-26T18:56:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.972+0000 7fa2b3317700 1 -- 172.21.15.102:0/1521831864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2ac100170 msgr2=0x7fa2ac195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.972+0000 7fa2b3317700 1 --2- 172.21.15.102:0/1521831864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2ac100170 0x7fa2ac195980 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400d900 tx=0x7fa2a400dcc0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.972+0000 7fa2b3317700 1 -- 172.21.15.102:0/1521831864 shutdown_connections 2024-03-26T18:56:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.972+0000 7fa2b3317700 1 --2- 172.21.15.102:0/1521831864 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7fa29406e220 0x7fa2940706d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.972+0000 7fa2b3317700 1 --2- 172.21.15.102:0/1521831864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2ac100170 0x7fa2ac195980 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.972+0000 7fa2b3317700 1 --2- 172.21.15.102:0/1521831864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2ac101370 0x7fa2ac195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.972+0000 7fa2b3317700 1 -- 172.21.15.102:0/1521831864 >> 172.21.15.102:0/1521831864 conn(0x7fa2ac0fb6e0 msgr2=0x7fa2ac1045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:03.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.973+0000 7fa2b3317700 1 -- 172.21.15.102:0/1521831864 shutdown_connections 2024-03-26T18:56:03.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:03.973+0000 7fa2b3317700 1 -- 172.21.15.102:0/1521831864 wait complete. 2024-03-26T18:56:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.101+0000 7f1c758b6700 1 -- 172.21.15.102:0/201094778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c70100260 msgr2=0x7f1c701006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.101+0000 7f1c758b6700 1 --2- 172.21.15.102:0/201094778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c70100260 0x7f1c701006b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009910 tx=0x7f1c64009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.101+0000 7f1c758b6700 1 -- 172.21.15.102:0/201094778 shutdown_connections 2024-03-26T18:56:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.101+0000 7f1c758b6700 1 --2- 172.21.15.102:0/201094778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c70100260 0x7f1c701006b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.101+0000 7f1c758b6700 1 --2- 172.21.15.102:0/201094778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c700ff060 0x7f1c700ff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.101+0000 7f1c758b6700 1 -- 172.21.15.102:0/201094778 >> 172.21.15.102:0/201094778 conn(0x7f1c700fa610 msgr2=0x7f1c700fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.101+0000 7f1c758b6700 1 -- 172.21.15.102:0/201094778 shutdown_connections 2024-03-26T18:56:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.101+0000 7f1c758b6700 1 -- 172.21.15.102:0/201094778 wait complete. 2024-03-26T18:56:04.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.101+0000 7f1c758b6700 1 Processor -- start 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.101+0000 7f1c758b6700 1 -- start start 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c758b6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c700ff060 0x7f1c700698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c758b6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c70100260 0x7f1c7019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c758b6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7019aca0 con 0x7f1c70100260 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c758b6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7019ade0 con 0x7f1c700ff060 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c6e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c70100260 0x7f1c7019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c6e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c70100260 0x7f1c7019a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41246/0 (socket says 172.21.15.102:41246) 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c6e7fc700 1 -- 172.21.15.102:0/1877613979 learned_addr learned my addr 172.21.15.102:0/1877613979 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c6effd700 1 --2- 172.21.15.102:0/1877613979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c700ff060 0x7f1c700698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c6e7fc700 1 -- 172.21.15.102:0/1877613979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c700ff060 msgr2=0x7f1c700698d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c6e7fc700 1 --2- 172.21.15.102:0/1877613979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c700ff060 0x7f1c700698d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c6e7fc700 1 -- 172.21.15.102:0/1877613979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640095f0 con 0x7f1c70100260 2024-03-26T18:56:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c6e7fc700 1 --2- 172.21.15.102:0/1877613979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c70100260 0x7f1c7019a6d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1c64006010 tx=0x7f1c6400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c748b4700 1 -- 172.21.15.102:0/1877613979 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c6401d070 con 0x7f1c70100260 2024-03-26T18:56:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c758b6700 1 -- 172.21.15.102:0/1877613979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7019b060 con 0x7f1c70100260 2024-03-26T18:56:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c748b4700 1 -- 172.21.15.102:0/1877613979 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1c6400f460 con 0x7f1c70100260 2024-03-26T18:56:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c748b4700 1 -- 172.21.15.102:0/1877613979 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c64021640 con 0x7f1c70100260 2024-03-26T18:56:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.102+0000 7f1c758b6700 1 -- 172.21.15.102:0/1877613979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7019b550 con 0x7f1c70100260 2024-03-26T18:56:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.104+0000 7f1c758b6700 1 -- 172.21.15.102:0/1877613979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c70107bb0 con 0x7f1c70100260 2024-03-26T18:56:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.104+0000 7f1c748b4700 1 -- 172.21.15.102:0/1877613979 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 30) v1 ==== 92174+0+0 (secure 0 0 0) 0x7f1c64003a60 con 0x7f1c70100260 2024-03-26T18:56:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.105+0000 7f1c748b4700 1 --2- 172.21.15.102:0/1877613979 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1c5806e220 0x7f1c580706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.105+0000 7f1c748b4700 1 -- 172.21.15.102:0/1877613979 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f1c6408f390 con 0x7f1c70100260 2024-03-26T18:56:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.105+0000 7f1c6effd700 1 --2- 172.21.15.102:0/1877613979 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1c5806e220 0x7f1c580706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.105+0000 7f1c6effd700 1 --2- 172.21.15.102:0/1877613979 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1c5806e220 0x7f1c580706d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1c701000c0 tx=0x7f1c6000b410 comp rx=0 tx=0).ready entity=mgr.14704 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:04.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.109+0000 7f1c748b4700 1 -- 172.21.15.102:0/1877613979 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f1c6405c380 con 0x7f1c70100260 2024-03-26T18:56:04.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.370+0000 7f1c758b6700 1 -- 172.21.15.102:0/1877613979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1c7019bb90 con 0x7f1c70100260 2024-03-26T18:56:04.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.371+0000 7f1c748b4700 1 -- 172.21.15.102:0/1877613979 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f1c6405bf10 con 0x7f1c70100260 2024-03-26T18:56:04.364 INFO:teuthology.orchestra.run.smithi102.stdout:e19 2024-03-26T18:56:04.364 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:56:04.364 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:56:04.364 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:56:04.364 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:56:04.364 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:56:04.364 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:56:04.364 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 19 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:55:27.824831+0000 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:56:04.365 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:04.366 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:56:04.367 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:56:04.367 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:56:04.367 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:56:04.367 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:04.367 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.373+0000 7f1c758b6700 1 -- 172.21.15.102:0/1877613979 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1c5806e220 msgr2=0x7f1c580706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.373+0000 7f1c758b6700 1 --2- 172.21.15.102:0/1877613979 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1c5806e220 0x7f1c580706d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1c701000c0 tx=0x7f1c6000b410 comp rx=0 tx=0).stop 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.373+0000 7f1c758b6700 1 -- 172.21.15.102:0/1877613979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c70100260 msgr2=0x7f1c7019a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.373+0000 7f1c758b6700 1 --2- 172.21.15.102:0/1877613979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c70100260 0x7f1c7019a6d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1c64006010 tx=0x7f1c6400bd70 comp rx=0 tx=0).stop 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.373+0000 7f1c758b6700 1 -- 172.21.15.102:0/1877613979 shutdown_connections 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.373+0000 7f1c758b6700 1 --2- 172.21.15.102:0/1877613979 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1c5806e220 0x7f1c580706d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.373+0000 7f1c758b6700 1 --2- 172.21.15.102:0/1877613979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c700ff060 0x7f1c700698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.373+0000 7f1c758b6700 1 --2- 172.21.15.102:0/1877613979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c70100260 0x7f1c7019a6d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.373+0000 7f1c758b6700 1 -- 172.21.15.102:0/1877613979 >> 172.21.15.102:0/1877613979 conn(0x7f1c700fa610 msgr2=0x7f1c70103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.373+0000 7f1c758b6700 1 -- 172.21.15.102:0/1877613979 shutdown_connections 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.373+0000 7f1c758b6700 1 -- 172.21.15.102:0/1877613979 wait complete. 2024-03-26T18:56:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 19 2024-03-26T18:56:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 -- 172.21.15.102:0/3777299431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec100280 msgr2=0x7f1cec1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 --2- 172.21.15.102:0/3777299431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec100280 0x7f1cec1006d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009910 tx=0x7f1cd8009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 -- 172.21.15.102:0/3777299431 shutdown_connections 2024-03-26T18:56:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 --2- 172.21.15.102:0/3777299431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec100280 0x7f1cec1006d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 --2- 172.21.15.102:0/3777299431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cec0ff080 0x7f1cec0ff490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 -- 172.21.15.102:0/3777299431 >> 172.21.15.102:0/3777299431 conn(0x7f1cec0fa630 msgr2=0x7f1cec0fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 -- 172.21.15.102:0/3777299431 shutdown_connections 2024-03-26T18:56:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 -- 172.21.15.102:0/3777299431 wait complete. 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 Processor -- start 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 -- start start 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec0ff080 0x7f1cec1948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cec100280 0x7f1cec194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec195400 con 0x7f1cec0ff080 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.513+0000 7f1cf2be9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec195540 con 0x7f1cec100280 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1cf0985700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec0ff080 0x7f1cec1948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1cebfff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cec100280 0x7f1cec194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1cf0985700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec0ff080 0x7f1cec1948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41266/0 (socket says 172.21.15.102:41266) 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1cf0985700 1 -- 172.21.15.102:0/2740644703 learned_addr learned my addr 172.21.15.102:0/2740644703 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1cf0985700 1 -- 172.21.15.102:0/2740644703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cec100280 msgr2=0x7f1cec194de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1cf0985700 1 --2- 172.21.15.102:0/2740644703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cec100280 0x7f1cec194de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1cf0985700 1 -- 172.21.15.102:0/2740644703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd80095f0 con 0x7f1cec0ff080 2024-03-26T18:56:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1cebfff700 1 --2- 172.21.15.102:0/2740644703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cec100280 0x7f1cec194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:56:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1cf0985700 1 --2- 172.21.15.102:0/2740644703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec0ff080 0x7f1cec1948a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000e8f0 tx=0x7f1ce000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1ce9ffb700 1 -- 172.21.15.102:0/2740644703 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ce0009790 con 0x7f1cec0ff080 2024-03-26T18:56:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1cf2be9700 1 -- 172.21.15.102:0/2740644703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec199ff0 con 0x7f1cec0ff080 2024-03-26T18:56:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1ce9ffb700 1 -- 172.21.15.102:0/2740644703 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1ce000c810 con 0x7f1cec0ff080 2024-03-26T18:56:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.514+0000 7f1ce9ffb700 1 -- 172.21.15.102:0/2740644703 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ce0003d20 con 0x7f1cec0ff080 2024-03-26T18:56:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.515+0000 7f1cf2be9700 1 -- 172.21.15.102:0/2740644703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec068ec0 con 0x7f1cec0ff080 2024-03-26T18:56:04.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.516+0000 7f1cf2be9700 1 -- 172.21.15.102:0/2740644703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec107bd0 con 0x7f1cec0ff080 2024-03-26T18:56:04.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.516+0000 7f1ce9ffb700 1 -- 172.21.15.102:0/2740644703 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 30) v1 ==== 92174+0+0 (secure 0 0 0) 0x7f1ce00041f0 con 0x7f1cec0ff080 2024-03-26T18:56:04.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.517+0000 7f1ce9ffb700 1 --2- 172.21.15.102:0/2740644703 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1cdc070430 0x7f1cdc0728e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:04.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.517+0000 7f1ce9ffb700 1 -- 172.21.15.102:0/2740644703 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f1ce0014070 con 0x7f1cec0ff080 2024-03-26T18:56:04.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.517+0000 7f1cebfff700 1 --2- 172.21.15.102:0/2740644703 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1cdc070430 0x7f1cdc0728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:04.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.518+0000 7f1cebfff700 1 --2- 172.21.15.102:0/2740644703 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1cdc070430 0x7f1cdc0728e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009fd0 tx=0x7f1cd8005940 comp rx=0 tx=0).ready entity=mgr.14704 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:04.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.522+0000 7f1ce9ffb700 1 -- 172.21.15.102:0/2740644703 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f1ce005b390 con 0x7f1cec0ff080 2024-03-26T18:56:04.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.750+0000 7f1cf2be9700 1 -- 172.21.15.102:0/2740644703 --> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cec069a90 con 0x7f1cdc070430 2024-03-26T18:56:04.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.751+0000 7f1ce9ffb700 1 -- 172.21.15.102:0/2740644703 <== mgr.14704 v2:172.21.15.102:6800/2831156761 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f1cec069a90 con 0x7f1cdc070430 2024-03-26T18:56:04.745 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:56:04.745 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:56:04.745 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:56:04.745 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:56:04.745 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T18:56:04.746 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr" 2024-03-26T18:56:04.746 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T18:56:04.746 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "2/23 daemons upgraded", 2024-03-26T18:56:04.746 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "", 2024-03-26T18:56:04.746 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:56:04.746 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:56:04.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.754+0000 7f1cf2be9700 1 -- 172.21.15.102:0/2740644703 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1cdc070430 msgr2=0x7f1cdc0728e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:04.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.754+0000 7f1cf2be9700 1 --2- 172.21.15.102:0/2740644703 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1cdc070430 0x7f1cdc0728e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009fd0 tx=0x7f1cd8005940 comp rx=0 tx=0).stop 2024-03-26T18:56:04.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.754+0000 7f1cf2be9700 1 -- 172.21.15.102:0/2740644703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec0ff080 msgr2=0x7f1cec1948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:04.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.754+0000 7f1cf2be9700 1 --2- 172.21.15.102:0/2740644703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec0ff080 0x7f1cec1948a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000e8f0 tx=0x7f1ce000ec00 comp rx=0 tx=0).stop 2024-03-26T18:56:04.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.754+0000 7f1cf2be9700 1 -- 172.21.15.102:0/2740644703 shutdown_connections 2024-03-26T18:56:04.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.754+0000 7f1cf2be9700 1 --2- 172.21.15.102:0/2740644703 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f1cdc070430 0x7f1cdc0728e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.754+0000 7f1cf2be9700 1 --2- 172.21.15.102:0/2740644703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec0ff080 0x7f1cec1948a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.754+0000 7f1cf2be9700 1 --2- 172.21.15.102:0/2740644703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cec100280 0x7f1cec194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.754+0000 7f1cf2be9700 1 -- 172.21.15.102:0/2740644703 >> 172.21.15.102:0/2740644703 conn(0x7f1cec0fa630 msgr2=0x7f1cec1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:04.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.754+0000 7f1cf2be9700 1 -- 172.21.15.102:0/2740644703 shutdown_connections 2024-03-26T18:56:04.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.754+0000 7f1cf2be9700 1 -- 172.21.15.102:0/2740644703 wait complete. 2024-03-26T18:56:04.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.865+0000 7f2f006d4700 1 -- 172.21.15.102:0/1907581664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ef80fe410 msgr2=0x7f2ef80fe820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.865+0000 7f2f006d4700 1 --2- 172.21.15.102:0/1907581664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ef80fe410 0x7f2ef80fe820 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009910 tx=0x7f2ef4009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.865+0000 7f2f006d4700 1 -- 172.21.15.102:0/1907581664 shutdown_connections 2024-03-26T18:56:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.865+0000 7f2f006d4700 1 --2- 172.21.15.102:0/1907581664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ef81013f0 0x7f2ef8101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.865+0000 7f2f006d4700 1 --2- 172.21.15.102:0/1907581664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ef80fe410 0x7f2ef80fe820 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.865+0000 7f2f006d4700 1 -- 172.21.15.102:0/1907581664 >> 172.21.15.102:0/1907581664 conn(0x7f2ef80f9d80 msgr2=0x7f2ef80fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.865+0000 7f2f006d4700 1 -- 172.21.15.102:0/1907581664 shutdown_connections 2024-03-26T18:56:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.865+0000 7f2f006d4700 1 -- 172.21.15.102:0/1907581664 wait complete. 2024-03-26T18:56:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2f006d4700 1 Processor -- start 2024-03-26T18:56:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2f006d4700 1 -- start start 2024-03-26T18:56:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2f006d4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ef80fe410 0x7f2ef8100d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2f006d4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ef81013f0 0x7f2ef80ff3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2f006d4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef80ff8e0 con 0x7f2ef80fe410 2024-03-26T18:56:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2f006d4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef80ffa20 con 0x7f2ef81013f0 2024-03-26T18:56:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2efe470700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ef80fe410 0x7f2ef8100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2efe470700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ef80fe410 0x7f2ef8100d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41278/0 (socket says 172.21.15.102:41278) 2024-03-26T18:56:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2efe470700 1 -- 172.21.15.102:0/36911873 learned_addr learned my addr 172.21.15.102:0/36911873 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2efdc6f700 1 --2- 172.21.15.102:0/36911873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ef81013f0 0x7f2ef80ff3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2efe470700 1 -- 172.21.15.102:0/36911873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ef81013f0 msgr2=0x7f2ef80ff3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2efe470700 1 --2- 172.21.15.102:0/36911873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ef81013f0 0x7f2ef80ff3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.866+0000 7f2efe470700 1 -- 172.21.15.102:0/36911873 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40095f0 con 0x7f2ef80fe410 2024-03-26T18:56:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.867+0000 7f2efe470700 1 --2- 172.21.15.102:0/36911873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ef80fe410 0x7f2ef8100d50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4000c00 tx=0x7f2ef400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.867+0000 7f2eef7fe700 1 -- 172.21.15.102:0/36911873 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef401d070 con 0x7f2ef80fe410 2024-03-26T18:56:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.867+0000 7f2f006d4700 1 -- 172.21.15.102:0/36911873 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef80ffca0 con 0x7f2ef80fe410 2024-03-26T18:56:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.867+0000 7f2eef7fe700 1 -- 172.21.15.102:0/36911873 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2ef4005dc0 con 0x7f2ef80fe410 2024-03-26T18:56:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.867+0000 7f2eef7fe700 1 -- 172.21.15.102:0/36911873 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef4022470 con 0x7f2ef80fe410 2024-03-26T18:56:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.867+0000 7f2f006d4700 1 -- 172.21.15.102:0/36911873 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef8100190 con 0x7f2ef80fe410 2024-03-26T18:56:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.868+0000 7f2f006d4700 1 -- 172.21.15.102:0/36911873 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef8108c90 con 0x7f2ef80fe410 2024-03-26T18:56:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.868+0000 7f2eef7fe700 1 -- 172.21.15.102:0/36911873 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 30) v1 ==== 92174+0+0 (secure 0 0 0) 0x7f2ef4003810 con 0x7f2ef80fe410 2024-03-26T18:56:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.869+0000 7f2eef7fe700 1 --2- 172.21.15.102:0/36911873 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2ee406e220 0x7f2ee40706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.869+0000 7f2eef7fe700 1 -- 172.21.15.102:0/36911873 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f2ef408f5f0 con 0x7f2ef80fe410 2024-03-26T18:56:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.869+0000 7f2efdc6f700 1 --2- 172.21.15.102:0/36911873 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2ee406e220 0x7f2ee40706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.869+0000 7f2efdc6f700 1 --2- 172.21.15.102:0/36911873 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2ee406e220 0x7f2ee40706d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009c60 tx=0x7f2ee8009260 comp rx=0 tx=0).ready entity=mgr.14704 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:04.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:04.873+0000 7f2eef7fe700 1 -- 172.21.15.102:0/36911873 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185430 (secure 0 0 0) 0x7f2ef405c5e0 con 0x7f2ef80fe410 2024-03-26T18:56:05.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:04 smithi102 ceph-mon[29002]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:05.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:04 smithi102 ceph-mon[29002]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:05.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:04 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:05.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:04 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:05.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:04 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:05.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:04 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:56:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:04 smithi102 ceph-mon[29002]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-26T18:56:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:04 smithi102 ceph-mon[29002]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-26T18:56:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:04 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1521831864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:56:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:04 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/1877613979' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:56:05.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.166+0000 7f2f006d4700 1 -- 172.21.15.102:0/36911873 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2ef819de00 con 0x7f2ef80fe410 2024-03-26T18:56:05.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.167+0000 7f2eef7fe700 1 -- 172.21.15.102:0/36911873 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f2ef405c170 con 0x7f2ef80fe410 2024-03-26T18:56:05.160 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T18:56:05.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.169+0000 7f2f006d4700 1 -- 172.21.15.102:0/36911873 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2ee406e220 msgr2=0x7f2ee40706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:05.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.169+0000 7f2f006d4700 1 --2- 172.21.15.102:0/36911873 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2ee406e220 0x7f2ee40706d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009c60 tx=0x7f2ee8009260 comp rx=0 tx=0).stop 2024-03-26T18:56:05.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.169+0000 7f2f006d4700 1 -- 172.21.15.102:0/36911873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ef80fe410 msgr2=0x7f2ef8100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:05.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.169+0000 7f2f006d4700 1 --2- 172.21.15.102:0/36911873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ef80fe410 0x7f2ef8100d50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4000c00 tx=0x7f2ef400f690 comp rx=0 tx=0).stop 2024-03-26T18:56:05.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.169+0000 7f2f006d4700 1 -- 172.21.15.102:0/36911873 shutdown_connections 2024-03-26T18:56:05.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.169+0000 7f2f006d4700 1 --2- 172.21.15.102:0/36911873 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f2ee406e220 0x7f2ee40706d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:05.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.169+0000 7f2f006d4700 1 --2- 172.21.15.102:0/36911873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ef80fe410 0x7f2ef8100d50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:05.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.169+0000 7f2f006d4700 1 --2- 172.21.15.102:0/36911873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ef81013f0 0x7f2ef80ff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:05.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.169+0000 7f2f006d4700 1 -- 172.21.15.102:0/36911873 >> 172.21.15.102:0/36911873 conn(0x7f2ef80f9d80 msgr2=0x7f2ef8104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:05.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.169+0000 7f2f006d4700 1 -- 172.21.15.102:0/36911873 shutdown_connections 2024-03-26T18:56:05.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:05.169+0000 7f2f006d4700 1 -- 172.21.15.102:0/36911873 wait complete. 2024-03-26T18:56:05.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:04 smithi190 ceph-mon[35043]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:05.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:04 smithi190 ceph-mon[35043]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:05.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:04 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:05.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:04 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:05.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:04 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:05.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:04 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:56:05.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:04 smithi190 ceph-mon[35043]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-26T18:56:05.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:04 smithi190 ceph-mon[35043]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-26T18:56:05.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:04 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/1521831864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:56:05.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:04 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/1877613979' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:56:06.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:56:06.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:56:06.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: pgmap v14: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 39 KiB/s rd, 1.6 MiB/s wr, 370 op/s 2024-03-26T18:56:06.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:56:06.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:56:06.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:56:06.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:56:06.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:06.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:06.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:06.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:06.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:06.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:06.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:05 smithi102 ceph-mon[29002]: from='client.? 172.21.15.102:0/36911873' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:56:06.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:56:06.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:56:06.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: pgmap v14: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 39 KiB/s rd, 1.6 MiB/s wr, 370 op/s 2024-03-26T18:56:06.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:56:06.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:56:06.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:56:06.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:56:06.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:06.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:06.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:06.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:06.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:06.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:06.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:05 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/36911873' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:56:07.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:06 smithi102 ceph-mon[29002]: Reconfiguring prometheus.smithi102 (dependencies changed)... 2024-03-26T18:56:07.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:06 smithi102 ceph-mon[29002]: Reconfiguring daemon prometheus.smithi102 on smithi102 2024-03-26T18:56:07.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:06 smithi190 ceph-mon[35043]: Reconfiguring prometheus.smithi102 (dependencies changed)... 2024-03-26T18:56:07.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:06 smithi190 ceph-mon[35043]: Reconfiguring daemon prometheus.smithi102 on smithi102 2024-03-26T18:56:08.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:07 smithi190 ceph-mon[35043]: pgmap v15: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 655 KiB/s wr, 171 op/s 2024-03-26T18:56:08.306 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:07 smithi102 ceph-mon[29002]: pgmap v15: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 655 KiB/s wr, 171 op/s 2024-03-26T18:56:09.941 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:09 smithi102 ceph-mon[29002]: pgmap v16: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 655 KiB/s wr, 171 op/s 2024-03-26T18:56:10.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:09 smithi190 ceph-mon[35043]: pgmap v16: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 655 KiB/s wr, 171 op/s 2024-03-26T18:56:11.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:11 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:11.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:11 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:11.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:11 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:56:11.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:11 smithi190 ceph-mon[35043]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:56:11.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:11 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:11.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:11 smithi190 ceph-mon[35043]: pgmap v17: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 655 KiB/s wr, 171 op/s 2024-03-26T18:56:11.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:11 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:11.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:11 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:11.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:11 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:56:11.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:11 smithi102 ceph-mon[29002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T18:56:11.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:11 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:11.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:11 smithi102 ceph-mon[29002]: pgmap v17: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 655 KiB/s wr, 171 op/s 2024-03-26T18:56:12.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:12 smithi190 ceph-mon[35043]: Upgrade: Updating mgr.smithi190.zrixos 2024-03-26T18:56:12.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:12 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.zrixos", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:56:12.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:12 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:56:12.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:12 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:12.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:12 smithi190 ceph-mon[35043]: Deploying daemon mgr.smithi190.zrixos on smithi190 2024-03-26T18:56:12.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:12 smithi102 ceph-mon[29002]: Upgrade: Updating mgr.smithi190.zrixos 2024-03-26T18:56:12.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:12 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.zrixos", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:56:12.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:12 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:56:12.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:12 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:12.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:12 smithi102 ceph-mon[29002]: Deploying daemon mgr.smithi190.zrixos on smithi190 2024-03-26T18:56:13.367 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:13 smithi190 ceph-mon[35043]: pgmap v18: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-26T18:56:13.367 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:13 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:56:13.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:13 smithi102 ceph-mon[29002]: pgmap v18: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-26T18:56:13.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:13 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:56:15.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:15 smithi190 ceph-mon[35043]: pgmap v19: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-26T18:56:15.875 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:15 smithi102 ceph-mon[29002]: pgmap v19: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-26T18:56:16.555 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:16 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:16.555 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:16 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:16.555 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:16 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:16.555 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:16 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:16.555 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:16 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:16.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:16 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:16.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:16 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:16.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:16 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:16.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:16 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:16.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:16 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:17.526 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:17 smithi190 ceph-mon[35043]: pgmap v20: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:17.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:17 smithi102 ceph-mon[29002]: pgmap v20: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:19.655 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:19 smithi190 ceph-mon[35043]: pgmap v21: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:19.655 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:19 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:19.655 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:19 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:19.871 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:19 smithi102 ceph-mon[29002]: pgmap v21: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:19.872 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:19 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:19.872 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:19 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:21.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:20 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:21.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:20 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:21.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:20 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:21.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:20 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:21.925 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:21 smithi190 ceph-mon[35043]: pgmap v22: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:21.925 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:21 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:21.925 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:21 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:22.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:21 smithi102 ceph-mon[29002]: pgmap v22: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:22.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:21 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:22.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:21 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:23.013 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:22 smithi190 ceph-mon[35043]: Standby manager daemon smithi190.zrixos restarted 2024-03-26T18:56:23.013 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:22 smithi190 ceph-mon[35043]: Standby manager daemon smithi190.zrixos started 2024-03-26T18:56:23.013 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:22 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/2245300152' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/crt"}]: dispatch 2024-03-26T18:56:23.013 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:22 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/2245300152' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:56:23.013 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:22 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/2245300152' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/key"}]: dispatch 2024-03-26T18:56:23.013 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:22 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/2245300152' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:56:23.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:22 smithi102 ceph-mon[29002]: Standby manager daemon smithi190.zrixos restarted 2024-03-26T18:56:23.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:22 smithi102 ceph-mon[29002]: Standby manager daemon smithi190.zrixos started 2024-03-26T18:56:23.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:22 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/2245300152' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/crt"}]: dispatch 2024-03-26T18:56:23.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:22 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/2245300152' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:56:23.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:22 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/2245300152' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/key"}]: dispatch 2024-03-26T18:56:23.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:22 smithi102 ceph-mon[29002]: from='mgr.? 172.21.15.190:0/2245300152' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:56:23.789 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: pgmap v23: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:23.789 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: mgrmap e31: smithi102.rsfpnv(active, since 40s), standbys: smithi190.zrixos 2024-03-26T18:56:23.789 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:23.789 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:23.790 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:23.790 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:56:23.790 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:23.790 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:23.790 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:56:23.790 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:23.790 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi102.rsfpnv"}]: dispatch 2024-03-26T18:56:23.790 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi102.rsfpnv"}]': finished 2024-03-26T18:56:23.790 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi190.zrixos"}]: dispatch 2024-03-26T18:56:23.790 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi190.zrixos"}]': finished 2024-03-26T18:56:23.790 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:23 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T18:56:24.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: pgmap v23: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:24.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: mgrmap e31: smithi102.rsfpnv(active, since 40s), standbys: smithi190.zrixos 2024-03-26T18:56:24.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:24.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:24.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:24.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:56:24.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:24.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:24.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:56:24.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:24.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi102.rsfpnv"}]: dispatch 2024-03-26T18:56:24.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi102.rsfpnv"}]': finished 2024-03-26T18:56:24.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi190.zrixos"}]: dispatch 2024-03-26T18:56:24.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi190.zrixos"}]': finished 2024-03-26T18:56:24.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:23 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T18:56:25.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:24 smithi190 ceph-mon[35043]: Upgrade: Setting container_image for all mgr 2024-03-26T18:56:25.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:24 smithi102 ceph-mon[29002]: Upgrade: Setting container_image for all mgr 2024-03-26T18:56:26.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:25 smithi190 ceph-mon[35043]: pgmap v24: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:26.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:25 smithi190 ceph-mon[35043]: Upgrade: Updating mon.smithi102 2024-03-26T18:56:26.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:25 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:26.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:25 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:56:26.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:25 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:56:26.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:25 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:26.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:25 smithi190 ceph-mon[35043]: Deploying daemon mon.smithi102 on smithi102 2024-03-26T18:56:26.119 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:25 smithi102 ceph-mon[29002]: pgmap v24: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:26.120 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:25 smithi102 ceph-mon[29002]: Upgrade: Updating mon.smithi102 2024-03-26T18:56:26.120 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:25 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:26.120 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:25 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:56:26.120 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:25 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:56:26.120 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:25 smithi102 ceph-mon[29002]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:26.120 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:25 smithi102 ceph-mon[29002]: Deploying daemon mon.smithi102 on smithi102 2024-03-26T18:56:27.050 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:26 smithi102 systemd[1]: Stopping Ceph mon.smithi102 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:56:27.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:27 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102[28998]: 2024-03-26T18:56:27.057+0000 7f0bca568700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi102 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T18:56:27.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:27 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102[28998]: 2024-03-26T18:56:27.057+0000 7f0bca568700 -1 mon.smithi102@0(leader) e2 *** Got Signal Terminated *** 2024-03-26T18:56:27.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:27 smithi102 podman[78148]: 2024-03-26 18:56:27.251904126 +0000 UTC m=+0.305908189 container died b5abd46c397361e721a12662f865e86dc7c3adeed8817d539e58ad91c23b5d42 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-26T18:56:27.835 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:27 smithi102 podman[78148]: 2024-03-26 18:56:27.66657707 +0000 UTC m=+0.720581131 container cleanup b5abd46c397361e721a12662f865e86dc7c3adeed8817d539e58ad91c23b5d42 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-26T18:56:27.835 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:27 smithi102 bash[78148]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102 2024-03-26T18:56:28.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:27 smithi102 podman[78173]: 2024-03-26 18:56:27.842902647 +0000 UTC m=+0.586199575 container remove b5abd46c397361e721a12662f865e86dc7c3adeed8817d539e58ad91c23b5d42 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T18:56:28.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi102.service: Deactivated successfully. 2024-03-26T18:56:28.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 systemd[1]: Stopped Ceph mon.smithi102 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:56:28.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi102.service: Consumed 11.514s CPU time. 2024-03-26T18:56:28.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 systemd[1]: Starting Ceph mon.smithi102 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:56:28.781 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 podman[78304]: 2024-03-26 18:56:28.419519966 +0000 UTC m=+0.017849801 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:56:28.782 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 podman[78304]: 2024-03-26 18:56:28.535569059 +0000 UTC m=+0.133898897 container create 9aaa700937607b9246599dc3dc8c91e72cc0eb42f6ebd347b4fc354fe774a4fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T18:56:28.782 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 podman[78304]: 2024-03-26 18:56:28.76341374 +0000 UTC m=+0.361743576 container init 9aaa700937607b9246599dc3dc8c91e72cc0eb42f6ebd347b4fc354fe774a4fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T18:56:28.782 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 podman[78304]: 2024-03-26 18:56:28.76660901 +0000 UTC m=+0.364938847 container start 9aaa700937607b9246599dc3dc8c91e72cc0eb42f6ebd347b4fc354fe774a4fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T18:56:29.065 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T18:56:29.065 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev), process ceph-mon, pid 2 2024-03-26T18:56:29.066 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: pidfile_write: ignore empty --pid-file 2024-03-26T18:56:29.066 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: load: jerasure load: lrc 2024-03-26T18:56:29.066 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: RocksDB version: 7.9.2 2024-03-26T18:56:29.066 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Git sha 0 2024-03-26T18:56:29.066 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Compile date 2024-03-20 16:36:06 2024-03-26T18:56:29.066 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: DB SUMMARY 2024-03-26T18:56:29.066 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: DB Session ID: G5GMUUF2B0S1EU8O91CJ 2024-03-26T18:56:29.066 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: CURRENT file: CURRENT 2024-03-26T18:56:29.066 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T18:56:29.066 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1501 Bytes 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi102/store.db dir, Total Num: 1, files: 000032.sst 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi102/store.db: 000030.log size: 775980 ; 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.error_if_exists: 0 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.create_if_missing: 0 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.paranoid_checks: 1 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.env: 0x55f2c31fec60 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.info_log: 0x55f2c56fb9a0 2024-03-26T18:56:29.067 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.statistics: (nil) 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.use_fsync: 0 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_log_file_size: 0 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.allow_fallocate: 1 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.use_direct_reads: 0 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T18:56:29.068 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T18:56:29.069 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.db_log_dir: 2024-03-26T18:56:29.069 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.wal_dir: 2024-03-26T18:56:29.069 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T18:56:29.070 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T18:56:29.070 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T18:56:29.070 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T18:56:29.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T18:56:29.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T18:56:29.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T18:56:29.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T18:56:29.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.write_buffer_manager: 0x55f2c56ce1e0 2024-03-26T18:56:29.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T18:56:29.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T18:56:29.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T18:56:29.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T18:56:29.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T18:56:29.071 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.unordered_write: 0 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.row_cache: None 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.wal_filter: None 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.two_write_queues: 0 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T18:56:29.072 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.wal_compression: 0 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.atomic_flush: 0 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.log_readahead_size: 0 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_background_jobs: 2 2024-03-26T18:56:29.073 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_background_compactions: -1 2024-03-26T18:56:29.074 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_subcompactions: 1 2024-03-26T18:56:29.074 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T18:56:29.074 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T18:56:29.074 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T18:56:29.074 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T18:56:29.074 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T18:56:29.074 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T18:56:29.074 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T18:56:29.074 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T18:56:29.075 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_open_files: -1 2024-03-26T18:56:29.075 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T18:56:29.075 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T18:56:29.075 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T18:56:29.075 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T18:56:29.075 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_background_flushes: -1 2024-03-26T18:56:29.075 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Compression algorithms supported: 2024-03-26T18:56:29.075 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T18:56:29.075 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: kZSTD supported: 0 2024-03-26T18:56:29.075 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: kXpressCompression supported: 0 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: kZlibCompression supported: 1 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: kSnappyCompression supported: 1 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: kLZ4Compression supported: 1 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: kBZip2Compression supported: 0 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi102/store.db/MANIFEST-000015 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.merge_operator: 2024-03-26T18:56:29.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_filter: None 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f2c56fbaa0) 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: index_type: 0 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: data_block_index_type: 0 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: index_shortening: 1 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T18:56:29.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: checksum: 4 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: no_block_cache: 0 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_cache: 0x55f2c5701090 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_cache_options: 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: capacity : 536870912 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: num_shard_bits : 4 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: strict_capacity_limit : 0 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_cache_compressed: (nil) 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: persistent_cache: (nil) 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_size: 4096 2024-03-26T18:56:29.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_size_deviation: 10 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_restart_interval: 16 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: index_block_restart_interval: 1 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: metadata_block_size: 4096 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: partition_filters: 0 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: use_delta_encoding: 1 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: filter_policy: bloomfilter 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: whole_key_filtering: 1 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: verify_compression: 0 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: format_version: 5 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: enable_index_compression: 1 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: block_align: 0 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: max_auto_readahead_size: 262144 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: prepopulate_block_cache: 0 2024-03-26T18:56:29.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: initial_auto_readahead_size: 8192 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compression: NoCompression 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.num_levels: 7 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T18:56:29.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T18:56:29.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T18:56:29.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T18:56:29.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T18:56:29.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T18:56:29.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T18:56:29.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T18:56:29.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T18:56:29.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T18:56:29.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T18:56:29.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T18:56:29.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T18:56:29.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T18:56:29.085 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T18:56:29.085 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.table_properties_collectors: 2024-03-26T18:56:29.085 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.inplace_update_support: 0 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.bloom_locality: 0 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.max_successive_merges: 0 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.ttl: 2592000 2024-03-26T18:56:29.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.enable_blob_files: false 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.min_blob_size: 0 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T18:56:29.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T18:56:29.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T18:56:29.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi102/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12126, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-03-26T18:56:29.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-03-26T18:56:29.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 286a59b9-b308-4ae2-b340-ff4d305fe399 2024-03-26T18:56:29.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711479388807123, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-03-26T18:56:29.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-03-26T18:56:29.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711479388810279, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 702096, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12127, "largest_seqno": 12571, "table_properties": {"data_size": 699217, "index_size": 1289, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 5229, "raw_average_key_size": 24, "raw_value_size": 694531, "raw_average_value_size": 3291, "num_data_blocks": 57, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711479388, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "286a59b9-b308-4ae2-b340-ff4d305fe399", "db_session_id": "G5GMUUF2B0S1EU8O91CJ", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-03-26T18:56:29.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711479388810399, "job": 1, "event": "recovery_finished"} 2024-03-26T18:56:29.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-03-26T18:56:29.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T18:56:29.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi102/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f2c57f4000 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: DB pointer 0x55f2c57de000 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: ** DB Stats ** 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:56:29.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-26T18:56:29.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T18:56:29.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T18:56:29.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: L0 1/0 685.64 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 359.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T18:56:29.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: L6 1/0 5.90 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-26T18:56:29.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Sum 2/0 6.57 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 359.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T18:56:29.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 359.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T18:56:29.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:56:29.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-26T18:56:29.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T18:56:29.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T18:56:29.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 359.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Cumulative compaction: 0.00 GB write, 36.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Interval compaction: 0.00 GB write, 36.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Block cache BinnedLRUCache@0x55f2c5701090#2 capacity: 512.00 MB usage: 2.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-26T18:56:29.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.66 KB,0.00012517%) IndexBlock(1,1.45 KB,0.000277162%) Misc(1,0.00 KB,0%) 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: starting mon.smithi102 rank 0 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi102 fsid ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: mon.smithi102@-1(???) e2 preinit fsid ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: mon.smithi102@-1(???).mds e19 new map 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: mon.smithi102@-1(???).mds e19 print_map 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: e19 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: legacy client fscid: 1 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Filesystem 'cephfs' (1) 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: fs_name cephfs 2024-03-26T18:56:29.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: epoch 19 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: modified 2024-03-26T18:55:27.824831+0000 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: tableserver 0 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: root 0 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: session_timeout 60 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: session_autoclose 300 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: max_file_size 1099511627776 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: max_xattr_size 65536 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: required_client_features {} 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: last_failure 0 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: last_failure_osd_epoch 47 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:56:29.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: max_mds 2 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: in 0,1 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: up {0=14490,1=24283} 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: failed 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: damaged 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: stopped 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: data_pools [3] 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: metadata_pool 2 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: inline_data disabled 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: balancer 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: bal_rank_mask -1 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: standby_count_wanted 1 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: qdb_cluster leader: 0 members: 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: [mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:29.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: [mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: Standby daemons: 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: [mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout: [mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: mon.smithi102@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: mon.smithi102@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: mon.smithi102@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: mon.smithi102@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 ceph-mon[78322]: mon.smithi102@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 bash[78304]: 9aaa700937607b9246599dc3dc8c91e72cc0eb42f6ebd347b4fc354fe774a4fd 2024-03-26T18:56:29.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:28 smithi102 systemd[1]: Started Ceph mon.smithi102 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:56:30.566 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:56:30.566 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:56:30.566 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:56:30.566 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: pgmap v26: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: mon.smithi102 calling monitor election 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: mon.smithi102 is new leader, mons smithi102,smithi190 in quorum (ranks 0,1) 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: monmap epoch 2 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: fsid ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: last_changed 2024-03-26T18:48:02.282584+0000 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: created 2024-03-26T18:45:22.493426+0000 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: min_mon_release 18 (reef) 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: election_strategy: 1 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.smithi102 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.smithi190 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:56:30.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: osdmap e49: 6 total, 6 up, 6 in 2024-03-26T18:56:30.568 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: mgrmap e31: smithi102.rsfpnv(active, since 47s), standbys: smithi190.zrixos 2024-03-26T18:56:30.568 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: overall HEALTH_OK 2024-03-26T18:56:30.568 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: from='mgr.14704 ' entity='' 2024-03-26T18:56:30.568 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:30 smithi102 ceph-mon[78322]: mgrmap e32: smithi102.rsfpnv(active, since 47s), standbys: smithi190.zrixos 2024-03-26T18:56:30.770 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:56:30.771 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:56:30.771 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: from='mgr.14704 172.21.15.102:0/105231804' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:56:30.771 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: pgmap v26: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:30.771 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: mon.smithi102 calling monitor election 2024-03-26T18:56:30.771 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: mon.smithi102 is new leader, mons smithi102,smithi190 in quorum (ranks 0,1) 2024-03-26T18:56:30.771 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: monmap epoch 2 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: fsid ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: last_changed 2024-03-26T18:48:02.282584+0000 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: created 2024-03-26T18:45:22.493426+0000 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: min_mon_release 18 (reef) 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: election_strategy: 1 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.smithi102 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.smithi190 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: osdmap e49: 6 total, 6 up, 6 in 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: mgrmap e31: smithi102.rsfpnv(active, since 47s), standbys: smithi190.zrixos 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: overall HEALTH_OK 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: from='mgr.14704 ' entity='' 2024-03-26T18:56:30.772 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:30 smithi190 ceph-mon[35043]: mgrmap e32: smithi102.rsfpnv(active, since 47s), standbys: smithi190.zrixos 2024-03-26T18:56:35.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 -- 172.21.15.102:0/3161580678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8908100280 msgr2=0x7f89081006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:35.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 --2- 172.21.15.102:0/3161580678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8908100280 0x7f89081006d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009910 tx=0x7f88f4009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:35.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 -- 172.21.15.102:0/3161580678 shutdown_connections 2024-03-26T18:56:35.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 --2- 172.21.15.102:0/3161580678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8908100280 0x7f89081006d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:35.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 --2- 172.21.15.102:0/3161580678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89080ff080 0x7f89080ff490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:35.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 -- 172.21.15.102:0/3161580678 >> 172.21.15.102:0/3161580678 conn(0x7f89080fa630 msgr2=0x7f89080fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:35.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 -- 172.21.15.102:0/3161580678 shutdown_connections 2024-03-26T18:56:35.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 -- 172.21.15.102:0/3161580678 wait complete. 2024-03-26T18:56:35.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 Processor -- start 2024-03-26T18:56:35.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 -- start start 2024-03-26T18:56:35.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89080ff080 0x7f8908103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:35.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8908100280 0x7f8908101660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:35.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8908101ba0 con 0x7f8908100280 2024-03-26T18:56:35.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.282+0000 7f890ef82700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8908101ce0 con 0x7f89080ff080 2024-03-26T18:56:35.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f8907fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8908100280 0x7f8908101660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:35.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f8907fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8908100280 0x7f8908101660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54350/0 (socket says 172.21.15.102:54350) 2024-03-26T18:56:35.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f8907fff700 1 -- 172.21.15.102:0/1147020539 learned_addr learned my addr 172.21.15.102:0/1147020539 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:35.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f890cd1e700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89080ff080 0x7f8908103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f8907fff700 1 -- 172.21.15.102:0/1147020539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89080ff080 msgr2=0x7f8908103010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f8907fff700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89080ff080 0x7f8908103010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f8907fff700 1 -- 172.21.15.102:0/1147020539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f40095f0 con 0x7f8908100280 2024-03-26T18:56:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f8907fff700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8908100280 0x7f8908101660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f88f4000c00 tx=0x7f88f400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:35.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f8905ffb700 1 -- 172.21.15.102:0/1147020539 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88f401d070 con 0x7f8908100280 2024-03-26T18:56:35.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f8905ffb700 1 -- 172.21.15.102:0/1147020539 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f88f4005dc0 con 0x7f8908100280 2024-03-26T18:56:35.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f890ef82700 1 -- 172.21.15.102:0/1147020539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8908101f60 con 0x7f8908100280 2024-03-26T18:56:35.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f8905ffb700 1 -- 172.21.15.102:0/1147020539 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88f40226a0 con 0x7f8908100280 2024-03-26T18:56:35.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.283+0000 7f890ef82700 1 -- 172.21.15.102:0/1147020539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8908102450 con 0x7f8908100280 2024-03-26T18:56:35.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.284+0000 7f890ef82700 1 -- 172.21.15.102:0/1147020539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8908107bd0 con 0x7f8908100280 2024-03-26T18:56:35.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.285+0000 7f8905ffb700 1 -- 172.21.15.102:0/1147020539 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 32) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f88f400faf0 con 0x7f8908100280 2024-03-26T18:56:35.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.285+0000 7f8905ffb700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 0x7f88f8070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:35.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.285+0000 7f890cd1e700 1 -- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 msgr2=0x7f88f8070a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2831156761 2024-03-26T18:56:35.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.285+0000 7f8905ffb700 1 -- 172.21.15.102:0/1147020539 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f88f408fa10 con 0x7f8908100280 2024-03-26T18:56:35.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.285+0000 7f890cd1e700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 0x7f88f8070a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T18:56:35.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.290+0000 7f8905ffb700 1 -- 172.21.15.102:0/1147020539 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f88f405c680 con 0x7f8908100280 2024-03-26T18:56:35.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.486+0000 7f890cd1e700 1 -- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 msgr2=0x7f88f8070a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2831156761 2024-03-26T18:56:35.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.486+0000 7f890cd1e700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 0x7f88f8070a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T18:56:35.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.515+0000 7f890ef82700 1 -- 172.21.15.102:0/1147020539 --> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8908069400 con 0x7f88f806e5a0 2024-03-26T18:56:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.887+0000 7f890cd1e700 1 -- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 msgr2=0x7f88f8070a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2831156761 2024-03-26T18:56:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:35.887+0000 7f890cd1e700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 0x7f88f8070a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T18:56:36.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:36.688+0000 7f890cd1e700 1 -- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 msgr2=0x7f88f8070a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2831156761 2024-03-26T18:56:36.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:36.688+0000 7f890cd1e700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 0x7f88f8070a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T18:56:37.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:37.185+0000 7f8905ffb700 1 -- 172.21.15.102:0/1147020539 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 33) v1 ==== 91842+0+0 (secure 0 0 0) 0x7f88f40038f0 con 0x7f8908100280 2024-03-26T18:56:37.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:37.186+0000 7f8905ffb700 1 -- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 msgr2=0x7f88f8070a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T18:56:37.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:37.186+0000 7f8905ffb700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 0x7f88f8070a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:37.383 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: Standby manager daemon smithi190.zrixos restarted 2024-03-26T18:56:37.383 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: Standby manager daemon smithi190.zrixos started 2024-03-26T18:56:37.383 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3185422108' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/crt"}]: dispatch 2024-03-26T18:56:37.383 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3185422108' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:56:37.383 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3185422108' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/key"}]: dispatch 2024-03-26T18:56:37.383 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: from='mgr.? 172.21.15.190:0/3185422108' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:56:37.383 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: Active manager daemon smithi102.rsfpnv restarted 2024-03-26T18:56:37.383 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: Activating manager daemon smithi102.rsfpnv 2024-03-26T18:56:37.383 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: osdmap e50: 6 total, 6 up, 6 in 2024-03-26T18:56:37.384 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: mgrmap e33: smithi102.rsfpnv(active, starting, since 0.00651316s), standbys: smithi190.zrixos 2024-03-26T18:56:37.384 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:56:37.384 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:56:37.384 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T18:56:37.384 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T18:56:37.384 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T18:56:37.384 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:37 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T18:56:37.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: Standby manager daemon smithi190.zrixos restarted 2024-03-26T18:56:37.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: Standby manager daemon smithi190.zrixos started 2024-03-26T18:56:37.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: from='mgr.? 172.21.15.190:0/3185422108' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/crt"}]: dispatch 2024-03-26T18:56:37.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: from='mgr.? 172.21.15.190:0/3185422108' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T18:56:37.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: from='mgr.? 172.21.15.190:0/3185422108' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.zrixos/key"}]: dispatch 2024-03-26T18:56:37.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: from='mgr.? 172.21.15.190:0/3185422108' entity='mgr.smithi190.zrixos' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T18:56:37.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: Active manager daemon smithi102.rsfpnv restarted 2024-03-26T18:56:37.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: Activating manager daemon smithi102.rsfpnv 2024-03-26T18:56:37.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: osdmap e50: 6 total, 6 up, 6 in 2024-03-26T18:56:37.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: mgrmap e33: smithi102.rsfpnv(active, starting, since 0.00651316s), standbys: smithi190.zrixos 2024-03-26T18:56:37.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:56:37.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:56:37.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T18:56:37.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T18:56:37.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T18:56:37.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:37 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T18:56:38.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.200+0000 7f8905ffb700 1 -- 172.21.15.102:0/1147020539 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mgrmap(e 34) v1 ==== 91969+0+0 (secure 0 0 0) 0x7f88f408dd70 con 0x7f8908100280 2024-03-26T18:56:38.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.201+0000 7f8905ffb700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f88f801d3c0 0x7f88f801d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:38.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.201+0000 7f8905ffb700 1 -- 172.21.15.102:0/1147020539 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8908069400 con 0x7f88f801d3c0 2024-03-26T18:56:38.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.202+0000 7f890cd1e700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f88f801d3c0 0x7f88f801d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:38.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.202+0000 7f890cd1e700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f88f801d3c0 0x7f88f801d790 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f89081000e0 tx=0x7f88fc009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:38.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.205+0000 7f8905ffb700 1 -- 172.21.15.102:0/1147020539 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f8908069400 con 0x7f88f801d3c0 2024-03-26T18:56:38.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 -- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f88f801d3c0 msgr2=0x7f88f801d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:38.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f88f801d3c0 0x7f88f801d790 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f89081000e0 tx=0x7f88fc009260 comp rx=0 tx=0).stop 2024-03-26T18:56:38.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 -- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8908100280 msgr2=0x7f8908101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:38.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8908100280 0x7f8908101660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f88f4000c00 tx=0x7f88f400f690 comp rx=0 tx=0).stop 2024-03-26T18:56:38.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 -- 172.21.15.102:0/1147020539 shutdown_connections 2024-03-26T18:56:38.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/2831156761,v1:172.21.15.102:6801/2831156761] conn(0x7f88f806e5a0 0x7f88f8070a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89080ff080 0x7f8908103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f88f801d3c0 0x7f88f801d790 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 --2- 172.21.15.102:0/1147020539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8908100280 0x7f8908101660 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 -- 172.21.15.102:0/1147020539 >> 172.21.15.102:0/1147020539 conn(0x7f89080fa630 msgr2=0x7f8908068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:38.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 -- 172.21.15.102:0/1147020539 shutdown_connections 2024-03-26T18:56:38.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.209+0000 7f890ef82700 1 -- 172.21.15.102:0/1147020539 wait complete. 2024-03-26T18:56:38.213 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:56:38.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.316+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/1594538704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c34101370 msgr2=0x7f1c341017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:38.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.316+0000 7f1c3b7c3700 1 --2- 172.21.15.102:0/1594538704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c34101370 0x7f1c341017c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009910 tx=0x7f1c24009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:38.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.316+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/1594538704 shutdown_connections 2024-03-26T18:56:38.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.316+0000 7f1c3b7c3700 1 --2- 172.21.15.102:0/1594538704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c34101370 0x7f1c341017c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.316+0000 7f1c3b7c3700 1 --2- 172.21.15.102:0/1594538704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34100170 0x7f1c34100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.316+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/1594538704 >> 172.21.15.102:0/1594538704 conn(0x7f1c340fb6e0 msgr2=0x7f1c340fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:38.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.316+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/1594538704 shutdown_connections 2024-03-26T18:56:38.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.316+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/1594538704 wait complete. 2024-03-26T18:56:38.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.317+0000 7f1c3b7c3700 1 Processor -- start 2024-03-26T18:56:38.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.317+0000 7f1c3b7c3700 1 -- start start 2024-03-26T18:56:38.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.317+0000 7f1c3b7c3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34100170 0x7f1c34195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:38.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.317+0000 7f1c3b7c3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c34101370 0x7f1c34195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:38.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.317+0000 7f1c3b7c3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c341964e0 con 0x7f1c34101370 2024-03-26T18:56:38.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.317+0000 7f1c3b7c3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c34196620 con 0x7f1c34100170 2024-03-26T18:56:38.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.317+0000 7f1c38d5e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c34101370 0x7f1c34195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:38.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.317+0000 7f1c38d5e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c34101370 0x7f1c34195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54470/0 (socket says 172.21.15.102:54470) 2024-03-26T18:56:38.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.317+0000 7f1c38d5e700 1 -- 172.21.15.102:0/3553524482 learned_addr learned my addr 172.21.15.102:0/3553524482 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:38.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.317+0000 7f1c3955f700 1 --2- 172.21.15.102:0/3553524482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34100170 0x7f1c34195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:38.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.318+0000 7f1c38d5e700 1 -- 172.21.15.102:0/3553524482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34100170 msgr2=0x7f1c34195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:38.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.318+0000 7f1c38d5e700 1 --2- 172.21.15.102:0/3553524482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34100170 0x7f1c34195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.318+0000 7f1c38d5e700 1 -- 172.21.15.102:0/3553524482 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c240095f0 con 0x7f1c34101370 2024-03-26T18:56:38.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.318+0000 7f1c38d5e700 1 --2- 172.21.15.102:0/3553524482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c34101370 0x7f1c34195ec0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1c24000c00 tx=0x7f1c2400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:38.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.318+0000 7f1c2a7fc700 1 -- 172.21.15.102:0/3553524482 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c2401d070 con 0x7f1c34101370 2024-03-26T18:56:38.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.318+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/3553524482 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3419b070 con 0x7f1c34101370 2024-03-26T18:56:38.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.318+0000 7f1c2a7fc700 1 -- 172.21.15.102:0/3553524482 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1c24005dc0 con 0x7f1c34101370 2024-03-26T18:56:38.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.318+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/3553524482 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3419b560 con 0x7f1c34101370 2024-03-26T18:56:38.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.318+0000 7f1c2a7fc700 1 -- 172.21.15.102:0/3553524482 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c240226a0 con 0x7f1c34101370 2024-03-26T18:56:38.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.319+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/3553524482 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c34108cc0 con 0x7f1c34101370 2024-03-26T18:56:38.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.320+0000 7f1c2a7fc700 1 -- 172.21.15.102:0/3553524482 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 34) v1 ==== 91969+0+0 (secure 0 0 0) 0x7f1c240038f0 con 0x7f1c34101370 2024-03-26T18:56:38.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.320+0000 7f1c2a7fc700 1 --2- 172.21.15.102:0/3553524482 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1c2006e180 0x7f1c20070630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:38.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.320+0000 7f1c3955f700 1 --2- 172.21.15.102:0/3553524482 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1c2006e180 0x7f1c20070630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:38.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.320+0000 7f1c2a7fc700 1 -- 172.21.15.102:0/3553524482 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1c2408f440 con 0x7f1c34101370 2024-03-26T18:56:38.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.320+0000 7f1c3955f700 1 --2- 172.21.15.102:0/3553524482 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1c2006e180 0x7f1c20070630 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1c341011d0 tx=0x7f1c30009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.325+0000 7f1c2a7fc700 1 -- 172.21.15.102:0/3553524482 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1c240588a0 con 0x7f1c34101370 2024-03-26T18:56:38.462 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi102.rsfpnv", "id": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi190.zrixos", "id": "smithi190.zrixos"}]: dispatch 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: Manager daemon smithi102.rsfpnv is now available 2024-03-26T18:56:38.463 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:38.464 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:56:38.464 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:56:38.464 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:38 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/trash_purge_schedule"}]: dispatch 2024-03-26T18:56:38.531 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi102.rsfpnv", "id": "smithi102.rsfpnv"}]: dispatch 2024-03-26T18:56:38.531 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr metadata", "who": "smithi190.zrixos", "id": "smithi190.zrixos"}]: dispatch 2024-03-26T18:56:38.531 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: Manager daemon smithi102.rsfpnv is now available 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/mirror_snapshot_schedule"}]: dispatch 2024-03-26T18:56:38.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi102.rsfpnv/trash_purge_schedule"}]: dispatch 2024-03-26T18:56:38.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.557+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/3553524482 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c34105cc0 con 0x7f1c2006e180 2024-03-26T18:56:38.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.558+0000 7f1c2a7fc700 1 -- 172.21.15.102:0/3553524482 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f1c34105cc0 con 0x7f1c2006e180 2024-03-26T18:56:38.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.560+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/3553524482 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1c2006e180 msgr2=0x7f1c20070630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:38.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.560+0000 7f1c3b7c3700 1 --2- 172.21.15.102:0/3553524482 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1c2006e180 0x7f1c20070630 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1c341011d0 tx=0x7f1c30009260 comp rx=0 tx=0).stop 2024-03-26T18:56:38.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.560+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/3553524482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c34101370 msgr2=0x7f1c34195ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:38.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.560+0000 7f1c3b7c3700 1 --2- 172.21.15.102:0/3553524482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c34101370 0x7f1c34195ec0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1c24000c00 tx=0x7f1c2400f690 comp rx=0 tx=0).stop 2024-03-26T18:56:38.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.560+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/3553524482 shutdown_connections 2024-03-26T18:56:38.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.560+0000 7f1c3b7c3700 1 --2- 172.21.15.102:0/3553524482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34100170 0x7f1c34195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.560+0000 7f1c3b7c3700 1 --2- 172.21.15.102:0/3553524482 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1c2006e180 0x7f1c20070630 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.560+0000 7f1c3b7c3700 1 --2- 172.21.15.102:0/3553524482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c34101370 0x7f1c34195ec0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.560+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/3553524482 >> 172.21.15.102:0/3553524482 conn(0x7f1c340fb6e0 msgr2=0x7f1c341045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.561+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/3553524482 shutdown_connections 2024-03-26T18:56:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.561+0000 7f1c3b7c3700 1 -- 172.21.15.102:0/3553524482 wait complete. 2024-03-26T18:56:38.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.666+0000 7f06fa57f700 1 -- 172.21.15.102:0/288761847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f40fe440 msgr2=0x7f06f40fe850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:38.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.666+0000 7f06fa57f700 1 --2- 172.21.15.102:0/288761847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f40fe440 0x7f06f40fe850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009910 tx=0x7f06e4009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:38.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.666+0000 7f06fa57f700 1 -- 172.21.15.102:0/288761847 shutdown_connections 2024-03-26T18:56:38.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.666+0000 7f06fa57f700 1 --2- 172.21.15.102:0/288761847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f40fed90 0x7f06f41014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.666+0000 7f06fa57f700 1 --2- 172.21.15.102:0/288761847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f40fe440 0x7f06f40fe850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.666+0000 7f06fa57f700 1 -- 172.21.15.102:0/288761847 >> 172.21.15.102:0/288761847 conn(0x7f06f40f9e50 msgr2=0x7f06f40fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:38.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.666+0000 7f06fa57f700 1 -- 172.21.15.102:0/288761847 shutdown_connections 2024-03-26T18:56:38.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.666+0000 7f06fa57f700 1 -- 172.21.15.102:0/288761847 wait complete. 2024-03-26T18:56:38.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.667+0000 7f06fa57f700 1 Processor -- start 2024-03-26T18:56:38.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.667+0000 7f06fa57f700 1 -- start start 2024-03-26T18:56:38.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.667+0000 7f06fa57f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f40fe440 0x7f06f4197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:38.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.667+0000 7f06fa57f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f40fed90 0x7f06f4198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:38.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.667+0000 7f06fa57f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f4198720 con 0x7f06f40fed90 2024-03-26T18:56:38.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.667+0000 7f06fa57f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f4198860 con 0x7f06f40fe440 2024-03-26T18:56:38.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.667+0000 7f06f37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f40fed90 0x7f06f4198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.667+0000 7f06f37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f40fed90 0x7f06f4198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46892/0 (socket says 172.21.15.102:46892) 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.667+0000 7f06f37fe700 1 -- 172.21.15.102:0/4045890695 learned_addr learned my addr 172.21.15.102:0/4045890695 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.667+0000 7f06f3fff700 1 --2- 172.21.15.102:0/4045890695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f40fe440 0x7f06f4197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.668+0000 7f06f37fe700 1 -- 172.21.15.102:0/4045890695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f40fe440 msgr2=0x7f06f4197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.668+0000 7f06f37fe700 1 --2- 172.21.15.102:0/4045890695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f40fe440 0x7f06f4197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.668+0000 7f06f37fe700 1 -- 172.21.15.102:0/4045890695 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e40095f0 con 0x7f06f40fed90 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.668+0000 7f06f37fe700 1 --2- 172.21.15.102:0/4045890695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f40fed90 0x7f06f4198100 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f06e800c840 tx=0x7f06e800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.668+0000 7f06f17fa700 1 -- 172.21.15.102:0/4045890695 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e80041f0 con 0x7f06f40fed90 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.668+0000 7f06f17fa700 1 -- 172.21.15.102:0/4045890695 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f06e8004ce0 con 0x7f06f40fed90 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.668+0000 7f06fa57f700 1 -- 172.21.15.102:0/4045890695 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f419d310 con 0x7f06f40fed90 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.668+0000 7f06f17fa700 1 -- 172.21.15.102:0/4045890695 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e8007740 con 0x7f06f40fed90 2024-03-26T18:56:38.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.668+0000 7f06fa57f700 1 -- 172.21.15.102:0/4045890695 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f419d860 con 0x7f06f40fed90 2024-03-26T18:56:38.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.669+0000 7f06fa57f700 1 -- 172.21.15.102:0/4045890695 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f41055a0 con 0x7f06f40fed90 2024-03-26T18:56:38.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.669+0000 7f06f17fa700 1 -- 172.21.15.102:0/4045890695 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 34) v1 ==== 91969+0+0 (secure 0 0 0) 0x7f06e8007a00 con 0x7f06f40fed90 2024-03-26T18:56:38.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.670+0000 7f06f17fa700 1 --2- 172.21.15.102:0/4045890695 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f06dc06e180 0x7f06dc070630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:38.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.670+0000 7f06f17fa700 1 -- 172.21.15.102:0/4045890695 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f06e808dc10 con 0x7f06f40fed90 2024-03-26T18:56:38.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.670+0000 7f06f3fff700 1 --2- 172.21.15.102:0/4045890695 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f06dc06e180 0x7f06dc070630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:38.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.670+0000 7f06f3fff700 1 --2- 172.21.15.102:0/4045890695 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f06dc06e180 0x7f06dc070630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f06e4006010 tx=0x7f06e400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:38.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.674+0000 7f06f17fa700 1 -- 172.21.15.102:0/4045890695 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f06e80571c0 con 0x7f06f40fed90 2024-03-26T18:56:38.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.901+0000 7f06fa57f700 1 -- 172.21.15.102:0/4045890695 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f06f4061190 con 0x7f06dc06e180 2024-03-26T18:56:38.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.913+0000 7f06f17fa700 1 -- 172.21.15.102:0/4045890695 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f06f4061190 con 0x7f06dc06e180 2024-03-26T18:56:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:56:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (8m) 17s ago 9m 25.1M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:56:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (9m) 17s ago 9m 7746k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:56:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (8m) 20s ago 8m 7642k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (9m) 17s ago 9m 6665k - 18.2.0 dc2bc1663786 bfac00217c68 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (8m) 20s ago 8m 6656k - 18.2.0 dc2bc1663786 7379f1f6b746 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (8m) 17s ago 9m 84.2M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (4m) 17s ago 4m 133M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (4m) 17s ago 4m 13.9M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (4m) 20s ago 4m 14.5M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (4m) 20s ago 4m 120M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (70s) 17s ago 11m 494M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (23s) 20s ago 8m 66.5M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (11m) 17s ago 11m 53.2M 2048M 18.2.0 dc2bc1663786 b5abd46c3973 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (8m) 20s ago 8m 40.7M 2048M 18.2.0 dc2bc1663786 6bd913d483f0 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (9m) 17s ago 9m 15.6M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:56:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (8m) 20s ago 8m 15.2M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:56:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (7m) 17s ago 7m 220M 1216M 18.2.0 dc2bc1663786 3e2065fb5fbc 2024-03-26T18:56:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (7m) 17s ago 7m 244M 1216M 18.2.0 dc2bc1663786 0bf773515cc1 2024-03-26T18:56:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (6m) 17s ago 6m 194M 1216M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:56:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (6m) 20s ago 6m 272M 2240M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:56:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (6m) 20s ago 6m 250M 2240M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:56:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (5m) 20s ago 5m 222M 2240M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:56:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (29s) 17s ago 9m 48.6M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T18:56:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.916+0000 7f06fa57f700 1 -- 172.21.15.102:0/4045890695 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f06dc06e180 msgr2=0x7f06dc070630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.916+0000 7f06fa57f700 1 --2- 172.21.15.102:0/4045890695 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f06dc06e180 0x7f06dc070630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f06e4006010 tx=0x7f06e400b540 comp rx=0 tx=0).stop 2024-03-26T18:56:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.916+0000 7f06fa57f700 1 -- 172.21.15.102:0/4045890695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f40fed90 msgr2=0x7f06f4198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.916+0000 7f06fa57f700 1 --2- 172.21.15.102:0/4045890695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f40fed90 0x7f06f4198100 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f06e800c840 tx=0x7f06e800cc00 comp rx=0 tx=0).stop 2024-03-26T18:56:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.917+0000 7f06fa57f700 1 -- 172.21.15.102:0/4045890695 shutdown_connections 2024-03-26T18:56:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.917+0000 7f06fa57f700 1 --2- 172.21.15.102:0/4045890695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f40fe440 0x7f06f4197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.917+0000 7f06fa57f700 1 --2- 172.21.15.102:0/4045890695 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f06dc06e180 0x7f06dc070630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.917+0000 7f06fa57f700 1 --2- 172.21.15.102:0/4045890695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f40fed90 0x7f06f4198100 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.917+0000 7f06fa57f700 1 -- 172.21.15.102:0/4045890695 >> 172.21.15.102:0/4045890695 conn(0x7f06f40f9e50 msgr2=0x7f06f41026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.917+0000 7f06fa57f700 1 -- 172.21.15.102:0/4045890695 shutdown_connections 2024-03-26T18:56:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:38.917+0000 7f06fa57f700 1 -- 172.21.15.102:0/4045890695 wait complete. 2024-03-26T18:56:39.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.022+0000 7ff256b52700 1 -- 172.21.15.102:0/1479786281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff250101390 msgr2=0x7ff2501017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:39.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.022+0000 7ff256b52700 1 --2- 172.21.15.102:0/1479786281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff250101390 0x7ff2501017e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009910 tx=0x7ff23c009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:39.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.022+0000 7ff256b52700 1 -- 172.21.15.102:0/1479786281 shutdown_connections 2024-03-26T18:56:39.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.022+0000 7ff256b52700 1 --2- 172.21.15.102:0/1479786281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff250101390 0x7ff2501017e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.022+0000 7ff256b52700 1 --2- 172.21.15.102:0/1479786281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff250100190 0x7ff2501005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.022+0000 7ff256b52700 1 -- 172.21.15.102:0/1479786281 >> 172.21.15.102:0/1479786281 conn(0x7ff2500fb720 msgr2=0x7ff2500fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff256b52700 1 -- 172.21.15.102:0/1479786281 shutdown_connections 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff256b52700 1 -- 172.21.15.102:0/1479786281 wait complete. 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff256b52700 1 Processor -- start 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff256b52700 1 -- start start 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff256b52700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff250100190 0x7ff250195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff256b52700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff250101390 0x7ff250195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff256b52700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2501965b0 con 0x7ff250101390 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff256b52700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2501966f0 con 0x7ff250100190 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff24ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff250101390 0x7ff250195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff24ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff250101390 0x7ff250195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46904/0 (socket says 172.21.15.102:46904) 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff24ffff700 1 -- 172.21.15.102:0/2364729638 learned_addr learned my addr 172.21.15.102:0/2364729638 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.023+0000 7ff2548ee700 1 --2- 172.21.15.102:0/2364729638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff250100190 0x7ff250195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:39.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.024+0000 7ff24ffff700 1 -- 172.21.15.102:0/2364729638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff250100190 msgr2=0x7ff250195a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:39.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.024+0000 7ff24ffff700 1 --2- 172.21.15.102:0/2364729638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff250100190 0x7ff250195a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.024+0000 7ff24ffff700 1 -- 172.21.15.102:0/2364729638 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c0095f0 con 0x7ff250101390 2024-03-26T18:56:39.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.024+0000 7ff24ffff700 1 --2- 172.21.15.102:0/2364729638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff250101390 0x7ff250195f90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00b5c0 tx=0x7ff23c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:39.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.024+0000 7ff24dffb700 1 -- 172.21.15.102:0/2364729638 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff23c01d070 con 0x7ff250101390 2024-03-26T18:56:39.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.024+0000 7ff24dffb700 1 -- 172.21.15.102:0/2364729638 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff23c022470 con 0x7ff250101390 2024-03-26T18:56:39.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.024+0000 7ff256b52700 1 -- 172.21.15.102:0/2364729638 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25019b140 con 0x7ff250101390 2024-03-26T18:56:39.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.024+0000 7ff24dffb700 1 -- 172.21.15.102:0/2364729638 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff23c00f690 con 0x7ff250101390 2024-03-26T18:56:39.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.024+0000 7ff256b52700 1 -- 172.21.15.102:0/2364729638 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25019b6b0 con 0x7ff250101390 2024-03-26T18:56:39.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.025+0000 7ff256b52700 1 -- 172.21.15.102:0/2364729638 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff250108d80 con 0x7ff250101390 2024-03-26T18:56:39.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.026+0000 7ff24dffb700 1 -- 172.21.15.102:0/2364729638 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 34) v1 ==== 91969+0+0 (secure 0 0 0) 0x7ff23c004b70 con 0x7ff250101390 2024-03-26T18:56:39.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.026+0000 7ff24dffb700 1 --2- 172.21.15.102:0/2364729638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff24006e180 0x7ff240070630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:39.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.026+0000 7ff2548ee700 1 --2- 172.21.15.102:0/2364729638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff24006e180 0x7ff240070630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:39.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.026+0000 7ff24dffb700 1 -- 172.21.15.102:0/2364729638 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff23c08f600 con 0x7ff250101390 2024-03-26T18:56:39.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.027+0000 7ff2548ee700 1 --2- 172.21.15.102:0/2364729638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff24006e180 0x7ff240070630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff244006fd0 tx=0x7ff244008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:39.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.031+0000 7ff24dffb700 1 -- 172.21.15.102:0/2364729638 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff23c058b90 con 0x7ff250101390 2024-03-26T18:56:39.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.219+0000 7ff24dffb700 1 -- 172.21.15.102:0/2364729638 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 35) v1 ==== 92265+0+0 (secure 0 0 0) 0x7ff23c058f60 con 0x7ff250101390 2024-03-26T18:56:39.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.341+0000 7ff256b52700 1 -- 172.21.15.102:0/2364729638 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff25019bd40 con 0x7ff250101390 2024-03-26T18:56:39.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.341+0000 7ff24dffb700 1 -- 172.21.15.102:0/2364729638 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+779 (secure 0 0 0) 0x7ff23c087fc0 con 0x7ff250101390 2024-03-26T18:56:39.334 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:56:39.334 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 1 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:56:39.335 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 11, 2024-03-26T18:56:39.336 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 3 2024-03-26T18:56:39.336 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:56:39.336 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:56:39.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.343+0000 7ff256b52700 1 -- 172.21.15.102:0/2364729638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff24006e180 msgr2=0x7ff240070630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.343+0000 7ff256b52700 1 --2- 172.21.15.102:0/2364729638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff24006e180 0x7ff240070630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff244006fd0 tx=0x7ff244008040 comp rx=0 tx=0).stop 2024-03-26T18:56:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.343+0000 7ff256b52700 1 -- 172.21.15.102:0/2364729638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff250101390 msgr2=0x7ff250195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.343+0000 7ff256b52700 1 --2- 172.21.15.102:0/2364729638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff250101390 0x7ff250195f90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00b5c0 tx=0x7ff23c00be20 comp rx=0 tx=0).stop 2024-03-26T18:56:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.343+0000 7ff256b52700 1 -- 172.21.15.102:0/2364729638 shutdown_connections 2024-03-26T18:56:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.343+0000 7ff256b52700 1 --2- 172.21.15.102:0/2364729638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff250100190 0x7ff250195a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.343+0000 7ff256b52700 1 --2- 172.21.15.102:0/2364729638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff24006e180 0x7ff240070630 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.343+0000 7ff256b52700 1 --2- 172.21.15.102:0/2364729638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff250101390 0x7ff250195f90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.343+0000 7ff256b52700 1 -- 172.21.15.102:0/2364729638 >> 172.21.15.102:0/2364729638 conn(0x7ff2500fb720 msgr2=0x7ff2501045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.344+0000 7ff256b52700 1 -- 172.21.15.102:0/2364729638 shutdown_connections 2024-03-26T18:56:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.344+0000 7ff256b52700 1 -- 172.21.15.102:0/2364729638 wait complete. 2024-03-26T18:56:39.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.449+0000 7f0dbfff3700 1 -- 172.21.15.102:0/346136335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8100250 msgr2=0x7f0db81006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:39.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.449+0000 7f0dbfff3700 1 --2- 172.21.15.102:0/346136335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8100250 0x7f0db81006a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009910 tx=0x7f0da8009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:39.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.449+0000 7f0dbfff3700 1 -- 172.21.15.102:0/346136335 shutdown_connections 2024-03-26T18:56:39.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.449+0000 7f0dbfff3700 1 --2- 172.21.15.102:0/346136335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8100250 0x7f0db81006a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.449+0000 7f0dbfff3700 1 --2- 172.21.15.102:0/346136335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db80ff050 0x7f0db80ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.449+0000 7f0dbfff3700 1 -- 172.21.15.102:0/346136335 >> 172.21.15.102:0/346136335 conn(0x7f0db80fa5c0 msgr2=0x7f0db80fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:39.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.449+0000 7f0dbfff3700 1 -- 172.21.15.102:0/346136335 shutdown_connections 2024-03-26T18:56:39.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.449+0000 7f0dbfff3700 1 -- 172.21.15.102:0/346136335 wait complete. 2024-03-26T18:56:39.444 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:39 smithi102 ceph-mon[78322]: [26/Mar/2024:18:56:37] ENGINE Bus STARTING 2024-03-26T18:56:39.445 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:39 smithi102 ceph-mon[78322]: [26/Mar/2024:18:56:37] ENGINE Serving on http://172.21.15.102:8765 2024-03-26T18:56:39.445 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:39 smithi102 ceph-mon[78322]: [26/Mar/2024:18:56:37] ENGINE Serving on https://172.21.15.102:7150 2024-03-26T18:56:39.445 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:39 smithi102 ceph-mon[78322]: [26/Mar/2024:18:56:37] ENGINE Bus STARTED 2024-03-26T18:56:39.445 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:39 smithi102 ceph-mon[78322]: mgrmap e34: smithi102.rsfpnv(active, since 1.02296s), standbys: smithi190.zrixos 2024-03-26T18:56:39.445 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:39 smithi102 ceph-mon[78322]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:39.445 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:39 smithi102 ceph-mon[78322]: pgmap v3: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:39.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.450+0000 7f0dbfff3700 1 Processor -- start 2024-03-26T18:56:39.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.450+0000 7f0dbfff3700 1 -- start start 2024-03-26T18:56:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.450+0000 7f0dbfff3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db80ff050 0x7f0db8195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.450+0000 7f0dbfff3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8100250 0x7f0db8195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.450+0000 7f0dbfff3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db81964c0 con 0x7f0db8100250 2024-03-26T18:56:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.450+0000 7f0dbfff3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8196600 con 0x7f0db80ff050 2024-03-26T18:56:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.450+0000 7f0dbd58e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8100250 0x7f0db8195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.450+0000 7f0dbd58e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8100250 0x7f0db8195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46936/0 (socket says 172.21.15.102:46936) 2024-03-26T18:56:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.450+0000 7f0dbd58e700 1 -- 172.21.15.102:0/2904704525 learned_addr learned my addr 172.21.15.102:0/2904704525 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.450+0000 7f0dbdd8f700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db80ff050 0x7f0db8195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.451+0000 7f0dbd58e700 1 -- 172.21.15.102:0/2904704525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db80ff050 msgr2=0x7f0db8195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.451+0000 7f0dbd58e700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db80ff050 0x7f0db8195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.451+0000 7f0dbd58e700 1 -- 172.21.15.102:0/2904704525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da80095f0 con 0x7f0db8100250 2024-03-26T18:56:39.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.451+0000 7f0dbdd8f700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db80ff050 0x7f0db8195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T18:56:39.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.451+0000 7f0dbd58e700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8100250 0x7f0db8195ea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0da800b5c0 tx=0x7f0da800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:39.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.451+0000 7f0daeffd700 1 -- 172.21.15.102:0/2904704525 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0da801d070 con 0x7f0db8100250 2024-03-26T18:56:39.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.451+0000 7f0daeffd700 1 -- 172.21.15.102:0/2904704525 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0da8022470 con 0x7f0db8100250 2024-03-26T18:56:39.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.451+0000 7f0dbfff3700 1 -- 172.21.15.102:0/2904704525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db819b050 con 0x7f0db8100250 2024-03-26T18:56:39.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.451+0000 7f0daeffd700 1 -- 172.21.15.102:0/2904704525 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0da800f690 con 0x7f0db8100250 2024-03-26T18:56:39.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.451+0000 7f0dbfff3700 1 -- 172.21.15.102:0/2904704525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db819b540 con 0x7f0db8100250 2024-03-26T18:56:39.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.452+0000 7f0dbfff3700 1 -- 172.21.15.102:0/2904704525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db8107ba0 con 0x7f0db8100250 2024-03-26T18:56:39.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.452+0000 7f0daeffd700 1 -- 172.21.15.102:0/2904704525 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 35) v1 ==== 92265+0+0 (secure 0 0 0) 0x7f0da8004b70 con 0x7f0db8100250 2024-03-26T18:56:39.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.453+0000 7f0daeffd700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0da406e500 0x7f0da40709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:39.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.453+0000 7f0daeffd700 1 -- 172.21.15.102:0/2904704525 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0da808fca0 con 0x7f0db8100250 2024-03-26T18:56:39.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.453+0000 7f0dbdd8f700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0da406e500 0x7f0da40709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:39.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.453+0000 7f0dbdd8f700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0da406e500 0x7f0da40709b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0db81000b0 tx=0x7f0db4005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:39.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.457+0000 7f0daeffd700 1 -- 172.21.15.102:0/2904704525 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0da8094070 con 0x7f0db8100250 2024-03-26T18:56:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:39 smithi190 ceph-mon[35043]: [26/Mar/2024:18:56:37] ENGINE Bus STARTING 2024-03-26T18:56:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:39 smithi190 ceph-mon[35043]: [26/Mar/2024:18:56:37] ENGINE Serving on http://172.21.15.102:8765 2024-03-26T18:56:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:39 smithi190 ceph-mon[35043]: [26/Mar/2024:18:56:37] ENGINE Serving on https://172.21.15.102:7150 2024-03-26T18:56:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:39 smithi190 ceph-mon[35043]: [26/Mar/2024:18:56:37] ENGINE Bus STARTED 2024-03-26T18:56:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:39 smithi190 ceph-mon[35043]: mgrmap e34: smithi102.rsfpnv(active, since 1.02296s), standbys: smithi190.zrixos 2024-03-26T18:56:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:39 smithi190 ceph-mon[35043]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:39 smithi190 ceph-mon[35043]: pgmap v3: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:39.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.711+0000 7f0dbfff3700 1 -- 172.21.15.102:0/2904704525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0db8107d60 con 0x7f0db8100250 2024-03-26T18:56:39.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.711+0000 7f0daeffd700 1 -- 172.21.15.102:0/2904704525 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1924 (secure 0 0 0) 0x7f0da805c940 con 0x7f0db8100250 2024-03-26T18:56:39.705 INFO:teuthology.orchestra.run.smithi102.stdout:e19 2024-03-26T18:56:39.705 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:56:39.705 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:56:39.705 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:56:39.705 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:56:39.705 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:56:39.705 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:56:39.705 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 19 2024-03-26T18:56:39.705 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:56:39.705 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:56:39.705 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:55:27.824831+0000 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:56:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 0 members: 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:56:39.707 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:56:39.708 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:56:39.708 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:39.708 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.714+0000 7f0dbfff3700 1 -- 172.21.15.102:0/2904704525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0da406e500 msgr2=0x7f0da40709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.714+0000 7f0dbfff3700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0da406e500 0x7f0da40709b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0db81000b0 tx=0x7f0db4005f90 comp rx=0 tx=0).stop 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.714+0000 7f0dbfff3700 1 -- 172.21.15.102:0/2904704525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8100250 msgr2=0x7f0db8195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.714+0000 7f0dbfff3700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8100250 0x7f0db8195ea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0da800b5c0 tx=0x7f0da800be20 comp rx=0 tx=0).stop 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.714+0000 7f0dbfff3700 1 -- 172.21.15.102:0/2904704525 shutdown_connections 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.714+0000 7f0dbfff3700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db80ff050 0x7f0db8195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.714+0000 7f0dbfff3700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0da406e500 0x7f0da40709b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.714+0000 7f0dbfff3700 1 --2- 172.21.15.102:0/2904704525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8100250 0x7f0db8195ea0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.714+0000 7f0dbfff3700 1 -- 172.21.15.102:0/2904704525 >> 172.21.15.102:0/2904704525 conn(0x7f0db80fa5c0 msgr2=0x7f0db8068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.714+0000 7f0dbfff3700 1 -- 172.21.15.102:0/2904704525 shutdown_connections 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.714+0000 7f0dbfff3700 1 -- 172.21.15.102:0/2904704525 wait complete. 2024-03-26T18:56:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 19 2024-03-26T18:56:39.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.826+0000 7f8b39b7b700 1 -- 172.21.15.102:0/2895725346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b341005e0 msgr2=0x7f8b341009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:39.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.826+0000 7f8b39b7b700 1 --2- 172.21.15.102:0/2895725346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b341005e0 0x7f8b341009f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8b28009910 tx=0x7f8b28009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:39.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.826+0000 7f8b39b7b700 1 -- 172.21.15.102:0/2895725346 shutdown_connections 2024-03-26T18:56:39.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.826+0000 7f8b39b7b700 1 --2- 172.21.15.102:0/2895725346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b340ff320 0x7f8b340ff770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.826+0000 7f8b39b7b700 1 --2- 172.21.15.102:0/2895725346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b341005e0 0x7f8b341009f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.826+0000 7f8b39b7b700 1 -- 172.21.15.102:0/2895725346 >> 172.21.15.102:0/2895725346 conn(0x7f8b340f9d80 msgr2=0x7f8b340fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:39.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.826+0000 7f8b39b7b700 1 -- 172.21.15.102:0/2895725346 shutdown_connections 2024-03-26T18:56:39.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.826+0000 7f8b39b7b700 1 -- 172.21.15.102:0/2895725346 wait complete. 2024-03-26T18:56:39.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.826+0000 7f8b39b7b700 1 Processor -- start 2024-03-26T18:56:39.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b39b7b700 1 -- start start 2024-03-26T18:56:39.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b39b7b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b340ff320 0x7f8b340fed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:39.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b39b7b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b341005e0 0x7f8b3419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:39.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b39b7b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3419ac70 con 0x7f8b341005e0 2024-03-26T18:56:39.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b39b7b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3419ade0 con 0x7f8b340ff320 2024-03-26T18:56:39.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b32ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b341005e0 0x7f8b3419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:39.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b32ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b341005e0 0x7f8b3419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46952/0 (socket says 172.21.15.102:46952) 2024-03-26T18:56:39.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b32ffd700 1 -- 172.21.15.102:0/3786908778 learned_addr learned my addr 172.21.15.102:0/3786908778 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:39.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b337fe700 1 --2- 172.21.15.102:0/3786908778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b340ff320 0x7f8b340fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:39.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b32ffd700 1 -- 172.21.15.102:0/3786908778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b340ff320 msgr2=0x7f8b340fed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:39.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b32ffd700 1 --2- 172.21.15.102:0/3786908778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b340ff320 0x7f8b340fed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:39.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b32ffd700 1 -- 172.21.15.102:0/3786908778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b280095f0 con 0x7f8b341005e0 2024-03-26T18:56:39.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b32ffd700 1 --2- 172.21.15.102:0/3786908778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b341005e0 0x7f8b3419a6a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400d900 tx=0x7f8b2400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:39.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b30ff9700 1 -- 172.21.15.102:0/3786908778 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b240041f0 con 0x7f8b341005e0 2024-03-26T18:56:39.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.827+0000 7f8b30ff9700 1 -- 172.21.15.102:0/3786908778 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8b24004ce0 con 0x7f8b341005e0 2024-03-26T18:56:39.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.828+0000 7f8b30ff9700 1 -- 172.21.15.102:0/3786908778 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b2400b750 con 0x7f8b341005e0 2024-03-26T18:56:39.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.828+0000 7f8b39b7b700 1 -- 172.21.15.102:0/3786908778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3419b0c0 con 0x7f8b341005e0 2024-03-26T18:56:39.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.828+0000 7f8b39b7b700 1 -- 172.21.15.102:0/3786908778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3419b610 con 0x7f8b341005e0 2024-03-26T18:56:39.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.829+0000 7f8b39b7b700 1 -- 172.21.15.102:0/3786908778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b34107b90 con 0x7f8b341005e0 2024-03-26T18:56:39.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.829+0000 7f8b30ff9700 1 -- 172.21.15.102:0/3786908778 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 35) v1 ==== 92265+0+0 (secure 0 0 0) 0x7f8b24004e50 con 0x7f8b341005e0 2024-03-26T18:56:39.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.829+0000 7f8b30ff9700 1 --2- 172.21.15.102:0/3786908778 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8b1c06e500 0x7f8b1c0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:39.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.829+0000 7f8b30ff9700 1 -- 172.21.15.102:0/3786908778 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8b2408e3b0 con 0x7f8b341005e0 2024-03-26T18:56:39.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.829+0000 7f8b337fe700 1 --2- 172.21.15.102:0/3786908778 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8b1c06e500 0x7f8b1c0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:39.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.830+0000 7f8b337fe700 1 --2- 172.21.15.102:0/3786908778 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8b1c06e500 0x7f8b1c0709b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8b28005090 tx=0x7f8b28005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:39.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:39.834+0000 7f8b30ff9700 1 -- 172.21.15.102:0/3786908778 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8b2405b020 con 0x7f8b341005e0 2024-03-26T18:56:40.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.055+0000 7f8b39b7b700 1 -- 172.21.15.102:0/3786908778 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b34104c80 con 0x7f8b1c06e500 2024-03-26T18:56:40.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.056+0000 7f8b30ff9700 1 -- 172.21.15.102:0/3786908778 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f8b34104c80 con 0x7f8b1c06e500 2024-03-26T18:56:40.049 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:56:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:56:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:56:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:56:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T18:56:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr" 2024-03-26T18:56:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T18:56:40.050 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "2/23 daemons upgraded", 2024-03-26T18:56:40.050 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "", 2024-03-26T18:56:40.050 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:56:40.050 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:56:40.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.058+0000 7f8b39b7b700 1 -- 172.21.15.102:0/3786908778 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8b1c06e500 msgr2=0x7f8b1c0709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:40.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.058+0000 7f8b39b7b700 1 --2- 172.21.15.102:0/3786908778 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8b1c06e500 0x7f8b1c0709b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8b28005090 tx=0x7f8b28005a90 comp rx=0 tx=0).stop 2024-03-26T18:56:40.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.058+0000 7f8b39b7b700 1 -- 172.21.15.102:0/3786908778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b341005e0 msgr2=0x7f8b3419a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:40.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.058+0000 7f8b39b7b700 1 --2- 172.21.15.102:0/3786908778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b341005e0 0x7f8b3419a6a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400d900 tx=0x7f8b2400dc10 comp rx=0 tx=0).stop 2024-03-26T18:56:40.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.058+0000 7f8b39b7b700 1 -- 172.21.15.102:0/3786908778 shutdown_connections 2024-03-26T18:56:40.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.058+0000 7f8b39b7b700 1 --2- 172.21.15.102:0/3786908778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b340ff320 0x7f8b340fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:40.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.058+0000 7f8b39b7b700 1 --2- 172.21.15.102:0/3786908778 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8b1c06e500 0x7f8b1c0709b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:40.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.058+0000 7f8b39b7b700 1 --2- 172.21.15.102:0/3786908778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b341005e0 0x7f8b3419a6a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:40.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.058+0000 7f8b39b7b700 1 -- 172.21.15.102:0/3786908778 >> 172.21.15.102:0/3786908778 conn(0x7f8b340f9d80 msgr2=0x7f8b34103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:40.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.059+0000 7f8b39b7b700 1 -- 172.21.15.102:0/3786908778 shutdown_connections 2024-03-26T18:56:40.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.059+0000 7f8b39b7b700 1 -- 172.21.15.102:0/3786908778 wait complete. 2024-03-26T18:56:40.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.163+0000 7f27a56bb700 1 -- 172.21.15.102:0/3883667875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a0100170 msgr2=0x7f27a0100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:40.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.163+0000 7f27a56bb700 1 --2- 172.21.15.102:0/3883667875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a0100170 0x7f27a0100580 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2790009910 tx=0x7f2790009c20 comp rx=0 tx=0).stop 2024-03-26T18:56:40.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.163+0000 7f27a56bb700 1 -- 172.21.15.102:0/3883667875 shutdown_connections 2024-03-26T18:56:40.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.163+0000 7f27a56bb700 1 --2- 172.21.15.102:0/3883667875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27a0101370 0x7f27a01017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:40.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.163+0000 7f27a56bb700 1 --2- 172.21.15.102:0/3883667875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a0100170 0x7f27a0100580 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:40.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.163+0000 7f27a56bb700 1 -- 172.21.15.102:0/3883667875 >> 172.21.15.102:0/3883667875 conn(0x7f27a00fb6e0 msgr2=0x7f27a00fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:40.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.163+0000 7f27a56bb700 1 -- 172.21.15.102:0/3883667875 shutdown_connections 2024-03-26T18:56:40.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.163+0000 7f27a56bb700 1 -- 172.21.15.102:0/3883667875 wait complete. 2024-03-26T18:56:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.163+0000 7f27a56bb700 1 Processor -- start 2024-03-26T18:56:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f27a56bb700 1 -- start start 2024-03-26T18:56:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f27a56bb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a0100170 0x7f27a0195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f27a56bb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27a0101370 0x7f27a0195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f27a56bb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a01964f0 con 0x7f27a0100170 2024-03-26T18:56:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f27a56bb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a0196630 con 0x7f27a0101370 2024-03-26T18:56:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f279effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a0100170 0x7f27a0195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f279effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a0100170 0x7f27a0195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46970/0 (socket says 172.21.15.102:46970) 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f279effd700 1 -- 172.21.15.102:0/2073296130 learned_addr learned my addr 172.21.15.102:0/2073296130 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f279e7fc700 1 --2- 172.21.15.102:0/2073296130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27a0101370 0x7f27a0195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f279effd700 1 -- 172.21.15.102:0/2073296130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27a0101370 msgr2=0x7f27a0195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f279effd700 1 --2- 172.21.15.102:0/2073296130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27a0101370 0x7f27a0195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f279effd700 1 -- 172.21.15.102:0/2073296130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27900095f0 con 0x7f27a0100170 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.164+0000 7f279effd700 1 --2- 172.21.15.102:0/2073296130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a0100170 0x7f27a0195990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f279000bda0 tx=0x7f279000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.165+0000 7f2787fff700 1 -- 172.21.15.102:0/2073296130 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f279001d070 con 0x7f27a0100170 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.165+0000 7f27a56bb700 1 -- 172.21.15.102:0/2073296130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a019b080 con 0x7f27a0100170 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.165+0000 7f2787fff700 1 -- 172.21.15.102:0/2073296130 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f279000f460 con 0x7f27a0100170 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.165+0000 7f2787fff700 1 -- 172.21.15.102:0/2073296130 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2790021620 con 0x7f27a0100170 2024-03-26T18:56:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.165+0000 7f27a56bb700 1 -- 172.21.15.102:0/2073296130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a019b570 con 0x7f27a0100170 2024-03-26T18:56:40.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.166+0000 7f27a56bb700 1 -- 172.21.15.102:0/2073296130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a0108cc0 con 0x7f27a0100170 2024-03-26T18:56:40.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.166+0000 7f2787fff700 1 -- 172.21.15.102:0/2073296130 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 35) v1 ==== 92265+0+0 (secure 0 0 0) 0x7f279000f5d0 con 0x7f27a0100170 2024-03-26T18:56:40.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.166+0000 7f2787fff700 1 --2- 172.21.15.102:0/2073296130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f278806e500 0x7f27880709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:56:40.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.166+0000 7f2787fff700 1 -- 172.21.15.102:0/2073296130 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f279008f860 con 0x7f27a0100170 2024-03-26T18:56:40.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.166+0000 7f279e7fc700 1 --2- 172.21.15.102:0/2073296130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f278806e500 0x7f27880709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:56:40.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.167+0000 7f279e7fc700 1 --2- 172.21.15.102:0/2073296130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f278806e500 0x7f27880709b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2794005950 tx=0x7f279400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:56:40.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.171+0000 7f2787fff700 1 -- 172.21.15.102:0/2073296130 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f279005c450 con 0x7f27a0100170 2024-03-26T18:56:40.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:40 smithi102 ceph-mon[78322]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:40.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:40 smithi102 ceph-mon[78322]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:40.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:40 smithi102 ceph-mon[78322]: pgmap v4: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:40.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:40 smithi102 ceph-mon[78322]: mgrmap e35: smithi102.rsfpnv(active, since 2s), standbys: smithi190.zrixos 2024-03-26T18:56:40.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:40 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2364729638' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:56:40.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:40 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2904704525' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:56:40.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.474+0000 7f27a56bb700 1 -- 172.21.15.102:0/2073296130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f27a0108e80 con 0x7f27a0100170 2024-03-26T18:56:40.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.474+0000 7f2787fff700 1 -- 172.21.15.102:0/2073296130 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f279005bfe0 con 0x7f27a0100170 2024-03-26T18:56:40.467 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T18:56:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.476+0000 7f27a56bb700 1 -- 172.21.15.102:0/2073296130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f278806e500 msgr2=0x7f27880709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.476+0000 7f27a56bb700 1 --2- 172.21.15.102:0/2073296130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f278806e500 0x7f27880709b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2794005950 tx=0x7f279400b410 comp rx=0 tx=0).stop 2024-03-26T18:56:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.476+0000 7f27a56bb700 1 -- 172.21.15.102:0/2073296130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a0100170 msgr2=0x7f27a0195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:56:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.476+0000 7f27a56bb700 1 --2- 172.21.15.102:0/2073296130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a0100170 0x7f27a0195990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f279000bda0 tx=0x7f279000bdd0 comp rx=0 tx=0).stop 2024-03-26T18:56:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.476+0000 7f27a56bb700 1 -- 172.21.15.102:0/2073296130 shutdown_connections 2024-03-26T18:56:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.476+0000 7f27a56bb700 1 --2- 172.21.15.102:0/2073296130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f278806e500 0x7f27880709b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.476+0000 7f27a56bb700 1 --2- 172.21.15.102:0/2073296130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a0100170 0x7f27a0195990 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.476+0000 7f27a56bb700 1 --2- 172.21.15.102:0/2073296130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27a0101370 0x7f27a0195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:56:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.476+0000 7f27a56bb700 1 -- 172.21.15.102:0/2073296130 >> 172.21.15.102:0/2073296130 conn(0x7f27a00fb6e0 msgr2=0x7f27a01045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:56:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.477+0000 7f27a56bb700 1 -- 172.21.15.102:0/2073296130 shutdown_connections 2024-03-26T18:56:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:56:40.477+0000 7f27a56bb700 1 -- 172.21.15.102:0/2073296130 wait complete. 2024-03-26T18:56:40.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:40 smithi190 ceph-mon[35043]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:40.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:40 smithi190 ceph-mon[35043]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:40.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:40 smithi190 ceph-mon[35043]: pgmap v4: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:40.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:40 smithi190 ceph-mon[35043]: mgrmap e35: smithi102.rsfpnv(active, since 2s), standbys: smithi190.zrixos 2024-03-26T18:56:40.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:40 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2364729638' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:56:40.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:40 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2904704525' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:56:41.480 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:41 smithi190 ceph-mon[35043]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:41.480 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:41 smithi190 ceph-mon[35043]: from='client.? 172.21.15.102:0/2073296130' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:56:41.480 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:41 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:41.481 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:41 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:41.512 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:41 smithi102 ceph-mon[78322]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:56:41.512 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:41 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2073296130' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:56:41.513 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:41.513 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:42.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:42 smithi102 ceph-mon[78322]: pgmap v5: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:42.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:42 smithi102 ceph-mon[78322]: mgrmap e36: smithi102.rsfpnv(active, since 4s), standbys: smithi190.zrixos 2024-03-26T18:56:42.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:42 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:42.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:42 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:42.475 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:42 smithi190 ceph-mon[35043]: pgmap v5: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:42.475 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:42 smithi190 ceph-mon[35043]: mgrmap e36: smithi102.rsfpnv(active, since 4s), standbys: smithi190.zrixos 2024-03-26T18:56:42.475 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:42 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:42.475 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:42 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:43.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:43.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:43.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:43 smithi102 ceph-mon[78322]: pgmap v6: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:44.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:43 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:44.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:43 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:44.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:43 smithi190 ceph-mon[35043]: pgmap v6: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:44.840 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:44 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:44.841 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:44 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:45.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:44 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:45.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:44 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:46.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:45 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:46.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:45 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:46.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:45 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:46.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:45 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:56:46.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:45 smithi190 ceph-mon[35043]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-26T18:56:46.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:45 smithi190 ceph-mon[35043]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-26T18:56:46.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:45 smithi190 ceph-mon[35043]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:56:46.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:45 smithi190 ceph-mon[35043]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:56:46.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:45 smithi190 ceph-mon[35043]: pgmap v7: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T18:56:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:56:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:45 smithi102 ceph-mon[78322]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-26T18:56:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:45 smithi102 ceph-mon[78322]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-26T18:56:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:45 smithi102 ceph-mon[78322]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:56:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:45 smithi102 ceph-mon[78322]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T18:56:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:45 smithi102 ceph-mon[78322]: pgmap v7: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T18:56:47.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:56:47.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:56:47.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:56:47.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:56:47.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:47.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:47.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:47.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:47.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:47.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:47.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:56:47.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T18:56:47.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:56:47.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-26T18:56:47.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: Updating smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:56:47.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: Updating smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T18:56:47.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:47.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:47.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:47.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:47.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:47.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:47.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:56:47.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:46 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T18:56:48.254 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:47 smithi190 ceph-mon[35043]: pgmap v8: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T18:56:48.254 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:47 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:48.254 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:47 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:56:48.254 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:47 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:56:48.254 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:47 smithi190 ceph-mon[35043]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:48.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:47 smithi102 ceph-mon[78322]: pgmap v8: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T18:56:48.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:47 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:48.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:47 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:56:48.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:47 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:56:48.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:47 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:56:49.277 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:48 smithi102 ceph-mon[78322]: Upgrade: Updating mon.smithi190 2024-03-26T18:56:49.277 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:48 smithi102 ceph-mon[78322]: Deploying daemon mon.smithi190 on smithi190 2024-03-26T18:56:49.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:48 smithi190 ceph-mon[35043]: Upgrade: Updating mon.smithi190 2024-03-26T18:56:49.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:48 smithi190 ceph-mon[35043]: Deploying daemon mon.smithi190 on smithi190 2024-03-26T18:56:49.987 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:49 smithi190 systemd[1]: Stopping Ceph mon.smithi190 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:56:49.988 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:49 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190[35023]: 2024-03-26T18:56:49.809+0000 7f108f0d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi190 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T18:56:49.988 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:49 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190[35023]: 2024-03-26T18:56:49.809+0000 7f108f0d6700 -1 mon.smithi190@1(peon) e2 *** Got Signal Terminated *** 2024-03-26T18:56:49.988 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:49 smithi190 podman[65525]: 2024-03-26 18:56:49.99454615 +0000 UTC m=+0.263862128 container died 6bd913d483f03a942f79191d50776785fe1f592fdd259df94b917eb70b8ef0fc (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190, CEPH_POINT_RELEASE=-18.2.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T18:56:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:50 smithi190 podman[65525]: 2024-03-26 18:56:50.335368738 +0000 UTC m=+0.604684721 container cleanup 6bd913d483f03a942f79191d50776785fe1f592fdd259df94b917eb70b8ef0fc (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) 2024-03-26T18:56:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:50 smithi190 bash[65525]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190 2024-03-26T18:56:50.952 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:50 smithi190 podman[65539]: 2024-03-26 18:56:50.545125168 +0000 UTC m=+0.548019277 container remove 6bd913d483f03a942f79191d50776785fe1f592fdd259df94b917eb70b8ef0fc (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.29.1) 2024-03-26T18:56:50.952 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:50 smithi190 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi190.service: Deactivated successfully. 2024-03-26T18:56:50.952 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:50 smithi190 systemd[1]: Stopped Ceph mon.smithi190 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:56:50.952 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:50 smithi190 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi190.service: Consumed 6.364s CPU time. 2024-03-26T18:56:51.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:50 smithi190 systemd[1]: Starting Ceph mon.smithi190 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:56:51.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 podman[65670]: 2024-03-26 18:56:51.077442301 +0000 UTC m=+0.017926108 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:56:51.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 podman[65670]: 2024-03-26 18:56:51.187756756 +0000 UTC m=+0.128240631 container create e80478bd14bd74e2ab2c34ae28dac57fbf5ab80749d3fb0d1108f2aefc8d69cf (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T18:56:51.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 podman[65670]: 2024-03-26 18:56:51.41560575 +0000 UTC m=+0.356089556 container init e80478bd14bd74e2ab2c34ae28dac57fbf5ab80749d3fb0d1108f2aefc8d69cf (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T18:56:51.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 podman[65670]: 2024-03-26 18:56:51.418854221 +0000 UTC m=+0.359338032 container start e80478bd14bd74e2ab2c34ae28dac57fbf5ab80749d3fb0d1108f2aefc8d69cf (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T18:56:51.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T18:56:51.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev), process ceph-mon, pid 2 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: pidfile_write: ignore empty --pid-file 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: load: jerasure load: lrc 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: RocksDB version: 7.9.2 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Git sha 0 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Compile date 2024-03-20 16:36:06 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: DB SUMMARY 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: DB Session ID: F7NCMQI2YBS8JA4TTC3T 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: CURRENT file: CURRENT 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1395 Bytes 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi190/store.db dir, Total Num: 1, files: 000027.sst 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi190/store.db: 000025.log size: 3740764 ; 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.error_if_exists: 0 2024-03-26T18:56:51.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.create_if_missing: 0 2024-03-26T18:56:51.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.paranoid_checks: 1 2024-03-26T18:56:51.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T18:56:51.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T18:56:51.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T18:56:51.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.env: 0x559565df1c60 2024-03-26T18:56:51.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T18:56:51.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.info_log: 0x5595674df9a0 2024-03-26T18:56:51.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.statistics: (nil) 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.use_fsync: 0 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_log_file_size: 0 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.allow_fallocate: 1 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.use_direct_reads: 0 2024-03-26T18:56:51.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.db_log_dir: 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.wal_dir: 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.write_buffer_manager: 0x5595674b21e0 2024-03-26T18:56:51.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.unordered_write: 0 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.row_cache: None 2024-03-26T18:56:51.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.wal_filter: None 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.two_write_queues: 0 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.wal_compression: 0 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.atomic_flush: 0 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.log_readahead_size: 0 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T18:56:51.787 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_background_jobs: 2 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_background_compactions: -1 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_subcompactions: 1 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T18:56:51.788 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_open_files: -1 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_background_flushes: -1 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Compression algorithms supported: 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: kZSTD supported: 0 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: kXpressCompression supported: 0 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: kZlibCompression supported: 1 2024-03-26T18:56:51.789 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: kSnappyCompression supported: 1 2024-03-26T18:56:51.790 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: kLZ4Compression supported: 1 2024-03-26T18:56:51.790 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: kBZip2Compression supported: 0 2024-03-26T18:56:51.790 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T18:56:51.790 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T18:56:51.790 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi190/store.db/MANIFEST-000010 2024-03-26T18:56:51.790 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T18:56:51.790 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T18:56:51.790 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.merge_operator: 2024-03-26T18:56:51.790 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_filter: None 2024-03-26T18:56:51.790 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T18:56:51.790 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5595674dfaa0) 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: cache_index_and_filter_blocks: 1 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: pin_top_level_index_and_filter: 1 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: index_type: 0 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: data_block_index_type: 0 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: index_shortening: 1 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: checksum: 4 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: no_block_cache: 0 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: block_cache: 0x5595674e5090 2024-03-26T18:56:51.791 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: block_cache_name: BinnedLRUCache 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: block_cache_options: 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: capacity : 536870912 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: num_shard_bits : 4 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: strict_capacity_limit : 0 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: high_pri_pool_ratio: 0.000 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: block_cache_compressed: (nil) 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: persistent_cache: (nil) 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: block_size: 4096 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: block_size_deviation: 10 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: block_restart_interval: 16 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: index_block_restart_interval: 1 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: metadata_block_size: 4096 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: partition_filters: 0 2024-03-26T18:56:51.792 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: use_delta_encoding: 1 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: filter_policy: bloomfilter 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: whole_key_filtering: 1 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: verify_compression: 0 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: read_amp_bytes_per_bit: 0 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: format_version: 5 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: enable_index_compression: 1 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: block_align: 0 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: max_auto_readahead_size: 262144 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: prepopulate_block_cache: 0 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: initial_auto_readahead_size: 8192 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T18:56:51.793 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compression: NoCompression 2024-03-26T18:56:51.794 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T18:56:51.794 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T18:56:51.794 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T18:56:51.794 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.num_levels: 7 2024-03-26T18:56:51.794 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T18:56:51.794 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T18:56:51.794 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T18:56:51.795 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T18:56:51.795 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T18:56:51.795 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T18:56:51.795 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T18:56:51.795 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T18:56:51.795 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T18:56:51.795 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T18:56:51.795 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T18:56:51.795 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T18:56:51.795 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T18:56:51.795 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T18:56:51.796 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T18:56:51.797 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.table_properties_collectors: 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.inplace_update_support: 0 2024-03-26T18:56:51.798 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.bloom_locality: 0 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.max_successive_merges: 0 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.ttl: 2592000 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.enable_blob_files: false 2024-03-26T18:56:51.799 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.min_blob_size: 0 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi190/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 12184, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 35bd36bf-1dff-43e6-bab5-598f82404b48 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711479411448390, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-26T18:56:51.800 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711479411456802, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 2312825, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12189, "largest_seqno": 13124, "table_properties": {"data_size": 2308144, "index_size": 2575, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1093, "raw_key_size": 10092, "raw_average_key_size": 24, "raw_value_size": 2299150, "raw_average_value_size": 5526, "num_data_blocks": 117, "num_entries": 416, "num_filter_entries": 416, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711479411, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "35bd36bf-1dff-43e6-bab5-598f82404b48", "db_session_id": "F7NCMQI2YBS8JA4TTC3T", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711479411456908, "job": 1, "event": "recovery_finished"} 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi190/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5595675d8000 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: DB pointer 0x5595675c2000 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: ** DB Stats ** 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T18:56:51.801 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: ** Compaction Stats [default] ** 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: L0 1/0 2.21 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 625.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: L6 1/0 5.90 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Sum 2/0 8.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 625.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 625.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: ** Compaction Stats [default] ** 2024-03-26T18:56:51.802 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T18:56:51.803 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T18:56:51.803 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 625.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-26T18:56:51.803 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: 2024-03-26T18:56:51.803 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T18:56:51.803 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: 2024-03-26T18:56:51.803 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T18:56:51.803 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-26T18:56:51.803 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T18:56:51.803 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T18:56:51.803 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T18:56:51.804 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T18:56:51.804 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Cumulative compaction: 0.00 GB write, 178.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T18:56:51.804 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Interval compaction: 0.00 GB write, 178.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T18:56:51.804 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T18:56:51.804 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Block cache BinnedLRUCache@0x5595674e5090#2 capacity: 512.00 MB usage: 3.98 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-26T18:56:51.804 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.16 KB,0.000220537%) IndexBlock(1,2.83 KB,0.000539422%) Misc(1,0.00 KB,0%) 2024-03-26T18:56:51.804 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: 2024-03-26T18:56:51.804 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T18:56:51.804 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: starting mon.smithi190 rank 1 at public addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] at bind addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi190 fsid ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:56:51.804 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: mon.smithi190@-1(???) e2 preinit fsid ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: mon.smithi190@-1(???).mds e19 new map 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: mon.smithi190@-1(???).mds e19 print_map 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: e19 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: legacy client fscid: 1 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Filesystem 'cephfs' (1) 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: fs_name cephfs 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: epoch 19 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: modified 2024-03-26T18:55:27.824831+0000 2024-03-26T18:56:51.805 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: tableserver 0 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: root 0 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: session_timeout 60 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: session_autoclose 300 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: max_file_size 1099511627776 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: max_xattr_size 65536 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: required_client_features {} 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: last_failure 0 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: last_failure_osd_epoch 47 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: max_mds 2 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: in 0,1 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: up {0=14490,1=24283} 2024-03-26T18:56:51.806 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: failed 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: damaged 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: stopped 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: data_pools [3] 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: metadata_pool 2 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: inline_data disabled 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: balancer 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: bal_rank_mask -1 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: standby_count_wanted 1 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: qdb_cluster leader: 0 members: 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: [mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: [mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: 2024-03-26T18:56:51.807 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: Standby daemons: 2024-03-26T18:56:51.808 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: 2024-03-26T18:56:51.808 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: [mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:51.808 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout: [mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:56:51.808 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: mon.smithi190@-1(???).osd e50 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-26T18:56:51.808 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: mon.smithi190@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-03-26T18:56:51.808 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: mon.smithi190@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-03-26T18:56:51.808 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: mon.smithi190@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-03-26T18:56:51.808 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 ceph-mon[65689]: mon.smithi190@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-03-26T18:56:51.808 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 bash[65670]: e80478bd14bd74e2ab2c34ae28dac57fbf5ab80749d3fb0d1108f2aefc8d69cf 2024-03-26T18:56:51.808 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:51 smithi190 systemd[1]: Started Ceph mon.smithi190 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:56:54.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: overall HEALTH_OK 2024-03-26T18:56:54.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:56:54.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:56:54.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: mon.smithi102 calling monitor election 2024-03-26T18:56:54.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: mon.smithi190 calling monitor election 2024-03-26T18:56:54.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: mon.smithi102 is new leader, mons smithi102,smithi190 in quorum (ranks 0,1) 2024-03-26T18:56:54.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: monmap epoch 3 2024-03-26T18:56:54.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: fsid ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: last_changed 2024-03-26T18:56:53.022008+0000 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: created 2024-03-26T18:45:22.493426+0000 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: min_mon_release 19 (squid) 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: election_strategy: 1 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.smithi102 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.smithi190 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: osdmap e50: 6 total, 6 up, 6 in 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: mgrmap e36: smithi102.rsfpnv(active, since 15s), standbys: smithi190.zrixos 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: overall HEALTH_OK 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:54.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:54 smithi190 ceph-mon[65689]: pgmap v11: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T18:56:54.331 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: overall HEALTH_OK 2024-03-26T18:56:54.331 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-03-26T18:56:54.331 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-03-26T18:56:54.331 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: mon.smithi102 calling monitor election 2024-03-26T18:56:54.331 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: mon.smithi190 calling monitor election 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: mon.smithi102 is new leader, mons smithi102,smithi190 in quorum (ranks 0,1) 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: monmap epoch 3 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: fsid ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: last_changed 2024-03-26T18:56:53.022008+0000 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: created 2024-03-26T18:45:22.493426+0000 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: min_mon_release 19 (squid) 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: election_strategy: 1 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.smithi102 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.smithi190 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: osdmap e50: 6 total, 6 up, 6 in 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: mgrmap e36: smithi102.rsfpnv(active, since 15s), standbys: smithi190.zrixos 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: overall HEALTH_OK 2024-03-26T18:56:54.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:54.333 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:54.333 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:56:54.333 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:54 smithi102 ceph-mon[78322]: pgmap v11: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T18:56:56.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:56 smithi190 ceph-mon[65689]: pgmap v12: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T18:56:56.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:56 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:56.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:56 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:56.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:56 smithi102 ceph-mon[78322]: pgmap v12: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T18:56:56.703 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:56 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:56.704 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:56 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:58.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:58 smithi190 ceph-mon[65689]: pgmap v13: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:58.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:58 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:58.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:56:58 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:58 smithi102 ceph-mon[78322]: pgmap v13: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:56:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:58 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:56:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:56:58 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:00.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:00 smithi190 ceph-mon[65689]: pgmap v14: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:00.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:00 smithi102 ceph-mon[78322]: pgmap v14: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:01.538 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:01.538 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:01.538 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:01.538 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:57:01.539 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:01.539 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:01 smithi102 ceph-mon[78322]: Reconfiguring mon.smithi102 (monmap changed)... 2024-03-26T18:57:01.539 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:57:01.539 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:57:01.539 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:01.539 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:01 smithi102 ceph-mon[78322]: Reconfiguring daemon mon.smithi102 on smithi102 2024-03-26T18:57:01.539 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:01 smithi102 ceph-mon[78322]: pgmap v15: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:01.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:01.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:01.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:01.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:57:01.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:01.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:01 smithi190 ceph-mon[65689]: Reconfiguring mon.smithi102 (monmap changed)... 2024-03-26T18:57:01.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:57:01.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:57:01.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:01.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:01 smithi190 ceph-mon[65689]: Reconfiguring daemon mon.smithi102 on smithi102 2024-03-26T18:57:01.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:01 smithi190 ceph-mon[65689]: pgmap v15: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:03.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:03.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:03.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:03 smithi102 ceph-mon[78322]: Reconfiguring mgr.smithi102.rsfpnv (monmap changed)... 2024-03-26T18:57:03.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.rsfpnv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:57:03.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:57:03.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:03.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:03 smithi102 ceph-mon[78322]: Reconfiguring daemon mgr.smithi102.rsfpnv on smithi102 2024-03-26T18:57:03.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:03.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:03.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:03 smithi190 ceph-mon[65689]: Reconfiguring mgr.smithi102.rsfpnv (monmap changed)... 2024-03-26T18:57:03.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.rsfpnv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:57:03.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:57:03.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:03.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:03 smithi190 ceph-mon[65689]: Reconfiguring daemon mgr.smithi102.rsfpnv on smithi102 2024-03-26T18:57:04.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:04 smithi102 ceph-mon[78322]: pgmap v16: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:04.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:04.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:04.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:57:04.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:57:04.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-26T18:57:04.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi102"}]: dispatch 2024-03-26T18:57:04.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:04.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:04 smithi190 ceph-mon[65689]: pgmap v16: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:04.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:04.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:04.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:57:04.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:57:04.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-26T18:57:04.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi102"}]: dispatch 2024-03-26T18:57:04.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:05.434 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:05 smithi102 ceph-mon[78322]: Reconfiguring ceph-exporter.smithi102 (monmap changed)... 2024-03-26T18:57:05.434 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:05 smithi102 ceph-mon[78322]: Unable to update caps for client.ceph-exporter.smithi102 2024-03-26T18:57:05.434 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:05 smithi102 ceph-mon[78322]: Reconfiguring daemon ceph-exporter.smithi102 on smithi102 2024-03-26T18:57:05.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:05 smithi190 ceph-mon[65689]: Reconfiguring ceph-exporter.smithi102 (monmap changed)... 2024-03-26T18:57:05.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:05 smithi190 ceph-mon[65689]: Unable to update caps for client.ceph-exporter.smithi102 2024-03-26T18:57:05.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:05 smithi190 ceph-mon[65689]: Reconfiguring daemon ceph-exporter.smithi102 on smithi102 2024-03-26T18:57:06.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:06 smithi102 ceph-mon[78322]: pgmap v17: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:06.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:06 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:06.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:06 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:06.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:06 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:57:06.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:06 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:06.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:06 smithi190 ceph-mon[65689]: pgmap v17: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:06.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:06 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:06.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:06 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:06.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:06 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:57:06.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:06 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:07.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:07 smithi190 ceph-mon[65689]: Reconfiguring crash.smithi102 (monmap changed)... 2024-03-26T18:57:07.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:07 smithi190 ceph-mon[65689]: Reconfiguring daemon crash.smithi102 on smithi102 2024-03-26T18:57:07.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:07 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:07.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:07 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:07.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:07 smithi102 ceph-mon[78322]: Reconfiguring crash.smithi102 (monmap changed)... 2024-03-26T18:57:07.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:07 smithi102 ceph-mon[78322]: Reconfiguring daemon crash.smithi102 on smithi102 2024-03-26T18:57:07.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:07 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:07.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:07 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:08.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:08 smithi102 ceph-mon[78322]: pgmap v18: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:08.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:08 smithi102 ceph-mon[78322]: Reconfiguring osd.0 (monmap changed)... 2024-03-26T18:57:08.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T18:57:08.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:08.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:08 smithi102 ceph-mon[78322]: Reconfiguring daemon osd.0 on smithi102 2024-03-26T18:57:08.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:57:08.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:08 smithi190 ceph-mon[65689]: pgmap v18: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:08.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:08 smithi190 ceph-mon[65689]: Reconfiguring osd.0 (monmap changed)... 2024-03-26T18:57:08.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T18:57:08.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:08.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:08 smithi190 ceph-mon[65689]: Reconfiguring daemon osd.0 on smithi102 2024-03-26T18:57:08.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:57:10.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:09 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:10.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:09 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:10.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:09 smithi102 ceph-mon[78322]: Reconfiguring osd.1 (monmap changed)... 2024-03-26T18:57:10.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:09 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T18:57:10.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:09 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:10.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:09 smithi102 ceph-mon[78322]: Reconfiguring daemon osd.1 on smithi102 2024-03-26T18:57:10.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:09 smithi102 ceph-mon[78322]: pgmap v19: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:10.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:09 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:10.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:09 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:10.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:09 smithi190 ceph-mon[65689]: Reconfiguring osd.1 (monmap changed)... 2024-03-26T18:57:10.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:09 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T18:57:10.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:09 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:10.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:09 smithi190 ceph-mon[65689]: Reconfiguring daemon osd.1 on smithi102 2024-03-26T18:57:10.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:09 smithi190 ceph-mon[65689]: pgmap v19: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.594+0000 7f6d2482c700 1 -- 172.21.15.102:0/703123678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d20101370 msgr2=0x7f6d201017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.594+0000 7f6d2482c700 1 --2- 172.21.15.102:0/703123678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d20101370 0x7f6d201017c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6d14009910 tx=0x7f6d14009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.594+0000 7f6d2482c700 1 -- 172.21.15.102:0/703123678 shutdown_connections 2024-03-26T18:57:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.594+0000 7f6d2482c700 1 --2- 172.21.15.102:0/703123678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d20101370 0x7f6d201017c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.594+0000 7f6d2482c700 1 --2- 172.21.15.102:0/703123678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d20100170 0x7f6d20100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.594+0000 7f6d2482c700 1 -- 172.21.15.102:0/703123678 >> 172.21.15.102:0/703123678 conn(0x7f6d200fb6e0 msgr2=0x7f6d200fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.594+0000 7f6d2482c700 1 -- 172.21.15.102:0/703123678 shutdown_connections 2024-03-26T18:57:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.594+0000 7f6d2482c700 1 -- 172.21.15.102:0/703123678 wait complete. 2024-03-26T18:57:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.594+0000 7f6d2482c700 1 Processor -- start 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d2482c700 1 -- start start 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d2482c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d20100170 0x7f6d20195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d2482c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d20101370 0x7f6d20195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d2482c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d201964e0 con 0x7f6d20100170 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d2482c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d20196620 con 0x7f6d20101370 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d1e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d20100170 0x7f6d20195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d1e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d20100170 0x7f6d20195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57254/0 (socket says 172.21.15.102:57254) 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d1e59c700 1 -- 172.21.15.102:0/982900909 learned_addr learned my addr 172.21.15.102:0/982900909 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d1dd9b700 1 --2- 172.21.15.102:0/982900909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d20101370 0x7f6d20195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d1e59c700 1 -- 172.21.15.102:0/982900909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d20101370 msgr2=0x7f6d20195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d1e59c700 1 --2- 172.21.15.102:0/982900909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d20101370 0x7f6d20195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d1e59c700 1 -- 172.21.15.102:0/982900909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d140095f0 con 0x7f6d20100170 2024-03-26T18:57:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d1e59c700 1 --2- 172.21.15.102:0/982900909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d20100170 0x7f6d20195980 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000d900 tx=0x7f6d1000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d0f7fe700 1 -- 172.21.15.102:0/982900909 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d100041f0 con 0x7f6d20100170 2024-03-26T18:57:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.595+0000 7f6d0f7fe700 1 -- 172.21.15.102:0/982900909 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6d10004350 con 0x7f6d20100170 2024-03-26T18:57:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.596+0000 7f6d2482c700 1 -- 172.21.15.102:0/982900909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2019b0d0 con 0x7f6d20100170 2024-03-26T18:57:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.596+0000 7f6d0f7fe700 1 -- 172.21.15.102:0/982900909 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d10003d20 con 0x7f6d20100170 2024-03-26T18:57:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.596+0000 7f6d2482c700 1 -- 172.21.15.102:0/982900909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2019b620 con 0x7f6d20100170 2024-03-26T18:57:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.597+0000 7f6d2482c700 1 -- 172.21.15.102:0/982900909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d20108cc0 con 0x7f6d20100170 2024-03-26T18:57:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.597+0000 7f6d0f7fe700 1 -- 172.21.15.102:0/982900909 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f6d10010460 con 0x7f6d20100170 2024-03-26T18:57:10.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.597+0000 7f6d0f7fe700 1 --2- 172.21.15.102:0/982900909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6d0806e5a0 0x7f6d08070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:10.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.597+0000 7f6d0f7fe700 1 -- 172.21.15.102:0/982900909 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6d1008e440 con 0x7f6d20100170 2024-03-26T18:57:10.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.597+0000 7f6d1dd9b700 1 --2- 172.21.15.102:0/982900909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6d0806e5a0 0x7f6d08070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:10.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.598+0000 7f6d1dd9b700 1 --2- 172.21.15.102:0/982900909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6d0806e5a0 0x7f6d08070a50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006010 tx=0x7f6d1400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:10.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.602+0000 7f6d0f7fe700 1 -- 172.21.15.102:0/982900909 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6d1005afd0 con 0x7f6d20100170 2024-03-26T18:57:10.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.829+0000 7f6d2482c700 1 -- 172.21.15.102:0/982900909 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d20105cc0 con 0x7f6d0806e5a0 2024-03-26T18:57:10.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.831+0000 7f6d0f7fe700 1 -- 172.21.15.102:0/982900909 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f6d20105cc0 con 0x7f6d0806e5a0 2024-03-26T18:57:10.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.834+0000 7f6d2482c700 1 -- 172.21.15.102:0/982900909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6d0806e5a0 msgr2=0x7f6d08070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.834+0000 7f6d2482c700 1 --2- 172.21.15.102:0/982900909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6d0806e5a0 0x7f6d08070a50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006010 tx=0x7f6d1400b540 comp rx=0 tx=0).stop 2024-03-26T18:57:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.834+0000 7f6d2482c700 1 -- 172.21.15.102:0/982900909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d20100170 msgr2=0x7f6d20195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.834+0000 7f6d2482c700 1 --2- 172.21.15.102:0/982900909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d20100170 0x7f6d20195980 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000d900 tx=0x7f6d1000dc10 comp rx=0 tx=0).stop 2024-03-26T18:57:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.834+0000 7f6d2482c700 1 -- 172.21.15.102:0/982900909 shutdown_connections 2024-03-26T18:57:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.834+0000 7f6d2482c700 1 --2- 172.21.15.102:0/982900909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6d0806e5a0 0x7f6d08070a50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.834+0000 7f6d2482c700 1 --2- 172.21.15.102:0/982900909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d20100170 0x7f6d20195980 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.834+0000 7f6d2482c700 1 --2- 172.21.15.102:0/982900909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d20101370 0x7f6d20195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.834+0000 7f6d2482c700 1 -- 172.21.15.102:0/982900909 >> 172.21.15.102:0/982900909 conn(0x7f6d200fb6e0 msgr2=0x7f6d201045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.835+0000 7f6d2482c700 1 -- 172.21.15.102:0/982900909 shutdown_connections 2024-03-26T18:57:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.835+0000 7f6d2482c700 1 -- 172.21.15.102:0/982900909 wait complete. 2024-03-26T18:57:10.838 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:57:10.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.944+0000 7efea2470700 1 -- 172.21.15.102:0/3114913270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe9c0ff050 msgr2=0x7efe9c0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:10.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.944+0000 7efea2470700 1 --2- 172.21.15.102:0/3114913270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe9c0ff050 0x7efe9c0ff460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efe90009910 tx=0x7efe90009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:10.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.944+0000 7efea2470700 1 -- 172.21.15.102:0/3114913270 shutdown_connections 2024-03-26T18:57:10.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.944+0000 7efea2470700 1 --2- 172.21.15.102:0/3114913270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe9c100250 0x7efe9c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:10.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.944+0000 7efea2470700 1 --2- 172.21.15.102:0/3114913270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe9c0ff050 0x7efe9c0ff460 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:10.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.944+0000 7efea2470700 1 -- 172.21.15.102:0/3114913270 >> 172.21.15.102:0/3114913270 conn(0x7efe9c0fa5e0 msgr2=0x7efe9c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:10.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.944+0000 7efea2470700 1 -- 172.21.15.102:0/3114913270 shutdown_connections 2024-03-26T18:57:10.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.944+0000 7efea2470700 1 -- 172.21.15.102:0/3114913270 wait complete. 2024-03-26T18:57:10.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.944+0000 7efea2470700 1 Processor -- start 2024-03-26T18:57:10.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efea2470700 1 -- start start 2024-03-26T18:57:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efea2470700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe9c0ff050 0x7efe9c102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efea2470700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe9c100250 0x7efe9c101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efea2470700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c101b70 con 0x7efe9c0ff050 2024-03-26T18:57:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efea2470700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c101cb0 con 0x7efe9c100250 2024-03-26T18:57:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efe9bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe9c0ff050 0x7efe9c102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efe9bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe9c0ff050 0x7efe9c102fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57264/0 (socket says 172.21.15.102:57264) 2024-03-26T18:57:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efe9bfff700 1 -- 172.21.15.102:0/1452291622 learned_addr learned my addr 172.21.15.102:0/1452291622 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efe9bfff700 1 -- 172.21.15.102:0/1452291622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe9c100250 msgr2=0x7efe9c101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efe9b7fe700 1 --2- 172.21.15.102:0/1452291622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe9c100250 0x7efe9c101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efe9bfff700 1 --2- 172.21.15.102:0/1452291622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe9c100250 0x7efe9c101630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efe9bfff700 1 -- 172.21.15.102:0/1452291622 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe900095f0 con 0x7efe9c0ff050 2024-03-26T18:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.945+0000 7efe9bfff700 1 --2- 172.21.15.102:0/1452291622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe9c0ff050 0x7efe9c102fe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efe9000bda0 tx=0x7efe9000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.946+0000 7efe997fa700 1 -- 172.21.15.102:0/1452291622 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe9001d070 con 0x7efe9c0ff050 2024-03-26T18:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.946+0000 7efea2470700 1 -- 172.21.15.102:0/1452291622 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c101f30 con 0x7efe9c0ff050 2024-03-26T18:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.946+0000 7efe997fa700 1 -- 172.21.15.102:0/1452291622 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efe9000f460 con 0x7efe9c0ff050 2024-03-26T18:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.946+0000 7efe997fa700 1 -- 172.21.15.102:0/1452291622 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe90021640 con 0x7efe9c0ff050 2024-03-26T18:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.946+0000 7efea2470700 1 -- 172.21.15.102:0/1452291622 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c102420 con 0x7efe9c0ff050 2024-03-26T18:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.947+0000 7efea2470700 1 -- 172.21.15.102:0/1452291622 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c107ba0 con 0x7efe9c0ff050 2024-03-26T18:57:10.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.948+0000 7efe997fa700 1 -- 172.21.15.102:0/1452291622 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7efe90003a60 con 0x7efe9c0ff050 2024-03-26T18:57:10.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.948+0000 7efe997fa700 1 --2- 172.21.15.102:0/1452291622 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efe8406e5a0 0x7efe84070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:10.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.948+0000 7efe997fa700 1 -- 172.21.15.102:0/1452291622 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efe9008f9c0 con 0x7efe9c0ff050 2024-03-26T18:57:10.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.948+0000 7efe9b7fe700 1 --2- 172.21.15.102:0/1452291622 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efe8406e5a0 0x7efe84070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:10.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.949+0000 7efe9b7fe700 1 --2- 172.21.15.102:0/1452291622 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efe8406e5a0 0x7efe84070a50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efe8c005950 tx=0x7efe8c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:10.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:10.953+0000 7efe997fa700 1 -- 172.21.15.102:0/1452291622 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efe9005c4d0 con 0x7efe9c0ff050 2024-03-26T18:57:11.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.183+0000 7efea2470700 1 -- 172.21.15.102:0/1452291622 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efe9c068ae0 con 0x7efe8406e5a0 2024-03-26T18:57:11.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.184+0000 7efe997fa700 1 -- 172.21.15.102:0/1452291622 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7efe9c068ae0 con 0x7efe8406e5a0 2024-03-26T18:57:11.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.186+0000 7efea2470700 1 -- 172.21.15.102:0/1452291622 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efe8406e5a0 msgr2=0x7efe84070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:11.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.186+0000 7efea2470700 1 --2- 172.21.15.102:0/1452291622 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efe8406e5a0 0x7efe84070a50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efe8c005950 tx=0x7efe8c0058e0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.186+0000 7efea2470700 1 -- 172.21.15.102:0/1452291622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe9c0ff050 msgr2=0x7efe9c102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:11.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.186+0000 7efea2470700 1 --2- 172.21.15.102:0/1452291622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe9c0ff050 0x7efe9c102fe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efe9000bda0 tx=0x7efe9000be80 comp rx=0 tx=0).stop 2024-03-26T18:57:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.186+0000 7efea2470700 1 -- 172.21.15.102:0/1452291622 shutdown_connections 2024-03-26T18:57:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.186+0000 7efea2470700 1 --2- 172.21.15.102:0/1452291622 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efe8406e5a0 0x7efe84070a50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.186+0000 7efea2470700 1 --2- 172.21.15.102:0/1452291622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe9c0ff050 0x7efe9c102fe0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.186+0000 7efea2470700 1 --2- 172.21.15.102:0/1452291622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe9c100250 0x7efe9c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.186+0000 7efea2470700 1 -- 172.21.15.102:0/1452291622 >> 172.21.15.102:0/1452291622 conn(0x7efe9c0fa5e0 msgr2=0x7efe9c0683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.186+0000 7efea2470700 1 -- 172.21.15.102:0/1452291622 shutdown_connections 2024-03-26T18:57:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.186+0000 7efea2470700 1 -- 172.21.15.102:0/1452291622 wait complete. 2024-03-26T18:57:11.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.292+0000 7f2eca2d5700 1 -- 172.21.15.102:0/2347673088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ec4100190 msgr2=0x7f2ec41005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:11.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.292+0000 7f2eca2d5700 1 --2- 172.21.15.102:0/2347673088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ec4100190 0x7f2ec41005a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009910 tx=0x7f2eb4009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:11.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.292+0000 7f2eca2d5700 1 -- 172.21.15.102:0/2347673088 shutdown_connections 2024-03-26T18:57:11.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.292+0000 7f2eca2d5700 1 --2- 172.21.15.102:0/2347673088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ec4101390 0x7f2ec41017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.292+0000 7f2eca2d5700 1 --2- 172.21.15.102:0/2347673088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ec4100190 0x7f2ec41005a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.292+0000 7f2eca2d5700 1 -- 172.21.15.102:0/2347673088 >> 172.21.15.102:0/2347673088 conn(0x7f2ec40fb720 msgr2=0x7f2ec40fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:11.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.292+0000 7f2eca2d5700 1 -- 172.21.15.102:0/2347673088 shutdown_connections 2024-03-26T18:57:11.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.292+0000 7f2eca2d5700 1 -- 172.21.15.102:0/2347673088 wait complete. 2024-03-26T18:57:11.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.292+0000 7f2eca2d5700 1 Processor -- start 2024-03-26T18:57:11.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.292+0000 7f2eca2d5700 1 -- start start 2024-03-26T18:57:11.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2eca2d5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ec4100190 0x7f2ec4195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:11.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2eca2d5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ec4101390 0x7f2ec4195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:11.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2eca2d5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec41964f0 con 0x7f2ec4100190 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2eca2d5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec4196630 con 0x7f2ec4101390 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2ec3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ec4100190 0x7f2ec4195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2ec3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ec4100190 0x7f2ec4195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57282/0 (socket says 172.21.15.102:57282) 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2ec3fff700 1 -- 172.21.15.102:0/1718056995 learned_addr learned my addr 172.21.15.102:0/1718056995 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2ec37fe700 1 --2- 172.21.15.102:0/1718056995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ec4101390 0x7f2ec4195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2ec3fff700 1 -- 172.21.15.102:0/1718056995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ec4101390 msgr2=0x7f2ec4195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2ec3fff700 1 --2- 172.21.15.102:0/1718056995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ec4101390 0x7f2ec4195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2ec3fff700 1 -- 172.21.15.102:0/1718056995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb40095f0 con 0x7f2ec4100190 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2ec3fff700 1 --2- 172.21.15.102:0/1718056995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ec4100190 0x7f2ec4195990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400bd70 tx=0x7f2eb400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2ec17fa700 1 -- 172.21.15.102:0/1718056995 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2eb401d070 con 0x7f2ec4100190 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2ec17fa700 1 -- 172.21.15.102:0/1718056995 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2eb4005dc0 con 0x7f2ec4100190 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2eca2d5700 1 -- 172.21.15.102:0/1718056995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec419b080 con 0x7f2ec4100190 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2ec17fa700 1 -- 172.21.15.102:0/1718056995 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2eb4021410 con 0x7f2ec4100190 2024-03-26T18:57:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.293+0000 7f2eca2d5700 1 -- 172.21.15.102:0/1718056995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec419b570 con 0x7f2ec4100190 2024-03-26T18:57:11.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.294+0000 7f2eca2d5700 1 -- 172.21.15.102:0/1718056995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec4108ce0 con 0x7f2ec4100190 2024-03-26T18:57:11.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.295+0000 7f2ec17fa700 1 -- 172.21.15.102:0/1718056995 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f2eb400f7f0 con 0x7f2ec4100190 2024-03-26T18:57:11.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.295+0000 7f2ec17fa700 1 --2- 172.21.15.102:0/1718056995 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2eac06e590 0x7f2eac070a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:11.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.295+0000 7f2ec17fa700 1 -- 172.21.15.102:0/1718056995 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2eb408f8f0 con 0x7f2ec4100190 2024-03-26T18:57:11.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.295+0000 7f2ec37fe700 1 --2- 172.21.15.102:0/1718056995 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2eac06e590 0x7f2eac070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:11.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.296+0000 7f2ec37fe700 1 --2- 172.21.15.102:0/1718056995 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2eac06e590 0x7f2eac070a40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8007c60 tx=0x7f2eb80073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:11.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.300+0000 7f2ec17fa700 1 -- 172.21.15.102:0/1718056995 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2eb405c400 con 0x7f2ec4100190 2024-03-26T18:57:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.516+0000 7f2eca2d5700 1 -- 172.21.15.102:0/1718056995 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ec419bc70 con 0x7f2eac06e590 2024-03-26T18:57:11.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.526+0000 7f2ec17fa700 1 -- 172.21.15.102:0/1718056995 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f2ec419bc70 con 0x7f2eac06e590 2024-03-26T18:57:11.519 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:57:11.519 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (8m) 28s ago 10m 24.9M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (10m) 28s ago 10m 7772k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (9m) 15s ago 9m 7688k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (10m) 28s ago 10m 6665k - 18.2.0 dc2bc1663786 bfac00217c68 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (9m) 15s ago 9m 6656k - 18.2.0 dc2bc1663786 7379f1f6b746 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (8m) 28s ago 9m 84.5M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (4m) 28s ago 4m 133M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (4m) 28s ago 4m 14.5M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (4m) 15s ago 4m 15.0M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (4m) 15s ago 4m 120M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (103s) 28s ago 11m 473M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (56s) 15s ago 9m 447M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (43s) 28s ago 11m 27.7M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T18:57:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (20s) 15s ago 9m 20.3M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T18:57:11.521 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (10m) 28s ago 10m 16.3M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:57:11.521 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (9m) 15s ago 9m 15.5M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:57:11.521 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (8m) 28s ago 8m 220M 1216M 18.2.0 dc2bc1663786 3e2065fb5fbc 2024-03-26T18:57:11.521 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (7m) 28s ago 7m 244M 1216M 18.2.0 dc2bc1663786 0bf773515cc1 2024-03-26T18:57:11.521 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (7m) 28s ago 7m 194M 1216M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:57:11.521 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (7m) 15s ago 7m 272M 2240M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:57:11.521 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (6m) 15s ago 6m 250M 2240M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:57:11.521 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (6m) 15s ago 6m 223M 2240M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:57:11.521 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (61s) 28s ago 9m 62.0M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T18:57:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.529+0000 7f2eca2d5700 1 -- 172.21.15.102:0/1718056995 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2eac06e590 msgr2=0x7f2eac070a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.529+0000 7f2eca2d5700 1 --2- 172.21.15.102:0/1718056995 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2eac06e590 0x7f2eac070a40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8007c60 tx=0x7f2eb80073d0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.529+0000 7f2eca2d5700 1 -- 172.21.15.102:0/1718056995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ec4100190 msgr2=0x7f2ec4195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.529+0000 7f2eca2d5700 1 --2- 172.21.15.102:0/1718056995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ec4100190 0x7f2ec4195990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400bd70 tx=0x7f2eb400bda0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.529+0000 7f2eca2d5700 1 -- 172.21.15.102:0/1718056995 shutdown_connections 2024-03-26T18:57:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.529+0000 7f2eca2d5700 1 --2- 172.21.15.102:0/1718056995 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2eac06e590 0x7f2eac070a40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.529+0000 7f2eca2d5700 1 --2- 172.21.15.102:0/1718056995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ec4100190 0x7f2ec4195990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.529+0000 7f2eca2d5700 1 --2- 172.21.15.102:0/1718056995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ec4101390 0x7f2ec4195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.529+0000 7f2eca2d5700 1 -- 172.21.15.102:0/1718056995 >> 172.21.15.102:0/1718056995 conn(0x7f2ec40fb720 msgr2=0x7f2ec41045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.529+0000 7f2eca2d5700 1 -- 172.21.15.102:0/1718056995 shutdown_connections 2024-03-26T18:57:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.529+0000 7f2eca2d5700 1 -- 172.21.15.102:0/1718056995 wait complete. 2024-03-26T18:57:11.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.634+0000 7fc335a02700 1 -- 172.21.15.102:0/82091221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc330103410 msgr2=0x7fc3301057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:11.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.634+0000 7fc335a02700 1 --2- 172.21.15.102:0/82091221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc330103410 0x7fc3301057f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc32400b0a0 tx=0x7fc32400b3b0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.634+0000 7fc335a02700 1 -- 172.21.15.102:0/82091221 shutdown_connections 2024-03-26T18:57:11.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.634+0000 7fc335a02700 1 --2- 172.21.15.102:0/82091221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc330103410 0x7fc3301057f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.634+0000 7fc335a02700 1 --2- 172.21.15.102:0/82091221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc330100af0 0x7fc330102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.634+0000 7fc335a02700 1 -- 172.21.15.102:0/82091221 >> 172.21.15.102:0/82091221 conn(0x7fc3300fa490 msgr2=0x7fc3300fc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:11.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.634+0000 7fc335a02700 1 -- 172.21.15.102:0/82091221 shutdown_connections 2024-03-26T18:57:11.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.634+0000 7fc335a02700 1 -- 172.21.15.102:0/82091221 wait complete. 2024-03-26T18:57:11.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.634+0000 7fc335a02700 1 Processor -- start 2024-03-26T18:57:11.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc335a02700 1 -- start start 2024-03-26T18:57:11.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc335a02700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc330100af0 0x7fc330100450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:11.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc335a02700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc330103410 0x7fc3300feaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:11.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc335a02700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3300fefe0 con 0x7fc330103410 2024-03-26T18:57:11.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc335a02700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3300ff120 con 0x7fc330100af0 2024-03-26T18:57:11.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc32e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc330103410 0x7fc3300feaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:11.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc32e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc330103410 0x7fc3300feaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57306/0 (socket says 172.21.15.102:57306) 2024-03-26T18:57:11.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc32e7fc700 1 -- 172.21.15.102:0/3731217109 learned_addr learned my addr 172.21.15.102:0/3731217109 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:11.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc32effd700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc330100af0 0x7fc330100450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:11.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc32e7fc700 1 -- 172.21.15.102:0/3731217109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc330100af0 msgr2=0x7fc330100450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:11.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc32e7fc700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc330100af0 0x7fc330100450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc32e7fc700 1 -- 172.21.15.102:0/3731217109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc324009d00 con 0x7fc330103410 2024-03-26T18:57:11.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc32effd700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc330100af0 0x7fc330100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:57:11.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.635+0000 7fc32e7fc700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc330103410 0x7fc3300feaa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc32400d010 tx=0x7fc324009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:11.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.636+0000 7fc334a00700 1 -- 172.21.15.102:0/3731217109 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc32400e040 con 0x7fc330103410 2024-03-26T18:57:11.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.636+0000 7fc334a00700 1 -- 172.21.15.102:0/3731217109 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc324003de0 con 0x7fc330103410 2024-03-26T18:57:11.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.636+0000 7fc335a02700 1 -- 172.21.15.102:0/3731217109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3300ff3a0 con 0x7fc330103410 2024-03-26T18:57:11.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.636+0000 7fc334a00700 1 -- 172.21.15.102:0/3731217109 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc32401d840 con 0x7fc330103410 2024-03-26T18:57:11.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.636+0000 7fc335a02700 1 -- 172.21.15.102:0/3731217109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3300ff890 con 0x7fc330103410 2024-03-26T18:57:11.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.637+0000 7fc335a02700 1 -- 172.21.15.102:0/3731217109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33018fa00 con 0x7fc330103410 2024-03-26T18:57:11.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.637+0000 7fc334a00700 1 -- 172.21.15.102:0/3731217109 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7fc324004040 con 0x7fc330103410 2024-03-26T18:57:11.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.637+0000 7fc334a00700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc31806e5a0 0x7fc318070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:11.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.638+0000 7fc334a00700 1 -- 172.21.15.102:0/3731217109 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc324090020 con 0x7fc330103410 2024-03-26T18:57:11.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.638+0000 7fc32effd700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc31806e5a0 0x7fc318070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:11.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.638+0000 7fc32effd700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc31806e5a0 0x7fc318070a50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc328011490 tx=0x7fc328009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:11.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.642+0000 7fc334a00700 1 -- 172.21.15.102:0/3731217109 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc32405cb30 con 0x7fc330103410 2024-03-26T18:57:11.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:11 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:11.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:11 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:11.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:11 smithi190 ceph-mon[65689]: Reconfiguring osd.2 (monmap changed)... 2024-03-26T18:57:11.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:11 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T18:57:11.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:11 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:11.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:11 smithi190 ceph-mon[65689]: Reconfiguring daemon osd.2 on smithi102 2024-03-26T18:57:11.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:11 smithi190 ceph-mon[65689]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:11.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:11 smithi190 ceph-mon[65689]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:11.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:11 smithi190 ceph-mon[65689]: pgmap v20: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:11.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:11 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:11.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:11 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:11.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:11 smithi102 ceph-mon[78322]: Reconfiguring osd.2 (monmap changed)... 2024-03-26T18:57:11.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:11 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T18:57:11.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:11 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:11.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:11 smithi102 ceph-mon[78322]: Reconfiguring daemon osd.2 on smithi102 2024-03-26T18:57:11.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:11 smithi102 ceph-mon[78322]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:11.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:11 smithi102 ceph-mon[78322]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:11.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:11 smithi102 ceph-mon[78322]: pgmap v20: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.935+0000 7fc335a02700 1 -- 172.21.15.102:0/3731217109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc33002ce10 con 0x7fc330103410 2024-03-26T18:57:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.936+0000 7fc334a00700 1 -- 172.21.15.102:0/3731217109 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7fc32405c6c0 con 0x7fc330103410 2024-03-26T18:57:11.929 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:57:11.929 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:57:11.929 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:57:11.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:57:11.929 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 10, 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 4 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:57:11.930 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:57:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.938+0000 7fc335a02700 1 -- 172.21.15.102:0/3731217109 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc31806e5a0 msgr2=0x7fc318070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.938+0000 7fc335a02700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc31806e5a0 0x7fc318070a50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc328011490 tx=0x7fc328009260 comp rx=0 tx=0).stop 2024-03-26T18:57:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.938+0000 7fc335a02700 1 -- 172.21.15.102:0/3731217109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc330103410 msgr2=0x7fc3300feaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.938+0000 7fc335a02700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc330103410 0x7fc3300feaa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc32400d010 tx=0x7fc324009750 comp rx=0 tx=0).stop 2024-03-26T18:57:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.938+0000 7fc335a02700 1 -- 172.21.15.102:0/3731217109 shutdown_connections 2024-03-26T18:57:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.938+0000 7fc335a02700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc330100af0 0x7fc330100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.938+0000 7fc335a02700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc31806e5a0 0x7fc318070a50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.938+0000 7fc335a02700 1 --2- 172.21.15.102:0/3731217109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc330103410 0x7fc3300feaa0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.938+0000 7fc335a02700 1 -- 172.21.15.102:0/3731217109 >> 172.21.15.102:0/3731217109 conn(0x7fc3300fa490 msgr2=0x7fc3300fc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.939+0000 7fc335a02700 1 -- 172.21.15.102:0/3731217109 shutdown_connections 2024-03-26T18:57:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:11.939+0000 7fc335a02700 1 -- 172.21.15.102:0/3731217109 wait complete. 2024-03-26T18:57:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.042+0000 7f93bd48a700 1 -- 172.21.15.102:0/702729735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b8100270 msgr2=0x7f93b81006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.042+0000 7f93bd48a700 1 --2- 172.21.15.102:0/702729735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b8100270 0x7f93b81006c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f93a0009910 tx=0x7f93a0009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.042+0000 7f93bd48a700 1 -- 172.21.15.102:0/702729735 shutdown_connections 2024-03-26T18:57:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.042+0000 7f93bd48a700 1 --2- 172.21.15.102:0/702729735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b8100270 0x7f93b81006c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.042+0000 7f93bd48a700 1 --2- 172.21.15.102:0/702729735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93b80ff070 0x7f93b80ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.042+0000 7f93bd48a700 1 -- 172.21.15.102:0/702729735 >> 172.21.15.102:0/702729735 conn(0x7f93b80fa600 msgr2=0x7f93b80fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.042+0000 7f93bd48a700 1 -- 172.21.15.102:0/702729735 shutdown_connections 2024-03-26T18:57:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.042+0000 7f93bd48a700 1 -- 172.21.15.102:0/702729735 wait complete. 2024-03-26T18:57:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.043+0000 7f93bd48a700 1 Processor -- start 2024-03-26T18:57:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.043+0000 7f93bd48a700 1 -- start start 2024-03-26T18:57:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.043+0000 7f93bd48a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b80ff070 0x7f93b8194880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.043+0000 7f93bd48a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93b8100270 0x7f93b8194dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.043+0000 7f93bd48a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b81953e0 con 0x7f93b80ff070 2024-03-26T18:57:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.043+0000 7f93bd48a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b8195520 con 0x7f93b8100270 2024-03-26T18:57:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.043+0000 7f93b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b80ff070 0x7f93b8194880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.043+0000 7f93b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b80ff070 0x7f93b8194880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57326/0 (socket says 172.21.15.102:57326) 2024-03-26T18:57:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.043+0000 7f93b6ffd700 1 -- 172.21.15.102:0/506083283 learned_addr learned my addr 172.21.15.102:0/506083283 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.043+0000 7f93ae7fc700 1 --2- 172.21.15.102:0/506083283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93b8100270 0x7f93b8194dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.044+0000 7f93b6ffd700 1 -- 172.21.15.102:0/506083283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93b8100270 msgr2=0x7f93b8194dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.044+0000 7f93b6ffd700 1 --2- 172.21.15.102:0/506083283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93b8100270 0x7f93b8194dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.044+0000 7f93b6ffd700 1 -- 172.21.15.102:0/506083283 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a00095f0 con 0x7f93b80ff070 2024-03-26T18:57:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.044+0000 7f93b6ffd700 1 --2- 172.21.15.102:0/506083283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b80ff070 0x7f93b8194880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f93a800e8f0 tx=0x7f93a800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.044+0000 7f93b4ff9700 1 -- 172.21.15.102:0/506083283 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93a8009790 con 0x7f93b80ff070 2024-03-26T18:57:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.044+0000 7f93b4ff9700 1 -- 172.21.15.102:0/506083283 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f93a800c810 con 0x7f93b80ff070 2024-03-26T18:57:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.044+0000 7f93bd48a700 1 -- 172.21.15.102:0/506083283 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b8199fd0 con 0x7f93b80ff070 2024-03-26T18:57:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.044+0000 7f93b4ff9700 1 -- 172.21.15.102:0/506083283 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93a8003d20 con 0x7f93b80ff070 2024-03-26T18:57:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.044+0000 7f93bd48a700 1 -- 172.21.15.102:0/506083283 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b819a490 con 0x7f93b80ff070 2024-03-26T18:57:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.045+0000 7f93bd48a700 1 -- 172.21.15.102:0/506083283 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b8107bc0 con 0x7f93b80ff070 2024-03-26T18:57:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.046+0000 7f93b4ff9700 1 -- 172.21.15.102:0/506083283 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f93a80041f0 con 0x7f93b80ff070 2024-03-26T18:57:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.046+0000 7f93b4ff9700 1 --2- 172.21.15.102:0/506083283 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f93a406e590 0x7f93a4070a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.046+0000 7f93b4ff9700 1 -- 172.21.15.102:0/506083283 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f93a8014070 con 0x7f93b80ff070 2024-03-26T18:57:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.046+0000 7f93ae7fc700 1 --2- 172.21.15.102:0/506083283 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f93a406e590 0x7f93a4070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.046+0000 7f93ae7fc700 1 --2- 172.21.15.102:0/506083283 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f93a406e590 0x7f93a4070a40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f93a000b6f0 tx=0x7f93a0005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:12.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.050+0000 7f93b4ff9700 1 -- 172.21.15.102:0/506083283 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f93a805b450 con 0x7f93b80ff070 2024-03-26T18:57:12.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.300+0000 7f93bd48a700 1 -- 172.21.15.102:0/506083283 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f93b819aff0 con 0x7f93b80ff070 2024-03-26T18:57:12.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.301+0000 7f93b4ff9700 1 -- 172.21.15.102:0/506083283 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1924 (secure 0 0 0) 0x7f93a805afe0 con 0x7f93b80ff070 2024-03-26T18:57:12.294 INFO:teuthology.orchestra.run.smithi102.stdout:e19 2024-03-26T18:57:12.294 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:57:12.294 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:57:12.294 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:57:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:57:12.294 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 19 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:55:27.824831+0000 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:57:12.295 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:57:12.296 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:57:12.297 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 0 members: 2024-03-26T18:57:12.297 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:57:12.297 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:57:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:57:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:57:12.297 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:57:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:57:12.297 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:57:12.297 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:57:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.303+0000 7f93bd48a700 1 -- 172.21.15.102:0/506083283 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f93a406e590 msgr2=0x7f93a4070a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.303+0000 7f93bd48a700 1 --2- 172.21.15.102:0/506083283 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f93a406e590 0x7f93a4070a40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f93a000b6f0 tx=0x7f93a0005a90 comp rx=0 tx=0).stop 2024-03-26T18:57:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.303+0000 7f93bd48a700 1 -- 172.21.15.102:0/506083283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b80ff070 msgr2=0x7f93b8194880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.303+0000 7f93bd48a700 1 --2- 172.21.15.102:0/506083283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b80ff070 0x7f93b8194880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f93a800e8f0 tx=0x7f93a800ec00 comp rx=0 tx=0).stop 2024-03-26T18:57:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.303+0000 7f93bd48a700 1 -- 172.21.15.102:0/506083283 shutdown_connections 2024-03-26T18:57:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.303+0000 7f93bd48a700 1 --2- 172.21.15.102:0/506083283 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f93a406e590 0x7f93a4070a40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.303+0000 7f93bd48a700 1 --2- 172.21.15.102:0/506083283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b80ff070 0x7f93b8194880 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.303+0000 7f93bd48a700 1 --2- 172.21.15.102:0/506083283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93b8100270 0x7f93b8194dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.303+0000 7f93bd48a700 1 -- 172.21.15.102:0/506083283 >> 172.21.15.102:0/506083283 conn(0x7f93b80fa600 msgr2=0x7f93b81034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.303+0000 7f93bd48a700 1 -- 172.21.15.102:0/506083283 shutdown_connections 2024-03-26T18:57:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.303+0000 7f93bd48a700 1 -- 172.21.15.102:0/506083283 wait complete. 2024-03-26T18:57:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 19 2024-03-26T18:57:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.426+0000 7ff73feca700 1 -- 172.21.15.102:0/4252471878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7380ff080 msgr2=0x7ff7380ff490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.426+0000 7ff73feca700 1 --2- 172.21.15.102:0/4252471878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7380ff080 0x7ff7380ff490 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff734009910 tx=0x7ff734009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:12.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.426+0000 7ff73feca700 1 -- 172.21.15.102:0/4252471878 shutdown_connections 2024-03-26T18:57:12.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.426+0000 7ff73feca700 1 --2- 172.21.15.102:0/4252471878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff738100280 0x7ff7381006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.426+0000 7ff73feca700 1 --2- 172.21.15.102:0/4252471878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7380ff080 0x7ff7380ff490 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.426+0000 7ff73feca700 1 -- 172.21.15.102:0/4252471878 >> 172.21.15.102:0/4252471878 conn(0x7ff7380fa630 msgr2=0x7ff7380fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:12.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.426+0000 7ff73feca700 1 -- 172.21.15.102:0/4252471878 shutdown_connections 2024-03-26T18:57:12.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.426+0000 7ff73feca700 1 -- 172.21.15.102:0/4252471878 wait complete. 2024-03-26T18:57:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73feca700 1 Processor -- start 2024-03-26T18:57:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73feca700 1 -- start start 2024-03-26T18:57:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73feca700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7380ff080 0x7ff7381959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73feca700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff738100280 0x7ff738195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73feca700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff738196520 con 0x7ff7380ff080 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73feca700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff738196660 con 0x7ff738100280 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73dc66700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7380ff080 0x7ff7381959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73dc66700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7380ff080 0x7ff7381959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57356/0 (socket says 172.21.15.102:57356) 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73dc66700 1 -- 172.21.15.102:0/884382900 learned_addr learned my addr 172.21.15.102:0/884382900 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73d465700 1 --2- 172.21.15.102:0/884382900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff738100280 0x7ff738195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73dc66700 1 -- 172.21.15.102:0/884382900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff738100280 msgr2=0x7ff738195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73dc66700 1 --2- 172.21.15.102:0/884382900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff738100280 0x7ff738195f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.427+0000 7ff73dc66700 1 -- 172.21.15.102:0/884382900 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7340095f0 con 0x7ff7380ff080 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.428+0000 7ff73dc66700 1 --2- 172.21.15.102:0/884382900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7380ff080 0x7ff7381959c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff73400bda0 tx=0x7ff73400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.428+0000 7ff72effd700 1 -- 172.21.15.102:0/884382900 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff73401d070 con 0x7ff7380ff080 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.428+0000 7ff72effd700 1 -- 172.21.15.102:0/884382900 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff73400f460 con 0x7ff7380ff080 2024-03-26T18:57:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.428+0000 7ff73feca700 1 -- 172.21.15.102:0/884382900 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff73819b0b0 con 0x7ff7380ff080 2024-03-26T18:57:12.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.428+0000 7ff73feca700 1 -- 172.21.15.102:0/884382900 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73819b5a0 con 0x7ff7380ff080 2024-03-26T18:57:12.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.428+0000 7ff72effd700 1 -- 172.21.15.102:0/884382900 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff734021640 con 0x7ff7380ff080 2024-03-26T18:57:12.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.429+0000 7ff73feca700 1 -- 172.21.15.102:0/884382900 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c005320 con 0x7ff7380ff080 2024-03-26T18:57:12.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.429+0000 7ff72effd700 1 -- 172.21.15.102:0/884382900 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7ff73400f5d0 con 0x7ff7380ff080 2024-03-26T18:57:12.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.430+0000 7ff72effd700 1 --2- 172.21.15.102:0/884382900 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff72406e5a0 0x7ff724070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:12.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.430+0000 7ff72effd700 1 -- 172.21.15.102:0/884382900 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff73408f890 con 0x7ff7380ff080 2024-03-26T18:57:12.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.430+0000 7ff73d465700 1 --2- 172.21.15.102:0/884382900 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff72406e5a0 0x7ff724070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:12.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.430+0000 7ff73d465700 1 --2- 172.21.15.102:0/884382900 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff72406e5a0 0x7ff724070a50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff728007c60 tx=0x7ff7280073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:12.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.435+0000 7ff72effd700 1 -- 172.21.15.102:0/884382900 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff73405c3a0 con 0x7ff7380ff080 2024-03-26T18:57:12.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.661+0000 7ff73feca700 1 -- 172.21.15.102:0/884382900 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff71c000bf0 con 0x7ff72406e5a0 2024-03-26T18:57:12.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.662+0000 7ff72effd700 1 -- 172.21.15.102:0/884382900 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7ff71c000bf0 con 0x7ff72406e5a0 2024-03-26T18:57:12.655 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:57:12.655 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:57:12.655 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:57:12.655 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:57:12.655 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T18:57:12.655 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T18:57:12.655 INFO:teuthology.orchestra.run.smithi102.stdout: "mon" 2024-03-26T18:57:12.655 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T18:57:12.656 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "4/23 daemons upgraded", 2024-03-26T18:57:12.656 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading mon daemons", 2024-03-26T18:57:12.656 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:57:12.656 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:57:12.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.664+0000 7ff73feca700 1 -- 172.21.15.102:0/884382900 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff72406e5a0 msgr2=0x7ff724070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:12.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.664+0000 7ff73feca700 1 --2- 172.21.15.102:0/884382900 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff72406e5a0 0x7ff724070a50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff728007c60 tx=0x7ff7280073d0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.664+0000 7ff73feca700 1 -- 172.21.15.102:0/884382900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7380ff080 msgr2=0x7ff7381959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:12.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.664+0000 7ff73feca700 1 --2- 172.21.15.102:0/884382900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7380ff080 0x7ff7381959c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff73400bda0 tx=0x7ff73400bdd0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.664+0000 7ff73feca700 1 -- 172.21.15.102:0/884382900 shutdown_connections 2024-03-26T18:57:12.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.664+0000 7ff73feca700 1 --2- 172.21.15.102:0/884382900 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff72406e5a0 0x7ff724070a50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.664+0000 7ff73feca700 1 --2- 172.21.15.102:0/884382900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7380ff080 0x7ff7381959c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.664+0000 7ff73feca700 1 --2- 172.21.15.102:0/884382900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff738100280 0x7ff738195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.664+0000 7ff73feca700 1 -- 172.21.15.102:0/884382900 >> 172.21.15.102:0/884382900 conn(0x7ff7380fa630 msgr2=0x7ff738068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:12.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.664+0000 7ff73feca700 1 -- 172.21.15.102:0/884382900 shutdown_connections 2024-03-26T18:57:12.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.664+0000 7ff73feca700 1 -- 172.21.15.102:0/884382900 wait complete. 2024-03-26T18:57:12.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.771+0000 7fc05d12e700 1 -- 172.21.15.102:0/404022437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc058100250 msgr2=0x7fc0581006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:12.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.771+0000 7fc05d12e700 1 --2- 172.21.15.102:0/404022437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc058100250 0x7fc0581006a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc048009910 tx=0x7fc048009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:12.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.771+0000 7fc05d12e700 1 -- 172.21.15.102:0/404022437 shutdown_connections 2024-03-26T18:57:12.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.771+0000 7fc05d12e700 1 --2- 172.21.15.102:0/404022437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc058100250 0x7fc0581006a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.771+0000 7fc05d12e700 1 --2- 172.21.15.102:0/404022437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ff050 0x7fc0580ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.771+0000 7fc05d12e700 1 -- 172.21.15.102:0/404022437 >> 172.21.15.102:0/404022437 conn(0x7fc0580fa600 msgr2=0x7fc0580fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:12.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.771+0000 7fc05d12e700 1 -- 172.21.15.102:0/404022437 shutdown_connections 2024-03-26T18:57:12.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.771+0000 7fc05d12e700 1 -- 172.21.15.102:0/404022437 wait complete. 2024-03-26T18:57:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.771+0000 7fc05d12e700 1 Processor -- start 2024-03-26T18:57:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc05d12e700 1 -- start start 2024-03-26T18:57:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc05d12e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ff050 0x7fc0580697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc05d12e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc058100250 0x7fc05819a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc05d12e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05819ac00 con 0x7fc058100250 2024-03-26T18:57:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc05d12e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05819ad40 con 0x7fc0580ff050 2024-03-26T18:57:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc05659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc058100250 0x7fc05819a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc056d9d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ff050 0x7fc0580697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc05659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc058100250 0x7fc05819a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57382/0 (socket says 172.21.15.102:57382) 2024-03-26T18:57:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc05659c700 1 -- 172.21.15.102:0/1944290697 learned_addr learned my addr 172.21.15.102:0/1944290697 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:12.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc05659c700 1 -- 172.21.15.102:0/1944290697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ff050 msgr2=0x7fc0580697c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:12.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc05659c700 1 --2- 172.21.15.102:0/1944290697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ff050 0x7fc0580697c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:12.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.772+0000 7fc05659c700 1 -- 172.21.15.102:0/1944290697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0480095f0 con 0x7fc058100250 2024-03-26T18:57:12.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.773+0000 7fc05659c700 1 --2- 172.21.15.102:0/1944290697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc058100250 0x7fc05819a6c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc04800bd70 tx=0x7fc04800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:12.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.773+0000 7fc03ffff700 1 -- 172.21.15.102:0/1944290697 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc04801d070 con 0x7fc058100250 2024-03-26T18:57:12.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.773+0000 7fc03ffff700 1 -- 172.21.15.102:0/1944290697 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc048005940 con 0x7fc058100250 2024-03-26T18:57:12.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.773+0000 7fc05d12e700 1 -- 172.21.15.102:0/1944290697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc05819afc0 con 0x7fc058100250 2024-03-26T18:57:12.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.773+0000 7fc03ffff700 1 -- 172.21.15.102:0/1944290697 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc048021430 con 0x7fc058100250 2024-03-26T18:57:12.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.773+0000 7fc05d12e700 1 -- 172.21.15.102:0/1944290697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05819b480 con 0x7fc058100250 2024-03-26T18:57:12.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.774+0000 7fc05d12e700 1 -- 172.21.15.102:0/1944290697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc058107ba0 con 0x7fc058100250 2024-03-26T18:57:12.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.774+0000 7fc03ffff700 1 -- 172.21.15.102:0/1944290697 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7fc04800f860 con 0x7fc058100250 2024-03-26T18:57:12.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.775+0000 7fc03ffff700 1 --2- 172.21.15.102:0/1944290697 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc04006e5a0 0x7fc040070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:12.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.775+0000 7fc03ffff700 1 -- 172.21.15.102:0/1944290697 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc04808fa50 con 0x7fc058100250 2024-03-26T18:57:12.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.775+0000 7fc056d9d700 1 --2- 172.21.15.102:0/1944290697 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc04006e5a0 0x7fc040070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:12.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.775+0000 7fc056d9d700 1 --2- 172.21.15.102:0/1944290697 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc04006e5a0 0x7fc040070a50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc0581000b0 tx=0x7fc04c00a3b0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:12.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:12.784+0000 7fc03ffff700 1 -- 172.21.15.102:0/1944290697 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc04805c560 con 0x7fc058100250 2024-03-26T18:57:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:12 smithi190 ceph-mon[65689]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:12 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3731217109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:12 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/506083283' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:57:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:12 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:12 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:12 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.tvsqaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:57:12.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:12 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:12.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:12 smithi102 ceph-mon[78322]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:12.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:12 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3731217109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:12.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:12 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/506083283' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:57:12.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:12 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:12.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:12 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:12.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:12 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.tvsqaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:57:12.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:12 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:13.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.089+0000 7fc05d12e700 1 -- 172.21.15.102:0/1944290697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc05819bac0 con 0x7fc058100250 2024-03-26T18:57:13.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.089+0000 7fc03ffff700 1 -- 172.21.15.102:0/1944290697 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fc04805c0f0 con 0x7fc058100250 2024-03-26T18:57:13.083 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T18:57:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.091+0000 7fc05d12e700 1 -- 172.21.15.102:0/1944290697 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc04006e5a0 msgr2=0x7fc040070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.091+0000 7fc05d12e700 1 --2- 172.21.15.102:0/1944290697 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc04006e5a0 0x7fc040070a50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc0581000b0 tx=0x7fc04c00a3b0 comp rx=0 tx=0).stop 2024-03-26T18:57:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.091+0000 7fc05d12e700 1 -- 172.21.15.102:0/1944290697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc058100250 msgr2=0x7fc05819a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.091+0000 7fc05d12e700 1 --2- 172.21.15.102:0/1944290697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc058100250 0x7fc05819a6c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc04800bd70 tx=0x7fc04800bda0 comp rx=0 tx=0).stop 2024-03-26T18:57:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.091+0000 7fc05d12e700 1 -- 172.21.15.102:0/1944290697 shutdown_connections 2024-03-26T18:57:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.091+0000 7fc05d12e700 1 --2- 172.21.15.102:0/1944290697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ff050 0x7fc0580697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.091+0000 7fc05d12e700 1 --2- 172.21.15.102:0/1944290697 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc04006e5a0 0x7fc040070a50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.091+0000 7fc05d12e700 1 --2- 172.21.15.102:0/1944290697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc058100250 0x7fc05819a6c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.091+0000 7fc05d12e700 1 -- 172.21.15.102:0/1944290697 >> 172.21.15.102:0/1944290697 conn(0x7fc0580fa600 msgr2=0x7fc058103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.091+0000 7fc05d12e700 1 -- 172.21.15.102:0/1944290697 shutdown_connections 2024-03-26T18:57:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:13.092+0000 7fc05d12e700 1 -- 172.21.15.102:0/1944290697 wait complete. 2024-03-26T18:57:13.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:13 smithi102 ceph-mon[78322]: Reconfiguring mds.cephfs.smithi102.tvsqaa (monmap changed)... 2024-03-26T18:57:13.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:13 smithi102 ceph-mon[78322]: Reconfiguring daemon mds.cephfs.smithi102.tvsqaa on smithi102 2024-03-26T18:57:13.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:13 smithi102 ceph-mon[78322]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:13.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:13 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1944290697' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:57:13.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:13 smithi102 ceph-mon[78322]: pgmap v21: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:14.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:13 smithi190 ceph-mon[65689]: Reconfiguring mds.cephfs.smithi102.tvsqaa (monmap changed)... 2024-03-26T18:57:14.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:13 smithi190 ceph-mon[65689]: Reconfiguring daemon mds.cephfs.smithi102.tvsqaa on smithi102 2024-03-26T18:57:14.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:13 smithi190 ceph-mon[65689]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:14.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:13 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1944290697' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:57:14.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:13 smithi190 ceph-mon[65689]: pgmap v21: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:15.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:15.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:15.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:15 smithi102 ceph-mon[78322]: Reconfiguring mds.cephfs.smithi102.hrbrsp (monmap changed)... 2024-03-26T18:57:15.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hrbrsp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:57:15.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:15.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:15 smithi102 ceph-mon[78322]: Reconfiguring daemon mds.cephfs.smithi102.hrbrsp on smithi102 2024-03-26T18:57:15.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:15.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:15.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:15 smithi190 ceph-mon[65689]: Reconfiguring mds.cephfs.smithi102.hrbrsp (monmap changed)... 2024-03-26T18:57:15.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hrbrsp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:57:15.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:15.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:15 smithi190 ceph-mon[65689]: Reconfiguring daemon mds.cephfs.smithi102.hrbrsp on smithi102 2024-03-26T18:57:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:16 smithi102 ceph-mon[78322]: pgmap v22: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:16 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:16 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:16 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:57:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:16 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:57:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:16 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-26T18:57:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:16 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi190"}]: dispatch 2024-03-26T18:57:16.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:16 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:16.377 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:16 smithi190 ceph-mon[65689]: pgmap v22: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:16.377 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:16 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:16.377 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:16 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:16.377 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:16 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:57:16.377 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:16 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T18:57:16.377 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:16 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-26T18:57:16.377 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:16 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi190"}]: dispatch 2024-03-26T18:57:16.377 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:16 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:17.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:17 smithi102 ceph-mon[78322]: Reconfiguring ceph-exporter.smithi190 (monmap changed)... 2024-03-26T18:57:17.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:17 smithi102 ceph-mon[78322]: Unable to update caps for client.ceph-exporter.smithi190 2024-03-26T18:57:17.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:17 smithi102 ceph-mon[78322]: Reconfiguring daemon ceph-exporter.smithi190 on smithi190 2024-03-26T18:57:17.400 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:17 smithi190 ceph-mon[65689]: Reconfiguring ceph-exporter.smithi190 (monmap changed)... 2024-03-26T18:57:17.400 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:17 smithi190 ceph-mon[65689]: Unable to update caps for client.ceph-exporter.smithi190 2024-03-26T18:57:17.400 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:17 smithi190 ceph-mon[65689]: Reconfiguring daemon ceph-exporter.smithi190 on smithi190 2024-03-26T18:57:18.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:18 smithi190 ceph-mon[65689]: pgmap v23: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:18.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:18 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:18.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:18 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:18.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:18 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:57:18.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:18 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:18.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:18 smithi102 ceph-mon[78322]: pgmap v23: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:18.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:18 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:18.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:18 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:18.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:18 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:57:18.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:18 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:19.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:19 smithi102 ceph-mon[78322]: Reconfiguring crash.smithi190 (monmap changed)... 2024-03-26T18:57:19.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:19 smithi102 ceph-mon[78322]: Reconfiguring daemon crash.smithi190 on smithi190 2024-03-26T18:57:19.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:19 smithi190 ceph-mon[65689]: Reconfiguring crash.smithi190 (monmap changed)... 2024-03-26T18:57:19.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:19 smithi190 ceph-mon[65689]: Reconfiguring daemon crash.smithi190 on smithi190 2024-03-26T18:57:20.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:20 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:20.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:20 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:20.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:20 smithi190 ceph-mon[65689]: Reconfiguring mgr.smithi190.zrixos (monmap changed)... 2024-03-26T18:57:20.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:20 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.zrixos", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:57:20.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:20 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:57:20.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:20 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:20.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:20 smithi190 ceph-mon[65689]: Reconfiguring daemon mgr.smithi190.zrixos on smithi190 2024-03-26T18:57:20.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:20 smithi190 ceph-mon[65689]: pgmap v24: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:20.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:20 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:20.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:20 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:20.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:20 smithi102 ceph-mon[78322]: Reconfiguring mgr.smithi190.zrixos (monmap changed)... 2024-03-26T18:57:20.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:20 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.zrixos", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T18:57:20.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:20 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T18:57:20.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:20 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:20.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:20 smithi102 ceph-mon[78322]: Reconfiguring daemon mgr.smithi190.zrixos on smithi190 2024-03-26T18:57:20.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:20 smithi102 ceph-mon[78322]: pgmap v24: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:21.875 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:21 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:21.875 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:21 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:21.875 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:21 smithi190 ceph-mon[65689]: Reconfiguring mon.smithi190 (monmap changed)... 2024-03-26T18:57:21.875 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:21 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:57:21.875 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:21 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:57:21.875 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:21 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:21.875 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:21 smithi190 ceph-mon[65689]: Reconfiguring daemon mon.smithi190 on smithi190 2024-03-26T18:57:21.876 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:21 smithi190 ceph-mon[65689]: pgmap v25: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:22.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:21 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:22.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:21 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:22.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:21 smithi102 ceph-mon[78322]: Reconfiguring mon.smithi190 (monmap changed)... 2024-03-26T18:57:22.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:21 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T18:57:22.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:21 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T18:57:22.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:21 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:22.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:21 smithi102 ceph-mon[78322]: Reconfiguring daemon mon.smithi190 on smithi190 2024-03-26T18:57:22.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:21 smithi102 ceph-mon[78322]: pgmap v25: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:23.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:57:23.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:23.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:23.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T18:57:23.029 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:23.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:57:23.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:23.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:23.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T18:57:23.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:23.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:23 smithi102 ceph-mon[78322]: Reconfiguring osd.3 (monmap changed)... 2024-03-26T18:57:23.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:23 smithi102 ceph-mon[78322]: Reconfiguring daemon osd.3 on smithi190 2024-03-26T18:57:23.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:23 smithi102 ceph-mon[78322]: pgmap v26: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:24.017 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:23 smithi190 ceph-mon[65689]: Reconfiguring osd.3 (monmap changed)... 2024-03-26T18:57:24.017 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:23 smithi190 ceph-mon[65689]: Reconfiguring daemon osd.3 on smithi190 2024-03-26T18:57:24.017 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:23 smithi190 ceph-mon[65689]: pgmap v26: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:25.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:25.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:25.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:25 smithi102 ceph-mon[78322]: Reconfiguring osd.4 (monmap changed)... 2024-03-26T18:57:25.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T18:57:25.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:25.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:25 smithi102 ceph-mon[78322]: Reconfiguring daemon osd.4 on smithi190 2024-03-26T18:57:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:25 smithi190 ceph-mon[65689]: Reconfiguring osd.4 (monmap changed)... 2024-03-26T18:57:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T18:57:25.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:25.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:25 smithi190 ceph-mon[65689]: Reconfiguring daemon osd.4 on smithi190 2024-03-26T18:57:26.378 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:26 smithi190 ceph-mon[65689]: pgmap v27: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:26.379 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:26.379 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:26.379 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T18:57:26.379 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:26.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:26 smithi102 ceph-mon[78322]: pgmap v27: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:26.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:26.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:26.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T18:57:26.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:27.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:27 smithi190 ceph-mon[65689]: Reconfiguring osd.5 (monmap changed)... 2024-03-26T18:57:27.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:27 smithi190 ceph-mon[65689]: Reconfiguring daemon osd.5 on smithi190 2024-03-26T18:57:27.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:27 smithi102 ceph-mon[78322]: Reconfiguring osd.5 (monmap changed)... 2024-03-26T18:57:27.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:27 smithi102 ceph-mon[78322]: Reconfiguring daemon osd.5 on smithi190 2024-03-26T18:57:28.347 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:28 smithi190 ceph-mon[65689]: pgmap v28: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:28.347 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:28 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:28.348 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:28 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:28.348 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:28 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.vykizb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:57:28.348 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:28 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:28.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:28 smithi102 ceph-mon[78322]: pgmap v28: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:28.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:28 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:28.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:28 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:28.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:28 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.vykizb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:57:28.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:28 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:29.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:29 smithi190 ceph-mon[65689]: Reconfiguring mds.cephfs.smithi190.vykizb (monmap changed)... 2024-03-26T18:57:29.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:29 smithi190 ceph-mon[65689]: Reconfiguring daemon mds.cephfs.smithi190.vykizb on smithi190 2024-03-26T18:57:29.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:29 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:29.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:29 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:29.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:29 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.hwsvdg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:57:29.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:29 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:29.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:29 smithi102 ceph-mon[78322]: Reconfiguring mds.cephfs.smithi190.vykizb (monmap changed)... 2024-03-26T18:57:29.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:29 smithi102 ceph-mon[78322]: Reconfiguring daemon mds.cephfs.smithi190.vykizb on smithi190 2024-03-26T18:57:29.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:29 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:29.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:29 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:29.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:29 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.hwsvdg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T18:57:29.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:29 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:30.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:30 smithi102 ceph-mon[78322]: Reconfiguring mds.cephfs.smithi190.hwsvdg (monmap changed)... 2024-03-26T18:57:30.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:30 smithi102 ceph-mon[78322]: Reconfiguring daemon mds.cephfs.smithi190.hwsvdg on smithi190 2024-03-26T18:57:30.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:30 smithi102 ceph-mon[78322]: pgmap v29: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:30.461 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:30 smithi190 ceph-mon[65689]: Reconfiguring mds.cephfs.smithi190.hwsvdg (monmap changed)... 2024-03-26T18:57:30.461 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:30 smithi190 ceph-mon[65689]: Reconfiguring daemon mds.cephfs.smithi190.hwsvdg on smithi190 2024-03-26T18:57:30.461 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:30 smithi190 ceph-mon[65689]: pgmap v29: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:31.514 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:31.514 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:31.514 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:57:31.514 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:31.514 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:31.514 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: Upgrade: Setting container_image for all mon 2024-03-26T18:57:31.514 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:31.514 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi102"}]: dispatch 2024-03-26T18:57:31.515 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi102"}]': finished 2024-03-26T18:57:31.515 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi190"}]: dispatch 2024-03-26T18:57:31.515 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi190"}]': finished 2024-03-26T18:57:31.515 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:31 smithi190 ceph-mon[65689]: pgmap v30: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:31.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:31.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:31.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:57:31.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:31.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:31.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: Upgrade: Setting container_image for all mon 2024-03-26T18:57:31.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:31.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi102"}]: dispatch 2024-03-26T18:57:31.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi102"}]': finished 2024-03-26T18:57:31.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi190"}]: dispatch 2024-03-26T18:57:31.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi190"}]': finished 2024-03-26T18:57:31.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:31 smithi102 ceph-mon[78322]: pgmap v30: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:33.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:33 smithi102 ceph-mon[78322]: Upgrade: Updating crash.smithi102 (1/2) 2024-03-26T18:57:33.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:33 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:33.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:33 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:57:33.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:33 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:33.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:33 smithi102 ceph-mon[78322]: Deploying daemon crash.smithi102 on smithi102 2024-03-26T18:57:33.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:33 smithi190 ceph-mon[65689]: Upgrade: Updating crash.smithi102 (1/2) 2024-03-26T18:57:33.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:33 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:33.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:33 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:57:33.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:33 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:33.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:33 smithi190 ceph-mon[65689]: Deploying daemon crash.smithi102 on smithi102 2024-03-26T18:57:34.357 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:34 smithi102 ceph-mon[78322]: pgmap v31: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:34.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:34 smithi190 ceph-mon[65689]: pgmap v31: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:36.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:36 smithi190 ceph-mon[65689]: pgmap v32: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:36.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:36 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:36.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:36 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:36.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:36 smithi102 ceph-mon[78322]: pgmap v32: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:36.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:36 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:36.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:36 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:38.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:38 smithi102 ceph-mon[78322]: pgmap v33: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:38.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:57:38.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:38.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:57:38.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:38.415 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:38 smithi190 ceph-mon[65689]: pgmap v33: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:38.415 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:57:38.415 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:38.415 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T18:57:38.415 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:39.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:39 smithi102 ceph-mon[78322]: Upgrade: Updating crash.smithi190 (2/2) 2024-03-26T18:57:39.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:39 smithi102 ceph-mon[78322]: Deploying daemon crash.smithi190 on smithi190 2024-03-26T18:57:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:39 smithi190 ceph-mon[65689]: Upgrade: Updating crash.smithi190 (2/2) 2024-03-26T18:57:39.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:39 smithi190 ceph-mon[65689]: Deploying daemon crash.smithi190 on smithi190 2024-03-26T18:57:40.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:40 smithi190 ceph-mon[65689]: pgmap v34: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:40.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:40 smithi102 ceph-mon[78322]: pgmap v34: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:42.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:42 smithi190 ceph-mon[65689]: pgmap v35: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:42.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:42 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:42.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:42 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:42.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:42 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:57:42.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:42 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:42.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:42 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:42 smithi102 ceph-mon[78322]: pgmap v35: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:42 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:42 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:42 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:57:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:42 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:42 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:43.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.200+0000 7f66fbee2700 1 -- 172.21.15.102:0/2957326829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f40ff320 msgr2=0x7f66f40ff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:43.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.200+0000 7f66fbee2700 1 --2- 172.21.15.102:0/2957326829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f40ff320 0x7f66f40ff770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009910 tx=0x7f66e8009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:43.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.200+0000 7f66fbee2700 1 -- 172.21.15.102:0/2957326829 shutdown_connections 2024-03-26T18:57:43.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.200+0000 7f66fbee2700 1 --2- 172.21.15.102:0/2957326829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f40ff320 0x7f66f40ff770 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.200+0000 7f66fbee2700 1 --2- 172.21.15.102:0/2957326829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66f41005e0 0x7f66f41009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.200+0000 7f66fbee2700 1 -- 172.21.15.102:0/2957326829 >> 172.21.15.102:0/2957326829 conn(0x7f66f40f9d80 msgr2=0x7f66f40fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:43.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.200+0000 7f66fbee2700 1 -- 172.21.15.102:0/2957326829 shutdown_connections 2024-03-26T18:57:43.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.200+0000 7f66fbee2700 1 -- 172.21.15.102:0/2957326829 wait complete. 2024-03-26T18:57:43.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66fbee2700 1 Processor -- start 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66fbee2700 1 -- start start 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66fbee2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f40ff320 0x7f66f40fed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66fbee2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66f41005e0 0x7f66f419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66fbee2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f419ac80 con 0x7f66f40ff320 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66fbee2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f419adc0 con 0x7f66f41005e0 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66f9c7e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f40ff320 0x7f66f40fed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66f9c7e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f40ff320 0x7f66f40fed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34730/0 (socket says 172.21.15.102:34730) 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66f9c7e700 1 -- 172.21.15.102:0/324060418 learned_addr learned my addr 172.21.15.102:0/324060418 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66f947d700 1 --2- 172.21.15.102:0/324060418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66f41005e0 0x7f66f419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66f9c7e700 1 -- 172.21.15.102:0/324060418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66f41005e0 msgr2=0x7f66f419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66f9c7e700 1 --2- 172.21.15.102:0/324060418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66f41005e0 0x7f66f419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66f9c7e700 1 -- 172.21.15.102:0/324060418 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e80095f0 con 0x7f66f40ff320 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.201+0000 7f66f9c7e700 1 --2- 172.21.15.102:0/324060418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f40ff320 0x7f66f40fed70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00efc0 tx=0x7f66ec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.202+0000 7f66f2ffd700 1 -- 172.21.15.102:0/324060418 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66ec018070 con 0x7f66f40ff320 2024-03-26T18:57:43.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.202+0000 7f66f2ffd700 1 -- 172.21.15.102:0/324060418 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f66ec00f040 con 0x7f66f40ff320 2024-03-26T18:57:43.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.202+0000 7f66fbee2700 1 -- 172.21.15.102:0/324060418 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f419b040 con 0x7f66f40ff320 2024-03-26T18:57:43.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.202+0000 7f66f2ffd700 1 -- 172.21.15.102:0/324060418 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66ec004490 con 0x7f66f40ff320 2024-03-26T18:57:43.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.202+0000 7f66fbee2700 1 -- 172.21.15.102:0/324060418 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f419b560 con 0x7f66f40ff320 2024-03-26T18:57:43.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.203+0000 7f66fbee2700 1 -- 172.21.15.102:0/324060418 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f4107b90 con 0x7f66f40ff320 2024-03-26T18:57:43.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.203+0000 7f66f2ffd700 1 -- 172.21.15.102:0/324060418 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f66ec004710 con 0x7f66f40ff320 2024-03-26T18:57:43.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.203+0000 7f66f2ffd700 1 --2- 172.21.15.102:0/324060418 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f66dc06e5a0 0x7f66dc070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:43.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.204+0000 7f66f2ffd700 1 -- 172.21.15.102:0/324060418 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f66ec08e8c0 con 0x7f66f40ff320 2024-03-26T18:57:43.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.204+0000 7f66f947d700 1 --2- 172.21.15.102:0/324060418 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f66dc06e5a0 0x7f66dc070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:43.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.204+0000 7f66f947d700 1 --2- 172.21.15.102:0/324060418 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f66dc06e5a0 0x7f66dc070a50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009fd0 tx=0x7f66e8005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.208+0000 7f66f2ffd700 1 -- 172.21.15.102:0/324060418 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f66ec05b3d0 con 0x7f66f40ff320 2024-03-26T18:57:43.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.444+0000 7f66fbee2700 1 -- 172.21.15.102:0/324060418 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f66f4104c80 con 0x7f66dc06e5a0 2024-03-26T18:57:43.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.445+0000 7f66f2ffd700 1 -- 172.21.15.102:0/324060418 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f66f4104c80 con 0x7f66dc06e5a0 2024-03-26T18:57:43.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.448+0000 7f66fbee2700 1 -- 172.21.15.102:0/324060418 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f66dc06e5a0 msgr2=0x7f66dc070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.448+0000 7f66fbee2700 1 --2- 172.21.15.102:0/324060418 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f66dc06e5a0 0x7f66dc070a50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009fd0 tx=0x7f66e8005940 comp rx=0 tx=0).stop 2024-03-26T18:57:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.448+0000 7f66fbee2700 1 -- 172.21.15.102:0/324060418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f40ff320 msgr2=0x7f66f40fed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.448+0000 7f66fbee2700 1 --2- 172.21.15.102:0/324060418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f40ff320 0x7f66f40fed70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00efc0 tx=0x7f66ec00c3a0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.448+0000 7f66fbee2700 1 -- 172.21.15.102:0/324060418 shutdown_connections 2024-03-26T18:57:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.448+0000 7f66fbee2700 1 --2- 172.21.15.102:0/324060418 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f66dc06e5a0 0x7f66dc070a50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.448+0000 7f66fbee2700 1 --2- 172.21.15.102:0/324060418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f40ff320 0x7f66f40fed70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.448+0000 7f66fbee2700 1 --2- 172.21.15.102:0/324060418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66f41005e0 0x7f66f419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.448+0000 7f66fbee2700 1 -- 172.21.15.102:0/324060418 >> 172.21.15.102:0/324060418 conn(0x7f66f40f9d80 msgr2=0x7f66f4103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.448+0000 7f66fbee2700 1 -- 172.21.15.102:0/324060418 shutdown_connections 2024-03-26T18:57:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.448+0000 7f66fbee2700 1 -- 172.21.15.102:0/324060418 wait complete. 2024-03-26T18:57:43.452 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:57:43.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.554+0000 7fd8179d3700 1 -- 172.21.15.102:0/2386069653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd810100190 msgr2=0x7fd8101005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:43.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.554+0000 7fd8179d3700 1 --2- 172.21.15.102:0/2386069653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd810100190 0x7fd8101005a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd80800b0a0 tx=0x7fd80800b3b0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.554+0000 7fd8179d3700 1 -- 172.21.15.102:0/2386069653 shutdown_connections 2024-03-26T18:57:43.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.554+0000 7fd8179d3700 1 --2- 172.21.15.102:0/2386069653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810101390 0x7fd8101017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.554+0000 7fd8179d3700 1 --2- 172.21.15.102:0/2386069653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd810100190 0x7fd8101005a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.554+0000 7fd8179d3700 1 -- 172.21.15.102:0/2386069653 >> 172.21.15.102:0/2386069653 conn(0x7fd8100fb720 msgr2=0x7fd8100fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:43.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.554+0000 7fd8179d3700 1 -- 172.21.15.102:0/2386069653 shutdown_connections 2024-03-26T18:57:43.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.554+0000 7fd8179d3700 1 -- 172.21.15.102:0/2386069653 wait complete. 2024-03-26T18:57:43.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd8179d3700 1 Processor -- start 2024-03-26T18:57:43.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd8179d3700 1 -- start start 2024-03-26T18:57:43.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd8179d3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810100190 0x7fd810195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd8179d3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd810101390 0x7fd810195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd8179d3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101964f0 con 0x7fd810101390 2024-03-26T18:57:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd8179d3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810196630 con 0x7fd810100190 2024-03-26T18:57:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd814f6e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd810101390 0x7fd810195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd814f6e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd810101390 0x7fd810195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34748/0 (socket says 172.21.15.102:34748) 2024-03-26T18:57:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd81576f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810100190 0x7fd810195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd814f6e700 1 -- 172.21.15.102:0/1338306640 learned_addr learned my addr 172.21.15.102:0/1338306640 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd81576f700 1 -- 172.21.15.102:0/1338306640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd810101390 msgr2=0x7fd810195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd81576f700 1 --2- 172.21.15.102:0/1338306640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd810101390 0x7fd810195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd81576f700 1 -- 172.21.15.102:0/1338306640 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd808009d00 con 0x7fd810100190 2024-03-26T18:57:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.555+0000 7fd814f6e700 1 --2- 172.21.15.102:0/1338306640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd810101390 0x7fd810195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T18:57:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.556+0000 7fd81576f700 1 --2- 172.21.15.102:0/1338306640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810100190 0x7fd810195990 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd808015040 tx=0x7fd808003c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.556+0000 7fd8067fc700 1 -- 172.21.15.102:0/1338306640 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd808021070 con 0x7fd810100190 2024-03-26T18:57:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.556+0000 7fd8179d3700 1 -- 172.21.15.102:0/1338306640 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81019b080 con 0x7fd810100190 2024-03-26T18:57:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.556+0000 7fd8067fc700 1 -- 172.21.15.102:0/1338306640 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd8080096a0 con 0x7fd810100190 2024-03-26T18:57:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.556+0000 7fd8067fc700 1 -- 172.21.15.102:0/1338306640 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80800e040 con 0x7fd810100190 2024-03-26T18:57:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.556+0000 7fd8179d3700 1 -- 172.21.15.102:0/1338306640 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81019b5c0 con 0x7fd810100190 2024-03-26T18:57:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.557+0000 7fd8179d3700 1 -- 172.21.15.102:0/1338306640 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd810108ce0 con 0x7fd810100190 2024-03-26T18:57:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.558+0000 7fd8067fc700 1 -- 172.21.15.102:0/1338306640 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7fd808004250 con 0x7fd810100190 2024-03-26T18:57:43.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.558+0000 7fd8067fc700 1 --2- 172.21.15.102:0/1338306640 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd7fc06e5a0 0x7fd7fc070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:43.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.558+0000 7fd8067fc700 1 -- 172.21.15.102:0/1338306640 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd80808f9e0 con 0x7fd810100190 2024-03-26T18:57:43.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.558+0000 7fd814f6e700 1 --2- 172.21.15.102:0/1338306640 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd7fc06e5a0 0x7fd7fc070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:43.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.558+0000 7fd814f6e700 1 --2- 172.21.15.102:0/1338306640 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd7fc06e5a0 0x7fd7fc070a50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd800011550 tx=0x7fd800009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:43.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.563+0000 7fd8067fc700 1 -- 172.21.15.102:0/1338306640 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd80805c4f0 con 0x7fd810100190 2024-03-26T18:57:43.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.790+0000 7fd8179d3700 1 -- 172.21.15.102:0/1338306640 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd810105ce0 con 0x7fd7fc06e5a0 2024-03-26T18:57:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.792+0000 7fd8067fc700 1 -- 172.21.15.102:0/1338306640 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7fd810105ce0 con 0x7fd7fc06e5a0 2024-03-26T18:57:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.794+0000 7fd8179d3700 1 -- 172.21.15.102:0/1338306640 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd7fc06e5a0 msgr2=0x7fd7fc070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.794+0000 7fd8179d3700 1 --2- 172.21.15.102:0/1338306640 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd7fc06e5a0 0x7fd7fc070a50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd800011550 tx=0x7fd800009260 comp rx=0 tx=0).stop 2024-03-26T18:57:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.794+0000 7fd8179d3700 1 -- 172.21.15.102:0/1338306640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810100190 msgr2=0x7fd810195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.794+0000 7fd8179d3700 1 --2- 172.21.15.102:0/1338306640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810100190 0x7fd810195990 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd808015040 tx=0x7fd808003c30 comp rx=0 tx=0).stop 2024-03-26T18:57:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.794+0000 7fd8179d3700 1 -- 172.21.15.102:0/1338306640 shutdown_connections 2024-03-26T18:57:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.794+0000 7fd8179d3700 1 --2- 172.21.15.102:0/1338306640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810100190 0x7fd810195990 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.794+0000 7fd8179d3700 1 --2- 172.21.15.102:0/1338306640 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd7fc06e5a0 0x7fd7fc070a50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.794+0000 7fd8179d3700 1 --2- 172.21.15.102:0/1338306640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd810101390 0x7fd810195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.794+0000 7fd8179d3700 1 -- 172.21.15.102:0/1338306640 >> 172.21.15.102:0/1338306640 conn(0x7fd8100fb720 msgr2=0x7fd8101045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:43.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.794+0000 7fd8179d3700 1 -- 172.21.15.102:0/1338306640 shutdown_connections 2024-03-26T18:57:43.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.794+0000 7fd8179d3700 1 -- 172.21.15.102:0/1338306640 wait complete. 2024-03-26T18:57:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.898+0000 7feb6210f700 1 -- 172.21.15.102:0/742732980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c103500 msgr2=0x7feb5c1058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.898+0000 7feb6210f700 1 --2- 172.21.15.102:0/742732980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c103500 0x7feb5c1058e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb50009910 tx=0x7feb50009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.898+0000 7feb6210f700 1 -- 172.21.15.102:0/742732980 shutdown_connections 2024-03-26T18:57:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.898+0000 7feb6210f700 1 --2- 172.21.15.102:0/742732980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c103500 0x7feb5c1058e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.898+0000 7feb6210f700 1 --2- 172.21.15.102:0/742732980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb5c100be0 0x7feb5c102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.898+0000 7feb6210f700 1 -- 172.21.15.102:0/742732980 >> 172.21.15.102:0/742732980 conn(0x7feb5c0fa600 msgr2=0x7feb5c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.899+0000 7feb6210f700 1 -- 172.21.15.102:0/742732980 shutdown_connections 2024-03-26T18:57:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.899+0000 7feb6210f700 1 -- 172.21.15.102:0/742732980 wait complete. 2024-03-26T18:57:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.899+0000 7feb6210f700 1 Processor -- start 2024-03-26T18:57:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.899+0000 7feb6210f700 1 -- start start 2024-03-26T18:57:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.899+0000 7feb6210f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c100be0 0x7feb5c195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.899+0000 7feb6210f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb5c103500 0x7feb5c195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb6210f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c1964e0 con 0x7feb5c100be0 2024-03-26T18:57:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb6210f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c196620 con 0x7feb5c103500 2024-03-26T18:57:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb5b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c100be0 0x7feb5c195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb5b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c100be0 0x7feb5c195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34760/0 (socket says 172.21.15.102:34760) 2024-03-26T18:57:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb5b7fe700 1 -- 172.21.15.102:0/2317784818 learned_addr learned my addr 172.21.15.102:0/2317784818 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb5affd700 1 --2- 172.21.15.102:0/2317784818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb5c103500 0x7feb5c195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb5b7fe700 1 -- 172.21.15.102:0/2317784818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb5c103500 msgr2=0x7feb5c195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb5b7fe700 1 --2- 172.21.15.102:0/2317784818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb5c103500 0x7feb5c195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb5b7fe700 1 -- 172.21.15.102:0/2317784818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb500095f0 con 0x7feb5c100be0 2024-03-26T18:57:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb5b7fe700 1 --2- 172.21.15.102:0/2317784818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c100be0 0x7feb5c195980 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00d900 tx=0x7feb4c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb58ff9700 1 -- 172.21.15.102:0/2317784818 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb4c0041f0 con 0x7feb5c100be0 2024-03-26T18:57:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb58ff9700 1 -- 172.21.15.102:0/2317784818 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feb4c004ce0 con 0x7feb5c100be0 2024-03-26T18:57:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb58ff9700 1 -- 172.21.15.102:0/2317784818 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb4c00b750 con 0x7feb5c100be0 2024-03-26T18:57:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb6210f700 1 -- 172.21.15.102:0/2317784818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5c19b0d0 con 0x7feb5c100be0 2024-03-26T18:57:43.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.900+0000 7feb6210f700 1 -- 172.21.15.102:0/2317784818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5c19b5d0 con 0x7feb5c100be0 2024-03-26T18:57:43.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.901+0000 7feb6210f700 1 -- 172.21.15.102:0/2317784818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5c18fb70 con 0x7feb5c100be0 2024-03-26T18:57:43.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.902+0000 7feb58ff9700 1 -- 172.21.15.102:0/2317784818 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7feb4c004e50 con 0x7feb5c100be0 2024-03-26T18:57:43.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.902+0000 7feb58ff9700 1 --2- 172.21.15.102:0/2317784818 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feb4406e5a0 0x7feb44070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:43.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.902+0000 7feb58ff9700 1 -- 172.21.15.102:0/2317784818 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7feb4c08e580 con 0x7feb5c100be0 2024-03-26T18:57:43.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.902+0000 7feb5affd700 1 --2- 172.21.15.102:0/2317784818 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feb4406e5a0 0x7feb44070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:43.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.903+0000 7feb5affd700 1 --2- 172.21.15.102:0/2317784818 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feb4406e5a0 0x7feb44070a50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feb5000b5c0 tx=0x7feb50005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:43.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:43.907+0000 7feb58ff9700 1 -- 172.21.15.102:0/2317784818 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feb4c05b090 con 0x7feb5c100be0 2024-03-26T18:57:44.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.123+0000 7feb6210f700 1 -- 172.21.15.102:0/2317784818 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7feb5c061190 con 0x7feb4406e5a0 2024-03-26T18:57:44.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.134+0000 7feb58ff9700 1 -- 172.21.15.102:0/2317784818 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7feb5c061190 con 0x7feb4406e5a0 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (9m) 61s ago 10m 24.9M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (10m) 61s ago 10m 7772k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (9m) 47s ago 9m 7688k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 starting - - - - 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 starting - - - - 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (9m) 61s ago 10m 84.5M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (5m) 61s ago 5m 133M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (5m) 61s ago 5m 14.5M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (5m) 47s ago 5m 15.0M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (5m) 47s ago 5m 120M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (2m) 61s ago 12m 473M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (89s) 47s ago 9m 447M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T18:57:44.128 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (75s) 61s ago 12m 27.7M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T18:57:44.129 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (53s) 47s ago 9m 20.3M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T18:57:44.129 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (10m) 61s ago 10m 16.3M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:57:44.129 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (9m) 47s ago 9m 15.5M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:57:44.129 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (8m) 61s ago 8m 220M 1216M 18.2.0 dc2bc1663786 3e2065fb5fbc 2024-03-26T18:57:44.129 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (8m) 61s ago 8m 244M 1216M 18.2.0 dc2bc1663786 0bf773515cc1 2024-03-26T18:57:44.129 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (8m) 61s ago 8m 194M 1216M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:57:44.129 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (7m) 47s ago 7m 272M 2240M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:57:44.129 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (7m) 47s ago 7m 250M 2240M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:57:44.129 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (6m) 47s ago 6m 223M 2240M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:57:44.129 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (94s) 61s ago 10m 62.0M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T18:57:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.137+0000 7feb6210f700 1 -- 172.21.15.102:0/2317784818 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feb4406e5a0 msgr2=0x7feb44070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.137+0000 7feb6210f700 1 --2- 172.21.15.102:0/2317784818 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feb4406e5a0 0x7feb44070a50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feb5000b5c0 tx=0x7feb50005ee0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.137+0000 7feb6210f700 1 -- 172.21.15.102:0/2317784818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c100be0 msgr2=0x7feb5c195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.137+0000 7feb6210f700 1 --2- 172.21.15.102:0/2317784818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c100be0 0x7feb5c195980 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00d900 tx=0x7feb4c00dc10 comp rx=0 tx=0).stop 2024-03-26T18:57:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.138+0000 7feb6210f700 1 -- 172.21.15.102:0/2317784818 shutdown_connections 2024-03-26T18:57:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.138+0000 7feb6210f700 1 --2- 172.21.15.102:0/2317784818 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feb4406e5a0 0x7feb44070a50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.138+0000 7feb6210f700 1 --2- 172.21.15.102:0/2317784818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c100be0 0x7feb5c195980 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.138+0000 7feb6210f700 1 --2- 172.21.15.102:0/2317784818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb5c103500 0x7feb5c195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.138+0000 7feb6210f700 1 -- 172.21.15.102:0/2317784818 >> 172.21.15.102:0/2317784818 conn(0x7feb5c0fa600 msgr2=0x7feb5c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.138+0000 7feb6210f700 1 -- 172.21.15.102:0/2317784818 shutdown_connections 2024-03-26T18:57:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.138+0000 7feb6210f700 1 -- 172.21.15.102:0/2317784818 wait complete. 2024-03-26T18:57:44.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.258+0000 7fe566068700 1 -- 172.21.15.102:0/525377990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe560101390 msgr2=0x7fe5601017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:44.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.258+0000 7fe566068700 1 --2- 172.21.15.102:0/525377990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe560101390 0x7fe5601017e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe548009910 tx=0x7fe548009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:44.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.258+0000 7fe566068700 1 -- 172.21.15.102:0/525377990 shutdown_connections 2024-03-26T18:57:44.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.258+0000 7fe566068700 1 --2- 172.21.15.102:0/525377990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe560101390 0x7fe5601017e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.258+0000 7fe566068700 1 --2- 172.21.15.102:0/525377990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe560100190 0x7fe5601005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.258+0000 7fe566068700 1 -- 172.21.15.102:0/525377990 >> 172.21.15.102:0/525377990 conn(0x7fe5600fb740 msgr2=0x7fe5600fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:44.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.258+0000 7fe566068700 1 -- 172.21.15.102:0/525377990 shutdown_connections 2024-03-26T18:57:44.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.258+0000 7fe566068700 1 -- 172.21.15.102:0/525377990 wait complete. 2024-03-26T18:57:44.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.258+0000 7fe566068700 1 Processor -- start 2024-03-26T18:57:44.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe566068700 1 -- start start 2024-03-26T18:57:44.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe566068700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe560100190 0x7fe560074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:44.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe566068700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe560101390 0x7fe560073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe566068700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe560073680 con 0x7fe560101390 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe566068700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5600737c0 con 0x7fe560100190 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe560101390 0x7fe560073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe560101390 0x7fe560073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34774/0 (socket says 172.21.15.102:34774) 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55effd700 1 -- 172.21.15.102:0/4025451691 learned_addr learned my addr 172.21.15.102:0/4025451691 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55effd700 1 -- 172.21.15.102:0/4025451691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe560100190 msgr2=0x7fe560074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55f7fe700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe560100190 0x7fe560074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55effd700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe560100190 0x7fe560074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55effd700 1 -- 172.21.15.102:0/4025451691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5480095f0 con 0x7fe560101390 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55f7fe700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe560100190 0x7fe560074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55effd700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe560101390 0x7fe560073140 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe54800bd70 tx=0x7fe54800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55cff9700 1 -- 172.21.15.102:0/4025451691 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54801d070 con 0x7fe560101390 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55cff9700 1 -- 172.21.15.102:0/4025451691 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe548022470 con 0x7fe560101390 2024-03-26T18:57:44.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.259+0000 7fe55cff9700 1 -- 172.21.15.102:0/4025451691 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54800f670 con 0x7fe560101390 2024-03-26T18:57:44.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.260+0000 7fe566068700 1 -- 172.21.15.102:0/4025451691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe560073a40 con 0x7fe560101390 2024-03-26T18:57:44.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.260+0000 7fe566068700 1 -- 172.21.15.102:0/4025451691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe560073f30 con 0x7fe560101390 2024-03-26T18:57:44.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.260+0000 7fe566068700 1 -- 172.21.15.102:0/4025451691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe560108d80 con 0x7fe560101390 2024-03-26T18:57:44.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.261+0000 7fe55cff9700 1 -- 172.21.15.102:0/4025451691 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7fe548022a50 con 0x7fe560101390 2024-03-26T18:57:44.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.261+0000 7fe55cff9700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe54c06e5a0 0x7fe54c070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:44.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.261+0000 7fe55cff9700 1 -- 172.21.15.102:0/4025451691 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe54808fc40 con 0x7fe560101390 2024-03-26T18:57:44.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.261+0000 7fe55f7fe700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe54c06e5a0 0x7fe54c070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:44.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.262+0000 7fe55f7fe700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe54c06e5a0 0x7fe54c070a50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe5601011f0 tx=0x7fe5500073c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:44.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.266+0000 7fe55cff9700 1 -- 172.21.15.102:0/4025451691 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe54805c880 con 0x7fe560101390 2024-03-26T18:57:44.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:44 smithi102 ceph-mon[78322]: pgmap v36: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:44.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:44 smithi190 ceph-mon[65689]: pgmap v36: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:44.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.560+0000 7fe566068700 1 -- 172.21.15.102:0/4025451691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe56019dee0 con 0x7fe560101390 2024-03-26T18:57:44.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.560+0000 7fe55cff9700 1 -- 172.21.15.102:0/4025451691 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7fe5480272c0 con 0x7fe560101390 2024-03-26T18:57:44.554 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:57:44.554 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:57:44.554 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:57:44.554 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:57:44.554 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:57:44.554 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:57:44.554 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:57:44.554 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:57:44.554 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-03-26T18:57:44.554 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:57:44.555 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:57:44.555 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:57:44.555 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:57:44.555 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:57:44.555 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 10, 2024-03-26T18:57:44.555 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 4 2024-03-26T18:57:44.555 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:57:44.555 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:57:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.562+0000 7fe566068700 1 -- 172.21.15.102:0/4025451691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe54c06e5a0 msgr2=0x7fe54c070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.562+0000 7fe566068700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe54c06e5a0 0x7fe54c070a50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe5601011f0 tx=0x7fe5500073c0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.563+0000 7fe566068700 1 -- 172.21.15.102:0/4025451691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe560101390 msgr2=0x7fe560073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.563+0000 7fe566068700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe560101390 0x7fe560073140 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe54800bd70 tx=0x7fe54800be50 comp rx=0 tx=0).stop 2024-03-26T18:57:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.563+0000 7fe566068700 1 -- 172.21.15.102:0/4025451691 shutdown_connections 2024-03-26T18:57:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.563+0000 7fe566068700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe560100190 0x7fe560074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.563+0000 7fe566068700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe54c06e5a0 0x7fe54c070a50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.563+0000 7fe566068700 1 --2- 172.21.15.102:0/4025451691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe560101390 0x7fe560073140 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.563+0000 7fe566068700 1 -- 172.21.15.102:0/4025451691 >> 172.21.15.102:0/4025451691 conn(0x7fe5600fb740 msgr2=0x7fe5601045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.563+0000 7fe566068700 1 -- 172.21.15.102:0/4025451691 shutdown_connections 2024-03-26T18:57:44.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.563+0000 7fe566068700 1 -- 172.21.15.102:0/4025451691 wait complete. 2024-03-26T18:57:44.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.667+0000 7f9c2b733700 1 -- 172.21.15.102:0/619986871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c24100ca0 msgr2=0x7f9c24103080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:44.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.667+0000 7f9c2b733700 1 --2- 172.21.15.102:0/619986871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c24100ca0 0x7f9c24103080 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009910 tx=0x7f9c20009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:44.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.667+0000 7f9c2b733700 1 -- 172.21.15.102:0/619986871 shutdown_connections 2024-03-26T18:57:44.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.667+0000 7f9c2b733700 1 --2- 172.21.15.102:0/619986871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c241035c0 0x7f9c241059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.667+0000 7f9c2b733700 1 --2- 172.21.15.102:0/619986871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c24100ca0 0x7f9c24103080 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.667+0000 7f9c2b733700 1 -- 172.21.15.102:0/619986871 >> 172.21.15.102:0/619986871 conn(0x7f9c240fa5e0 msgr2=0x7f9c240fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:44.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.667+0000 7f9c2b733700 1 -- 172.21.15.102:0/619986871 shutdown_connections 2024-03-26T18:57:44.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.667+0000 7f9c2b733700 1 -- 172.21.15.102:0/619986871 wait complete. 2024-03-26T18:57:44.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c2b733700 1 Processor -- start 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c2b733700 1 -- start start 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c2b733700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c24100ca0 0x7f9c24195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c2b733700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c241035c0 0x7f9c24195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c2b733700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c241964b0 con 0x7f9c241035c0 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c2b733700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c241965f0 con 0x7f9c24100ca0 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c28cce700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c241035c0 0x7f9c24195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c28cce700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c241035c0 0x7f9c24195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34790/0 (socket says 172.21.15.102:34790) 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c28cce700 1 -- 172.21.15.102:0/1279419893 learned_addr learned my addr 172.21.15.102:0/1279419893 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c294cf700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c24100ca0 0x7f9c24195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c28cce700 1 -- 172.21.15.102:0/1279419893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c24100ca0 msgr2=0x7f9c24195950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c28cce700 1 --2- 172.21.15.102:0/1279419893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c24100ca0 0x7f9c24195950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.668+0000 7f9c28cce700 1 -- 172.21.15.102:0/1279419893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c200095f0 con 0x7f9c241035c0 2024-03-26T18:57:44.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.669+0000 7f9c28cce700 1 --2- 172.21.15.102:0/1279419893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c241035c0 0x7f9c24195e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400e920 tx=0x7f9c1400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:44.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.669+0000 7f9c1a7fc700 1 -- 172.21.15.102:0/1279419893 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c1400c9d0 con 0x7f9c241035c0 2024-03-26T18:57:44.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.669+0000 7f9c2b733700 1 -- 172.21.15.102:0/1279419893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2419b0a0 con 0x7f9c241035c0 2024-03-26T18:57:44.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.669+0000 7f9c1a7fc700 1 -- 172.21.15.102:0/1279419893 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9c14010430 con 0x7f9c241035c0 2024-03-26T18:57:44.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.669+0000 7f9c1a7fc700 1 -- 172.21.15.102:0/1279419893 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c14018650 con 0x7f9c241035c0 2024-03-26T18:57:44.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.669+0000 7f9c2b733700 1 -- 172.21.15.102:0/1279419893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2419b5f0 con 0x7f9c241035c0 2024-03-26T18:57:44.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.670+0000 7f9c2b733700 1 -- 172.21.15.102:0/1279419893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2418fb70 con 0x7f9c241035c0 2024-03-26T18:57:44.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.670+0000 7f9c1a7fc700 1 -- 172.21.15.102:0/1279419893 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f9c140105a0 con 0x7f9c241035c0 2024-03-26T18:57:44.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.671+0000 7f9c1a7fc700 1 --2- 172.21.15.102:0/1279419893 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c1006e5a0 0x7f9c10070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:44.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.671+0000 7f9c1a7fc700 1 -- 172.21.15.102:0/1279419893 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9c14014070 con 0x7f9c241035c0 2024-03-26T18:57:44.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.671+0000 7f9c294cf700 1 --2- 172.21.15.102:0/1279419893 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c1006e5a0 0x7f9c10070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:44.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.671+0000 7f9c294cf700 1 --2- 172.21.15.102:0/1279419893 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c1006e5a0 0x7f9c10070a50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009fd0 tx=0x7f9c20005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:44.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.675+0000 7f9c1a7fc700 1 -- 172.21.15.102:0/1279419893 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9c14053740 con 0x7f9c241035c0 2024-03-26T18:57:44.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.939+0000 7f9c2b733700 1 -- 172.21.15.102:0/1279419893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9c2402ce10 con 0x7f9c241035c0 2024-03-26T18:57:44.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.940+0000 7f9c1a7fc700 1 -- 172.21.15.102:0/1279419893 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1924 (secure 0 0 0) 0x7f9c1405b430 con 0x7f9c241035c0 2024-03-26T18:57:44.934 INFO:teuthology.orchestra.run.smithi102.stdout:e19 2024-03-26T18:57:44.934 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:57:44.934 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:57:44.934 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:57:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:57:44.934 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:57:44.934 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:57:44.934 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 19 2024-03-26T18:57:44.934 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:57:44.934 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:55:27.824831+0000 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:57:44.935 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 0 members: 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:57:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:57:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:57:44.937 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:57:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:57:44.937 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:57:44.937 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:57:44.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.943+0000 7f9c2b733700 1 -- 172.21.15.102:0/1279419893 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c1006e5a0 msgr2=0x7f9c10070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.943+0000 7f9c2b733700 1 --2- 172.21.15.102:0/1279419893 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c1006e5a0 0x7f9c10070a50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009fd0 tx=0x7f9c20005940 comp rx=0 tx=0).stop 2024-03-26T18:57:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.943+0000 7f9c2b733700 1 -- 172.21.15.102:0/1279419893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c241035c0 msgr2=0x7f9c24195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.943+0000 7f9c2b733700 1 --2- 172.21.15.102:0/1279419893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c241035c0 0x7f9c24195e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400e920 tx=0x7f9c1400ec30 comp rx=0 tx=0).stop 2024-03-26T18:57:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.943+0000 7f9c2b733700 1 -- 172.21.15.102:0/1279419893 shutdown_connections 2024-03-26T18:57:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.943+0000 7f9c2b733700 1 --2- 172.21.15.102:0/1279419893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c24100ca0 0x7f9c24195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.943+0000 7f9c2b733700 1 --2- 172.21.15.102:0/1279419893 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c1006e5a0 0x7f9c10070a50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.943+0000 7f9c2b733700 1 --2- 172.21.15.102:0/1279419893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c241035c0 0x7f9c24195e90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.943+0000 7f9c2b733700 1 -- 172.21.15.102:0/1279419893 >> 172.21.15.102:0/1279419893 conn(0x7f9c240fa5e0 msgr2=0x7f9c240fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.943+0000 7f9c2b733700 1 -- 172.21.15.102:0/1279419893 shutdown_connections 2024-03-26T18:57:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:44.943+0000 7f9c2b733700 1 -- 172.21.15.102:0/1279419893 wait complete. 2024-03-26T18:57:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 19 2024-03-26T18:57:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.059+0000 7f491ea7d700 1 -- 172.21.15.102:0/3257803936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4918100cb0 msgr2=0x7f4918103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.059+0000 7f491ea7d700 1 --2- 172.21.15.102:0/3257803936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4918100cb0 0x7f4918103090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f490c009910 tx=0x7f490c009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 -- 172.21.15.102:0/3257803936 shutdown_connections 2024-03-26T18:57:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 --2- 172.21.15.102:0/3257803936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49181035d0 0x7f49181059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 --2- 172.21.15.102:0/3257803936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4918100cb0 0x7f4918103090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 -- 172.21.15.102:0/3257803936 >> 172.21.15.102:0/3257803936 conn(0x7f49180fa610 msgr2=0x7f49180fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 -- 172.21.15.102:0/3257803936 shutdown_connections 2024-03-26T18:57:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 -- 172.21.15.102:0/3257803936 wait complete. 2024-03-26T18:57:45.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 Processor -- start 2024-03-26T18:57:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 -- start start 2024-03-26T18:57:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4918100cb0 0x7f4918195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49181035d0 0x7f4918195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4918196470 con 0x7f4918100cb0 2024-03-26T18:57:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491ea7d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49181965b0 con 0x7f49181035d0 2024-03-26T18:57:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491c819700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4918100cb0 0x7f4918195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491c819700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4918100cb0 0x7f4918195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34800/0 (socket says 172.21.15.102:34800) 2024-03-26T18:57:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.060+0000 7f491c819700 1 -- 172.21.15.102:0/902566234 learned_addr learned my addr 172.21.15.102:0/902566234 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.061+0000 7f4917fff700 1 --2- 172.21.15.102:0/902566234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49181035d0 0x7f4918195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.061+0000 7f491c819700 1 -- 172.21.15.102:0/902566234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49181035d0 msgr2=0x7f4918195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.061+0000 7f491c819700 1 --2- 172.21.15.102:0/902566234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49181035d0 0x7f4918195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.061+0000 7f491c819700 1 -- 172.21.15.102:0/902566234 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490c0095f0 con 0x7f4918100cb0 2024-03-26T18:57:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.061+0000 7f491c819700 1 --2- 172.21.15.102:0/902566234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4918100cb0 0x7f4918195910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f490c000c00 tx=0x7f490c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.061+0000 7f4915ffb700 1 -- 172.21.15.102:0/902566234 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f490c01d070 con 0x7f4918100cb0 2024-03-26T18:57:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.061+0000 7f4915ffb700 1 -- 172.21.15.102:0/902566234 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f490c005dc0 con 0x7f4918100cb0 2024-03-26T18:57:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.061+0000 7f491ea7d700 1 -- 172.21.15.102:0/902566234 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491819b000 con 0x7f4918100cb0 2024-03-26T18:57:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.061+0000 7f4915ffb700 1 -- 172.21.15.102:0/902566234 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f490c0226a0 con 0x7f4918100cb0 2024-03-26T18:57:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.061+0000 7f491ea7d700 1 -- 172.21.15.102:0/902566234 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491819b4f0 con 0x7f4918100cb0 2024-03-26T18:57:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.062+0000 7f491ea7d700 1 -- 172.21.15.102:0/902566234 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491818fb80 con 0x7f4918100cb0 2024-03-26T18:57:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.063+0000 7f4915ffb700 1 -- 172.21.15.102:0/902566234 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f490c0038f0 con 0x7f4918100cb0 2024-03-26T18:57:45.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.063+0000 7f4915ffb700 1 --2- 172.21.15.102:0/902566234 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f490806e590 0x7f4908070a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:45.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.063+0000 7f4915ffb700 1 -- 172.21.15.102:0/902566234 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f490c08fac0 con 0x7f4918100cb0 2024-03-26T18:57:45.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.063+0000 7f4917fff700 1 --2- 172.21.15.102:0/902566234 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f490806e590 0x7f4908070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:45.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.064+0000 7f4917fff700 1 --2- 172.21.15.102:0/902566234 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f490806e590 0x7f4908070a40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4904009c60 tx=0x7f4904009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:45.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.068+0000 7f4915ffb700 1 -- 172.21.15.102:0/902566234 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f490c05c650 con 0x7f4918100cb0 2024-03-26T18:57:45.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.296+0000 7f491ea7d700 1 -- 172.21.15.102:0/902566234 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4918061190 con 0x7f490806e590 2024-03-26T18:57:45.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.297+0000 7f4915ffb700 1 -- 172.21.15.102:0/902566234 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f4918061190 con 0x7f490806e590 2024-03-26T18:57:45.290 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:57:45.290 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:57:45.290 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:57:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:57:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T18:57:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T18:57:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "mon" 2024-03-26T18:57:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T18:57:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "5/23 daemons upgraded", 2024-03-26T18:57:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading crash daemons", 2024-03-26T18:57:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:57:45.291 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:57:45.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.299+0000 7f491ea7d700 1 -- 172.21.15.102:0/902566234 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f490806e590 msgr2=0x7f4908070a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:45.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.299+0000 7f491ea7d700 1 --2- 172.21.15.102:0/902566234 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f490806e590 0x7f4908070a40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4904009c60 tx=0x7f4904009260 comp rx=0 tx=0).stop 2024-03-26T18:57:45.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.299+0000 7f491ea7d700 1 -- 172.21.15.102:0/902566234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4918100cb0 msgr2=0x7f4918195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:45.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.299+0000 7f491ea7d700 1 --2- 172.21.15.102:0/902566234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4918100cb0 0x7f4918195910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f490c000c00 tx=0x7f490c00f690 comp rx=0 tx=0).stop 2024-03-26T18:57:45.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.299+0000 7f491ea7d700 1 -- 172.21.15.102:0/902566234 shutdown_connections 2024-03-26T18:57:45.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.299+0000 7f491ea7d700 1 --2- 172.21.15.102:0/902566234 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f490806e590 0x7f4908070a40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.299+0000 7f491ea7d700 1 --2- 172.21.15.102:0/902566234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4918100cb0 0x7f4918195910 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.299+0000 7f491ea7d700 1 --2- 172.21.15.102:0/902566234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49181035d0 0x7f4918195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.299+0000 7f491ea7d700 1 -- 172.21.15.102:0/902566234 >> 172.21.15.102:0/902566234 conn(0x7f49180fa610 msgr2=0x7f49180fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:45.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.300+0000 7f491ea7d700 1 -- 172.21.15.102:0/902566234 shutdown_connections 2024-03-26T18:57:45.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.300+0000 7f491ea7d700 1 -- 172.21.15.102:0/902566234 wait complete. 2024-03-26T18:57:45.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.411+0000 7fce99120700 1 -- 172.21.15.102:0/4284040531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce940ff050 msgr2=0x7fce940ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:45.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.411+0000 7fce99120700 1 --2- 172.21.15.102:0/4284040531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce940ff050 0x7fce940ff460 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fce84009910 tx=0x7fce84009c20 comp rx=0 tx=0).stop 2024-03-26T18:57:45.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.411+0000 7fce99120700 1 -- 172.21.15.102:0/4284040531 shutdown_connections 2024-03-26T18:57:45.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.411+0000 7fce99120700 1 --2- 172.21.15.102:0/4284040531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce94100250 0x7fce941006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.411+0000 7fce99120700 1 --2- 172.21.15.102:0/4284040531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce940ff050 0x7fce940ff460 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.411+0000 7fce99120700 1 -- 172.21.15.102:0/4284040531 >> 172.21.15.102:0/4284040531 conn(0x7fce940fa5e0 msgr2=0x7fce940fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:45.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.411+0000 7fce99120700 1 -- 172.21.15.102:0/4284040531 shutdown_connections 2024-03-26T18:57:45.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.411+0000 7fce99120700 1 -- 172.21.15.102:0/4284040531 wait complete. 2024-03-26T18:57:45.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.412+0000 7fce99120700 1 Processor -- start 2024-03-26T18:57:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.412+0000 7fce99120700 1 -- start start 2024-03-26T18:57:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.412+0000 7fce99120700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce940ff050 0x7fce94194890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.412+0000 7fce99120700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce94100250 0x7fce94194dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.412+0000 7fce99120700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce941953f0 con 0x7fce94100250 2024-03-26T18:57:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.412+0000 7fce99120700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce94195530 con 0x7fce940ff050 2024-03-26T18:57:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce9259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce94100250 0x7fce94194dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce9259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce94100250 0x7fce94194dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34828/0 (socket says 172.21.15.102:34828) 2024-03-26T18:57:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce9259c700 1 -- 172.21.15.102:0/931146713 learned_addr learned my addr 172.21.15.102:0/931146713 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:57:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce92d9d700 1 --2- 172.21.15.102:0/931146713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce940ff050 0x7fce94194890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce9259c700 1 -- 172.21.15.102:0/931146713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce940ff050 msgr2=0x7fce94194890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce9259c700 1 --2- 172.21.15.102:0/931146713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce940ff050 0x7fce94194890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce9259c700 1 -- 172.21.15.102:0/931146713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce840095f0 con 0x7fce94100250 2024-03-26T18:57:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce9259c700 1 --2- 172.21.15.102:0/931146713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce94100250 0x7fce94194dd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fce8800c8a0 tx=0x7fce8800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce7bfff700 1 -- 172.21.15.102:0/931146713 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce880078e0 con 0x7fce94100250 2024-03-26T18:57:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce7bfff700 1 -- 172.21.15.102:0/931146713 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fce8800f450 con 0x7fce94100250 2024-03-26T18:57:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce99120700 1 -- 172.21.15.102:0/931146713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce94199fe0 con 0x7fce94100250 2024-03-26T18:57:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce7bfff700 1 -- 172.21.15.102:0/931146713 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce8800e5e0 con 0x7fce94100250 2024-03-26T18:57:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.413+0000 7fce99120700 1 -- 172.21.15.102:0/931146713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce9419a500 con 0x7fce94100250 2024-03-26T18:57:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.414+0000 7fce99120700 1 -- 172.21.15.102:0/931146713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce94107ba0 con 0x7fce94100250 2024-03-26T18:57:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.415+0000 7fce7bfff700 1 -- 172.21.15.102:0/931146713 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 36) v1 ==== 92321+0+0 (secure 0 0 0) 0x7fce8800f5c0 con 0x7fce94100250 2024-03-26T18:57:45.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.415+0000 7fce7bfff700 1 --2- 172.21.15.102:0/931146713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce7c06e5a0 0x7fce7c070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:57:45.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.415+0000 7fce7bfff700 1 -- 172.21.15.102:0/931146713 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fce8808e5f0 con 0x7fce94100250 2024-03-26T18:57:45.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.415+0000 7fce92d9d700 1 --2- 172.21.15.102:0/931146713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce7c06e5a0 0x7fce7c070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:57:45.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.416+0000 7fce92d9d700 1 --2- 172.21.15.102:0/931146713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce7c06e5a0 0x7fce7c070a50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fce84006010 tx=0x7fce8400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:57:45.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.420+0000 7fce7bfff700 1 -- 172.21.15.102:0/931146713 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fce8805b100 con 0x7fce94100250 2024-03-26T18:57:45.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:45 smithi190 ceph-mon[65689]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:45.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:45 smithi190 ceph-mon[65689]: from='client.44129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:45.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:45 smithi190 ceph-mon[65689]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:45.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:45 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/4025451691' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:45.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:45.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:45.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:45 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1279419893' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:57:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:45 smithi102 ceph-mon[78322]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:45 smithi102 ceph-mon[78322]: from='client.44129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:45 smithi102 ceph-mon[78322]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:45 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/4025451691' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:45.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:45 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1279419893' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:57:45.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.712+0000 7fce99120700 1 -- 172.21.15.102:0/931146713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fce94107d60 con 0x7fce94100250 2024-03-26T18:57:45.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.712+0000 7fce7bfff700 1 -- 172.21.15.102:0/931146713 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fce8805ac90 con 0x7fce94100250 2024-03-26T18:57:45.706 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T18:57:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.716+0000 7fce99120700 1 -- 172.21.15.102:0/931146713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce7c06e5a0 msgr2=0x7fce7c070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.716+0000 7fce99120700 1 --2- 172.21.15.102:0/931146713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce7c06e5a0 0x7fce7c070a50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fce84006010 tx=0x7fce8400b540 comp rx=0 tx=0).stop 2024-03-26T18:57:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.716+0000 7fce99120700 1 -- 172.21.15.102:0/931146713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce94100250 msgr2=0x7fce94194dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:57:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.716+0000 7fce99120700 1 --2- 172.21.15.102:0/931146713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce94100250 0x7fce94194dd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fce8800c8a0 tx=0x7fce8800cbb0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.716+0000 7fce99120700 1 -- 172.21.15.102:0/931146713 shutdown_connections 2024-03-26T18:57:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.716+0000 7fce99120700 1 --2- 172.21.15.102:0/931146713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce940ff050 0x7fce94194890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.716+0000 7fce99120700 1 --2- 172.21.15.102:0/931146713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce7c06e5a0 0x7fce7c070a50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.716+0000 7fce99120700 1 --2- 172.21.15.102:0/931146713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce94100250 0x7fce94194dd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:57:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.716+0000 7fce99120700 1 -- 172.21.15.102:0/931146713 >> 172.21.15.102:0/931146713 conn(0x7fce940fa5e0 msgr2=0x7fce94103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:57:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.716+0000 7fce99120700 1 -- 172.21.15.102:0/931146713 shutdown_connections 2024-03-26T18:57:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:57:45.716+0000 7fce99120700 1 -- 172.21.15.102:0/931146713 wait complete. 2024-03-26T18:57:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:46 smithi190 ceph-mon[65689]: pgmap v37: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:46 smithi190 ceph-mon[65689]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:46 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/931146713' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:57:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:46 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:46 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:46.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:46 smithi102 ceph-mon[78322]: pgmap v37: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:46.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:46 smithi102 ceph-mon[78322]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:57:46.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:46 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/931146713' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:57:46.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:46.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:47.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:47 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:47.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:47 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:47.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:47 smithi190 ceph-mon[65689]: pgmap v38: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:48.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:47 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:48.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:47 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:48.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:47 smithi102 ceph-mon[78322]: pgmap v38: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:49.013 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:49.014 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:49.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:49.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:50.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:50 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:50.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:50 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:50.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:50 smithi102 ceph-mon[78322]: pgmap v39: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:50 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:50 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:50 smithi190 ceph-mon[65689]: pgmap v39: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:52.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:52.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:52.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:52.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:57:52.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: pgmap v40: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: Upgrade: Setting container_image for all crash 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi102"}]: dispatch 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi102"}]': finished 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi190"}]: dispatch 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi190"}]': finished 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T18:57:52.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:52 smithi102 ceph-mon[78322]: Upgrade: osd.0 is safe to restart 2024-03-26T18:57:52.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:52.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: pgmap v40: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: Upgrade: Setting container_image for all crash 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi102"}]: dispatch 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi102"}]': finished 2024-03-26T18:57:52.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi190"}]: dispatch 2024-03-26T18:57:52.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi190"}]': finished 2024-03-26T18:57:52.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T18:57:52.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T18:57:52.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:52 smithi190 ceph-mon[65689]: Upgrade: osd.0 is safe to restart 2024-03-26T18:57:53.373 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:57:53.373 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:53.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T18:57:53.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:53.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:57:53.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:57:53.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T18:57:53.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:57:54.315 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:54 smithi102 ceph-mon[78322]: Upgrade: Updating osd.0 2024-03-26T18:57:54.316 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:54 smithi102 ceph-mon[78322]: Deploying daemon osd.0 on smithi102 2024-03-26T18:57:54.316 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:54 smithi102 ceph-mon[78322]: pgmap v41: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:54.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:54 smithi190 ceph-mon[65689]: Upgrade: Updating osd.0 2024-03-26T18:57:54.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:54 smithi190 ceph-mon[65689]: Deploying daemon osd.0 on smithi102 2024-03-26T18:57:54.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:54 smithi190 ceph-mon[65689]: pgmap v41: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:56.260 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:56 smithi102 systemd[1]: Stopping Ceph osd.0 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:57:56.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:56 smithi190 ceph-mon[65689]: pgmap v42: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:56.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:56 smithi102 ceph-mon[78322]: pgmap v42: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:56.626 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:56 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[43870]: 2024-03-26T18:57:56.385+0000 7f520e37e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T18:57:56.626 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:56 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[43870]: 2024-03-26T18:57:56.385+0000 7f520e37e700 -1 osd.0 50 *** Got signal Terminated *** 2024-03-26T18:57:56.626 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:56 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[43870]: 2024-03-26T18:57:56.385+0000 7f520e37e700 -1 osd.0 50 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T18:57:57.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:57 smithi190 ceph-mon[65689]: osd.0 marked itself down and dead 2024-03-26T18:57:57.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:57 smithi102 ceph-mon[78322]: osd.0 marked itself down and dead 2024-03-26T18:57:58.126 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:57 smithi102 podman[84365]: 2024-03-26 18:57:57.796170133 +0000 UTC m=+1.528219694 container died 3e2065fb5fbc5208fe4528ed292e71b09bdd4c97d8318e699dd15ff14bfdf840 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, RELEASE=HEAD, org.label-schema.license=GPLv2) 2024-03-26T18:57:58.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:58 smithi190 ceph-mon[65689]: pgmap v43: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:58.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:58 smithi190 ceph-mon[65689]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T18:57:58.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:58 smithi190 ceph-mon[65689]: osdmap e51: 6 total, 5 up, 6 in 2024-03-26T18:57:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:58 smithi102 ceph-mon[78322]: pgmap v43: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:57:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:58 smithi102 ceph-mon[78322]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T18:57:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:58 smithi102 ceph-mon[78322]: osdmap e51: 6 total, 5 up, 6 in 2024-03-26T18:57:58.626 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:58 smithi102 podman[84365]: 2024-03-26 18:57:58.203067371 +0000 UTC m=+1.935116972 container cleanup 3e2065fb5fbc5208fe4528ed292e71b09bdd4c97d8318e699dd15ff14bfdf840 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0) 2024-03-26T18:57:58.626 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:58 smithi102 bash[84365]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0 2024-03-26T18:57:58.627 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:58 smithi102 podman[84383]: 2024-03-26 18:57:58.413191164 +0000 UTC m=+0.614110949 container remove 3e2065fb5fbc5208fe4528ed292e71b09bdd4c97d8318e699dd15ff14bfdf840 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0, GIT_CLEAN=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-26T18:57:59.117 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:58 smithi102 podman[84477]: 2024-03-26 18:57:58.714278079 +0000 UTC m=+0.018064718 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:57:59.117 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:58 smithi102 podman[84477]: 2024-03-26 18:57:58.814209156 +0000 UTC m=+0.117995831 container create c0f9ba172bec485bb1db809f504421ae71810bdb99e86d6eed0d62e2154fa871 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T18:57:59.118 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:59 smithi102 podman[84477]: 2024-03-26 18:57:59.033858148 +0000 UTC m=+0.337644788 container init c0f9ba172bec485bb1db809f504421ae71810bdb99e86d6eed0d62e2154fa871 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-26T18:57:59.118 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:59 smithi102 podman[84477]: 2024-03-26 18:57:59.037600797 +0000 UTC m=+0.341387432 container start c0f9ba172bec485bb1db809f504421ae71810bdb99e86d6eed0d62e2154fa871 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-03-26T18:57:59.118 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:59 smithi102 podman[84477]: 2024-03-26 18:57:59.124953778 +0000 UTC m=+0.428740412 container attach c0f9ba172bec485bb1db809f504421ae71810bdb99e86d6eed0d62e2154fa871 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T18:57:59.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:57:59 smithi102 ceph-mon[78322]: osdmap e52: 6 total, 5 up, 6 in 2024-03-26T18:57:59.376 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:59 smithi102 podman[84477]: 2024-03-26 18:57:59.328358214 +0000 UTC m=+0.632144845 container died c0f9ba172bec485bb1db809f504421ae71810bdb99e86d6eed0d62e2154fa871 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T18:57:59.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:57:59 smithi190 ceph-mon[65689]: osdmap e52: 6 total, 5 up, 6 in 2024-03-26T18:58:00.282 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:59 smithi102 podman[84503]: 2024-03-26 18:57:59.954427162 +0000 UTC m=+0.620998658 container remove c0f9ba172bec485bb1db809f504421ae71810bdb99e86d6eed0d62e2154fa871 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-26T18:58:00.282 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:59 smithi102 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-26T18:58:00.282 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:59 smithi102 systemd[1]: Stopped Ceph osd.0 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:58:00.282 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:57:59 smithi102 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.0.service: Consumed 55.277s CPU time. 2024-03-26T18:58:00.282 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:00 smithi102 systemd[1]: Starting Ceph osd.0 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:58:00.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:00 smithi102 ceph-mon[78322]: pgmap v46: 65 pgs: 34 peering, 31 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:00.532 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:00 smithi102 ceph-mon[78322]: Health check failed: Reduced data availability: 27 pgs peering (PG_AVAILABILITY) 2024-03-26T18:58:00.532 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:00 smithi102 podman[84606]: 2024-03-26 18:58:00.310161291 +0000 UTC m=+0.018112645 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:58:00.532 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:00 smithi102 podman[84606]: 2024-03-26 18:58:00.431018342 +0000 UTC m=+0.138969720 container create 834676b110376c869531780ed3a85e06094ee02e9a2d41b2d9cedeecac22c3af (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T18:58:00.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:00 smithi190 ceph-mon[65689]: pgmap v46: 65 pgs: 34 peering, 31 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:00.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:00 smithi190 ceph-mon[65689]: Health check failed: Reduced data availability: 27 pgs peering (PG_AVAILABILITY) 2024-03-26T18:58:00.991 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:00 smithi102 podman[84606]: 2024-03-26 18:58:00.64176398 +0000 UTC m=+0.349715316 container init 834676b110376c869531780ed3a85e06094ee02e9a2d41b2d9cedeecac22c3af (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T18:58:00.991 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:00 smithi102 podman[84606]: 2024-03-26 18:58:00.645122346 +0000 UTC m=+0.353073692 container start 834676b110376c869531780ed3a85e06094ee02e9a2d41b2d9cedeecac22c3af (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T18:58:00.991 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:00 smithi102 podman[84606]: 2024-03-26 18:58:00.716699017 +0000 UTC m=+0.424650356 container attach 834676b110376c869531780ed3a85e06094ee02e9a2d41b2d9cedeecac22c3af (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T18:58:01.376 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate[84621]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T18:58:01.376 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate[84621]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T18:58:01.376 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 bash[84606]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T18:58:01.376 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 bash[84606]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T18:58:01.376 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 bash[84606]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T18:58:01.377 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 bash[84606]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T18:58:01.377 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 bash[84606]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-26T18:58:01.377 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 bash[84606]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T18:58:01.377 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 bash[84606]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-26T18:58:01.377 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate[84621]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T18:58:01.377 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate[84621]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T18:58:01.377 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate[84621]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-26T18:58:01.377 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate[84621]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T18:58:01.377 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate[84621]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-26T18:58:01.377 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:01 smithi102 podman[84606]: 2024-03-26 18:58:01.115970497 +0000 UTC m=+0.823921827 container died 834676b110376c869531780ed3a85e06094ee02e9a2d41b2d9cedeecac22c3af (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T18:58:02.250 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:02 smithi102 podman[84751]: 2024-03-26 18:58:02.006921562 +0000 UTC m=+0.882655798 container remove 834676b110376c869531780ed3a85e06094ee02e9a2d41b2d9cedeecac22c3af (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T18:58:02.250 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:02 smithi102 podman[84817]: 2024-03-26 18:58:02.123641134 +0000 UTC m=+0.018216732 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:58:02.536 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:02 smithi102 ceph-mon[78322]: pgmap v47: 65 pgs: 34 peering, 31 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:02.536 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:02 smithi102 podman[84817]: 2024-03-26 18:58:02.257910033 +0000 UTC m=+0.152485631 container create aec2fe1c7a8b97a70c36828c7bac441a8667be85d75b900c607ae7e87e99939f (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_BRANCH=HEAD) 2024-03-26T18:58:02.536 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:02 smithi102 podman[84817]: 2024-03-26 18:58:02.452128612 +0000 UTC m=+0.346704211 container init aec2fe1c7a8b97a70c36828c7bac441a8667be85d75b900c607ae7e87e99939f (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.schema-version=1.0) 2024-03-26T18:58:02.536 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:02 smithi102 podman[84817]: 2024-03-26 18:58:02.455071543 +0000 UTC m=+0.349647149 container start aec2fe1c7a8b97a70c36828c7bac441a8667be85d75b900c607ae7e87e99939f (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T18:58:02.536 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:02 smithi102 ceph-osd[84836]: -- 172.21.15.102:0/1434182671 <== mon.0 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55961e441680 con 0x55961e40ac00 2024-03-26T18:58:02.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:02 smithi190 ceph-mon[65689]: pgmap v47: 65 pgs: 34 peering, 31 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:02.876 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:02 smithi102 bash[84817]: aec2fe1c7a8b97a70c36828c7bac441a8667be85d75b900c607ae7e87e99939f 2024-03-26T18:58:02.876 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:02 smithi102 systemd[1]: Started Ceph osd.0 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:58:03.376 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:02 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[84832]: 2024-03-26T18:58:02.998+0000 7f79b29c37c0 -1 Falling back to public interface 2024-03-26T18:58:03.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:03.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:03.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:58:03.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:03 smithi102 ceph-mon[78322]: pgmap v48: 65 pgs: 11 active+undersized+degraded, 12 peering, 11 active+undersized, 31 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 39/354 objects degraded (11.017%) 2024-03-26T18:58:04.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:04.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:04.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:58:04.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:03 smithi190 ceph-mon[65689]: pgmap v48: 65 pgs: 11 active+undersized+degraded, 12 peering, 11 active+undersized, 31 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 39/354 objects degraded (11.017%) 2024-03-26T18:58:04.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:04 smithi102 ceph-mon[78322]: Health check failed: Degraded data redundancy: 39/354 objects degraded (11.017%), 11 pgs degraded (PG_DEGRADED) 2024-03-26T18:58:05.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:04 smithi190 ceph-mon[65689]: Health check failed: Degraded data redundancy: 39/354 objects degraded (11.017%), 11 pgs degraded (PG_DEGRADED) 2024-03-26T18:58:05.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:05 smithi102 ceph-mon[78322]: pgmap v49: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 69/354 objects degraded (19.492%) 2024-03-26T18:58:06.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:05 smithi190 ceph-mon[65689]: pgmap v49: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 69/354 objects degraded (19.492%) 2024-03-26T18:58:06.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:06 smithi102 ceph-mon[78322]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs peering) 2024-03-26T18:58:07.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:06 smithi190 ceph-mon[65689]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs peering) 2024-03-26T18:58:07.867 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:07 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[84832]: 2024-03-26T18:58:07.434+0000 7f79b29c37c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-26T18:58:07.867 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:07 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[84832]: 2024-03-26T18:58:07.873+0000 7f79b29c37c0 -1 osd.0 50 log_to_monitors true 2024-03-26T18:58:08.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:08 smithi190 ceph-mon[65689]: pgmap v50: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 69/354 objects degraded (19.492%) 2024-03-26T18:58:08.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:08.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:58:08.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:08.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:08.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:08 smithi190 ceph-mon[65689]: from='osd.0 [v2:172.21.15.102:6802/3021596332,v1:172.21.15.102:6803/3021596332]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T18:58:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:08 smithi102 ceph-mon[78322]: pgmap v50: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 69/354 objects degraded (19.492%) 2024-03-26T18:58:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:58:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:08 smithi102 ceph-mon[78322]: from='osd.0 [v2:172.21.15.102:6802/3021596332,v1:172.21.15.102:6803/3021596332]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T18:58:09.126 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 18:58:08 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[84832]: 2024-03-26T18:58:08.756+0000 7f79a3e1b700 -1 osd.0 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T18:58:09.720 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:09 smithi102 ceph-mon[78322]: mgrmap e37: smithi102.rsfpnv(active, since 91s), standbys: smithi190.zrixos 2024-03-26T18:58:09.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:09 smithi102 ceph-mon[78322]: from='osd.0 [v2:172.21.15.102:6802/3021596332,v1:172.21.15.102:6803/3021596332]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T18:58:09.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:09 smithi102 ceph-mon[78322]: osdmap e53: 6 total, 5 up, 6 in 2024-03-26T18:58:09.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:09 smithi102 ceph-mon[78322]: from='osd.0 [v2:172.21.15.102:6802/3021596332,v1:172.21.15.102:6803/3021596332]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:58:09.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:09 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:09.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:09 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:09.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:09 smithi102 ceph-mon[78322]: Health check update: Degraded data redundancy: 69/354 objects degraded (19.492%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T18:58:09.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:09 smithi102 ceph-mon[78322]: pgmap v52: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 614 B/s wr, 0 op/s; 69/354 objects degraded (19.492%) 2024-03-26T18:58:10.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:09 smithi190 ceph-mon[65689]: mgrmap e37: smithi102.rsfpnv(active, since 91s), standbys: smithi190.zrixos 2024-03-26T18:58:10.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:09 smithi190 ceph-mon[65689]: from='osd.0 [v2:172.21.15.102:6802/3021596332,v1:172.21.15.102:6803/3021596332]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T18:58:10.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:09 smithi190 ceph-mon[65689]: osdmap e53: 6 total, 5 up, 6 in 2024-03-26T18:58:10.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:09 smithi190 ceph-mon[65689]: from='osd.0 [v2:172.21.15.102:6802/3021596332,v1:172.21.15.102:6803/3021596332]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:58:10.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:09 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:10.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:09 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:10.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:09 smithi190 ceph-mon[65689]: Health check update: Degraded data redundancy: 69/354 objects degraded (19.492%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T18:58:10.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:09 smithi190 ceph-mon[65689]: pgmap v52: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 614 B/s wr, 0 op/s; 69/354 objects degraded (19.492%) 2024-03-26T18:58:10.692 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:10 smithi102 ceph-mon[78322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T18:58:10.692 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:10 smithi102 ceph-mon[78322]: osd.0 [v2:172.21.15.102:6802/3021596332,v1:172.21.15.102:6803/3021596332] boot 2024-03-26T18:58:10.693 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:10 smithi102 ceph-mon[78322]: osdmap e54: 6 total, 6 up, 6 in 2024-03-26T18:58:10.693 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:10 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:58:11.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:10 smithi190 ceph-mon[65689]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T18:58:11.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:10 smithi190 ceph-mon[65689]: osd.0 [v2:172.21.15.102:6802/3021596332,v1:172.21.15.102:6803/3021596332] boot 2024-03-26T18:58:11.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:10 smithi190 ceph-mon[65689]: osdmap e54: 6 total, 6 up, 6 in 2024-03-26T18:58:11.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:10 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T18:58:11.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:11 smithi102 ceph-mon[78322]: osdmap e55: 6 total, 6 up, 6 in 2024-03-26T18:58:11.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:11 smithi102 ceph-mon[78322]: pgmap v55: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s; 69/354 objects degraded (19.492%) 2024-03-26T18:58:12.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:11 smithi190 ceph-mon[65689]: osdmap e55: 6 total, 6 up, 6 in 2024-03-26T18:58:12.030 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:11 smithi190 ceph-mon[65689]: pgmap v55: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s; 69/354 objects degraded (19.492%) 2024-03-26T18:58:13.337 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:13.337 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:13.337 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:58:13.338 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:58:13.338 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:13.338 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:58:13.338 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:13.338 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:13.338 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:13.338 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T18:58:13.338 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T18:58:13.338 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:13 smithi190 ceph-mon[65689]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-03-26T18:58:13.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:13.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:13.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:58:13.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:58:13.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:13.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:58:13.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:13.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:13.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:13.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T18:58:13.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T18:58:13.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:13 smithi102 ceph-mon[78322]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-03-26T18:58:14.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:14 smithi102 ceph-mon[78322]: pgmap v56: 65 pgs: 12 active+undersized+degraded, 8 active+undersized, 45 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s; 42/354 objects degraded (11.864%) 2024-03-26T18:58:14.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:14 smithi190 ceph-mon[65689]: pgmap v56: 65 pgs: 12 active+undersized+degraded, 8 active+undersized, 45 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s; 42/354 objects degraded (11.864%) 2024-03-26T18:58:15.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:15 smithi102 ceph-mon[78322]: Health check update: Degraded data redundancy: 42/354 objects degraded (11.864%), 12 pgs degraded (PG_DEGRADED) 2024-03-26T18:58:15.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:15 smithi190 ceph-mon[65689]: Health check update: Degraded data redundancy: 42/354 objects degraded (11.864%), 12 pgs degraded (PG_DEGRADED) 2024-03-26T18:58:15.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.840+0000 7ff35cfff700 1 -- 172.21.15.102:0/1950636215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3580ff090 msgr2=0x7ff3580ff4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:15.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.840+0000 7ff35cfff700 1 --2- 172.21.15.102:0/1950636215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3580ff090 0x7ff3580ff4a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff348009910 tx=0x7ff348009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:15.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.840+0000 7ff35cfff700 1 -- 172.21.15.102:0/1950636215 shutdown_connections 2024-03-26T18:58:15.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.840+0000 7ff35cfff700 1 --2- 172.21.15.102:0/1950636215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff358100290 0x7ff3581006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.840+0000 7ff35cfff700 1 --2- 172.21.15.102:0/1950636215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3580ff090 0x7ff3580ff4a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.840+0000 7ff35cfff700 1 -- 172.21.15.102:0/1950636215 >> 172.21.15.102:0/1950636215 conn(0x7ff3580fa620 msgr2=0x7ff3580fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.840+0000 7ff35cfff700 1 -- 172.21.15.102:0/1950636215 shutdown_connections 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.840+0000 7ff35cfff700 1 -- 172.21.15.102:0/1950636215 wait complete. 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35cfff700 1 Processor -- start 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35cfff700 1 -- start start 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35cfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3580ff090 0x7ff358103020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35cfff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff358100290 0x7ff358101670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35cfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff358101bb0 con 0x7ff3580ff090 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35cfff700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff358101cf0 con 0x7ff358100290 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3580ff090 0x7ff358103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3580ff090 0x7ff358103020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51408/0 (socket says 172.21.15.102:51408) 2024-03-26T18:58:15.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35659c700 1 -- 172.21.15.102:0/3121494486 learned_addr learned my addr 172.21.15.102:0/3121494486 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff355d9b700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff358100290 0x7ff358101670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35659c700 1 -- 172.21.15.102:0/3121494486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff358100290 msgr2=0x7ff358101670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35659c700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff358100290 0x7ff358101670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff35659c700 1 -- 172.21.15.102:0/3121494486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3480095f0 con 0x7ff3580ff090 2024-03-26T18:58:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.841+0000 7ff355d9b700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff358100290 0x7ff358101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T18:58:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.842+0000 7ff35659c700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3580ff090 0x7ff358103020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff34800bda0 tx=0x7ff34800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.842+0000 7ff3477fe700 1 -- 172.21.15.102:0/3121494486 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff34801d070 con 0x7ff3580ff090 2024-03-26T18:58:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.842+0000 7ff35cfff700 1 -- 172.21.15.102:0/3121494486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff358101f70 con 0x7ff3580ff090 2024-03-26T18:58:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.842+0000 7ff3477fe700 1 -- 172.21.15.102:0/3121494486 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff34800f460 con 0x7ff3580ff090 2024-03-26T18:58:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.842+0000 7ff3477fe700 1 -- 172.21.15.102:0/3121494486 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff348021640 con 0x7ff3580ff090 2024-03-26T18:58:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.842+0000 7ff35cfff700 1 -- 172.21.15.102:0/3121494486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff358102460 con 0x7ff3580ff090 2024-03-26T18:58:15.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.843+0000 7ff35cfff700 1 -- 172.21.15.102:0/3121494486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff358107be0 con 0x7ff3580ff090 2024-03-26T18:58:15.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.843+0000 7ff3477fe700 1 -- 172.21.15.102:0/3121494486 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff348003a60 con 0x7ff3580ff090 2024-03-26T18:58:15.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.844+0000 7ff3477fe700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff34006e460 0x7ff340070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:15.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.844+0000 7ff3477fe700 1 -- 172.21.15.102:0/3121494486 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff34808f6c0 con 0x7ff3580ff090 2024-03-26T18:58:15.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.844+0000 7ff355d9b700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff34006e460 0x7ff340070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:15.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.844+0000 7ff355d9b700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff34006e460 0x7ff340070910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff34c005d90 tx=0x7ff34c005d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:15.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:15.848+0000 7ff3477fe700 1 -- 172.21.15.102:0/3121494486 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff34805c3f0 con 0x7ff3580ff090 2024-03-26T18:58:16.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.066+0000 7ff35cfff700 1 -- 172.21.15.102:0/3121494486 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff358069460 con 0x7ff34006e460 2024-03-26T18:58:16.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.068+0000 7ff3477fe700 1 -- 172.21.15.102:0/3121494486 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ff358069460 con 0x7ff34006e460 2024-03-26T18:58:16.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.070+0000 7ff35cfff700 1 -- 172.21.15.102:0/3121494486 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff34006e460 msgr2=0x7ff340070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.070+0000 7ff35cfff700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff34006e460 0x7ff340070910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff34c005d90 tx=0x7ff34c005d20 comp rx=0 tx=0).stop 2024-03-26T18:58:16.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.070+0000 7ff35cfff700 1 -- 172.21.15.102:0/3121494486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3580ff090 msgr2=0x7ff358103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.070+0000 7ff35cfff700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3580ff090 0x7ff358103020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff34800bda0 tx=0x7ff34800be80 comp rx=0 tx=0).stop 2024-03-26T18:58:16.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.070+0000 7ff35cfff700 1 -- 172.21.15.102:0/3121494486 shutdown_connections 2024-03-26T18:58:16.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.070+0000 7ff35cfff700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff34006e460 0x7ff340070910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.070+0000 7ff35cfff700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3580ff090 0x7ff358103020 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.070+0000 7ff35cfff700 1 --2- 172.21.15.102:0/3121494486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff358100290 0x7ff358101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.070+0000 7ff35cfff700 1 -- 172.21.15.102:0/3121494486 >> 172.21.15.102:0/3121494486 conn(0x7ff3580fa620 msgr2=0x7ff358068d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:16.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.070+0000 7ff35cfff700 1 -- 172.21.15.102:0/3121494486 shutdown_connections 2024-03-26T18:58:16.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.070+0000 7ff35cfff700 1 -- 172.21.15.102:0/3121494486 wait complete. 2024-03-26T18:58:16.074 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:58:16.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:16 smithi102 ceph-mon[78322]: pgmap v57: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 43 B/s, 0 objects/s recovering 2024-03-26T18:58:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.175+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2137710616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bf0101390 msgr2=0x7f2bf01017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.175+0000 7f2bf69b1700 1 --2- 172.21.15.102:0/2137710616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bf0101390 0x7f2bf01017e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009910 tx=0x7f2be4009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.175+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2137710616 shutdown_connections 2024-03-26T18:58:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.175+0000 7f2bf69b1700 1 --2- 172.21.15.102:0/2137710616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bf0101390 0x7f2bf01017e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.175+0000 7f2bf69b1700 1 --2- 172.21.15.102:0/2137710616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf0100190 0x7f2bf01005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.175+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2137710616 >> 172.21.15.102:0/2137710616 conn(0x7f2bf00fb720 msgr2=0x7f2bf00fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.175+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2137710616 shutdown_connections 2024-03-26T18:58:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.175+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2137710616 wait complete. 2024-03-26T18:58:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.176+0000 7f2bf69b1700 1 Processor -- start 2024-03-26T18:58:16.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.176+0000 7f2bf69b1700 1 -- start start 2024-03-26T18:58:16.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.176+0000 7f2bf69b1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bf0100190 0x7f2bf01959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:16.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.176+0000 7f2bf69b1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf0101390 0x7f2bf0195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:16.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.176+0000 7f2bf69b1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf0196500 con 0x7f2bf0100190 2024-03-26T18:58:16.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.176+0000 7f2bf69b1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf0196640 con 0x7f2bf0101390 2024-03-26T18:58:16.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.176+0000 7f2beffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bf0100190 0x7f2bf01959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:16.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.176+0000 7f2beffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bf0100190 0x7f2bf01959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51434/0 (socket says 172.21.15.102:51434) 2024-03-26T18:58:16.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.176+0000 7f2beffff700 1 -- 172.21.15.102:0/2311515082 learned_addr learned my addr 172.21.15.102:0/2311515082 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.177+0000 7f2bef7fe700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf0101390 0x7f2bf0195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.177+0000 7f2beffff700 1 -- 172.21.15.102:0/2311515082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf0101390 msgr2=0x7f2bf0195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.177+0000 7f2beffff700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf0101390 0x7f2bf0195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.177+0000 7f2beffff700 1 -- 172.21.15.102:0/2311515082 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40095f0 con 0x7f2bf0100190 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.177+0000 7f2bef7fe700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf0101390 0x7f2bf0195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.177+0000 7f2beffff700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bf0100190 0x7f2bf01959a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2be000d900 tx=0x7f2be000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.177+0000 7f2bed7fa700 1 -- 172.21.15.102:0/2311515082 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2be00041f0 con 0x7f2bf0100190 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.177+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2311515082 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf019b0f0 con 0x7f2bf0100190 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.177+0000 7f2bed7fa700 1 -- 172.21.15.102:0/2311515082 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2be0004350 con 0x7f2bf0100190 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.177+0000 7f2bed7fa700 1 -- 172.21.15.102:0/2311515082 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2be0003da0 con 0x7f2bf0100190 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.177+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2311515082 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf019b640 con 0x7f2bf0100190 2024-03-26T18:58:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.178+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2311515082 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf0108ce0 con 0x7f2bf0100190 2024-03-26T18:58:16.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.178+0000 7f2bed7fa700 1 -- 172.21.15.102:0/2311515082 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2be0009730 con 0x7f2bf0100190 2024-03-26T18:58:16.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.179+0000 7f2bed7fa700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2bd806e460 0x7f2bd8070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:16.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.179+0000 7f2bed7fa700 1 -- 172.21.15.102:0/2311515082 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2be008e500 con 0x7f2bf0100190 2024-03-26T18:58:16.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.179+0000 7f2bef7fe700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2bd806e460 0x7f2bd8070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:16.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.179+0000 7f2bef7fe700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2bd806e460 0x7f2bd8070910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2be4005b40 tx=0x7f2be4005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.183+0000 7f2bed7fa700 1 -- 172.21.15.102:0/2311515082 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2be005b230 con 0x7f2bf0100190 2024-03-26T18:58:16.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.397+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2311515082 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bf0105ce0 con 0x7f2bd806e460 2024-03-26T18:58:16.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.398+0000 7f2bed7fa700 1 -- 172.21.15.102:0/2311515082 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2bf0105ce0 con 0x7f2bd806e460 2024-03-26T18:58:16.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.400+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2311515082 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2bd806e460 msgr2=0x7f2bd8070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.400+0000 7f2bf69b1700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2bd806e460 0x7f2bd8070910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2be4005b40 tx=0x7f2be4005a90 comp rx=0 tx=0).stop 2024-03-26T18:58:16.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.400+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2311515082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bf0100190 msgr2=0x7f2bf01959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.400+0000 7f2bf69b1700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bf0100190 0x7f2bf01959a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2be000d900 tx=0x7f2be000dcc0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.401+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2311515082 shutdown_connections 2024-03-26T18:58:16.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.401+0000 7f2bf69b1700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2bd806e460 0x7f2bd8070910 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.401+0000 7f2bf69b1700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bf0100190 0x7f2bf01959a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.401+0000 7f2bf69b1700 1 --2- 172.21.15.102:0/2311515082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf0101390 0x7f2bf0195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.401+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2311515082 >> 172.21.15.102:0/2311515082 conn(0x7f2bf00fb720 msgr2=0x7f2bf01045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:16.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.401+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2311515082 shutdown_connections 2024-03-26T18:58:16.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.401+0000 7f2bf69b1700 1 -- 172.21.15.102:0/2311515082 wait complete. 2024-03-26T18:58:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.503+0000 7f5901b1e700 1 -- 172.21.15.102:0/1036248242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58fc101350 msgr2=0x7f58fc1017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.503+0000 7f5901b1e700 1 --2- 172.21.15.102:0/1036248242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58fc101350 0x7f58fc1017a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009910 tx=0x7f58f0009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.503+0000 7f5901b1e700 1 -- 172.21.15.102:0/1036248242 shutdown_connections 2024-03-26T18:58:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.503+0000 7f5901b1e700 1 --2- 172.21.15.102:0/1036248242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58fc101350 0x7f58fc1017a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.503+0000 7f5901b1e700 1 --2- 172.21.15.102:0/1036248242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc100150 0x7f58fc100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.503+0000 7f5901b1e700 1 -- 172.21.15.102:0/1036248242 >> 172.21.15.102:0/1036248242 conn(0x7f58fc0fb6e0 msgr2=0x7f58fc0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.504+0000 7f5901b1e700 1 -- 172.21.15.102:0/1036248242 shutdown_connections 2024-03-26T18:58:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.504+0000 7f5901b1e700 1 -- 172.21.15.102:0/1036248242 wait complete. 2024-03-26T18:58:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.504+0000 7f5901b1e700 1 Processor -- start 2024-03-26T18:58:16.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.504+0000 7f5901b1e700 1 -- start start 2024-03-26T18:58:16.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f5901b1e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc100150 0x7f58fc074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:16.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f5901b1e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58fc101350 0x7f58fc073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:16.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f5901b1e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc073680 con 0x7f58fc101350 2024-03-26T18:58:16.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f5901b1e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc0737c0 con 0x7f58fc100150 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f58faffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58fc101350 0x7f58fc073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f58faffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58fc101350 0x7f58fc073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51454/0 (socket says 172.21.15.102:51454) 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f58faffd700 1 -- 172.21.15.102:0/3229022561 learned_addr learned my addr 172.21.15.102:0/3229022561 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f58fb7fe700 1 --2- 172.21.15.102:0/3229022561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc100150 0x7f58fc074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f58faffd700 1 -- 172.21.15.102:0/3229022561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc100150 msgr2=0x7f58fc074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f58faffd700 1 --2- 172.21.15.102:0/3229022561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc100150 0x7f58fc074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f58faffd700 1 -- 172.21.15.102:0/3229022561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f00095f0 con 0x7f58fc101350 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f58faffd700 1 --2- 172.21.15.102:0/3229022561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58fc101350 0x7f58fc073140 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f58f0006010 tx=0x7f58f000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.505+0000 7f58f8ff9700 1 -- 172.21.15.102:0/3229022561 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58f001d070 con 0x7f58fc101350 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.506+0000 7f5901b1e700 1 -- 172.21.15.102:0/3229022561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58fc073a40 con 0x7f58fc101350 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.506+0000 7f58f8ff9700 1 -- 172.21.15.102:0/3229022561 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f58f000f460 con 0x7f58fc101350 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.506+0000 7f58f8ff9700 1 -- 172.21.15.102:0/3229022561 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58f0021640 con 0x7f58fc101350 2024-03-26T18:58:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.506+0000 7f5901b1e700 1 -- 172.21.15.102:0/3229022561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58fc073f30 con 0x7f58fc101350 2024-03-26T18:58:16.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.507+0000 7f5901b1e700 1 -- 172.21.15.102:0/3229022561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58fc108d40 con 0x7f58fc101350 2024-03-26T18:58:16.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.507+0000 7f58f8ff9700 1 -- 172.21.15.102:0/3229022561 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f58f002b440 con 0x7f58fc101350 2024-03-26T18:58:16.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.508+0000 7f58f8ff9700 1 --2- 172.21.15.102:0/3229022561 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58e406e460 0x7f58e4070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.508+0000 7f58f8ff9700 1 -- 172.21.15.102:0/3229022561 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f58f008f790 con 0x7f58fc101350 2024-03-26T18:58:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.508+0000 7f58fb7fe700 1 --2- 172.21.15.102:0/3229022561 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58e406e460 0x7f58e4070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.508+0000 7f58fb7fe700 1 --2- 172.21.15.102:0/3229022561 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58e406e460 0x7f58e4070910 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58fc1011b0 tx=0x7f58ec009190 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:16.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.513+0000 7f58f8ff9700 1 -- 172.21.15.102:0/3229022561 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f58f005c4c0 con 0x7f58fc101350 2024-03-26T18:58:16.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:16 smithi190 ceph-mon[65689]: pgmap v57: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 43 B/s, 0 objects/s recovering 2024-03-26T18:58:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.720+0000 7f5901b1e700 1 -- 172.21.15.102:0/3229022561 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f58fc105ca0 con 0x7f58e406e460 2024-03-26T18:58:16.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.731+0000 7f58f8ff9700 1 -- 172.21.15.102:0/3229022561 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f58fc105ca0 con 0x7f58e406e460 2024-03-26T18:58:16.724 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:58:16.724 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (9m) 8s ago 11m 24.9M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:58:16.724 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (11m) 8s ago 11m 7889k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:58:16.724 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (10m) 31s ago 10m 7755k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (41s) 8s ago 11m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (36s) 31s ago 10m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (9m) 8s ago 11m 83.4M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (5m) 8s ago 5m 132M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (5m) 8s ago 5m 15.6M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (5m) 31s ago 5m 15.3M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (5m) 31s ago 5m 120M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (2m) 8s ago 12m 497M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (2m) 31s ago 10m 449M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (108s) 8s ago 12m 42.1M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (85s) 31s ago 10m 32.4M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (11m) 8s ago 11m 16.3M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (10m) 31s ago 10m 15.5M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:58:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (14s) 8s ago 9m 12.5M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T18:58:16.726 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (9m) 8s ago 9m 246M 1216M 18.2.0 dc2bc1663786 0bf773515cc1 2024-03-26T18:58:16.726 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (8m) 8s ago 8m 192M 1216M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:58:16.726 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (8m) 31s ago 8m 272M 2240M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:58:16.726 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (7m) 31s ago 7m 251M 2240M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:58:16.726 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (7m) 31s ago 7m 225M 2240M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:58:16.726 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (2m) 8s ago 10m 67.5M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T18:58:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.733+0000 7f5901b1e700 1 -- 172.21.15.102:0/3229022561 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58e406e460 msgr2=0x7f58e4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.733+0000 7f5901b1e700 1 --2- 172.21.15.102:0/3229022561 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58e406e460 0x7f58e4070910 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58fc1011b0 tx=0x7f58ec009190 comp rx=0 tx=0).stop 2024-03-26T18:58:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.733+0000 7f5901b1e700 1 -- 172.21.15.102:0/3229022561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58fc101350 msgr2=0x7f58fc073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.733+0000 7f5901b1e700 1 --2- 172.21.15.102:0/3229022561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58fc101350 0x7f58fc073140 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f58f0006010 tx=0x7f58f000bd70 comp rx=0 tx=0).stop 2024-03-26T18:58:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.733+0000 7f5901b1e700 1 -- 172.21.15.102:0/3229022561 shutdown_connections 2024-03-26T18:58:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.733+0000 7f5901b1e700 1 --2- 172.21.15.102:0/3229022561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc100150 0x7f58fc074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.733+0000 7f5901b1e700 1 --2- 172.21.15.102:0/3229022561 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58e406e460 0x7f58e4070910 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.733+0000 7f5901b1e700 1 --2- 172.21.15.102:0/3229022561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58fc101350 0x7f58fc073140 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.733+0000 7f5901b1e700 1 -- 172.21.15.102:0/3229022561 >> 172.21.15.102:0/3229022561 conn(0x7f58fc0fb6e0 msgr2=0x7f58fc104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.733+0000 7f5901b1e700 1 -- 172.21.15.102:0/3229022561 shutdown_connections 2024-03-26T18:58:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.733+0000 7f5901b1e700 1 -- 172.21.15.102:0/3229022561 wait complete. 2024-03-26T18:58:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.836+0000 7f595a7f5700 1 -- 172.21.15.102:0/160426185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59541002a0 msgr2=0x7f59541006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.836+0000 7f595a7f5700 1 --2- 172.21.15.102:0/160426185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59541002a0 0x7f59541006f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5948009910 tx=0x7f5948009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.836+0000 7f595a7f5700 1 -- 172.21.15.102:0/160426185 shutdown_connections 2024-03-26T18:58:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.836+0000 7f595a7f5700 1 --2- 172.21.15.102:0/160426185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59541002a0 0x7f59541006f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.836+0000 7f595a7f5700 1 --2- 172.21.15.102:0/160426185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59540ff0a0 0x7f59540ff4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.836+0000 7f595a7f5700 1 -- 172.21.15.102:0/160426185 >> 172.21.15.102:0/160426185 conn(0x7f59540fa630 msgr2=0x7f59540fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.836+0000 7f595a7f5700 1 -- 172.21.15.102:0/160426185 shutdown_connections 2024-03-26T18:58:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.836+0000 7f595a7f5700 1 -- 172.21.15.102:0/160426185 wait complete. 2024-03-26T18:58:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.837+0000 7f595a7f5700 1 Processor -- start 2024-03-26T18:58:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.837+0000 7f595a7f5700 1 -- start start 2024-03-26T18:58:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.837+0000 7f595a7f5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59540ff0a0 0x7f5954195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.837+0000 7f595a7f5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59541002a0 0x7f5954195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.837+0000 7f595a7f5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59541964f0 con 0x7f59541002a0 2024-03-26T18:58:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.837+0000 7f595a7f5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5954196630 con 0x7f59540ff0a0 2024-03-26T18:58:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.837+0000 7f59537fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59541002a0 0x7f5954195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.837+0000 7f59537fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59541002a0 0x7f5954195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51460/0 (socket says 172.21.15.102:51460) 2024-03-26T18:58:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.837+0000 7f59537fe700 1 -- 172.21.15.102:0/1394558245 learned_addr learned my addr 172.21.15.102:0/1394558245 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.837+0000 7f5953fff700 1 --2- 172.21.15.102:0/1394558245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59540ff0a0 0x7f5954195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.838+0000 7f59537fe700 1 -- 172.21.15.102:0/1394558245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59540ff0a0 msgr2=0x7f5954195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.838+0000 7f59537fe700 1 --2- 172.21.15.102:0/1394558245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59540ff0a0 0x7f5954195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.838+0000 7f59537fe700 1 -- 172.21.15.102:0/1394558245 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59480095f0 con 0x7f59541002a0 2024-03-26T18:58:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.838+0000 7f59537fe700 1 --2- 172.21.15.102:0/1394558245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59541002a0 0x7f5954195ed0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5948000c00 tx=0x7f594800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.838+0000 7f59517fa700 1 -- 172.21.15.102:0/1394558245 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f594801d070 con 0x7f59541002a0 2024-03-26T18:58:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.838+0000 7f59517fa700 1 -- 172.21.15.102:0/1394558245 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5948005dc0 con 0x7f59541002a0 2024-03-26T18:58:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.838+0000 7f595a7f5700 1 -- 172.21.15.102:0/1394558245 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595419b080 con 0x7f59541002a0 2024-03-26T18:58:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.838+0000 7f59517fa700 1 -- 172.21.15.102:0/1394558245 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59480226a0 con 0x7f59541002a0 2024-03-26T18:58:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.838+0000 7f595a7f5700 1 -- 172.21.15.102:0/1394558245 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f595419b570 con 0x7f59541002a0 2024-03-26T18:58:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.839+0000 7f595a7f5700 1 -- 172.21.15.102:0/1394558245 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5954107bf0 con 0x7f59541002a0 2024-03-26T18:58:16.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.839+0000 7f59517fa700 1 -- 172.21.15.102:0/1394558245 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f59480038f0 con 0x7f59541002a0 2024-03-26T18:58:16.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.840+0000 7f59517fa700 1 --2- 172.21.15.102:0/1394558245 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f593c06e460 0x7f593c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:16.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.840+0000 7f59517fa700 1 -- 172.21.15.102:0/1394558245 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f594808f8d0 con 0x7f59541002a0 2024-03-26T18:58:16.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.840+0000 7f5953fff700 1 --2- 172.21.15.102:0/1394558245 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f593c06e460 0x7f593c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:16.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.840+0000 7f5953fff700 1 --2- 172.21.15.102:0/1394558245 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f593c06e460 0x7f593c070910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5954100100 tx=0x7f5944009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:16.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:16.844+0000 7f59517fa700 1 -- 172.21.15.102:0/1394558245 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f594805c600 con 0x7f59541002a0 2024-03-26T18:58:17.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.125+0000 7f595a7f5700 1 -- 172.21.15.102:0/1394558245 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5954107de0 con 0x7f59541002a0 2024-03-26T18:58:17.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.126+0000 7f59517fa700 1 -- 172.21.15.102:0/1394558245 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f5948003680 con 0x7f59541002a0 2024-03-26T18:58:17.119 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:58:17.119 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:58:17.119 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:58:17.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:58:17.119 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:58:17.119 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:58:17.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 5, 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 1 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 9, 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 5 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:58:17.120 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:58:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.128+0000 7f595a7f5700 1 -- 172.21.15.102:0/1394558245 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f593c06e460 msgr2=0x7f593c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.128+0000 7f595a7f5700 1 --2- 172.21.15.102:0/1394558245 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f593c06e460 0x7f593c070910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5954100100 tx=0x7f5944009260 comp rx=0 tx=0).stop 2024-03-26T18:58:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.128+0000 7f595a7f5700 1 -- 172.21.15.102:0/1394558245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59541002a0 msgr2=0x7f5954195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.128+0000 7f595a7f5700 1 --2- 172.21.15.102:0/1394558245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59541002a0 0x7f5954195ed0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5948000c00 tx=0x7f594800f690 comp rx=0 tx=0).stop 2024-03-26T18:58:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.128+0000 7f595a7f5700 1 -- 172.21.15.102:0/1394558245 shutdown_connections 2024-03-26T18:58:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.128+0000 7f595a7f5700 1 --2- 172.21.15.102:0/1394558245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59540ff0a0 0x7f5954195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.128+0000 7f595a7f5700 1 --2- 172.21.15.102:0/1394558245 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f593c06e460 0x7f593c070910 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.128+0000 7f595a7f5700 1 --2- 172.21.15.102:0/1394558245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59541002a0 0x7f5954195ed0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.128+0000 7f595a7f5700 1 -- 172.21.15.102:0/1394558245 >> 172.21.15.102:0/1394558245 conn(0x7f59540fa630 msgr2=0x7f5954068d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.128+0000 7f595a7f5700 1 -- 172.21.15.102:0/1394558245 shutdown_connections 2024-03-26T18:58:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.128+0000 7f595a7f5700 1 -- 172.21.15.102:0/1394558245 wait complete. 2024-03-26T18:58:17.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.233+0000 7fd82fd90700 1 -- 172.21.15.102:0/1703660409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8280ff050 msgr2=0x7fd8280ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.233+0000 7fd82fd90700 1 --2- 172.21.15.102:0/1703660409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8280ff050 0x7fd8280ff460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd818009910 tx=0x7fd818009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:17.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.233+0000 7fd82fd90700 1 -- 172.21.15.102:0/1703660409 shutdown_connections 2024-03-26T18:58:17.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.233+0000 7fd82fd90700 1 --2- 172.21.15.102:0/1703660409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd828100250 0x7fd8281006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.233+0000 7fd82fd90700 1 --2- 172.21.15.102:0/1703660409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8280ff050 0x7fd8280ff460 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.233+0000 7fd82fd90700 1 -- 172.21.15.102:0/1703660409 >> 172.21.15.102:0/1703660409 conn(0x7fd8280fa5e0 msgr2=0x7fd8280fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.233+0000 7fd82fd90700 1 -- 172.21.15.102:0/1703660409 shutdown_connections 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.233+0000 7fd82fd90700 1 -- 172.21.15.102:0/1703660409 wait complete. 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82fd90700 1 Processor -- start 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82fd90700 1 -- start start 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82fd90700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8280ff050 0x7fd828068f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82fd90700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828100250 0x7fd82819caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82fd90700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd828196560 con 0x7fd828100250 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82fd90700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82819d030 con 0x7fd8280ff050 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82d32b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828100250 0x7fd82819caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82d32b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828100250 0x7fd82819caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51480/0 (socket says 172.21.15.102:51480) 2024-03-26T18:58:17.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82d32b700 1 -- 172.21.15.102:0/1545619472 learned_addr learned my addr 172.21.15.102:0/1545619472 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82db2c700 1 --2- 172.21.15.102:0/1545619472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8280ff050 0x7fd828068f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82d32b700 1 -- 172.21.15.102:0/1545619472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8280ff050 msgr2=0x7fd828068f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82d32b700 1 --2- 172.21.15.102:0/1545619472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8280ff050 0x7fd828068f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82d32b700 1 -- 172.21.15.102:0/1545619472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8180095f0 con 0x7fd828100250 2024-03-26T18:58:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.234+0000 7fd82d32b700 1 --2- 172.21.15.102:0/1545619472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828100250 0x7fd82819caf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd82400d900 tx=0x7fd82400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.235+0000 7fd81effd700 1 -- 172.21.15.102:0/1545619472 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd824009b10 con 0x7fd828100250 2024-03-26T18:58:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.235+0000 7fd81effd700 1 -- 172.21.15.102:0/1545619472 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd82400b6e0 con 0x7fd828100250 2024-03-26T18:58:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.235+0000 7fd82fd90700 1 -- 172.21.15.102:0/1545619472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82819d2e0 con 0x7fd828100250 2024-03-26T18:58:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.235+0000 7fd81effd700 1 -- 172.21.15.102:0/1545619472 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd82400f5f0 con 0x7fd828100250 2024-03-26T18:58:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.235+0000 7fd82fd90700 1 -- 172.21.15.102:0/1545619472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82819d830 con 0x7fd828100250 2024-03-26T18:58:17.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.236+0000 7fd82fd90700 1 -- 172.21.15.102:0/1545619472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd828107ba0 con 0x7fd828100250 2024-03-26T18:58:17.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.236+0000 7fd81effd700 1 -- 172.21.15.102:0/1545619472 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd824009c70 con 0x7fd828100250 2024-03-26T18:58:17.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.236+0000 7fd81effd700 1 --2- 172.21.15.102:0/1545619472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81406e460 0x7fd814070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:17.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.237+0000 7fd81effd700 1 -- 172.21.15.102:0/1545619472 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd82408e250 con 0x7fd828100250 2024-03-26T18:58:17.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.237+0000 7fd82db2c700 1 --2- 172.21.15.102:0/1545619472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81406e460 0x7fd814070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:17.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.237+0000 7fd82db2c700 1 --2- 172.21.15.102:0/1545619472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81406e460 0x7fd814070910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd818006010 tx=0x7fd81800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:17.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.241+0000 7fd81effd700 1 -- 172.21.15.102:0/1545619472 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd82405b000 con 0x7fd828100250 2024-03-26T18:58:17.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:17 smithi102 ceph-mon[78322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/354 objects degraded (11.864%), 12 pgs degraded) 2024-03-26T18:58:17.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:17 smithi102 ceph-mon[78322]: Cluster is now healthy 2024-03-26T18:58:17.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:17 smithi102 ceph-mon[78322]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:17.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:17 smithi102 ceph-mon[78322]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:58:17.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.487+0000 7fd82fd90700 1 -- 172.21.15.102:0/1545619472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd82819dec0 con 0x7fd828100250 2024-03-26T18:58:17.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.487+0000 7fd81effd700 1 -- 172.21.15.102:0/1545619472 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1922 (secure 0 0 0) 0x7fd8240167a0 con 0x7fd828100250 2024-03-26T18:58:17.481 INFO:teuthology.orchestra.run.smithi102.stdout:e20 2024-03-26T18:58:17.481 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:58:17.481 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:58:17.481 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:58:17.481 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:58:17.481 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:58:17.481 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:58:17.481 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 20 2024-03-26T18:58:17.481 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:58:16.067523+0000 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:58:17.482 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 14490 members: 14490,24283 2024-03-26T18:58:17.483 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:58:17.484 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:58:17.484 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:58:17.484 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:58:17.484 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:58:17.484 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:58:17.484 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:58:17.484 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:58:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.489+0000 7fd82fd90700 1 -- 172.21.15.102:0/1545619472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81406e460 msgr2=0x7fd814070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.489+0000 7fd82fd90700 1 --2- 172.21.15.102:0/1545619472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81406e460 0x7fd814070910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd818006010 tx=0x7fd81800b540 comp rx=0 tx=0).stop 2024-03-26T18:58:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.489+0000 7fd82fd90700 1 -- 172.21.15.102:0/1545619472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828100250 msgr2=0x7fd82819caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.489+0000 7fd82fd90700 1 --2- 172.21.15.102:0/1545619472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828100250 0x7fd82819caf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd82400d900 tx=0x7fd82400dcc0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.489+0000 7fd82fd90700 1 -- 172.21.15.102:0/1545619472 shutdown_connections 2024-03-26T18:58:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.489+0000 7fd82fd90700 1 --2- 172.21.15.102:0/1545619472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8280ff050 0x7fd828068f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.489+0000 7fd82fd90700 1 --2- 172.21.15.102:0/1545619472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81406e460 0x7fd814070910 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.489+0000 7fd82fd90700 1 --2- 172.21.15.102:0/1545619472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828100250 0x7fd82819caf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.489+0000 7fd82fd90700 1 -- 172.21.15.102:0/1545619472 >> 172.21.15.102:0/1545619472 conn(0x7fd8280fa5e0 msgr2=0x7fd828103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.489+0000 7fd82fd90700 1 -- 172.21.15.102:0/1545619472 shutdown_connections 2024-03-26T18:58:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.490+0000 7fd82fd90700 1 -- 172.21.15.102:0/1545619472 wait complete. 2024-03-26T18:58:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 20 2024-03-26T18:58:17.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:17 smithi190 ceph-mon[65689]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/354 objects degraded (11.864%), 12 pgs degraded) 2024-03-26T18:58:17.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:17 smithi190 ceph-mon[65689]: Cluster is now healthy 2024-03-26T18:58:17.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:17 smithi190 ceph-mon[65689]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:17.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:17 smithi190 ceph-mon[65689]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:58:17.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.595+0000 7f21d0274700 1 -- 172.21.15.102:0/3409527980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21c8100150 msgr2=0x7f21c8100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.595+0000 7f21d0274700 1 --2- 172.21.15.102:0/3409527980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21c8100150 0x7f21c8100560 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f21b8009910 tx=0x7f21b8009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:17.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.595+0000 7f21d0274700 1 -- 172.21.15.102:0/3409527980 shutdown_connections 2024-03-26T18:58:17.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.595+0000 7f21d0274700 1 --2- 172.21.15.102:0/3409527980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c8101350 0x7f21c81017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.595+0000 7f21d0274700 1 --2- 172.21.15.102:0/3409527980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21c8100150 0x7f21c8100560 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.595+0000 7f21d0274700 1 -- 172.21.15.102:0/3409527980 >> 172.21.15.102:0/3409527980 conn(0x7f21c80fb6e0 msgr2=0x7f21c80fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.595+0000 7f21d0274700 1 -- 172.21.15.102:0/3409527980 shutdown_connections 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.595+0000 7f21d0274700 1 -- 172.21.15.102:0/3409527980 wait complete. 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.595+0000 7f21d0274700 1 Processor -- start 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.595+0000 7f21d0274700 1 -- start start 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21d0274700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21c8100150 0x7f21c81959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21d0274700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c8101350 0x7f21c8195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21d0274700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c8196500 con 0x7f21c8100150 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21d0274700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c8196640 con 0x7f21c8101350 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21ce010700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21c8100150 0x7f21c81959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21ce010700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21c8100150 0x7f21c81959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51494/0 (socket says 172.21.15.102:51494) 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21ce010700 1 -- 172.21.15.102:0/3278780135 learned_addr learned my addr 172.21.15.102:0/3278780135 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21cd80f700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c8101350 0x7f21c8195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21ce010700 1 -- 172.21.15.102:0/3278780135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c8101350 msgr2=0x7f21c8195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21ce010700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c8101350 0x7f21c8195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21ce010700 1 -- 172.21.15.102:0/3278780135 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b80095f0 con 0x7f21c8100150 2024-03-26T18:58:17.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21cd80f700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c8101350 0x7f21c8195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T18:58:17.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21ce010700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21c8100150 0x7f21c81959a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f21b800bd70 tx=0x7f21b800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:17.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21bf7fe700 1 -- 172.21.15.102:0/3278780135 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21b801d070 con 0x7f21c8100150 2024-03-26T18:58:17.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21d0274700 1 -- 172.21.15.102:0/3278780135 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c819b090 con 0x7f21c8100150 2024-03-26T18:58:17.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21bf7fe700 1 -- 172.21.15.102:0/3278780135 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f21b800f460 con 0x7f21c8100150 2024-03-26T18:58:17.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21bf7fe700 1 -- 172.21.15.102:0/3278780135 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21b8021620 con 0x7f21c8100150 2024-03-26T18:58:17.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.596+0000 7f21d0274700 1 -- 172.21.15.102:0/3278780135 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c819b580 con 0x7f21c8100150 2024-03-26T18:58:17.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.597+0000 7f21d0274700 1 -- 172.21.15.102:0/3278780135 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c8108ca0 con 0x7f21c8100150 2024-03-26T18:58:17.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.598+0000 7f21bf7fe700 1 -- 172.21.15.102:0/3278780135 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f21b800fa40 con 0x7f21c8100150 2024-03-26T18:58:17.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.598+0000 7f21bf7fe700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f21b406e460 0x7f21b4070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:17.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.598+0000 7f21bf7fe700 1 -- 172.21.15.102:0/3278780135 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f21b808f6d0 con 0x7f21c8100150 2024-03-26T18:58:17.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.598+0000 7f21cd80f700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f21b406e460 0x7f21b4070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:17.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.599+0000 7f21cd80f700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f21b406e460 0x7f21b4070910 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005fd0 tx=0x7f21c4005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:17.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.603+0000 7f21bf7fe700 1 -- 172.21.15.102:0/3278780135 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f21b805c400 con 0x7f21c8100150 2024-03-26T18:58:17.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.825+0000 7f21d0274700 1 -- 172.21.15.102:0/3278780135 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21c8105ca0 con 0x7f21b406e460 2024-03-26T18:58:17.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.827+0000 7f21bf7fe700 1 -- 172.21.15.102:0/3278780135 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f21c8105ca0 con 0x7f21b406e460 2024-03-26T18:58:17.820 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:58:17.820 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:58:17.820 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:58:17.820 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:58:17.820 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T18:58:17.820 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T18:58:17.820 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-03-26T18:58:17.821 INFO:teuthology.orchestra.run.smithi102.stdout: "mon" 2024-03-26T18:58:17.821 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T18:58:17.821 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "7/23 daemons upgraded", 2024-03-26T18:58:17.821 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T18:58:17.821 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:58:17.821 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:58:17.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.829+0000 7f21d0274700 1 -- 172.21.15.102:0/3278780135 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f21b406e460 msgr2=0x7f21b4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.829+0000 7f21d0274700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f21b406e460 0x7f21b4070910 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005fd0 tx=0x7f21c4005ee0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.829+0000 7f21d0274700 1 -- 172.21.15.102:0/3278780135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21c8100150 msgr2=0x7f21c81959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.829+0000 7f21d0274700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21c8100150 0x7f21c81959a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f21b800bd70 tx=0x7f21b800be50 comp rx=0 tx=0).stop 2024-03-26T18:58:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.829+0000 7f21d0274700 1 -- 172.21.15.102:0/3278780135 shutdown_connections 2024-03-26T18:58:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.829+0000 7f21d0274700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f21b406e460 0x7f21b4070910 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.829+0000 7f21d0274700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21c8100150 0x7f21c81959a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.829+0000 7f21d0274700 1 --2- 172.21.15.102:0/3278780135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c8101350 0x7f21c8195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.829+0000 7f21d0274700 1 -- 172.21.15.102:0/3278780135 >> 172.21.15.102:0/3278780135 conn(0x7f21c80fb6e0 msgr2=0x7f21c8104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.829+0000 7f21d0274700 1 -- 172.21.15.102:0/3278780135 shutdown_connections 2024-03-26T18:58:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.829+0000 7f21d0274700 1 -- 172.21.15.102:0/3278780135 wait complete. 2024-03-26T18:58:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.933+0000 7f056f7ab700 1 -- 172.21.15.102:0/1803939953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0568103520 msgr2=0x7f0568105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.933+0000 7f056f7ab700 1 --2- 172.21.15.102:0/1803939953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0568103520 0x7f0568105900 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0558009910 tx=0x7f0558009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.933+0000 7f056f7ab700 1 -- 172.21.15.102:0/1803939953 shutdown_connections 2024-03-26T18:58:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.933+0000 7f056f7ab700 1 --2- 172.21.15.102:0/1803939953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0568103520 0x7f0568105900 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.933+0000 7f056f7ab700 1 --2- 172.21.15.102:0/1803939953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0568100c00 0x7f0568102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.933+0000 7f056f7ab700 1 -- 172.21.15.102:0/1803939953 >> 172.21.15.102:0/1803939953 conn(0x7f05680fa600 msgr2=0x7f05680fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.933+0000 7f056f7ab700 1 -- 172.21.15.102:0/1803939953 shutdown_connections 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.933+0000 7f056f7ab700 1 -- 172.21.15.102:0/1803939953 wait complete. 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.934+0000 7f056f7ab700 1 Processor -- start 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.934+0000 7f056f7ab700 1 -- start start 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.934+0000 7f056f7ab700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0568100c00 0x7f0568195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.934+0000 7f056f7ab700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0568103520 0x7f0568195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.934+0000 7f056f7ab700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05681964d0 con 0x7f0568100c00 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.934+0000 7f056f7ab700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0568196610 con 0x7f0568103520 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.934+0000 7f056d547700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0568100c00 0x7f0568195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.934+0000 7f056d547700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0568100c00 0x7f0568195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51512/0 (socket says 172.21.15.102:51512) 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.934+0000 7f056d547700 1 -- 172.21.15.102:0/1875668406 learned_addr learned my addr 172.21.15.102:0/1875668406 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.934+0000 7f056cd46700 1 --2- 172.21.15.102:0/1875668406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0568103520 0x7f0568195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.935+0000 7f056d547700 1 -- 172.21.15.102:0/1875668406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0568103520 msgr2=0x7f0568195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.935+0000 7f056d547700 1 --2- 172.21.15.102:0/1875668406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0568103520 0x7f0568195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.935+0000 7f056d547700 1 -- 172.21.15.102:0/1875668406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05580095f0 con 0x7f0568100c00 2024-03-26T18:58:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.935+0000 7f056d547700 1 --2- 172.21.15.102:0/1875668406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0568100c00 0x7f0568195970 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f056400d8d0 tx=0x7f056400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.935+0000 7f055e7fc700 1 -- 172.21.15.102:0/1875668406 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05640098a0 con 0x7f0568100c00 2024-03-26T18:58:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.935+0000 7f055e7fc700 1 -- 172.21.15.102:0/1875668406 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0564010460 con 0x7f0568100c00 2024-03-26T18:58:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.935+0000 7f056f7ab700 1 -- 172.21.15.102:0/1875668406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056819b0c0 con 0x7f0568100c00 2024-03-26T18:58:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.935+0000 7f055e7fc700 1 -- 172.21.15.102:0/1875668406 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056400f5f0 con 0x7f0568100c00 2024-03-26T18:58:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.935+0000 7f056f7ab700 1 -- 172.21.15.102:0/1875668406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056819b610 con 0x7f0568100c00 2024-03-26T18:58:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.936+0000 7f056f7ab700 1 -- 172.21.15.102:0/1875668406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056818fb90 con 0x7f0568100c00 2024-03-26T18:58:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.936+0000 7f055e7fc700 1 -- 172.21.15.102:0/1875668406 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f0564009a00 con 0x7f0568100c00 2024-03-26T18:58:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.937+0000 7f055e7fc700 1 --2- 172.21.15.102:0/1875668406 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f055406e460 0x7f0554070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.937+0000 7f055e7fc700 1 -- 172.21.15.102:0/1875668406 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f056408e1b0 con 0x7f0568100c00 2024-03-26T18:58:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.937+0000 7f056cd46700 1 --2- 172.21.15.102:0/1875668406 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f055406e460 0x7f0554070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:17.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.937+0000 7f056cd46700 1 --2- 172.21.15.102:0/1875668406 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f055406e460 0x7f0554070910 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0558006010 tx=0x7f055800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:17.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:17.941+0000 7f055e7fc700 1 -- 172.21.15.102:0/1875668406 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f056405af60 con 0x7f0568100c00 2024-03-26T18:58:18.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.225+0000 7f056f7ab700 1 -- 172.21.15.102:0/1875668406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f056802cc30 con 0x7f0568100c00 2024-03-26T18:58:18.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.226+0000 7f055e7fc700 1 -- 172.21.15.102:0/1875668406 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f056405aaf0 con 0x7f0568100c00 2024-03-26T18:58:18.219 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T18:58:18.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.228+0000 7f056f7ab700 1 -- 172.21.15.102:0/1875668406 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f055406e460 msgr2=0x7f0554070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:18.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.228+0000 7f056f7ab700 1 --2- 172.21.15.102:0/1875668406 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f055406e460 0x7f0554070910 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0558006010 tx=0x7f055800b540 comp rx=0 tx=0).stop 2024-03-26T18:58:18.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.228+0000 7f056f7ab700 1 -- 172.21.15.102:0/1875668406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0568100c00 msgr2=0x7f0568195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:18.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.228+0000 7f056f7ab700 1 --2- 172.21.15.102:0/1875668406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0568100c00 0x7f0568195970 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f056400d8d0 tx=0x7f056400dbe0 comp rx=0 tx=0).stop 2024-03-26T18:58:18.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.228+0000 7f056f7ab700 1 -- 172.21.15.102:0/1875668406 shutdown_connections 2024-03-26T18:58:18.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.228+0000 7f056f7ab700 1 --2- 172.21.15.102:0/1875668406 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f055406e460 0x7f0554070910 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:18.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.228+0000 7f056f7ab700 1 --2- 172.21.15.102:0/1875668406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0568100c00 0x7f0568195970 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:18.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.228+0000 7f056f7ab700 1 --2- 172.21.15.102:0/1875668406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0568103520 0x7f0568195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:18.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.228+0000 7f056f7ab700 1 -- 172.21.15.102:0/1875668406 >> 172.21.15.102:0/1875668406 conn(0x7f05680fa600 msgr2=0x7f05680fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:18.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.228+0000 7f056f7ab700 1 -- 172.21.15.102:0/1875668406 shutdown_connections 2024-03-26T18:58:18.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:18.228+0000 7f056f7ab700 1 -- 172.21.15.102:0/1875668406 wait complete. 2024-03-26T18:58:18.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:18 smithi102 ceph-mon[78322]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:18.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:18 smithi102 ceph-mon[78322]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:18.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:18 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1394558245' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:18.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:18 smithi102 ceph-mon[78322]: pgmap v58: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 35 B/s, 0 objects/s recovering 2024-03-26T18:58:18.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:18 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1545619472' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:58:18.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:18 smithi190 ceph-mon[65689]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:18.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:18 smithi190 ceph-mon[65689]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:18.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:18 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1394558245' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:18.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:18 smithi190 ceph-mon[65689]: pgmap v58: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 35 B/s, 0 objects/s recovering 2024-03-26T18:58:18.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:18 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1545619472' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:58:19.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:19 smithi102 ceph-mon[78322]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:19.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:19 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1875668406' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:58:19.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:19 smithi190 ceph-mon[65689]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:19.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:19 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1875668406' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:58:20.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:20 smithi102 ceph-mon[78322]: pgmap v59: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 29 B/s, 0 objects/s recovering 2024-03-26T18:58:20.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:20 smithi190 ceph-mon[65689]: pgmap v59: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 29 B/s, 0 objects/s recovering 2024-03-26T18:58:22.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:22 smithi190 ceph-mon[65689]: pgmap v60: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 26 B/s, 0 objects/s recovering 2024-03-26T18:58:22.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:22 smithi102 ceph-mon[78322]: pgmap v60: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 26 B/s, 0 objects/s recovering 2024-03-26T18:58:23.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:23.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:58:23.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:23.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:23.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:58:23.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:24.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:24 smithi102 ceph-mon[78322]: pgmap v61: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-26T18:58:24.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:24 smithi190 ceph-mon[65689]: pgmap v61: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-26T18:58:26.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:26 smithi102 ceph-mon[78322]: pgmap v62: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-26T18:58:26.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:26 smithi190 ceph-mon[65689]: pgmap v62: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-26T18:58:27.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:27 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T18:58:27.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:27 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T18:58:28.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:28 smithi102 ceph-mon[78322]: pgmap v63: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:28.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:28 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T18:58:28.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:28 smithi102 ceph-mon[78322]: Upgrade: osd.1 is safe to restart 2024-03-26T18:58:28.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:28 smithi190 ceph-mon[65689]: pgmap v63: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:28.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:28 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T18:58:28.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:28 smithi190 ceph-mon[65689]: Upgrade: osd.1 is safe to restart 2024-03-26T18:58:30.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:29 smithi102 ceph-mon[78322]: Upgrade: Updating osd.1 2024-03-26T18:58:30.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:29 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:30.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:29 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T18:58:30.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:29 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:58:30.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:29 smithi102 ceph-mon[78322]: Deploying daemon osd.1 on smithi102 2024-03-26T18:58:30.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:29 smithi102 ceph-mon[78322]: pgmap v64: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:30.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:29 smithi190 ceph-mon[65689]: Upgrade: Updating osd.1 2024-03-26T18:58:30.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:29 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:30.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:29 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T18:58:30.280 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:29 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:58:30.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:29 smithi190 ceph-mon[65689]: Deploying daemon osd.1 on smithi102 2024-03-26T18:58:30.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:29 smithi190 ceph-mon[65689]: pgmap v64: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:32.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:32 smithi102 ceph-mon[78322]: pgmap v65: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:32.377 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:32 smithi102 systemd[1]: Stopping Ceph osd.1 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:58:32.377 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:32 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[49428]: 2024-03-26T18:58:32.325+0000 7fcce91b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T18:58:32.377 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:32 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[49428]: 2024-03-26T18:58:32.325+0000 7fcce91b8700 -1 osd.1 55 *** Got signal Terminated *** 2024-03-26T18:58:32.377 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:32 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[49428]: 2024-03-26T18:58:32.325+0000 7fcce91b8700 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T18:58:32.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:32 smithi190 ceph-mon[65689]: pgmap v65: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:33.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:33 smithi190 ceph-mon[65689]: osd.1 marked itself down and dead 2024-03-26T18:58:33.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:33 smithi102 ceph-mon[78322]: osd.1 marked itself down and dead 2024-03-26T18:58:34.002 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:33 smithi102 podman[87645]: 2024-03-26 18:58:33.654854147 +0000 UTC m=+1.424138974 container died 0bf773515cc1a04046fd09bc19b92e8dcf91f84375e7fb42adeca179544d6b50 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_BRANCH=HEAD) 2024-03-26T18:58:34.002 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:34 smithi102 podman[87645]: 2024-03-26 18:58:34.009697679 +0000 UTC m=+1.778982493 container cleanup 0bf773515cc1a04046fd09bc19b92e8dcf91f84375e7fb42adeca179544d6b50 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T18:58:34.276 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:34 smithi102 bash[87645]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1 2024-03-26T18:58:34.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:34 smithi190 ceph-mon[65689]: pgmap v66: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:34.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:34 smithi190 ceph-mon[65689]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T18:58:34.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:34 smithi190 ceph-mon[65689]: osdmap e56: 6 total, 5 up, 6 in 2024-03-26T18:58:34.615 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:34 smithi102 ceph-mon[78322]: pgmap v66: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:34.615 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:34 smithi102 ceph-mon[78322]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T18:58:34.615 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:34 smithi102 ceph-mon[78322]: osdmap e56: 6 total, 5 up, 6 in 2024-03-26T18:58:34.616 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:34 smithi102 podman[87665]: 2024-03-26 18:58:34.303022421 +0000 UTC m=+0.640326862 container remove 0bf773515cc1a04046fd09bc19b92e8dcf91f84375e7fb42adeca179544d6b50 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=-18.2.0, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T18:58:34.874 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:34 smithi102 podman[87754]: 2024-03-26 18:58:34.526435892 +0000 UTC m=+0.018531791 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:58:34.874 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:34 smithi102 podman[87754]: 2024-03-26 18:58:34.653873046 +0000 UTC m=+0.145968917 container create b3268f248f7e67977da340ed75d0726abf67f9bc50c6314ea818e6e3b1518d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-26T18:58:34.874 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:34 smithi102 podman[87754]: 2024-03-26 18:58:34.881355724 +0000 UTC m=+0.373451607 container init b3268f248f7e67977da340ed75d0726abf67f9bc50c6314ea818e6e3b1518d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T18:58:35.126 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:34 smithi102 podman[87754]: 2024-03-26 18:58:34.884769145 +0000 UTC m=+0.376865026 container start b3268f248f7e67977da340ed75d0726abf67f9bc50c6314ea818e6e3b1518d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T18:58:35.126 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:34 smithi102 podman[87754]: 2024-03-26 18:58:34.973095576 +0000 UTC m=+0.465191462 container attach b3268f248f7e67977da340ed75d0726abf67f9bc50c6314ea818e6e3b1518d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T18:58:35.468 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:35 smithi102 ceph-mon[78322]: osdmap e57: 6 total, 5 up, 6 in 2024-03-26T18:58:35.468 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:35 smithi102 podman[87754]: 2024-03-26 18:58:35.182412269 +0000 UTC m=+0.674508148 container died b3268f248f7e67977da340ed75d0726abf67f9bc50c6314ea818e6e3b1518d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240320.145742-3605160, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-26T18:58:35.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:35 smithi190 ceph-mon[65689]: osdmap e57: 6 total, 5 up, 6 in 2024-03-26T18:58:36.126 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:35 smithi102 podman[87780]: 2024-03-26 18:58:35.74369723 +0000 UTC m=+0.551445376 container remove b3268f248f7e67977da340ed75d0726abf67f9bc50c6314ea818e6e3b1518d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T18:58:36.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:35 smithi102 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-26T18:58:36.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:35 smithi102 systemd[1]: Stopped Ceph osd.1 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:58:36.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:35 smithi102 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.1.service: Consumed 1min 2.473s CPU time. 2024-03-26T18:58:36.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:35 smithi102 systemd[1]: Starting Ceph osd.1 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:58:36.457 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:36 smithi102 ceph-mon[78322]: pgmap v69: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:36.457 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:36 smithi102 ceph-mon[78322]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-26T18:58:36.458 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 podman[87886]: 2024-03-26 18:58:36.093163214 +0000 UTC m=+0.017656023 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:58:36.458 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 podman[87886]: 2024-03-26 18:58:36.21166751 +0000 UTC m=+0.136160292 container create ac04c2cefcfc2401074e20bfee2e32c27338617b19f57f7b6c645c62d319ba17 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-26T18:58:36.458 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 podman[87886]: 2024-03-26 18:58:36.464573387 +0000 UTC m=+0.389066170 container init ac04c2cefcfc2401074e20bfee2e32c27338617b19f57f7b6c645c62d319ba17 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T18:58:36.751 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 podman[87886]: 2024-03-26 18:58:36.46835406 +0000 UTC m=+0.392846847 container start ac04c2cefcfc2401074e20bfee2e32c27338617b19f57f7b6c645c62d319ba17 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T18:58:36.751 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 podman[87886]: 2024-03-26 18:58:36.564389459 +0000 UTC m=+0.488882249 container attach ac04c2cefcfc2401074e20bfee2e32c27338617b19f57f7b6c645c62d319ba17 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T18:58:36.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:36 smithi190 ceph-mon[65689]: pgmap v69: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:36.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:36 smithi190 ceph-mon[65689]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-26T18:58:37.126 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate[87901]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T18:58:37.126 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate[87901]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T18:58:37.126 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 bash[87886]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T18:58:37.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 bash[87886]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T18:58:37.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 bash[87886]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T18:58:37.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 bash[87886]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T18:58:37.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 bash[87886]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-26T18:58:37.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 bash[87886]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T18:58:37.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 bash[87886]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-26T18:58:37.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate[87901]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T18:58:37.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate[87901]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T18:58:37.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate[87901]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-26T18:58:37.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate[87901]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T18:58:37.127 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate[87901]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-26T18:58:37.128 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:36 smithi102 podman[87886]: 2024-03-26 18:58:36.93519871 +0000 UTC m=+0.859691493 container died ac04c2cefcfc2401074e20bfee2e32c27338617b19f57f7b6c645c62d319ba17 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T18:58:37.626 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:37 smithi102 podman[88041]: 2024-03-26 18:58:37.502091589 +0000 UTC m=+0.561840602 container remove ac04c2cefcfc2401074e20bfee2e32c27338617b19f57f7b6c645c62d319ba17 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T18:58:37.981 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:37 smithi102 podman[88106]: 2024-03-26 18:58:37.6157964 +0000 UTC m=+0.018148733 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:58:37.981 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:37 smithi102 podman[88106]: 2024-03-26 18:58:37.736298179 +0000 UTC m=+0.138650541 container create 3149459e0c02ee53695e81d55c84d24dd5345b9bed3f7f5ef54c1244ecb829c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T18:58:38.271 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:37 smithi102 podman[88106]: 2024-03-26 18:58:37.989384048 +0000 UTC m=+0.391736371 container init 3149459e0c02ee53695e81d55c84d24dd5345b9bed3f7f5ef54c1244ecb829c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T18:58:38.271 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:37 smithi102 podman[88106]: 2024-03-26 18:58:37.992599253 +0000 UTC m=+0.394951582 container start 3149459e0c02ee53695e81d55c84d24dd5345b9bed3f7f5ef54c1244ecb829c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-26T18:58:38.271 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:38 smithi102 ceph-osd[88125]: -- 172.21.15.102:0/1594991861 <== mon.0 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561a29a35680 con 0x561a299ff000 2024-03-26T18:58:38.271 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:38 smithi102 bash[88106]: 3149459e0c02ee53695e81d55c84d24dd5345b9bed3f7f5ef54c1244ecb829c9 2024-03-26T18:58:38.271 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:38 smithi102 systemd[1]: Started Ceph osd.1 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:58:38.528 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:38 smithi102 ceph-mon[78322]: pgmap v70: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:38.529 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:38.529 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:58:38.529 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:38.529 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:38.529 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:58:38.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:38 smithi190 ceph-mon[65689]: pgmap v70: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:38.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:38.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:58:38.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:38.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:38.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:58:38.876 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:38 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[88121]: 2024-03-26T18:58:38.535+0000 7faa7bf277c0 -1 Falling back to public interface 2024-03-26T18:58:39.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:39 smithi102 ceph-mon[78322]: Health check failed: Degraded data redundancy: 55/354 objects degraded (15.537%), 16 pgs degraded (PG_DEGRADED) 2024-03-26T18:58:39.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:39 smithi102 ceph-mon[78322]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-26T18:58:39.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:39 smithi190 ceph-mon[65689]: Health check failed: Degraded data redundancy: 55/354 objects degraded (15.537%), 16 pgs degraded (PG_DEGRADED) 2024-03-26T18:58:39.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:39 smithi190 ceph-mon[65689]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-26T18:58:40.591 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:40 smithi102 ceph-mon[78322]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 55/354 objects degraded (15.537%) 2024-03-26T18:58:40.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:40 smithi190 ceph-mon[65689]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 55/354 objects degraded (15.537%) 2024-03-26T18:58:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:42 smithi102 ceph-mon[78322]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 55/354 objects degraded (15.537%) 2024-03-26T18:58:42.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:42 smithi190 ceph-mon[65689]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 55/354 objects degraded (15.537%) 2024-03-26T18:58:43.376 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:43 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[88121]: 2024-03-26T18:58:43.094+0000 7faa7bf277c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-26T18:58:43.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:43 smithi102 ceph-mon[78322]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:58:43.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:43.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:43.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:43 smithi102 ceph-mon[78322]: pgmap v73: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 55/354 objects degraded (15.537%) 2024-03-26T18:58:43.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:43 smithi190 ceph-mon[65689]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:active} 2 up:standby 2024-03-26T18:58:43.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:43.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:43.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:43 smithi190 ceph-mon[65689]: pgmap v73: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 55/354 objects degraded (15.537%) 2024-03-26T18:58:43.876 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:43 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[88121]: 2024-03-26T18:58:43.598+0000 7faa7bf277c0 -1 osd.1 55 log_to_monitors true 2024-03-26T18:58:44.626 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 18:58:44 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[88121]: 2024-03-26T18:58:44.365+0000 7faa6d37f700 -1 osd.1 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T18:58:44.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:44 smithi102 ceph-mon[78322]: from='osd.1 [v2:172.21.15.102:6810/3336954503,v1:172.21.15.102:6811/3336954503]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T18:58:44.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:44 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:44.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:44 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:44.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:44 smithi190 ceph-mon[65689]: from='osd.1 [v2:172.21.15.102:6810/3336954503,v1:172.21.15.102:6811/3336954503]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T18:58:44.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:44 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:44.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:44 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:45.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:45 smithi102 ceph-mon[78322]: from='osd.1 [v2:172.21.15.102:6810/3336954503,v1:172.21.15.102:6811/3336954503]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T18:58:45.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:45 smithi102 ceph-mon[78322]: osdmap e58: 6 total, 5 up, 6 in 2024-03-26T18:58:45.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:45 smithi102 ceph-mon[78322]: from='osd.1 [v2:172.21.15.102:6810/3336954503,v1:172.21.15.102:6811/3336954503]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:58:45.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:45 smithi102 ceph-mon[78322]: pgmap v75: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 55/354 objects degraded (15.537%) 2024-03-26T18:58:45.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:45 smithi190 ceph-mon[65689]: from='osd.1 [v2:172.21.15.102:6810/3336954503,v1:172.21.15.102:6811/3336954503]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T18:58:45.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:45 smithi190 ceph-mon[65689]: osdmap e58: 6 total, 5 up, 6 in 2024-03-26T18:58:45.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:45 smithi190 ceph-mon[65689]: from='osd.1 [v2:172.21.15.102:6810/3336954503,v1:172.21.15.102:6811/3336954503]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:58:45.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:45 smithi190 ceph-mon[65689]: pgmap v75: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 55/354 objects degraded (15.537%) 2024-03-26T18:58:46.439 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:46 smithi102 ceph-mon[78322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T18:58:46.440 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:46 smithi102 ceph-mon[78322]: osd.1 [v2:172.21.15.102:6810/3336954503,v1:172.21.15.102:6811/3336954503] boot 2024-03-26T18:58:46.440 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:46 smithi102 ceph-mon[78322]: osdmap e59: 6 total, 6 up, 6 in 2024-03-26T18:58:46.440 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:58:46.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:46 smithi190 ceph-mon[65689]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T18:58:46.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:46 smithi190 ceph-mon[65689]: osd.1 [v2:172.21.15.102:6810/3336954503,v1:172.21.15.102:6811/3336954503] boot 2024-03-26T18:58:46.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:46 smithi190 ceph-mon[65689]: osdmap e59: 6 total, 6 up, 6 in 2024-03-26T18:58:46.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:46 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T18:58:47.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:47 smithi102 ceph-mon[78322]: osdmap e60: 6 total, 6 up, 6 in 2024-03-26T18:58:47.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:47 smithi102 ceph-mon[78322]: pgmap v78: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 55/354 objects degraded (15.537%) 2024-03-26T18:58:47.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:47 smithi190 ceph-mon[65689]: osdmap e60: 6 total, 6 up, 6 in 2024-03-26T18:58:47.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:47 smithi190 ceph-mon[65689]: pgmap v78: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 55/354 objects degraded (15.537%) 2024-03-26T18:58:48.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.338+0000 7f87bfd76700 1 -- 172.21.15.102:0/2150872238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b8100be0 msgr2=0x7f87b8102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:48.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.338+0000 7f87bfd76700 1 --2- 172.21.15.102:0/2150872238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b8100be0 0x7f87b8102fc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009910 tx=0x7f87a8009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.338+0000 7f87bfd76700 1 -- 172.21.15.102:0/2150872238 shutdown_connections 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.338+0000 7f87bfd76700 1 --2- 172.21.15.102:0/2150872238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87b8103500 0x7f87b81058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.338+0000 7f87bfd76700 1 --2- 172.21.15.102:0/2150872238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b8100be0 0x7f87b8102fc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.338+0000 7f87bfd76700 1 -- 172.21.15.102:0/2150872238 >> 172.21.15.102:0/2150872238 conn(0x7f87b80fa5e0 msgr2=0x7f87b80fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.338+0000 7f87bfd76700 1 -- 172.21.15.102:0/2150872238 shutdown_connections 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.338+0000 7f87bfd76700 1 -- 172.21.15.102:0/2150872238 wait complete. 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bfd76700 1 Processor -- start 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bfd76700 1 -- start start 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bfd76700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87b8100be0 0x7f87b8195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bfd76700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b8103500 0x7f87b8195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bfd76700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b8196480 con 0x7f87b8103500 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bfd76700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b81965c0 con 0x7f87b8100be0 2024-03-26T18:58:48.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bd311700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b8103500 0x7f87b8195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bd311700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b8103500 0x7f87b8195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40532/0 (socket says 172.21.15.102:40532) 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bd311700 1 -- 172.21.15.102:0/225711130 learned_addr learned my addr 172.21.15.102:0/225711130 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bdb12700 1 --2- 172.21.15.102:0/225711130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87b8100be0 0x7f87b8195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bd311700 1 -- 172.21.15.102:0/225711130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87b8100be0 msgr2=0x7f87b8195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bd311700 1 --2- 172.21.15.102:0/225711130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87b8100be0 0x7f87b8195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bd311700 1 -- 172.21.15.102:0/225711130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a80095f0 con 0x7f87b8103500 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.339+0000 7f87bd311700 1 --2- 172.21.15.102:0/225711130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b8103500 0x7f87b8195e60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f87b400d8d0 tx=0x7f87b400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.340+0000 7f87aeffd700 1 -- 172.21.15.102:0/225711130 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87b40098a0 con 0x7f87b8103500 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.340+0000 7f87bfd76700 1 -- 172.21.15.102:0/225711130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b819b070 con 0x7f87b8103500 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.340+0000 7f87aeffd700 1 -- 172.21.15.102:0/225711130 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f87b4010460 con 0x7f87b8103500 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.340+0000 7f87bfd76700 1 -- 172.21.15.102:0/225711130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b819b5c0 con 0x7f87b8103500 2024-03-26T18:58:48.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.340+0000 7f87aeffd700 1 -- 172.21.15.102:0/225711130 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87b400f5f0 con 0x7f87b8103500 2024-03-26T18:58:48.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.341+0000 7f87bfd76700 1 -- 172.21.15.102:0/225711130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b818fb70 con 0x7f87b8103500 2024-03-26T18:58:48.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.342+0000 7f87aeffd700 1 -- 172.21.15.102:0/225711130 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f87b4009a00 con 0x7f87b8103500 2024-03-26T18:58:48.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.342+0000 7f87aeffd700 1 --2- 172.21.15.102:0/225711130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f87a406e460 0x7f87a4070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:48.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.342+0000 7f87aeffd700 1 -- 172.21.15.102:0/225711130 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f87b408e250 con 0x7f87b8103500 2024-03-26T18:58:48.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.342+0000 7f87bdb12700 1 --2- 172.21.15.102:0/225711130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f87a406e460 0x7f87a4070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:48.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.342+0000 7f87bdb12700 1 --2- 172.21.15.102:0/225711130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f87a406e460 0x7f87a4070910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f87a8006010 tx=0x7f87a800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:48.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.351+0000 7f87aeffd700 1 -- 172.21.15.102:0/225711130 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f87b405af80 con 0x7f87b8103500 2024-03-26T18:58:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.577+0000 7f87bfd76700 1 -- 172.21.15.102:0/225711130 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f87b8061190 con 0x7f87a406e460 2024-03-26T18:58:48.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.578+0000 7f87aeffd700 1 -- 172.21.15.102:0/225711130 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f87b8061190 con 0x7f87a406e460 2024-03-26T18:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.581+0000 7f87bfd76700 1 -- 172.21.15.102:0/225711130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f87a406e460 msgr2=0x7f87a4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.581+0000 7f87bfd76700 1 --2- 172.21.15.102:0/225711130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f87a406e460 0x7f87a4070910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f87a8006010 tx=0x7f87a800b540 comp rx=0 tx=0).stop 2024-03-26T18:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.581+0000 7f87bfd76700 1 -- 172.21.15.102:0/225711130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b8103500 msgr2=0x7f87b8195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.581+0000 7f87bfd76700 1 --2- 172.21.15.102:0/225711130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b8103500 0x7f87b8195e60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f87b400d8d0 tx=0x7f87b400dbe0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.581+0000 7f87bfd76700 1 -- 172.21.15.102:0/225711130 shutdown_connections 2024-03-26T18:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.581+0000 7f87bfd76700 1 --2- 172.21.15.102:0/225711130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87b8100be0 0x7f87b8195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.581+0000 7f87bfd76700 1 --2- 172.21.15.102:0/225711130 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f87a406e460 0x7f87a4070910 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.581+0000 7f87bfd76700 1 --2- 172.21.15.102:0/225711130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b8103500 0x7f87b8195e60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.581+0000 7f87bfd76700 1 -- 172.21.15.102:0/225711130 >> 172.21.15.102:0/225711130 conn(0x7f87b80fa5e0 msgr2=0x7f87b80fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:48.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.581+0000 7f87bfd76700 1 -- 172.21.15.102:0/225711130 shutdown_connections 2024-03-26T18:58:48.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.581+0000 7f87bfd76700 1 -- 172.21.15.102:0/225711130 wait complete. 2024-03-26T18:58:48.584 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:58:48.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:48.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:48.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:58:48.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:58:48.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:48.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:58:48.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:48.585 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:48.586 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:48.586 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T18:58:48.586 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T18:58:48.586 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:48 smithi102 ceph-mon[78322]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T18:58:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.688+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/482158133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe34100170 msgr2=0x7fbe34100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.688+0000 7fbe3a2d7700 1 --2- 172.21.15.102:0/482158133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe34100170 0x7fbe34100580 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009910 tx=0x7fbe24009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.689+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/482158133 shutdown_connections 2024-03-26T18:58:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.689+0000 7fbe3a2d7700 1 --2- 172.21.15.102:0/482158133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe34101370 0x7fbe341017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.689+0000 7fbe3a2d7700 1 --2- 172.21.15.102:0/482158133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe34100170 0x7fbe34100580 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.689+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/482158133 >> 172.21.15.102:0/482158133 conn(0x7fbe340fb6e0 msgr2=0x7fbe340fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.689+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/482158133 shutdown_connections 2024-03-26T18:58:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.689+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/482158133 wait complete. 2024-03-26T18:58:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.689+0000 7fbe3a2d7700 1 Processor -- start 2024-03-26T18:58:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.689+0000 7fbe3a2d7700 1 -- start start 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe3a2d7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe34100170 0x7fbe34195a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe3a2d7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe34101370 0x7fbe34195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe3a2d7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe34196570 con 0x7fbe34100170 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe3a2d7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe341966b0 con 0x7fbe34101370 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe33fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe34100170 0x7fbe34195a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe33fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe34100170 0x7fbe34195a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53544/0 (socket says 172.21.15.102:53544) 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe33fff700 1 -- 172.21.15.102:0/1313296838 learned_addr learned my addr 172.21.15.102:0/1313296838 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe337fe700 1 --2- 172.21.15.102:0/1313296838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe34101370 0x7fbe34195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe33fff700 1 -- 172.21.15.102:0/1313296838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe34101370 msgr2=0x7fbe34195f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe33fff700 1 --2- 172.21.15.102:0/1313296838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe34101370 0x7fbe34195f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe33fff700 1 -- 172.21.15.102:0/1313296838 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe240095f0 con 0x7fbe34100170 2024-03-26T18:58:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe33fff700 1 --2- 172.21.15.102:0/1313296838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe34100170 0x7fbe34195a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400bda0 tx=0x7fbe2400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe317fa700 1 -- 172.21.15.102:0/1313296838 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe2401d070 con 0x7fbe34100170 2024-03-26T18:58:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe317fa700 1 -- 172.21.15.102:0/1313296838 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbe24022470 con 0x7fbe34100170 2024-03-26T18:58:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/1313296838 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3419b100 con 0x7fbe34100170 2024-03-26T18:58:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.690+0000 7fbe317fa700 1 -- 172.21.15.102:0/1313296838 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe2400f650 con 0x7fbe34100170 2024-03-26T18:58:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.691+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/1313296838 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3419b670 con 0x7fbe34100170 2024-03-26T18:58:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.692+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/1313296838 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe34108d40 con 0x7fbe34100170 2024-03-26T18:58:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.692+0000 7fbe317fa700 1 -- 172.21.15.102:0/1313296838 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fbe2400f8c0 con 0x7fbe34100170 2024-03-26T18:58:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.692+0000 7fbe317fa700 1 --2- 172.21.15.102:0/1313296838 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fbe1c06e460 0x7fbe1c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.692+0000 7fbe317fa700 1 -- 172.21.15.102:0/1313296838 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbe2408fab0 con 0x7fbe34100170 2024-03-26T18:58:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.692+0000 7fbe337fe700 1 --2- 172.21.15.102:0/1313296838 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fbe1c06e460 0x7fbe1c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.693+0000 7fbe337fe700 1 --2- 172.21.15.102:0/1313296838 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fbe1c06e460 0x7fbe1c070910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009a10 tx=0x7fbe28009190 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:48.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.697+0000 7fbe317fa700 1 -- 172.21.15.102:0/1313296838 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbe2405c890 con 0x7fbe34100170 2024-03-26T18:58:48.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:48.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:48.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:58:48.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:58:48.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:48.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:58:48.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:48.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:48.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:48.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T18:58:48.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T18:58:48.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:48 smithi190 ceph-mon[65689]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T18:58:48.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.921+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/1313296838 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe34105cc0 con 0x7fbe1c06e460 2024-03-26T18:58:48.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.922+0000 7fbe317fa700 1 -- 172.21.15.102:0/1313296838 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fbe34105cc0 con 0x7fbe1c06e460 2024-03-26T18:58:48.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.924+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/1313296838 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fbe1c06e460 msgr2=0x7fbe1c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:48.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.924+0000 7fbe3a2d7700 1 --2- 172.21.15.102:0/1313296838 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fbe1c06e460 0x7fbe1c070910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009a10 tx=0x7fbe28009190 comp rx=0 tx=0).stop 2024-03-26T18:58:48.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.924+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/1313296838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe34100170 msgr2=0x7fbe34195a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:48.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.924+0000 7fbe3a2d7700 1 --2- 172.21.15.102:0/1313296838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe34100170 0x7fbe34195a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400bda0 tx=0x7fbe2400bdd0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.924+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/1313296838 shutdown_connections 2024-03-26T18:58:48.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.924+0000 7fbe3a2d7700 1 --2- 172.21.15.102:0/1313296838 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fbe1c06e460 0x7fbe1c070910 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.924+0000 7fbe3a2d7700 1 --2- 172.21.15.102:0/1313296838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe34100170 0x7fbe34195a10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.924+0000 7fbe3a2d7700 1 --2- 172.21.15.102:0/1313296838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe34101370 0x7fbe34195f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:48.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.924+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/1313296838 >> 172.21.15.102:0/1313296838 conn(0x7fbe340fb6e0 msgr2=0x7fbe341045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:48.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.925+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/1313296838 shutdown_connections 2024-03-26T18:58:48.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:48.925+0000 7fbe3a2d7700 1 -- 172.21.15.102:0/1313296838 wait complete. 2024-03-26T18:58:49.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.026+0000 7fc736191700 1 -- 172.21.15.102:0/2076265284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc730101350 msgr2=0x7fc7301017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:49.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.026+0000 7fc736191700 1 --2- 172.21.15.102:0/2076265284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc730101350 0x7fc7301017a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc724009910 tx=0x7fc724009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:49.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.026+0000 7fc736191700 1 -- 172.21.15.102:0/2076265284 shutdown_connections 2024-03-26T18:58:49.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.026+0000 7fc736191700 1 --2- 172.21.15.102:0/2076265284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc730101350 0x7fc7301017a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.026+0000 7fc736191700 1 --2- 172.21.15.102:0/2076265284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc730100150 0x7fc730100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.026+0000 7fc736191700 1 -- 172.21.15.102:0/2076265284 >> 172.21.15.102:0/2076265284 conn(0x7fc7300fb6e0 msgr2=0x7fc7300fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:49.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.027+0000 7fc736191700 1 -- 172.21.15.102:0/2076265284 shutdown_connections 2024-03-26T18:58:49.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.027+0000 7fc736191700 1 -- 172.21.15.102:0/2076265284 wait complete. 2024-03-26T18:58:49.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.027+0000 7fc736191700 1 Processor -- start 2024-03-26T18:58:49.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.027+0000 7fc736191700 1 -- start start 2024-03-26T18:58:49.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.027+0000 7fc736191700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc730100150 0x7fc730195a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:49.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.027+0000 7fc736191700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc730101350 0x7fc730195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:49.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.027+0000 7fc736191700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc730196570 con 0x7fc730101350 2024-03-26T18:58:49.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc736191700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7301966b0 con 0x7fc730100150 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc72effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc730101350 0x7fc730195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc72effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc730101350 0x7fc730195f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53558/0 (socket says 172.21.15.102:53558) 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc72effd700 1 -- 172.21.15.102:0/1170598350 learned_addr learned my addr 172.21.15.102:0/1170598350 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc72f7fe700 1 --2- 172.21.15.102:0/1170598350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc730100150 0x7fc730195a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc72effd700 1 -- 172.21.15.102:0/1170598350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc730100150 msgr2=0x7fc730195a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc72effd700 1 --2- 172.21.15.102:0/1170598350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc730100150 0x7fc730195a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc72effd700 1 -- 172.21.15.102:0/1170598350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7240095f0 con 0x7fc730101350 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc72effd700 1 --2- 172.21.15.102:0/1170598350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc730101350 0x7fc730195f50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc724006010 tx=0x7fc72400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc72cff9700 1 -- 172.21.15.102:0/1170598350 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc72401d070 con 0x7fc730101350 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc72cff9700 1 -- 172.21.15.102:0/1170598350 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc72400f460 con 0x7fc730101350 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc736191700 1 -- 172.21.15.102:0/1170598350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73019b100 con 0x7fc730101350 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc72cff9700 1 -- 172.21.15.102:0/1170598350 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc724021640 con 0x7fc730101350 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.028+0000 7fc736191700 1 -- 172.21.15.102:0/1170598350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73019b670 con 0x7fc730101350 2024-03-26T18:58:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.029+0000 7fc736191700 1 -- 172.21.15.102:0/1170598350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc730108d40 con 0x7fc730101350 2024-03-26T18:58:49.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.030+0000 7fc72cff9700 1 -- 172.21.15.102:0/1170598350 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc724004b20 con 0x7fc730101350 2024-03-26T18:58:49.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.030+0000 7fc72cff9700 1 --2- 172.21.15.102:0/1170598350 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc71806e460 0x7fc718070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:49.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.030+0000 7fc72f7fe700 1 --2- 172.21.15.102:0/1170598350 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc71806e460 0x7fc718070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:49.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.030+0000 7fc72cff9700 1 -- 172.21.15.102:0/1170598350 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc72408f880 con 0x7fc730101350 2024-03-26T18:58:49.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.031+0000 7fc72f7fe700 1 --2- 172.21.15.102:0/1170598350 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc71806e460 0x7fc718070910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc720006fd0 tx=0x7fc720008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:49.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.035+0000 7fc72cff9700 1 -- 172.21.15.102:0/1170598350 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc72405c5b0 con 0x7fc730101350 2024-03-26T18:58:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.242+0000 7fc736191700 1 -- 172.21.15.102:0/1170598350 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc730105ca0 con 0x7fc71806e460 2024-03-26T18:58:49.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.255+0000 7fc72cff9700 1 -- 172.21.15.102:0/1170598350 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fc730105ca0 con 0x7fc71806e460 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (10m) 6s ago 11m 24.9M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (12m) 6s ago 12m 8144k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (11m) 64s ago 11m 7755k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (73s) 6s ago 12m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (68s) 64s ago 10m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (10m) 6s ago 11m 85.8M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (6m) 6s ago 6m 132M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (6m) 6s ago 6m 15.9M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (6m) 64s ago 6m 15.3M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (6m) 64s ago 6m 120M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (3m) 6s ago 13m 500M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (2m) 64s ago 10m 449M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T18:58:49.249 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (2m) 6s ago 13m 44.4M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T18:58:49.250 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (118s) 64s ago 10m 32.4M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T18:58:49.250 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (11m) 6s ago 11m 17.0M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:58:49.250 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (10m) 64s ago 10m 15.5M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:58:49.250 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (47s) 6s ago 10m 85.4M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T18:58:49.250 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (11s) 6s ago 9m 15.1M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T18:58:49.250 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (9m) 6s ago 9m 193M 1216M 18.2.0 dc2bc1663786 38676c8c0459 2024-03-26T18:58:49.250 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (8m) 64s ago 8m 272M 2240M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:58:49.250 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (8m) 64s ago 8m 251M 2240M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:58:49.250 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (7m) 64s ago 7m 225M 2240M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:58:49.250 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (2m) 6s ago 11m 74.2M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T18:58:49.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.258+0000 7fc736191700 1 -- 172.21.15.102:0/1170598350 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc71806e460 msgr2=0x7fc718070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:49.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.258+0000 7fc736191700 1 --2- 172.21.15.102:0/1170598350 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc71806e460 0x7fc718070910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc720006fd0 tx=0x7fc720008040 comp rx=0 tx=0).stop 2024-03-26T18:58:49.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.258+0000 7fc736191700 1 -- 172.21.15.102:0/1170598350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc730101350 msgr2=0x7fc730195f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.258+0000 7fc736191700 1 --2- 172.21.15.102:0/1170598350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc730101350 0x7fc730195f50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc724006010 tx=0x7fc72400be20 comp rx=0 tx=0).stop 2024-03-26T18:58:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.258+0000 7fc736191700 1 -- 172.21.15.102:0/1170598350 shutdown_connections 2024-03-26T18:58:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.258+0000 7fc736191700 1 --2- 172.21.15.102:0/1170598350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc730100150 0x7fc730195a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.258+0000 7fc736191700 1 --2- 172.21.15.102:0/1170598350 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc71806e460 0x7fc718070910 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.258+0000 7fc736191700 1 --2- 172.21.15.102:0/1170598350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc730101350 0x7fc730195f50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.258+0000 7fc736191700 1 -- 172.21.15.102:0/1170598350 >> 172.21.15.102:0/1170598350 conn(0x7fc7300fb6e0 msgr2=0x7fc730104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.258+0000 7fc736191700 1 -- 172.21.15.102:0/1170598350 shutdown_connections 2024-03-26T18:58:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.258+0000 7fc736191700 1 -- 172.21.15.102:0/1170598350 wait complete. 2024-03-26T18:58:49.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.360+0000 7f4315306700 1 -- 172.21.15.102:0/2000493451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4310100270 msgr2=0x7f43101006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:49.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.360+0000 7f4315306700 1 --2- 172.21.15.102:0/2000493451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4310100270 0x7f43101006c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4304009910 tx=0x7f4304009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.361+0000 7f4315306700 1 -- 172.21.15.102:0/2000493451 shutdown_connections 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.361+0000 7f4315306700 1 --2- 172.21.15.102:0/2000493451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4310100270 0x7f43101006c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.361+0000 7f4315306700 1 --2- 172.21.15.102:0/2000493451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43100ff070 0x7f43100ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.361+0000 7f4315306700 1 -- 172.21.15.102:0/2000493451 >> 172.21.15.102:0/2000493451 conn(0x7f43100fa600 msgr2=0x7f43100fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.361+0000 7f4315306700 1 -- 172.21.15.102:0/2000493451 shutdown_connections 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.361+0000 7f4315306700 1 -- 172.21.15.102:0/2000493451 wait complete. 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.361+0000 7f4315306700 1 Processor -- start 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.361+0000 7f4315306700 1 -- start start 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f4315306700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43100ff070 0x7f43100698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f4315306700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4310100270 0x7f431019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f4315306700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431019ac90 con 0x7f43100ff070 2024-03-26T18:58:49.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f4315306700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431019ae00 con 0x7f4310100270 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f430effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43100ff070 0x7f43100698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f430effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43100ff070 0x7f43100698d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53562/0 (socket says 172.21.15.102:53562) 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f430effd700 1 -- 172.21.15.102:0/649982446 learned_addr learned my addr 172.21.15.102:0/649982446 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f430e7fc700 1 --2- 172.21.15.102:0/649982446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4310100270 0x7f431019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f430effd700 1 -- 172.21.15.102:0/649982446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4310100270 msgr2=0x7f431019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f430effd700 1 --2- 172.21.15.102:0/649982446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4310100270 0x7f431019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f430effd700 1 -- 172.21.15.102:0/649982446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43040095f0 con 0x7f43100ff070 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f430effd700 1 --2- 172.21.15.102:0/649982446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43100ff070 0x7f43100698d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f430000d900 tx=0x7f430000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f42f7fff700 1 -- 172.21.15.102:0/649982446 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43000041f0 con 0x7f43100ff070 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f4315306700 1 -- 172.21.15.102:0/649982446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431019b0e0 con 0x7f43100ff070 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f42f7fff700 1 -- 172.21.15.102:0/649982446 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4300004350 con 0x7f43100ff070 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f42f7fff700 1 -- 172.21.15.102:0/649982446 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4300003da0 con 0x7f43100ff070 2024-03-26T18:58:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.362+0000 7f4315306700 1 -- 172.21.15.102:0/649982446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431019b630 con 0x7f43100ff070 2024-03-26T18:58:49.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.363+0000 7f4315306700 1 -- 172.21.15.102:0/649982446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4310107bc0 con 0x7f43100ff070 2024-03-26T18:58:49.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.364+0000 7f42f7fff700 1 -- 172.21.15.102:0/649982446 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4300009730 con 0x7f43100ff070 2024-03-26T18:58:49.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.364+0000 7f42f7fff700 1 --2- 172.21.15.102:0/649982446 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f42f806e460 0x7f42f8070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:49.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.364+0000 7f42f7fff700 1 -- 172.21.15.102:0/649982446 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f430008e340 con 0x7f43100ff070 2024-03-26T18:58:49.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.364+0000 7f430e7fc700 1 --2- 172.21.15.102:0/649982446 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f42f806e460 0x7f42f8070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:49.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.365+0000 7f430e7fc700 1 --2- 172.21.15.102:0/649982446 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f42f806e460 0x7f42f8070910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4310069e30 tx=0x7f430400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:49.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.369+0000 7f42f7fff700 1 -- 172.21.15.102:0/649982446 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f430005b070 con 0x7f43100ff070 2024-03-26T18:58:49.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.657+0000 7f4315306700 1 -- 172.21.15.102:0/649982446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f431019b270 con 0x7f43100ff070 2024-03-26T18:58:49.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.657+0000 7f42f7fff700 1 -- 172.21.15.102:0/649982446 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f431019b270 con 0x7f43100ff070 2024-03-26T18:58:49.651 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:58:49.651 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:58:49.651 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:58:49.651 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:58:49.651 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:58:49.651 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:58:49.651 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:58:49.651 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:58:49.652 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4, 2024-03-26T18:58:49.652 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:58:49.652 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:58:49.652 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:58:49.652 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:58:49.652 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:58:49.652 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:58:49.652 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 8, 2024-03-26T18:58:49.652 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 6 2024-03-26T18:58:49.652 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:58:49.652 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:58:49.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.659+0000 7f4315306700 1 -- 172.21.15.102:0/649982446 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f42f806e460 msgr2=0x7f42f8070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:49.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.659+0000 7f4315306700 1 --2- 172.21.15.102:0/649982446 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f42f806e460 0x7f42f8070910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4310069e30 tx=0x7f430400b540 comp rx=0 tx=0).stop 2024-03-26T18:58:49.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.659+0000 7f4315306700 1 -- 172.21.15.102:0/649982446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43100ff070 msgr2=0x7f43100698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:49.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.659+0000 7f4315306700 1 --2- 172.21.15.102:0/649982446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43100ff070 0x7f43100698d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f430000d900 tx=0x7f430000dcc0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.660+0000 7f4315306700 1 -- 172.21.15.102:0/649982446 shutdown_connections 2024-03-26T18:58:49.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.660+0000 7f4315306700 1 --2- 172.21.15.102:0/649982446 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f42f806e460 0x7f42f8070910 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.660+0000 7f4315306700 1 --2- 172.21.15.102:0/649982446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43100ff070 0x7f43100698d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.660+0000 7f4315306700 1 --2- 172.21.15.102:0/649982446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4310100270 0x7f431019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.660+0000 7f4315306700 1 -- 172.21.15.102:0/649982446 >> 172.21.15.102:0/649982446 conn(0x7f43100fa600 msgr2=0x7f43101034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:49.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.660+0000 7f4315306700 1 -- 172.21.15.102:0/649982446 shutdown_connections 2024-03-26T18:58:49.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.660+0000 7f4315306700 1 -- 172.21.15.102:0/649982446 wait complete. 2024-03-26T18:58:49.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.763+0000 7f66673b2700 1 -- 172.21.15.102:0/1547039960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660101360 msgr2=0x7f66601017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:49.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.763+0000 7f66673b2700 1 --2- 172.21.15.102:0/1547039960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660101360 0x7f66601017b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6650009910 tx=0x7f6650009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:49.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.763+0000 7f66673b2700 1 -- 172.21.15.102:0/1547039960 shutdown_connections 2024-03-26T18:58:49.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.763+0000 7f66673b2700 1 --2- 172.21.15.102:0/1547039960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660101360 0x7f66601017b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.763+0000 7f66673b2700 1 --2- 172.21.15.102:0/1547039960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6660100160 0x7f6660100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.763+0000 7f66673b2700 1 -- 172.21.15.102:0/1547039960 >> 172.21.15.102:0/1547039960 conn(0x7f66600fb6f0 msgr2=0x7f66600fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:49.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:49 smithi102 ceph-mon[78322]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:49.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:49 smithi102 ceph-mon[78322]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:49.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:49 smithi102 ceph-mon[78322]: pgmap v79: 65 pgs: 7 active+undersized+degraded, 5 active+undersized, 53 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 24/354 objects degraded (6.780%) 2024-03-26T18:58:49.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:49 smithi102 ceph-mon[78322]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.763+0000 7f66673b2700 1 -- 172.21.15.102:0/1547039960 shutdown_connections 2024-03-26T18:58:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.763+0000 7f66673b2700 1 -- 172.21.15.102:0/1547039960 wait complete. 2024-03-26T18:58:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f66673b2700 1 Processor -- start 2024-03-26T18:58:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f66673b2700 1 -- start start 2024-03-26T18:58:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f66673b2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660100160 0x7f6660195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f66673b2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6660101360 0x7f6660195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f66673b2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6660196430 con 0x7f6660100160 2024-03-26T18:58:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f66673b2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6660196570 con 0x7f6660101360 2024-03-26T18:58:49.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f666514e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660100160 0x7f6660195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f666514e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660100160 0x7f6660195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53578/0 (socket says 172.21.15.102:53578) 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f666514e700 1 -- 172.21.15.102:0/422391684 learned_addr learned my addr 172.21.15.102:0/422391684 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f666494d700 1 --2- 172.21.15.102:0/422391684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6660101360 0x7f6660195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f666514e700 1 -- 172.21.15.102:0/422391684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6660101360 msgr2=0x7f6660195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f666514e700 1 --2- 172.21.15.102:0/422391684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6660101360 0x7f6660195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f666514e700 1 -- 172.21.15.102:0/422391684 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66500095f0 con 0x7f6660100160 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.764+0000 7f666514e700 1 --2- 172.21.15.102:0/422391684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660100160 0x7f6660195960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f665800cfd0 tx=0x7f665800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.765+0000 7f66567fc700 1 -- 172.21.15.102:0/422391684 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f665800ad70 con 0x7f6660100160 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.765+0000 7f66673b2700 1 -- 172.21.15.102:0/422391684 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f666019afd0 con 0x7f6660100160 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.765+0000 7f66567fc700 1 -- 172.21.15.102:0/422391684 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6658004020 con 0x7f6660100160 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.765+0000 7f66567fc700 1 -- 172.21.15.102:0/422391684 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6658012650 con 0x7f6660100160 2024-03-26T18:58:49.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.765+0000 7f66673b2700 1 -- 172.21.15.102:0/422391684 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f666019b4f0 con 0x7f6660100160 2024-03-26T18:58:49.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.766+0000 7f66673b2700 1 -- 172.21.15.102:0/422391684 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6660108cb0 con 0x7f6660100160 2024-03-26T18:58:49.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.766+0000 7f66567fc700 1 -- 172.21.15.102:0/422391684 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6658004190 con 0x7f6660100160 2024-03-26T18:58:49.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.767+0000 7f66567fc700 1 --2- 172.21.15.102:0/422391684 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f664c06e460 0x7f664c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:49.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.767+0000 7f66567fc700 1 -- 172.21.15.102:0/422391684 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f665808e610 con 0x7f6660100160 2024-03-26T18:58:49.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.767+0000 7f666494d700 1 --2- 172.21.15.102:0/422391684 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f664c06e460 0x7f664c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:49.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.767+0000 7f666494d700 1 --2- 172.21.15.102:0/422391684 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f664c06e460 0x7f664c070910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f665000b5c0 tx=0x7f6650005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:49.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:49.772+0000 7f66567fc700 1 -- 172.21.15.102:0/422391684 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f665805b340 con 0x7f6660100160 2024-03-26T18:58:49.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:49 smithi190 ceph-mon[65689]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:50.019 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:49 smithi190 ceph-mon[65689]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:50.020 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:49 smithi190 ceph-mon[65689]: pgmap v79: 65 pgs: 7 active+undersized+degraded, 5 active+undersized, 53 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 24/354 objects degraded (6.780%) 2024-03-26T18:58:50.020 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:49 smithi190 ceph-mon[65689]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:50.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.017+0000 7f66673b2700 1 -- 172.21.15.102:0/422391684 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f666019bbb0 con 0x7f6660100160 2024-03-26T18:58:50.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.017+0000 7f66567fc700 1 -- 172.21.15.102:0/422391684 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1905 (secure 0 0 0) 0x7f665805aed0 con 0x7f6660100160 2024-03-26T18:58:50.021 INFO:teuthology.orchestra.run.smithi102.stdout:e21 2024-03-26T18:58:50.021 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:58:50.021 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:58:50.021 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:58:50.021 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:58:50.021 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:58:50.021 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:58:50.021 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 21 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:58:42.198340+0000 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:58:50.022 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 14490 members: 24283,14490 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:58:50.023 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:58:50.024 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:58:50.024 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:58:50.024 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:58:50.024 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:58:50.024 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:58:50.024 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:58:50.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.019+0000 7f66673b2700 1 -- 172.21.15.102:0/422391684 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f664c06e460 msgr2=0x7f664c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:50.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.019+0000 7f66673b2700 1 --2- 172.21.15.102:0/422391684 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f664c06e460 0x7f664c070910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f665000b5c0 tx=0x7f6650005960 comp rx=0 tx=0).stop 2024-03-26T18:58:50.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.019+0000 7f66673b2700 1 -- 172.21.15.102:0/422391684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660100160 msgr2=0x7f6660195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:50.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.019+0000 7f66673b2700 1 --2- 172.21.15.102:0/422391684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660100160 0x7f6660195960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f665800cfd0 tx=0x7f665800a340 comp rx=0 tx=0).stop 2024-03-26T18:58:50.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.019+0000 7f66673b2700 1 -- 172.21.15.102:0/422391684 shutdown_connections 2024-03-26T18:58:50.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.019+0000 7f66673b2700 1 --2- 172.21.15.102:0/422391684 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f664c06e460 0x7f664c070910 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.019+0000 7f66673b2700 1 --2- 172.21.15.102:0/422391684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660100160 0x7f6660195960 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.019+0000 7f66673b2700 1 --2- 172.21.15.102:0/422391684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6660101360 0x7f6660195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.019+0000 7f66673b2700 1 -- 172.21.15.102:0/422391684 >> 172.21.15.102:0/422391684 conn(0x7f66600fb6f0 msgr2=0x7f6660104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:50.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.019+0000 7f66673b2700 1 -- 172.21.15.102:0/422391684 shutdown_connections 2024-03-26T18:58:50.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.019+0000 7f66673b2700 1 -- 172.21.15.102:0/422391684 wait complete. 2024-03-26T18:58:50.026 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 21 2024-03-26T18:58:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.125+0000 7f994b662700 1 -- 172.21.15.102:0/1030386561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9944100170 msgr2=0x7f9944100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.125+0000 7f994b662700 1 --2- 172.21.15.102:0/1030386561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9944100170 0x7f9944100580 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9940009910 tx=0x7f9940009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.125+0000 7f994b662700 1 -- 172.21.15.102:0/1030386561 shutdown_connections 2024-03-26T18:58:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.125+0000 7f994b662700 1 --2- 172.21.15.102:0/1030386561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9944101370 0x7f99441017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.125+0000 7f994b662700 1 --2- 172.21.15.102:0/1030386561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9944100170 0x7f9944100580 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.125+0000 7f994b662700 1 -- 172.21.15.102:0/1030386561 >> 172.21.15.102:0/1030386561 conn(0x7f99440fb6e0 msgr2=0x7f99440fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.125+0000 7f994b662700 1 -- 172.21.15.102:0/1030386561 shutdown_connections 2024-03-26T18:58:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.125+0000 7f994b662700 1 -- 172.21.15.102:0/1030386561 wait complete. 2024-03-26T18:58:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f994b662700 1 Processor -- start 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f994b662700 1 -- start start 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f994b662700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9944101370 0x7f9944074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f994b662700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9944073140 0x7f99440735b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f994b662700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9944073af0 con 0x7f9944101370 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f994b662700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9944073c60 con 0x7f9944073140 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f99493fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9944101370 0x7f9944074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f99493fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9944101370 0x7f9944074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53592/0 (socket says 172.21.15.102:53592) 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f99493fe700 1 -- 172.21.15.102:0/2454619546 learned_addr learned my addr 172.21.15.102:0/2454619546 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f99493fe700 1 -- 172.21.15.102:0/2454619546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9944073140 msgr2=0x7f99440735b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f99493fe700 1 --2- 172.21.15.102:0/2454619546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9944073140 0x7f99440735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f99493fe700 1 -- 172.21.15.102:0/2454619546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99400095f0 con 0x7f9944101370 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.126+0000 7f99493fe700 1 --2- 172.21.15.102:0/2454619546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9944101370 0x7f9944074af0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f99441011d0 tx=0x7f994000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.127+0000 7f993a7fc700 1 -- 172.21.15.102:0/2454619546 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f994001d070 con 0x7f9944101370 2024-03-26T18:58:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.127+0000 7f993a7fc700 1 -- 172.21.15.102:0/2454619546 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9940022470 con 0x7f9944101370 2024-03-26T18:58:50.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.127+0000 7f994b662700 1 -- 172.21.15.102:0/2454619546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9944073ee0 con 0x7f9944101370 2024-03-26T18:58:50.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.127+0000 7f993a7fc700 1 -- 172.21.15.102:0/2454619546 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f994000f670 con 0x7f9944101370 2024-03-26T18:58:50.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.127+0000 7f994b662700 1 -- 172.21.15.102:0/2454619546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994419de60 con 0x7f9944101370 2024-03-26T18:58:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.128+0000 7f994b662700 1 -- 172.21.15.102:0/2454619546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9944108cc0 con 0x7f9944101370 2024-03-26T18:58:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.128+0000 7f993a7fc700 1 -- 172.21.15.102:0/2454619546 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9940004af0 con 0x7f9944101370 2024-03-26T18:58:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.128+0000 7f993a7fc700 1 --2- 172.21.15.102:0/2454619546 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f993006e460 0x7f9930070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.129+0000 7f993a7fc700 1 -- 172.21.15.102:0/2454619546 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f994008fa90 con 0x7f9944101370 2024-03-26T18:58:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.129+0000 7f9948bfd700 1 --2- 172.21.15.102:0/2454619546 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f993006e460 0x7f9930070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.129+0000 7f9948bfd700 1 --2- 172.21.15.102:0/2454619546 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f993006e460 0x7f9930070910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f993400a990 tx=0x7f9934005c80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:50.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.133+0000 7f993a7fc700 1 -- 172.21.15.102:0/2454619546 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f994005c990 con 0x7f9944101370 2024-03-26T18:58:50.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.350+0000 7f994b662700 1 -- 172.21.15.102:0/2454619546 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9944105cc0 con 0x7f993006e460 2024-03-26T18:58:50.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.351+0000 7f993a7fc700 1 -- 172.21.15.102:0/2454619546 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9944105cc0 con 0x7f993006e460 2024-03-26T18:58:50.345 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:58:50.345 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:58:50.345 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:58:50.345 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:58:50.345 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T18:58:50.345 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T18:58:50.345 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-03-26T18:58:50.345 INFO:teuthology.orchestra.run.smithi102.stdout: "mon" 2024-03-26T18:58:50.345 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T18:58:50.346 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "8/23 daemons upgraded", 2024-03-26T18:58:50.346 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T18:58:50.346 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:58:50.346 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:58:50.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.354+0000 7f994b662700 1 -- 172.21.15.102:0/2454619546 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f993006e460 msgr2=0x7f9930070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:50.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.354+0000 7f994b662700 1 --2- 172.21.15.102:0/2454619546 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f993006e460 0x7f9930070910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f993400a990 tx=0x7f9934005c80 comp rx=0 tx=0).stop 2024-03-26T18:58:50.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.354+0000 7f994b662700 1 -- 172.21.15.102:0/2454619546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9944101370 msgr2=0x7f9944074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:50.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.354+0000 7f994b662700 1 --2- 172.21.15.102:0/2454619546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9944101370 0x7f9944074af0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f99441011d0 tx=0x7f994000be20 comp rx=0 tx=0).stop 2024-03-26T18:58:50.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.354+0000 7f994b662700 1 -- 172.21.15.102:0/2454619546 shutdown_connections 2024-03-26T18:58:50.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.354+0000 7f994b662700 1 --2- 172.21.15.102:0/2454619546 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f993006e460 0x7f9930070910 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.354+0000 7f994b662700 1 --2- 172.21.15.102:0/2454619546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9944101370 0x7f9944074af0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.354+0000 7f994b662700 1 --2- 172.21.15.102:0/2454619546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9944073140 0x7f99440735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.354+0000 7f994b662700 1 -- 172.21.15.102:0/2454619546 >> 172.21.15.102:0/2454619546 conn(0x7f99440fb6e0 msgr2=0x7f99441045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:50.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.354+0000 7f994b662700 1 -- 172.21.15.102:0/2454619546 shutdown_connections 2024-03-26T18:58:50.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.354+0000 7f994b662700 1 -- 172.21.15.102:0/2454619546 wait complete. 2024-03-26T18:58:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.457+0000 7fe77da80700 1 -- 172.21.15.102:0/234502722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7780ff050 msgr2=0x7fe7780ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.457+0000 7fe77da80700 1 --2- 172.21.15.102:0/234502722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7780ff050 0x7fe7780ff460 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe770009910 tx=0x7fe770009c20 comp rx=0 tx=0).stop 2024-03-26T18:58:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.457+0000 7fe77da80700 1 -- 172.21.15.102:0/234502722 shutdown_connections 2024-03-26T18:58:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.457+0000 7fe77da80700 1 --2- 172.21.15.102:0/234502722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe778100250 0x7fe7781006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.457+0000 7fe77da80700 1 --2- 172.21.15.102:0/234502722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7780ff050 0x7fe7780ff460 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.457+0000 7fe77da80700 1 -- 172.21.15.102:0/234502722 >> 172.21.15.102:0/234502722 conn(0x7fe7780fa5e0 msgr2=0x7fe7780fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.457+0000 7fe77da80700 1 -- 172.21.15.102:0/234502722 shutdown_connections 2024-03-26T18:58:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.457+0000 7fe77da80700 1 -- 172.21.15.102:0/234502722 wait complete. 2024-03-26T18:58:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe77da80700 1 Processor -- start 2024-03-26T18:58:50.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe77da80700 1 -- start start 2024-03-26T18:58:50.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe77da80700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7780ff050 0x7fe778068f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:50.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe77da80700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe778100250 0x7fe77819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:50.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe77da80700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77819ac30 con 0x7fe778100250 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe77da80700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77819ad70 con 0x7fe7780ff050 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe776ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe778100250 0x7fe77819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe776ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe778100250 0x7fe77819a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53608/0 (socket says 172.21.15.102:53608) 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe776ffd700 1 -- 172.21.15.102:0/502399079 learned_addr learned my addr 172.21.15.102:0/502399079 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe7777fe700 1 --2- 172.21.15.102:0/502399079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7780ff050 0x7fe778068f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe776ffd700 1 -- 172.21.15.102:0/502399079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7780ff050 msgr2=0x7fe778068f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe776ffd700 1 --2- 172.21.15.102:0/502399079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7780ff050 0x7fe778068f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.458+0000 7fe776ffd700 1 -- 172.21.15.102:0/502399079 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7700095f0 con 0x7fe778100250 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.459+0000 7fe776ffd700 1 --2- 172.21.15.102:0/502399079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe778100250 0x7fe77819a6f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00cfd0 tx=0x7fe76c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.459+0000 7fe774ff9700 1 -- 172.21.15.102:0/502399079 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe76c00ad70 con 0x7fe778100250 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.459+0000 7fe774ff9700 1 -- 172.21.15.102:0/502399079 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe76c0128d0 con 0x7fe778100250 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.459+0000 7fe77da80700 1 -- 172.21.15.102:0/502399079 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77819b050 con 0x7fe778100250 2024-03-26T18:58:50.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.459+0000 7fe774ff9700 1 -- 172.21.15.102:0/502399079 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe76c011880 con 0x7fe778100250 2024-03-26T18:58:50.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.459+0000 7fe77da80700 1 -- 172.21.15.102:0/502399079 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77819b570 con 0x7fe778100250 2024-03-26T18:58:50.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.460+0000 7fe77da80700 1 -- 172.21.15.102:0/502399079 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe778107ba0 con 0x7fe778100250 2024-03-26T18:58:50.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.460+0000 7fe774ff9700 1 -- 172.21.15.102:0/502399079 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe76c012420 con 0x7fe778100250 2024-03-26T18:58:50.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.461+0000 7fe774ff9700 1 --2- 172.21.15.102:0/502399079 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe76006e460 0x7fe760070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:58:50.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.461+0000 7fe774ff9700 1 -- 172.21.15.102:0/502399079 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe76c08e840 con 0x7fe778100250 2024-03-26T18:58:50.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.461+0000 7fe7777fe700 1 --2- 172.21.15.102:0/502399079 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe76006e460 0x7fe760070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:58:50.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.461+0000 7fe7777fe700 1 --2- 172.21.15.102:0/502399079 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe76006e460 0x7fe760070910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe770006010 tx=0x7fe77000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:58:50.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.465+0000 7fe774ff9700 1 -- 172.21.15.102:0/502399079 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe76c05b570 con 0x7fe778100250 2024-03-26T18:58:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:50 smithi102 ceph-mon[78322]: Health check update: Degraded data redundancy: 24/354 objects degraded (6.780%), 7 pgs degraded (PG_DEGRADED) 2024-03-26T18:58:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:50 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/649982446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:50 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/422391684' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:58:50.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.759+0000 7fe77da80700 1 -- 172.21.15.102:0/502399079 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe77819c090 con 0x7fe778100250 2024-03-26T18:58:50.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.760+0000 7fe774ff9700 1 -- 172.21.15.102:0/502399079 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+568 (secure 0 0 0) 0x7fe76c05b100 con 0x7fe778100250 2024-03-26T18:58:50.753 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_WARN Degraded data redundancy: 24/354 objects degraded (6.780%), 7 pgs degraded 2024-03-26T18:58:50.753 INFO:teuthology.orchestra.run.smithi102.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 24/354 objects degraded (6.780%), 7 pgs degraded 2024-03-26T18:58:50.753 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.4 is active+undersized+degraded, acting [0,4] 2024-03-26T18:58:50.753 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-03-26T18:58:50.753 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.9 is active+undersized+degraded, acting [4,0] 2024-03-26T18:58:50.754 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.a is active+undersized+degraded, acting [4,3] 2024-03-26T18:58:50.754 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.d is active+undersized+degraded, acting [3,2] 2024-03-26T18:58:50.754 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.15 is active+undersized+degraded, acting [3,0] 2024-03-26T18:58:50.754 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1b is active+undersized+degraded, acting [0,5] 2024-03-26T18:58:50.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.762+0000 7fe77da80700 1 -- 172.21.15.102:0/502399079 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe76006e460 msgr2=0x7fe760070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:50.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.762+0000 7fe77da80700 1 --2- 172.21.15.102:0/502399079 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe76006e460 0x7fe760070910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe770006010 tx=0x7fe77000b540 comp rx=0 tx=0).stop 2024-03-26T18:58:50.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.762+0000 7fe77da80700 1 -- 172.21.15.102:0/502399079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe778100250 msgr2=0x7fe77819a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:58:50.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.762+0000 7fe77da80700 1 --2- 172.21.15.102:0/502399079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe778100250 0x7fe77819a6f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00cfd0 tx=0x7fe76c00a340 comp rx=0 tx=0).stop 2024-03-26T18:58:50.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.762+0000 7fe77da80700 1 -- 172.21.15.102:0/502399079 shutdown_connections 2024-03-26T18:58:50.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.762+0000 7fe77da80700 1 --2- 172.21.15.102:0/502399079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7780ff050 0x7fe778068f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.762+0000 7fe77da80700 1 --2- 172.21.15.102:0/502399079 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe76006e460 0x7fe760070910 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.762+0000 7fe77da80700 1 --2- 172.21.15.102:0/502399079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe778100250 0x7fe77819a6f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:58:50.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.762+0000 7fe77da80700 1 -- 172.21.15.102:0/502399079 >> 172.21.15.102:0/502399079 conn(0x7fe7780fa5e0 msgr2=0x7fe778103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:58:50.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.762+0000 7fe77da80700 1 -- 172.21.15.102:0/502399079 shutdown_connections 2024-03-26T18:58:50.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:58:50.762+0000 7fe77da80700 1 -- 172.21.15.102:0/502399079 wait complete. 2024-03-26T18:58:50.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:50 smithi190 ceph-mon[65689]: Health check update: Degraded data redundancy: 24/354 objects degraded (6.780%), 7 pgs degraded (PG_DEGRADED) 2024-03-26T18:58:50.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:50 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/649982446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:58:50.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:50 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/422391684' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:58:51.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:51 smithi190 ceph-mon[65689]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:51.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:51 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/502399079' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:58:51.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:51 smithi190 ceph-mon[65689]: pgmap v80: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:51.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:51 smithi102 ceph-mon[78322]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:58:51.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:51 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/502399079' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:58:51.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:51 smithi102 ceph-mon[78322]: pgmap v80: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:52 smithi102 ceph-mon[78322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/354 objects degraded (6.780%), 7 pgs degraded) 2024-03-26T18:58:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:52 smithi102 ceph-mon[78322]: Cluster is now healthy 2024-03-26T18:58:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:58:52.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:52 smithi190 ceph-mon[65689]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/354 objects degraded (6.780%), 7 pgs degraded) 2024-03-26T18:58:52.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:52 smithi190 ceph-mon[65689]: Cluster is now healthy 2024-03-26T18:58:52.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:52.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:58:53.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:53 smithi190 ceph-mon[65689]: pgmap v81: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:53.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:53 smithi102 ceph-mon[78322]: pgmap v81: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:56.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:56 smithi190 ceph-mon[65689]: pgmap v82: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:56.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:56 smithi102 ceph-mon[78322]: pgmap v82: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:58 smithi102 ceph-mon[78322]: pgmap v83: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:58:58 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:58:58.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:58 smithi190 ceph-mon[65689]: pgmap v83: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:58:58.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:58:58 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:00.581 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:00 smithi102 ceph-mon[78322]: pgmap v84: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:00.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:00 smithi190 ceph-mon[65689]: pgmap v84: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:02.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:02 smithi190 ceph-mon[65689]: pgmap v85: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:02.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:02 smithi102 ceph-mon[78322]: pgmap v85: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:03.609 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T18:59:03.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T18:59:04.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:04 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T18:59:04.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:04 smithi102 ceph-mon[78322]: Upgrade: osd.2 is safe to restart 2024-03-26T18:59:04.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:04 smithi102 ceph-mon[78322]: pgmap v86: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:04.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:04.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T18:59:04.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:59:04.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:04 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T18:59:04.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:04 smithi190 ceph-mon[65689]: Upgrade: osd.2 is safe to restart 2024-03-26T18:59:04.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:04 smithi190 ceph-mon[65689]: pgmap v86: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:04.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:04.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T18:59:04.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:59:05.508 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:05 smithi102 ceph-mon[78322]: Upgrade: Updating osd.2 2024-03-26T18:59:05.508 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:05 smithi102 ceph-mon[78322]: Deploying daemon osd.2 on smithi102 2024-03-26T18:59:05.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:05 smithi190 ceph-mon[65689]: Upgrade: Updating osd.2 2024-03-26T18:59:05.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:05 smithi190 ceph-mon[65689]: Deploying daemon osd.2 on smithi102 2024-03-26T18:59:06.510 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:06 smithi102 ceph-mon[78322]: pgmap v87: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:06.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:06 smithi190 ceph-mon[65689]: pgmap v87: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:07.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:07 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:59:07.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:07 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:59:07.876 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:07 smithi102 systemd[1]: Stopping Ceph osd.2 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:59:07.876 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:07 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[54930]: 2024-03-26T18:59:07.753+0000 7f6e4d4cf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T18:59:07.876 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:07 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[54930]: 2024-03-26T18:59:07.753+0000 7f6e4d4cf700 -1 osd.2 60 *** Got signal Terminated *** 2024-03-26T18:59:07.876 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:07 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[54930]: 2024-03-26T18:59:07.753+0000 7f6e4d4cf700 -1 osd.2 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T18:59:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:08 smithi102 ceph-mon[78322]: pgmap v88: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:08 smithi102 ceph-mon[78322]: osd.2 marked itself down and dead 2024-03-26T18:59:08.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:08 smithi190 ceph-mon[65689]: pgmap v88: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:08.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:08 smithi190 ceph-mon[65689]: osd.2 marked itself down and dead 2024-03-26T18:59:09.126 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:08 smithi102 podman[90988]: 2024-03-26 18:59:08.876397808 +0000 UTC m=+1.222332546 container died 38676c8c045953634822c7aeab57bae36e62ef08686ae09625c8344764e9880c (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T18:59:09.480 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:09 smithi102 ceph-mon[78322]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T18:59:09.480 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:09 smithi102 ceph-mon[78322]: osdmap e61: 6 total, 5 up, 6 in 2024-03-26T18:59:09.481 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:09 smithi102 podman[90988]: 2024-03-26 18:59:09.203623577 +0000 UTC m=+1.549558306 container cleanup 38676c8c045953634822c7aeab57bae36e62ef08686ae09625c8344764e9880c (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2) 2024-03-26T18:59:09.481 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:09 smithi102 bash[90988]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2 2024-03-26T18:59:09.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:09 smithi190 ceph-mon[65689]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T18:59:09.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:09 smithi190 ceph-mon[65689]: osdmap e61: 6 total, 5 up, 6 in 2024-03-26T18:59:09.876 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:09 smithi102 podman[91005]: 2024-03-26 18:59:09.487813662 +0000 UTC m=+0.605224317 container remove 38676c8c045953634822c7aeab57bae36e62ef08686ae09625c8344764e9880c (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-03-26T18:59:10.151 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:09 smithi102 podman[91098]: 2024-03-26 18:59:09.800978693 +0000 UTC m=+0.018041485 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:59:10.151 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:09 smithi102 podman[91098]: 2024-03-26 18:59:09.91436729 +0000 UTC m=+0.131430074 container create 527aaee2f39650a474c7f87e96df8cfafa148dab641fcc8bea0fa80f11391b3e (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T18:59:10.151 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:10 smithi102 podman[91098]: 2024-03-26 18:59:10.158921596 +0000 UTC m=+0.375984383 container init 527aaee2f39650a474c7f87e96df8cfafa148dab641fcc8bea0fa80f11391b3e (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-26T18:59:10.446 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:10 smithi102 ceph-mon[78322]: pgmap v90: 65 pgs: 8 peering, 6 stale+active+clean, 51 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:10.447 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:10 smithi102 ceph-mon[78322]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-26T18:59:10.447 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:10 smithi102 ceph-mon[78322]: osdmap e62: 6 total, 5 up, 6 in 2024-03-26T18:59:10.447 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:10 smithi102 podman[91098]: 2024-03-26 18:59:10.162372272 +0000 UTC m=+0.379435066 container start 527aaee2f39650a474c7f87e96df8cfafa148dab641fcc8bea0fa80f11391b3e (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240320.145742-3605160, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-26T18:59:10.447 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:10 smithi102 podman[91098]: 2024-03-26 18:59:10.242182874 +0000 UTC m=+0.459245658 container attach 527aaee2f39650a474c7f87e96df8cfafa148dab641fcc8bea0fa80f11391b3e (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T18:59:10.746 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:10 smithi102 podman[91098]: 2024-03-26 18:59:10.454506598 +0000 UTC m=+0.671569374 container died 527aaee2f39650a474c7f87e96df8cfafa148dab641fcc8bea0fa80f11391b3e (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T18:59:10.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:10 smithi190 ceph-mon[65689]: pgmap v90: 65 pgs: 8 peering, 6 stale+active+clean, 51 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:10.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:10 smithi190 ceph-mon[65689]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-26T18:59:10.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:10 smithi190 ceph-mon[65689]: osdmap e62: 6 total, 5 up, 6 in 2024-03-26T18:59:11.376 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:11 smithi102 podman[91124]: 2024-03-26 18:59:11.0210679 +0000 UTC m=+0.557071224 container remove 527aaee2f39650a474c7f87e96df8cfafa148dab641fcc8bea0fa80f11391b3e (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240320.145742-3605160, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-26T18:59:11.376 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:11 smithi102 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-26T18:59:11.376 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:11 smithi102 systemd[1]: Stopped Ceph osd.2 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:59:11.376 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:11 smithi102 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.2.service: Consumed 49.652s CPU time. 2024-03-26T18:59:11.376 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:11 smithi102 systemd[1]: Starting Ceph osd.2 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:59:11.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:11 smithi102 ceph-mon[78322]: pgmap v92: 65 pgs: 3 active+undersized+degraded, 8 peering, 2 stale+active+clean, 4 active+undersized, 48 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 13/354 objects degraded (3.672%) 2024-03-26T18:59:11.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:11 smithi102 ceph-mon[78322]: Health check failed: Degraded data redundancy: 13/354 objects degraded (3.672%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T18:59:11.626 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:11 smithi102 podman[91228]: 2024-03-26 18:59:11.368670601 +0000 UTC m=+0.018654012 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:59:11.626 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:11 smithi102 podman[91228]: 2024-03-26 18:59:11.489100232 +0000 UTC m=+0.139083673 container create f64a9108bbe12050b6a39ead3d19824321247a5ae765023e4a5735c9e8141a05 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240320.145742-3605160, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T18:59:11.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:11 smithi190 ceph-mon[65689]: pgmap v92: 65 pgs: 3 active+undersized+degraded, 8 peering, 2 stale+active+clean, 4 active+undersized, 48 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 13/354 objects degraded (3.672%) 2024-03-26T18:59:11.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:11 smithi190 ceph-mon[65689]: Health check failed: Degraded data redundancy: 13/354 objects degraded (3.672%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T18:59:12.126 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:11 smithi102 podman[91228]: 2024-03-26 18:59:11.734605111 +0000 UTC m=+0.384588524 container init f64a9108bbe12050b6a39ead3d19824321247a5ae765023e4a5735c9e8141a05 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T18:59:12.126 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:11 smithi102 podman[91228]: 2024-03-26 18:59:11.737929385 +0000 UTC m=+0.387912788 container start f64a9108bbe12050b6a39ead3d19824321247a5ae765023e4a5735c9e8141a05 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-26T18:59:12.126 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:11 smithi102 podman[91228]: 2024-03-26 18:59:11.81719242 +0000 UTC m=+0.467175819 container attach f64a9108bbe12050b6a39ead3d19824321247a5ae765023e4a5735c9e8141a05 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-26T18:59:12.485 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate[91243]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T18:59:12.485 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate[91243]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T18:59:12.485 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 bash[91228]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T18:59:12.485 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 bash[91228]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T18:59:12.485 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 bash[91228]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T18:59:12.486 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 bash[91228]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T18:59:12.486 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 bash[91228]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-26T18:59:12.486 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 bash[91228]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T18:59:12.486 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 bash[91228]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-26T18:59:12.486 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate[91243]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T18:59:12.486 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate[91243]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T18:59:12.486 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate[91243]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-26T18:59:12.486 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate[91243]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T18:59:12.486 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate[91243]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-26T18:59:12.486 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 podman[91228]: 2024-03-26 18:59:12.200115722 +0000 UTC m=+0.850099122 container died f64a9108bbe12050b6a39ead3d19824321247a5ae765023e4a5735c9e8141a05 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T18:59:13.064 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:12 smithi102 podman[91368]: 2024-03-26 18:59:12.807999582 +0000 UTC m=+0.603001192 container remove f64a9108bbe12050b6a39ead3d19824321247a5ae765023e4a5735c9e8141a05 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T18:59:13.064 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:13 smithi102 podman[91433]: 2024-03-26 18:59:12.923803272 +0000 UTC m=+0.018059783 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:59:13.065 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:13 smithi102 podman[91433]: 2024-03-26 18:59:13.071974395 +0000 UTC m=+0.166230905 container create 60206f27cb55553818b5f86f0d61a42786e62eeb3ce3e59bff3bb021cdc4fa2b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-26T18:59:13.342 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:13 smithi102 podman[91433]: 2024-03-26 18:59:13.249823741 +0000 UTC m=+0.344080251 container init 60206f27cb55553818b5f86f0d61a42786e62eeb3ce3e59bff3bb021cdc4fa2b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T18:59:13.343 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:13 smithi102 podman[91433]: 2024-03-26 18:59:13.252949627 +0000 UTC m=+0.347206141 container start 60206f27cb55553818b5f86f0d61a42786e62eeb3ce3e59bff3bb021cdc4fa2b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-26T18:59:13.343 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:13 smithi102 ceph-osd[91452]: -- 172.21.15.102:0/3624135929 <== mon.0 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55722bd13680 con 0x55722bcdcc00 2024-03-26T18:59:13.626 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:13 smithi102 bash[91433]: 60206f27cb55553818b5f86f0d61a42786e62eeb3ce3e59bff3bb021cdc4fa2b 2024-03-26T18:59:13.626 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:13 smithi102 systemd[1]: Started Ceph osd.2 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:59:14.126 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:13 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[91448]: 2024-03-26T18:59:13.797+0000 7fbfccf6f7c0 -1 Falling back to public interface 2024-03-26T18:59:14.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:14 smithi190 ceph-mon[65689]: pgmap v93: 65 pgs: 9 active+undersized+degraded, 8 peering, 10 active+undersized, 38 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 30/354 objects degraded (8.475%) 2024-03-26T18:59:14.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:14.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:14.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:59:14.612 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:14 smithi102 ceph-mon[78322]: pgmap v93: 65 pgs: 9 active+undersized+degraded, 8 peering, 10 active+undersized, 38 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 30/354 objects degraded (8.475%) 2024-03-26T18:59:14.612 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:14.612 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:14.612 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:59:15.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:15 smithi102 ceph-mon[78322]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-03-26T18:59:15.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:15 smithi190 ceph-mon[65689]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-03-26T18:59:16.606 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:16 smithi102 ceph-mon[78322]: pgmap v94: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 42/354 objects degraded (11.864%) 2024-03-26T18:59:16.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:16 smithi190 ceph-mon[65689]: pgmap v94: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 42/354 objects degraded (11.864%) 2024-03-26T18:59:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:17 smithi102 ceph-mon[78322]: pgmap v95: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 42/354 objects degraded (11.864%) 2024-03-26T18:59:17.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:17 smithi190 ceph-mon[65689]: pgmap v95: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 42/354 objects degraded (11.864%) 2024-03-26T18:59:18.037 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:17 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[91448]: 2024-03-26T18:59:17.679+0000 7fbfccf6f7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-26T18:59:18.376 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:18 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[91448]: 2024-03-26T18:59:18.084+0000 7fbfccf6f7c0 -1 osd.2 60 log_to_monitors true 2024-03-26T18:59:18.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:18 smithi190 ceph-mon[65689]: from='osd.2 [v2:172.21.15.102:6818/618947256,v1:172.21.15.102:6819/618947256]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T18:59:18.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:18 smithi102 ceph-mon[78322]: from='osd.2 [v2:172.21.15.102:6818/618947256,v1:172.21.15.102:6819/618947256]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T18:59:18.876 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 18:59:18 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[91448]: 2024-03-26T18:59:18.416+0000 7fbfbe3c7700 -1 osd.2 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T18:59:19.777 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:19 smithi102 ceph-mon[78322]: from='osd.2 [v2:172.21.15.102:6818/618947256,v1:172.21.15.102:6819/618947256]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T18:59:19.777 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:19 smithi102 ceph-mon[78322]: osdmap e63: 6 total, 5 up, 6 in 2024-03-26T18:59:19.777 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:19 smithi102 ceph-mon[78322]: from='osd.2 [v2:172.21.15.102:6818/618947256,v1:172.21.15.102:6819/618947256]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:59:19.778 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:19 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:19.778 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:19 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:19.778 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:19 smithi102 ceph-mon[78322]: Health check update: Degraded data redundancy: 42/354 objects degraded (11.864%), 13 pgs degraded (PG_DEGRADED) 2024-03-26T18:59:19.778 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:19 smithi102 ceph-mon[78322]: pgmap v97: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 42/354 objects degraded (11.864%) 2024-03-26T18:59:19.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:19 smithi190 ceph-mon[65689]: from='osd.2 [v2:172.21.15.102:6818/618947256,v1:172.21.15.102:6819/618947256]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T18:59:19.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:19 smithi190 ceph-mon[65689]: osdmap e63: 6 total, 5 up, 6 in 2024-03-26T18:59:19.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:19 smithi190 ceph-mon[65689]: from='osd.2 [v2:172.21.15.102:6818/618947256,v1:172.21.15.102:6819/618947256]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-26T18:59:19.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:19 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:19.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:19 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:19.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:19 smithi190 ceph-mon[65689]: Health check update: Degraded data redundancy: 42/354 objects degraded (11.864%), 13 pgs degraded (PG_DEGRADED) 2024-03-26T18:59:19.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:19 smithi190 ceph-mon[65689]: pgmap v97: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 42/354 objects degraded (11.864%) 2024-03-26T18:59:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:20 smithi102 ceph-mon[78322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T18:59:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:20 smithi102 ceph-mon[78322]: osd.2 [v2:172.21.15.102:6818/618947256,v1:172.21.15.102:6819/618947256] boot 2024-03-26T18:59:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:20 smithi102 ceph-mon[78322]: osdmap e64: 6 total, 6 up, 6 in 2024-03-26T18:59:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:20 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:59:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:20 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:20 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:20 smithi102 ceph-mon[78322]: osdmap e65: 6 total, 6 up, 6 in 2024-03-26T18:59:20.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:20 smithi190 ceph-mon[65689]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T18:59:20.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:20 smithi190 ceph-mon[65689]: osd.2 [v2:172.21.15.102:6818/618947256,v1:172.21.15.102:6819/618947256] boot 2024-03-26T18:59:20.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:20 smithi190 ceph-mon[65689]: osdmap e64: 6 total, 6 up, 6 in 2024-03-26T18:59:20.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:20 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T18:59:20.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:20 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:20.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:20 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:20.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:20 smithi190 ceph-mon[65689]: osdmap e65: 6 total, 6 up, 6 in 2024-03-26T18:59:20.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.886+0000 7fc6916ad700 1 -- 172.21.15.102:0/3282142026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc68c100c60 msgr2=0x7fc68c103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.886+0000 7fc6916ad700 1 --2- 172.21.15.102:0/3282142026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc68c100c60 0x7fc68c103040 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc680009910 tx=0x7fc680009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.886+0000 7fc6916ad700 1 -- 172.21.15.102:0/3282142026 shutdown_connections 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.886+0000 7fc6916ad700 1 --2- 172.21.15.102:0/3282142026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc68c103580 0x7fc68c105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.886+0000 7fc6916ad700 1 --2- 172.21.15.102:0/3282142026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc68c100c60 0x7fc68c103040 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.886+0000 7fc6916ad700 1 -- 172.21.15.102:0/3282142026 >> 172.21.15.102:0/3282142026 conn(0x7fc68c0fa600 msgr2=0x7fc68c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.886+0000 7fc6916ad700 1 -- 172.21.15.102:0/3282142026 shutdown_connections 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.886+0000 7fc6916ad700 1 -- 172.21.15.102:0/3282142026 wait complete. 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.887+0000 7fc6916ad700 1 Processor -- start 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.887+0000 7fc6916ad700 1 -- start start 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.887+0000 7fc6916ad700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc68c100c60 0x7fc68c1005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.887+0000 7fc6916ad700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc68c103580 0x7fc68c0fec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.887+0000 7fc6916ad700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c0ff150 con 0x7fc68c100c60 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.887+0000 7fc6916ad700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c0ff290 con 0x7fc68c103580 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc68affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc68c100c60 0x7fc68c1005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc68affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc68c100c60 0x7fc68c1005c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45608/0 (socket says 172.21.15.102:45608) 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc68affd700 1 -- 172.21.15.102:0/551966484 learned_addr learned my addr 172.21.15.102:0/551966484 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc68a7fc700 1 --2- 172.21.15.102:0/551966484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc68c103580 0x7fc68c0fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc68affd700 1 -- 172.21.15.102:0/551966484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc68c103580 msgr2=0x7fc68c0fec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc68affd700 1 --2- 172.21.15.102:0/551966484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc68c103580 0x7fc68c0fec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc68affd700 1 -- 172.21.15.102:0/551966484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6800095f0 con 0x7fc68c100c60 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc68affd700 1 --2- 172.21.15.102:0/551966484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc68c100c60 0x7fc68c1005c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc680006010 tx=0x7fc68000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc673fff700 1 -- 172.21.15.102:0/551966484 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc68001d070 con 0x7fc68c100c60 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc673fff700 1 -- 172.21.15.102:0/551966484 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc68000f460 con 0x7fc68c100c60 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc673fff700 1 -- 172.21.15.102:0/551966484 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc680021640 con 0x7fc68c100c60 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc6916ad700 1 -- 172.21.15.102:0/551966484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc68c0ff510 con 0x7fc68c100c60 2024-03-26T18:59:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.888+0000 7fc6916ad700 1 -- 172.21.15.102:0/551966484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68c0ff9b0 con 0x7fc68c100c60 2024-03-26T18:59:20.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.889+0000 7fc6916ad700 1 -- 172.21.15.102:0/551966484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68c18fb70 con 0x7fc68c100c60 2024-03-26T18:59:20.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.890+0000 7fc673fff700 1 -- 172.21.15.102:0/551966484 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc68000fab0 con 0x7fc68c100c60 2024-03-26T18:59:20.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.890+0000 7fc673fff700 1 --2- 172.21.15.102:0/551966484 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc67406e460 0x7fc674070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:20.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.891+0000 7fc68a7fc700 1 --2- 172.21.15.102:0/551966484 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc67406e460 0x7fc674070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:20.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.891+0000 7fc673fff700 1 -- 172.21.15.102:0/551966484 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc68008f740 con 0x7fc68c100c60 2024-03-26T18:59:20.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.891+0000 7fc68a7fc700 1 --2- 172.21.15.102:0/551966484 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc67406e460 0x7fc674070910 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc67c005950 tx=0x7fc67c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:20.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:20.895+0000 7fc673fff700 1 -- 172.21.15.102:0/551966484 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc68005c3d0 con 0x7fc68c100c60 2024-03-26T18:59:21.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.123+0000 7fc6916ad700 1 -- 172.21.15.102:0/551966484 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc68c061190 con 0x7fc67406e460 2024-03-26T18:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.124+0000 7fc673fff700 1 -- 172.21.15.102:0/551966484 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fc68c061190 con 0x7fc67406e460 2024-03-26T18:59:21.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.126+0000 7fc6916ad700 1 -- 172.21.15.102:0/551966484 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc67406e460 msgr2=0x7fc674070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.126+0000 7fc6916ad700 1 --2- 172.21.15.102:0/551966484 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc67406e460 0x7fc674070910 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc67c005950 tx=0x7fc67c00b410 comp rx=0 tx=0).stop 2024-03-26T18:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.126+0000 7fc6916ad700 1 -- 172.21.15.102:0/551966484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc68c100c60 msgr2=0x7fc68c1005c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.126+0000 7fc6916ad700 1 --2- 172.21.15.102:0/551966484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc68c100c60 0x7fc68c1005c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc680006010 tx=0x7fc68000bd70 comp rx=0 tx=0).stop 2024-03-26T18:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.126+0000 7fc6916ad700 1 -- 172.21.15.102:0/551966484 shutdown_connections 2024-03-26T18:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.126+0000 7fc6916ad700 1 --2- 172.21.15.102:0/551966484 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc67406e460 0x7fc674070910 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.126+0000 7fc6916ad700 1 --2- 172.21.15.102:0/551966484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc68c100c60 0x7fc68c1005c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.126+0000 7fc6916ad700 1 --2- 172.21.15.102:0/551966484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc68c103580 0x7fc68c0fec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.126+0000 7fc6916ad700 1 -- 172.21.15.102:0/551966484 >> 172.21.15.102:0/551966484 conn(0x7fc68c0fa600 msgr2=0x7fc68c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.126+0000 7fc6916ad700 1 -- 172.21.15.102:0/551966484 shutdown_connections 2024-03-26T18:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.126+0000 7fc6916ad700 1 -- 172.21.15.102:0/551966484 wait complete. 2024-03-26T18:59:21.130 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:59:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.231+0000 7f4334f88700 1 -- 172.21.15.102:0/1359448362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4330103570 msgr2=0x7f4330105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.231+0000 7f4334f88700 1 --2- 172.21.15.102:0/1359448362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4330103570 0x7f4330105950 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4324009910 tx=0x7f4324009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.232+0000 7f4334f88700 1 -- 172.21.15.102:0/1359448362 shutdown_connections 2024-03-26T18:59:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.232+0000 7f4334f88700 1 --2- 172.21.15.102:0/1359448362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4330103570 0x7f4330105950 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.232+0000 7f4334f88700 1 --2- 172.21.15.102:0/1359448362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4330100c50 0x7f4330103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.232+0000 7f4334f88700 1 -- 172.21.15.102:0/1359448362 >> 172.21.15.102:0/1359448362 conn(0x7f43300fa610 msgr2=0x7f43300fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.232+0000 7f4334f88700 1 -- 172.21.15.102:0/1359448362 shutdown_connections 2024-03-26T18:59:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.232+0000 7f4334f88700 1 -- 172.21.15.102:0/1359448362 wait complete. 2024-03-26T18:59:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.232+0000 7f4334f88700 1 Processor -- start 2024-03-26T18:59:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.232+0000 7f4334f88700 1 -- start start 2024-03-26T18:59:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f4334f88700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4330100c50 0x7f4330195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f4334f88700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4330103570 0x7f4330195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f4334f88700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43301964c0 con 0x7f4330103570 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f4334f88700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4330196600 con 0x7f4330100c50 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f432dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4330103570 0x7f4330195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f432dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4330103570 0x7f4330195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45630/0 (socket says 172.21.15.102:45630) 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f432dd9b700 1 -- 172.21.15.102:0/2287734790 learned_addr learned my addr 172.21.15.102:0/2287734790 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f432e59c700 1 --2- 172.21.15.102:0/2287734790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4330100c50 0x7f4330195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f432dd9b700 1 -- 172.21.15.102:0/2287734790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4330100c50 msgr2=0x7f4330195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f432dd9b700 1 --2- 172.21.15.102:0/2287734790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4330100c50 0x7f4330195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f432dd9b700 1 -- 172.21.15.102:0/2287734790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43240095f0 con 0x7f4330103570 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f432dd9b700 1 --2- 172.21.15.102:0/2287734790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4330103570 0x7f4330195ea0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4324000c00 tx=0x7f432400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f431f7fe700 1 -- 172.21.15.102:0/2287734790 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f432401d070 con 0x7f4330103570 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f431f7fe700 1 -- 172.21.15.102:0/2287734790 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4324005dc0 con 0x7f4330103570 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f431f7fe700 1 -- 172.21.15.102:0/2287734790 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43240226a0 con 0x7f4330103570 2024-03-26T18:59:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.233+0000 7f4334f88700 1 -- 172.21.15.102:0/2287734790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433019b050 con 0x7f4330103570 2024-03-26T18:59:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.234+0000 7f4334f88700 1 -- 172.21.15.102:0/2287734790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433019b4f0 con 0x7f4330103570 2024-03-26T18:59:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.234+0000 7f4334f88700 1 -- 172.21.15.102:0/2287734790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433018fb80 con 0x7f4330103570 2024-03-26T18:59:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.235+0000 7f431f7fe700 1 -- 172.21.15.102:0/2287734790 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f43240038f0 con 0x7f4330103570 2024-03-26T18:59:21.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.236+0000 7f431f7fe700 1 --2- 172.21.15.102:0/2287734790 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f431806e460 0x7f4318070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:21.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.236+0000 7f431f7fe700 1 -- 172.21.15.102:0/2287734790 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f432408fcd0 con 0x7f4330103570 2024-03-26T18:59:21.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.236+0000 7f432e59c700 1 --2- 172.21.15.102:0/2287734790 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f431806e460 0x7f4318070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:21.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.236+0000 7f432e59c700 1 --2- 172.21.15.102:0/2287734790 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f431806e460 0x7f4318070910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4320009c60 tx=0x7f4320009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:21.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.240+0000 7f431f7fe700 1 -- 172.21.15.102:0/2287734790 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f432405ca20 con 0x7f4330103570 2024-03-26T18:59:21.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.465+0000 7f4334f88700 1 -- 172.21.15.102:0/2287734790 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4330061190 con 0x7f431806e460 2024-03-26T18:59:21.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.466+0000 7f431f7fe700 1 -- 172.21.15.102:0/2287734790 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f4330061190 con 0x7f431806e460 2024-03-26T18:59:21.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.468+0000 7f4334f88700 1 -- 172.21.15.102:0/2287734790 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f431806e460 msgr2=0x7f4318070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.468+0000 7f4334f88700 1 --2- 172.21.15.102:0/2287734790 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f431806e460 0x7f4318070910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4320009c60 tx=0x7f4320009260 comp rx=0 tx=0).stop 2024-03-26T18:59:21.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.468+0000 7f4334f88700 1 -- 172.21.15.102:0/2287734790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4330103570 msgr2=0x7f4330195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.468+0000 7f4334f88700 1 --2- 172.21.15.102:0/2287734790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4330103570 0x7f4330195ea0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4324000c00 tx=0x7f432400f690 comp rx=0 tx=0).stop 2024-03-26T18:59:21.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.469+0000 7f4334f88700 1 -- 172.21.15.102:0/2287734790 shutdown_connections 2024-03-26T18:59:21.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.469+0000 7f4334f88700 1 --2- 172.21.15.102:0/2287734790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4330100c50 0x7f4330195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.469+0000 7f4334f88700 1 --2- 172.21.15.102:0/2287734790 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f431806e460 0x7f4318070910 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.469+0000 7f4334f88700 1 --2- 172.21.15.102:0/2287734790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4330103570 0x7f4330195ea0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.469+0000 7f4334f88700 1 -- 172.21.15.102:0/2287734790 >> 172.21.15.102:0/2287734790 conn(0x7f43300fa610 msgr2=0x7f43300fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:21.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.469+0000 7f4334f88700 1 -- 172.21.15.102:0/2287734790 shutdown_connections 2024-03-26T18:59:21.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.469+0000 7f4334f88700 1 -- 172.21.15.102:0/2287734790 wait complete. 2024-03-26T18:59:21.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.578+0000 7fe2f3950700 1 -- 172.21.15.102:0/3969548820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2ec100250 msgr2=0x7fe2ec1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.578+0000 7fe2f3950700 1 --2- 172.21.15.102:0/3969548820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2ec100250 0x7fe2ec1006a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009910 tx=0x7fe2e0009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:21.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.578+0000 7fe2f3950700 1 -- 172.21.15.102:0/3969548820 shutdown_connections 2024-03-26T18:59:21.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.578+0000 7fe2f3950700 1 --2- 172.21.15.102:0/3969548820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2ec100250 0x7fe2ec1006a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.578+0000 7fe2f3950700 1 --2- 172.21.15.102:0/3969548820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2ec0ff050 0x7fe2ec0ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.578+0000 7fe2f3950700 1 -- 172.21.15.102:0/3969548820 >> 172.21.15.102:0/3969548820 conn(0x7fe2ec0fa5e0 msgr2=0x7fe2ec0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:21.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.578+0000 7fe2f3950700 1 -- 172.21.15.102:0/3969548820 shutdown_connections 2024-03-26T18:59:21.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.578+0000 7fe2f3950700 1 -- 172.21.15.102:0/3969548820 wait complete. 2024-03-26T18:59:21.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.578+0000 7fe2f3950700 1 Processor -- start 2024-03-26T18:59:21.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.578+0000 7fe2f3950700 1 -- start start 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f3950700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2ec0ff050 0x7fe2ec195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f3950700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2ec100250 0x7fe2ec195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f3950700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec196460 con 0x7fe2ec100250 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f3950700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec1965a0 con 0x7fe2ec0ff050 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f0eeb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2ec100250 0x7fe2ec195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f0eeb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2ec100250 0x7fe2ec195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45656/0 (socket says 172.21.15.102:45656) 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f0eeb700 1 -- 172.21.15.102:0/2228551351 learned_addr learned my addr 172.21.15.102:0/2228551351 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f16ec700 1 --2- 172.21.15.102:0/2228551351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2ec0ff050 0x7fe2ec195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f0eeb700 1 -- 172.21.15.102:0/2228551351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2ec0ff050 msgr2=0x7fe2ec195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f0eeb700 1 --2- 172.21.15.102:0/2228551351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2ec0ff050 0x7fe2ec195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f0eeb700 1 -- 172.21.15.102:0/2228551351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e00095f0 con 0x7fe2ec100250 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f0eeb700 1 --2- 172.21.15.102:0/2228551351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2ec100250 0x7fe2ec195ed0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000bd70 tx=0x7fe2e000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:21.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2ea7fc700 1 -- 172.21.15.102:0/2228551351 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2e001d070 con 0x7fe2ec100250 2024-03-26T18:59:21.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2ea7fc700 1 -- 172.21.15.102:0/2228551351 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe2e0005940 con 0x7fe2ec100250 2024-03-26T18:59:21.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f3950700 1 -- 172.21.15.102:0/2228551351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2ec19b000 con 0x7fe2ec100250 2024-03-26T18:59:21.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2ea7fc700 1 -- 172.21.15.102:0/2228551351 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2e0021430 con 0x7fe2ec100250 2024-03-26T18:59:21.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.579+0000 7fe2f3950700 1 -- 172.21.15.102:0/2228551351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ec19b4c0 con 0x7fe2ec100250 2024-03-26T18:59:21.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.580+0000 7fe2f3950700 1 -- 172.21.15.102:0/2228551351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2ec107ba0 con 0x7fe2ec100250 2024-03-26T18:59:21.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.581+0000 7fe2ea7fc700 1 -- 172.21.15.102:0/2228551351 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe2e000f860 con 0x7fe2ec100250 2024-03-26T18:59:21.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.581+0000 7fe2ea7fc700 1 --2- 172.21.15.102:0/2228551351 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe2d406e460 0x7fe2d4070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:21.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.581+0000 7fe2ea7fc700 1 -- 172.21.15.102:0/2228551351 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe2e008f700 con 0x7fe2ec100250 2024-03-26T18:59:21.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.581+0000 7fe2f16ec700 1 --2- 172.21.15.102:0/2228551351 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe2d406e460 0x7fe2d4070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:21.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.582+0000 7fe2f16ec700 1 --2- 172.21.15.102:0/2228551351 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe2d406e460 0x7fe2d4070910 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec1000b0 tx=0x7fe2e4007a30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:21.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.586+0000 7fe2ea7fc700 1 -- 172.21.15.102:0/2228551351 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe2e005c430 con 0x7fe2ec100250 2024-03-26T18:59:21.697 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:21 smithi102 ceph-mon[78322]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:21.697 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:21 smithi102 ceph-mon[78322]: pgmap v100: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 42/354 objects degraded (11.864%) 2024-03-26T18:59:21.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:21 smithi190 ceph-mon[65689]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:21.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:21 smithi190 ceph-mon[65689]: pgmap v100: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 42/354 objects degraded (11.864%) 2024-03-26T18:59:21.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.805+0000 7fe2f3950700 1 -- 172.21.15.102:0/2228551351 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2ec068ae0 con 0x7fe2d406e460 2024-03-26T18:59:21.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.816+0000 7fe2ea7fc700 1 -- 172.21.15.102:0/2228551351 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fe2ec068ae0 con 0x7fe2d406e460 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (10m) 3s ago 12m 25.0M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (12m) 3s ago 12m 8354k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (11m) 96s ago 11m 7755k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (106s) 3s ago 12m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (101s) 96s ago 11m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (10m) 3s ago 12m 86.6M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (6m) 3s ago 6m 132M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (6m) 3s ago 6m 16.1M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (6m) 96s ago 6m 15.3M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (6m) 96s ago 6m 120M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:59:21.810 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (3m) 3s ago 13m 504M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (3m) 96s ago 11m 449M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (2m) 3s ago 13m 46.2M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (2m) 96s ago 11m 32.4M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (12m) 3s ago 12m 16.3M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (11m) 96s ago 11m 15.5M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (79s) 3s ago 10m 86.0M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (44s) 3s ago 10m 95.7M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (8s) 3s ago 9m 11.7M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (9m) 96s ago 9m 272M 2240M 18.2.0 dc2bc1663786 3cd9522e1208 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (8m) 96s ago 8m 251M 2240M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (8m) 96s ago 8m 225M 2240M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:59:21.811 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (3m) 3s ago 11m 74.3M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T18:59:21.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.818+0000 7fe2f3950700 1 -- 172.21.15.102:0/2228551351 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe2d406e460 msgr2=0x7fe2d4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.818+0000 7fe2f3950700 1 --2- 172.21.15.102:0/2228551351 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe2d406e460 0x7fe2d4070910 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec1000b0 tx=0x7fe2e4007a30 comp rx=0 tx=0).stop 2024-03-26T18:59:21.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.818+0000 7fe2f3950700 1 -- 172.21.15.102:0/2228551351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2ec100250 msgr2=0x7fe2ec195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.818+0000 7fe2f3950700 1 --2- 172.21.15.102:0/2228551351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2ec100250 0x7fe2ec195ed0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000bd70 tx=0x7fe2e000bda0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.819+0000 7fe2f3950700 1 -- 172.21.15.102:0/2228551351 shutdown_connections 2024-03-26T18:59:21.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.819+0000 7fe2f3950700 1 --2- 172.21.15.102:0/2228551351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2ec0ff050 0x7fe2ec195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.819+0000 7fe2f3950700 1 --2- 172.21.15.102:0/2228551351 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe2d406e460 0x7fe2d4070910 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.819+0000 7fe2f3950700 1 --2- 172.21.15.102:0/2228551351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2ec100250 0x7fe2ec195ed0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.819+0000 7fe2f3950700 1 -- 172.21.15.102:0/2228551351 >> 172.21.15.102:0/2228551351 conn(0x7fe2ec0fa5e0 msgr2=0x7fe2ec0683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:21.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.819+0000 7fe2f3950700 1 -- 172.21.15.102:0/2228551351 shutdown_connections 2024-03-26T18:59:21.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.819+0000 7fe2f3950700 1 -- 172.21.15.102:0/2228551351 wait complete. 2024-03-26T18:59:21.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.925+0000 7f96c2044700 1 -- 172.21.15.102:0/4236995620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96bc100280 msgr2=0x7f96bc1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.925+0000 7f96c2044700 1 --2- 172.21.15.102:0/4236995620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96bc100280 0x7f96bc1006d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009910 tx=0x7f96b0009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:21.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.925+0000 7f96c2044700 1 -- 172.21.15.102:0/4236995620 shutdown_connections 2024-03-26T18:59:21.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.925+0000 7f96c2044700 1 --2- 172.21.15.102:0/4236995620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96bc100280 0x7f96bc1006d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.925+0000 7f96c2044700 1 --2- 172.21.15.102:0/4236995620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96bc0ff080 0x7f96bc0ff490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.925+0000 7f96c2044700 1 -- 172.21.15.102:0/4236995620 >> 172.21.15.102:0/4236995620 conn(0x7f96bc0fa630 msgr2=0x7f96bc0fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:21.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.925+0000 7f96c2044700 1 -- 172.21.15.102:0/4236995620 shutdown_connections 2024-03-26T18:59:21.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.926+0000 7f96c2044700 1 -- 172.21.15.102:0/4236995620 wait complete. 2024-03-26T18:59:21.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.926+0000 7f96c2044700 1 Processor -- start 2024-03-26T18:59:21.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.926+0000 7f96c2044700 1 -- start start 2024-03-26T18:59:21.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96c2044700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96bc0ff080 0x7f96bc195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:21.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96c2044700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96bc100280 0x7f96bc195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:21.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96c2044700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc1964c0 con 0x7f96bc100280 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96c2044700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc196600 con 0x7f96bc0ff080 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96baffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96bc100280 0x7f96bc195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96baffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96bc100280 0x7f96bc195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45664/0 (socket says 172.21.15.102:45664) 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96baffd700 1 -- 172.21.15.102:0/1072366178 learned_addr learned my addr 172.21.15.102:0/1072366178 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96bb7fe700 1 --2- 172.21.15.102:0/1072366178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96bc0ff080 0x7f96bc195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96baffd700 1 -- 172.21.15.102:0/1072366178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96bc0ff080 msgr2=0x7f96bc195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96baffd700 1 --2- 172.21.15.102:0/1072366178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96bc0ff080 0x7f96bc195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96baffd700 1 -- 172.21.15.102:0/1072366178 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b00095f0 con 0x7f96bc100280 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96baffd700 1 --2- 172.21.15.102:0/1072366178 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96bc100280 0x7f96bc195ea0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f96b0000c00 tx=0x7f96b000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.927+0000 7f96b8ff9700 1 -- 172.21.15.102:0/1072366178 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b001d070 con 0x7f96bc100280 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.928+0000 7f96b8ff9700 1 -- 172.21.15.102:0/1072366178 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f96b0005dc0 con 0x7f96bc100280 2024-03-26T18:59:21.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.928+0000 7f96c2044700 1 -- 172.21.15.102:0/1072366178 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96bc19b050 con 0x7f96bc100280 2024-03-26T18:59:21.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.928+0000 7f96b8ff9700 1 -- 172.21.15.102:0/1072366178 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b00226a0 con 0x7f96bc100280 2024-03-26T18:59:21.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.928+0000 7f96c2044700 1 -- 172.21.15.102:0/1072366178 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96bc19b540 con 0x7f96bc100280 2024-03-26T18:59:21.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.929+0000 7f96c2044700 1 -- 172.21.15.102:0/1072366178 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96bc108ce0 con 0x7f96bc100280 2024-03-26T18:59:21.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.930+0000 7f96b8ff9700 1 -- 172.21.15.102:0/1072366178 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f96b00038f0 con 0x7f96bc100280 2024-03-26T18:59:21.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.930+0000 7f96b8ff9700 1 --2- 172.21.15.102:0/1072366178 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96a406e460 0x7f96a4070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:21.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.931+0000 7f96b8ff9700 1 -- 172.21.15.102:0/1072366178 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f96b008fb60 con 0x7f96bc100280 2024-03-26T18:59:21.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.931+0000 7f96bb7fe700 1 --2- 172.21.15.102:0/1072366178 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96a406e460 0x7f96a4070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:21.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.931+0000 7f96bb7fe700 1 --2- 172.21.15.102:0/1072366178 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96a406e460 0x7f96a4070910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f96bc1000e0 tx=0x7f96ac009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:21.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:21.934+0000 7f96b8ff9700 1 -- 172.21.15.102:0/1072366178 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f96b005c9b0 con 0x7f96bc100280 2024-03-26T18:59:22.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.232+0000 7f96c2044700 1 -- 172.21.15.102:0/1072366178 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f96bc108ed0 con 0x7f96bc100280 2024-03-26T18:59:22.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.233+0000 7f96b8ff9700 1 -- 172.21.15.102:0/1072366178 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f96b005c540 con 0x7f96bc100280 2024-03-26T18:59:22.227 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:59:22.227 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:59:22.227 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:59:22.227 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:59:22.227 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:59:22.227 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:59:22.227 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:59:22.227 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:59:22.227 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 3, 2024-03-26T18:59:22.227 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 3 2024-03-26T18:59:22.228 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:59:22.228 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:59:22.228 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:59:22.228 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:59:22.228 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:59:22.228 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 7, 2024-03-26T18:59:22.228 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 7 2024-03-26T18:59:22.228 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:59:22.228 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:59:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.235+0000 7f96c2044700 1 -- 172.21.15.102:0/1072366178 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96a406e460 msgr2=0x7f96a4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.235+0000 7f96c2044700 1 --2- 172.21.15.102:0/1072366178 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96a406e460 0x7f96a4070910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f96bc1000e0 tx=0x7f96ac009260 comp rx=0 tx=0).stop 2024-03-26T18:59:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.235+0000 7f96c2044700 1 -- 172.21.15.102:0/1072366178 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96bc100280 msgr2=0x7f96bc195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.235+0000 7f96c2044700 1 --2- 172.21.15.102:0/1072366178 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96bc100280 0x7f96bc195ea0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f96b0000c00 tx=0x7f96b000f690 comp rx=0 tx=0).stop 2024-03-26T18:59:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.235+0000 7f96c2044700 1 -- 172.21.15.102:0/1072366178 shutdown_connections 2024-03-26T18:59:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.235+0000 7f96c2044700 1 --2- 172.21.15.102:0/1072366178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96bc0ff080 0x7f96bc195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.235+0000 7f96c2044700 1 --2- 172.21.15.102:0/1072366178 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96a406e460 0x7f96a4070910 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.235+0000 7f96c2044700 1 --2- 172.21.15.102:0/1072366178 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96bc100280 0x7f96bc195ea0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.235+0000 7f96c2044700 1 -- 172.21.15.102:0/1072366178 >> 172.21.15.102:0/1072366178 conn(0x7f96bc0fa630 msgr2=0x7f96bc1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.235+0000 7f96c2044700 1 -- 172.21.15.102:0/1072366178 shutdown_connections 2024-03-26T18:59:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.235+0000 7f96c2044700 1 -- 172.21.15.102:0/1072366178 wait complete. 2024-03-26T18:59:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.349+0000 7f2a324a0700 1 -- 172.21.15.102:0/1640163511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a2c100c20 msgr2=0x7f2a2c103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.349+0000 7f2a324a0700 1 --2- 172.21.15.102:0/1640163511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a2c100c20 0x7f2a2c103000 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c009910 tx=0x7f2a1c009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.349+0000 7f2a324a0700 1 -- 172.21.15.102:0/1640163511 shutdown_connections 2024-03-26T18:59:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.349+0000 7f2a324a0700 1 --2- 172.21.15.102:0/1640163511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a2c103540 0x7f2a2c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.349+0000 7f2a324a0700 1 --2- 172.21.15.102:0/1640163511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a2c100c20 0x7f2a2c103000 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.349+0000 7f2a324a0700 1 -- 172.21.15.102:0/1640163511 >> 172.21.15.102:0/1640163511 conn(0x7f2a2c0fa5e0 msgr2=0x7f2a2c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.349+0000 7f2a324a0700 1 -- 172.21.15.102:0/1640163511 shutdown_connections 2024-03-26T18:59:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.349+0000 7f2a324a0700 1 -- 172.21.15.102:0/1640163511 wait complete. 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.349+0000 7f2a324a0700 1 Processor -- start 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.349+0000 7f2a324a0700 1 -- start start 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a324a0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a2c100c20 0x7f2a2c100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a324a0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a2c103540 0x7f2a2c0febd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a324a0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c0ff110 con 0x7f2a2c103540 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a324a0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c0ff250 con 0x7f2a2c100c20 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a2b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a2c103540 0x7f2a2c0febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a2b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a2c103540 0x7f2a2c0febd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45694/0 (socket says 172.21.15.102:45694) 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a2b7fe700 1 -- 172.21.15.102:0/994522450 learned_addr learned my addr 172.21.15.102:0/994522450 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a2bfff700 1 --2- 172.21.15.102:0/994522450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a2c100c20 0x7f2a2c100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a2b7fe700 1 -- 172.21.15.102:0/994522450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a2c100c20 msgr2=0x7f2a2c100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a2b7fe700 1 --2- 172.21.15.102:0/994522450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a2c100c20 0x7f2a2c100580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a2b7fe700 1 -- 172.21.15.102:0/994522450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c0095f0 con 0x7f2a2c103540 2024-03-26T18:59:22.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.350+0000 7f2a2b7fe700 1 --2- 172.21.15.102:0/994522450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a2c103540 0x7f2a2c0febd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000d8d0 tx=0x7f2a2000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:22.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.351+0000 7f2a297fa700 1 -- 172.21.15.102:0/994522450 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a200098a0 con 0x7f2a2c103540 2024-03-26T18:59:22.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.351+0000 7f2a324a0700 1 -- 172.21.15.102:0/994522450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c0ff530 con 0x7f2a2c103540 2024-03-26T18:59:22.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.351+0000 7f2a297fa700 1 -- 172.21.15.102:0/994522450 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2a20010460 con 0x7f2a2c103540 2024-03-26T18:59:22.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.351+0000 7f2a297fa700 1 -- 172.21.15.102:0/994522450 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2000f5f0 con 0x7f2a2c103540 2024-03-26T18:59:22.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.351+0000 7f2a324a0700 1 -- 172.21.15.102:0/994522450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c19d990 con 0x7f2a2c103540 2024-03-26T18:59:22.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.352+0000 7f2a12ffd700 1 -- 172.21.15.102:0/994522450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2c108c90 con 0x7f2a2c103540 2024-03-26T18:59:22.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.352+0000 7f2a297fa700 1 -- 172.21.15.102:0/994522450 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2a20009a00 con 0x7f2a2c103540 2024-03-26T18:59:22.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.353+0000 7f2a297fa700 1 --2- 172.21.15.102:0/994522450 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2a1406e460 0x7f2a14070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:22.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.353+0000 7f2a2bfff700 1 --2- 172.21.15.102:0/994522450 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2a1406e460 0x7f2a14070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:22.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.353+0000 7f2a297fa700 1 -- 172.21.15.102:0/994522450 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2a2008e300 con 0x7f2a2c103540 2024-03-26T18:59:22.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.353+0000 7f2a2bfff700 1 --2- 172.21.15.102:0/994522450 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2a1406e460 0x7f2a14070910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c006010 tx=0x7f2a1c00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:22.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.357+0000 7f2a297fa700 1 -- 172.21.15.102:0/994522450 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2a2005af90 con 0x7f2a2c103540 2024-03-26T18:59:22.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.606+0000 7f2a12ffd700 1 -- 172.21.15.102:0/994522450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2a2c108e20 con 0x7f2a2c103540 2024-03-26T18:59:22.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.607+0000 7f2a297fa700 1 -- 172.21.15.102:0/994522450 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1905 (secure 0 0 0) 0x7f2a2005ab20 con 0x7f2a2c103540 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:e21 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 21 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:58:42.198340+0000 2024-03-26T18:59:22.601 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:59:22.602 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 14490 members: 24283,14490 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:59:22.603 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:59:22.604 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:59:22.604 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:59:22.604 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:59:22.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.609+0000 7f2a12ffd700 1 -- 172.21.15.102:0/994522450 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2a1406e460 msgr2=0x7f2a14070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.609+0000 7f2a12ffd700 1 --2- 172.21.15.102:0/994522450 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2a1406e460 0x7f2a14070910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c006010 tx=0x7f2a1c00b540 comp rx=0 tx=0).stop 2024-03-26T18:59:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.609+0000 7f2a12ffd700 1 -- 172.21.15.102:0/994522450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a2c103540 msgr2=0x7f2a2c0febd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.609+0000 7f2a12ffd700 1 --2- 172.21.15.102:0/994522450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a2c103540 0x7f2a2c0febd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000d8d0 tx=0x7f2a2000dbe0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.609+0000 7f2a12ffd700 1 -- 172.21.15.102:0/994522450 shutdown_connections 2024-03-26T18:59:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.609+0000 7f2a12ffd700 1 --2- 172.21.15.102:0/994522450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a2c100c20 0x7f2a2c100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.609+0000 7f2a12ffd700 1 --2- 172.21.15.102:0/994522450 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2a1406e460 0x7f2a14070910 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.609+0000 7f2a12ffd700 1 --2- 172.21.15.102:0/994522450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a2c103540 0x7f2a2c0febd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.609+0000 7f2a12ffd700 1 -- 172.21.15.102:0/994522450 >> 172.21.15.102:0/994522450 conn(0x7f2a2c0fa5e0 msgr2=0x7f2a2c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.609+0000 7f2a12ffd700 1 -- 172.21.15.102:0/994522450 shutdown_connections 2024-03-26T18:59:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.609+0000 7f2a12ffd700 1 -- 172.21.15.102:0/994522450 wait complete. 2024-03-26T18:59:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 21 2024-03-26T18:59:22.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.715+0000 7fe0e25cd700 1 -- 172.21.15.102:0/686463460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0dc1014b0 msgr2=0x7fe0dc101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:22.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.715+0000 7fe0e25cd700 1 --2- 172.21.15.102:0/686463460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0dc1014b0 0x7fe0dc101900 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0009910 tx=0x7fe0d0009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:22.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.715+0000 7fe0e25cd700 1 -- 172.21.15.102:0/686463460 shutdown_connections 2024-03-26T18:59:22.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.715+0000 7fe0e25cd700 1 --2- 172.21.15.102:0/686463460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0dc1014b0 0x7fe0dc101900 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.715+0000 7fe0e25cd700 1 --2- 172.21.15.102:0/686463460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0dc0fe430 0x7fe0dc0fe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.715+0000 7fe0e25cd700 1 -- 172.21.15.102:0/686463460 >> 172.21.15.102:0/686463460 conn(0x7fe0dc0f9d80 msgr2=0x7fe0dc0fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:22.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.715+0000 7fe0e25cd700 1 -- 172.21.15.102:0/686463460 shutdown_connections 2024-03-26T18:59:22.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.715+0000 7fe0e25cd700 1 -- 172.21.15.102:0/686463460 wait complete. 2024-03-26T18:59:22.709 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:22 smithi102 ceph-mon[78322]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:22.709 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:22 smithi102 ceph-mon[78322]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:22.710 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:22 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1072366178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:22.710 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:22.710 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:59:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.715+0000 7fe0e25cd700 1 Processor -- start 2024-03-26T18:59:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.715+0000 7fe0e25cd700 1 -- start start 2024-03-26T18:59:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0e25cd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0dc0fe430 0x7fe0dc195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0e25cd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0dc1014b0 0x7fe0dc195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0e25cd700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc1964c0 con 0x7fe0dc0fe430 2024-03-26T18:59:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0e25cd700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc196600 con 0x7fe0dc1014b0 2024-03-26T18:59:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0dbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0dc0fe430 0x7fe0dc195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0dbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0dc0fe430 0x7fe0dc195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45718/0 (socket says 172.21.15.102:45718) 2024-03-26T18:59:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0dbfff700 1 -- 172.21.15.102:0/221329673 learned_addr learned my addr 172.21.15.102:0/221329673 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0db7fe700 1 --2- 172.21.15.102:0/221329673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0dc1014b0 0x7fe0dc195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0dbfff700 1 -- 172.21.15.102:0/221329673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0dc1014b0 msgr2=0x7fe0dc195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0dbfff700 1 --2- 172.21.15.102:0/221329673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0dc1014b0 0x7fe0dc195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0dbfff700 1 -- 172.21.15.102:0/221329673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d00095f0 con 0x7fe0dc0fe430 2024-03-26T18:59:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.716+0000 7fe0dbfff700 1 --2- 172.21.15.102:0/221329673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0dc0fe430 0x7fe0dc195960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00d900 tx=0x7fe0cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.717+0000 7fe0d97fa700 1 -- 172.21.15.102:0/221329673 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0cc0041f0 con 0x7fe0dc0fe430 2024-03-26T18:59:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.717+0000 7fe0d97fa700 1 -- 172.21.15.102:0/221329673 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe0cc004350 con 0x7fe0dc0fe430 2024-03-26T18:59:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.717+0000 7fe0e25cd700 1 -- 172.21.15.102:0/221329673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0dc19b0b0 con 0x7fe0dc0fe430 2024-03-26T18:59:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.717+0000 7fe0d97fa700 1 -- 172.21.15.102:0/221329673 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0cc003da0 con 0x7fe0dc0fe430 2024-03-26T18:59:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.717+0000 7fe0e25cd700 1 -- 172.21.15.102:0/221329673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0dc19b600 con 0x7fe0dc0fe430 2024-03-26T18:59:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.718+0000 7fe0e25cd700 1 -- 172.21.15.102:0/221329673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0dc108c90 con 0x7fe0dc0fe430 2024-03-26T18:59:22.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.718+0000 7fe0d97fa700 1 -- 172.21.15.102:0/221329673 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe0cc009730 con 0x7fe0dc0fe430 2024-03-26T18:59:22.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.719+0000 7fe0d97fa700 1 --2- 172.21.15.102:0/221329673 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe0c406e460 0x7fe0c4070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:22.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.719+0000 7fe0db7fe700 1 --2- 172.21.15.102:0/221329673 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe0c406e460 0x7fe0c4070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:22.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.719+0000 7fe0d97fa700 1 -- 172.21.15.102:0/221329673 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe0cc08e340 con 0x7fe0dc0fe430 2024-03-26T18:59:22.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.719+0000 7fe0db7fe700 1 --2- 172.21.15.102:0/221329673 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe0c406e460 0x7fe0c4070910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0006010 tx=0x7fe0d000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:22.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.723+0000 7fe0d97fa700 1 -- 172.21.15.102:0/221329673 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe0cc05b070 con 0x7fe0dc0fe430 2024-03-26T18:59:22.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:22 smithi190 ceph-mon[65689]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:22.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:22 smithi190 ceph-mon[65689]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:22.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:22 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1072366178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:22.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:22.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:59:22.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.950+0000 7fe0e25cd700 1 -- 172.21.15.102:0/221329673 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0dc105e00 con 0x7fe0c406e460 2024-03-26T18:59:22.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.951+0000 7fe0d97fa700 1 -- 172.21.15.102:0/221329673 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe0dc105e00 con 0x7fe0c406e460 2024-03-26T18:59:22.946 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:59:22.946 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:59:22.947 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:59:22.947 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:59:22.947 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T18:59:22.947 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T18:59:22.947 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-03-26T18:59:22.947 INFO:teuthology.orchestra.run.smithi102.stdout: "mon" 2024-03-26T18:59:22.947 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T18:59:22.947 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "9/23 daemons upgraded", 2024-03-26T18:59:22.947 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T18:59:22.947 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:59:22.947 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:59:22.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.953+0000 7fe0e25cd700 1 -- 172.21.15.102:0/221329673 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe0c406e460 msgr2=0x7fe0c4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:22.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.953+0000 7fe0e25cd700 1 --2- 172.21.15.102:0/221329673 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe0c406e460 0x7fe0c4070910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0006010 tx=0x7fe0d000b540 comp rx=0 tx=0).stop 2024-03-26T18:59:22.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.953+0000 7fe0e25cd700 1 -- 172.21.15.102:0/221329673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0dc0fe430 msgr2=0x7fe0dc195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:22.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.953+0000 7fe0e25cd700 1 --2- 172.21.15.102:0/221329673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0dc0fe430 0x7fe0dc195960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00d900 tx=0x7fe0cc00dcc0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.953+0000 7fe0e25cd700 1 -- 172.21.15.102:0/221329673 shutdown_connections 2024-03-26T18:59:22.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.953+0000 7fe0e25cd700 1 --2- 172.21.15.102:0/221329673 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe0c406e460 0x7fe0c4070910 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.953+0000 7fe0e25cd700 1 --2- 172.21.15.102:0/221329673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0dc0fe430 0x7fe0dc195960 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.953+0000 7fe0e25cd700 1 --2- 172.21.15.102:0/221329673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0dc1014b0 0x7fe0dc195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:22.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.953+0000 7fe0e25cd700 1 -- 172.21.15.102:0/221329673 >> 172.21.15.102:0/221329673 conn(0x7fe0dc0f9d80 msgr2=0x7fe0dc1046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:22.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.953+0000 7fe0e25cd700 1 -- 172.21.15.102:0/221329673 shutdown_connections 2024-03-26T18:59:22.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:22.953+0000 7fe0e25cd700 1 -- 172.21.15.102:0/221329673 wait complete. 2024-03-26T18:59:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.059+0000 7f36482fe700 1 -- 172.21.15.102:0/4069672871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3640100610 msgr2=0x7f3640100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.059+0000 7f36482fe700 1 --2- 172.21.15.102:0/4069672871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3640100610 0x7f3640100a20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3638009910 tx=0x7f3638009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.059+0000 7f36482fe700 1 -- 172.21.15.102:0/4069672871 shutdown_connections 2024-03-26T18:59:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.059+0000 7f36482fe700 1 --2- 172.21.15.102:0/4069672871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36400ff350 0x7f36400ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.059+0000 7f36482fe700 1 --2- 172.21.15.102:0/4069672871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3640100610 0x7f3640100a20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.059+0000 7f36482fe700 1 -- 172.21.15.102:0/4069672871 >> 172.21.15.102:0/4069672871 conn(0x7f36400f9db0 msgr2=0x7f36400fc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.059+0000 7f36482fe700 1 -- 172.21.15.102:0/4069672871 shutdown_connections 2024-03-26T18:59:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.059+0000 7f36482fe700 1 -- 172.21.15.102:0/4069672871 wait complete. 2024-03-26T18:59:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f36482fe700 1 Processor -- start 2024-03-26T18:59:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f36482fe700 1 -- start start 2024-03-26T18:59:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f36482fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36400ff350 0x7f36401030f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f36482fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3640100610 0x7f3640101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f36482fe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3640101c80 con 0x7f3640100610 2024-03-26T18:59:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f36482fe700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3640101dc0 con 0x7f36400ff350 2024-03-26T18:59:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f3645899700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3640100610 0x7f3640101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f3645899700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3640100610 0x7f3640101740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45736/0 (socket says 172.21.15.102:45736) 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f3645899700 1 -- 172.21.15.102:0/3454619047 learned_addr learned my addr 172.21.15.102:0/3454619047 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f364609a700 1 --2- 172.21.15.102:0/3454619047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36400ff350 0x7f36401030f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f3645899700 1 -- 172.21.15.102:0/3454619047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36400ff350 msgr2=0x7f36401030f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f3645899700 1 --2- 172.21.15.102:0/3454619047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36400ff350 0x7f36401030f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f3645899700 1 -- 172.21.15.102:0/3454619047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36380095f0 con 0x7f3640100610 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.060+0000 7f3645899700 1 --2- 172.21.15.102:0/3454619047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3640100610 0x7f3640101740 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f363000b4b0 tx=0x7f363000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.061+0000 7f36377fe700 1 -- 172.21.15.102:0/3454619047 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3630011800 con 0x7f3640100610 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.061+0000 7f36377fe700 1 -- 172.21.15.102:0/3454619047 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3630011e10 con 0x7f3640100610 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.061+0000 7f36377fe700 1 -- 172.21.15.102:0/3454619047 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f363000f2e0 con 0x7f3640100610 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.061+0000 7f36482fe700 1 -- 172.21.15.102:0/3454619047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36401020a0 con 0x7f3640100610 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.061+0000 7f36482fe700 1 -- 172.21.15.102:0/3454619047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36401025a0 con 0x7f3640100610 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.062+0000 7f36482fe700 1 -- 172.21.15.102:0/3454619047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3640107c40 con 0x7f3640100610 2024-03-26T18:59:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.062+0000 7f36377fe700 1 -- 172.21.15.102:0/3454619047 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3630010380 con 0x7f3640100610 2024-03-26T18:59:23.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.062+0000 7f36377fe700 1 --2- 172.21.15.102:0/3454619047 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f362c06e460 0x7f362c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:23.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.063+0000 7f36377fe700 1 -- 172.21.15.102:0/3454619047 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f363008e030 con 0x7f3640100610 2024-03-26T18:59:23.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.063+0000 7f364609a700 1 --2- 172.21.15.102:0/3454619047 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f362c06e460 0x7f362c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:23.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.063+0000 7f364609a700 1 --2- 172.21.15.102:0/3454619047 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f362c06e460 0x7f362c070910 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f36380059d0 tx=0x7f3638005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:23.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.067+0000 7f36377fe700 1 -- 172.21.15.102:0/3454619047 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f363005ad60 con 0x7f3640100610 2024-03-26T18:59:23.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.349+0000 7f36482fe700 1 -- 172.21.15.102:0/3454619047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3640107e00 con 0x7f3640100610 2024-03-26T18:59:23.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.349+0000 7f36377fe700 1 -- 172.21.15.102:0/3454619047 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+907 (secure 0 0 0) 0x7f363005a8f0 con 0x7f3640100610 2024-03-26T18:59:23.343 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_WARN Degraded data redundancy: 42/354 objects degraded (11.864%), 13 pgs degraded 2024-03-26T18:59:23.343 INFO:teuthology.orchestra.run.smithi102.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 42/354 objects degraded (11.864%), 13 pgs degraded 2024-03-26T18:59:23.343 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1 is active+undersized+degraded, acting [1,0] 2024-03-26T18:59:23.343 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.3 is active+undersized+degraded, acting [5,1] 2024-03-26T18:59:23.343 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.7 is active+undersized+degraded, acting [3,4] 2024-03-26T18:59:23.343 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.c is active+undersized+degraded, acting [3,0] 2024-03-26T18:59:23.343 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-03-26T18:59:23.344 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.e is active+undersized+degraded, acting [0,3] 2024-03-26T18:59:23.344 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-03-26T18:59:23.344 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-03-26T18:59:23.344 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.16 is active+undersized+degraded, acting [5,3] 2024-03-26T18:59:23.344 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.17 is active+undersized+degraded, acting [5,1] 2024-03-26T18:59:23.344 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.19 is active+undersized+degraded, acting [0,4] 2024-03-26T18:59:23.344 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1c is active+undersized+degraded, acting [4,5] 2024-03-26T18:59:23.344 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-03-26T18:59:23.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.351+0000 7f36482fe700 1 -- 172.21.15.102:0/3454619047 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f362c06e460 msgr2=0x7f362c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:23.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.351+0000 7f36482fe700 1 --2- 172.21.15.102:0/3454619047 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f362c06e460 0x7f362c070910 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f36380059d0 tx=0x7f3638005960 comp rx=0 tx=0).stop 2024-03-26T18:59:23.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.351+0000 7f36482fe700 1 -- 172.21.15.102:0/3454619047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3640100610 msgr2=0x7f3640101740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:23.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.351+0000 7f36482fe700 1 --2- 172.21.15.102:0/3454619047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3640100610 0x7f3640101740 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f363000b4b0 tx=0x7f363000b870 comp rx=0 tx=0).stop 2024-03-26T18:59:23.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.351+0000 7f36482fe700 1 -- 172.21.15.102:0/3454619047 shutdown_connections 2024-03-26T18:59:23.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.351+0000 7f36482fe700 1 --2- 172.21.15.102:0/3454619047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36400ff350 0x7f36401030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:23.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.351+0000 7f36482fe700 1 --2- 172.21.15.102:0/3454619047 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f362c06e460 0x7f362c070910 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:23.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.351+0000 7f36482fe700 1 --2- 172.21.15.102:0/3454619047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3640100610 0x7f3640101740 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:23.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.351+0000 7f36482fe700 1 -- 172.21.15.102:0/3454619047 >> 172.21.15.102:0/3454619047 conn(0x7f36400f9db0 msgr2=0x7f36400fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:23.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.351+0000 7f36482fe700 1 -- 172.21.15.102:0/3454619047 shutdown_connections 2024-03-26T18:59:23.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:23.351+0000 7f36482fe700 1 -- 172.21.15.102:0/3454619047 wait complete. 2024-03-26T18:59:23.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/994522450' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:59:23.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: pgmap v101: 65 pgs: 8 active+undersized+degraded, 9 active+undersized, 48 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 26/354 objects degraded (7.345%) 2024-03-26T18:59:23.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:23 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3454619047' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:59:23.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/994522450' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:59:23.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: pgmap v101: 65 pgs: 8 active+undersized+degraded, 9 active+undersized, 48 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 26/354 objects degraded (7.345%) 2024-03-26T18:59:23.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:23 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3454619047' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:59:24.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:24 smithi190 ceph-mon[65689]: Health check update: Degraded data redundancy: 26/354 objects degraded (7.345%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T18:59:24.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:24 smithi102 ceph-mon[78322]: Health check update: Degraded data redundancy: 26/354 objects degraded (7.345%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T18:59:25.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:25 smithi190 ceph-mon[65689]: pgmap v102: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:25.852 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:25 smithi102 ceph-mon[78322]: pgmap v102: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:26.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:26 smithi102 ceph-mon[78322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/354 objects degraded (7.345%), 8 pgs degraded) 2024-03-26T18:59:26.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:26 smithi102 ceph-mon[78322]: Cluster is now healthy 2024-03-26T18:59:26.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:26 smithi190 ceph-mon[65689]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/354 objects degraded (7.345%), 8 pgs degraded) 2024-03-26T18:59:26.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:26 smithi190 ceph-mon[65689]: Cluster is now healthy 2024-03-26T18:59:27.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:27 smithi190 ceph-mon[65689]: pgmap v103: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:27.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:27 smithi102 ceph-mon[78322]: pgmap v103: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:30.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:30 smithi102 ceph-mon[78322]: pgmap v104: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:30.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:30 smithi190 ceph-mon[65689]: pgmap v104: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:32.545 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:32 smithi190 ceph-mon[65689]: pgmap v105: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:32.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:32 smithi102 ceph-mon[78322]: pgmap v105: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:33.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:33 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:33.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:33 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:34.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:34 smithi102 ceph-mon[78322]: pgmap v106: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:34.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:34 smithi190 ceph-mon[65689]: pgmap v106: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:36.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:36 smithi102 ceph-mon[78322]: pgmap v107: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:36.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:36 smithi190 ceph-mon[65689]: pgmap v107: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:38.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:38 smithi190 ceph-mon[65689]: pgmap v108: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:38.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:38.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:59:38.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T18:59:38.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:38 smithi102 ceph-mon[78322]: pgmap v108: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:38.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:38.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:59:38.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T18:59:39.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:39 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T18:59:39.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:39 smithi190 ceph-mon[65689]: Upgrade: osd.3 is safe to restart 2024-03-26T18:59:39.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:39 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T18:59:39.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:39 smithi102 ceph-mon[78322]: Upgrade: osd.3 is safe to restart 2024-03-26T18:59:40.562 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:40 smithi190 ceph-mon[65689]: pgmap v109: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:40.562 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:40 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:40.562 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:40 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T18:59:40.562 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:40 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:59:40.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:40 smithi102 ceph-mon[78322]: pgmap v109: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:40.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:40 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:40.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:40 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T18:59:40.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:40 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:59:41.413 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:41 smithi190 ceph-mon[65689]: Upgrade: Updating osd.3 2024-03-26T18:59:41.413 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:41 smithi190 ceph-mon[65689]: Deploying daemon osd.3 on smithi190 2024-03-26T18:59:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:41 smithi102 ceph-mon[78322]: Upgrade: Updating osd.3 2024-03-26T18:59:41.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:41 smithi102 ceph-mon[78322]: Deploying daemon osd.3 on smithi190 2024-03-26T18:59:42.451 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:42 smithi190 ceph-mon[65689]: pgmap v110: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:42 smithi102 ceph-mon[78322]: pgmap v110: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:43.417 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:43 smithi190 systemd[1]: Stopping Ceph osd.3 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:59:43.781 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:43 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[41121]: 2024-03-26T18:59:43.423+0000 7f0d30c15700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T18:59:43.781 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:43 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[41121]: 2024-03-26T18:59:43.423+0000 7f0d30c15700 -1 osd.3 65 *** Got signal Terminated *** 2024-03-26T18:59:43.781 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:43 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[41121]: 2024-03-26T18:59:43.423+0000 7f0d30c15700 -1 osd.3 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T18:59:44.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:44 smithi102 ceph-mon[78322]: pgmap v111: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:44.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:44 smithi102 ceph-mon[78322]: osd.3 marked itself down and dead 2024-03-26T18:59:44.746 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:44 smithi190 ceph-mon[65689]: pgmap v111: 65 pgs: 65 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:44.746 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:44 smithi190 ceph-mon[65689]: osd.3 marked itself down and dead 2024-03-26T18:59:45.030 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:44 smithi190 podman[70525]: 2024-03-26 18:59:44.753127299 +0000 UTC m=+1.434358671 container died 3cd9522e1208513f4be329172ae7e64898ae3d85a940f9f75fb9151524e4ea4d (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20231212) 2024-03-26T18:59:45.296 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:45 smithi190 podman[70525]: 2024-03-26 18:59:45.224101084 +0000 UTC m=+1.905332494 container cleanup 3cd9522e1208513f4be329172ae7e64898ae3d85a940f9f75fb9151524e4ea4d (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD) 2024-03-26T18:59:45.296 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:45 smithi190 bash[70525]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3 2024-03-26T18:59:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:45 smithi102 ceph-mon[78322]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T18:59:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:45 smithi102 ceph-mon[78322]: osdmap e66: 6 total, 5 up, 6 in 2024-03-26T18:59:45.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:45 smithi190 ceph-mon[65689]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T18:59:45.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:45 smithi190 ceph-mon[65689]: osdmap e66: 6 total, 5 up, 6 in 2024-03-26T18:59:45.781 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:45 smithi190 podman[70543]: 2024-03-26 18:59:45.500378993 +0000 UTC m=+0.741146866 container remove 3cd9522e1208513f4be329172ae7e64898ae3d85a940f9f75fb9151524e4ea4d (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux ) 2024-03-26T18:59:46.154 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:45 smithi190 podman[70634]: 2024-03-26 18:59:45.749152704 +0000 UTC m=+0.017698968 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:59:46.154 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:45 smithi190 podman[70634]: 2024-03-26 18:59:45.859527357 +0000 UTC m=+0.128073604 container create f6f32787c00ca913172e8c93e5b6e1031af3fabaf91c497bcaddc6cdd70478df (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T18:59:46.155 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:46 smithi190 podman[70634]: 2024-03-26 18:59:46.061892491 +0000 UTC m=+0.330438738 container init f6f32787c00ca913172e8c93e5b6e1031af3fabaf91c497bcaddc6cdd70478df (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T18:59:46.155 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:46 smithi190 podman[70634]: 2024-03-26 18:59:46.065332336 +0000 UTC m=+0.333878589 container start f6f32787c00ca913172e8c93e5b6e1031af3fabaf91c497bcaddc6cdd70478df (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T18:59:46.155 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:46 smithi190 podman[70634]: 2024-03-26 18:59:46.161714846 +0000 UTC m=+0.430261112 container attach f6f32787c00ca913172e8c93e5b6e1031af3fabaf91c497bcaddc6cdd70478df (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T18:59:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:46 smithi102 ceph-mon[78322]: pgmap v113: 65 pgs: 9 peering, 14 stale+active+clean, 42 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:46.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:46 smithi102 ceph-mon[78322]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-26T18:59:46.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:46 smithi102 ceph-mon[78322]: osdmap e67: 6 total, 5 up, 6 in 2024-03-26T18:59:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:46 smithi190 ceph-mon[65689]: pgmap v113: 65 pgs: 9 peering, 14 stale+active+clean, 42 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:46 smithi190 ceph-mon[65689]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-26T18:59:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:46 smithi190 ceph-mon[65689]: osdmap e67: 6 total, 5 up, 6 in 2024-03-26T18:59:46.531 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:46 smithi190 podman[70634]: 2024-03-26 18:59:46.352626903 +0000 UTC m=+0.621173152 container died f6f32787c00ca913172e8c93e5b6e1031af3fabaf91c497bcaddc6cdd70478df (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T18:59:47.281 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:46 smithi190 podman[70660]: 2024-03-26 18:59:46.998949714 +0000 UTC m=+0.640362971 container remove f6f32787c00ca913172e8c93e5b6e1031af3fabaf91c497bcaddc6cdd70478df (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T18:59:47.281 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:47 smithi190 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-26T18:59:47.281 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:47 smithi190 systemd[1]: Stopped Ceph osd.3 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:59:47.281 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:47 smithi190 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.3.service: Consumed 1min 9.690s CPU time. 2024-03-26T18:59:47.281 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:47 smithi190 systemd[1]: Starting Ceph osd.3 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T18:59:47.693 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:47 smithi190 podman[70758]: 2024-03-26 18:59:47.343654428 +0000 UTC m=+0.018573212 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:59:47.693 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:47 smithi190 podman[70758]: 2024-03-26 18:59:47.441754327 +0000 UTC m=+0.116673090 container create 2772c9981c63b6c9b7b783e466c39bfe7cea34fce623ce829f4eec485b7cc191 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5) 2024-03-26T18:59:47.693 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:47 smithi190 podman[70758]: 2024-03-26 18:59:47.652514263 +0000 UTC m=+0.327433044 container init 2772c9981c63b6c9b7b783e466c39bfe7cea34fce623ce829f4eec485b7cc191 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-26T18:59:47.693 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:47 smithi190 podman[70758]: 2024-03-26 18:59:47.655658612 +0000 UTC m=+0.330577374 container start 2772c9981c63b6c9b7b783e466c39bfe7cea34fce623ce829f4eec485b7cc191 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-26T18:59:48.030 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:47 smithi190 podman[70758]: 2024-03-26 18:59:47.735872297 +0000 UTC m=+0.410791069 container attach 2772c9981c63b6c9b7b783e466c39bfe7cea34fce623ce829f4eec485b7cc191 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T18:59:48.431 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:48 smithi190 ceph-mon[65689]: pgmap v115: 65 pgs: 9 peering, 14 stale+active+clean, 42 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate[70774]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate[70774]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 bash[70758]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 bash[70758]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 bash[70758]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 bash[70758]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 bash[70758]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 bash[70758]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 bash[70758]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate[70774]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate[70774]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate[70774]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate[70774]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T18:59:48.432 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate[70774]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-26T18:59:48.433 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 podman[70758]: 2024-03-26 18:59:48.125605842 +0000 UTC m=+0.800524615 container died 2772c9981c63b6c9b7b783e466c39bfe7cea34fce623ce829f4eec485b7cc191 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T18:59:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:48 smithi102 ceph-mon[78322]: pgmap v115: 65 pgs: 9 peering, 14 stale+active+clean, 42 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-03-26T18:59:49.030 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 podman[70895]: 2024-03-26 18:59:48.739873396 +0000 UTC m=+0.606845934 container remove 2772c9981c63b6c9b7b783e466c39bfe7cea34fce623ce829f4eec485b7cc191 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-activate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T18:59:49.030 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 podman[70956]: 2024-03-26 18:59:48.940521095 +0000 UTC m=+0.101272244 container create 03955331c9621e6f8ad50bf62790749950242db6f6a2affac1561e4fe89d0f57 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-26T18:59:49.030 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:48 smithi190 podman[70956]: 2024-03-26 18:59:48.856896612 +0000 UTC m=+0.017647774 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T18:59:49.530 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:49 smithi190 podman[70956]: 2024-03-26 18:59:49.151274681 +0000 UTC m=+0.312025830 container init 03955331c9621e6f8ad50bf62790749950242db6f6a2affac1561e4fe89d0f57 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-26T18:59:49.531 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:49 smithi190 podman[70956]: 2024-03-26 18:59:49.154212931 +0000 UTC m=+0.314964081 container start 03955331c9621e6f8ad50bf62790749950242db6f6a2affac1561e4fe89d0f57 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.vendor=CentOS) 2024-03-26T18:59:49.531 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:49 smithi190 ceph-osd[70976]: -- 172.21.15.190:0/210592467 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555bb5e69680 con 0x555bb5e33000 2024-03-26T18:59:49.531 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:49 smithi190 bash[70956]: 03955331c9621e6f8ad50bf62790749950242db6f6a2affac1561e4fe89d0f57 2024-03-26T18:59:49.531 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:49 smithi190 systemd[1]: Started Ceph osd.3 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T18:59:50.030 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:49 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[70972]: 2024-03-26T18:59:49.705+0000 7fd81e1ef7c0 -1 Falling back to public interface 2024-03-26T18:59:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:50 smithi190 ceph-mon[65689]: pgmap v116: 65 pgs: 14 active+undersized+degraded, 9 peering, 15 active+undersized, 27 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 53/354 objects degraded (14.972%) 2024-03-26T18:59:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:50 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:50 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:50.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:50 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:59:50.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:50 smithi190 ceph-mon[65689]: Health check failed: Degraded data redundancy: 53/354 objects degraded (14.972%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T18:59:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:50 smithi102 ceph-mon[78322]: pgmap v116: 65 pgs: 14 active+undersized+degraded, 9 peering, 15 active+undersized, 27 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 53/354 objects degraded (14.972%) 2024-03-26T18:59:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:50 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:50 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:50 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:59:50.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:50 smithi102 ceph-mon[78322]: Health check failed: Degraded data redundancy: 53/354 objects degraded (14.972%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T18:59:51.624 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:51 smithi190 ceph-mon[65689]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-26T18:59:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:51 smithi102 ceph-mon[78322]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-26T18:59:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:52 smithi102 ceph-mon[78322]: pgmap v117: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 74/354 objects degraded (20.904%) 2024-03-26T18:59:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:52.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:59:52.730 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:52 smithi190 ceph-mon[65689]: pgmap v117: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 74/354 objects degraded (20.904%) 2024-03-26T18:59:52.731 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:52.731 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:52.731 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:52.731 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T18:59:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.477+0000 7f8fba409700 1 -- 172.21.15.102:0/3055259034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fb4103570 msgr2=0x7f8fb4105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.477+0000 7f8fba409700 1 --2- 172.21.15.102:0/3055259034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fb4103570 0x7f8fb4105950 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009910 tx=0x7f8fa8009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.477+0000 7f8fba409700 1 -- 172.21.15.102:0/3055259034 shutdown_connections 2024-03-26T18:59:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.477+0000 7f8fba409700 1 --2- 172.21.15.102:0/3055259034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fb4103570 0x7f8fb4105950 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.477+0000 7f8fba409700 1 --2- 172.21.15.102:0/3055259034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fb4100c50 0x7f8fb4103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.477+0000 7f8fba409700 1 -- 172.21.15.102:0/3055259034 >> 172.21.15.102:0/3055259034 conn(0x7f8fb40fa610 msgr2=0x7f8fb40fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.477+0000 7f8fba409700 1 -- 172.21.15.102:0/3055259034 shutdown_connections 2024-03-26T18:59:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.477+0000 7f8fba409700 1 -- 172.21.15.102:0/3055259034 wait complete. 2024-03-26T18:59:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.477+0000 7f8fba409700 1 Processor -- start 2024-03-26T18:59:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.477+0000 7f8fba409700 1 -- start start 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fba409700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fb4100c50 0x7f8fb4195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fba409700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fb4103570 0x7f8fb4195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fba409700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb41964c0 con 0x7f8fb4100c50 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fba409700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb4196600 con 0x7f8fb4103570 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fb3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fb4100c50 0x7f8fb4195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fb3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fb4100c50 0x7f8fb4195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45646/0 (socket says 172.21.15.102:45646) 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fb3fff700 1 -- 172.21.15.102:0/2129624843 learned_addr learned my addr 172.21.15.102:0/2129624843 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fb37fe700 1 --2- 172.21.15.102:0/2129624843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fb4103570 0x7f8fb4195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fb3fff700 1 -- 172.21.15.102:0/2129624843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fb4103570 msgr2=0x7f8fb4195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fb3fff700 1 --2- 172.21.15.102:0/2129624843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fb4103570 0x7f8fb4195ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fb3fff700 1 -- 172.21.15.102:0/2129624843 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa80095f0 con 0x7f8fb4100c50 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.478+0000 7f8fb3fff700 1 --2- 172.21.15.102:0/2129624843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fb4100c50 0x7f8fb4195960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400e920 tx=0x7f8fa400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.479+0000 7f8fb17fa700 1 -- 172.21.15.102:0/2129624843 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fa400c9d0 con 0x7f8fb4100c50 2024-03-26T18:59:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.479+0000 7f8fba409700 1 -- 172.21.15.102:0/2129624843 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb419b0b0 con 0x7f8fb4100c50 2024-03-26T18:59:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.479+0000 7f8fb17fa700 1 -- 172.21.15.102:0/2129624843 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8fa4010430 con 0x7f8fb4100c50 2024-03-26T18:59:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.479+0000 7f8fb17fa700 1 -- 172.21.15.102:0/2129624843 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fa4018650 con 0x7f8fb4100c50 2024-03-26T18:59:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.479+0000 7f8fba409700 1 -- 172.21.15.102:0/2129624843 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb419b600 con 0x7f8fb4100c50 2024-03-26T18:59:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.480+0000 7f8fba409700 1 -- 172.21.15.102:0/2129624843 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb418fb80 con 0x7f8fb4100c50 2024-03-26T18:59:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.480+0000 7f8fb17fa700 1 -- 172.21.15.102:0/2129624843 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8fa40105a0 con 0x7f8fb4100c50 2024-03-26T18:59:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.481+0000 7f8fb17fa700 1 --2- 172.21.15.102:0/2129624843 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f9c06e460 0x7f8f9c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.481+0000 7f8fb17fa700 1 -- 172.21.15.102:0/2129624843 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8fa4014070 con 0x7f8fb4100c50 2024-03-26T18:59:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.481+0000 7f8fb37fe700 1 --2- 172.21.15.102:0/2129624843 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f9c06e460 0x7f8f9c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.481+0000 7f8fb37fe700 1 --2- 172.21.15.102:0/2129624843 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f9c06e460 0x7f8f9c070910 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009fd0 tx=0x7f8fa8005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.486+0000 7f8fb17fa700 1 -- 172.21.15.102:0/2129624843 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8fa4053740 con 0x7f8fb4100c50 2024-03-26T18:59:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.703+0000 7f8fba409700 1 -- 172.21.15.102:0/2129624843 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fb4061190 con 0x7f8f9c06e460 2024-03-26T18:59:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.704+0000 7f8fb17fa700 1 -- 172.21.15.102:0/2129624843 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f8fb4061190 con 0x7f8f9c06e460 2024-03-26T18:59:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.706+0000 7f8fba409700 1 -- 172.21.15.102:0/2129624843 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f9c06e460 msgr2=0x7f8f9c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.706+0000 7f8fba409700 1 --2- 172.21.15.102:0/2129624843 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f9c06e460 0x7f8f9c070910 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009fd0 tx=0x7f8fa8005940 comp rx=0 tx=0).stop 2024-03-26T18:59:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.706+0000 7f8fba409700 1 -- 172.21.15.102:0/2129624843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fb4100c50 msgr2=0x7f8fb4195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.706+0000 7f8fba409700 1 --2- 172.21.15.102:0/2129624843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fb4100c50 0x7f8fb4195960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400e920 tx=0x7f8fa400ec30 comp rx=0 tx=0).stop 2024-03-26T18:59:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.707+0000 7f8fba409700 1 -- 172.21.15.102:0/2129624843 shutdown_connections 2024-03-26T18:59:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.707+0000 7f8fba409700 1 --2- 172.21.15.102:0/2129624843 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f9c06e460 0x7f8f9c070910 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.707+0000 7f8fba409700 1 --2- 172.21.15.102:0/2129624843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fb4100c50 0x7f8fb4195960 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.707+0000 7f8fba409700 1 --2- 172.21.15.102:0/2129624843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fb4103570 0x7f8fb4195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.707+0000 7f8fba409700 1 -- 172.21.15.102:0/2129624843 >> 172.21.15.102:0/2129624843 conn(0x7f8fb40fa610 msgr2=0x7f8fb40fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.707+0000 7f8fba409700 1 -- 172.21.15.102:0/2129624843 shutdown_connections 2024-03-26T18:59:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.707+0000 7f8fba409700 1 -- 172.21.15.102:0/2129624843 wait complete. 2024-03-26T18:59:53.710 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T18:59:53.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.811+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/17088245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3d0103500 msgr2=0x7fe3d01058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:53.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.811+0000 7fe3d6ef8700 1 --2- 172.21.15.102:0/17088245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3d0103500 0x7fe3d01058e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009910 tx=0x7fe3c4009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:53.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.811+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/17088245 shutdown_connections 2024-03-26T18:59:53.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.811+0000 7fe3d6ef8700 1 --2- 172.21.15.102:0/17088245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3d0103500 0x7fe3d01058e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:53.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.811+0000 7fe3d6ef8700 1 --2- 172.21.15.102:0/17088245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3d0100be0 0x7fe3d0102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:53.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.811+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/17088245 >> 172.21.15.102:0/17088245 conn(0x7fe3d00fa5e0 msgr2=0x7fe3d00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:53.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.811+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/17088245 shutdown_connections 2024-03-26T18:59:53.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.811+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/17088245 wait complete. 2024-03-26T18:59:53.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3d6ef8700 1 Processor -- start 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3d6ef8700 1 -- start start 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3d6ef8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3d0100be0 0x7fe3d0100540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3d6ef8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3d0103500 0x7fe3d00feb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3d6ef8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d00ff0d0 con 0x7fe3d0103500 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3d6ef8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d00ff210 con 0x7fe3d0100be0 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3cffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3d0103500 0x7fe3d00feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3cffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3d0103500 0x7fe3d00feb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45666/0 (socket says 172.21.15.102:45666) 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3cffff700 1 -- 172.21.15.102:0/2342788822 learned_addr learned my addr 172.21.15.102:0/2342788822 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3d4c94700 1 --2- 172.21.15.102:0/2342788822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3d0100be0 0x7fe3d0100540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3cffff700 1 -- 172.21.15.102:0/2342788822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3d0100be0 msgr2=0x7fe3d0100540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3cffff700 1 --2- 172.21.15.102:0/2342788822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3d0100be0 0x7fe3d0100540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3cffff700 1 -- 172.21.15.102:0/2342788822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c40095f0 con 0x7fe3d0103500 2024-03-26T18:59:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.812+0000 7fe3cffff700 1 --2- 172.21.15.102:0/2342788822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3d0103500 0x7fe3d00feb90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4006010 tx=0x7fe3c400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:53.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.813+0000 7fe3cdffb700 1 -- 172.21.15.102:0/2342788822 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3c401d070 con 0x7fe3d0103500 2024-03-26T18:59:53.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.813+0000 7fe3cdffb700 1 -- 172.21.15.102:0/2342788822 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe3c400f460 con 0x7fe3d0103500 2024-03-26T18:59:53.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.813+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/2342788822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d00ff490 con 0x7fe3d0103500 2024-03-26T18:59:53.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.813+0000 7fe3cdffb700 1 -- 172.21.15.102:0/2342788822 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3c4021640 con 0x7fe3d0103500 2024-03-26T18:59:53.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.813+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/2342788822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d00ff980 con 0x7fe3d0103500 2024-03-26T18:59:53.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.814+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/2342788822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d018fb70 con 0x7fe3d0103500 2024-03-26T18:59:53.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.815+0000 7fe3cdffb700 1 -- 172.21.15.102:0/2342788822 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe3c400fab0 con 0x7fe3d0103500 2024-03-26T18:59:53.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.815+0000 7fe3cdffb700 1 --2- 172.21.15.102:0/2342788822 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe3b806e460 0x7fe3b8070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:53.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.815+0000 7fe3cdffb700 1 -- 172.21.15.102:0/2342788822 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe3c408f600 con 0x7fe3d0103500 2024-03-26T18:59:53.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.815+0000 7fe3d4c94700 1 --2- 172.21.15.102:0/2342788822 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe3b806e460 0x7fe3b8070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:53.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.815+0000 7fe3d4c94700 1 --2- 172.21.15.102:0/2342788822 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe3b806e460 0x7fe3b8070910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009550 tx=0x7fe3c80091b0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:53.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:53.820+0000 7fe3cdffb700 1 -- 172.21.15.102:0/2342788822 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe3c405c3b0 con 0x7fe3d0103500 2024-03-26T18:59:54.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.035+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/2342788822 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3d0061190 con 0x7fe3b806e460 2024-03-26T18:59:54.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.037+0000 7fe3cdffb700 1 -- 172.21.15.102:0/2342788822 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fe3d0061190 con 0x7fe3b806e460 2024-03-26T18:59:54.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.039+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/2342788822 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe3b806e460 msgr2=0x7fe3b8070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.039+0000 7fe3d6ef8700 1 --2- 172.21.15.102:0/2342788822 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe3b806e460 0x7fe3b8070910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009550 tx=0x7fe3c80091b0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.039+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/2342788822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3d0103500 msgr2=0x7fe3d00feb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.039+0000 7fe3d6ef8700 1 --2- 172.21.15.102:0/2342788822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3d0103500 0x7fe3d00feb90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4006010 tx=0x7fe3c400bd70 comp rx=0 tx=0).stop 2024-03-26T18:59:54.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.039+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/2342788822 shutdown_connections 2024-03-26T18:59:54.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.039+0000 7fe3d6ef8700 1 --2- 172.21.15.102:0/2342788822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3d0100be0 0x7fe3d0100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.039+0000 7fe3d6ef8700 1 --2- 172.21.15.102:0/2342788822 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe3b806e460 0x7fe3b8070910 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.039+0000 7fe3d6ef8700 1 --2- 172.21.15.102:0/2342788822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3d0103500 0x7fe3d00feb90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.039+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/2342788822 >> 172.21.15.102:0/2342788822 conn(0x7fe3d00fa5e0 msgr2=0x7fe3d00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:54.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.039+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/2342788822 shutdown_connections 2024-03-26T18:59:54.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.039+0000 7fe3d6ef8700 1 -- 172.21.15.102:0/2342788822 wait complete. 2024-03-26T18:59:54.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.142+0000 7f4218d18700 1 -- 172.21.15.102:0/928238064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4214102130 msgr2=0x7f42140fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.142+0000 7f4218d18700 1 --2- 172.21.15.102:0/928238064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4214102130 0x7f42140fe260 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4208009910 tx=0x7f4208009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:54.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.142+0000 7f4218d18700 1 -- 172.21.15.102:0/928238064 shutdown_connections 2024-03-26T18:59:54.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.142+0000 7f4218d18700 1 --2- 172.21.15.102:0/928238064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4214102130 0x7f42140fe260 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.142+0000 7f4218d18700 1 --2- 172.21.15.102:0/928238064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42141017e0 0x7f4214101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.142+0000 7f4218d18700 1 -- 172.21.15.102:0/928238064 >> 172.21.15.102:0/928238064 conn(0x7f42140f9e50 msgr2=0x7f42140fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:54.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.143+0000 7f4218d18700 1 -- 172.21.15.102:0/928238064 shutdown_connections 2024-03-26T18:59:54.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.143+0000 7f4218d18700 1 -- 172.21.15.102:0/928238064 wait complete. 2024-03-26T18:59:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.143+0000 7f4218d18700 1 Processor -- start 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.143+0000 7f4218d18700 1 -- start start 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f4218d18700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42141017e0 0x7f4214195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f4218d18700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4214102130 0x7f4214195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f4218d18700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42141964b0 con 0x7f42141017e0 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f4218d18700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42141965f0 con 0x7f4214102130 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f421259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42141017e0 0x7f4214195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f421259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42141017e0 0x7f4214195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45686/0 (socket says 172.21.15.102:45686) 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f421259c700 1 -- 172.21.15.102:0/2204589032 learned_addr learned my addr 172.21.15.102:0/2204589032 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f4211d9b700 1 --2- 172.21.15.102:0/2204589032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4214102130 0x7f4214195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f421259c700 1 -- 172.21.15.102:0/2204589032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4214102130 msgr2=0x7f4214195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f421259c700 1 --2- 172.21.15.102:0/2204589032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4214102130 0x7f4214195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f421259c700 1 -- 172.21.15.102:0/2204589032 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42080095f0 con 0x7f42141017e0 2024-03-26T18:59:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f421259c700 1 --2- 172.21.15.102:0/2204589032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42141017e0 0x7f4214195950 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f420400d900 tx=0x7f420400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f42037fe700 1 -- 172.21.15.102:0/2204589032 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42040041f0 con 0x7f42141017e0 2024-03-26T18:59:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f42037fe700 1 -- 172.21.15.102:0/2204589032 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4204004ce0 con 0x7f42141017e0 2024-03-26T18:59:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f4218d18700 1 -- 172.21.15.102:0/2204589032 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421419b0a0 con 0x7f42141017e0 2024-03-26T18:59:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f42037fe700 1 -- 172.21.15.102:0/2204589032 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f420400b750 con 0x7f42141017e0 2024-03-26T18:59:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.144+0000 7f4218d18700 1 -- 172.21.15.102:0/2204589032 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421419b5f0 con 0x7f42141017e0 2024-03-26T18:59:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.145+0000 7f4218d18700 1 -- 172.21.15.102:0/2204589032 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4214108c50 con 0x7f42141017e0 2024-03-26T18:59:54.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.146+0000 7f42037fe700 1 -- 172.21.15.102:0/2204589032 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4204004830 con 0x7f42141017e0 2024-03-26T18:59:54.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.146+0000 7f42037fe700 1 --2- 172.21.15.102:0/2204589032 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f41fc06e460 0x7f41fc070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:54.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.146+0000 7f42037fe700 1 -- 172.21.15.102:0/2204589032 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f420408e340 con 0x7f42141017e0 2024-03-26T18:59:54.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.146+0000 7f4211d9b700 1 --2- 172.21.15.102:0/2204589032 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f41fc06e460 0x7f41fc070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:54.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.147+0000 7f4211d9b700 1 --2- 172.21.15.102:0/2204589032 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f41fc06e460 0x7f41fc070910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4208006010 tx=0x7f4208005ec0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:54.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.151+0000 7f42037fe700 1 -- 172.21.15.102:0/2204589032 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f420405b070 con 0x7f42141017e0 2024-03-26T18:59:54.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.363+0000 7f4218d18700 1 -- 172.21.15.102:0/2204589032 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4214105ed0 con 0x7f41fc06e460 2024-03-26T18:59:54.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.373+0000 7f42037fe700 1 -- 172.21.15.102:0/2204589032 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f4214105ed0 con 0x7f41fc06e460 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (11m) 35s ago 12m 25.0M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (13m) 35s ago 13m 8354k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (12m) 2s ago 12m 7940k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (2m) 35s ago 13m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (2m) 2s ago 12m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (11m) 35s ago 12m 86.6M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (7m) 35s ago 7m 132M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (7m) 35s ago 7m 16.1M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (7m) 2s ago 7m 16.3M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (7m) 2s ago 7m 120M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (4m) 35s ago 14m 504M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T18:59:54.367 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (3m) 2s ago 11m 450M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T18:59:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (3m) 35s ago 14m 46.2M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T18:59:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (3m) 2s ago 11m 38.2M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T18:59:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (13m) 35s ago 13m 16.3M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T18:59:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (12m) 2s ago 11m 15.5M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T18:59:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (112s) 35s ago 11m 86.0M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T18:59:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (76s) 35s ago 10m 95.7M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T18:59:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (41s) 35s ago 10m 11.7M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T18:59:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (5s) 2s ago 9m 12.3M 2240M 19.0.0-2311-g36051606 497bb87bc5da 03955331c962 2024-03-26T18:59:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (9m) 2s ago 9m 251M 2240M 18.2.0 dc2bc1663786 f2a02dcd6fee 2024-03-26T18:59:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (8m) 2s ago 8m 223M 2240M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T18:59:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (3m) 35s ago 12m 74.3M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T18:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.375+0000 7f4218d18700 1 -- 172.21.15.102:0/2204589032 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f41fc06e460 msgr2=0x7f41fc070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.375+0000 7f4218d18700 1 --2- 172.21.15.102:0/2204589032 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f41fc06e460 0x7f41fc070910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4208006010 tx=0x7f4208005ec0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.375+0000 7f4218d18700 1 -- 172.21.15.102:0/2204589032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42141017e0 msgr2=0x7f4214195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.375+0000 7f4218d18700 1 --2- 172.21.15.102:0/2204589032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42141017e0 0x7f4214195950 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f420400d900 tx=0x7f420400dc10 comp rx=0 tx=0).stop 2024-03-26T18:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.375+0000 7f4218d18700 1 -- 172.21.15.102:0/2204589032 shutdown_connections 2024-03-26T18:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.375+0000 7f4218d18700 1 --2- 172.21.15.102:0/2204589032 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f41fc06e460 0x7f41fc070910 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.375+0000 7f4218d18700 1 --2- 172.21.15.102:0/2204589032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42141017e0 0x7f4214195950 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.375+0000 7f4218d18700 1 --2- 172.21.15.102:0/2204589032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4214102130 0x7f4214195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.375+0000 7f4218d18700 1 -- 172.21.15.102:0/2204589032 >> 172.21.15.102:0/2204589032 conn(0x7f42140f9e50 msgr2=0x7f42141047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.376+0000 7f4218d18700 1 -- 172.21.15.102:0/2204589032 shutdown_connections 2024-03-26T18:59:54.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.376+0000 7f4218d18700 1 -- 172.21.15.102:0/2204589032 wait complete. 2024-03-26T18:59:54.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.479+0000 7f09a648f700 1 -- 172.21.15.102:0/2763064105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a0100170 msgr2=0x7f09a0100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.479+0000 7f09a648f700 1 --2- 172.21.15.102:0/2763064105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a0100170 0x7f09a0100580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0988009910 tx=0x7f0988009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:54.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.479+0000 7f09a648f700 1 -- 172.21.15.102:0/2763064105 shutdown_connections 2024-03-26T18:59:54.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.479+0000 7f09a648f700 1 --2- 172.21.15.102:0/2763064105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09a0101370 0x7f09a01017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.479+0000 7f09a648f700 1 --2- 172.21.15.102:0/2763064105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a0100170 0x7f09a0100580 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.479+0000 7f09a648f700 1 -- 172.21.15.102:0/2763064105 >> 172.21.15.102:0/2763064105 conn(0x7f09a00fb720 msgr2=0x7f09a00fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:54.473 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:54 smithi102 ceph-mon[78322]: pgmap v118: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 74/354 objects degraded (20.904%) 2024-03-26T18:59:54.473 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:54 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:54.473 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:54 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.479+0000 7f09a648f700 1 -- 172.21.15.102:0/2763064105 shutdown_connections 2024-03-26T18:59:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.479+0000 7f09a648f700 1 -- 172.21.15.102:0/2763064105 wait complete. 2024-03-26T18:59:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.480+0000 7f09a648f700 1 Processor -- start 2024-03-26T18:59:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.480+0000 7f09a648f700 1 -- start start 2024-03-26T18:59:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.480+0000 7f09a648f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09a0100170 0x7f09a0195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.480+0000 7f09a648f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a0101370 0x7f09a0195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.480+0000 7f09a648f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a01964f0 con 0x7f09a0101370 2024-03-26T18:59:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.480+0000 7f09a648f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a0196630 con 0x7f09a0100170 2024-03-26T18:59:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.480+0000 7f099f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a0101370 0x7f09a0195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.480+0000 7f099f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a0101370 0x7f09a0195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45704/0 (socket says 172.21.15.102:45704) 2024-03-26T18:59:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.480+0000 7f099f7fe700 1 -- 172.21.15.102:0/2757813972 learned_addr learned my addr 172.21.15.102:0/2757813972 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.480+0000 7f099ffff700 1 --2- 172.21.15.102:0/2757813972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09a0100170 0x7f09a0195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.481+0000 7f099f7fe700 1 -- 172.21.15.102:0/2757813972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09a0100170 msgr2=0x7f09a0195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.481+0000 7f099f7fe700 1 --2- 172.21.15.102:0/2757813972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09a0100170 0x7f09a0195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.481+0000 7f099f7fe700 1 -- 172.21.15.102:0/2757813972 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09880095f0 con 0x7f09a0101370 2024-03-26T18:59:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.481+0000 7f099f7fe700 1 --2- 172.21.15.102:0/2757813972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a0101370 0x7f09a0195ed0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f099000e920 tx=0x7f099000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.481+0000 7f099d7fa700 1 -- 172.21.15.102:0/2757813972 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f099000c9d0 con 0x7f09a0101370 2024-03-26T18:59:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.481+0000 7f099d7fa700 1 -- 172.21.15.102:0/2757813972 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0990010430 con 0x7f09a0101370 2024-03-26T18:59:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.481+0000 7f09a648f700 1 -- 172.21.15.102:0/2757813972 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a019b0e0 con 0x7f09a0101370 2024-03-26T18:59:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.481+0000 7f099d7fa700 1 -- 172.21.15.102:0/2757813972 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0990018650 con 0x7f09a0101370 2024-03-26T18:59:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.481+0000 7f09a648f700 1 -- 172.21.15.102:0/2757813972 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a019b630 con 0x7f09a0101370 2024-03-26T18:59:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.482+0000 7f09a648f700 1 -- 172.21.15.102:0/2757813972 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a0108cc0 con 0x7f09a0101370 2024-03-26T18:59:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.483+0000 7f099d7fa700 1 -- 172.21.15.102:0/2757813972 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f099000cb30 con 0x7f09a0101370 2024-03-26T18:59:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.483+0000 7f099d7fa700 1 --2- 172.21.15.102:0/2757813972 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f098c06e4b0 0x7f098c070960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.483+0000 7f099d7fa700 1 -- 172.21.15.102:0/2757813972 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0990014070 con 0x7f09a0101370 2024-03-26T18:59:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.483+0000 7f099ffff700 1 --2- 172.21.15.102:0/2757813972 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f098c06e4b0 0x7f098c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.484+0000 7f099ffff700 1 --2- 172.21.15.102:0/2757813972 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f098c06e4b0 0x7f098c070960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f098800b6f0 tx=0x7f0988005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.488+0000 7f099d7fa700 1 -- 172.21.15.102:0/2757813972 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0990053740 con 0x7f09a0101370 2024-03-26T18:59:54.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:54 smithi190 ceph-mon[65689]: pgmap v118: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 74/354 objects degraded (20.904%) 2024-03-26T18:59:54.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:54 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:54.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:54 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:54.531 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:54 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[70972]: 2024-03-26T18:59:54.281+0000 7fd81e1ef7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-26T18:59:54.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.771+0000 7f09a648f700 1 -- 172.21.15.102:0/2757813972 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f09a0108e80 con 0x7f09a0101370 2024-03-26T18:59:54.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.771+0000 7f099d7fa700 1 -- 172.21.15.102:0/2757813972 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f099005b3a0 con 0x7f09a0101370 2024-03-26T18:59:54.765 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:59:54.765 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T18:59:54.765 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:59:54.765 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:59:54.765 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T18:59:54.765 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T18:59:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:59:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T18:59:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-03-26T18:59:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 3 2024-03-26T18:59:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:59:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T18:59:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T18:59:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T18:59:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T18:59:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6, 2024-03-26T18:59:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 7 2024-03-26T18:59:54.767 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T18:59:54.767 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:59:54.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.774+0000 7f09a648f700 1 -- 172.21.15.102:0/2757813972 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f098c06e4b0 msgr2=0x7f098c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.774+0000 7f09a648f700 1 --2- 172.21.15.102:0/2757813972 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f098c06e4b0 0x7f098c070960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f098800b6f0 tx=0x7f0988005940 comp rx=0 tx=0).stop 2024-03-26T18:59:54.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.774+0000 7f09a648f700 1 -- 172.21.15.102:0/2757813972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a0101370 msgr2=0x7f09a0195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.774+0000 7f09a648f700 1 --2- 172.21.15.102:0/2757813972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a0101370 0x7f09a0195ed0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f099000e920 tx=0x7f099000ec30 comp rx=0 tx=0).stop 2024-03-26T18:59:54.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.774+0000 7f09a648f700 1 -- 172.21.15.102:0/2757813972 shutdown_connections 2024-03-26T18:59:54.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.774+0000 7f09a648f700 1 --2- 172.21.15.102:0/2757813972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09a0100170 0x7f09a0195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.774+0000 7f09a648f700 1 --2- 172.21.15.102:0/2757813972 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f098c06e4b0 0x7f098c070960 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.774+0000 7f09a648f700 1 --2- 172.21.15.102:0/2757813972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a0101370 0x7f09a0195ed0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.774+0000 7f09a648f700 1 -- 172.21.15.102:0/2757813972 >> 172.21.15.102:0/2757813972 conn(0x7f09a00fb720 msgr2=0x7f09a01045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:54.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.774+0000 7f09a648f700 1 -- 172.21.15.102:0/2757813972 shutdown_connections 2024-03-26T18:59:54.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.774+0000 7f09a648f700 1 -- 172.21.15.102:0/2757813972 wait complete. 2024-03-26T18:59:54.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.878+0000 7ff92aff1700 1 -- 172.21.15.102:0/3433675494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff924100260 msgr2=0x7ff9241006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.878+0000 7ff92aff1700 1 --2- 172.21.15.102:0/3433675494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff924100260 0x7ff9241006b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff914009910 tx=0x7ff914009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.878+0000 7ff92aff1700 1 -- 172.21.15.102:0/3433675494 shutdown_connections 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.878+0000 7ff92aff1700 1 --2- 172.21.15.102:0/3433675494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff924100260 0x7ff9241006b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.878+0000 7ff92aff1700 1 --2- 172.21.15.102:0/3433675494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9240ff060 0x7ff9240ff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.878+0000 7ff92aff1700 1 -- 172.21.15.102:0/3433675494 >> 172.21.15.102:0/3433675494 conn(0x7ff9240fa610 msgr2=0x7ff9240fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.878+0000 7ff92aff1700 1 -- 172.21.15.102:0/3433675494 shutdown_connections 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.878+0000 7ff92aff1700 1 -- 172.21.15.102:0/3433675494 wait complete. 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.879+0000 7ff92aff1700 1 Processor -- start 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.879+0000 7ff92aff1700 1 -- start start 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.879+0000 7ff92aff1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9240ff060 0x7ff9240698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.879+0000 7ff92aff1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff924100260 0x7ff92419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.879+0000 7ff92aff1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92419aca0 con 0x7ff924100260 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.879+0000 7ff92aff1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92419ade0 con 0x7ff9240ff060 2024-03-26T18:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.879+0000 7ff923fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff924100260 0x7ff92419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.879+0000 7ff923fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff924100260 0x7ff92419a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45722/0 (socket says 172.21.15.102:45722) 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.879+0000 7ff923fff700 1 -- 172.21.15.102:0/4058306049 learned_addr learned my addr 172.21.15.102:0/4058306049 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.879+0000 7ff928d8d700 1 --2- 172.21.15.102:0/4058306049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9240ff060 0x7ff9240698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.880+0000 7ff923fff700 1 -- 172.21.15.102:0/4058306049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9240ff060 msgr2=0x7ff9240698d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.880+0000 7ff923fff700 1 --2- 172.21.15.102:0/4058306049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9240ff060 0x7ff9240698d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.880+0000 7ff923fff700 1 -- 172.21.15.102:0/4058306049 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9140095f0 con 0x7ff924100260 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.880+0000 7ff923fff700 1 --2- 172.21.15.102:0/4058306049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff924100260 0x7ff92419a6d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff914000c00 tx=0x7ff91400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.880+0000 7ff921ffb700 1 -- 172.21.15.102:0/4058306049 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91401d070 con 0x7ff924100260 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.880+0000 7ff921ffb700 1 -- 172.21.15.102:0/4058306049 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff914005dc0 con 0x7ff924100260 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.880+0000 7ff92aff1700 1 -- 172.21.15.102:0/4058306049 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92419b060 con 0x7ff924100260 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.880+0000 7ff921ffb700 1 -- 172.21.15.102:0/4058306049 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9140226a0 con 0x7ff924100260 2024-03-26T18:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.880+0000 7ff92aff1700 1 -- 172.21.15.102:0/4058306049 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92419b550 con 0x7ff924100260 2024-03-26T18:59:54.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.881+0000 7ff92aff1700 1 -- 172.21.15.102:0/4058306049 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff924107bb0 con 0x7ff924100260 2024-03-26T18:59:54.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.882+0000 7ff921ffb700 1 -- 172.21.15.102:0/4058306049 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff91400faf0 con 0x7ff924100260 2024-03-26T18:59:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.882+0000 7ff921ffb700 1 --2- 172.21.15.102:0/4058306049 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff91006e460 0x7ff910070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.882+0000 7ff921ffb700 1 -- 172.21.15.102:0/4058306049 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff91408f7e0 con 0x7ff924100260 2024-03-26T18:59:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.882+0000 7ff928d8d700 1 --2- 172.21.15.102:0/4058306049 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff91006e460 0x7ff910070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.882+0000 7ff928d8d700 1 --2- 172.21.15.102:0/4058306049 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff91006e460 0x7ff910070910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff9241000c0 tx=0x7ff91c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:54.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:54.887+0000 7ff921ffb700 1 -- 172.21.15.102:0/4058306049 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff91405c590 con 0x7ff924100260 2024-03-26T18:59:55.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.132+0000 7ff92aff1700 1 -- 172.21.15.102:0/4058306049 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff92419bbe0 con 0x7ff924100260 2024-03-26T18:59:55.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.132+0000 7ff921ffb700 1 -- 172.21.15.102:0/4058306049 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1905 (secure 0 0 0) 0x7ff914003680 con 0x7ff924100260 2024-03-26T18:59:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:e21 2024-03-26T18:59:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T18:59:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:59:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T18:59:55.126 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:59:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T18:59:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 21 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:58:42.198340+0000 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T18:59:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T18:59:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T18:59:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 14490 members: 24283,14490 2024-03-26T18:59:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:59:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:59:55.129 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:59:55.129 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:59:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T18:59:55.129 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T18:59:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:59:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T18:59:55.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.134+0000 7ff92aff1700 1 -- 172.21.15.102:0/4058306049 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff91006e460 msgr2=0x7ff910070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:55.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.134+0000 7ff92aff1700 1 --2- 172.21.15.102:0/4058306049 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff91006e460 0x7ff910070910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff9241000c0 tx=0x7ff91c00b040 comp rx=0 tx=0).stop 2024-03-26T18:59:55.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.134+0000 7ff92aff1700 1 -- 172.21.15.102:0/4058306049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff924100260 msgr2=0x7ff92419a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:55.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.134+0000 7ff92aff1700 1 --2- 172.21.15.102:0/4058306049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff924100260 0x7ff92419a6d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff914000c00 tx=0x7ff91400f690 comp rx=0 tx=0).stop 2024-03-26T18:59:55.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.135+0000 7ff92aff1700 1 -- 172.21.15.102:0/4058306049 shutdown_connections 2024-03-26T18:59:55.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.135+0000 7ff92aff1700 1 --2- 172.21.15.102:0/4058306049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9240ff060 0x7ff9240698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.135+0000 7ff92aff1700 1 --2- 172.21.15.102:0/4058306049 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff91006e460 0x7ff910070910 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.135+0000 7ff92aff1700 1 --2- 172.21.15.102:0/4058306049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff924100260 0x7ff92419a6d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.135+0000 7ff92aff1700 1 -- 172.21.15.102:0/4058306049 >> 172.21.15.102:0/4058306049 conn(0x7ff9240fa610 msgr2=0x7ff924103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:55.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.135+0000 7ff92aff1700 1 -- 172.21.15.102:0/4058306049 shutdown_connections 2024-03-26T18:59:55.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.135+0000 7ff92aff1700 1 -- 172.21.15.102:0/4058306049 wait complete. 2024-03-26T18:59:55.131 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 21 2024-03-26T18:59:55.188 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:54 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[70972]: 2024-03-26T18:59:54.807+0000 7fd81e1ef7c0 -1 osd.3 65 log_to_monitors true 2024-03-26T18:59:55.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.240+0000 7fb4268c6700 1 -- 172.21.15.102:0/454218144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb420103560 msgr2=0x7fb420105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.240+0000 7fb4268c6700 1 --2- 172.21.15.102:0/454218144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb420103560 0x7fb420105940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb414009910 tx=0x7fb414009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.240+0000 7fb4268c6700 1 -- 172.21.15.102:0/454218144 shutdown_connections 2024-03-26T18:59:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.240+0000 7fb4268c6700 1 --2- 172.21.15.102:0/454218144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb420103560 0x7fb420105940 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.240+0000 7fb4268c6700 1 --2- 172.21.15.102:0/454218144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb420100c40 0x7fb420103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.240+0000 7fb4268c6700 1 -- 172.21.15.102:0/454218144 >> 172.21.15.102:0/454218144 conn(0x7fb4200fa5e0 msgr2=0x7fb4200fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.240+0000 7fb4268c6700 1 -- 172.21.15.102:0/454218144 shutdown_connections 2024-03-26T18:59:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.240+0000 7fb4268c6700 1 -- 172.21.15.102:0/454218144 wait complete. 2024-03-26T18:59:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.240+0000 7fb4268c6700 1 Processor -- start 2024-03-26T18:59:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.240+0000 7fb4268c6700 1 -- start start 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb4268c6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb420100c40 0x7fb420197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb4268c6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb420103560 0x7fb420198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb4268c6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4201986b0 con 0x7fb420103560 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb4268c6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4201987f0 con 0x7fb420100c40 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb41f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb420103560 0x7fb420198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb41f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb420103560 0x7fb420198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45752/0 (socket says 172.21.15.102:45752) 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb41f7fe700 1 -- 172.21.15.102:0/474767438 learned_addr learned my addr 172.21.15.102:0/474767438 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb41ffff700 1 --2- 172.21.15.102:0/474767438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb420100c40 0x7fb420197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb41f7fe700 1 -- 172.21.15.102:0/474767438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb420100c40 msgr2=0x7fb420197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb41f7fe700 1 --2- 172.21.15.102:0/474767438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb420100c40 0x7fb420197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb41f7fe700 1 -- 172.21.15.102:0/474767438 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4140095f0 con 0x7fb420103560 2024-03-26T18:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb41f7fe700 1 --2- 172.21.15.102:0/474767438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb420103560 0x7fb420198090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb414000c00 tx=0x7fb41400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb41d7fa700 1 -- 172.21.15.102:0/474767438 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb41401d070 con 0x7fb420103560 2024-03-26T18:59:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb41d7fa700 1 -- 172.21.15.102:0/474767438 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb414005dc0 con 0x7fb420103560 2024-03-26T18:59:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.241+0000 7fb4268c6700 1 -- 172.21.15.102:0/474767438 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42019d240 con 0x7fb420103560 2024-03-26T18:59:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.242+0000 7fb41d7fa700 1 -- 172.21.15.102:0/474767438 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4140226a0 con 0x7fb420103560 2024-03-26T18:59:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.242+0000 7fb4268c6700 1 -- 172.21.15.102:0/474767438 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42019d730 con 0x7fb420103560 2024-03-26T18:59:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.243+0000 7fb4268c6700 1 -- 172.21.15.102:0/474767438 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4200fc1c0 con 0x7fb420103560 2024-03-26T18:59:55.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.243+0000 7fb41d7fa700 1 -- 172.21.15.102:0/474767438 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb4140038f0 con 0x7fb420103560 2024-03-26T18:59:55.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.243+0000 7fb41d7fa700 1 --2- 172.21.15.102:0/474767438 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb40806e460 0x7fb408070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:55.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.244+0000 7fb41d7fa700 1 -- 172.21.15.102:0/474767438 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb41408f8f0 con 0x7fb420103560 2024-03-26T18:59:55.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.244+0000 7fb41ffff700 1 --2- 172.21.15.102:0/474767438 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb40806e460 0x7fb408070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:55.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.244+0000 7fb41ffff700 1 --2- 172.21.15.102:0/474767438 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb40806e460 0x7fb408070910 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb410009c60 tx=0x7fb410009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:55.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.248+0000 7fb41d7fa700 1 -- 172.21.15.102:0/474767438 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb41405c620 con 0x7fb420103560 2024-03-26T18:59:55.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:55 smithi102 ceph-mon[78322]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:55.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:55 smithi102 ceph-mon[78322]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:55.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:55 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2757813972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:55.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:55 smithi102 ceph-mon[78322]: from='osd.3 [v2:172.21.15.190:6800/2294582584,v1:172.21.15.190:6801/2294582584]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T18:59:55.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:55 smithi102 ceph-mon[78322]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T18:59:55.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:55 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/4058306049' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:59:55.447 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 18:59:55 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[70972]: 2024-03-26T18:59:55.347+0000 7fd80f647700 -1 osd.3 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T18:59:55.447 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:55 smithi190 ceph-mon[65689]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:55.447 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:55 smithi190 ceph-mon[65689]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:55.447 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:55 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2757813972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:55.447 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:55 smithi190 ceph-mon[65689]: from='osd.3 [v2:172.21.15.190:6800/2294582584,v1:172.21.15.190:6801/2294582584]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T18:59:55.447 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:55 smithi190 ceph-mon[65689]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T18:59:55.447 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:55 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/4058306049' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T18:59:55.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.468+0000 7fb4268c6700 1 -- 172.21.15.102:0/474767438 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb420061190 con 0x7fb40806e460 2024-03-26T18:59:55.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.470+0000 7fb41d7fa700 1 -- 172.21.15.102:0/474767438 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fb420061190 con 0x7fb40806e460 2024-03-26T18:59:55.463 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T18:59:55.463 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T18:59:55.463 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T18:59:55.463 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T18:59:55.463 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T18:59:55.463 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T18:59:55.463 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-03-26T18:59:55.463 INFO:teuthology.orchestra.run.smithi102.stdout: "mon" 2024-03-26T18:59:55.463 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T18:59:55.463 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "10/23 daemons upgraded", 2024-03-26T18:59:55.464 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T18:59:55.464 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T18:59:55.464 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T18:59:55.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.472+0000 7fb4268c6700 1 -- 172.21.15.102:0/474767438 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb40806e460 msgr2=0x7fb408070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:55.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.472+0000 7fb4268c6700 1 --2- 172.21.15.102:0/474767438 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb40806e460 0x7fb408070910 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb410009c60 tx=0x7fb410009260 comp rx=0 tx=0).stop 2024-03-26T18:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.472+0000 7fb4268c6700 1 -- 172.21.15.102:0/474767438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb420103560 msgr2=0x7fb420198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.472+0000 7fb4268c6700 1 --2- 172.21.15.102:0/474767438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb420103560 0x7fb420198090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb414000c00 tx=0x7fb41400f690 comp rx=0 tx=0).stop 2024-03-26T18:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.472+0000 7fb4268c6700 1 -- 172.21.15.102:0/474767438 shutdown_connections 2024-03-26T18:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.472+0000 7fb4268c6700 1 --2- 172.21.15.102:0/474767438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb420100c40 0x7fb420197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.472+0000 7fb4268c6700 1 --2- 172.21.15.102:0/474767438 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb40806e460 0x7fb408070910 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.472+0000 7fb4268c6700 1 --2- 172.21.15.102:0/474767438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb420103560 0x7fb420198090 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.472+0000 7fb4268c6700 1 -- 172.21.15.102:0/474767438 >> 172.21.15.102:0/474767438 conn(0x7fb4200fa5e0 msgr2=0x7fb4200ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.472+0000 7fb4268c6700 1 -- 172.21.15.102:0/474767438 shutdown_connections 2024-03-26T18:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.472+0000 7fb4268c6700 1 -- 172.21.15.102:0/474767438 wait complete. 2024-03-26T18:59:55.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.576+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/3709884331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f18101370 msgr2=0x7f1f181017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:55.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.576+0000 7f1f1cc4a700 1 --2- 172.21.15.102:0/3709884331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f18101370 0x7f1f181017c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1f08009910 tx=0x7f1f08009c20 comp rx=0 tx=0).stop 2024-03-26T18:59:55.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.576+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/3709884331 shutdown_connections 2024-03-26T18:59:55.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.576+0000 7f1f1cc4a700 1 --2- 172.21.15.102:0/3709884331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f18101370 0x7f1f181017c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.576+0000 7f1f1cc4a700 1 --2- 172.21.15.102:0/3709884331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f18100170 0x7f1f18100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.576+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/3709884331 >> 172.21.15.102:0/3709884331 conn(0x7f1f180fb6e0 msgr2=0x7f1f180fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:55.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.576+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/3709884331 shutdown_connections 2024-03-26T18:59:55.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.576+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/3709884331 wait complete. 2024-03-26T18:59:55.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.577+0000 7f1f1cc4a700 1 Processor -- start 2024-03-26T18:59:55.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.577+0000 7f1f1cc4a700 1 -- start start 2024-03-26T18:59:55.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.577+0000 7f1f1cc4a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f18100170 0x7f1f18195a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:55.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.577+0000 7f1f1cc4a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f18101370 0x7f1f18195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:55.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.577+0000 7f1f1cc4a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f18196590 con 0x7f1f18101370 2024-03-26T18:59:55.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f1cc4a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f181966d0 con 0x7f1f18100170 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f15d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f18101370 0x7f1f18195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f15d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f18101370 0x7f1f18195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45764/0 (socket says 172.21.15.102:45764) 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f15d9b700 1 -- 172.21.15.102:0/4171106977 learned_addr learned my addr 172.21.15.102:0/4171106977 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f1659c700 1 --2- 172.21.15.102:0/4171106977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f18100170 0x7f1f18195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f15d9b700 1 -- 172.21.15.102:0/4171106977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f18100170 msgr2=0x7f1f18195a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f15d9b700 1 --2- 172.21.15.102:0/4171106977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f18100170 0x7f1f18195a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f15d9b700 1 -- 172.21.15.102:0/4171106977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f080095f0 con 0x7f1f18101370 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f15d9b700 1 --2- 172.21.15.102:0/4171106977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f18101370 0x7f1f18195f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f0800b5c0 tx=0x7f1f0800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f077fe700 1 -- 172.21.15.102:0/4171106977 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f0801d070 con 0x7f1f18101370 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/4171106977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1819b120 con 0x7f1f18101370 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f077fe700 1 -- 172.21.15.102:0/4171106977 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1f08022470 con 0x7f1f18101370 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f077fe700 1 -- 172.21.15.102:0/4171106977 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f0800f690 con 0x7f1f18101370 2024-03-26T18:59:55.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.578+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/4171106977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1819b690 con 0x7f1f18101370 2024-03-26T18:59:55.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.579+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/4171106977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f18108d60 con 0x7f1f18101370 2024-03-26T18:59:55.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.580+0000 7f1f077fe700 1 -- 172.21.15.102:0/4171106977 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f1f0800f900 con 0x7f1f18101370 2024-03-26T18:59:55.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.580+0000 7f1f077fe700 1 --2- 172.21.15.102:0/4171106977 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1f0006e460 0x7f1f00070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T18:59:55.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.580+0000 7f1f077fe700 1 -- 172.21.15.102:0/4171106977 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1f0808fbc0 con 0x7f1f18101370 2024-03-26T18:59:55.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.580+0000 7f1f1659c700 1 --2- 172.21.15.102:0/4171106977 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1f0006e460 0x7f1f00070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T18:59:55.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.581+0000 7f1f1659c700 1 --2- 172.21.15.102:0/4171106977 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1f0006e460 0x7f1f00070910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1f181011d0 tx=0x7f1f0c009190 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T18:59:55.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.585+0000 7f1f077fe700 1 -- 172.21.15.102:0/4171106977 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1f0805c960 con 0x7f1f18101370 2024-03-26T18:59:55.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.867+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/4171106977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1f1819bcd0 con 0x7f1f18101370 2024-03-26T18:59:55.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.868+0000 7f1f077fe700 1 -- 172.21.15.102:0/4171106977 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1383 (secure 0 0 0) 0x7f1f0805c4f0 con 0x7f1f18101370 2024-03-26T18:59:55.861 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 74/354 objects degraded (20.904%), 20 pgs degraded 2024-03-26T18:59:55.861 INFO:teuthology.orchestra.run.smithi102.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-26T18:59:55.861 INFO:teuthology.orchestra.run.smithi102.stdout: osd.3 (root=default,host=smithi190) is down 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 74/354 objects degraded (20.904%), 20 pgs degraded 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 1.0 is active+undersized+degraded, acting [0,1] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.7 is active+undersized+degraded, acting [4,2] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.8 is active+undersized+degraded, acting [5,0] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.a is active+undersized+degraded, acting [1,4] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.d is active+undersized+degraded, acting [1,2] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.e is active+undersized+degraded, acting [2,0] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-03-26T18:59:55.862 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.12 is active+undersized+degraded, acting [1,0] 2024-03-26T18:59:55.863 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.14 is active+undersized+degraded, acting [4,5] 2024-03-26T18:59:55.863 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-03-26T18:59:55.863 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-03-26T18:59:55.863 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-03-26T18:59:55.863 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1a is active+undersized+degraded, acting [4,5] 2024-03-26T18:59:55.863 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1d is active+undersized+degraded, acting [5,0] 2024-03-26T18:59:55.863 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-03-26T18:59:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.870+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/4171106977 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1f0006e460 msgr2=0x7f1f00070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.870+0000 7f1f1cc4a700 1 --2- 172.21.15.102:0/4171106977 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1f0006e460 0x7f1f00070910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1f181011d0 tx=0x7f1f0c009190 comp rx=0 tx=0).stop 2024-03-26T18:59:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.870+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/4171106977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f18101370 msgr2=0x7f1f18195f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T18:59:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.870+0000 7f1f1cc4a700 1 --2- 172.21.15.102:0/4171106977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f18101370 0x7f1f18195f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f0800b5c0 tx=0x7f1f0800bd70 comp rx=0 tx=0).stop 2024-03-26T18:59:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.870+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/4171106977 shutdown_connections 2024-03-26T18:59:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.870+0000 7f1f1cc4a700 1 --2- 172.21.15.102:0/4171106977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f18100170 0x7f1f18195a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.870+0000 7f1f1cc4a700 1 --2- 172.21.15.102:0/4171106977 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1f0006e460 0x7f1f00070910 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.870+0000 7f1f1cc4a700 1 --2- 172.21.15.102:0/4171106977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f18101370 0x7f1f18195f70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T18:59:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.870+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/4171106977 >> 172.21.15.102:0/4171106977 conn(0x7f1f180fb6e0 msgr2=0x7f1f181045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T18:59:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.870+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/4171106977 shutdown_connections 2024-03-26T18:59:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T18:59:55.870+0000 7f1f1cc4a700 1 -- 172.21.15.102:0/4171106977 wait complete. 2024-03-26T18:59:56.509 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:56 smithi190 ceph-mon[65689]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:56.509 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:56 smithi190 ceph-mon[65689]: pgmap v119: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 74/354 objects degraded (20.904%) 2024-03-26T18:59:56.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:56 smithi190 ceph-mon[65689]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T18:59:56.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:56 smithi190 ceph-mon[65689]: osdmap e68: 6 total, 5 up, 6 in 2024-03-26T18:59:56.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:56 smithi190 ceph-mon[65689]: from='osd.3 [v2:172.21.15.190:6800/2294582584,v1:172.21.15.190:6801/2294582584]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:59:56.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:56 smithi190 ceph-mon[65689]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:59:56.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:56 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/4171106977' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:59:56.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:56 smithi102 ceph-mon[78322]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:56.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:56 smithi102 ceph-mon[78322]: pgmap v119: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 300 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 74/354 objects degraded (20.904%) 2024-03-26T18:59:56.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:56 smithi102 ceph-mon[78322]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T18:59:56.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:56 smithi102 ceph-mon[78322]: osdmap e68: 6 total, 5 up, 6 in 2024-03-26T18:59:56.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:56 smithi102 ceph-mon[78322]: from='osd.3 [v2:172.21.15.190:6800/2294582584,v1:172.21.15.190:6801/2294582584]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:59:56.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:56 smithi102 ceph-mon[78322]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T18:59:56.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:56 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/4171106977' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T18:59:57.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:57.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T18:59:57.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: osd.3 [v2:172.21.15.190:6800/2294582584,v1:172.21.15.190:6801/2294582584] boot 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: osdmap e69: 6 total, 6 up, 6 in 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:57.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T18:59:57.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: osd.3 [v2:172.21.15.190:6800/2294582584,v1:172.21.15.190:6801/2294582584] boot 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: osdmap e69: 6 total, 6 up, 6 in 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:57.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:57.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T18:59:57.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T18:59:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:58 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T18:59:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:58 smithi102 ceph-mon[78322]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-03-26T18:59:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:58 smithi102 ceph-mon[78322]: pgmap v122: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 74/354 objects degraded (20.904%) 2024-03-26T18:59:58.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:58 smithi102 ceph-mon[78322]: osdmap e70: 6 total, 6 up, 6 in 2024-03-26T18:59:58.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:58 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T18:59:58.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:58 smithi190 ceph-mon[65689]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-03-26T18:59:58.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:58 smithi190 ceph-mon[65689]: pgmap v122: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 74/354 objects degraded (20.904%) 2024-03-26T18:59:58.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:58 smithi190 ceph-mon[65689]: osdmap e70: 6 total, 6 up, 6 in 2024-03-26T18:59:59.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 18:59:59 smithi102 ceph-mon[78322]: Health check update: Degraded data redundancy: 74/354 objects degraded (20.904%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T18:59:59.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 18:59:59 smithi190 ceph-mon[65689]: Health check update: Degraded data redundancy: 74/354 objects degraded (20.904%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T19:00:00.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pgmap v124: 65 pgs: 14 active+undersized+degraded, 9 active+undersized, 42 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 57/354 objects degraded (16.102%) 2024-03-26T19:00:00.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: Health detail: HEALTH_WARN Degraded data redundancy: 57/354 objects degraded (16.102%), 14 pgs degraded 2024-03-26T19:00:00.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: [WRN] PG_DEGRADED: Degraded data redundancy: 57/354 objects degraded (16.102%), 14 pgs degraded 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 1.0 is active+undersized+degraded, acting [0,1] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.7 is active+undersized+degraded, acting [4,2] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.8 is active+undersized+degraded, acting [5,0] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.a is active+undersized+degraded, acting [1,4] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.b is active+undersized+degraded, acting [4,5] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.d is active+undersized+degraded, acting [1,2] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.12 is active+undersized+degraded, acting [1,0] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.14 is active+undersized+degraded, acting [4,5] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.1a is active+undersized+degraded, acting [4,5] 2024-03-26T19:00:00.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:00 smithi102 ceph-mon[78322]: pg 2.1d is active+undersized+degraded, acting [5,0] 2024-03-26T19:00:00.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pgmap v124: 65 pgs: 14 active+undersized+degraded, 9 active+undersized, 42 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 57/354 objects degraded (16.102%) 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: Health detail: HEALTH_WARN Degraded data redundancy: 57/354 objects degraded (16.102%), 14 pgs degraded 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: [WRN] PG_DEGRADED: Degraded data redundancy: 57/354 objects degraded (16.102%), 14 pgs degraded 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 1.0 is active+undersized+degraded, acting [0,1] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.7 is active+undersized+degraded, acting [4,2] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.8 is active+undersized+degraded, acting [5,0] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.a is active+undersized+degraded, acting [1,4] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.b is active+undersized+degraded, acting [4,5] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.d is active+undersized+degraded, acting [1,2] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.12 is active+undersized+degraded, acting [1,0] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.14 is active+undersized+degraded, acting [4,5] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-03-26T19:00:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.1a is active+undersized+degraded, acting [4,5] 2024-03-26T19:00:00.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:00 smithi190 ceph-mon[65689]: pg 2.1d is active+undersized+degraded, acting [5,0] 2024-03-26T19:00:01.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:01 smithi102 ceph-mon[78322]: pgmap v125: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:01.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:01 smithi190 ceph-mon[65689]: pgmap v125: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:02.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:02 smithi102 ceph-mon[78322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/354 objects degraded (16.102%), 14 pgs degraded) 2024-03-26T19:00:02.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:02 smithi102 ceph-mon[78322]: Cluster is now healthy 2024-03-26T19:00:02.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:02 smithi190 ceph-mon[65689]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/354 objects degraded (16.102%), 14 pgs degraded) 2024-03-26T19:00:02.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:02 smithi190 ceph-mon[65689]: Cluster is now healthy 2024-03-26T19:00:03.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:03 smithi102 ceph-mon[78322]: pgmap v126: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:03.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:03 smithi190 ceph-mon[65689]: pgmap v126: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:06.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:06 smithi102 ceph-mon[78322]: pgmap v127: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:06.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:06 smithi190 ceph-mon[65689]: pgmap v127: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:08.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:08 smithi190 ceph-mon[65689]: pgmap v128: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:08.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:08.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:00:08.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:08 smithi102 ceph-mon[78322]: pgmap v128: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:00:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:10.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:10 smithi190 ceph-mon[65689]: pgmap v129: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:10.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:10 smithi102 ceph-mon[78322]: pgmap v129: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:12.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:12 smithi190 ceph-mon[65689]: pgmap v130: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:12.529 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:12 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T19:00:12.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:12 smithi102 ceph-mon[78322]: pgmap v130: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:12.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:12 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T19:00:13.594 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:13 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T19:00:13.595 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:13 smithi190 ceph-mon[65689]: Upgrade: osd.4 is safe to restart 2024-03-26T19:00:13.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:13 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T19:00:13.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:13 smithi102 ceph-mon[78322]: Upgrade: osd.4 is safe to restart 2024-03-26T19:00:14.452 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:14 smithi190 ceph-mon[65689]: pgmap v131: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:14.452 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:14.453 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T19:00:14.453 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:00:14.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:14 smithi102 ceph-mon[78322]: pgmap v131: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:14.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:14.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T19:00:14.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:00:15.488 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:15 smithi190 ceph-mon[65689]: Upgrade: Updating osd.4 2024-03-26T19:00:15.489 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:15 smithi190 ceph-mon[65689]: Deploying daemon osd.4 on smithi190 2024-03-26T19:00:15.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:15 smithi102 ceph-mon[78322]: Upgrade: Updating osd.4 2024-03-26T19:00:15.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:15 smithi102 ceph-mon[78322]: Deploying daemon osd.4 on smithi190 2024-03-26T19:00:16.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:16 smithi102 ceph-mon[78322]: pgmap v132: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:16.712 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:16 smithi190 ceph-mon[65689]: pgmap v132: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:17.030 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:16 smithi190 systemd[1]: Stopping Ceph osd.4 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:00:17.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:17 smithi190 ceph-mon[65689]: osd.4 marked itself down and dead 2024-03-26T19:00:17.531 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:17 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[45750]: 2024-03-26T19:00:17.174+0000 7f86577a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T19:00:17.531 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:17 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[45750]: 2024-03-26T19:00:17.174+0000 7f86577a0700 -1 osd.4 70 *** Got signal Terminated *** 2024-03-26T19:00:17.531 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:17 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[45750]: 2024-03-26T19:00:17.174+0000 7f86577a0700 -1 osd.4 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T19:00:17.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:17 smithi102 ceph-mon[78322]: osd.4 marked itself down and dead 2024-03-26T19:00:17.952 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:17 smithi190 podman[73390]: 2024-03-26 19:00:17.688037489 +0000 UTC m=+0.628946619 container died f2a02dcd6fee4e4d4633cf82d4fdc32ef60bb3cfe240f230a8cdbf0654226687 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-03-26T19:00:18.280 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:18 smithi190 podman[73390]: 2024-03-26 19:00:18.057995666 +0000 UTC m=+0.998904796 container cleanup f2a02dcd6fee4e4d4633cf82d4fdc32ef60bb3cfe240f230a8cdbf0654226687 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, ceph=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0) 2024-03-26T19:00:18.280 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:18 smithi190 bash[73390]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4 2024-03-26T19:00:18.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:18 smithi102 ceph-mon[78322]: pgmap v133: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:18.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:18 smithi102 ceph-mon[78322]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T19:00:18.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:18 smithi102 ceph-mon[78322]: osdmap e71: 6 total, 5 up, 6 in 2024-03-26T19:00:18.660 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:18 smithi190 ceph-mon[65689]: pgmap v133: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:18.661 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:18 smithi190 ceph-mon[65689]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T19:00:18.661 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:18 smithi190 ceph-mon[65689]: osdmap e71: 6 total, 5 up, 6 in 2024-03-26T19:00:18.661 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:18 smithi190 podman[73409]: 2024-03-26 19:00:18.334312549 +0000 UTC m=+0.642331481 container remove f2a02dcd6fee4e4d4633cf82d4fdc32ef60bb3cfe240f230a8cdbf0654226687 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.0, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T19:00:18.988 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:18 smithi190 podman[73499]: 2024-03-26 19:00:18.57100765 +0000 UTC m=+0.018500834 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:00:18.989 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:18 smithi190 podman[73499]: 2024-03-26 19:00:18.702183133 +0000 UTC m=+0.149676298 container create 1bc1660bff438527c986b3afaff371a70d28feb1676f0435e40869bf726ac7f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T19:00:18.989 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:18 smithi190 podman[73499]: 2024-03-26 19:00:18.913230593 +0000 UTC m=+0.360723757 container init 1bc1660bff438527c986b3afaff371a70d28feb1676f0435e40869bf726ac7f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.build-date=20240319) 2024-03-26T19:00:18.989 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:18 smithi190 podman[73499]: 2024-03-26 19:00:18.91646279 +0000 UTC m=+0.363955957 container start 1bc1660bff438527c986b3afaff371a70d28feb1676f0435e40869bf726ac7f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T19:00:18.989 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:18 smithi190 podman[73499]: 2024-03-26 19:00:18.99619649 +0000 UTC m=+0.443689657 container attach 1bc1660bff438527c986b3afaff371a70d28feb1676f0435e40869bf726ac7f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T19:00:19.280 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:19 smithi190 podman[73499]: 2024-03-26 19:00:19.209592746 +0000 UTC m=+0.657085901 container died 1bc1660bff438527c986b3afaff371a70d28feb1676f0435e40869bf726ac7f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.schema-version=1.0) 2024-03-26T19:00:19.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:19 smithi102 ceph-mon[78322]: osdmap e72: 6 total, 5 up, 6 in 2024-03-26T19:00:19.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:19 smithi190 ceph-mon[65689]: osdmap e72: 6 total, 5 up, 6 in 2024-03-26T19:00:20.235 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:19 smithi190 podman[73525]: 2024-03-26 19:00:19.816244484 +0000 UTC m=+0.601729116 container remove 1bc1660bff438527c986b3afaff371a70d28feb1676f0435e40869bf726ac7f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240320.145742-3605160, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T19:00:20.235 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:19 smithi190 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-26T19:00:20.235 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:19 smithi190 systemd[1]: Stopped Ceph osd.4 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T19:00:20.235 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:19 smithi190 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.4.service: Consumed 1min 1.196s CPU time. 2024-03-26T19:00:20.235 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 systemd[1]: Starting Ceph osd.4 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:00:20.487 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:20 smithi190 ceph-mon[65689]: pgmap v136: 65 pgs: 4 active+undersized+degraded, 5 peering, 2 stale+active+clean, 6 active+undersized, 48 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 14/354 objects degraded (3.955%) 2024-03-26T19:00:20.487 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:20 smithi190 ceph-mon[65689]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T19:00:20.487 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:20 smithi190 ceph-mon[65689]: Health check failed: Degraded data redundancy: 14/354 objects degraded (3.955%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T19:00:20.488 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 podman[73625]: 2024-03-26 19:00:20.242435912 +0000 UTC m=+0.096329610 container create 33fd8c59c02d4f45fcee23004634851f9e8991fa48f7e5d9d7e889c54c37f27c (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True) 2024-03-26T19:00:20.488 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 podman[73625]: 2024-03-26 19:00:20.164012432 +0000 UTC m=+0.017906130 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:00:20.488 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 podman[73625]: 2024-03-26 19:00:20.495034967 +0000 UTC m=+0.348928665 container init 33fd8c59c02d4f45fcee23004634851f9e8991fa48f7e5d9d7e889c54c37f27c (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T19:00:20.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:20 smithi102 ceph-mon[78322]: pgmap v136: 65 pgs: 4 active+undersized+degraded, 5 peering, 2 stale+active+clean, 6 active+undersized, 48 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 14/354 objects degraded (3.955%) 2024-03-26T19:00:20.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:20 smithi102 ceph-mon[78322]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T19:00:20.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:20 smithi102 ceph-mon[78322]: Health check failed: Degraded data redundancy: 14/354 objects degraded (3.955%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T19:00:20.780 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 podman[73625]: 2024-03-26 19:00:20.498224856 +0000 UTC m=+0.352118556 container start 33fd8c59c02d4f45fcee23004634851f9e8991fa48f7e5d9d7e889c54c37f27c (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-26T19:00:20.780 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 podman[73625]: 2024-03-26 19:00:20.603458158 +0000 UTC m=+0.457351858 container attach 33fd8c59c02d4f45fcee23004634851f9e8991fa48f7e5d9d7e889c54c37f27c (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T19:00:21.249 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate[73641]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T19:00:21.249 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate[73641]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T19:00:21.249 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 bash[73625]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T19:00:21.249 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 bash[73625]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T19:00:21.249 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 bash[73625]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T19:00:21.249 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 bash[73625]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T19:00:21.250 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 bash[73625]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-03-26T19:00:21.250 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 bash[73625]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T19:00:21.250 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 bash[73625]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T19:00:21.250 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate[73641]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T19:00:21.250 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate[73641]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T19:00:21.250 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate[73641]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-03-26T19:00:21.250 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate[73641]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T19:00:21.250 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate[73641]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T19:00:21.250 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:20 smithi190 podman[73625]: 2024-03-26 19:00:20.964933461 +0000 UTC m=+0.818827152 container died 33fd8c59c02d4f45fcee23004634851f9e8991fa48f7e5d9d7e889c54c37f27c (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-26T19:00:21.864 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:21 smithi190 podman[73761]: 2024-03-26 19:00:21.556861973 +0000 UTC m=+0.586440428 container remove 33fd8c59c02d4f45fcee23004634851f9e8991fa48f7e5d9d7e889c54c37f27c (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T19:00:21.864 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:21 smithi190 podman[73826]: 2024-03-26 19:00:21.765977183 +0000 UTC m=+0.114477301 container create 23e140cd8961fed85f06b4a21ea6e977aaff572cc0a952bac22f0e5fc8b56235 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T19:00:21.864 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:21 smithi190 podman[73826]: 2024-03-26 19:00:21.669295165 +0000 UTC m=+0.017795313 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:00:22.280 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:22 smithi190 podman[73826]: 2024-03-26 19:00:22.001535715 +0000 UTC m=+0.350035837 container init 23e140cd8961fed85f06b4a21ea6e977aaff572cc0a952bac22f0e5fc8b56235 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T19:00:22.280 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:22 smithi190 podman[73826]: 2024-03-26 19:00:22.004511664 +0000 UTC m=+0.353011786 container start 23e140cd8961fed85f06b4a21ea6e977aaff572cc0a952bac22f0e5fc8b56235 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T19:00:22.281 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:22 smithi190 ceph-osd[73844]: -- 172.21.15.190:0/133128276 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55569b49b680 con 0x55569b465000 2024-03-26T19:00:22.281 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:22 smithi190 bash[73826]: 23e140cd8961fed85f06b4a21ea6e977aaff572cc0a952bac22f0e5fc8b56235 2024-03-26T19:00:22.281 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:22 smithi190 systemd[1]: Started Ceph osd.4 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T19:00:22.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:22 smithi102 ceph-mon[78322]: pgmap v137: 65 pgs: 14 active+undersized+degraded, 5 peering, 15 active+undersized, 31 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 54/354 objects degraded (15.254%) 2024-03-26T19:00:22.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:22.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:22.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:00:22.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:22.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:00:22.780 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:22 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:00:22.557+0000 7efe1df0d7c0 -1 Falling back to public interface 2024-03-26T19:00:22.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:22 smithi190 ceph-mon[65689]: pgmap v137: 65 pgs: 14 active+undersized+degraded, 5 peering, 15 active+undersized, 31 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 54/354 objects degraded (15.254%) 2024-03-26T19:00:22.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:22.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:22.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:00:22.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:22.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:00:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:23 smithi190 ceph-mon[65689]: pgmap v138: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 60/354 objects degraded (16.949%) 2024-03-26T19:00:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:23 smithi190 ceph-mon[65689]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T19:00:23.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:23 smithi102 ceph-mon[78322]: pgmap v138: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 60/354 objects degraded (16.949%) 2024-03-26T19:00:23.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:23 smithi102 ceph-mon[78322]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T19:00:25.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.984+0000 7fa0788aa700 1 -- 172.21.15.102:0/2315803892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa074100c00 msgr2=0x7fa074102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.984+0000 7fa0788aa700 1 --2- 172.21.15.102:0/2315803892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa074100c00 0x7fa074102fe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa064009910 tx=0x7fa064009c20 comp rx=0 tx=0).stop 2024-03-26T19:00:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.984+0000 7fa0788aa700 1 -- 172.21.15.102:0/2315803892 shutdown_connections 2024-03-26T19:00:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.984+0000 7fa0788aa700 1 --2- 172.21.15.102:0/2315803892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa074103520 0x7fa074105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.984+0000 7fa0788aa700 1 --2- 172.21.15.102:0/2315803892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa074100c00 0x7fa074102fe0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.984+0000 7fa0788aa700 1 -- 172.21.15.102:0/2315803892 >> 172.21.15.102:0/2315803892 conn(0x7fa0740fa600 msgr2=0x7fa0740fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.984+0000 7fa0788aa700 1 -- 172.21.15.102:0/2315803892 shutdown_connections 2024-03-26T19:00:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.984+0000 7fa0788aa700 1 -- 172.21.15.102:0/2315803892 wait complete. 2024-03-26T19:00:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.984+0000 7fa0788aa700 1 Processor -- start 2024-03-26T19:00:25.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa0788aa700 1 -- start start 2024-03-26T19:00:25.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa0788aa700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa074100c00 0x7fa074100560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:25.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa0788aa700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa074103520 0x7fa0740febb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:25.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa0788aa700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740ff0f0 con 0x7fa074100c00 2024-03-26T19:00:25.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa0788aa700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740ff230 con 0x7fa074103520 2024-03-26T19:00:25.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa07259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa074100c00 0x7fa074100560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:25.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa07259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa074100c00 0x7fa074100560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57974/0 (socket says 172.21.15.102:57974) 2024-03-26T19:00:25.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa07259c700 1 -- 172.21.15.102:0/1906768057 learned_addr learned my addr 172.21.15.102:0/1906768057 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:25.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa069d9b700 1 --2- 172.21.15.102:0/1906768057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa074103520 0x7fa0740febb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:25.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa07259c700 1 -- 172.21.15.102:0/1906768057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa074103520 msgr2=0x7fa0740febb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:25.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa07259c700 1 --2- 172.21.15.102:0/1906768057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa074103520 0x7fa0740febb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:25.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.985+0000 7fa07259c700 1 -- 172.21.15.102:0/1906768057 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0640095f0 con 0x7fa074100c00 2024-03-26T19:00:25.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.986+0000 7fa07259c700 1 --2- 172.21.15.102:0/1906768057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa074100c00 0x7fa074100560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa06400b5c0 tx=0x7fa06400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:25.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.986+0000 7fa06bfff700 1 -- 172.21.15.102:0/1906768057 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa06401d070 con 0x7fa074100c00 2024-03-26T19:00:25.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.986+0000 7fa06bfff700 1 -- 172.21.15.102:0/1906768057 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa06400f460 con 0x7fa074100c00 2024-03-26T19:00:25.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.986+0000 7fa0788aa700 1 -- 172.21.15.102:0/1906768057 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0740ff4b0 con 0x7fa074100c00 2024-03-26T19:00:25.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.986+0000 7fa06bfff700 1 -- 172.21.15.102:0/1906768057 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa064021640 con 0x7fa074100c00 2024-03-26T19:00:25.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.986+0000 7fa0788aa700 1 -- 172.21.15.102:0/1906768057 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0740ff9a0 con 0x7fa074100c00 2024-03-26T19:00:25.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.987+0000 7fa0788aa700 1 -- 172.21.15.102:0/1906768057 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07418fbb0 con 0x7fa074100c00 2024-03-26T19:00:25.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.987+0000 7fa06bfff700 1 -- 172.21.15.102:0/1906768057 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa06400fa60 con 0x7fa074100c00 2024-03-26T19:00:25.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.988+0000 7fa06bfff700 1 --2- 172.21.15.102:0/1906768057 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa06006e460 0x7fa060070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:25.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.988+0000 7fa06bfff700 1 -- 172.21.15.102:0/1906768057 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa06408f5f0 con 0x7fa074100c00 2024-03-26T19:00:25.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.988+0000 7fa069d9b700 1 --2- 172.21.15.102:0/1906768057 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa06006e460 0x7fa060070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:25.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.988+0000 7fa069d9b700 1 --2- 172.21.15.102:0/1906768057 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa06006e460 0x7fa060070910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa05c004630 tx=0x7fa05c00b3c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:25.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:25.992+0000 7fa06bfff700 1 -- 172.21.15.102:0/1906768057 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa06405c3a0 con 0x7fa074100c00 2024-03-26T19:00:26.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.214+0000 7fa0788aa700 1 -- 172.21.15.102:0/1906768057 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa074061190 con 0x7fa06006e460 2024-03-26T19:00:26.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.215+0000 7fa06bfff700 1 -- 172.21.15.102:0/1906768057 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fa074061190 con 0x7fa06006e460 2024-03-26T19:00:26.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.218+0000 7fa0788aa700 1 -- 172.21.15.102:0/1906768057 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa06006e460 msgr2=0x7fa060070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:26.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.218+0000 7fa0788aa700 1 --2- 172.21.15.102:0/1906768057 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa06006e460 0x7fa060070910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa05c004630 tx=0x7fa05c00b3c0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.218+0000 7fa0788aa700 1 -- 172.21.15.102:0/1906768057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa074100c00 msgr2=0x7fa074100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.218+0000 7fa0788aa700 1 --2- 172.21.15.102:0/1906768057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa074100c00 0x7fa074100560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa06400b5c0 tx=0x7fa06400bd70 comp rx=0 tx=0).stop 2024-03-26T19:00:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.218+0000 7fa0788aa700 1 -- 172.21.15.102:0/1906768057 shutdown_connections 2024-03-26T19:00:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.218+0000 7fa0788aa700 1 --2- 172.21.15.102:0/1906768057 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa06006e460 0x7fa060070910 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.218+0000 7fa0788aa700 1 --2- 172.21.15.102:0/1906768057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa074100c00 0x7fa074100560 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.218+0000 7fa0788aa700 1 --2- 172.21.15.102:0/1906768057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa074103520 0x7fa0740febb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.218+0000 7fa0788aa700 1 -- 172.21.15.102:0/1906768057 >> 172.21.15.102:0/1906768057 conn(0x7fa0740fa600 msgr2=0x7fa0740fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.218+0000 7fa0788aa700 1 -- 172.21.15.102:0/1906768057 shutdown_connections 2024-03-26T19:00:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.218+0000 7fa0788aa700 1 -- 172.21.15.102:0/1906768057 wait complete. 2024-03-26T19:00:26.222 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T19:00:26.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.327+0000 7feb14ac1700 1 -- 172.21.15.102:0/4183901120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb10103560 msgr2=0x7feb10105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:26.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.327+0000 7feb14ac1700 1 --2- 172.21.15.102:0/4183901120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb10103560 0x7feb10105940 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feb04009910 tx=0x7feb04009c20 comp rx=0 tx=0).stop 2024-03-26T19:00:26.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.327+0000 7feb14ac1700 1 -- 172.21.15.102:0/4183901120 shutdown_connections 2024-03-26T19:00:26.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.327+0000 7feb14ac1700 1 --2- 172.21.15.102:0/4183901120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb10103560 0x7feb10105940 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.321 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:26.321 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:26.321 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:26 smithi102 ceph-mon[78322]: pgmap v139: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 60/354 objects degraded (16.949%) 2024-03-26T19:00:26.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.327+0000 7feb14ac1700 1 --2- 172.21.15.102:0/4183901120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb10100c40 0x7feb10103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.327+0000 7feb14ac1700 1 -- 172.21.15.102:0/4183901120 >> 172.21.15.102:0/4183901120 conn(0x7feb100fa600 msgr2=0x7feb100fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.327+0000 7feb14ac1700 1 -- 172.21.15.102:0/4183901120 shutdown_connections 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.327+0000 7feb14ac1700 1 -- 172.21.15.102:0/4183901120 wait complete. 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.328+0000 7feb14ac1700 1 Processor -- start 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.328+0000 7feb14ac1700 1 -- start start 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.328+0000 7feb14ac1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb10100c40 0x7feb10195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.328+0000 7feb14ac1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb10103560 0x7feb10195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.328+0000 7feb14ac1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb101964b0 con 0x7feb10100c40 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.328+0000 7feb14ac1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb101965f0 con 0x7feb10103560 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.329+0000 7feb0dd9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb10103560 0x7feb10195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.329+0000 7feb0e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb10100c40 0x7feb10195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.329+0000 7feb0e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb10100c40 0x7feb10195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57992/0 (socket says 172.21.15.102:57992) 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.329+0000 7feb0e59c700 1 -- 172.21.15.102:0/3838281472 learned_addr learned my addr 172.21.15.102:0/3838281472 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:26.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.329+0000 7feb0e59c700 1 -- 172.21.15.102:0/3838281472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb10103560 msgr2=0x7feb10195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:26.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.329+0000 7feb0e59c700 1 --2- 172.21.15.102:0/3838281472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb10103560 0x7feb10195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.329+0000 7feb0e59c700 1 -- 172.21.15.102:0/3838281472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb040095f0 con 0x7feb10100c40 2024-03-26T19:00:26.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.329+0000 7feb0e59c700 1 --2- 172.21.15.102:0/3838281472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb10100c40 0x7feb10195950 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feb0000e920 tx=0x7feb0000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:26.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.329+0000 7feaff7fe700 1 -- 172.21.15.102:0/3838281472 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb0000c9d0 con 0x7feb10100c40 2024-03-26T19:00:26.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.329+0000 7feaff7fe700 1 -- 172.21.15.102:0/3838281472 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feb00010430 con 0x7feb10100c40 2024-03-26T19:00:26.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.329+0000 7feb14ac1700 1 -- 172.21.15.102:0/3838281472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1019b0a0 con 0x7feb10100c40 2024-03-26T19:00:26.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.330+0000 7feaff7fe700 1 -- 172.21.15.102:0/3838281472 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb00018650 con 0x7feb10100c40 2024-03-26T19:00:26.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.330+0000 7feb14ac1700 1 -- 172.21.15.102:0/3838281472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1019b4e0 con 0x7feb10100c40 2024-03-26T19:00:26.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.331+0000 7feb14ac1700 1 -- 172.21.15.102:0/3838281472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb1018fb70 con 0x7feb10100c40 2024-03-26T19:00:26.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.332+0000 7feaff7fe700 1 -- 172.21.15.102:0/3838281472 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7feb000105a0 con 0x7feb10100c40 2024-03-26T19:00:26.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.332+0000 7feaff7fe700 1 --2- 172.21.15.102:0/3838281472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feaf806e460 0x7feaf8070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:26.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.332+0000 7feaff7fe700 1 -- 172.21.15.102:0/3838281472 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6500+0+0 (secure 0 0 0) 0x7feb00014070 con 0x7feb10100c40 2024-03-26T19:00:26.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.332+0000 7feb0dd9b700 1 --2- 172.21.15.102:0/3838281472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feaf806e460 0x7feaf8070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:26.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.333+0000 7feb0dd9b700 1 --2- 172.21.15.102:0/3838281472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feaf806e460 0x7feaf8070910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb0400b6f0 tx=0x7feb04005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:26.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.337+0000 7feaff7fe700 1 -- 172.21.15.102:0/3838281472 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feb00053740 con 0x7feb10100c40 2024-03-26T19:00:26.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:26.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:26.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:26 smithi190 ceph-mon[65689]: pgmap v139: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 60/354 objects degraded (16.949%) 2024-03-26T19:00:26.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.558+0000 7feb14ac1700 1 -- 172.21.15.102:0/3838281472 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feb10061190 con 0x7feaf806e460 2024-03-26T19:00:26.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.559+0000 7feaff7fe700 1 -- 172.21.15.102:0/3838281472 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7feb10061190 con 0x7feaf806e460 2024-03-26T19:00:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.562+0000 7feb14ac1700 1 -- 172.21.15.102:0/3838281472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feaf806e460 msgr2=0x7feaf8070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.562+0000 7feb14ac1700 1 --2- 172.21.15.102:0/3838281472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feaf806e460 0x7feaf8070910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb0400b6f0 tx=0x7feb04005a90 comp rx=0 tx=0).stop 2024-03-26T19:00:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.562+0000 7feb14ac1700 1 -- 172.21.15.102:0/3838281472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb10100c40 msgr2=0x7feb10195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.562+0000 7feb14ac1700 1 --2- 172.21.15.102:0/3838281472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb10100c40 0x7feb10195950 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feb0000e920 tx=0x7feb0000ec30 comp rx=0 tx=0).stop 2024-03-26T19:00:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.562+0000 7feb14ac1700 1 -- 172.21.15.102:0/3838281472 shutdown_connections 2024-03-26T19:00:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.562+0000 7feb14ac1700 1 --2- 172.21.15.102:0/3838281472 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7feaf806e460 0x7feaf8070910 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.562+0000 7feb14ac1700 1 --2- 172.21.15.102:0/3838281472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb10100c40 0x7feb10195950 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.562+0000 7feb14ac1700 1 --2- 172.21.15.102:0/3838281472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb10103560 0x7feb10195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.562+0000 7feb14ac1700 1 -- 172.21.15.102:0/3838281472 >> 172.21.15.102:0/3838281472 conn(0x7feb100fa600 msgr2=0x7feb100fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:26.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.562+0000 7feb14ac1700 1 -- 172.21.15.102:0/3838281472 shutdown_connections 2024-03-26T19:00:26.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.562+0000 7feb14ac1700 1 -- 172.21.15.102:0/3838281472 wait complete. 2024-03-26T19:00:26.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.671+0000 7f3e30503700 1 -- 172.21.15.102:0/576450547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e28100fe0 msgr2=0x7f3e280ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.671+0000 7f3e30503700 1 --2- 172.21.15.102:0/576450547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e28100fe0 0x7f3e280ff3f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3e18009910 tx=0x7f3e18009c20 comp rx=0 tx=0).stop 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 -- 172.21.15.102:0/576450547 shutdown_connections 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 --2- 172.21.15.102:0/576450547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e28100fe0 0x7f3e280ff3f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 --2- 172.21.15.102:0/576450547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e28100690 0x7f3e28100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 -- 172.21.15.102:0/576450547 >> 172.21.15.102:0/576450547 conn(0x7f3e280f9e50 msgr2=0x7f3e280fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 -- 172.21.15.102:0/576450547 shutdown_connections 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 -- 172.21.15.102:0/576450547 wait complete. 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 Processor -- start 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 -- start start 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e28100690 0x7f3e28195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e28100fe0 0x7f3e28195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:26.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e281964b0 con 0x7f3e28100fe0 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.672+0000 7f3e30503700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e281965f0 con 0x7f3e28100690 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e2da9e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e28100fe0 0x7f3e28195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e2da9e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e28100fe0 0x7f3e28195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58002/0 (socket says 172.21.15.102:58002) 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e2da9e700 1 -- 172.21.15.102:0/3831599390 learned_addr learned my addr 172.21.15.102:0/3831599390 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e2e29f700 1 --2- 172.21.15.102:0/3831599390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e28100690 0x7f3e28195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e2da9e700 1 -- 172.21.15.102:0/3831599390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e28100690 msgr2=0x7f3e28195950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e2da9e700 1 --2- 172.21.15.102:0/3831599390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e28100690 0x7f3e28195950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e2da9e700 1 -- 172.21.15.102:0/3831599390 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e180095f0 con 0x7f3e28100fe0 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e2da9e700 1 --2- 172.21.15.102:0/3831599390 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e28100fe0 0x7f3e28195e90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3e18006010 tx=0x7f3e1800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e1f7fe700 1 -- 172.21.15.102:0/3831599390 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e1801d070 con 0x7f3e28100fe0 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e1f7fe700 1 -- 172.21.15.102:0/3831599390 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3e1800f460 con 0x7f3e28100fe0 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e30503700 1 -- 172.21.15.102:0/3831599390 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2819b040 con 0x7f3e28100fe0 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e1f7fe700 1 -- 172.21.15.102:0/3831599390 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e18021640 con 0x7f3e28100fe0 2024-03-26T19:00:26.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.673+0000 7f3e30503700 1 -- 172.21.15.102:0/3831599390 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2819b530 con 0x7f3e28100fe0 2024-03-26T19:00:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.674+0000 7f3e30503700 1 -- 172.21.15.102:0/3831599390 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e28107b40 con 0x7f3e28100fe0 2024-03-26T19:00:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.675+0000 7f3e1f7fe700 1 -- 172.21.15.102:0/3831599390 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3e1800fab0 con 0x7f3e28100fe0 2024-03-26T19:00:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.675+0000 7f3e1f7fe700 1 --2- 172.21.15.102:0/3831599390 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3e1406e460 0x7f3e14070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.675+0000 7f3e1f7fe700 1 -- 172.21.15.102:0/3831599390 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3e1808f6c0 con 0x7f3e28100fe0 2024-03-26T19:00:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.675+0000 7f3e2e29f700 1 --2- 172.21.15.102:0/3831599390 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3e1406e460 0x7f3e14070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.676+0000 7f3e2e29f700 1 --2- 172.21.15.102:0/3831599390 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3e1406e460 0x7f3e14070910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e28068d60 tx=0x7f3e2400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:26.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.680+0000 7f3e1f7fe700 1 -- 172.21.15.102:0/3831599390 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3e1805c3f0 con 0x7f3e28100fe0 2024-03-26T19:00:26.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.888+0000 7f3e30503700 1 -- 172.21.15.102:0/3831599390 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e280fe990 con 0x7f3e1406e460 2024-03-26T19:00:26.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.899+0000 7f3e1f7fe700 1 -- 172.21.15.102:0/3831599390 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f3e280fe990 con 0x7f3e1406e460 2024-03-26T19:00:26.893 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T19:00:26.893 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (12m) 68s ago 13m 25.0M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T19:00:26.893 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (13m) 68s ago 13m 8354k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T19:00:26.893 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (12m) 1s ago 12m 8232k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T19:00:26.893 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (2m) 68s ago 13m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T19:00:26.893 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (2m) 1s ago 12m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T19:00:26.893 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (11m) 68s ago 13m 86.6M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T19:00:26.893 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (7m) 68s ago 7m 132M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (8m) 68s ago 8m 16.1M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (7m) 1s ago 7m 16.5M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (8m) 1s ago 8m 120M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (4m) 68s ago 14m 504M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (4m) 1s ago 12m 450M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (3m) 68s ago 15m 46.2M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (3m) 1s ago 12m 35.4M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (13m) 68s ago 13m 16.3M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (12m) 1s ago 12m 15.1M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (2m) 68s ago 11m 86.0M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (109s) 68s ago 11m 95.7M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (73s) 68s ago 10m 11.7M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T19:00:26.894 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (38s) 1s ago 10m 96.6M 2240M 19.0.0-2311-g36051606 497bb87bc5da 03955331c962 2024-03-26T19:00:26.895 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (5s) 1s ago 9m 12.2M 2240M 19.0.0-2311-g36051606 497bb87bc5da 23e140cd8961 2024-03-26T19:00:26.895 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (9m) 1s ago 9m 223M 2240M 18.2.0 dc2bc1663786 1be709fc0288 2024-03-26T19:00:26.895 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (4m) 68s ago 13m 74.3M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T19:00:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.902+0000 7f3e30503700 1 -- 172.21.15.102:0/3831599390 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3e1406e460 msgr2=0x7f3e14070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.902+0000 7f3e30503700 1 --2- 172.21.15.102:0/3831599390 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3e1406e460 0x7f3e14070910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e28068d60 tx=0x7f3e2400b410 comp rx=0 tx=0).stop 2024-03-26T19:00:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.902+0000 7f3e30503700 1 -- 172.21.15.102:0/3831599390 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e28100fe0 msgr2=0x7f3e28195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.902+0000 7f3e30503700 1 --2- 172.21.15.102:0/3831599390 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e28100fe0 0x7f3e28195e90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3e18006010 tx=0x7f3e1800bd70 comp rx=0 tx=0).stop 2024-03-26T19:00:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.902+0000 7f3e30503700 1 -- 172.21.15.102:0/3831599390 shutdown_connections 2024-03-26T19:00:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.902+0000 7f3e30503700 1 --2- 172.21.15.102:0/3831599390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e28100690 0x7f3e28195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.902+0000 7f3e30503700 1 --2- 172.21.15.102:0/3831599390 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3e1406e460 0x7f3e14070910 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.902+0000 7f3e30503700 1 --2- 172.21.15.102:0/3831599390 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e28100fe0 0x7f3e28195e90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.902+0000 7f3e30503700 1 -- 172.21.15.102:0/3831599390 >> 172.21.15.102:0/3831599390 conn(0x7f3e280f9e50 msgr2=0x7f3e280fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.902+0000 7f3e30503700 1 -- 172.21.15.102:0/3831599390 shutdown_connections 2024-03-26T19:00:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:26.902+0000 7f3e30503700 1 -- 172.21.15.102:0/3831599390 wait complete. 2024-03-26T19:00:26.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.005+0000 7f9a83e64700 1 -- 172.21.15.102:0/2340794358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7c0ff050 msgr2=0x7f9a7c0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:26.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.005+0000 7f9a83e64700 1 --2- 172.21.15.102:0/2340794358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7c0ff050 0x7f9a7c0ff460 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009910 tx=0x7f9a78009c20 comp rx=0 tx=0).stop 2024-03-26T19:00:26.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.005+0000 7f9a83e64700 1 -- 172.21.15.102:0/2340794358 shutdown_connections 2024-03-26T19:00:26.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.005+0000 7f9a83e64700 1 --2- 172.21.15.102:0/2340794358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a7c100250 0x7f9a7c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.005+0000 7f9a83e64700 1 --2- 172.21.15.102:0/2340794358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7c0ff050 0x7f9a7c0ff460 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:26.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.005+0000 7f9a83e64700 1 -- 172.21.15.102:0/2340794358 >> 172.21.15.102:0/2340794358 conn(0x7f9a7c0fa5e0 msgr2=0x7f9a7c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:26.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.005+0000 7f9a83e64700 1 -- 172.21.15.102:0/2340794358 shutdown_connections 2024-03-26T19:00:26.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.005+0000 7f9a83e64700 1 -- 172.21.15.102:0/2340794358 wait complete. 2024-03-26T19:00:27.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.006+0000 7f9a83e64700 1 Processor -- start 2024-03-26T19:00:27.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.006+0000 7f9a83e64700 1 -- start start 2024-03-26T19:00:27.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.006+0000 7f9a83e64700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7c0ff050 0x7f9a7c102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:27.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.006+0000 7f9a83e64700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a7c100250 0x7f9a7c101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:27.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.006+0000 7f9a83e64700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c101b70 con 0x7f9a7c0ff050 2024-03-26T19:00:27.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.006+0000 7f9a83e64700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c101cb0 con 0x7f9a7c100250 2024-03-26T19:00:27.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.006+0000 7f9a81c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7c0ff050 0x7f9a7c102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:27.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.006+0000 7f9a81c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7c0ff050 0x7f9a7c102fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58022/0 (socket says 172.21.15.102:58022) 2024-03-26T19:00:27.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.006+0000 7f9a81c00700 1 -- 172.21.15.102:0/3821809435 learned_addr learned my addr 172.21.15.102:0/3821809435 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:27.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.006+0000 7f9a813ff700 1 --2- 172.21.15.102:0/3821809435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a7c100250 0x7f9a7c101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:27.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.007+0000 7f9a81c00700 1 -- 172.21.15.102:0/3821809435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a7c100250 msgr2=0x7f9a7c101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:27.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.007+0000 7f9a81c00700 1 --2- 172.21.15.102:0/3821809435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a7c100250 0x7f9a7c101630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.007+0000 7f9a81c00700 1 -- 172.21.15.102:0/3821809435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a780095f0 con 0x7f9a7c0ff050 2024-03-26T19:00:27.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.007+0000 7f9a81c00700 1 --2- 172.21.15.102:0/3821809435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7c0ff050 0x7f9a7c102fe0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800bda0 tx=0x7f9a7800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:27.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.007+0000 7f9a72ffd700 1 -- 172.21.15.102:0/3821809435 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a7801d070 con 0x7f9a7c0ff050 2024-03-26T19:00:27.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.007+0000 7f9a72ffd700 1 -- 172.21.15.102:0/3821809435 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9a78022470 con 0x7f9a7c0ff050 2024-03-26T19:00:27.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.007+0000 7f9a83e64700 1 -- 172.21.15.102:0/3821809435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c101f30 con 0x7f9a7c0ff050 2024-03-26T19:00:27.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.007+0000 7f9a72ffd700 1 -- 172.21.15.102:0/3821809435 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a7800f650 con 0x7f9a7c0ff050 2024-03-26T19:00:27.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.007+0000 7f9a83e64700 1 -- 172.21.15.102:0/3821809435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c102420 con 0x7f9a7c0ff050 2024-03-26T19:00:27.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.008+0000 7f9a83e64700 1 -- 172.21.15.102:0/3821809435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c107c20 con 0x7f9a7c0ff050 2024-03-26T19:00:27.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.009+0000 7f9a72ffd700 1 -- 172.21.15.102:0/3821809435 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9a78004af0 con 0x7f9a7c0ff050 2024-03-26T19:00:27.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.009+0000 7f9a72ffd700 1 --2- 172.21.15.102:0/3821809435 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9a6806e460 0x7f9a68070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:27.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.009+0000 7f9a72ffd700 1 -- 172.21.15.102:0/3821809435 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9a7808fa60 con 0x7f9a7c0ff050 2024-03-26T19:00:27.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.009+0000 7f9a813ff700 1 --2- 172.21.15.102:0/3821809435 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9a6806e460 0x7f9a68070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:27.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.010+0000 7f9a813ff700 1 --2- 172.21.15.102:0/3821809435 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9a6806e460 0x7f9a68070910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009a10 tx=0x7f9a6c009190 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:27.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.014+0000 7f9a72ffd700 1 -- 172.21.15.102:0/3821809435 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9a7805c840 con 0x7f9a7c0ff050 2024-03-26T19:00:27.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.300+0000 7f9a83e64700 1 -- 172.21.15.102:0/3821809435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9a7c19df60 con 0x7f9a7c0ff050 2024-03-26T19:00:27.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.300+0000 7f9a72ffd700 1 -- 172.21.15.102:0/3821809435 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f9a7805c3d0 con 0x7f9a7c0ff050 2024-03-26T19:00:27.294 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:00:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T19:00:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:00:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:00:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T19:00:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:00:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:00:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T19:00:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-03-26T19:00:27.295 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 4 2024-03-26T19:00:27.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:00:27.295 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T19:00:27.295 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T19:00:27.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:00:27.295 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T19:00:27.295 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 5, 2024-03-26T19:00:27.295 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 8 2024-03-26T19:00:27.295 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T19:00:27.295 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:00:27.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.302+0000 7f9a83e64700 1 -- 172.21.15.102:0/3821809435 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9a6806e460 msgr2=0x7f9a68070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:27.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.302+0000 7f9a83e64700 1 --2- 172.21.15.102:0/3821809435 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9a6806e460 0x7f9a68070910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009a10 tx=0x7f9a6c009190 comp rx=0 tx=0).stop 2024-03-26T19:00:27.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.302+0000 7f9a83e64700 1 -- 172.21.15.102:0/3821809435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7c0ff050 msgr2=0x7f9a7c102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:27.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.302+0000 7f9a83e64700 1 --2- 172.21.15.102:0/3821809435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7c0ff050 0x7f9a7c102fe0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800bda0 tx=0x7f9a7800bdd0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.303+0000 7f9a83e64700 1 -- 172.21.15.102:0/3821809435 shutdown_connections 2024-03-26T19:00:27.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.303+0000 7f9a83e64700 1 --2- 172.21.15.102:0/3821809435 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9a6806e460 0x7f9a68070910 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.303+0000 7f9a83e64700 1 --2- 172.21.15.102:0/3821809435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7c0ff050 0x7f9a7c102fe0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.303+0000 7f9a83e64700 1 --2- 172.21.15.102:0/3821809435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a7c100250 0x7f9a7c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.303+0000 7f9a83e64700 1 -- 172.21.15.102:0/3821809435 >> 172.21.15.102:0/3821809435 conn(0x7f9a7c0fa5e0 msgr2=0x7f9a7c0683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:27.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.303+0000 7f9a83e64700 1 -- 172.21.15.102:0/3821809435 shutdown_connections 2024-03-26T19:00:27.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.303+0000 7f9a83e64700 1 -- 172.21.15.102:0/3821809435 wait complete. 2024-03-26T19:00:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.406+0000 7fa37455d700 1 -- 172.21.15.102:0/1239837863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa36c103560 msgr2=0x7fa36c105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.406+0000 7fa37455d700 1 --2- 172.21.15.102:0/1239837863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa36c103560 0x7fa36c105940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa368009910 tx=0x7fa368009c20 comp rx=0 tx=0).stop 2024-03-26T19:00:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.406+0000 7fa37455d700 1 -- 172.21.15.102:0/1239837863 shutdown_connections 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.406+0000 7fa37455d700 1 --2- 172.21.15.102:0/1239837863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa36c103560 0x7fa36c105940 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.406+0000 7fa37455d700 1 --2- 172.21.15.102:0/1239837863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa36c100c40 0x7fa36c103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.406+0000 7fa37455d700 1 -- 172.21.15.102:0/1239837863 >> 172.21.15.102:0/1239837863 conn(0x7fa36c0fa600 msgr2=0x7fa36c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.406+0000 7fa37455d700 1 -- 172.21.15.102:0/1239837863 shutdown_connections 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.406+0000 7fa37455d700 1 -- 172.21.15.102:0/1239837863 wait complete. 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa37455d700 1 Processor -- start 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa37455d700 1 -- start start 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa37455d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa36c100c40 0x7fa36c195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa37455d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa36c103560 0x7fa36c195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa37455d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c1964b0 con 0x7fa36c100c40 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa37455d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c1965f0 con 0x7fa36c103560 2024-03-26T19:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa3722f9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa36c100c40 0x7fa36c195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa3722f9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa36c100c40 0x7fa36c195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58042/0 (socket says 172.21.15.102:58042) 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa3722f9700 1 -- 172.21.15.102:0/3245311285 learned_addr learned my addr 172.21.15.102:0/3245311285 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa371af8700 1 --2- 172.21.15.102:0/3245311285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa36c103560 0x7fa36c195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa3722f9700 1 -- 172.21.15.102:0/3245311285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa36c103560 msgr2=0x7fa36c195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.407+0000 7fa3722f9700 1 --2- 172.21.15.102:0/3245311285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa36c103560 0x7fa36c195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.408+0000 7fa3722f9700 1 -- 172.21.15.102:0/3245311285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3680095f0 con 0x7fa36c100c40 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.408+0000 7fa3722f9700 1 --2- 172.21.15.102:0/3245311285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa36c100c40 0x7fa36c195950 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00d8d0 tx=0x7fa35c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.408+0000 7fa3637fe700 1 -- 172.21.15.102:0/3245311285 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa35c0098a0 con 0x7fa36c100c40 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.408+0000 7fa3637fe700 1 -- 172.21.15.102:0/3245311285 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa35c010460 con 0x7fa36c100c40 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.408+0000 7fa37455d700 1 -- 172.21.15.102:0/3245311285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36c19b0a0 con 0x7fa36c100c40 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.408+0000 7fa3637fe700 1 -- 172.21.15.102:0/3245311285 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa35c00f5f0 con 0x7fa36c100c40 2024-03-26T19:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.408+0000 7fa37455d700 1 -- 172.21.15.102:0/3245311285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36c19b5f0 con 0x7fa36c100c40 2024-03-26T19:00:27.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.409+0000 7fa37455d700 1 -- 172.21.15.102:0/3245311285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36c18fb70 con 0x7fa36c100c40 2024-03-26T19:00:27.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.409+0000 7fa3637fe700 1 -- 172.21.15.102:0/3245311285 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa35c009a00 con 0x7fa36c100c40 2024-03-26T19:00:27.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.410+0000 7fa3637fe700 1 --2- 172.21.15.102:0/3245311285 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa35806e460 0x7fa358070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:27.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.410+0000 7fa371af8700 1 --2- 172.21.15.102:0/3245311285 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa35806e460 0x7fa358070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:27.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.410+0000 7fa3637fe700 1 -- 172.21.15.102:0/3245311285 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa35c08e1b0 con 0x7fa36c100c40 2024-03-26T19:00:27.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.410+0000 7fa371af8700 1 --2- 172.21.15.102:0/3245311285 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa35806e460 0x7fa358070910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa368006010 tx=0x7fa36800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:27.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.414+0000 7fa3637fe700 1 -- 172.21.15.102:0/3245311285 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa35c05af60 con 0x7fa36c100c40 2024-03-26T19:00:27.530 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:27 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:00:27.171+0000 7efe1df0d7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-26T19:00:27.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:27 smithi190 ceph-mon[65689]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:27.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:27 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:27.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:27 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:27.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:27 smithi102 ceph-mon[78322]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:27.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:27 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:27.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:27 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:27.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.660+0000 7fa37455d700 1 -- 172.21.15.102:0/3245311285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa36c061960 con 0x7fa36c100c40 2024-03-26T19:00:27.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.660+0000 7fa3637fe700 1 -- 172.21.15.102:0/3245311285 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1905 (secure 0 0 0) 0x7fa35c016790 con 0x7fa36c100c40 2024-03-26T19:00:27.653 INFO:teuthology.orchestra.run.smithi102.stdout:e21 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 21 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:58:42.198340+0000 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T19:00:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T19:00:27.655 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 14490 members: 24283,14490 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:00:27.656 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:00:27.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.662+0000 7fa37455d700 1 -- 172.21.15.102:0/3245311285 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa35806e460 msgr2=0x7fa358070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:27.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.662+0000 7fa37455d700 1 --2- 172.21.15.102:0/3245311285 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa35806e460 0x7fa358070910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa368006010 tx=0x7fa36800b540 comp rx=0 tx=0).stop 2024-03-26T19:00:27.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.662+0000 7fa37455d700 1 -- 172.21.15.102:0/3245311285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa36c100c40 msgr2=0x7fa36c195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:27.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.662+0000 7fa37455d700 1 --2- 172.21.15.102:0/3245311285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa36c100c40 0x7fa36c195950 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00d8d0 tx=0x7fa35c00dbe0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.662+0000 7fa37455d700 1 -- 172.21.15.102:0/3245311285 shutdown_connections 2024-03-26T19:00:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.662+0000 7fa37455d700 1 --2- 172.21.15.102:0/3245311285 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa35806e460 0x7fa358070910 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.662+0000 7fa37455d700 1 --2- 172.21.15.102:0/3245311285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa36c100c40 0x7fa36c195950 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.662+0000 7fa37455d700 1 --2- 172.21.15.102:0/3245311285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa36c103560 0x7fa36c195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.662+0000 7fa37455d700 1 -- 172.21.15.102:0/3245311285 >> 172.21.15.102:0/3245311285 conn(0x7fa36c0fa600 msgr2=0x7fa36c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.662+0000 7fa37455d700 1 -- 172.21.15.102:0/3245311285 shutdown_connections 2024-03-26T19:00:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.662+0000 7fa37455d700 1 -- 172.21.15.102:0/3245311285 wait complete. 2024-03-26T19:00:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 21 2024-03-26T19:00:27.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.767+0000 7fe42d2a8700 1 -- 172.21.15.102:0/38231530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4280ff050 msgr2=0x7fe4280ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:27.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.767+0000 7fe42d2a8700 1 --2- 172.21.15.102:0/38231530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4280ff050 0x7fe4280ff460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe418009910 tx=0x7fe418009c20 comp rx=0 tx=0).stop 2024-03-26T19:00:27.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.767+0000 7fe42d2a8700 1 -- 172.21.15.102:0/38231530 shutdown_connections 2024-03-26T19:00:27.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.767+0000 7fe42d2a8700 1 --2- 172.21.15.102:0/38231530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe428100250 0x7fe4281006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.767+0000 7fe42d2a8700 1 --2- 172.21.15.102:0/38231530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4280ff050 0x7fe4280ff460 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.767+0000 7fe42d2a8700 1 -- 172.21.15.102:0/38231530 >> 172.21.15.102:0/38231530 conn(0x7fe4280fa600 msgr2=0x7fe4280fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:27.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.767+0000 7fe42d2a8700 1 -- 172.21.15.102:0/38231530 shutdown_connections 2024-03-26T19:00:27.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.767+0000 7fe42d2a8700 1 -- 172.21.15.102:0/38231530 wait complete. 2024-03-26T19:00:27.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.768+0000 7fe42d2a8700 1 Processor -- start 2024-03-26T19:00:27.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.768+0000 7fe42d2a8700 1 -- start start 2024-03-26T19:00:27.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.768+0000 7fe42d2a8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4280ff050 0x7fe428195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:27.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.768+0000 7fe42d2a8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe428100250 0x7fe428195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.768+0000 7fe42d2a8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4281964d0 con 0x7fe4280ff050 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.768+0000 7fe42d2a8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe428196610 con 0x7fe428100250 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.768+0000 7fe426ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4280ff050 0x7fe428195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.768+0000 7fe426ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4280ff050 0x7fe428195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58064/0 (socket says 172.21.15.102:58064) 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.768+0000 7fe426ffd700 1 -- 172.21.15.102:0/21533140 learned_addr learned my addr 172.21.15.102:0/21533140 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.768+0000 7fe4267fc700 1 --2- 172.21.15.102:0/21533140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe428100250 0x7fe428195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.769+0000 7fe426ffd700 1 -- 172.21.15.102:0/21533140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe428100250 msgr2=0x7fe428195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.769+0000 7fe426ffd700 1 --2- 172.21.15.102:0/21533140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe428100250 0x7fe428195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.769+0000 7fe426ffd700 1 -- 172.21.15.102:0/21533140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4180095f0 con 0x7fe4280ff050 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.769+0000 7fe426ffd700 1 --2- 172.21.15.102:0/21533140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4280ff050 0x7fe428195970 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe41800bda0 tx=0x7fe41800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.769+0000 7fe40ffff700 1 -- 172.21.15.102:0/21533140 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe41801d070 con 0x7fe4280ff050 2024-03-26T19:00:27.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.769+0000 7fe42d2a8700 1 -- 172.21.15.102:0/21533140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42819b060 con 0x7fe4280ff050 2024-03-26T19:00:27.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.769+0000 7fe40ffff700 1 -- 172.21.15.102:0/21533140 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe41800f460 con 0x7fe4280ff050 2024-03-26T19:00:27.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.769+0000 7fe40ffff700 1 -- 172.21.15.102:0/21533140 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe418021620 con 0x7fe4280ff050 2024-03-26T19:00:27.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.769+0000 7fe42d2a8700 1 -- 172.21.15.102:0/21533140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42819b550 con 0x7fe4280ff050 2024-03-26T19:00:27.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.770+0000 7fe42d2a8700 1 -- 172.21.15.102:0/21533140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe428107ba0 con 0x7fe4280ff050 2024-03-26T19:00:27.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.770+0000 7fe40ffff700 1 -- 172.21.15.102:0/21533140 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe41800f5d0 con 0x7fe4280ff050 2024-03-26T19:00:27.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.771+0000 7fe40ffff700 1 --2- 172.21.15.102:0/21533140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe41006e460 0x7fe410070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:27.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.771+0000 7fe4267fc700 1 --2- 172.21.15.102:0/21533140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe41006e460 0x7fe410070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:27.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.771+0000 7fe40ffff700 1 -- 172.21.15.102:0/21533140 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe41808f6c0 con 0x7fe4280ff050 2024-03-26T19:00:27.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.771+0000 7fe4267fc700 1 --2- 172.21.15.102:0/21533140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe41006e460 0x7fe410070910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe41c005950 tx=0x7fe41c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:27.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.775+0000 7fe40ffff700 1 -- 172.21.15.102:0/21533140 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe41805c3f0 con 0x7fe4280ff050 2024-03-26T19:00:27.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.996+0000 7fe42d2a8700 1 -- 172.21.15.102:0/21533140 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe428069350 con 0x7fe41006e460 2024-03-26T19:00:27.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:27.998+0000 7fe40ffff700 1 -- 172.21.15.102:0/21533140 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fe428069350 con 0x7fe41006e460 2024-03-26T19:00:27.991 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:00:27.991 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T19:00:27.992 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T19:00:27.992 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T19:00:27.992 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T19:00:27.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T19:00:27.992 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-03-26T19:00:27.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon" 2024-03-26T19:00:27.992 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T19:00:27.992 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "11/23 daemons upgraded", 2024-03-26T19:00:27.992 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T19:00:27.992 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T19:00:27.992 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:00:27.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.000+0000 7fe42d2a8700 1 -- 172.21.15.102:0/21533140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe41006e460 msgr2=0x7fe410070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:27.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.000+0000 7fe42d2a8700 1 --2- 172.21.15.102:0/21533140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe41006e460 0x7fe410070910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe41c005950 tx=0x7fe41c00b410 comp rx=0 tx=0).stop 2024-03-26T19:00:27.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.000+0000 7fe42d2a8700 1 -- 172.21.15.102:0/21533140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4280ff050 msgr2=0x7fe428195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:27.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.000+0000 7fe42d2a8700 1 --2- 172.21.15.102:0/21533140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4280ff050 0x7fe428195970 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe41800bda0 tx=0x7fe41800bdd0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.000+0000 7fe42d2a8700 1 -- 172.21.15.102:0/21533140 shutdown_connections 2024-03-26T19:00:27.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.000+0000 7fe42d2a8700 1 --2- 172.21.15.102:0/21533140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe41006e460 0x7fe410070910 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.000+0000 7fe42d2a8700 1 --2- 172.21.15.102:0/21533140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4280ff050 0x7fe428195970 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.000+0000 7fe42d2a8700 1 --2- 172.21.15.102:0/21533140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe428100250 0x7fe428195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:27.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.000+0000 7fe42d2a8700 1 -- 172.21.15.102:0/21533140 >> 172.21.15.102:0/21533140 conn(0x7fe4280fa600 msgr2=0x7fe428068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:27.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.000+0000 7fe42d2a8700 1 -- 172.21.15.102:0/21533140 shutdown_connections 2024-03-26T19:00:27.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.000+0000 7fe42d2a8700 1 -- 172.21.15.102:0/21533140 wait complete. 2024-03-26T19:00:28.030 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:27 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:00:27.687+0000 7efe1df0d7c0 -1 osd.4 70 log_to_monitors true 2024-03-26T19:00:28.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.104+0000 7faf9490a700 1 -- 172.21.15.102:0/2479491901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf90100260 msgr2=0x7faf901006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:28.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.104+0000 7faf9490a700 1 --2- 172.21.15.102:0/2479491901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf90100260 0x7faf901006b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faf84009910 tx=0x7faf84009c20 comp rx=0 tx=0).stop 2024-03-26T19:00:28.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.104+0000 7faf9490a700 1 -- 172.21.15.102:0/2479491901 shutdown_connections 2024-03-26T19:00:28.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.104+0000 7faf9490a700 1 --2- 172.21.15.102:0/2479491901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf90100260 0x7faf901006b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:28.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.104+0000 7faf9490a700 1 --2- 172.21.15.102:0/2479491901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf900ff060 0x7faf900ff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:28.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.104+0000 7faf9490a700 1 -- 172.21.15.102:0/2479491901 >> 172.21.15.102:0/2479491901 conn(0x7faf900fa610 msgr2=0x7faf900fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:28.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.104+0000 7faf9490a700 1 -- 172.21.15.102:0/2479491901 shutdown_connections 2024-03-26T19:00:28.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.104+0000 7faf9490a700 1 -- 172.21.15.102:0/2479491901 wait complete. 2024-03-26T19:00:28.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.105+0000 7faf9490a700 1 Processor -- start 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.105+0000 7faf9490a700 1 -- start start 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.105+0000 7faf9490a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf900ff060 0x7faf90195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.105+0000 7faf9490a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf90100260 0x7faf90195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.105+0000 7faf9490a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf901964f0 con 0x7faf900ff060 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.105+0000 7faf9490a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf90196630 con 0x7faf90100260 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.105+0000 7faf8e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf900ff060 0x7faf90195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.105+0000 7faf8e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf900ff060 0x7faf90195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58088/0 (socket says 172.21.15.102:58088) 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.105+0000 7faf8e59c700 1 -- 172.21.15.102:0/384458009 learned_addr learned my addr 172.21.15.102:0/384458009 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.106+0000 7faf8dd9b700 1 --2- 172.21.15.102:0/384458009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf90100260 0x7faf90195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.106+0000 7faf8e59c700 1 -- 172.21.15.102:0/384458009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf90100260 msgr2=0x7faf90195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.106+0000 7faf8e59c700 1 --2- 172.21.15.102:0/384458009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf90100260 0x7faf90195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.106+0000 7faf8e59c700 1 -- 172.21.15.102:0/384458009 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf840095f0 con 0x7faf900ff060 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.106+0000 7faf8e59c700 1 --2- 172.21.15.102:0/384458009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf900ff060 0x7faf90195990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faf8000e8d0 tx=0x7faf8000ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:28.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.106+0000 7faf7f7fe700 1 -- 172.21.15.102:0/384458009 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf80009770 con 0x7faf900ff060 2024-03-26T19:00:28.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.106+0000 7faf7f7fe700 1 -- 172.21.15.102:0/384458009 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faf8000c810 con 0x7faf900ff060 2024-03-26T19:00:28.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.106+0000 7faf9490a700 1 -- 172.21.15.102:0/384458009 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9019b0e0 con 0x7faf900ff060 2024-03-26T19:00:28.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.106+0000 7faf7f7fe700 1 -- 172.21.15.102:0/384458009 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf80003da0 con 0x7faf900ff060 2024-03-26T19:00:28.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.106+0000 7faf9490a700 1 -- 172.21.15.102:0/384458009 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf9019b630 con 0x7faf900ff060 2024-03-26T19:00:28.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.107+0000 7faf9490a700 1 -- 172.21.15.102:0/384458009 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf90107bd0 con 0x7faf900ff060 2024-03-26T19:00:28.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.108+0000 7faf7f7fe700 1 -- 172.21.15.102:0/384458009 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7faf80021440 con 0x7faf900ff060 2024-03-26T19:00:28.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.108+0000 7faf7f7fe700 1 --2- 172.21.15.102:0/384458009 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7faf7806e460 0x7faf78070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:28.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.108+0000 7faf7f7fe700 1 -- 172.21.15.102:0/384458009 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faf80014070 con 0x7faf900ff060 2024-03-26T19:00:28.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.108+0000 7faf8dd9b700 1 --2- 172.21.15.102:0/384458009 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7faf7806e460 0x7faf78070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:28.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.108+0000 7faf8dd9b700 1 --2- 172.21.15.102:0/384458009 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7faf7806e460 0x7faf78070910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faf8400b6f0 tx=0x7faf84005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:28.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.112+0000 7faf7f7fe700 1 -- 172.21.15.102:0/384458009 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faf8005b530 con 0x7faf900ff060 2024-03-26T19:00:28.381 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:28 smithi190 ceph-mon[65689]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:28.382 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:28 smithi190 ceph-mon[65689]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:28.382 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:28 smithi190 ceph-mon[65689]: pgmap v140: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 60/354 objects degraded (16.949%) 2024-03-26T19:00:28.382 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:28 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3821809435' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:00:28.382 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:28 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3245311285' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:00:28.382 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:28 smithi190 ceph-mon[65689]: from='osd.4 [v2:172.21.15.190:6808/2940868243,v1:172.21.15.190:6809/2940868243]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T19:00:28.382 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:28 smithi190 ceph-mon[65689]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T19:00:28.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.398+0000 7faf9490a700 1 -- 172.21.15.102:0/384458009 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faf90107d90 con 0x7faf900ff060 2024-03-26T19:00:28.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.399+0000 7faf7f7fe700 1 -- 172.21.15.102:0/384458009 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1162 (secure 0 0 0) 0x7faf8000ce60 con 0x7faf900ff060 2024-03-26T19:00:28.392 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 60/354 objects degraded (16.949%), 16 pgs degraded 2024-03-26T19:00:28.392 INFO:teuthology.orchestra.run.smithi102.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-26T19:00:28.392 INFO:teuthology.orchestra.run.smithi102.stdout: osd.4 (root=default,host=smithi190) is down 2024-03-26T19:00:28.392 INFO:teuthology.orchestra.run.smithi102.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 60/354 objects degraded (16.949%), 16 pgs degraded 2024-03-26T19:00:28.392 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2024-03-26T19:00:28.392 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.5 is active+undersized+degraded, acting [3,0] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.6 is active+undersized+degraded, acting [1,3] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.7 is active+undersized+degraded, acting [3,2] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.9 is active+undersized+degraded, acting [1,0] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.b is active+undersized+degraded, acting [3,5] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.f is active+undersized+degraded, acting [0,5] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.11 is active+undersized+degraded, acting [3,1] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.13 is active+undersized+degraded, acting [0,2] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.18 is active+undersized+degraded, acting [5,3] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.19 is active+undersized+degraded, acting [0,2] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1a is active+undersized+degraded, acting [3,5] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-03-26T19:00:28.393 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1f is active+undersized+degraded, acting [0,3] 2024-03-26T19:00:28.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.401+0000 7faf9490a700 1 -- 172.21.15.102:0/384458009 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7faf7806e460 msgr2=0x7faf78070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:28.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.401+0000 7faf9490a700 1 --2- 172.21.15.102:0/384458009 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7faf7806e460 0x7faf78070910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faf8400b6f0 tx=0x7faf84005a90 comp rx=0 tx=0).stop 2024-03-26T19:00:28.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.401+0000 7faf9490a700 1 -- 172.21.15.102:0/384458009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf900ff060 msgr2=0x7faf90195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:28.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.401+0000 7faf9490a700 1 --2- 172.21.15.102:0/384458009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf900ff060 0x7faf90195990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faf8000e8d0 tx=0x7faf8000ebe0 comp rx=0 tx=0).stop 2024-03-26T19:00:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.401+0000 7faf9490a700 1 -- 172.21.15.102:0/384458009 shutdown_connections 2024-03-26T19:00:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.401+0000 7faf9490a700 1 --2- 172.21.15.102:0/384458009 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7faf7806e460 0x7faf78070910 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.401+0000 7faf9490a700 1 --2- 172.21.15.102:0/384458009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf900ff060 0x7faf90195990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.401+0000 7faf9490a700 1 --2- 172.21.15.102:0/384458009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf90100260 0x7faf90195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.401+0000 7faf9490a700 1 -- 172.21.15.102:0/384458009 >> 172.21.15.102:0/384458009 conn(0x7faf900fa610 msgr2=0x7faf90068d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.401+0000 7faf9490a700 1 -- 172.21.15.102:0/384458009 shutdown_connections 2024-03-26T19:00:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:28.401+0000 7faf9490a700 1 -- 172.21.15.102:0/384458009 wait complete. 2024-03-26T19:00:28.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:28 smithi102 ceph-mon[78322]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:28.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:28 smithi102 ceph-mon[78322]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:28.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:28 smithi102 ceph-mon[78322]: pgmap v140: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 60/354 objects degraded (16.949%) 2024-03-26T19:00:28.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:28 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3821809435' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:00:28.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:28 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3245311285' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:00:28.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:28 smithi102 ceph-mon[78322]: from='osd.4 [v2:172.21.15.190:6808/2940868243,v1:172.21.15.190:6809/2940868243]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T19:00:28.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:28 smithi102 ceph-mon[78322]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T19:00:29.435 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:00:29 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:00:29.235+0000 7efe0b35d700 -1 osd.4 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T19:00:29.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:29 smithi190 ceph-mon[65689]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:29.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:29 smithi190 ceph-mon[65689]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T19:00:29.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:29 smithi190 ceph-mon[65689]: osdmap e73: 6 total, 5 up, 6 in 2024-03-26T19:00:29.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:29 smithi190 ceph-mon[65689]: from='osd.4 [v2:172.21.15.190:6808/2940868243,v1:172.21.15.190:6809/2940868243]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T19:00:29.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:29 smithi190 ceph-mon[65689]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T19:00:29.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:29 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/384458009' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:00:29.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:29 smithi190 ceph-mon[65689]: Health check update: Degraded data redundancy: 60/354 objects degraded (16.949%), 16 pgs degraded (PG_DEGRADED) 2024-03-26T19:00:29.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:29 smithi190 ceph-mon[65689]: from='osd.4 ' entity='osd.4' 2024-03-26T19:00:29.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:29 smithi102 ceph-mon[78322]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:29.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:29 smithi102 ceph-mon[78322]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T19:00:29.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:29 smithi102 ceph-mon[78322]: osdmap e73: 6 total, 5 up, 6 in 2024-03-26T19:00:29.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:29 smithi102 ceph-mon[78322]: from='osd.4 [v2:172.21.15.190:6808/2940868243,v1:172.21.15.190:6809/2940868243]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T19:00:29.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:29 smithi102 ceph-mon[78322]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T19:00:29.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:29 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/384458009' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:00:29.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:29 smithi102 ceph-mon[78322]: Health check update: Degraded data redundancy: 60/354 objects degraded (16.949%), 16 pgs degraded (PG_DEGRADED) 2024-03-26T19:00:29.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:29 smithi102 ceph-mon[78322]: from='osd.4 ' entity='osd.4' 2024-03-26T19:00:30.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: pgmap v142: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 60/354 objects degraded (16.949%) 2024-03-26T19:00:30.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: osd.4 [v2:172.21.15.190:6808/2940868243,v1:172.21.15.190:6809/2940868243] boot 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: osdmap e74: 6 total, 6 up, 6 in 2024-03-26T19:00:30.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:30 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T19:00:30.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: pgmap v142: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 60/354 objects degraded (16.949%) 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T19:00:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: osd.4 [v2:172.21.15.190:6808/2940868243,v1:172.21.15.190:6809/2940868243] boot 2024-03-26T19:00:30.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: osdmap e74: 6 total, 6 up, 6 in 2024-03-26T19:00:30.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:30 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T19:00:31.600 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:31 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T19:00:31.600 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:31 smithi102 ceph-mon[78322]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-26T19:00:31.600 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:31 smithi102 ceph-mon[78322]: osdmap e75: 6 total, 6 up, 6 in 2024-03-26T19:00:31.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:31 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T19:00:31.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:31 smithi190 ceph-mon[65689]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-26T19:00:31.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:31 smithi190 ceph-mon[65689]: osdmap e75: 6 total, 6 up, 6 in 2024-03-26T19:00:32.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:32 smithi102 ceph-mon[78322]: pgmap v145: 65 pgs: 10 peering, 11 active+undersized+degraded, 13 active+undersized, 31 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 35/354 objects degraded (9.887%) 2024-03-26T19:00:32.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:32 smithi190 ceph-mon[65689]: pgmap v145: 65 pgs: 10 peering, 11 active+undersized+degraded, 13 active+undersized, 31 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 35/354 objects degraded (9.887%) 2024-03-26T19:00:34.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:34 smithi102 ceph-mon[78322]: pgmap v146: 65 pgs: 10 peering, 10 active+undersized+degraded, 10 active+undersized, 35 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 31/354 objects degraded (8.757%) 2024-03-26T19:00:34.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:34 smithi102 ceph-mon[78322]: Health check update: Degraded data redundancy: 31/354 objects degraded (8.757%), 10 pgs degraded (PG_DEGRADED) 2024-03-26T19:00:34.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:34 smithi190 ceph-mon[65689]: pgmap v146: 65 pgs: 10 peering, 10 active+undersized+degraded, 10 active+undersized, 35 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 31/354 objects degraded (8.757%) 2024-03-26T19:00:34.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:34 smithi190 ceph-mon[65689]: Health check update: Degraded data redundancy: 31/354 objects degraded (8.757%), 10 pgs degraded (PG_DEGRADED) 2024-03-26T19:00:35.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:35 smithi102 ceph-mon[78322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/354 objects degraded (8.757%), 10 pgs degraded) 2024-03-26T19:00:35.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:35 smithi102 ceph-mon[78322]: Cluster is now healthy 2024-03-26T19:00:35.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:35 smithi190 ceph-mon[65689]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/354 objects degraded (8.757%), 10 pgs degraded) 2024-03-26T19:00:35.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:35 smithi190 ceph-mon[65689]: Cluster is now healthy 2024-03-26T19:00:36.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:36 smithi102 ceph-mon[78322]: pgmap v147: 65 pgs: 10 peering, 55 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:36.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:36 smithi190 ceph-mon[65689]: pgmap v147: 65 pgs: 10 peering, 55 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:38.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:38 smithi190 ceph-mon[65689]: pgmap v148: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:38.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:38.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:00:38.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:38 smithi102 ceph-mon[78322]: pgmap v148: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:38.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:38.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:00:40.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:40 smithi190 ceph-mon[65689]: pgmap v149: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:40.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:40 smithi102 ceph-mon[78322]: pgmap v149: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:42.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:42 smithi190 ceph-mon[65689]: pgmap v150: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:42.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:42 smithi102 ceph-mon[78322]: pgmap v150: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:43.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:43.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:44.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:44 smithi102 ceph-mon[78322]: pgmap v151: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:44.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:44 smithi190 ceph-mon[65689]: pgmap v151: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:45.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T19:00:45.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T19:00:46.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:46 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T19:00:46.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:46 smithi190 ceph-mon[65689]: Upgrade: osd.5 is safe to restart 2024-03-26T19:00:46.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:46 smithi190 ceph-mon[65689]: pgmap v152: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:46.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:46 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:46.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:46 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T19:00:46.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:46 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:00:46.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:46 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T19:00:46.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:46 smithi102 ceph-mon[78322]: Upgrade: osd.5 is safe to restart 2024-03-26T19:00:46.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:46 smithi102 ceph-mon[78322]: pgmap v152: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:46.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:46.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T19:00:46.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:46 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:00:47.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:47 smithi102 ceph-mon[78322]: Upgrade: Updating osd.5 2024-03-26T19:00:47.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:47 smithi102 ceph-mon[78322]: Deploying daemon osd.5 on smithi190 2024-03-26T19:00:47.747 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:47 smithi190 ceph-mon[65689]: Upgrade: Updating osd.5 2024-03-26T19:00:47.747 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:47 smithi190 ceph-mon[65689]: Deploying daemon osd.5 on smithi190 2024-03-26T19:00:48.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:48 smithi102 ceph-mon[78322]: pgmap v153: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:48.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:48 smithi190 ceph-mon[65689]: pgmap v153: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:49.701 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:49 smithi190 ceph-mon[65689]: pgmap v154: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:49.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:49 smithi190 systemd[1]: Stopping Ceph osd.5 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:00:49.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:49 smithi102 ceph-mon[78322]: pgmap v154: 65 pgs: 65 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:50.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:49 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[50392]: 2024-03-26T19:00:49.707+0000 7fd825e1c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T19:00:50.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:49 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[50392]: 2024-03-26T19:00:49.707+0000 7fd825e1c700 -1 osd.5 75 *** Got signal Terminated *** 2024-03-26T19:00:50.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:49 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[50392]: 2024-03-26T19:00:49.707+0000 7fd825e1c700 -1 osd.5 75 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T19:00:50.778 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:50 smithi102 ceph-mon[78322]: osd.5 marked itself down and dead 2024-03-26T19:00:50.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:50 smithi190 ceph-mon[65689]: osd.5 marked itself down and dead 2024-03-26T19:00:51.281 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:50 smithi190 podman[76279]: 2024-03-26 19:00:50.987609054 +0000 UTC m=+1.388172184 container died 1be709fc0288b6c5815cba7a6474e696423458045c02a535f8e30f20c730211d (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD) 2024-03-26T19:00:51.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:51 smithi190 podman[76279]: 2024-03-26 19:00:51.390815608 +0000 UTC m=+1.791378730 container cleanup 1be709fc0288b6c5815cba7a6474e696423458045c02a535f8e30f20c730211d (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-03-26T19:00:51.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:51 smithi190 bash[76279]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5 2024-03-26T19:00:51.643 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:51 smithi190 ceph-mon[65689]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T19:00:51.644 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:51 smithi190 ceph-mon[65689]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-26T19:00:51.644 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:51 smithi190 ceph-mon[65689]: osdmap e76: 6 total, 5 up, 6 in 2024-03-26T19:00:51.644 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:51 smithi190 ceph-mon[65689]: pgmap v156: 65 pgs: 13 stale+active+clean, 52 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:51.791 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:51 smithi102 ceph-mon[78322]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T19:00:51.792 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:51 smithi102 ceph-mon[78322]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-26T19:00:51.792 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:51 smithi102 ceph-mon[78322]: osdmap e76: 6 total, 5 up, 6 in 2024-03-26T19:00:51.792 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:51 smithi102 ceph-mon[78322]: pgmap v156: 65 pgs: 13 stale+active+clean, 52 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:51.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:51 smithi190 podman[76296]: 2024-03-26 19:00:51.650613656 +0000 UTC m=+0.663330832 container remove 1be709fc0288b6c5815cba7a6474e696423458045c02a535f8e30f20c730211d (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.0, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212) 2024-03-26T19:00:52.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:51 smithi190 podman[76387]: 2024-03-26 19:00:51.874484393 +0000 UTC m=+0.017857254 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:00:52.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:51 smithi190 podman[76387]: 2024-03-26 19:00:51.976426982 +0000 UTC m=+0.119799844 container create e0aa59ff3c0688c046df34cd706e80334a2a37703c024d5cf9ef0d23579138c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T19:00:52.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:52 smithi190 podman[76387]: 2024-03-26 19:00:52.220460957 +0000 UTC m=+0.363833818 container init e0aa59ff3c0688c046df34cd706e80334a2a37703c024d5cf9ef0d23579138c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240320.145742-3605160, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-03-26T19:00:52.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:52 smithi190 podman[76387]: 2024-03-26 19:00:52.223677584 +0000 UTC m=+0.367050445 container start e0aa59ff3c0688c046df34cd706e80334a2a37703c024d5cf9ef0d23579138c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T19:00:52.515 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:52 smithi190 ceph-mon[65689]: osdmap e77: 6 total, 5 up, 6 in 2024-03-26T19:00:52.515 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:52.515 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:00:52.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:52 smithi190 podman[76387]: 2024-03-26 19:00:52.328934467 +0000 UTC m=+0.472307332 container attach e0aa59ff3c0688c046df34cd706e80334a2a37703c024d5cf9ef0d23579138c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T19:00:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:52 smithi102 ceph-mon[78322]: osdmap e77: 6 total, 5 up, 6 in 2024-03-26T19:00:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:52.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:00:52.780 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:52 smithi190 podman[76387]: 2024-03-26 19:00:52.522911896 +0000 UTC m=+0.666284761 container died e0aa59ff3c0688c046df34cd706e80334a2a37703c024d5cf9ef0d23579138c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2) 2024-03-26T19:00:53.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:53 smithi190 podman[76413]: 2024-03-26 19:00:53.082030167 +0000 UTC m=+0.553787177 container remove e0aa59ff3c0688c046df34cd706e80334a2a37703c024d5cf9ef0d23579138c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T19:00:53.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:53 smithi190 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-26T19:00:53.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:53 smithi190 systemd[1]: Stopped Ceph osd.5 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T19:00:53.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:53 smithi190 systemd[1]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.5.service: Consumed 58.801s CPU time. 2024-03-26T19:00:53.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:53 smithi190 systemd[1]: Starting Ceph osd.5 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:00:53.648 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:53 smithi190 ceph-mon[65689]: pgmap v158: 65 pgs: 13 stale+active+clean, 52 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:53.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:53 smithi190 podman[76516]: 2024-03-26 19:00:53.525309193 +0000 UTC m=+0.112817424 container create 3c5fb3a9cbfe83b3a03b562462e94d9a948e85d280d314a5caaa97b402a4c8cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T19:00:53.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:53 smithi190 podman[76516]: 2024-03-26 19:00:53.43076769 +0000 UTC m=+0.018275935 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:00:53.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:53 smithi102 ceph-mon[78322]: pgmap v158: 65 pgs: 13 stale+active+clean, 52 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-26T19:00:54.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:53 smithi190 podman[76516]: 2024-03-26 19:00:53.786044774 +0000 UTC m=+0.373553007 container init 3c5fb3a9cbfe83b3a03b562462e94d9a948e85d280d314a5caaa97b402a4c8cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T19:00:54.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:53 smithi190 podman[76516]: 2024-03-26 19:00:53.78914897 +0000 UTC m=+0.376657210 container start 3c5fb3a9cbfe83b3a03b562462e94d9a948e85d280d314a5caaa97b402a4c8cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-26T19:00:54.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:53 smithi190 podman[76516]: 2024-03-26 19:00:53.886020916 +0000 UTC m=+0.473529150 container attach 3c5fb3a9cbfe83b3a03b562462e94d9a948e85d280d314a5caaa97b402a4c8cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T19:00:54.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate[76531]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T19:00:54.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate[76531]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T19:00:54.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 bash[76516]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T19:00:54.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 bash[76516]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T19:00:54.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 bash[76516]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T19:00:54.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 bash[76516]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T19:00:54.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 bash[76516]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-03-26T19:00:54.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 bash[76516]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T19:00:54.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 bash[76516]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-26T19:00:54.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate[76531]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T19:00:54.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate[76531]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T19:00:54.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate[76531]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-03-26T19:00:54.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate[76531]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T19:00:54.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate[76531]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-26T19:00:54.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 podman[76516]: 2024-03-26 19:00:54.256897419 +0000 UTC m=+0.844405652 container died 3c5fb3a9cbfe83b3a03b562462e94d9a948e85d280d314a5caaa97b402a4c8cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T19:00:55.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:54 smithi190 podman[76662]: 2024-03-26 19:00:54.839254051 +0000 UTC m=+0.577029998 container remove 3c5fb3a9cbfe83b3a03b562462e94d9a948e85d280d314a5caaa97b402a4c8cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-26T19:00:55.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:55 smithi190 podman[76724]: 2024-03-26 19:00:55.048290097 +0000 UTC m=+0.114440008 container create 60afb0e4da18d26a13aca2d153eea7b2cce51f55cf9c741be2670074139f57cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-26T19:00:55.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:55 smithi190 podman[76724]: 2024-03-26 19:00:54.951256338 +0000 UTC m=+0.017406255 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:00:55.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:55 smithi190 ceph-mon[65689]: pgmap v159: 65 pgs: 8 active+undersized+degraded, 3 stale+active+clean, 10 active+undersized, 44 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 23/354 objects degraded (6.497%) 2024-03-26T19:00:55.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:55 smithi190 ceph-mon[65689]: Health check failed: Degraded data redundancy: 23/354 objects degraded (6.497%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T19:00:55.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:55 smithi190 podman[76724]: 2024-03-26 19:00:55.275769663 +0000 UTC m=+0.341919575 container init 60afb0e4da18d26a13aca2d153eea7b2cce51f55cf9c741be2670074139f57cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T19:00:55.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:55 smithi190 podman[76724]: 2024-03-26 19:00:55.278977368 +0000 UTC m=+0.345127280 container start 60afb0e4da18d26a13aca2d153eea7b2cce51f55cf9c741be2670074139f57cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-26T19:00:55.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:55 smithi190 ceph-osd[76743]: -- 172.21.15.190:0/1017710550 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e774f55680 con 0x55e774f1ec00 2024-03-26T19:00:55.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:55 smithi190 bash[76724]: 60afb0e4da18d26a13aca2d153eea7b2cce51f55cf9c741be2670074139f57cc 2024-03-26T19:00:55.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:55 smithi190 systemd[1]: Started Ceph osd.5 for ed4e027c-eba0-11ee-95d0-87774f69a715. 2024-03-26T19:00:55.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:55 smithi102 ceph-mon[78322]: pgmap v159: 65 pgs: 8 active+undersized+degraded, 3 stale+active+clean, 10 active+undersized, 44 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 23/354 objects degraded (6.497%) 2024-03-26T19:00:55.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:55 smithi102 ceph-mon[78322]: Health check failed: Degraded data redundancy: 23/354 objects degraded (6.497%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T19:00:56.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:00:55 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:00:55.820+0000 7f9eb2ee37c0 -1 Falling back to public interface 2024-03-26T19:00:56.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:56 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:56.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:56 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:56.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:56 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:00:56.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:56 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:56.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:56 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:56.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:56 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:00:57.775 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:57 smithi190 ceph-mon[65689]: pgmap v160: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 54/354 objects degraded (15.254%) 2024-03-26T19:00:57.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:57 smithi102 ceph-mon[78322]: pgmap v160: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 54/354 objects degraded (15.254%) 2024-03-26T19:00:58.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.520+0000 7faba32eb700 1 -- 172.21.15.102:0/3145049763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0fe7d0 msgr2=0x7fab9c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:58.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.520+0000 7faba32eb700 1 --2- 172.21.15.102:0/3145049763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0fe7d0 0x7fab9c0febe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fab9800b0a0 tx=0x7fab9800b3b0 comp rx=0 tx=0).stop 2024-03-26T19:00:58.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.520+0000 7faba32eb700 1 -- 172.21.15.102:0/3145049763 shutdown_connections 2024-03-26T19:00:58.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.520+0000 7faba32eb700 1 --2- 172.21.15.102:0/3145049763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab9c0ff9d0 0x7fab9c0ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:58.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.520+0000 7faba32eb700 1 --2- 172.21.15.102:0/3145049763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0fe7d0 0x7fab9c0febe0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:58.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.520+0000 7faba32eb700 1 -- 172.21.15.102:0/3145049763 >> 172.21.15.102:0/3145049763 conn(0x7fab9c0f9d80 msgr2=0x7fab9c0fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:58.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.520+0000 7faba32eb700 1 -- 172.21.15.102:0/3145049763 shutdown_connections 2024-03-26T19:00:58.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.520+0000 7faba32eb700 1 -- 172.21.15.102:0/3145049763 wait complete. 2024-03-26T19:00:58.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba32eb700 1 Processor -- start 2024-03-26T19:00:58.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba32eb700 1 -- start start 2024-03-26T19:00:58.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba32eb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0fe7d0 0x7fab9c195a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:58.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba32eb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab9c0ff9d0 0x7fab9c195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba32eb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c196570 con 0x7fab9c0fe7d0 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba32eb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c1966b0 con 0x7fab9c0ff9d0 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba1087700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0fe7d0 0x7fab9c195a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba1087700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0fe7d0 0x7fab9c195a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52624/0 (socket says 172.21.15.102:52624) 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba1087700 1 -- 172.21.15.102:0/385192752 learned_addr learned my addr 172.21.15.102:0/385192752 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba0886700 1 --2- 172.21.15.102:0/385192752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab9c0ff9d0 0x7fab9c195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba1087700 1 -- 172.21.15.102:0/385192752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab9c0ff9d0 msgr2=0x7fab9c195f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba1087700 1 --2- 172.21.15.102:0/385192752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab9c0ff9d0 0x7fab9c195f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.521+0000 7faba1087700 1 -- 172.21.15.102:0/385192752 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab98009d00 con 0x7fab9c0fe7d0 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.522+0000 7faba1087700 1 --2- 172.21.15.102:0/385192752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0fe7d0 0x7fab9c195a10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fab980077b0 tx=0x7fab98009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.522+0000 7fab927fc700 1 -- 172.21.15.102:0/385192752 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab9800e040 con 0x7fab9c0fe7d0 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.522+0000 7fab927fc700 1 -- 172.21.15.102:0/385192752 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fab98003de0 con 0x7fab9c0fe7d0 2024-03-26T19:00:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.522+0000 7faba32eb700 1 -- 172.21.15.102:0/385192752 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9c19b100 con 0x7fab9c0fe7d0 2024-03-26T19:00:58.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.522+0000 7fab927fc700 1 -- 172.21.15.102:0/385192752 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab9801d7f0 con 0x7fab9c0fe7d0 2024-03-26T19:00:58.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.522+0000 7faba32eb700 1 -- 172.21.15.102:0/385192752 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9c19b5f0 con 0x7fab9c0fe7d0 2024-03-26T19:00:58.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.523+0000 7faba32eb700 1 -- 172.21.15.102:0/385192752 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9c106a90 con 0x7fab9c0fe7d0 2024-03-26T19:00:58.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.523+0000 7fab927fc700 1 -- 172.21.15.102:0/385192752 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fab98004780 con 0x7fab9c0fe7d0 2024-03-26T19:00:58.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.524+0000 7fab927fc700 1 --2- 172.21.15.102:0/385192752 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab8806e460 0x7fab88070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:58.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.524+0000 7fab927fc700 1 -- 172.21.15.102:0/385192752 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fab9808fd90 con 0x7fab9c0fe7d0 2024-03-26T19:00:58.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.524+0000 7faba0886700 1 --2- 172.21.15.102:0/385192752 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab8806e460 0x7fab88070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:58.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.524+0000 7faba0886700 1 --2- 172.21.15.102:0/385192752 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab8806e460 0x7fab88070910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fab8c00bca0 tx=0x7fab8c00b490 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:58.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.529+0000 7fab927fc700 1 -- 172.21.15.102:0/385192752 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab9805cac0 con 0x7fab9c0fe7d0 2024-03-26T19:00:58.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.745+0000 7faba32eb700 1 -- 172.21.15.102:0/385192752 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fab9c0fc060 con 0x7fab8806e460 2024-03-26T19:00:58.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.746+0000 7fab927fc700 1 -- 172.21.15.102:0/385192752 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fab9c0fc060 con 0x7fab8806e460 2024-03-26T19:00:58.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.748+0000 7faba32eb700 1 -- 172.21.15.102:0/385192752 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab8806e460 msgr2=0x7fab88070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:58.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.748+0000 7faba32eb700 1 --2- 172.21.15.102:0/385192752 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab8806e460 0x7fab88070910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fab8c00bca0 tx=0x7fab8c00b490 comp rx=0 tx=0).stop 2024-03-26T19:00:58.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.748+0000 7faba32eb700 1 -- 172.21.15.102:0/385192752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0fe7d0 msgr2=0x7fab9c195a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:58.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.748+0000 7faba32eb700 1 --2- 172.21.15.102:0/385192752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0fe7d0 0x7fab9c195a10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fab980077b0 tx=0x7fab98009760 comp rx=0 tx=0).stop 2024-03-26T19:00:58.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.748+0000 7faba32eb700 1 -- 172.21.15.102:0/385192752 shutdown_connections 2024-03-26T19:00:58.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.748+0000 7faba32eb700 1 --2- 172.21.15.102:0/385192752 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab8806e460 0x7fab88070910 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:58.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.748+0000 7faba32eb700 1 --2- 172.21.15.102:0/385192752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0fe7d0 0x7fab9c195a10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:58.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.748+0000 7faba32eb700 1 --2- 172.21.15.102:0/385192752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab9c0ff9d0 0x7fab9c195f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:58.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.748+0000 7faba32eb700 1 -- 172.21.15.102:0/385192752 >> 172.21.15.102:0/385192752 conn(0x7fab9c0f9d80 msgr2=0x7fab9c0fb950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:58.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.748+0000 7faba32eb700 1 -- 172.21.15.102:0/385192752 shutdown_connections 2024-03-26T19:00:58.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.748+0000 7faba32eb700 1 -- 172.21.15.102:0/385192752 wait complete. 2024-03-26T19:00:58.752 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T19:00:58.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.854+0000 7f366a0aa700 1 -- 172.21.15.102:0/2008728668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3664100bf0 msgr2=0x7f3664102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:58.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.854+0000 7f366a0aa700 1 --2- 172.21.15.102:0/2008728668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3664100bf0 0x7f3664102fd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3654009910 tx=0x7f3654009c20 comp rx=0 tx=0).stop 2024-03-26T19:00:58.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.854+0000 7f366a0aa700 1 -- 172.21.15.102:0/2008728668 shutdown_connections 2024-03-26T19:00:58.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.854+0000 7f366a0aa700 1 --2- 172.21.15.102:0/2008728668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3664103510 0x7f36641058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:58.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.854+0000 7f366a0aa700 1 --2- 172.21.15.102:0/2008728668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3664100bf0 0x7f3664102fd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.854+0000 7f366a0aa700 1 -- 172.21.15.102:0/2008728668 >> 172.21.15.102:0/2008728668 conn(0x7f36640fa610 msgr2=0x7f36640fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.854+0000 7f366a0aa700 1 -- 172.21.15.102:0/2008728668 shutdown_connections 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.854+0000 7f366a0aa700 1 -- 172.21.15.102:0/2008728668 wait complete. 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.854+0000 7f366a0aa700 1 Processor -- start 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.854+0000 7f366a0aa700 1 -- start start 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f366a0aa700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3664100bf0 0x7f3664195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f366a0aa700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3664103510 0x7f3664195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f366a0aa700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36641964e0 con 0x7f3664100bf0 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f366a0aa700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3664196620 con 0x7f3664103510 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f36637fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3664100bf0 0x7f3664195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f36637fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3664100bf0 0x7f3664195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33268/0 (socket says 172.21.15.102:33268) 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f36637fe700 1 -- 172.21.15.102:0/160806566 learned_addr learned my addr 172.21.15.102:0/160806566 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:58.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f3662ffd700 1 --2- 172.21.15.102:0/160806566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3664103510 0x7f3664195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:58.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f36637fe700 1 -- 172.21.15.102:0/160806566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3664103510 msgr2=0x7f3664195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:58.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f36637fe700 1 --2- 172.21.15.102:0/160806566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3664103510 0x7f3664195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:58.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f36637fe700 1 -- 172.21.15.102:0/160806566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36540095f0 con 0x7f3664100bf0 2024-03-26T19:00:58.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f36637fe700 1 --2- 172.21.15.102:0/160806566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3664100bf0 0x7f3664195980 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3654000c00 tx=0x7f365400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:58.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f3660ff9700 1 -- 172.21.15.102:0/160806566 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f365401d070 con 0x7f3664100bf0 2024-03-26T19:00:58.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f3660ff9700 1 -- 172.21.15.102:0/160806566 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3654005dc0 con 0x7f3664100bf0 2024-03-26T19:00:58.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f3660ff9700 1 -- 172.21.15.102:0/160806566 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36540226a0 con 0x7f3664100bf0 2024-03-26T19:00:58.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f366a0aa700 1 -- 172.21.15.102:0/160806566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366419b070 con 0x7f3664100bf0 2024-03-26T19:00:58.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.855+0000 7f366a0aa700 1 -- 172.21.15.102:0/160806566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366419b560 con 0x7f3664100bf0 2024-03-26T19:00:58.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.856+0000 7f366a0aa700 1 -- 172.21.15.102:0/160806566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366418fba0 con 0x7f3664100bf0 2024-03-26T19:00:58.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.857+0000 7f3660ff9700 1 -- 172.21.15.102:0/160806566 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f365400faf0 con 0x7f3664100bf0 2024-03-26T19:00:58.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.857+0000 7f3660ff9700 1 --2- 172.21.15.102:0/160806566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f364c06e460 0x7f364c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:58.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.857+0000 7f3660ff9700 1 -- 172.21.15.102:0/160806566 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f365408f820 con 0x7f3664100bf0 2024-03-26T19:00:58.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.857+0000 7f3662ffd700 1 --2- 172.21.15.102:0/160806566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f364c06e460 0x7f364c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:58.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.858+0000 7f3662ffd700 1 --2- 172.21.15.102:0/160806566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f364c06e460 0x7f364c070910 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3658009c60 tx=0x7f3658009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:58.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:58.862+0000 7f3660ff9700 1 -- 172.21.15.102:0/160806566 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f365405c5d0 con 0x7f3664100bf0 2024-03-26T19:00:59.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.084+0000 7f366a0aa700 1 -- 172.21.15.102:0/160806566 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3664061190 con 0x7f364c06e460 2024-03-26T19:00:59.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.085+0000 7f3660ff9700 1 -- 172.21.15.102:0/160806566 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3664061190 con 0x7f364c06e460 2024-03-26T19:00:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.088+0000 7f366a0aa700 1 -- 172.21.15.102:0/160806566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f364c06e460 msgr2=0x7f364c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.088+0000 7f366a0aa700 1 --2- 172.21.15.102:0/160806566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f364c06e460 0x7f364c070910 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3658009c60 tx=0x7f3658009260 comp rx=0 tx=0).stop 2024-03-26T19:00:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.088+0000 7f366a0aa700 1 -- 172.21.15.102:0/160806566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3664100bf0 msgr2=0x7f3664195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.088+0000 7f366a0aa700 1 --2- 172.21.15.102:0/160806566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3664100bf0 0x7f3664195980 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3654000c00 tx=0x7f365400f690 comp rx=0 tx=0).stop 2024-03-26T19:00:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.088+0000 7f366a0aa700 1 -- 172.21.15.102:0/160806566 shutdown_connections 2024-03-26T19:00:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.088+0000 7f366a0aa700 1 --2- 172.21.15.102:0/160806566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f364c06e460 0x7f364c070910 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.088+0000 7f366a0aa700 1 --2- 172.21.15.102:0/160806566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3664100bf0 0x7f3664195980 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.088+0000 7f366a0aa700 1 --2- 172.21.15.102:0/160806566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3664103510 0x7f3664195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.088+0000 7f366a0aa700 1 -- 172.21.15.102:0/160806566 >> 172.21.15.102:0/160806566 conn(0x7f36640fa610 msgr2=0x7f36640fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.088+0000 7f366a0aa700 1 -- 172.21.15.102:0/160806566 shutdown_connections 2024-03-26T19:00:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.088+0000 7f366a0aa700 1 -- 172.21.15.102:0/160806566 wait complete. 2024-03-26T19:00:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.192+0000 7f231df60700 1 -- 172.21.15.102:0/2743748873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23180ff050 msgr2=0x7f23180ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.192+0000 7f231df60700 1 --2- 172.21.15.102:0/2743748873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23180ff050 0x7f23180ff460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2308009910 tx=0x7f2308009c20 comp rx=0 tx=0).stop 2024-03-26T19:00:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.192+0000 7f231df60700 1 -- 172.21.15.102:0/2743748873 shutdown_connections 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.192+0000 7f231df60700 1 --2- 172.21.15.102:0/2743748873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2318100250 0x7f23181006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.192+0000 7f231df60700 1 --2- 172.21.15.102:0/2743748873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23180ff050 0x7f23180ff460 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.192+0000 7f231df60700 1 -- 172.21.15.102:0/2743748873 >> 172.21.15.102:0/2743748873 conn(0x7f23180fa5e0 msgr2=0x7f23180fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.192+0000 7f231df60700 1 -- 172.21.15.102:0/2743748873 shutdown_connections 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.192+0000 7f231df60700 1 -- 172.21.15.102:0/2743748873 wait complete. 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f231df60700 1 Processor -- start 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f231df60700 1 -- start start 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f231df60700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23180ff050 0x7f2318194890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f231df60700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2318100250 0x7f2318194dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f231df60700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23181953f0 con 0x7f23180ff050 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f231df60700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2318195530 con 0x7f2318100250 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f23177fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23180ff050 0x7f2318194890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f23177fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23180ff050 0x7f2318194890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33282/0 (socket says 172.21.15.102:33282) 2024-03-26T19:00:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f23177fe700 1 -- 172.21.15.102:0/4163129713 learned_addr learned my addr 172.21.15.102:0/4163129713 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f230ffff700 1 --2- 172.21.15.102:0/4163129713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2318100250 0x7f2318194dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f23177fe700 1 -- 172.21.15.102:0/4163129713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2318100250 msgr2=0x7f2318194dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f23177fe700 1 --2- 172.21.15.102:0/4163129713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2318100250 0x7f2318194dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f23177fe700 1 -- 172.21.15.102:0/4163129713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080095f0 con 0x7f23180ff050 2024-03-26T19:00:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.193+0000 7f23177fe700 1 --2- 172.21.15.102:0/4163129713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23180ff050 0x7f2318194890 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f230800bda0 tx=0x7f230800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.194+0000 7f23157fa700 1 -- 172.21.15.102:0/4163129713 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f230801d070 con 0x7f23180ff050 2024-03-26T19:00:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.194+0000 7f231df60700 1 -- 172.21.15.102:0/4163129713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2318199f80 con 0x7f23180ff050 2024-03-26T19:00:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.194+0000 7f23157fa700 1 -- 172.21.15.102:0/4163129713 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f230800f460 con 0x7f23180ff050 2024-03-26T19:00:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.194+0000 7f23157fa700 1 -- 172.21.15.102:0/4163129713 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2308021620 con 0x7f23180ff050 2024-03-26T19:00:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.194+0000 7f231df60700 1 -- 172.21.15.102:0/4163129713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231819a3f0 con 0x7f23180ff050 2024-03-26T19:00:59.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.195+0000 7f231df60700 1 -- 172.21.15.102:0/4163129713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2318107ba0 con 0x7f23180ff050 2024-03-26T19:00:59.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.195+0000 7f23157fa700 1 -- 172.21.15.102:0/4163129713 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f230800f5d0 con 0x7f23180ff050 2024-03-26T19:00:59.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.196+0000 7f23157fa700 1 --2- 172.21.15.102:0/4163129713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f230406e460 0x7f2304070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:59.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.196+0000 7f23157fa700 1 -- 172.21.15.102:0/4163129713 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f230808f6c0 con 0x7f23180ff050 2024-03-26T19:00:59.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.196+0000 7f230ffff700 1 --2- 172.21.15.102:0/4163129713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f230406e460 0x7f2304070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:59.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.196+0000 7f230ffff700 1 --2- 172.21.15.102:0/4163129713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f230406e460 0x7f2304070910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2300005950 tx=0x7f230000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:59.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.200+0000 7f23157fa700 1 -- 172.21.15.102:0/4163129713 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f230805c3f0 con 0x7f23180ff050 2024-03-26T19:00:59.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.410+0000 7f231df60700 1 -- 172.21.15.102:0/4163129713 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2318104ba0 con 0x7f230406e460 2024-03-26T19:00:59.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.421+0000 7f23157fa700 1 -- 172.21.15.102:0/4163129713 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f2318104ba0 con 0x7f230406e460 2024-03-26T19:00:59.414 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (12m) 100s ago 13m 25.0M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (14m) 100s ago 14m 8354k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (13m) 1s ago 13m 8514k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (3m) 100s ago 14m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (3m) 1s ago 13m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (12m) 100s ago 13m 86.6M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (8m) 100s ago 8m 132M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (8m) 100s ago 8m 16.1M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (8m) 1s ago 8m 16.8M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (8m) 1s ago 8m 120M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (5m) 100s ago 15m 504M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (4m) 1s ago 13m 451M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T19:00:59.415 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (4m) 100s ago 15m 46.2M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T19:00:59.416 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (4m) 1s ago 12m 36.6M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T19:00:59.416 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (14m) 100s ago 14m 16.3M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T19:00:59.416 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (13m) 1s ago 13m 15.1M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T19:00:59.416 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (2m) 100s ago 12m 86.0M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T19:00:59.416 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (2m) 100s ago 11m 95.7M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T19:00:59.416 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (106s) 100s ago 11m 11.7M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T19:00:59.416 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (70s) 1s ago 10m 98.2M 2240M 19.0.0-2311-g36051606 497bb87bc5da 03955331c962 2024-03-26T19:00:59.416 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (37s) 1s ago 10m 102M 2240M 19.0.0-2311-g36051606 497bb87bc5da 23e140cd8961 2024-03-26T19:00:59.416 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (4s) 1s ago 10m 12.6M 2240M 19.0.0-2311-g36051606 497bb87bc5da 60afb0e4da18 2024-03-26T19:00:59.416 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (4m) 100s ago 13m 74.3M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T19:00:59.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.423+0000 7f231df60700 1 -- 172.21.15.102:0/4163129713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f230406e460 msgr2=0x7f2304070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.423+0000 7f231df60700 1 --2- 172.21.15.102:0/4163129713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f230406e460 0x7f2304070910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2300005950 tx=0x7f230000b410 comp rx=0 tx=0).stop 2024-03-26T19:00:59.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.423+0000 7f231df60700 1 -- 172.21.15.102:0/4163129713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23180ff050 msgr2=0x7f2318194890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.423+0000 7f231df60700 1 --2- 172.21.15.102:0/4163129713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23180ff050 0x7f2318194890 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f230800bda0 tx=0x7f230800bdd0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.423+0000 7f231df60700 1 -- 172.21.15.102:0/4163129713 shutdown_connections 2024-03-26T19:00:59.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.423+0000 7f231df60700 1 --2- 172.21.15.102:0/4163129713 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f230406e460 0x7f2304070910 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.423+0000 7f231df60700 1 --2- 172.21.15.102:0/4163129713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23180ff050 0x7f2318194890 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.423+0000 7f231df60700 1 --2- 172.21.15.102:0/4163129713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2318100250 0x7f2318194dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.423+0000 7f231df60700 1 -- 172.21.15.102:0/4163129713 >> 172.21.15.102:0/4163129713 conn(0x7f23180fa5e0 msgr2=0x7f2318103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:59.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.424+0000 7f231df60700 1 -- 172.21.15.102:0/4163129713 shutdown_connections 2024-03-26T19:00:59.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.424+0000 7f231df60700 1 -- 172.21.15.102:0/4163129713 wait complete. 2024-03-26T19:00:59.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.530+0000 7fc13efbe700 1 -- 172.21.15.102:0/4225199058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc138100150 msgr2=0x7fc138100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.530+0000 7fc13efbe700 1 --2- 172.21.15.102:0/4225199058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc138100150 0x7fc138100560 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc13000b0a0 tx=0x7fc13000b3b0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.530+0000 7fc13efbe700 1 -- 172.21.15.102:0/4225199058 shutdown_connections 2024-03-26T19:00:59.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.530+0000 7fc13efbe700 1 --2- 172.21.15.102:0/4225199058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc138101350 0x7fc1381017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.530+0000 7fc13efbe700 1 --2- 172.21.15.102:0/4225199058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc138100150 0x7fc138100560 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.530+0000 7fc13efbe700 1 -- 172.21.15.102:0/4225199058 >> 172.21.15.102:0/4225199058 conn(0x7fc1380fb6e0 msgr2=0x7fc1380fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:59.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.530+0000 7fc13efbe700 1 -- 172.21.15.102:0/4225199058 shutdown_connections 2024-03-26T19:00:59.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.530+0000 7fc13efbe700 1 -- 172.21.15.102:0/4225199058 wait complete. 2024-03-26T19:00:59.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:59.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:59.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:59 smithi102 ceph-mon[78322]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:59.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:59 smithi102 ceph-mon[78322]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:59.525 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:00:59 smithi102 ceph-mon[78322]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 54/354 objects degraded (15.254%) 2024-03-26T19:00:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc13efbe700 1 Processor -- start 2024-03-26T19:00:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc13efbe700 1 -- start start 2024-03-26T19:00:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc13efbe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc138100150 0x7fc138074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc13efbe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc138101350 0x7fc138073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc13efbe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc138073680 con 0x7fc138101350 2024-03-26T19:00:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc13efbe700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1380737c0 con 0x7fc138100150 2024-03-26T19:00:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc137fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc138101350 0x7fc138073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc137fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc138101350 0x7fc138073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33300/0 (socket says 172.21.15.102:33300) 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc137fff700 1 -- 172.21.15.102:0/3249998120 learned_addr learned my addr 172.21.15.102:0/3249998120 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc13cd5a700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc138100150 0x7fc138074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc13cd5a700 1 -- 172.21.15.102:0/3249998120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc138101350 msgr2=0x7fc138073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc13cd5a700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc138101350 0x7fc138073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc13cd5a700 1 -- 172.21.15.102:0/3249998120 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc130009d00 con 0x7fc138100150 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc137fff700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc138101350 0x7fc138073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.531+0000 7fc13cd5a700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc138100150 0x7fc138074af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc130015040 tx=0x7fc130003c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.532+0000 7fc135ffb700 1 -- 172.21.15.102:0/3249998120 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc130021070 con 0x7fc138100150 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.532+0000 7fc13efbe700 1 -- 172.21.15.102:0/3249998120 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc138073a40 con 0x7fc138100150 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.532+0000 7fc135ffb700 1 -- 172.21.15.102:0/3249998120 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc1300096a0 con 0x7fc138100150 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.532+0000 7fc135ffb700 1 -- 172.21.15.102:0/3249998120 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc13000e040 con 0x7fc138100150 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.532+0000 7fc13efbe700 1 -- 172.21.15.102:0/3249998120 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc138073f00 con 0x7fc138100150 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.533+0000 7fc13efbe700 1 -- 172.21.15.102:0/3249998120 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc138108ca0 con 0x7fc138100150 2024-03-26T19:00:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.533+0000 7fc135ffb700 1 -- 172.21.15.102:0/3249998120 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc130004250 con 0x7fc138100150 2024-03-26T19:00:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.534+0000 7fc135ffb700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc12406e460 0x7fc124070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.534+0000 7fc135ffb700 1 -- 172.21.15.102:0/3249998120 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc13008f7f0 con 0x7fc138100150 2024-03-26T19:00:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.534+0000 7fc137fff700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc12406e460 0x7fc124070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.534+0000 7fc137fff700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc12406e460 0x7fc124070910 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc128011550 tx=0x7fc128009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:59.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.539+0000 7fc135ffb700 1 -- 172.21.15.102:0/3249998120 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc13005c480 con 0x7fc138100150 2024-03-26T19:00:59.780 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:59.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:00:59.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:59 smithi190 ceph-mon[65689]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:59.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:59 smithi190 ceph-mon[65689]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:00:59.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:00:59 smithi190 ceph-mon[65689]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 54/354 objects degraded (15.254%) 2024-03-26T19:00:59.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.821+0000 7fc13efbe700 1 -- 172.21.15.102:0/3249998120 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc138108e90 con 0x7fc138100150 2024-03-26T19:00:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.821+0000 7fc135ffb700 1 -- 172.21.15.102:0/3249998120 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+700 (secure 0 0 0) 0x7fc13005c010 con 0x7fc138100150 2024-03-26T19:00:59.815 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:00:59.815 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T19:00:59.815 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:00:59.815 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:00:59.815 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T19:00:59.816 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:00:59.816 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:00:59.816 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T19:00:59.816 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 5 2024-03-26T19:00:59.816 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:00:59.816 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T19:00:59.816 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T19:00:59.816 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:00:59.816 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T19:00:59.816 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4, 2024-03-26T19:00:59.816 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 9 2024-03-26T19:00:59.817 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T19:00:59.817 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:00:59.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.823+0000 7fc13efbe700 1 -- 172.21.15.102:0/3249998120 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc12406e460 msgr2=0x7fc124070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.823+0000 7fc13efbe700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc12406e460 0x7fc124070910 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc128011550 tx=0x7fc128009260 comp rx=0 tx=0).stop 2024-03-26T19:00:59.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.823+0000 7fc13efbe700 1 -- 172.21.15.102:0/3249998120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc138100150 msgr2=0x7fc138074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.823+0000 7fc13efbe700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc138100150 0x7fc138074af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc130015040 tx=0x7fc130003c30 comp rx=0 tx=0).stop 2024-03-26T19:00:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.823+0000 7fc13efbe700 1 -- 172.21.15.102:0/3249998120 shutdown_connections 2024-03-26T19:00:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.823+0000 7fc13efbe700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc138100150 0x7fc138074af0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.823+0000 7fc13efbe700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc12406e460 0x7fc124070910 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.823+0000 7fc13efbe700 1 --2- 172.21.15.102:0/3249998120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc138101350 0x7fc138073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.823+0000 7fc13efbe700 1 -- 172.21.15.102:0/3249998120 >> 172.21.15.102:0/3249998120 conn(0x7fc1380fb6e0 msgr2=0x7fc138104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.824+0000 7fc13efbe700 1 -- 172.21.15.102:0/3249998120 shutdown_connections 2024-03-26T19:00:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.824+0000 7fc13efbe700 1 -- 172.21.15.102:0/3249998120 wait complete. 2024-03-26T19:00:59.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.927+0000 7f5df48ac700 1 -- 172.21.15.102:0/2075785405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5df0100c40 msgr2=0x7f5df0103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.927+0000 7f5df48ac700 1 --2- 172.21.15.102:0/2075785405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5df0100c40 0x7f5df0103020 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5de800b0a0 tx=0x7f5de800b3b0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.927+0000 7f5df48ac700 1 -- 172.21.15.102:0/2075785405 shutdown_connections 2024-03-26T19:00:59.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.927+0000 7f5df48ac700 1 --2- 172.21.15.102:0/2075785405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5df0103560 0x7f5df0105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.927+0000 7f5df48ac700 1 --2- 172.21.15.102:0/2075785405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5df0100c40 0x7f5df0103020 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.927+0000 7f5df48ac700 1 -- 172.21.15.102:0/2075785405 >> 172.21.15.102:0/2075785405 conn(0x7f5df00fa600 msgr2=0x7f5df00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:00:59.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.927+0000 7f5df48ac700 1 -- 172.21.15.102:0/2075785405 shutdown_connections 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.927+0000 7f5df48ac700 1 -- 172.21.15.102:0/2075785405 wait complete. 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5df48ac700 1 Processor -- start 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5df48ac700 1 -- start start 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5df48ac700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5df0100c40 0x7f5df0195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5df48ac700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5df0103560 0x7f5df0195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5df48ac700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df01964b0 con 0x7f5df0103560 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5df48ac700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df01965f0 con 0x7f5df0100c40 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5dedd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5df0103560 0x7f5df0195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5dedd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5df0103560 0x7f5df0195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33318/0 (socket says 172.21.15.102:33318) 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5dedd9b700 1 -- 172.21.15.102:0/2666233863 learned_addr learned my addr 172.21.15.102:0/2666233863 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5dee59c700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5df0100c40 0x7f5df0195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5dedd9b700 1 -- 172.21.15.102:0/2666233863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5df0100c40 msgr2=0x7f5df0195950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5dedd9b700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5df0100c40 0x7f5df0195950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:00:59.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5dedd9b700 1 -- 172.21.15.102:0/2666233863 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de8009d00 con 0x7f5df0103560 2024-03-26T19:00:59.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5dee59c700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5df0100c40 0x7f5df0195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T19:00:59.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.928+0000 7f5dedd9b700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5df0103560 0x7f5df0195e90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800e920 tx=0x7f5dd800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:59.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.929+0000 7f5de77fe700 1 -- 172.21.15.102:0/2666233863 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd800c9d0 con 0x7f5df0103560 2024-03-26T19:00:59.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.929+0000 7f5de77fe700 1 -- 172.21.15.102:0/2666233863 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5dd8010430 con 0x7f5df0103560 2024-03-26T19:00:59.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.929+0000 7f5df48ac700 1 -- 172.21.15.102:0/2666233863 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df019b0a0 con 0x7f5df0103560 2024-03-26T19:00:59.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.929+0000 7f5de77fe700 1 -- 172.21.15.102:0/2666233863 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd8018650 con 0x7f5df0103560 2024-03-26T19:00:59.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.929+0000 7f5df48ac700 1 -- 172.21.15.102:0/2666233863 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df019b670 con 0x7f5df0103560 2024-03-26T19:00:59.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.930+0000 7f5df48ac700 1 -- 172.21.15.102:0/2666233863 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df018fb70 con 0x7f5df0103560 2024-03-26T19:00:59.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.930+0000 7f5de77fe700 1 -- 172.21.15.102:0/2666233863 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5dd80105a0 con 0x7f5df0103560 2024-03-26T19:00:59.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.931+0000 7f5de77fe700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5ddc06e460 0x7f5ddc070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:00:59.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.931+0000 7f5de77fe700 1 -- 172.21.15.102:0/2666233863 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5dd8014070 con 0x7f5df0103560 2024-03-26T19:00:59.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.931+0000 7f5dee59c700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5ddc06e460 0x7f5ddc070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:00:59.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.931+0000 7f5dee59c700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5ddc06e460 0x7f5ddc070910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009fd0 tx=0x7f5de80060c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:00:59.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:00:59.935+0000 7f5de77fe700 1 -- 172.21.15.102:0/2666233863 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5dd8053740 con 0x7f5df0103560 2024-03-26T19:01:00.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.186+0000 7f5df48ac700 1 -- 172.21.15.102:0/2666233863 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5df002cc30 con 0x7f5df0103560 2024-03-26T19:01:00.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.186+0000 7f5de77fe700 1 -- 172.21.15.102:0/2666233863 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1905 (secure 0 0 0) 0x7f5dd805b380 con 0x7f5df0103560 2024-03-26T19:01:00.180 INFO:teuthology.orchestra.run.smithi102.stdout:e21 2024-03-26T19:01:00.180 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T19:01:00.180 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:01:00.180 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T19:01:00.180 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 21 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T18:58:42.198340+0000 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T19:01:00.181 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T19:01:00.182 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 14490 members: 24283,14490 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:01:00.183 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:01:00.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.188+0000 7f5df48ac700 1 -- 172.21.15.102:0/2666233863 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5ddc06e460 msgr2=0x7f5ddc070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:00.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.188+0000 7f5df48ac700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5ddc06e460 0x7f5ddc070910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009fd0 tx=0x7f5de80060c0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.188+0000 7f5df48ac700 1 -- 172.21.15.102:0/2666233863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5df0103560 msgr2=0x7f5df0195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:00.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.188+0000 7f5df48ac700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5df0103560 0x7f5df0195e90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800e920 tx=0x7f5dd800ec30 comp rx=0 tx=0).stop 2024-03-26T19:01:00.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.188+0000 7f5df48ac700 1 -- 172.21.15.102:0/2666233863 shutdown_connections 2024-03-26T19:01:00.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.188+0000 7f5df48ac700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5df0100c40 0x7f5df0195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.188+0000 7f5df48ac700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5ddc06e460 0x7f5ddc070910 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.188+0000 7f5df48ac700 1 --2- 172.21.15.102:0/2666233863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5df0103560 0x7f5df0195e90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.188+0000 7f5df48ac700 1 -- 172.21.15.102:0/2666233863 >> 172.21.15.102:0/2666233863 conn(0x7f5df00fa600 msgr2=0x7f5df00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:00.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.188+0000 7f5df48ac700 1 -- 172.21.15.102:0/2666233863 shutdown_connections 2024-03-26T19:01:00.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.188+0000 7f5df48ac700 1 -- 172.21.15.102:0/2666233863 wait complete. 2024-03-26T19:01:00.185 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 21 2024-03-26T19:01:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.293+0000 7fe4dcabb700 1 -- 172.21.15.102:0/792989405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4d80ff060 msgr2=0x7fe4d80ff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.293+0000 7fe4dcabb700 1 --2- 172.21.15.102:0/792989405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4d80ff060 0x7fe4d80ff470 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009910 tx=0x7fe4cc009c20 comp rx=0 tx=0).stop 2024-03-26T19:01:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.293+0000 7fe4dcabb700 1 -- 172.21.15.102:0/792989405 shutdown_connections 2024-03-26T19:01:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.293+0000 7fe4dcabb700 1 --2- 172.21.15.102:0/792989405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4d8100260 0x7fe4d81006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.293+0000 7fe4dcabb700 1 --2- 172.21.15.102:0/792989405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4d80ff060 0x7fe4d80ff470 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.293+0000 7fe4dcabb700 1 -- 172.21.15.102:0/792989405 >> 172.21.15.102:0/792989405 conn(0x7fe4d80fa610 msgr2=0x7fe4d80fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:00.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.293+0000 7fe4dcabb700 1 -- 172.21.15.102:0/792989405 shutdown_connections 2024-03-26T19:01:00.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.293+0000 7fe4dcabb700 1 -- 172.21.15.102:0/792989405 wait complete. 2024-03-26T19:01:00.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.294+0000 7fe4dcabb700 1 Processor -- start 2024-03-26T19:01:00.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.294+0000 7fe4dcabb700 1 -- start start 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.294+0000 7fe4dcabb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4d80ff060 0x7fe4d8102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.294+0000 7fe4dcabb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4d8100260 0x7fe4d8101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.294+0000 7fe4dcabb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d8101b80 con 0x7fe4d80ff060 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.294+0000 7fe4dcabb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d8101cc0 con 0x7fe4d8100260 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.294+0000 7fe4d659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4d80ff060 0x7fe4d8102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.294+0000 7fe4d659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4d80ff060 0x7fe4d8102ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33342/0 (socket says 172.21.15.102:33342) 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.294+0000 7fe4d659c700 1 -- 172.21.15.102:0/1434742 learned_addr learned my addr 172.21.15.102:0/1434742 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.294+0000 7fe4d5d9b700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4d8100260 0x7fe4d8101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.295+0000 7fe4d659c700 1 -- 172.21.15.102:0/1434742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4d8100260 msgr2=0x7fe4d8101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.295+0000 7fe4d659c700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4d8100260 0x7fe4d8101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.295+0000 7fe4d659c700 1 -- 172.21.15.102:0/1434742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc0095f0 con 0x7fe4d80ff060 2024-03-26T19:01:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.295+0000 7fe4d5d9b700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4d8100260 0x7fe4d8101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T19:01:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.295+0000 7fe4d659c700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4d80ff060 0x7fe4d8102ff0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc000c00 tx=0x7fe4cc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.295+0000 7fe4c77fe700 1 -- 172.21.15.102:0/1434742 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4cc01d070 con 0x7fe4d80ff060 2024-03-26T19:01:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.295+0000 7fe4c77fe700 1 -- 172.21.15.102:0/1434742 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe4cc005dc0 con 0x7fe4d80ff060 2024-03-26T19:01:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.295+0000 7fe4c77fe700 1 -- 172.21.15.102:0/1434742 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4cc022470 con 0x7fe4d80ff060 2024-03-26T19:01:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.295+0000 7fe4dcabb700 1 -- 172.21.15.102:0/1434742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d8101f40 con 0x7fe4d80ff060 2024-03-26T19:01:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.295+0000 7fe4dcabb700 1 -- 172.21.15.102:0/1434742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d8102430 con 0x7fe4d80ff060 2024-03-26T19:01:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.296+0000 7fe4dcabb700 1 -- 172.21.15.102:0/1434742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d8107bb0 con 0x7fe4d80ff060 2024-03-26T19:01:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.297+0000 7fe4c77fe700 1 -- 172.21.15.102:0/1434742 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe4cc003810 con 0x7fe4d80ff060 2024-03-26T19:01:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.297+0000 7fe4c77fe700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe4c006e460 0x7fe4c0070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.297+0000 7fe4c77fe700 1 -- 172.21.15.102:0/1434742 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe4cc08f820 con 0x7fe4d80ff060 2024-03-26T19:01:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.297+0000 7fe4d5d9b700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe4c006e460 0x7fe4c0070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.297+0000 7fe4d5d9b700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe4c006e460 0x7fe4c0070910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8011490 tx=0x7fe4c8009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:00.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.302+0000 7fe4c77fe700 1 -- 172.21.15.102:0/1434742 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe4cc05c5d0 con 0x7fe4d80ff060 2024-03-26T19:01:00.452 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:01:00 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:01:00.265+0000 7f9eb2ee37c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-26T19:01:00.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.520+0000 7fe4dcabb700 1 -- 172.21.15.102:0/1434742 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4d8069480 con 0x7fe4c006e460 2024-03-26T19:01:00.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.521+0000 7fe4c77fe700 1 -- 172.21.15.102:0/1434742 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe4d8069480 con 0x7fe4c006e460 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: "mon", 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: "osd" 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "12/23 daemons upgraded", 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T19:01:00.515 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T19:01:00.516 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:01:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.523+0000 7fe4dcabb700 1 -- 172.21.15.102:0/1434742 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe4c006e460 msgr2=0x7fe4c0070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.523+0000 7fe4dcabb700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe4c006e460 0x7fe4c0070910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8011490 tx=0x7fe4c8009260 comp rx=0 tx=0).stop 2024-03-26T19:01:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.523+0000 7fe4dcabb700 1 -- 172.21.15.102:0/1434742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4d80ff060 msgr2=0x7fe4d8102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.523+0000 7fe4dcabb700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4d80ff060 0x7fe4d8102ff0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc000c00 tx=0x7fe4cc00f690 comp rx=0 tx=0).stop 2024-03-26T19:01:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.523+0000 7fe4dcabb700 1 -- 172.21.15.102:0/1434742 shutdown_connections 2024-03-26T19:01:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.523+0000 7fe4dcabb700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe4c006e460 0x7fe4c0070910 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.523+0000 7fe4dcabb700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4d80ff060 0x7fe4d8102ff0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.523+0000 7fe4dcabb700 1 --2- 172.21.15.102:0/1434742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4d8100260 0x7fe4d8101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.523+0000 7fe4dcabb700 1 -- 172.21.15.102:0/1434742 >> 172.21.15.102:0/1434742 conn(0x7fe4d80fa610 msgr2=0x7fe4d8068d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:00.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.523+0000 7fe4dcabb700 1 -- 172.21.15.102:0/1434742 shutdown_connections 2024-03-26T19:01:00.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.523+0000 7fe4dcabb700 1 -- 172.21.15.102:0/1434742 wait complete. 2024-03-26T19:01:00.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 -- 172.21.15.102:0/1011844421 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c4100270 msgr2=0x7fc8c41006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:00.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 --2- 172.21.15.102:0/1011844421 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c4100270 0x7fc8c41006c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009910 tx=0x7fc8b4009c20 comp rx=0 tx=0).stop 2024-03-26T19:01:00.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 -- 172.21.15.102:0/1011844421 shutdown_connections 2024-03-26T19:01:00.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 --2- 172.21.15.102:0/1011844421 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c4100270 0x7fc8c41006c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 --2- 172.21.15.102:0/1011844421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c40ff070 0x7fc8c40ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 -- 172.21.15.102:0/1011844421 >> 172.21.15.102:0/1011844421 conn(0x7fc8c40fa600 msgr2=0x7fc8c40fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 -- 172.21.15.102:0/1011844421 shutdown_connections 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 -- 172.21.15.102:0/1011844421 wait complete. 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 Processor -- start 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 -- start start 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c40ff070 0x7fc8c4103000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c4100270 0x7fc8c4101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c4101b90 con 0x7fc8c4100270 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.626+0000 7fc8cb517700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c4101cd0 con 0x7fc8c40ff070 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8c8ab2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c4100270 0x7fc8c4101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8c8ab2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c4100270 0x7fc8c4101650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33354/0 (socket says 172.21.15.102:33354) 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8c8ab2700 1 -- 172.21.15.102:0/3024396961 learned_addr learned my addr 172.21.15.102:0/3024396961 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8c92b3700 1 --2- 172.21.15.102:0/3024396961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c40ff070 0x7fc8c4103000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8c8ab2700 1 -- 172.21.15.102:0/3024396961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c40ff070 msgr2=0x7fc8c4103000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:00.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8c8ab2700 1 --2- 172.21.15.102:0/3024396961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c40ff070 0x7fc8c4103000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8c8ab2700 1 -- 172.21.15.102:0/3024396961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b40095f0 con 0x7fc8c4100270 2024-03-26T19:01:00.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8c8ab2700 1 --2- 172.21.15.102:0/3024396961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c4100270 0x7fc8c4101650 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4000c00 tx=0x7fc8b400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:00.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8ba7fc700 1 -- 172.21.15.102:0/3024396961 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8b401d070 con 0x7fc8c4100270 2024-03-26T19:01:00.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8ba7fc700 1 -- 172.21.15.102:0/3024396961 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc8b4005dc0 con 0x7fc8c4100270 2024-03-26T19:01:00.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8cb517700 1 -- 172.21.15.102:0/3024396961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c4101f50 con 0x7fc8c4100270 2024-03-26T19:01:00.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8ba7fc700 1 -- 172.21.15.102:0/3024396961 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8b40226a0 con 0x7fc8c4100270 2024-03-26T19:01:00.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.627+0000 7fc8cb517700 1 -- 172.21.15.102:0/3024396961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c4102440 con 0x7fc8c4100270 2024-03-26T19:01:00.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.628+0000 7fc8cb517700 1 -- 172.21.15.102:0/3024396961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c4107bc0 con 0x7fc8c4100270 2024-03-26T19:01:00.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.629+0000 7fc8ba7fc700 1 -- 172.21.15.102:0/3024396961 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc8b400faf0 con 0x7fc8c4100270 2024-03-26T19:01:00.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.629+0000 7fc8ba7fc700 1 --2- 172.21.15.102:0/3024396961 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc8b006e460 0x7fc8b0070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:00.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.629+0000 7fc8ba7fc700 1 -- 172.21.15.102:0/3024396961 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc8b408f820 con 0x7fc8c4100270 2024-03-26T19:01:00.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.629+0000 7fc8c92b3700 1 --2- 172.21.15.102:0/3024396961 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc8b006e460 0x7fc8b0070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:00.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.629+0000 7fc8c92b3700 1 --2- 172.21.15.102:0/3024396961 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc8b006e460 0x7fc8b0070910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc8c41000d0 tx=0x7fc8c0009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:00.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.634+0000 7fc8ba7fc700 1 -- 172.21.15.102:0/3024396961 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc8b405c5d0 con 0x7fc8c4100270 2024-03-26T19:01:00.780 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:01:00 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:01:00.674+0000 7f9eb2ee37c0 -1 osd.5 75 log_to_monitors true 2024-03-26T19:01:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:00 smithi190 ceph-mon[65689]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:00 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3249998120' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:00.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:00 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2666233863' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:01:00.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:00 smithi102 ceph-mon[78322]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:00.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:00.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:00.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:00 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3249998120' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:00.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:00 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2666233863' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:01:00.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.915+0000 7fc8cb517700 1 -- 172.21.15.102:0/3024396961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc8c4107db0 con 0x7fc8c4100270 2024-03-26T19:01:00.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.915+0000 7fc8ba7fc700 1 -- 172.21.15.102:0/3024396961 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1290 (secure 0 0 0) 0x7fc8b405c160 con 0x7fc8c4100270 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 54/354 objects degraded (15.254%), 14 pgs degraded 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout: osd.5 (root=default,host=smithi190) is down 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 54/354 objects degraded (15.254%), 14 pgs degraded 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.2 is active+undersized+degraded, acting [1,0] 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.3 is active+undersized+degraded, acting [2,1] 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.8 is active+undersized+degraded, acting [3,0] 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.b is active+undersized+degraded, acting [3,4] 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-03-26T19:01:00.909 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.14 is active+undersized+degraded, acting [3,4] 2024-03-26T19:01:00.910 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.16 is active+undersized+degraded, acting [3,2] 2024-03-26T19:01:00.910 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.17 is active+undersized+degraded, acting [2,1] 2024-03-26T19:01:00.910 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.18 is active+undersized+degraded, acting [4,3] 2024-03-26T19:01:00.910 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1a is active+undersized+degraded, acting [3,4] 2024-03-26T19:01:00.910 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1b is active+undersized+degraded, acting [1,0] 2024-03-26T19:01:00.910 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1c is active+undersized+degraded, acting [4,2] 2024-03-26T19:01:00.910 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1d is active+undersized+degraded, acting [3,0] 2024-03-26T19:01:00.910 INFO:teuthology.orchestra.run.smithi102.stdout: pg 2.1e is active+undersized+degraded, acting [2,0] 2024-03-26T19:01:00.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.917+0000 7fc8cb517700 1 -- 172.21.15.102:0/3024396961 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc8b006e460 msgr2=0x7fc8b0070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:00.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.917+0000 7fc8cb517700 1 --2- 172.21.15.102:0/3024396961 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc8b006e460 0x7fc8b0070910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc8c41000d0 tx=0x7fc8c0009260 comp rx=0 tx=0).stop 2024-03-26T19:01:00.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.917+0000 7fc8cb517700 1 -- 172.21.15.102:0/3024396961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c4100270 msgr2=0x7fc8c4101650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:00.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.917+0000 7fc8cb517700 1 --2- 172.21.15.102:0/3024396961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c4100270 0x7fc8c4101650 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4000c00 tx=0x7fc8b400f690 comp rx=0 tx=0).stop 2024-03-26T19:01:00.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.917+0000 7fc8cb517700 1 -- 172.21.15.102:0/3024396961 shutdown_connections 2024-03-26T19:01:00.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.917+0000 7fc8cb517700 1 --2- 172.21.15.102:0/3024396961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c40ff070 0x7fc8c4103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.917+0000 7fc8cb517700 1 --2- 172.21.15.102:0/3024396961 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc8b006e460 0x7fc8b0070910 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.917+0000 7fc8cb517700 1 --2- 172.21.15.102:0/3024396961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c4100270 0x7fc8c4101650 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:00.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.917+0000 7fc8cb517700 1 -- 172.21.15.102:0/3024396961 >> 172.21.15.102:0/3024396961 conn(0x7fc8c40fa600 msgr2=0x7fc8c4068d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:00.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.917+0000 7fc8cb517700 1 -- 172.21.15.102:0/3024396961 shutdown_connections 2024-03-26T19:01:00.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:00.917+0000 7fc8cb517700 1 -- 172.21.15.102:0/3024396961 wait complete. 2024-03-26T19:01:01.704 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:01 smithi190 ceph-mon[65689]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:01.705 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:01 smithi190 ceph-mon[65689]: from='osd.5 [v2:172.21.15.190:6816/3222089930,v1:172.21.15.190:6817/3222089930]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T19:01:01.705 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:01 smithi190 ceph-mon[65689]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T19:01:01.705 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:01 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3024396961' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:01:01.705 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:01 smithi190 ceph-mon[65689]: pgmap v162: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 54/354 objects degraded (15.254%) 2024-03-26T19:01:01.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:01 smithi102 ceph-mon[78322]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:01.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:01 smithi102 ceph-mon[78322]: from='osd.5 [v2:172.21.15.190:6816/3222089930,v1:172.21.15.190:6817/3222089930]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T19:01:01.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:01 smithi102 ceph-mon[78322]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T19:01:01.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:01 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3024396961' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:01:01.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:01 smithi102 ceph-mon[78322]: pgmap v162: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 300 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 54/354 objects degraded (15.254%) 2024-03-26T19:01:02.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:02 smithi102 ceph-mon[78322]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T19:01:02.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:02 smithi102 ceph-mon[78322]: osdmap e78: 6 total, 5 up, 6 in 2024-03-26T19:01:02.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:02 smithi102 ceph-mon[78322]: from='osd.5 [v2:172.21.15.190:6816/3222089930,v1:172.21.15.190:6817/3222089930]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T19:01:02.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:02 smithi102 ceph-mon[78322]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T19:01:02.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:02 smithi102 ceph-mon[78322]: from='osd.5 ' entity='osd.5' 2024-03-26T19:01:02.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:01:02 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:01:02.422+0000 7f9ea0333700 -1 osd.5 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T19:01:02.676 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:02 smithi190 ceph-mon[65689]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T19:01:02.676 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:02 smithi190 ceph-mon[65689]: osdmap e78: 6 total, 5 up, 6 in 2024-03-26T19:01:02.676 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:02 smithi190 ceph-mon[65689]: from='osd.5 [v2:172.21.15.190:6816/3222089930,v1:172.21.15.190:6817/3222089930]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T19:01:02.677 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:02 smithi190 ceph-mon[65689]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-26T19:01:02.677 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:02 smithi190 ceph-mon[65689]: from='osd.5 ' entity='osd.5' 2024-03-26T19:01:04.034 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:04.034 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:04.034 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:04.034 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:04.034 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:04.034 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:04.034 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:04.034 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:04.034 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: Upgrade: Setting container_image for all osd 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-26T19:01:04.035 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: Upgrade: Setting require_osd_release to 19 squid 2024-03-26T19:01:04.036 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-26T19:01:04.036 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: pgmap v164: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 54/354 objects degraded (15.254%) 2024-03-26T19:01:04.036 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T19:01:04.036 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-26T19:01:04.036 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-26T19:01:04.036 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: osd.5 [v2:172.21.15.190:6816/3222089930,v1:172.21.15.190:6817/3222089930] boot 2024-03-26T19:01:04.036 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: osdmap e79: 6 total, 6 up, 6 in 2024-03-26T19:01:04.036 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T19:01:04.036 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:04.036 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-26T19:01:04.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:04.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:04.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:04.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:04.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:04.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:04.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:04.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: Upgrade: Setting container_image for all osd 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-26T19:01:04.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: Upgrade: Setting require_osd_release to 19 squid 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: pgmap v164: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 54/354 objects degraded (15.254%) 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: osd.5 [v2:172.21.15.190:6816/3222089930,v1:172.21.15.190:6817/3222089930] boot 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: osdmap e79: 6 total, 6 up, 6 in 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:04.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-26T19:01:05.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: Upgrade: Scaling down filesystem cephfs 2024-03-26T19:01:05.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: Health check update: Degraded data redundancy: 54/354 objects degraded (15.254%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T19:01:05.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: stopping daemon mds.cephfs.smithi190.vykizb 2024-03-26T19:01:05.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:stopping} 2 up:standby 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-03-26T19:01:05.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:04 smithi190 ceph-mon[65689]: osdmap e80: 6 total, 6 up, 6 in 2024-03-26T19:01:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: Upgrade: Scaling down filesystem cephfs 2024-03-26T19:01:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: Health check update: Degraded data redundancy: 54/354 objects degraded (15.254%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T19:01:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: stopping daemon mds.cephfs.smithi190.vykizb 2024-03-26T19:01:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-26T19:01:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:stopping} 2 up:standby 2024-03-26T19:01:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:05.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:05.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:05.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:05.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:05.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-03-26T19:01:05.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:04 smithi102 ceph-mon[78322]: osdmap e80: 6 total, 6 up, 6 in 2024-03-26T19:01:06.031 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:05 smithi190 ceph-mon[65689]: pgmap v167: 65 pgs: 3 peering, 13 active+undersized+degraded, 12 active+undersized, 37 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 50/354 objects degraded (14.124%) 2024-03-26T19:01:06.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:05 smithi102 ceph-mon[78322]: pgmap v167: 65 pgs: 3 peering, 13 active+undersized+degraded, 12 active+undersized, 37 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 50/354 objects degraded (14.124%) 2024-03-26T19:01:08.530 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:08 smithi190 ceph-mon[65689]: pgmap v168: 65 pgs: 3 peering, 3 active+undersized+degraded, 2 active+undersized, 57 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 6/354 objects degraded (1.695%) 2024-03-26T19:01:08.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:08.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:01:08.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:08 smithi102 ceph-mon[78322]: pgmap v168: 65 pgs: 3 peering, 3 active+undersized+degraded, 2 active+undersized, 57 active+clean; 300 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 6/354 objects degraded (1.695%) 2024-03-26T19:01:08.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:08.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:01:09.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:09 smithi190 ceph-mon[65689]: Health check update: Degraded data redundancy: 6/354 objects degraded (1.695%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T19:01:09.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:09 smithi102 ceph-mon[78322]: Health check update: Degraded data redundancy: 6/354 objects degraded (1.695%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T19:01:10.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:10 smithi102 ceph-mon[78322]: pgmap v169: 65 pgs: 2 peering, 63 active+clean; 301 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 132 KiB/s wr, 48 op/s 2024-03-26T19:01:10.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:10 smithi102 ceph-mon[78322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/354 objects degraded (1.695%), 3 pgs degraded) 2024-03-26T19:01:10.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:10 smithi102 ceph-mon[78322]: Cluster is now healthy 2024-03-26T19:01:10.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:10 smithi190 ceph-mon[65689]: pgmap v169: 65 pgs: 2 peering, 63 active+clean; 301 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 132 KiB/s wr, 48 op/s 2024-03-26T19:01:10.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:10 smithi190 ceph-mon[65689]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/354 objects degraded (1.695%), 3 pgs degraded) 2024-03-26T19:01:10.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:10 smithi190 ceph-mon[65689]: Cluster is now healthy 2024-03-26T19:01:12.307 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:12 smithi102 ceph-mon[78322]: pgmap v170: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 208 KiB/s wr, 70 op/s 2024-03-26T19:01:12.582 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:12 smithi190 ceph-mon[65689]: pgmap v170: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 208 KiB/s wr, 70 op/s 2024-03-26T19:01:13.574 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:13 smithi190 ceph-mon[65689]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:stopping} 2 up:standby 2024-03-26T19:01:13.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:13 smithi102 ceph-mon[78322]: fsmap cephfs:2 {0=cephfs.smithi102.hrbrsp=up:active,1=cephfs.smithi190.vykizb=up:stopping} 2 up:standby 2024-03-26T19:01:14.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:14 smithi102 ceph-mon[78322]: pgmap v171: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 169 KiB/s wr, 57 op/s 2024-03-26T19:01:14.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:14.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:14.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:14.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:14.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:14.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:14.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:14.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:14.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:14 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:14.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:14 smithi190 ceph-mon[65689]: pgmap v171: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 169 KiB/s wr, 57 op/s 2024-03-26T19:01:14.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:14.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:14.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:14.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:14.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:14.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:14.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:14.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:14.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:14 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:15.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:15 smithi102 ceph-mon[78322]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-03-26T19:01:15.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:15 smithi190 ceph-mon[65689]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-03-26T19:01:16.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:16 smithi102 ceph-mon[78322]: pgmap v172: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 226 KiB/s wr, 91 op/s 2024-03-26T19:01:16.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:16 smithi190 ceph-mon[65689]: pgmap v172: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 226 KiB/s wr, 91 op/s 2024-03-26T19:01:18.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:18 smithi102 ceph-mon[78322]: pgmap v173: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 KiB/s wr, 82 op/s 2024-03-26T19:01:18.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:18 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:18.630 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:18 smithi190 ceph-mon[65689]: pgmap v173: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 KiB/s wr, 82 op/s 2024-03-26T19:01:18.631 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:18 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:20.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:20 smithi102 ceph-mon[78322]: pgmap v174: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 KiB/s wr, 82 op/s 2024-03-26T19:01:20.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:20 smithi190 ceph-mon[65689]: pgmap v174: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 KiB/s wr, 82 op/s 2024-03-26T19:01:22.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:22 smithi102 ceph-mon[78322]: pgmap v175: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 121 KiB/s wr, 51 op/s 2024-03-26T19:01:22.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:22.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:01:22.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:22 smithi190 ceph-mon[65689]: pgmap v175: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 121 KiB/s wr, 51 op/s 2024-03-26T19:01:22.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:22.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:01:24.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:24 smithi102 ceph-mon[78322]: pgmap v176: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 68 KiB/s wr, 35 op/s 2024-03-26T19:01:24.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:24 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:24.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:24 smithi190 ceph-mon[65689]: pgmap v176: 65 pgs: 65 active+clean; 302 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 68 KiB/s wr, 35 op/s 2024-03-26T19:01:24.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:24 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:25.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:25.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:25.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:25.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:25.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:25.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:25.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:25.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:25 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:25.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:25.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:25.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:25.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:25.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:25.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:25.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:25.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:25 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:26.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:26 smithi102 ceph-mon[78322]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-03-26T19:01:26.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:26 smithi102 ceph-mon[78322]: pgmap v177: 65 pgs: 65 active+clean; 286 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 68 KiB/s wr, 37 op/s 2024-03-26T19:01:26.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:26 smithi190 ceph-mon[65689]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-03-26T19:01:26.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:26 smithi190 ceph-mon[65689]: pgmap v177: 65 pgs: 65 active+clean; 286 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 68 KiB/s wr, 37 op/s 2024-03-26T19:01:28.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:28 smithi102 ceph-mon[78322]: pgmap v178: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T19:01:28.744 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:28 smithi190 ceph-mon[65689]: pgmap v178: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T19:01:29.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:29 smithi190 ceph-mon[65689]: pgmap v179: 65 pgs: 65 active+clean; 254 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T19:01:29.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:29 smithi102 ceph-mon[78322]: pgmap v179: 65 pgs: 65 active+clean; 254 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T19:01:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.035+0000 7ff8266dc700 1 -- 172.21.15.102:0/2861969327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff820100c60 msgr2=0x7ff820103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.035+0000 7ff8266dc700 1 --2- 172.21.15.102:0/2861969327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff820100c60 0x7ff820103040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff814009910 tx=0x7ff814009c20 comp rx=0 tx=0).stop 2024-03-26T19:01:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.035+0000 7ff8266dc700 1 -- 172.21.15.102:0/2861969327 shutdown_connections 2024-03-26T19:01:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.035+0000 7ff8266dc700 1 --2- 172.21.15.102:0/2861969327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff820103580 0x7ff820105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.035+0000 7ff8266dc700 1 --2- 172.21.15.102:0/2861969327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff820100c60 0x7ff820103040 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.035+0000 7ff8266dc700 1 -- 172.21.15.102:0/2861969327 >> 172.21.15.102:0/2861969327 conn(0x7ff8200fa5e0 msgr2=0x7ff8200fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.035+0000 7ff8266dc700 1 -- 172.21.15.102:0/2861969327 shutdown_connections 2024-03-26T19:01:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.035+0000 7ff8266dc700 1 -- 172.21.15.102:0/2861969327 wait complete. 2024-03-26T19:01:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.036+0000 7ff8266dc700 1 Processor -- start 2024-03-26T19:01:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.036+0000 7ff8266dc700 1 -- start start 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.036+0000 7ff8266dc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff820100c60 0x7ff820195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.036+0000 7ff8266dc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff820103580 0x7ff820195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.036+0000 7ff8266dc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8201964b0 con 0x7ff820100c60 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.036+0000 7ff8266dc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8201965f0 con 0x7ff820103580 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.036+0000 7ff81ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff820100c60 0x7ff820195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.036+0000 7ff81ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff820100c60 0x7ff820195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55376/0 (socket says 172.21.15.102:55376) 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.036+0000 7ff81ffff700 1 -- 172.21.15.102:0/3997890643 learned_addr learned my addr 172.21.15.102:0/3997890643 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.037+0000 7ff81f7fe700 1 --2- 172.21.15.102:0/3997890643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff820103580 0x7ff820195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.037+0000 7ff81ffff700 1 -- 172.21.15.102:0/3997890643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff820103580 msgr2=0x7ff820195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.037+0000 7ff81ffff700 1 --2- 172.21.15.102:0/3997890643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff820103580 0x7ff820195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.037+0000 7ff81ffff700 1 -- 172.21.15.102:0/3997890643 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8140095f0 con 0x7ff820100c60 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.037+0000 7ff81ffff700 1 --2- 172.21.15.102:0/3997890643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff820100c60 0x7ff820195950 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff814006010 tx=0x7ff81400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.037+0000 7ff81d7fa700 1 -- 172.21.15.102:0/3997890643 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff81401d070 con 0x7ff820100c60 2024-03-26T19:01:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.037+0000 7ff81d7fa700 1 -- 172.21.15.102:0/3997890643 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff81400f460 con 0x7ff820100c60 2024-03-26T19:01:31.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.037+0000 7ff8266dc700 1 -- 172.21.15.102:0/3997890643 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82019b040 con 0x7ff820100c60 2024-03-26T19:01:31.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.037+0000 7ff81d7fa700 1 -- 172.21.15.102:0/3997890643 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff814021640 con 0x7ff820100c60 2024-03-26T19:01:31.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.037+0000 7ff8266dc700 1 -- 172.21.15.102:0/3997890643 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82019b530 con 0x7ff820100c60 2024-03-26T19:01:31.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.038+0000 7ff8266dc700 1 -- 172.21.15.102:0/3997890643 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff82018fb90 con 0x7ff820100c60 2024-03-26T19:01:31.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.039+0000 7ff81d7fa700 1 -- 172.21.15.102:0/3997890643 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff81400fab0 con 0x7ff820100c60 2024-03-26T19:01:31.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.039+0000 7ff81d7fa700 1 --2- 172.21.15.102:0/3997890643 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff80806e460 0x7ff808070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:31.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.039+0000 7ff81d7fa700 1 -- 172.21.15.102:0/3997890643 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff81408f6a0 con 0x7ff820100c60 2024-03-26T19:01:31.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.039+0000 7ff81f7fe700 1 --2- 172.21.15.102:0/3997890643 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff80806e460 0x7ff808070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:31.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.040+0000 7ff81f7fe700 1 --2- 172.21.15.102:0/3997890643 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff80806e460 0x7ff808070910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff810004630 tx=0x7ff81000b3c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:31.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.044+0000 7ff81d7fa700 1 -- 172.21.15.102:0/3997890643 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff81405c3d0 con 0x7ff820100c60 2024-03-26T19:01:31.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.261+0000 7ff8266dc700 1 -- 172.21.15.102:0/3997890643 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff820061190 con 0x7ff80806e460 2024-03-26T19:01:31.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.262+0000 7ff81d7fa700 1 -- 172.21.15.102:0/3997890643 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff820061190 con 0x7ff80806e460 2024-03-26T19:01:31.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.264+0000 7ff8266dc700 1 -- 172.21.15.102:0/3997890643 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff80806e460 msgr2=0x7ff808070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.264+0000 7ff8266dc700 1 --2- 172.21.15.102:0/3997890643 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff80806e460 0x7ff808070910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff810004630 tx=0x7ff81000b3c0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.264+0000 7ff8266dc700 1 -- 172.21.15.102:0/3997890643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff820100c60 msgr2=0x7ff820195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.264+0000 7ff8266dc700 1 --2- 172.21.15.102:0/3997890643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff820100c60 0x7ff820195950 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff814006010 tx=0x7ff81400bd70 comp rx=0 tx=0).stop 2024-03-26T19:01:31.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.264+0000 7ff8266dc700 1 -- 172.21.15.102:0/3997890643 shutdown_connections 2024-03-26T19:01:31.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.264+0000 7ff8266dc700 1 --2- 172.21.15.102:0/3997890643 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff80806e460 0x7ff808070910 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.264+0000 7ff8266dc700 1 --2- 172.21.15.102:0/3997890643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff820100c60 0x7ff820195950 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.264+0000 7ff8266dc700 1 --2- 172.21.15.102:0/3997890643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff820103580 0x7ff820195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.264+0000 7ff8266dc700 1 -- 172.21.15.102:0/3997890643 >> 172.21.15.102:0/3997890643 conn(0x7ff8200fa5e0 msgr2=0x7ff8200fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:31.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.264+0000 7ff8266dc700 1 -- 172.21.15.102:0/3997890643 shutdown_connections 2024-03-26T19:01:31.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.264+0000 7ff8266dc700 1 -- 172.21.15.102:0/3997890643 wait complete. 2024-03-26T19:01:31.267 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T19:01:31.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.367+0000 7f367a271700 1 -- 172.21.15.102:0/191259337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3674100250 msgr2=0x7f36741006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.367+0000 7f367a271700 1 --2- 172.21.15.102:0/191259337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3674100250 0x7f36741006a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3668009910 tx=0x7f3668009c20 comp rx=0 tx=0).stop 2024-03-26T19:01:31.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.367+0000 7f367a271700 1 -- 172.21.15.102:0/191259337 shutdown_connections 2024-03-26T19:01:31.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.367+0000 7f367a271700 1 --2- 172.21.15.102:0/191259337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3674100250 0x7f36741006a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.367+0000 7f367a271700 1 --2- 172.21.15.102:0/191259337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36740ff050 0x7f36740ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.367+0000 7f367a271700 1 -- 172.21.15.102:0/191259337 >> 172.21.15.102:0/191259337 conn(0x7f36740fa5c0 msgr2=0x7f36740fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:31.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.367+0000 7f367a271700 1 -- 172.21.15.102:0/191259337 shutdown_connections 2024-03-26T19:01:31.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.367+0000 7f367a271700 1 -- 172.21.15.102:0/191259337 wait complete. 2024-03-26T19:01:31.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.368+0000 7f367a271700 1 Processor -- start 2024-03-26T19:01:31.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.368+0000 7f367a271700 1 -- start start 2024-03-26T19:01:31.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.368+0000 7f367a271700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36740ff050 0x7f3674197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:31.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.368+0000 7f367a271700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3674100250 0x7f3674198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:31.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.368+0000 7f367a271700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36741987b0 con 0x7f3674100250 2024-03-26T19:01:31.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.368+0000 7f367a271700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367419d1c0 con 0x7f36740ff050 2024-03-26T19:01:31.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.368+0000 7f36737fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3674100250 0x7f3674198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:31.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.368+0000 7f36737fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3674100250 0x7f3674198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55402/0 (socket says 172.21.15.102:55402) 2024-03-26T19:01:31.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.368+0000 7f36737fe700 1 -- 172.21.15.102:0/3007851975 learned_addr learned my addr 172.21.15.102:0/3007851975 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:01:31.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.369+0000 7f3673fff700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36740ff050 0x7f3674197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.369+0000 7f36737fe700 1 -- 172.21.15.102:0/3007851975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36740ff050 msgr2=0x7f3674197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.369+0000 7f36737fe700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36740ff050 0x7f3674197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.369+0000 7f36737fe700 1 -- 172.21.15.102:0/3007851975 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36680095f0 con 0x7f3674100250 2024-03-26T19:01:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.369+0000 7f3673fff700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36740ff050 0x7f3674197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:01:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.369+0000 7f36737fe700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3674100250 0x7f3674198190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f366800b5c0 tx=0x7f366800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.369+0000 7f36717fa700 1 -- 172.21.15.102:0/3007851975 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f366801d070 con 0x7f3674100250 2024-03-26T19:01:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.369+0000 7f36717fa700 1 -- 172.21.15.102:0/3007851975 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3668022470 con 0x7f3674100250 2024-03-26T19:01:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.369+0000 7f367a271700 1 -- 172.21.15.102:0/3007851975 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367419d360 con 0x7f3674100250 2024-03-26T19:01:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.369+0000 7f36717fa700 1 -- 172.21.15.102:0/3007851975 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f366800f690 con 0x7f3674100250 2024-03-26T19:01:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.369+0000 7f367a271700 1 -- 172.21.15.102:0/3007851975 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367419d8a0 con 0x7f3674100250 2024-03-26T19:01:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.370+0000 7f367a271700 1 -- 172.21.15.102:0/3007851975 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3674107c40 con 0x7f3674100250 2024-03-26T19:01:31.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.370+0000 7f36717fa700 1 -- 172.21.15.102:0/3007851975 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3668004b70 con 0x7f3674100250 2024-03-26T19:01:31.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.371+0000 7f36717fa700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f365c06e460 0x7f365c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:31.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.371+0000 7f3673fff700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f365c06e460 0x7f365c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:31.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.371+0000 7f36717fa700 1 -- 172.21.15.102:0/3007851975 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f366808fb30 con 0x7f3674100250 2024-03-26T19:01:31.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.371+0000 7f3673fff700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f365c06e460 0x7f365c070910 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3674199130 tx=0x7f3664008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:31.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.376+0000 7f36717fa700 1 -- 172.21.15.102:0/3007851975 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3668058f00 con 0x7f3674100250 2024-03-26T19:01:31.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.592+0000 7f367a271700 1 -- 172.21.15.102:0/3007851975 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3674069460 con 0x7f365c06e460 2024-03-26T19:01:31.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.593+0000 7f36717fa700 1 -- 172.21.15.102:0/3007851975 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3674069460 con 0x7f365c06e460 2024-03-26T19:01:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.595+0000 7f367a271700 1 -- 172.21.15.102:0/3007851975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f365c06e460 msgr2=0x7f365c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.595+0000 7f367a271700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f365c06e460 0x7f365c070910 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3674199130 tx=0x7f3664008040 comp rx=0 tx=0).stop 2024-03-26T19:01:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.595+0000 7f367a271700 1 -- 172.21.15.102:0/3007851975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3674100250 msgr2=0x7f3674198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.595+0000 7f367a271700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3674100250 0x7f3674198190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f366800b5c0 tx=0x7f366800be20 comp rx=0 tx=0).stop 2024-03-26T19:01:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.595+0000 7f367a271700 1 -- 172.21.15.102:0/3007851975 shutdown_connections 2024-03-26T19:01:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.595+0000 7f367a271700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36740ff050 0x7f3674197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.595+0000 7f367a271700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f365c06e460 0x7f365c070910 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.595+0000 7f367a271700 1 --2- 172.21.15.102:0/3007851975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3674100250 0x7f3674198190 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.595+0000 7f367a271700 1 -- 172.21.15.102:0/3007851975 >> 172.21.15.102:0/3007851975 conn(0x7f36740fa5c0 msgr2=0x7f3674068d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.595+0000 7f367a271700 1 -- 172.21.15.102:0/3007851975 shutdown_connections 2024-03-26T19:01:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.595+0000 7f367a271700 1 -- 172.21.15.102:0/3007851975 wait complete. 2024-03-26T19:01:31.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.699+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/3021678078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b8100650 msgr2=0x7fd1b8100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.699+0000 7fd1bd9c8700 1 --2- 172.21.15.102:0/3021678078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b8100650 0x7fd1b8100a60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009910 tx=0x7fd1a8009c20 comp rx=0 tx=0).stop 2024-03-26T19:01:31.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.699+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/3021678078 shutdown_connections 2024-03-26T19:01:31.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.699+0000 7fd1bd9c8700 1 --2- 172.21.15.102:0/3021678078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1b80ff300 0x7fd1b80ff770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.699+0000 7fd1bd9c8700 1 --2- 172.21.15.102:0/3021678078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b8100650 0x7fd1b8100a60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.699+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/3021678078 >> 172.21.15.102:0/3021678078 conn(0x7fd1b80f9d80 msgr2=0x7fd1b80fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:31.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.699+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/3021678078 shutdown_connections 2024-03-26T19:01:31.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.699+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/3021678078 wait complete. 2024-03-26T19:01:31.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1bd9c8700 1 Processor -- start 2024-03-26T19:01:31.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1bd9c8700 1 -- start start 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1bd9c8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b80ff300 0x7fd1b8197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1bd9c8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1b8100650 0x7fd1b81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1bd9c8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b8198710 con 0x7fd1b80ff300 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1bd9c8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b8198850 con 0x7fd1b8100650 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b80ff300 0x7fd1b8197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b80ff300 0x7fd1b8197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55422/0 (socket says 172.21.15.102:55422) 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1b6ffd700 1 -- 172.21.15.102:0/2449861204 learned_addr learned my addr 172.21.15.102:0/2449861204 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1b67fc700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1b8100650 0x7fd1b81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1b6ffd700 1 -- 172.21.15.102:0/2449861204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1b8100650 msgr2=0x7fd1b81980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1b6ffd700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1b8100650 0x7fd1b81980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1b6ffd700 1 -- 172.21.15.102:0/2449861204 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a80095f0 con 0x7fd1b80ff300 2024-03-26T19:01:31.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.700+0000 7fd1b67fc700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1b8100650 0x7fd1b81980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:01:31.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.701+0000 7fd1b6ffd700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b80ff300 0x7fd1b8197bb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8000c00 tx=0x7fd1a800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:31.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.701+0000 7fd1bc9c6700 1 -- 172.21.15.102:0/2449861204 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1a801d070 con 0x7fd1b80ff300 2024-03-26T19:01:31.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.701+0000 7fd1bc9c6700 1 -- 172.21.15.102:0/2449861204 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd1a8005dc0 con 0x7fd1b80ff300 2024-03-26T19:01:31.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.701+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/2449861204 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b819d2a0 con 0x7fd1b80ff300 2024-03-26T19:01:31.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.701+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/2449861204 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b819d790 con 0x7fd1b80ff300 2024-03-26T19:01:31.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.701+0000 7fd1bc9c6700 1 -- 172.21.15.102:0/2449861204 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1a80226a0 con 0x7fd1b80ff300 2024-03-26T19:01:31.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.702+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/2449861204 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b8107bc0 con 0x7fd1b80ff300 2024-03-26T19:01:31.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.702+0000 7fd1bc9c6700 1 -- 172.21.15.102:0/2449861204 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd1a8003910 con 0x7fd1b80ff300 2024-03-26T19:01:31.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.703+0000 7fd1bc9c6700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd1a406e460 0x7fd1a4070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:31.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.703+0000 7fd1b67fc700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd1a406e460 0x7fd1a4070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:31.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.703+0000 7fd1bc9c6700 1 -- 172.21.15.102:0/2449861204 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd1a808f900 con 0x7fd1b80ff300 2024-03-26T19:01:31.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.703+0000 7fd1b67fc700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd1a406e460 0x7fd1a4070910 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8100380 tx=0x7fd1a0009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:31.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.708+0000 7fd1bc9c6700 1 -- 172.21.15.102:0/2449861204 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd1a805c6b0 con 0x7fd1b80ff300 2024-03-26T19:01:31.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.913+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/2449861204 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1b819dee0 con 0x7fd1a406e460 2024-03-26T19:01:31.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.924+0000 7fd1bc9c6700 1 -- 172.21.15.102:0/2449861204 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fd1b819dee0 con 0x7fd1a406e460 2024-03-26T19:01:31.918 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T19:01:31.918 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (13m) 2m ago 14m 25.0M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T19:01:31.918 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (14m) 2m ago 14m 8354k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (13m) 33s ago 13m 8514k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (3m) 2m ago 14m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (3m) 33s ago 13m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (12m) 2m ago 14m 86.6M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (9m) 2m ago 9m 132M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (9m) 2m ago 9m 16.1M - 18.2.0 dc2bc1663786 ed4f3f7aad6f 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (8m) 33s ago 8m 16.8M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (9m) 33s ago 9m 120M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (6m) 2m ago 15m 504M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (5m) 33s ago 13m 451M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (5m) 2m ago 16m 46.2M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T19:01:31.919 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (4m) 33s ago 13m 36.6M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T19:01:31.920 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (14m) 2m ago 14m 16.3M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T19:01:31.920 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (13m) 33s ago 13m 15.1M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T19:01:31.920 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (3m) 2m ago 12m 86.0M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T19:01:31.920 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (2m) 2m ago 12m 95.7M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T19:01:31.920 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (2m) 2m ago 11m 11.7M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T19:01:31.920 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (103s) 33s ago 11m 98.2M 2240M 19.0.0-2311-g36051606 497bb87bc5da 03955331c962 2024-03-26T19:01:31.920 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (70s) 33s ago 11m 102M 2240M 19.0.0-2311-g36051606 497bb87bc5da 23e140cd8961 2024-03-26T19:01:31.920 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (36s) 33s ago 10m 12.6M 2240M 19.0.0-2311-g36051606 497bb87bc5da 60afb0e4da18 2024-03-26T19:01:31.920 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (5m) 2m ago 14m 74.3M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T19:01:31.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.926+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/2449861204 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd1a406e460 msgr2=0x7fd1a4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.926+0000 7fd1bd9c8700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd1a406e460 0x7fd1a4070910 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8100380 tx=0x7fd1a0009260 comp rx=0 tx=0).stop 2024-03-26T19:01:31.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.926+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/2449861204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b80ff300 msgr2=0x7fd1b8197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:31.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.926+0000 7fd1bd9c8700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b80ff300 0x7fd1b8197bb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8000c00 tx=0x7fd1a800f740 comp rx=0 tx=0).stop 2024-03-26T19:01:31.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.926+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/2449861204 shutdown_connections 2024-03-26T19:01:31.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.926+0000 7fd1bd9c8700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd1a406e460 0x7fd1a4070910 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.926+0000 7fd1bd9c8700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b80ff300 0x7fd1b8197bb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.926+0000 7fd1bd9c8700 1 --2- 172.21.15.102:0/2449861204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1b8100650 0x7fd1b81980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:31.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.926+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/2449861204 >> 172.21.15.102:0/2449861204 conn(0x7fd1b80f9d80 msgr2=0x7fd1b80fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:31.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.926+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/2449861204 shutdown_connections 2024-03-26T19:01:31.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:31.926+0000 7fd1bd9c8700 1 -- 172.21.15.102:0/2449861204 wait complete. 2024-03-26T19:01:32.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.029+0000 7f761643d700 1 -- 172.21.15.102:0/118913625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76100ff080 msgr2=0x7f76100ff490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:32.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.029+0000 7f761643d700 1 --2- 172.21.15.102:0/118913625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76100ff080 0x7f76100ff490 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7600009910 tx=0x7f7600009c20 comp rx=0 tx=0).stop 2024-03-26T19:01:32.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.029+0000 7f761643d700 1 -- 172.21.15.102:0/118913625 shutdown_connections 2024-03-26T19:01:32.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.029+0000 7f761643d700 1 --2- 172.21.15.102:0/118913625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7610100280 0x7f76101006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.029+0000 7f761643d700 1 --2- 172.21.15.102:0/118913625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76100ff080 0x7f76100ff490 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.029+0000 7f761643d700 1 -- 172.21.15.102:0/118913625 >> 172.21.15.102:0/118913625 conn(0x7f76100fa630 msgr2=0x7f76100fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:32.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.029+0000 7f761643d700 1 -- 172.21.15.102:0/118913625 shutdown_connections 2024-03-26T19:01:32.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.029+0000 7f761643d700 1 -- 172.21.15.102:0/118913625 wait complete. 2024-03-26T19:01:32.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.029+0000 7f761643d700 1 Processor -- start 2024-03-26T19:01:32.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.029+0000 7f761643d700 1 -- start start 2024-03-26T19:01:32.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f761643d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76100ff080 0x7f7610194880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:32.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f761643d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7610100280 0x7f7610194dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:32.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f761643d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76101953e0 con 0x7f7610100280 2024-03-26T19:01:32.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f761643d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7610195520 con 0x7f76100ff080 2024-03-26T19:01:32.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f760f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7610100280 0x7f7610194dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:32.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f760f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7610100280 0x7f7610194dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55428/0 (socket says 172.21.15.102:55428) 2024-03-26T19:01:32.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f760f7fe700 1 -- 172.21.15.102:0/758125839 learned_addr learned my addr 172.21.15.102:0/758125839 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:01:32.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f760f7fe700 1 -- 172.21.15.102:0/758125839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76100ff080 msgr2=0x7f7610194880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:32.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f760ffff700 1 --2- 172.21.15.102:0/758125839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76100ff080 0x7f7610194880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:32.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f760f7fe700 1 --2- 172.21.15.102:0/758125839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76100ff080 0x7f7610194880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f760f7fe700 1 -- 172.21.15.102:0/758125839 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76000095f0 con 0x7f7610100280 2024-03-26T19:01:32.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.030+0000 7f760f7fe700 1 --2- 172.21.15.102:0/758125839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7610100280 0x7f7610194dc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f760400c8a0 tx=0x7f760400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:32.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.031+0000 7f760d7fa700 1 -- 172.21.15.102:0/758125839 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f760400cea0 con 0x7f7610100280 2024-03-26T19:01:32.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.031+0000 7f760d7fa700 1 -- 172.21.15.102:0/758125839 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7604004830 con 0x7f7610100280 2024-03-26T19:01:32.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.031+0000 7f760d7fa700 1 -- 172.21.15.102:0/758125839 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76040055b0 con 0x7f7610100280 2024-03-26T19:01:32.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.031+0000 7f761643d700 1 -- 172.21.15.102:0/758125839 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7610199fd0 con 0x7f7610100280 2024-03-26T19:01:32.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.031+0000 7f761643d700 1 -- 172.21.15.102:0/758125839 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761019a4f0 con 0x7f7610100280 2024-03-26T19:01:32.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.032+0000 7f761643d700 1 -- 172.21.15.102:0/758125839 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7610107bd0 con 0x7f7610100280 2024-03-26T19:01:32.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.032+0000 7f760d7fa700 1 -- 172.21.15.102:0/758125839 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7604007ad0 con 0x7f7610100280 2024-03-26T19:01:32.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.033+0000 7f760d7fa700 1 --2- 172.21.15.102:0/758125839 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75f806e460 0x7f75f8070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:32.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.033+0000 7f760d7fa700 1 -- 172.21.15.102:0/758125839 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f760408e000 con 0x7f7610100280 2024-03-26T19:01:32.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.033+0000 7f760ffff700 1 --2- 172.21.15.102:0/758125839 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75f806e460 0x7f75f8070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:32.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.034+0000 7f760ffff700 1 --2- 172.21.15.102:0/758125839 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75f806e460 0x7f75f8070910 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f76000059d0 tx=0x7f7600005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:32.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.037+0000 7f760d7fa700 1 -- 172.21.15.102:0/758125839 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f760405adb0 con 0x7f7610100280 2024-03-26T19:01:32.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.322+0000 7f761643d700 1 -- 172.21.15.102:0/758125839 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7610107dc0 con 0x7f7610100280 2024-03-26T19:01:32.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.324+0000 7f760d7fa700 1 -- 172.21.15.102:0/758125839 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+701 (secure 0 0 0) 0x7f760401d2d0 con 0x7f7610100280 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 6 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-03-26T19:01:32.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:01:32.319 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T19:01:32.319 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4, 2024-03-26T19:01:32.319 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 10 2024-03-26T19:01:32.319 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T19:01:32.319 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:01:32.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.326+0000 7f761643d700 1 -- 172.21.15.102:0/758125839 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75f806e460 msgr2=0x7f75f8070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:32.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.326+0000 7f761643d700 1 --2- 172.21.15.102:0/758125839 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75f806e460 0x7f75f8070910 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f76000059d0 tx=0x7f7600005960 comp rx=0 tx=0).stop 2024-03-26T19:01:32.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.326+0000 7f761643d700 1 -- 172.21.15.102:0/758125839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7610100280 msgr2=0x7f7610194dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:32.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.326+0000 7f761643d700 1 --2- 172.21.15.102:0/758125839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7610100280 0x7f7610194dc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f760400c8a0 tx=0x7f760400cc60 comp rx=0 tx=0).stop 2024-03-26T19:01:32.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.326+0000 7f761643d700 1 -- 172.21.15.102:0/758125839 shutdown_connections 2024-03-26T19:01:32.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.326+0000 7f761643d700 1 --2- 172.21.15.102:0/758125839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76100ff080 0x7f7610194880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.326+0000 7f761643d700 1 --2- 172.21.15.102:0/758125839 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75f806e460 0x7f75f8070910 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.326+0000 7f761643d700 1 --2- 172.21.15.102:0/758125839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7610100280 0x7f7610194dc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.326+0000 7f761643d700 1 -- 172.21.15.102:0/758125839 >> 172.21.15.102:0/758125839 conn(0x7f76100fa630 msgr2=0x7f76101034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:32.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.326+0000 7f761643d700 1 -- 172.21.15.102:0/758125839 shutdown_connections 2024-03-26T19:01:32.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.326+0000 7f761643d700 1 -- 172.21.15.102:0/758125839 wait complete. 2024-03-26T19:01:32.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.435+0000 7f3422e42700 1 -- 172.21.15.102:0/2300623520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f341c0fe790 msgr2=0x7f341c105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:32.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.435+0000 7f3422e42700 1 --2- 172.21.15.102:0/2300623520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f341c0fe790 0x7f341c105960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f340c009940 tx=0x7f340c009c50 comp rx=0 tx=0).stop 2024-03-26T19:01:32.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.435+0000 7f3422e42700 1 -- 172.21.15.102:0/2300623520 shutdown_connections 2024-03-26T19:01:32.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.435+0000 7f3422e42700 1 --2- 172.21.15.102:0/2300623520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f341c0fe790 0x7f341c105960 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.435+0000 7f3422e42700 1 --2- 172.21.15.102:0/2300623520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f341c0fdd90 0x7f341c0fe1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.435+0000 7f3422e42700 1 -- 172.21.15.102:0/2300623520 >> 172.21.15.102:0/2300623520 conn(0x7f341c0f9940 msgr2=0x7f341c0fbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:32.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.435+0000 7f3422e42700 1 -- 172.21.15.102:0/2300623520 shutdown_connections 2024-03-26T19:01:32.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.435+0000 7f3422e42700 1 -- 172.21.15.102:0/2300623520 wait complete. 2024-03-26T19:01:32.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.435+0000 7f3422e42700 1 Processor -- start 2024-03-26T19:01:32.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.435+0000 7f3422e42700 1 -- start start 2024-03-26T19:01:32.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3422e42700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f341c0fdd90 0x7f341c195a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:32.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3422e42700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f341c0fe790 0x7f341c195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:32.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3422e42700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c1965d0 con 0x7f341c0fe790 2024-03-26T19:01:32.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3422e42700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c196710 con 0x7f341c0fdd90 2024-03-26T19:01:32.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3413fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f341c0fe790 0x7f341c195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:32.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3413fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f341c0fe790 0x7f341c195fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55450/0 (socket says 172.21.15.102:55450) 2024-03-26T19:01:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3413fff700 1 -- 172.21.15.102:0/3366955490 learned_addr learned my addr 172.21.15.102:0/3366955490 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:01:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3420bde700 1 --2- 172.21.15.102:0/3366955490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f341c0fdd90 0x7f341c195a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3413fff700 1 -- 172.21.15.102:0/3366955490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f341c0fdd90 msgr2=0x7f341c195a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3413fff700 1 --2- 172.21.15.102:0/3366955490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f341c0fdd90 0x7f341c195a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3413fff700 1 -- 172.21.15.102:0/3366955490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c0095f0 con 0x7f341c0fe790 2024-03-26T19:01:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3413fff700 1 --2- 172.21.15.102:0/3366955490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f341c0fe790 0x7f341c195fb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f340c0055f0 tx=0x7f340c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3411ffb700 1 -- 172.21.15.102:0/3366955490 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f340c01d070 con 0x7f341c0fe790 2024-03-26T19:01:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3422e42700 1 -- 172.21.15.102:0/3366955490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c19b160 con 0x7f341c0fe790 2024-03-26T19:01:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3411ffb700 1 -- 172.21.15.102:0/3366955490 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f340c003ba0 con 0x7f341c0fe790 2024-03-26T19:01:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.436+0000 7f3411ffb700 1 -- 172.21.15.102:0/3366955490 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f340c021640 con 0x7f341c0fe790 2024-03-26T19:01:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.437+0000 7f3422e42700 1 -- 172.21.15.102:0/3366955490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c19b6d0 con 0x7f341c0fe790 2024-03-26T19:01:32.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.437+0000 7f3422e42700 1 -- 172.21.15.102:0/3366955490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341c18fba0 con 0x7f341c0fe790 2024-03-26T19:01:32.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.438+0000 7f3411ffb700 1 -- 172.21.15.102:0/3366955490 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f340c003d10 con 0x7f341c0fe790 2024-03-26T19:01:32.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.438+0000 7f3411ffb700 1 --2- 172.21.15.102:0/3366955490 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f340406e460 0x7f3404070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:32.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.439+0000 7f3420bde700 1 --2- 172.21.15.102:0/3366955490 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f340406e460 0x7f3404070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:32.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.439+0000 7f3411ffb700 1 -- 172.21.15.102:0/3366955490 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f340c08f7d0 con 0x7f341c0fe790 2024-03-26T19:01:32.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.439+0000 7f3420bde700 1 --2- 172.21.15.102:0/3366955490 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f340406e460 0x7f3404070910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3418006fd0 tx=0x7f3418008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:32.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.443+0000 7f3411ffb700 1 -- 172.21.15.102:0/3366955490 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f340c05c500 con 0x7f341c0fe790 2024-03-26T19:01:32.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:32 smithi102 ceph-mon[78322]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:32.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:32 smithi102 ceph-mon[78322]: pgmap v180: 65 pgs: 65 active+clean; 198 MiB data, 884 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 2.9 KiB/s wr, 7 op/s 2024-03-26T19:01:32.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.690+0000 7f3422e42700 1 -- 172.21.15.102:0/3366955490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f341c02cc30 con 0x7f341c0fe790 2024-03-26T19:01:32.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.691+0000 7f3411ffb700 1 -- 172.21.15.102:0/3366955490 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+1918 (secure 0 0 0) 0x7f340c05c090 con 0x7f341c0fe790 2024-03-26T19:01:32.685 INFO:teuthology.orchestra.run.smithi102.stdout:e23 2024-03-26T19:01:32.685 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T19:01:32.685 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:01:32.685 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T19:01:32.685 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:01:32.685 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T19:01:32.685 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 23 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T19:01:12.201799+0000 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:01:32.686 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 1 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490,1=24283} 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 14490 members: 14490 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{1:24283} state up:stopping seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:01:32.687 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:01:32.688 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:01:32.688 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T19:01:32.688 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:01:32.688 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:14502} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2674848848,v1:172.21.15.102:6827/2674848848] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:01:32.688 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.693+0000 7f3422e42700 1 -- 172.21.15.102:0/3366955490 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f340406e460 msgr2=0x7f3404070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.693+0000 7f3422e42700 1 --2- 172.21.15.102:0/3366955490 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f340406e460 0x7f3404070910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3418006fd0 tx=0x7f3418008040 comp rx=0 tx=0).stop 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.693+0000 7f3422e42700 1 -- 172.21.15.102:0/3366955490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f341c0fe790 msgr2=0x7f341c195fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.693+0000 7f3422e42700 1 --2- 172.21.15.102:0/3366955490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f341c0fe790 0x7f341c195fb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f340c0055f0 tx=0x7f340c0056d0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.693+0000 7f3422e42700 1 -- 172.21.15.102:0/3366955490 shutdown_connections 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.693+0000 7f3422e42700 1 --2- 172.21.15.102:0/3366955490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f341c0fdd90 0x7f341c195a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.693+0000 7f3422e42700 1 --2- 172.21.15.102:0/3366955490 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f340406e460 0x7f3404070910 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.693+0000 7f3422e42700 1 --2- 172.21.15.102:0/3366955490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f341c0fe790 0x7f341c195fb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.693+0000 7f3422e42700 1 -- 172.21.15.102:0/3366955490 >> 172.21.15.102:0/3366955490 conn(0x7f341c0f9940 msgr2=0x7f341c1041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.693+0000 7f3422e42700 1 -- 172.21.15.102:0/3366955490 shutdown_connections 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.693+0000 7f3422e42700 1 -- 172.21.15.102:0/3366955490 wait complete. 2024-03-26T19:01:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 23 2024-03-26T19:01:32.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:32 smithi190 ceph-mon[65689]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:32.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:32 smithi190 ceph-mon[65689]: pgmap v180: 65 pgs: 65 active+clean; 198 MiB data, 884 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 2.9 KiB/s wr, 7 op/s 2024-03-26T19:01:32.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.798+0000 7f7d84969700 1 -- 172.21.15.102:0/892477345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d80100190 msgr2=0x7f7d801005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:32.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.798+0000 7f7d84969700 1 --2- 172.21.15.102:0/892477345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d80100190 0x7f7d801005a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7d74009910 tx=0x7f7d74009c20 comp rx=0 tx=0).stop 2024-03-26T19:01:32.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.798+0000 7f7d84969700 1 -- 172.21.15.102:0/892477345 shutdown_connections 2024-03-26T19:01:32.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.798+0000 7f7d84969700 1 --2- 172.21.15.102:0/892477345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d80101390 0x7f7d801017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.798+0000 7f7d84969700 1 --2- 172.21.15.102:0/892477345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d80100190 0x7f7d801005a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.798+0000 7f7d84969700 1 -- 172.21.15.102:0/892477345 >> 172.21.15.102:0/892477345 conn(0x7f7d800fb720 msgr2=0x7f7d800fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:32.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.798+0000 7f7d84969700 1 -- 172.21.15.102:0/892477345 shutdown_connections 2024-03-26T19:01:32.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.798+0000 7f7d84969700 1 -- 172.21.15.102:0/892477345 wait complete. 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d84969700 1 Processor -- start 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d84969700 1 -- start start 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d84969700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d80100190 0x7f7d80195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d84969700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d80101390 0x7f7d80195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d84969700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d801964f0 con 0x7f7d80101390 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d84969700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80196630 con 0x7f7d80100190 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d7dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d80101390 0x7f7d80195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d7dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d80101390 0x7f7d80195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55480/0 (socket says 172.21.15.102:55480) 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d7dd9b700 1 -- 172.21.15.102:0/2190529814 learned_addr learned my addr 172.21.15.102:0/2190529814 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d7e59c700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d80100190 0x7f7d80195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d7dd9b700 1 -- 172.21.15.102:0/2190529814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d80100190 msgr2=0x7f7d80195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d7dd9b700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d80100190 0x7f7d80195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d7dd9b700 1 -- 172.21.15.102:0/2190529814 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d740095f0 con 0x7f7d80101390 2024-03-26T19:01:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.799+0000 7f7d7e59c700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d80100190 0x7f7d80195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:01:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.800+0000 7f7d7dd9b700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d80101390 0x7f7d80195ed0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000e920 tx=0x7f7d7000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.800+0000 7f7d6f7fe700 1 -- 172.21.15.102:0/2190529814 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d70009790 con 0x7f7d80101390 2024-03-26T19:01:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.800+0000 7f7d6f7fe700 1 -- 172.21.15.102:0/2190529814 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7d7000cba0 con 0x7f7d80101390 2024-03-26T19:01:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.800+0000 7f7d84969700 1 -- 172.21.15.102:0/2190529814 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8019b0e0 con 0x7f7d80101390 2024-03-26T19:01:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.800+0000 7f7d6f7fe700 1 -- 172.21.15.102:0/2190529814 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d70018630 con 0x7f7d80101390 2024-03-26T19:01:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.800+0000 7f7d84969700 1 -- 172.21.15.102:0/2190529814 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8019b630 con 0x7f7d80101390 2024-03-26T19:01:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.801+0000 7f7d84969700 1 -- 172.21.15.102:0/2190529814 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d80108ce0 con 0x7f7d80101390 2024-03-26T19:01:32.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.801+0000 7f7d6f7fe700 1 -- 172.21.15.102:0/2190529814 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7d7000cd10 con 0x7f7d80101390 2024-03-26T19:01:32.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.802+0000 7f7d6f7fe700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f7d6806e460 0x7f7d68070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:32.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.802+0000 7f7d7e59c700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f7d6806e460 0x7f7d68070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:32.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.802+0000 7f7d6f7fe700 1 -- 172.21.15.102:0/2190529814 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7d70014070 con 0x7f7d80101390 2024-03-26T19:01:32.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.802+0000 7f7d7e59c700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f7d6806e460 0x7f7d68070910 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7d74000c00 tx=0x7f7d74009f70 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:32.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:32.806+0000 7f7d6f7fe700 1 -- 172.21.15.102:0/2190529814 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7d70053740 con 0x7f7d80101390 2024-03-26T19:01:33.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.024+0000 7f7d84969700 1 -- 172.21.15.102:0/2190529814 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d80105ce0 con 0x7f7d6806e460 2024-03-26T19:01:33.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.025+0000 7f7d6f7fe700 1 -- 172.21.15.102:0/2190529814 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7d80105ce0 con 0x7f7d6806e460 2024-03-26T19:01:33.019 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:01:33.019 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T19:01:33.019 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T19:01:33.019 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T19:01:33.019 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T19:01:33.019 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T19:01:33.020 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-03-26T19:01:33.020 INFO:teuthology.orchestra.run.smithi102.stdout: "mon", 2024-03-26T19:01:33.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd" 2024-03-26T19:01:33.020 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T19:01:33.020 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "12/23 daemons upgraded", 2024-03-26T19:01:33.020 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T19:01:33.020 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T19:01:33.020 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:01:33.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.028+0000 7f7d84969700 1 -- 172.21.15.102:0/2190529814 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f7d6806e460 msgr2=0x7f7d68070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:33.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.028+0000 7f7d84969700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f7d6806e460 0x7f7d68070910 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7d74000c00 tx=0x7f7d74009f70 comp rx=0 tx=0).stop 2024-03-26T19:01:33.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.028+0000 7f7d84969700 1 -- 172.21.15.102:0/2190529814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d80101390 msgr2=0x7f7d80195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:33.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.028+0000 7f7d84969700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d80101390 0x7f7d80195ed0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000e920 tx=0x7f7d7000ece0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.028+0000 7f7d84969700 1 -- 172.21.15.102:0/2190529814 shutdown_connections 2024-03-26T19:01:33.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.028+0000 7f7d84969700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d80100190 0x7f7d80195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.028+0000 7f7d84969700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f7d6806e460 0x7f7d68070910 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.028+0000 7f7d84969700 1 --2- 172.21.15.102:0/2190529814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d80101390 0x7f7d80195ed0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.028+0000 7f7d84969700 1 -- 172.21.15.102:0/2190529814 >> 172.21.15.102:0/2190529814 conn(0x7f7d800fb720 msgr2=0x7f7d801045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:33.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.028+0000 7f7d84969700 1 -- 172.21.15.102:0/2190529814 shutdown_connections 2024-03-26T19:01:33.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.028+0000 7f7d84969700 1 -- 172.21.15.102:0/2190529814 wait complete. 2024-03-26T19:01:33.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.133+0000 7fb472022700 1 -- 172.21.15.102:0/468536524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c0ff050 msgr2=0x7fb46c0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:33.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.133+0000 7fb472022700 1 --2- 172.21.15.102:0/468536524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c0ff050 0x7fb46c0ff460 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb46400b0a0 tx=0x7fb46400b3b0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.133+0000 7fb472022700 1 -- 172.21.15.102:0/468536524 shutdown_connections 2024-03-26T19:01:33.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.133+0000 7fb472022700 1 --2- 172.21.15.102:0/468536524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb46c100250 0x7fb46c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.133+0000 7fb472022700 1 --2- 172.21.15.102:0/468536524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c0ff050 0x7fb46c0ff460 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.133+0000 7fb472022700 1 -- 172.21.15.102:0/468536524 >> 172.21.15.102:0/468536524 conn(0x7fb46c0fa5e0 msgr2=0x7fb46c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:33.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.133+0000 7fb472022700 1 -- 172.21.15.102:0/468536524 shutdown_connections 2024-03-26T19:01:33.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.133+0000 7fb472022700 1 -- 172.21.15.102:0/468536524 wait complete. 2024-03-26T19:01:33.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.133+0000 7fb472022700 1 Processor -- start 2024-03-26T19:01:33.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb472022700 1 -- start start 2024-03-26T19:01:33.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb472022700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c0ff050 0x7fb46c1959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb472022700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb46c100250 0x7fb46c195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb472022700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c196490 con 0x7fb46c0ff050 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb472022700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c1965d0 con 0x7fb46c100250 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb46b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c0ff050 0x7fb46c1959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb46b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c0ff050 0x7fb46c1959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55504/0 (socket says 172.21.15.102:55504) 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb46b7fe700 1 -- 172.21.15.102:0/3481302275 learned_addr learned my addr 172.21.15.102:0/3481302275 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb46affd700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb46c100250 0x7fb46c195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb46b7fe700 1 -- 172.21.15.102:0/3481302275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb46c100250 msgr2=0x7fb46c195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb46b7fe700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb46c100250 0x7fb46c195f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb46b7fe700 1 -- 172.21.15.102:0/3481302275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb464009d00 con 0x7fb46c0ff050 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb46affd700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb46c100250 0x7fb46c195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb46b7fe700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c0ff050 0x7fb46c1959c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb46400b070 tx=0x7fb464003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:33.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.134+0000 7fb468ff9700 1 -- 172.21.15.102:0/3481302275 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb46400e040 con 0x7fb46c0ff050 2024-03-26T19:01:33.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.135+0000 7fb468ff9700 1 -- 172.21.15.102:0/3481302275 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb46400ce60 con 0x7fb46c0ff050 2024-03-26T19:01:33.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.135+0000 7fb472022700 1 -- 172.21.15.102:0/3481302275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46c19b030 con 0x7fb46c0ff050 2024-03-26T19:01:33.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.135+0000 7fb468ff9700 1 -- 172.21.15.102:0/3481302275 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb46401da90 con 0x7fb46c0ff050 2024-03-26T19:01:33.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.135+0000 7fb472022700 1 -- 172.21.15.102:0/3481302275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c19b570 con 0x7fb46c0ff050 2024-03-26T19:01:33.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.135+0000 7fb472022700 1 -- 172.21.15.102:0/3481302275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c107ba0 con 0x7fb46c0ff050 2024-03-26T19:01:33.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.136+0000 7fb468ff9700 1 -- 172.21.15.102:0/3481302275 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb464004250 con 0x7fb46c0ff050 2024-03-26T19:01:33.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.136+0000 7fb468ff9700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb45406e460 0x7fb454070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:01:33.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.136+0000 7fb468ff9700 1 -- 172.21.15.102:0/3481302275 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb46408ffd0 con 0x7fb46c0ff050 2024-03-26T19:01:33.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.136+0000 7fb46affd700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb45406e460 0x7fb454070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:01:33.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.137+0000 7fb46affd700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb45406e460 0x7fb454070910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb460005fd0 tx=0x7fb460005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:01:33.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.141+0000 7fb468ff9700 1 -- 172.21.15.102:0/3481302275 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb46405cd80 con 0x7fb46c0ff050 2024-03-26T19:01:33.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.428+0000 7fb472022700 1 -- 172.21.15.102:0/3481302275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb46c19bcc0 con 0x7fb46c0ff050 2024-03-26T19:01:33.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.428+0000 7fb468ff9700 1 -- 172.21.15.102:0/3481302275 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fb46405c910 con 0x7fb46c0ff050 2024-03-26T19:01:33.423 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T19:01:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.430+0000 7fb472022700 1 -- 172.21.15.102:0/3481302275 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb45406e460 msgr2=0x7fb454070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.430+0000 7fb472022700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb45406e460 0x7fb454070910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb460005fd0 tx=0x7fb460005e30 comp rx=0 tx=0).stop 2024-03-26T19:01:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.430+0000 7fb472022700 1 -- 172.21.15.102:0/3481302275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c0ff050 msgr2=0x7fb46c1959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:01:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.430+0000 7fb472022700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c0ff050 0x7fb46c1959c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb46400b070 tx=0x7fb464003ce0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.430+0000 7fb472022700 1 -- 172.21.15.102:0/3481302275 shutdown_connections 2024-03-26T19:01:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.430+0000 7fb472022700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb45406e460 0x7fb454070910 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.430+0000 7fb472022700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c0ff050 0x7fb46c1959c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.430+0000 7fb472022700 1 --2- 172.21.15.102:0/3481302275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb46c100250 0x7fb46c195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:01:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.430+0000 7fb472022700 1 -- 172.21.15.102:0/3481302275 >> 172.21.15.102:0/3481302275 conn(0x7fb46c0fa5e0 msgr2=0x7fb46c0683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:01:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.431+0000 7fb472022700 1 -- 172.21.15.102:0/3481302275 shutdown_connections 2024-03-26T19:01:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:01:33.431+0000 7fb472022700 1 -- 172.21.15.102:0/3481302275 wait complete. 2024-03-26T19:01:33.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:33 smithi102 ceph-mon[78322]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:33.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:33 smithi102 ceph-mon[78322]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:33.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:33 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/758125839' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:33.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:33 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3366955490' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:01:33.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:33 smithi190 ceph-mon[65689]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:33.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:33 smithi190 ceph-mon[65689]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:33.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:33 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/758125839' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:33.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:33 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3366955490' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:01:34.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:34 smithi102 ceph-mon[78322]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:34.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:34 smithi102 ceph-mon[78322]: pgmap v181: 65 pgs: 65 active+clean; 182 MiB data, 836 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 938 B/s wr, 7 op/s 2024-03-26T19:01:34.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:34 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3481302275' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:01:34.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:34 smithi190 ceph-mon[65689]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:01:34.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:34 smithi190 ceph-mon[65689]: pgmap v181: 65 pgs: 65 active+clean; 182 MiB data, 836 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 938 B/s wr, 7 op/s 2024-03-26T19:01:34.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:34 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3481302275' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:01:35.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:35.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:35.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:35.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:35.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:35.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:35.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:35.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:35.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:35.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:35.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:35.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:35.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:35.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:35.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:35.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:35.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:35.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:36.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:36 smithi102 ceph-mon[78322]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-03-26T19:01:36.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:36 smithi102 ceph-mon[78322]: pgmap v182: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 938 B/s wr, 7 op/s 2024-03-26T19:01:36.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:36 smithi190 ceph-mon[65689]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-03-26T19:01:36.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:36 smithi190 ceph-mon[65689]: pgmap v182: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 938 B/s wr, 7 op/s 2024-03-26T19:01:37.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:37 smithi102 ceph-mon[78322]: daemon mds.cephfs.smithi190.vykizb finished stopping rank 1 in filesystem cephfs (now has 1 ranks) 2024-03-26T19:01:37.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:37 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:01:37.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:37 smithi190 ceph-mon[65689]: daemon mds.cephfs.smithi190.vykizb finished stopping rank 1 in filesystem cephfs (now has 1 ranks) 2024-03-26T19:01:37.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:37 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:01:38.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:38 smithi102 ceph-mon[78322]: pgmap v183: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1023 B/s wr, 6 op/s 2024-03-26T19:01:38.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:38 smithi102 ceph-mon[78322]: mds.1 [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] down:stopped 2024-03-26T19:01:38.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:38 smithi102 ceph-mon[78322]: fsmap cephfs:1 {0=cephfs.smithi102.hrbrsp=up:active} 2 up:standby 2024-03-26T19:01:38.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:38 smithi190 ceph-mon[65689]: pgmap v183: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1023 B/s wr, 6 op/s 2024-03-26T19:01:38.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:38 smithi190 ceph-mon[65689]: mds.1 [v2:172.21.15.190:6824/2769333078,v1:172.21.15.190:6825/2769333078] down:stopped 2024-03-26T19:01:38.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:38 smithi190 ceph-mon[65689]: fsmap cephfs:1 {0=cephfs.smithi102.hrbrsp=up:active} 2 up:standby 2024-03-26T19:01:39.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:39 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.190:6824/925070513,v1:172.21.15.190:6825/925070513] up:boot 2024-03-26T19:01:39.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:39 smithi102 ceph-mon[78322]: fsmap cephfs:1 {0=cephfs.smithi102.hrbrsp=up:active} 3 up:standby 2024-03-26T19:01:39.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:39 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T19:01:39.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:39 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.190:6824/925070513,v1:172.21.15.190:6825/925070513] up:boot 2024-03-26T19:01:39.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:39 smithi190 ceph-mon[65689]: fsmap cephfs:1 {0=cephfs.smithi102.hrbrsp=up:active} 3 up:standby 2024-03-26T19:01:39.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:39 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T19:01:40.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:40 smithi102 ceph-mon[78322]: pgmap v184: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 426 B/s wr, 2 op/s 2024-03-26T19:01:40.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:40 smithi190 ceph-mon[65689]: pgmap v184: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 426 B/s wr, 2 op/s 2024-03-26T19:01:42.602 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:42 smithi190 ceph-mon[65689]: pgmap v185: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 426 B/s wr, 2 op/s 2024-03-26T19:01:42.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:42 smithi102 ceph-mon[78322]: pgmap v185: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 426 B/s wr, 2 op/s 2024-03-26T19:01:44.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:44 smithi102 ceph-mon[78322]: pgmap v186: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-03-26T19:01:44.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:44 smithi190 ceph-mon[65689]: pgmap v186: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-03-26T19:01:45.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:45.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:45.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:45.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:45.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:45.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:45.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:45.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:45.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:45.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:45 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi102.tvsqaa"]}]: dispatch 2024-03-26T19:01:45.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:45.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:45.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:01:45.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:45.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:45.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:45.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:45.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:45.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:01:45.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:45 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi102.tvsqaa"]}]: dispatch 2024-03-26T19:01:46.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:46 smithi102 ceph-mon[78322]: Upgrade: It appears safe to stop mds.cephfs.smithi102.tvsqaa 2024-03-26T19:01:46.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:46 smithi102 ceph-mon[78322]: pgmap v187: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T19:01:46.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:46 smithi190 ceph-mon[65689]: Upgrade: It appears safe to stop mds.cephfs.smithi102.tvsqaa 2024-03-26T19:01:46.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:46 smithi190 ceph-mon[65689]: pgmap v187: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T19:01:48.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:47 smithi102 ceph-mon[78322]: Upgrade: Updating mds.cephfs.smithi102.tvsqaa 2024-03-26T19:01:48.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:47 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:48.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:47 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.tvsqaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T19:01:48.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:47 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:48.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:47 smithi102 ceph-mon[78322]: Deploying daemon mds.cephfs.smithi102.tvsqaa on smithi102 2024-03-26T19:01:48.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:47 smithi102 ceph-mon[78322]: pgmap v188: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T19:01:48.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:47 smithi190 ceph-mon[65689]: Upgrade: Updating mds.cephfs.smithi102.tvsqaa 2024-03-26T19:01:48.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:47 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:48.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:47 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.tvsqaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T19:01:48.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:47 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:01:48.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:47 smithi190 ceph-mon[65689]: Deploying daemon mds.cephfs.smithi102.tvsqaa on smithi102 2024-03-26T19:01:48.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:47 smithi190 ceph-mon[65689]: pgmap v188: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T19:01:50.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:50 smithi102 ceph-mon[78322]: osdmap e81: 6 total, 6 up, 6 in 2024-03-26T19:01:50.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:50 smithi102 ceph-mon[78322]: fsmap cephfs:1 {0=cephfs.smithi102.hrbrsp=up:active} 2 up:standby 2024-03-26T19:01:50.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:50 smithi102 ceph-mon[78322]: pgmap v190: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:01:50.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:50 smithi190 ceph-mon[65689]: osdmap e81: 6 total, 6 up, 6 in 2024-03-26T19:01:50.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:50 smithi190 ceph-mon[65689]: fsmap cephfs:1 {0=cephfs.smithi102.hrbrsp=up:active} 2 up:standby 2024-03-26T19:01:50.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:50 smithi190 ceph-mon[65689]: pgmap v190: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:01:52.513 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:52 smithi102 ceph-mon[78322]: pgmap v191: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:01:52.513 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:52 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:01:52.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:52 smithi190 ceph-mon[65689]: pgmap v191: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:01:52.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:52 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:01:53.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:53.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:53.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:53.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:53 smithi102 ceph-mon[78322]: pgmap v192: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:01:53.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:53 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] up:boot 2024-03-26T19:01:53.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:53 smithi102 ceph-mon[78322]: fsmap cephfs:1 {0=cephfs.smithi102.hrbrsp=up:active} 3 up:standby 2024-03-26T19:01:53.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T19:01:54.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:54.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:54.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:01:54.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:53 smithi190 ceph-mon[65689]: pgmap v192: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:01:54.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:53 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] up:boot 2024-03-26T19:01:54.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:53 smithi190 ceph-mon[65689]: fsmap cephfs:1 {0=cephfs.smithi102.hrbrsp=up:active} 3 up:standby 2024-03-26T19:01:54.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T19:01:56.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:56 smithi102 ceph-mon[78322]: pgmap v193: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:01:56.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:56 smithi190 ceph-mon[65689]: pgmap v193: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:01:58.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:58 smithi102 ceph-mon[78322]: pgmap v194: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:01:58.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:58 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:58.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:01:58 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:58.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:58 smithi190 ceph-mon[65689]: pgmap v194: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:01:58.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:58 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:01:58.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:01:58 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:00.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:00.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:00.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:00 smithi102 ceph-mon[78322]: pgmap v195: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:00.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:00.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:00.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:00 smithi190 ceph-mon[65689]: pgmap v195: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:02.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:02 smithi102 ceph-mon[78322]: pgmap v196: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:02.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:02 smithi190 ceph-mon[65689]: pgmap v196: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:03.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.543+0000 7f3bb964f700 1 -- 172.21.15.102:0/1397454666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb40ff050 msgr2=0x7f3bb40ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:03.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.543+0000 7f3bb964f700 1 --2- 172.21.15.102:0/1397454666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb40ff050 0x7f3bb40ff460 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009910 tx=0x7f3ba4009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:03.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.543+0000 7f3bb964f700 1 -- 172.21.15.102:0/1397454666 shutdown_connections 2024-03-26T19:02:03.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.543+0000 7f3bb964f700 1 --2- 172.21.15.102:0/1397454666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb4100250 0x7f3bb41006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:03.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.543+0000 7f3bb964f700 1 --2- 172.21.15.102:0/1397454666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb40ff050 0x7f3bb40ff460 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:03.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.543+0000 7f3bb964f700 1 -- 172.21.15.102:0/1397454666 >> 172.21.15.102:0/1397454666 conn(0x7f3bb40fa600 msgr2=0x7f3bb40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:03.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.543+0000 7f3bb964f700 1 -- 172.21.15.102:0/1397454666 shutdown_connections 2024-03-26T19:02:03.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.544+0000 7f3bb964f700 1 -- 172.21.15.102:0/1397454666 wait complete. 2024-03-26T19:02:03.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.544+0000 7f3bb964f700 1 Processor -- start 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.544+0000 7f3bb964f700 1 -- start start 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb964f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb40ff050 0x7f3bb40697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb964f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb4100250 0x7f3bb419cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb964f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4196540 con 0x7f3bb4100250 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb964f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb419d010 con 0x7f3bb40ff050 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb27fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb4100250 0x7f3bb419cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb27fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb4100250 0x7f3bb419cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41406/0 (socket says 172.21.15.102:41406) 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb27fc700 1 -- 172.21.15.102:0/3483756668 learned_addr learned my addr 172.21.15.102:0/3483756668 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb2ffd700 1 --2- 172.21.15.102:0/3483756668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb40ff050 0x7f3bb40697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb27fc700 1 -- 172.21.15.102:0/3483756668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb40ff050 msgr2=0x7f3bb40697c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb27fc700 1 --2- 172.21.15.102:0/3483756668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb40ff050 0x7f3bb40697c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb27fc700 1 -- 172.21.15.102:0/3483756668 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba40095f0 con 0x7f3bb4100250 2024-03-26T19:02:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.545+0000 7f3bb27fc700 1 --2- 172.21.15.102:0/3483756668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb4100250 0x7f3bb419cad0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800c8d0 tx=0x7f3ba800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:03.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.546+0000 7f3b9bfff700 1 -- 172.21.15.102:0/3483756668 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ba8007a90 con 0x7f3bb4100250 2024-03-26T19:02:03.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.546+0000 7f3bb964f700 1 -- 172.21.15.102:0/3483756668 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb419d2c0 con 0x7f3bb4100250 2024-03-26T19:02:03.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.546+0000 7f3b9bfff700 1 -- 172.21.15.102:0/3483756668 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3ba8004ce0 con 0x7f3bb4100250 2024-03-26T19:02:03.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.546+0000 7f3b9bfff700 1 -- 172.21.15.102:0/3483756668 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ba80055f0 con 0x7f3bb4100250 2024-03-26T19:02:03.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.546+0000 7f3bb964f700 1 -- 172.21.15.102:0/3483756668 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb419d810 con 0x7f3bb4100250 2024-03-26T19:02:03.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.547+0000 7f3bb964f700 1 -- 172.21.15.102:0/3483756668 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb4107ba0 con 0x7f3bb4100250 2024-03-26T19:02:03.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.547+0000 7f3b9bfff700 1 -- 172.21.15.102:0/3483756668 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3ba800f450 con 0x7f3bb4100250 2024-03-26T19:02:03.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.548+0000 7f3b9bfff700 1 --2- 172.21.15.102:0/3483756668 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3b9c06e460 0x7f3b9c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:03.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.548+0000 7f3b9bfff700 1 -- 172.21.15.102:0/3483756668 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3ba808dfc0 con 0x7f3bb4100250 2024-03-26T19:02:03.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.548+0000 7f3bb2ffd700 1 --2- 172.21.15.102:0/3483756668 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3b9c06e460 0x7f3b9c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:03.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.548+0000 7f3bb2ffd700 1 --2- 172.21.15.102:0/3483756668 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3b9c06e460 0x7f3b9c070910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4006010 tx=0x7f3ba400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:03.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.553+0000 7f3b9bfff700 1 -- 172.21.15.102:0/3483756668 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3ba805ae50 con 0x7f3bb4100250 2024-03-26T19:02:03.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.777+0000 7f3bb964f700 1 -- 172.21.15.102:0/3483756668 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bb4104ba0 con 0x7f3b9c06e460 2024-03-26T19:02:03.772 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: Detected new or changed devices on smithi102 2024-03-26T19:02:03.772 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:03.772 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:03.772 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:03.772 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:02:03.772 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:03.772 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:03.773 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:03.773 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:03.773 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:03.773 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:03.773 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi102.hrbrsp"]}]: dispatch 2024-03-26T19:02:03.773 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: Upgrade: It appears safe to stop mds.cephfs.smithi102.hrbrsp 2024-03-26T19:02:03.773 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:03 smithi102 ceph-mon[78322]: pgmap v197: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:03.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.779+0000 7f3b9bfff700 1 -- 172.21.15.102:0/3483756668 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3bb4104ba0 con 0x7f3b9c06e460 2024-03-26T19:02:03.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.781+0000 7f3bb964f700 1 -- 172.21.15.102:0/3483756668 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3b9c06e460 msgr2=0x7f3b9c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:03.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.781+0000 7f3bb964f700 1 --2- 172.21.15.102:0/3483756668 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3b9c06e460 0x7f3b9c070910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4006010 tx=0x7f3ba400b540 comp rx=0 tx=0).stop 2024-03-26T19:02:03.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.781+0000 7f3bb964f700 1 -- 172.21.15.102:0/3483756668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb4100250 msgr2=0x7f3bb419cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:03.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.781+0000 7f3bb964f700 1 --2- 172.21.15.102:0/3483756668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb4100250 0x7f3bb419cad0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800c8d0 tx=0x7f3ba800cc90 comp rx=0 tx=0).stop 2024-03-26T19:02:03.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.781+0000 7f3bb964f700 1 -- 172.21.15.102:0/3483756668 shutdown_connections 2024-03-26T19:02:03.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.781+0000 7f3bb964f700 1 --2- 172.21.15.102:0/3483756668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb40ff050 0x7f3bb40697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:03.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.781+0000 7f3bb964f700 1 --2- 172.21.15.102:0/3483756668 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3b9c06e460 0x7f3b9c070910 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:03.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.781+0000 7f3bb964f700 1 --2- 172.21.15.102:0/3483756668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb4100250 0x7f3bb419cad0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:03.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.781+0000 7f3bb964f700 1 -- 172.21.15.102:0/3483756668 >> 172.21.15.102:0/3483756668 conn(0x7f3bb40fa600 msgr2=0x7f3bb4103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:03.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.781+0000 7f3bb964f700 1 -- 172.21.15.102:0/3483756668 shutdown_connections 2024-03-26T19:02:03.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.781+0000 7f3bb964f700 1 -- 172.21.15.102:0/3483756668 wait complete. 2024-03-26T19:02:03.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: Detected new or changed devices on smithi102 2024-03-26T19:02:03.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:03.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:03.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:03.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:02:03.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:03.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:03.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:03.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:03.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:03.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:03.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi102.hrbrsp"]}]: dispatch 2024-03-26T19:02:03.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: Upgrade: It appears safe to stop mds.cephfs.smithi102.hrbrsp 2024-03-26T19:02:03.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:03 smithi190 ceph-mon[65689]: pgmap v197: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:03.785 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T19:02:03.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.887+0000 7fa524c4b700 1 -- 172.21.15.102:0/3561651592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5200ff060 msgr2=0x7fa5200ff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:03.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.887+0000 7fa524c4b700 1 --2- 172.21.15.102:0/3561651592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5200ff060 0x7fa5200ff470 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa514009910 tx=0x7fa514009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:03.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.887+0000 7fa524c4b700 1 -- 172.21.15.102:0/3561651592 shutdown_connections 2024-03-26T19:02:03.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.887+0000 7fa524c4b700 1 --2- 172.21.15.102:0/3561651592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520100260 0x7fa5201006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:03.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.887+0000 7fa524c4b700 1 --2- 172.21.15.102:0/3561651592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5200ff060 0x7fa5200ff470 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:03.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.887+0000 7fa524c4b700 1 -- 172.21.15.102:0/3561651592 >> 172.21.15.102:0/3561651592 conn(0x7fa5200fa610 msgr2=0x7fa5200fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:03.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.887+0000 7fa524c4b700 1 -- 172.21.15.102:0/3561651592 shutdown_connections 2024-03-26T19:02:03.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.887+0000 7fa524c4b700 1 -- 172.21.15.102:0/3561651592 wait complete. 2024-03-26T19:02:03.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.888+0000 7fa524c4b700 1 Processor -- start 2024-03-26T19:02:03.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.888+0000 7fa524c4b700 1 -- start start 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.888+0000 7fa524c4b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5200ff060 0x7fa5200698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.888+0000 7fa524c4b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520100260 0x7fa52019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.888+0000 7fa524c4b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52019ad60 con 0x7fa5200ff060 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.888+0000 7fa524c4b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52019aea0 con 0x7fa520100260 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.888+0000 7fa51e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5200ff060 0x7fa5200698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.888+0000 7fa51e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5200ff060 0x7fa5200698d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41420/0 (socket says 172.21.15.102:41420) 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.888+0000 7fa51e59c700 1 -- 172.21.15.102:0/2546875013 learned_addr learned my addr 172.21.15.102:0/2546875013 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.889+0000 7fa51dd9b700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520100260 0x7fa52019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.889+0000 7fa51e59c700 1 -- 172.21.15.102:0/2546875013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520100260 msgr2=0x7fa52019a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.889+0000 7fa51e59c700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520100260 0x7fa52019a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.889+0000 7fa51e59c700 1 -- 172.21.15.102:0/2546875013 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5140095f0 con 0x7fa5200ff060 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.889+0000 7fa51dd9b700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520100260 0x7fa52019a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T19:02:03.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.889+0000 7fa51e59c700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5200ff060 0x7fa5200698d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa51400bda0 tx=0x7fa51400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:03.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.889+0000 7fa50f7fe700 1 -- 172.21.15.102:0/2546875013 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa51401d070 con 0x7fa5200ff060 2024-03-26T19:02:03.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.889+0000 7fa50f7fe700 1 -- 172.21.15.102:0/2546875013 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa514004c50 con 0x7fa5200ff060 2024-03-26T19:02:03.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.889+0000 7fa524c4b700 1 -- 172.21.15.102:0/2546875013 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52019b120 con 0x7fa5200ff060 2024-03-26T19:02:03.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.889+0000 7fa50f7fe700 1 -- 172.21.15.102:0/2546875013 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa51400f690 con 0x7fa5200ff060 2024-03-26T19:02:03.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.889+0000 7fa524c4b700 1 -- 172.21.15.102:0/2546875013 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52019b690 con 0x7fa5200ff060 2024-03-26T19:02:03.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.890+0000 7fa524c4b700 1 -- 172.21.15.102:0/2546875013 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa520107c30 con 0x7fa5200ff060 2024-03-26T19:02:03.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.891+0000 7fa50f7fe700 1 -- 172.21.15.102:0/2546875013 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa5140229b0 con 0x7fa5200ff060 2024-03-26T19:02:03.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.891+0000 7fa50f7fe700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa50806e460 0x7fa508070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:03.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.891+0000 7fa50f7fe700 1 -- 172.21.15.102:0/2546875013 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa51408fa70 con 0x7fa5200ff060 2024-03-26T19:02:03.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.891+0000 7fa51dd9b700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa50806e460 0x7fa508070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:03.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.892+0000 7fa51dd9b700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa50806e460 0x7fa508070910 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa520069e50 tx=0x7fa510008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:03.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:03.896+0000 7fa50f7fe700 1 -- 172.21.15.102:0/2546875013 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa51405c8c0 con 0x7fa5200ff060 2024-03-26T19:02:04.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.117+0000 7fa524c4b700 1 -- 172.21.15.102:0/2546875013 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa520104bb0 con 0x7fa50806e460 2024-03-26T19:02:04.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.118+0000 7fa50f7fe700 1 -- 172.21.15.102:0/2546875013 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa520104bb0 con 0x7fa50806e460 2024-03-26T19:02:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.120+0000 7fa524c4b700 1 -- 172.21.15.102:0/2546875013 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa50806e460 msgr2=0x7fa508070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.120+0000 7fa524c4b700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa50806e460 0x7fa508070910 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa520069e50 tx=0x7fa510008040 comp rx=0 tx=0).stop 2024-03-26T19:02:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.120+0000 7fa524c4b700 1 -- 172.21.15.102:0/2546875013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5200ff060 msgr2=0x7fa5200698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.120+0000 7fa524c4b700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5200ff060 0x7fa5200698d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa51400bda0 tx=0x7fa51400be80 comp rx=0 tx=0).stop 2024-03-26T19:02:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.120+0000 7fa524c4b700 1 -- 172.21.15.102:0/2546875013 shutdown_connections 2024-03-26T19:02:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.120+0000 7fa524c4b700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa50806e460 0x7fa508070910 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.120+0000 7fa524c4b700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5200ff060 0x7fa5200698d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.120+0000 7fa524c4b700 1 --2- 172.21.15.102:0/2546875013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520100260 0x7fa52019a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.120+0000 7fa524c4b700 1 -- 172.21.15.102:0/2546875013 >> 172.21.15.102:0/2546875013 conn(0x7fa5200fa610 msgr2=0x7fa520103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.120+0000 7fa524c4b700 1 -- 172.21.15.102:0/2546875013 shutdown_connections 2024-03-26T19:02:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.120+0000 7fa524c4b700 1 -- 172.21.15.102:0/2546875013 wait complete. 2024-03-26T19:02:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.224+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2165475543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00ff070 msgr2=0x7fd5c00ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.224+0000 7fd5c5ac0700 1 --2- 172.21.15.102:0/2165475543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00ff070 0x7fd5c00ff480 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009910 tx=0x7fd5b8009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.224+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2165475543 shutdown_connections 2024-03-26T19:02:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.224+0000 7fd5c5ac0700 1 --2- 172.21.15.102:0/2165475543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5c0100270 0x7fd5c01006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.224+0000 7fd5c5ac0700 1 --2- 172.21.15.102:0/2165475543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00ff070 0x7fd5c00ff480 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.224+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2165475543 >> 172.21.15.102:0/2165475543 conn(0x7fd5c00fa600 msgr2=0x7fd5c00fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:04.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.224+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2165475543 shutdown_connections 2024-03-26T19:02:04.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.225+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2165475543 wait complete. 2024-03-26T19:02:04.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.225+0000 7fd5c5ac0700 1 Processor -- start 2024-03-26T19:02:04.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.225+0000 7fd5c5ac0700 1 -- start start 2024-03-26T19:02:04.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.225+0000 7fd5c5ac0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00ff070 0x7fd5c01959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:04.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.225+0000 7fd5c5ac0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5c0100270 0x7fd5c0195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:04.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.225+0000 7fd5c5ac0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c0196480 con 0x7fd5c00ff070 2024-03-26T19:02:04.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.225+0000 7fd5c5ac0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c01965c0 con 0x7fd5c0100270 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5bf7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00ff070 0x7fd5c01959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5bf7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00ff070 0x7fd5c01959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41430/0 (socket says 172.21.15.102:41430) 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5bf7fe700 1 -- 172.21.15.102:0/2844986715 learned_addr learned my addr 172.21.15.102:0/2844986715 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5beffd700 1 --2- 172.21.15.102:0/2844986715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5c0100270 0x7fd5c0195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5bf7fe700 1 -- 172.21.15.102:0/2844986715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5c0100270 msgr2=0x7fd5c0195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5bf7fe700 1 --2- 172.21.15.102:0/2844986715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5c0100270 0x7fd5c0195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5bf7fe700 1 -- 172.21.15.102:0/2844986715 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b80095f0 con 0x7fd5c00ff070 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5bf7fe700 1 --2- 172.21.15.102:0/2844986715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00ff070 0x7fd5c01959b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd5b800bda0 tx=0x7fd5b800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5bcff9700 1 -- 172.21.15.102:0/2844986715 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5b801d070 con 0x7fd5c00ff070 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5bcff9700 1 -- 172.21.15.102:0/2844986715 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd5b8022470 con 0x7fd5c00ff070 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5bcff9700 1 -- 172.21.15.102:0/2844986715 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5b800f670 con 0x7fd5c00ff070 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2844986715 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c019b020 con 0x7fd5c00ff070 2024-03-26T19:02:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.226+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2844986715 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c019b4e0 con 0x7fd5c00ff070 2024-03-26T19:02:04.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.227+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2844986715 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c0107bc0 con 0x7fd5c00ff070 2024-03-26T19:02:04.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.228+0000 7fd5bcff9700 1 -- 172.21.15.102:0/2844986715 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd5b8004af0 con 0x7fd5c00ff070 2024-03-26T19:02:04.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.228+0000 7fd5bcff9700 1 --2- 172.21.15.102:0/2844986715 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd5a806e460 0x7fd5a8070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:04.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.228+0000 7fd5bcff9700 1 -- 172.21.15.102:0/2844986715 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd5b808f960 con 0x7fd5c00ff070 2024-03-26T19:02:04.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.228+0000 7fd5beffd700 1 --2- 172.21.15.102:0/2844986715 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd5a806e460 0x7fd5a8070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:04.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.229+0000 7fd5beffd700 1 --2- 172.21.15.102:0/2844986715 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd5a806e460 0x7fd5a8070910 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400f440 tx=0x7fd5b40079e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:04.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.233+0000 7fd5bcff9700 1 -- 172.21.15.102:0/2844986715 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd5b805c8e0 con 0x7fd5c00ff070 2024-03-26T19:02:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.454+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2844986715 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5c00694c0 con 0x7fd5a806e460 2024-03-26T19:02:04.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.466+0000 7fd5bcff9700 1 -- 172.21.15.102:0/2844986715 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fd5c00694c0 con 0x7fd5a806e460 2024-03-26T19:02:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (13m) 6s ago 15m 25.7M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (15m) 6s ago 15m 8820k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (14m) 66s ago 14m 8514k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (4m) 6s ago 15m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (4m) 66s ago 14m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (13m) 6s ago 14m 85.5M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (9m) 6s ago 9m 132M - 18.2.0 dc2bc1663786 b181bfa43233 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (12s) 6s ago 9m 11.3M - 19.0.0-2311-g36051606 497bb87bc5da 63b8127a3849 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (9m) 66s ago 9m 16.8M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (9m) 66s ago 9m 120M - 18.2.0 dc2bc1663786 7e951a3db344 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (6m) 6s ago 16m 508M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (5m) 66s ago 14m 451M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (5m) 6s ago 16m 49.1M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T19:02:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (5m) 66s ago 14m 36.6M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T19:02:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (15m) 6s ago 15m 16.5M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T19:02:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (14m) 66s ago 14m 15.1M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T19:02:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (4m) 6s ago 13m 100M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T19:02:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (3m) 6s ago 12m 107M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T19:02:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (2m) 6s ago 12m 90.8M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T19:02:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (2m) 66s ago 11m 98.2M 2240M 19.0.0-2311-g36051606 497bb87bc5da 03955331c962 2024-03-26T19:02:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (102s) 66s ago 11m 102M 2240M 19.0.0-2311-g36051606 497bb87bc5da 23e140cd8961 2024-03-26T19:02:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (69s) 66s ago 11m 12.6M 2240M 19.0.0-2311-g36051606 497bb87bc5da 60afb0e4da18 2024-03-26T19:02:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (5m) 6s ago 14m 78.5M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T19:02:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.468+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2844986715 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd5a806e460 msgr2=0x7fd5a8070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.468+0000 7fd5c5ac0700 1 --2- 172.21.15.102:0/2844986715 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd5a806e460 0x7fd5a8070910 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400f440 tx=0x7fd5b40079e0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.468+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2844986715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00ff070 msgr2=0x7fd5c01959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.468+0000 7fd5c5ac0700 1 --2- 172.21.15.102:0/2844986715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00ff070 0x7fd5c01959b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd5b800bda0 tx=0x7fd5b800bdd0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.468+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2844986715 shutdown_connections 2024-03-26T19:02:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.468+0000 7fd5c5ac0700 1 --2- 172.21.15.102:0/2844986715 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd5a806e460 0x7fd5a8070910 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.468+0000 7fd5c5ac0700 1 --2- 172.21.15.102:0/2844986715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00ff070 0x7fd5c01959b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.468+0000 7fd5c5ac0700 1 --2- 172.21.15.102:0/2844986715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5c0100270 0x7fd5c0195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.468+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2844986715 >> 172.21.15.102:0/2844986715 conn(0x7fd5c00fa600 msgr2=0x7fd5c0068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:04.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.468+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2844986715 shutdown_connections 2024-03-26T19:02:04.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.468+0000 7fd5c5ac0700 1 -- 172.21.15.102:0/2844986715 wait complete. 2024-03-26T19:02:04.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.575+0000 7fc17e47d700 1 -- 172.21.15.102:0/3701296746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178102150 msgr2=0x7fc1780fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:04.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.575+0000 7fc17e47d700 1 --2- 172.21.15.102:0/3701296746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178102150 0x7fc1780fe1f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc168009910 tx=0x7fc168009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:04.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.575+0000 7fc17e47d700 1 -- 172.21.15.102:0/3701296746 shutdown_connections 2024-03-26T19:02:04.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.575+0000 7fc17e47d700 1 --2- 172.21.15.102:0/3701296746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178102150 0x7fc1780fe1f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.575+0000 7fc17e47d700 1 --2- 172.21.15.102:0/3701296746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc178101770 0x7fc178101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.575+0000 7fc17e47d700 1 -- 172.21.15.102:0/3701296746 >> 172.21.15.102:0/3701296746 conn(0x7fc1780f9d80 msgr2=0x7fc1780fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:04.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.575+0000 7fc17e47d700 1 -- 172.21.15.102:0/3701296746 shutdown_connections 2024-03-26T19:02:04.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.575+0000 7fc17e47d700 1 -- 172.21.15.102:0/3701296746 wait complete. 2024-03-26T19:02:04.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.575+0000 7fc17e47d700 1 Processor -- start 2024-03-26T19:02:04.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.575+0000 7fc17e47d700 1 -- start start 2024-03-26T19:02:04.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc17e47d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc178101770 0x7fc1781959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:04.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc17e47d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178102150 0x7fc178195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc17e47d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc178196550 con 0x7fc178102150 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc17e47d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc178196690 con 0x7fc178101770 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc1777fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178102150 0x7fc178195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc1777fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178102150 0x7fc178195f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41458/0 (socket says 172.21.15.102:41458) 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc1777fe700 1 -- 172.21.15.102:0/4181007851 learned_addr learned my addr 172.21.15.102:0/4181007851 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc177fff700 1 --2- 172.21.15.102:0/4181007851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc178101770 0x7fc1781959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc1777fe700 1 -- 172.21.15.102:0/4181007851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc178101770 msgr2=0x7fc1781959f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc1777fe700 1 --2- 172.21.15.102:0/4181007851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc178101770 0x7fc1781959f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc1777fe700 1 -- 172.21.15.102:0/4181007851 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1680095f0 con 0x7fc178102150 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.576+0000 7fc1777fe700 1 --2- 172.21.15.102:0/4181007851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178102150 0x7fc178195f30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc16800bda0 tx=0x7fc16800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.577+0000 7fc1757fa700 1 -- 172.21.15.102:0/4181007851 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc16801d070 con 0x7fc178102150 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.577+0000 7fc17e47d700 1 -- 172.21.15.102:0/4181007851 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17819b0e0 con 0x7fc178102150 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.577+0000 7fc1757fa700 1 -- 172.21.15.102:0/4181007851 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc168022470 con 0x7fc178102150 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.577+0000 7fc1757fa700 1 -- 172.21.15.102:0/4181007851 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc16800f650 con 0x7fc178102150 2024-03-26T19:02:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.577+0000 7fc17e47d700 1 -- 172.21.15.102:0/4181007851 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17819b650 con 0x7fc178102150 2024-03-26T19:02:04.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.578+0000 7fc17e47d700 1 -- 172.21.15.102:0/4181007851 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc178108d40 con 0x7fc178102150 2024-03-26T19:02:04.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.578+0000 7fc1757fa700 1 -- 172.21.15.102:0/4181007851 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc168004af0 con 0x7fc178102150 2024-03-26T19:02:04.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.578+0000 7fc1757fa700 1 --2- 172.21.15.102:0/4181007851 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc16406e460 0x7fc164070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:04.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.579+0000 7fc1757fa700 1 -- 172.21.15.102:0/4181007851 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc16808f960 con 0x7fc178102150 2024-03-26T19:02:04.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.579+0000 7fc177fff700 1 --2- 172.21.15.102:0/4181007851 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc16406e460 0x7fc164070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:04.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.579+0000 7fc177fff700 1 --2- 172.21.15.102:0/4181007851 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc16406e460 0x7fc164070910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc1780680b0 tx=0x7fc160009190 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:04.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.583+0000 7fc1757fa700 1 -- 172.21.15.102:0/4181007851 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc16805c7c0 con 0x7fc178102150 2024-03-26T19:02:04.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.943+0000 7fc17e47d700 1 -- 172.21.15.102:0/4181007851 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc178108f00 con 0x7fc178102150 2024-03-26T19:02:04.939 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:02:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T19:02:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:02:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:02:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T19:02:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:02:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:02:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T19:02:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 6 2024-03-26T19:02:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:02:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T19:02:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 3, 2024-03-26T19:02:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 1 2024-03-26T19:02:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:02:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T19:02:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 3, 2024-03-26T19:02:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 11 2024-03-26T19:02:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T19:02:04.940 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.943+0000 7fc1757fa700 1 -- 172.21.15.102:0/4181007851 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fc16805c350 con 0x7fc178102150 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.946+0000 7fc17e47d700 1 -- 172.21.15.102:0/4181007851 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc16406e460 msgr2=0x7fc164070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.946+0000 7fc17e47d700 1 --2- 172.21.15.102:0/4181007851 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc16406e460 0x7fc164070910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc1780680b0 tx=0x7fc160009190 comp rx=0 tx=0).stop 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.946+0000 7fc17e47d700 1 -- 172.21.15.102:0/4181007851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178102150 msgr2=0x7fc178195f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.946+0000 7fc17e47d700 1 --2- 172.21.15.102:0/4181007851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178102150 0x7fc178195f30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc16800bda0 tx=0x7fc16800bdd0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.946+0000 7fc17e47d700 1 -- 172.21.15.102:0/4181007851 shutdown_connections 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.946+0000 7fc17e47d700 1 --2- 172.21.15.102:0/4181007851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc178101770 0x7fc1781959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.946+0000 7fc17e47d700 1 --2- 172.21.15.102:0/4181007851 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc16406e460 0x7fc164070910 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.946+0000 7fc17e47d700 1 --2- 172.21.15.102:0/4181007851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178102150 0x7fc178195f30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.946+0000 7fc17e47d700 1 -- 172.21.15.102:0/4181007851 >> 172.21.15.102:0/4181007851 conn(0x7fc1780f9d80 msgr2=0x7fc178104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.946+0000 7fc17e47d700 1 -- 172.21.15.102:0/4181007851 shutdown_connections 2024-03-26T19:02:04.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:04.946+0000 7fc17e47d700 1 -- 172.21.15.102:0/4181007851 wait complete. 2024-03-26T19:02:05.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.059+0000 7f68b169a700 1 -- 172.21.15.102:0/3640513017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac0ff050 msgr2=0x7f68ac0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:05.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.059+0000 7f68b169a700 1 --2- 172.21.15.102:0/3640513017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac0ff050 0x7f68ac0ff460 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f689c009910 tx=0x7f689c009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:05.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.059+0000 7f68b169a700 1 -- 172.21.15.102:0/3640513017 shutdown_connections 2024-03-26T19:02:05.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.059+0000 7f68b169a700 1 --2- 172.21.15.102:0/3640513017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68ac100250 0x7f68ac1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.059+0000 7f68b169a700 1 --2- 172.21.15.102:0/3640513017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac0ff050 0x7f68ac0ff460 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.059+0000 7f68b169a700 1 -- 172.21.15.102:0/3640513017 >> 172.21.15.102:0/3640513017 conn(0x7f68ac0fa600 msgr2=0x7f68ac0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:05.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.059+0000 7f68b169a700 1 -- 172.21.15.102:0/3640513017 shutdown_connections 2024-03-26T19:02:05.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.059+0000 7f68b169a700 1 -- 172.21.15.102:0/3640513017 wait complete. 2024-03-26T19:02:05.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68b169a700 1 Processor -- start 2024-03-26T19:02:05.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68b169a700 1 -- start start 2024-03-26T19:02:05.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68b169a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac0ff050 0x7f68ac0697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:05.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68b169a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68ac100250 0x7f68ac19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:05.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68b169a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac19aca0 con 0x7f68ac0ff050 2024-03-26T19:02:05.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68b169a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac19ade0 con 0x7f68ac100250 2024-03-26T19:02:05.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68aaffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac0ff050 0x7f68ac0697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68aaffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac0ff050 0x7f68ac0697c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41476/0 (socket says 172.21.15.102:41476) 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68aaffd700 1 -- 172.21.15.102:0/1833605159 learned_addr learned my addr 172.21.15.102:0/1833605159 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68aa7fc700 1 --2- 172.21.15.102:0/1833605159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68ac100250 0x7f68ac19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68aaffd700 1 -- 172.21.15.102:0/1833605159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68ac100250 msgr2=0x7f68ac19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68aaffd700 1 --2- 172.21.15.102:0/1833605159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68ac100250 0x7f68ac19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68aaffd700 1 -- 172.21.15.102:0/1833605159 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c0095f0 con 0x7f68ac0ff050 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.060+0000 7f68aaffd700 1 --2- 172.21.15.102:0/1833605159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac0ff050 0x7f68ac0697c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f689c00bda0 tx=0x7f689c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.061+0000 7f6893fff700 1 -- 172.21.15.102:0/1833605159 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689c01d070 con 0x7f68ac0ff050 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.061+0000 7f6893fff700 1 -- 172.21.15.102:0/1833605159 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f689c00f460 con 0x7f68ac0ff050 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.061+0000 7f68b169a700 1 -- 172.21.15.102:0/1833605159 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac19b060 con 0x7f68ac0ff050 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.061+0000 7f6893fff700 1 -- 172.21.15.102:0/1833605159 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689c021620 con 0x7f68ac0ff050 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.061+0000 7f68b169a700 1 -- 172.21.15.102:0/1833605159 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac19b550 con 0x7f68ac0ff050 2024-03-26T19:02:05.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.062+0000 7f68b169a700 1 -- 172.21.15.102:0/1833605159 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ac107ba0 con 0x7f68ac0ff050 2024-03-26T19:02:05.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.062+0000 7f6893fff700 1 -- 172.21.15.102:0/1833605159 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f689c00f5d0 con 0x7f68ac0ff050 2024-03-26T19:02:05.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.063+0000 7f6893fff700 1 --2- 172.21.15.102:0/1833605159 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f689406e460 0x7f6894070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:05.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.063+0000 7f6893fff700 1 -- 172.21.15.102:0/1833605159 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f689c08f610 con 0x7f68ac0ff050 2024-03-26T19:02:05.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.063+0000 7f68aa7fc700 1 --2- 172.21.15.102:0/1833605159 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f689406e460 0x7f6894070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:05.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.063+0000 7f68aa7fc700 1 --2- 172.21.15.102:0/1833605159 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f689406e460 0x7f6894070910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f68ac069d40 tx=0x7f68a000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:05.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.067+0000 7f6893fff700 1 -- 172.21.15.102:0/1833605159 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f689c05c3c0 con 0x7f68ac0ff050 2024-03-26T19:02:05.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.320+0000 7f68b169a700 1 -- 172.21.15.102:0/1833605159 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f68ac19bb90 con 0x7f68ac0ff050 2024-03-26T19:02:05.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.320+0000 7f6893fff700 1 -- 172.21.15.102:0/1833605159 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1891 (secure 0 0 0) 0x7f689c05bf50 con 0x7f68ac0ff050 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:e27 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 24 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T19:01:37.203002+0000 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T19:02:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 47 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 1 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:in 0 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=14490} 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 1 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T19:02:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 14490 members: 14490 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{0:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.102:6828/989894576,v1:172.21.15.102:6829/989894576] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:34436} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:02:05.317 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.190:6824/925070513,v1:172.21.15.190:6825/925070513] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:02:05.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.323+0000 7f68b169a700 1 -- 172.21.15.102:0/1833605159 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f689406e460 msgr2=0x7f6894070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:05.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.323+0000 7f68b169a700 1 --2- 172.21.15.102:0/1833605159 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f689406e460 0x7f6894070910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f68ac069d40 tx=0x7f68a000b410 comp rx=0 tx=0).stop 2024-03-26T19:02:05.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.323+0000 7f68b169a700 1 -- 172.21.15.102:0/1833605159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac0ff050 msgr2=0x7f68ac0697c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:05.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.323+0000 7f68b169a700 1 --2- 172.21.15.102:0/1833605159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac0ff050 0x7f68ac0697c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f689c00bda0 tx=0x7f689c00bdd0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.323+0000 7f68b169a700 1 -- 172.21.15.102:0/1833605159 shutdown_connections 2024-03-26T19:02:05.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.323+0000 7f68b169a700 1 --2- 172.21.15.102:0/1833605159 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f689406e460 0x7f6894070910 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.323+0000 7f68b169a700 1 --2- 172.21.15.102:0/1833605159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac0ff050 0x7f68ac0697c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.323+0000 7f68b169a700 1 --2- 172.21.15.102:0/1833605159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68ac100250 0x7f68ac19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.323+0000 7f68b169a700 1 -- 172.21.15.102:0/1833605159 >> 172.21.15.102:0/1833605159 conn(0x7f68ac0fa600 msgr2=0x7f68ac103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:05.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.323+0000 7f68b169a700 1 -- 172.21.15.102:0/1833605159 shutdown_connections 2024-03-26T19:02:05.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.323+0000 7f68b169a700 1 -- 172.21.15.102:0/1833605159 wait complete. 2024-03-26T19:02:05.319 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-03-26T19:02:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.430+0000 7f29673ca700 1 -- 172.21.15.102:0/302181253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2960101370 msgr2=0x7f29601017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.430+0000 7f29673ca700 1 --2- 172.21.15.102:0/302181253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2960101370 0x7f29601017c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f295c009910 tx=0x7f295c009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.430+0000 7f29673ca700 1 -- 172.21.15.102:0/302181253 shutdown_connections 2024-03-26T19:02:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.430+0000 7f29673ca700 1 --2- 172.21.15.102:0/302181253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2960101370 0x7f29601017c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.430+0000 7f29673ca700 1 --2- 172.21.15.102:0/302181253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2960100170 0x7f2960100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.430+0000 7f29673ca700 1 -- 172.21.15.102:0/302181253 >> 172.21.15.102:0/302181253 conn(0x7f29600fb6e0 msgr2=0x7f29600fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.430+0000 7f29673ca700 1 -- 172.21.15.102:0/302181253 shutdown_connections 2024-03-26T19:02:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.430+0000 7f29673ca700 1 -- 172.21.15.102:0/302181253 wait complete. 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f29673ca700 1 Processor -- start 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f29673ca700 1 -- start start 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f29673ca700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2960100170 0x7f2960074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f29673ca700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2960101370 0x7f2960073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f29673ca700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2960073680 con 0x7f2960100170 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f29673ca700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29600737c0 con 0x7f2960101370 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f2965166700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2960100170 0x7f2960074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f2965166700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2960100170 0x7f2960074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41492/0 (socket says 172.21.15.102:41492) 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f2965166700 1 -- 172.21.15.102:0/347794315 learned_addr learned my addr 172.21.15.102:0/347794315 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f2964965700 1 --2- 172.21.15.102:0/347794315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2960101370 0x7f2960073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f2965166700 1 -- 172.21.15.102:0/347794315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2960101370 msgr2=0x7f2960073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f2965166700 1 --2- 172.21.15.102:0/347794315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2960101370 0x7f2960073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.431+0000 7f2965166700 1 -- 172.21.15.102:0/347794315 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c0095f0 con 0x7f2960100170 2024-03-26T19:02:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.432+0000 7f2965166700 1 --2- 172.21.15.102:0/347794315 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2960100170 0x7f2960074af0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f295000d900 tx=0x7f295000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.432+0000 7f29567fc700 1 -- 172.21.15.102:0/347794315 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2950009900 con 0x7f2960100170 2024-03-26T19:02:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.432+0000 7f29567fc700 1 -- 172.21.15.102:0/347794315 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2950010460 con 0x7f2960100170 2024-03-26T19:02:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.432+0000 7f29673ca700 1 -- 172.21.15.102:0/347794315 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2960073aa0 con 0x7f2960100170 2024-03-26T19:02:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.432+0000 7f29567fc700 1 -- 172.21.15.102:0/347794315 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f295000f5b0 con 0x7f2960100170 2024-03-26T19:02:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.432+0000 7f29673ca700 1 -- 172.21.15.102:0/347794315 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296019da00 con 0x7f2960100170 2024-03-26T19:02:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.433+0000 7f29673ca700 1 -- 172.21.15.102:0/347794315 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2960108ce0 con 0x7f2960100170 2024-03-26T19:02:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.433+0000 7f29567fc700 1 -- 172.21.15.102:0/347794315 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f29500105d0 con 0x7f2960100170 2024-03-26T19:02:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.434+0000 7f29567fc700 1 --2- 172.21.15.102:0/347794315 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f294c06e460 0x7f294c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.434+0000 7f29567fc700 1 -- 172.21.15.102:0/347794315 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f295008de70 con 0x7f2960100170 2024-03-26T19:02:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.434+0000 7f2964965700 1 --2- 172.21.15.102:0/347794315 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f294c06e460 0x7f294c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.434+0000 7f2964965700 1 --2- 172.21.15.102:0/347794315 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f294c06e460 0x7f294c070910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f295c006010 tx=0x7f295c00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:05.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.438+0000 7f29567fc700 1 -- 172.21.15.102:0/347794315 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f295005ac20 con 0x7f2960100170 2024-03-26T19:02:05.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:05 smithi102 ceph-mon[78322]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:05.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:05 smithi102 ceph-mon[78322]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:05.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:05 smithi102 ceph-mon[78322]: Upgrade: Updating mds.cephfs.smithi102.hrbrsp 2024-03-26T19:02:05.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:05 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:05.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:05 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hrbrsp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T19:02:05.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:05 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:05.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:05 smithi102 ceph-mon[78322]: Deploying daemon mds.cephfs.smithi102.hrbrsp on smithi102 2024-03-26T19:02:05.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:05 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/4181007851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:05.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.660+0000 7f29673ca700 1 -- 172.21.15.102:0/347794315 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2960105cc0 con 0x7f294c06e460 2024-03-26T19:02:05.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.661+0000 7f29567fc700 1 -- 172.21.15.102:0/347794315 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2960105cc0 con 0x7f294c06e460 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: "mon", 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: "osd" 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "13/23 daemons upgraded", 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading mds daemons", 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T19:02:05.656 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:02:05.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.663+0000 7f29673ca700 1 -- 172.21.15.102:0/347794315 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f294c06e460 msgr2=0x7f294c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.663+0000 7f29673ca700 1 --2- 172.21.15.102:0/347794315 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f294c06e460 0x7f294c070910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f295c006010 tx=0x7f295c00b540 comp rx=0 tx=0).stop 2024-03-26T19:02:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.663+0000 7f29673ca700 1 -- 172.21.15.102:0/347794315 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2960100170 msgr2=0x7f2960074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.663+0000 7f29673ca700 1 --2- 172.21.15.102:0/347794315 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2960100170 0x7f2960074af0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f295000d900 tx=0x7f295000dc10 comp rx=0 tx=0).stop 2024-03-26T19:02:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.663+0000 7f29673ca700 1 -- 172.21.15.102:0/347794315 shutdown_connections 2024-03-26T19:02:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.663+0000 7f29673ca700 1 --2- 172.21.15.102:0/347794315 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f294c06e460 0x7f294c070910 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.663+0000 7f29673ca700 1 --2- 172.21.15.102:0/347794315 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2960100170 0x7f2960074af0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.663+0000 7f29673ca700 1 --2- 172.21.15.102:0/347794315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2960101370 0x7f2960073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.663+0000 7f29673ca700 1 -- 172.21.15.102:0/347794315 >> 172.21.15.102:0/347794315 conn(0x7f29600fb6e0 msgr2=0x7f29601045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.663+0000 7f29673ca700 1 -- 172.21.15.102:0/347794315 shutdown_connections 2024-03-26T19:02:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.663+0000 7f29673ca700 1 -- 172.21.15.102:0/347794315 wait complete. 2024-03-26T19:02:05.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.769+0000 7fce399a4700 1 -- 172.21.15.102:0/3728487100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce34100250 msgr2=0x7fce341006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:05.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.769+0000 7fce399a4700 1 --2- 172.21.15.102:0/3728487100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce34100250 0x7fce341006a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fce1c009910 tx=0x7fce1c009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:05.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.769+0000 7fce399a4700 1 -- 172.21.15.102:0/3728487100 shutdown_connections 2024-03-26T19:02:05.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.769+0000 7fce399a4700 1 --2- 172.21.15.102:0/3728487100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce34100250 0x7fce341006a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.769+0000 7fce399a4700 1 --2- 172.21.15.102:0/3728487100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce340ff050 0x7fce340ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.769+0000 7fce399a4700 1 -- 172.21.15.102:0/3728487100 >> 172.21.15.102:0/3728487100 conn(0x7fce340fa5e0 msgr2=0x7fce340fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:05.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.769+0000 7fce399a4700 1 -- 172.21.15.102:0/3728487100 shutdown_connections 2024-03-26T19:02:05.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.769+0000 7fce399a4700 1 -- 172.21.15.102:0/3728487100 wait complete. 2024-03-26T19:02:05.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.769+0000 7fce399a4700 1 Processor -- start 2024-03-26T19:02:05.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.769+0000 7fce399a4700 1 -- start start 2024-03-26T19:02:05.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce399a4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340ff050 0x7fce34068f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:05.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce399a4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce34100250 0x7fce3419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:05.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce399a4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3419aca0 con 0x7fce340ff050 2024-03-26T19:02:05.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce399a4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3419ade0 con 0x7fce34100250 2024-03-26T19:02:05.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce32ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340ff050 0x7fce34068f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:05.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce32ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340ff050 0x7fce34068f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41496/0 (socket says 172.21.15.102:41496) 2024-03-26T19:02:05.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce32ffd700 1 -- 172.21.15.102:0/967833296 learned_addr learned my addr 172.21.15.102:0/967833296 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce2a7fc700 1 --2- 172.21.15.102:0/967833296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce34100250 0x7fce3419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce32ffd700 1 -- 172.21.15.102:0/967833296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce34100250 msgr2=0x7fce3419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce32ffd700 1 --2- 172.21.15.102:0/967833296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce34100250 0x7fce3419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce32ffd700 1 -- 172.21.15.102:0/967833296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1c0095f0 con 0x7fce340ff050 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce32ffd700 1 --2- 172.21.15.102:0/967833296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340ff050 0x7fce34068f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fce2400d900 tx=0x7fce2400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce389a2700 1 -- 172.21.15.102:0/967833296 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce240041f0 con 0x7fce340ff050 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce389a2700 1 -- 172.21.15.102:0/967833296 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fce24004350 con 0x7fce340ff050 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce399a4700 1 -- 172.21.15.102:0/967833296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3419b0c0 con 0x7fce340ff050 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce389a2700 1 -- 172.21.15.102:0/967833296 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce24003d20 con 0x7fce340ff050 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.770+0000 7fce399a4700 1 -- 172.21.15.102:0/967833296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3419b610 con 0x7fce340ff050 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.771+0000 7fce399a4700 1 -- 172.21.15.102:0/967833296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce34107ba0 con 0x7fce340ff050 2024-03-26T19:02:05.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.772+0000 7fce389a2700 1 -- 172.21.15.102:0/967833296 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fce24009730 con 0x7fce340ff050 2024-03-26T19:02:05.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.772+0000 7fce389a2700 1 --2- 172.21.15.102:0/967833296 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce2006e460 0x7fce20070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:05.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.773+0000 7fce2a7fc700 1 --2- 172.21.15.102:0/967833296 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce2006e460 0x7fce20070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:05.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.773+0000 7fce389a2700 1 -- 172.21.15.102:0/967833296 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fce2408e1c0 con 0x7fce340ff050 2024-03-26T19:02:05.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.773+0000 7fce2a7fc700 1 --2- 172.21.15.102:0/967833296 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce2006e460 0x7fce20070910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fce340694b0 tx=0x7fce1c00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:05.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:05.777+0000 7fce389a2700 1 -- 172.21.15.102:0/967833296 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fce2405af70 con 0x7fce340ff050 2024-03-26T19:02:05.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:05 smithi190 ceph-mon[65689]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:05.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:05 smithi190 ceph-mon[65689]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:05.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:05 smithi190 ceph-mon[65689]: Upgrade: Updating mds.cephfs.smithi102.hrbrsp 2024-03-26T19:02:05.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:05 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:05.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:05 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hrbrsp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T19:02:05.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:05 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:05.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:05 smithi190 ceph-mon[65689]: Deploying daemon mds.cephfs.smithi102.hrbrsp on smithi102 2024-03-26T19:02:05.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:05 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/4181007851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:06.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.066+0000 7fce399a4700 1 -- 172.21.15.102:0/967833296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fce34107dc0 con 0x7fce340ff050 2024-03-26T19:02:06.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.067+0000 7fce389a2700 1 -- 172.21.15.102:0/967833296 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fce2405ab00 con 0x7fce340ff050 2024-03-26T19:02:06.061 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T19:02:06.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.069+0000 7fce399a4700 1 -- 172.21.15.102:0/967833296 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce2006e460 msgr2=0x7fce20070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:06.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.069+0000 7fce399a4700 1 --2- 172.21.15.102:0/967833296 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce2006e460 0x7fce20070910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fce340694b0 tx=0x7fce1c00b540 comp rx=0 tx=0).stop 2024-03-26T19:02:06.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.069+0000 7fce399a4700 1 -- 172.21.15.102:0/967833296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340ff050 msgr2=0x7fce34068f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:06.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.069+0000 7fce399a4700 1 --2- 172.21.15.102:0/967833296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340ff050 0x7fce34068f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fce2400d900 tx=0x7fce2400dc10 comp rx=0 tx=0).stop 2024-03-26T19:02:06.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.069+0000 7fce399a4700 1 -- 172.21.15.102:0/967833296 shutdown_connections 2024-03-26T19:02:06.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.069+0000 7fce399a4700 1 --2- 172.21.15.102:0/967833296 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fce2006e460 0x7fce20070910 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:06.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.069+0000 7fce399a4700 1 --2- 172.21.15.102:0/967833296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340ff050 0x7fce34068f30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:06.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.069+0000 7fce399a4700 1 --2- 172.21.15.102:0/967833296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce34100250 0x7fce3419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:06.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.069+0000 7fce399a4700 1 -- 172.21.15.102:0/967833296 >> 172.21.15.102:0/967833296 conn(0x7fce340fa5e0 msgr2=0x7fce34103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:06.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.069+0000 7fce399a4700 1 -- 172.21.15.102:0/967833296 shutdown_connections 2024-03-26T19:02:06.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:06.069+0000 7fce399a4700 1 -- 172.21.15.102:0/967833296 wait complete. 2024-03-26T19:02:06.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:06 smithi102 ceph-mon[78322]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:06.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:06 smithi102 ceph-mon[78322]: pgmap v198: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:06.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:06 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1833605159' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:02:06.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:06 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/967833296' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:02:06.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:06 smithi190 ceph-mon[65689]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:06.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:06 smithi190 ceph-mon[65689]: pgmap v198: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:06.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:06 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1833605159' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:02:06.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:06 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/967833296' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:02:06.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:06 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102[78318]: 2024-03-26T19:02:06.597+0000 7f9567616700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T19:02:07.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:07 smithi102 ceph-mon[78322]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:07.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:07 smithi102 ceph-mon[78322]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T19:02:07.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:07 smithi102 ceph-mon[78322]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T19:02:07.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:07 smithi102 ceph-mon[78322]: osdmap e82: 6 total, 6 up, 6 in 2024-03-26T19:02:07.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:07 smithi102 ceph-mon[78322]: Standby daemon mds.cephfs.smithi190.hwsvdg assigned to filesystem cephfs as rank 0 2024-03-26T19:02:07.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:07 smithi102 ceph-mon[78322]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T19:02:07.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:07 smithi102 ceph-mon[78322]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-03-26T19:02:07.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:07 smithi102 ceph-mon[78322]: fsmap cephfs:1/1 {0=cephfs.smithi190.hwsvdg=up:replay} 2 up:standby 2024-03-26T19:02:07.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:07 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:07.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:07 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:02:07.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:07 smithi190 ceph-mon[65689]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:07.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:07 smithi190 ceph-mon[65689]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T19:02:07.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:07 smithi190 ceph-mon[65689]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T19:02:07.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:07 smithi190 ceph-mon[65689]: osdmap e82: 6 total, 6 up, 6 in 2024-03-26T19:02:07.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:07 smithi190 ceph-mon[65689]: Standby daemon mds.cephfs.smithi190.hwsvdg assigned to filesystem cephfs as rank 0 2024-03-26T19:02:07.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:07 smithi190 ceph-mon[65689]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T19:02:07.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:07 smithi190 ceph-mon[65689]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-03-26T19:02:07.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:07 smithi190 ceph-mon[65689]: fsmap cephfs:1/1 {0=cephfs.smithi190.hwsvdg=up:replay} 2 up:standby 2024-03-26T19:02:07.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:07 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:07.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:07 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:02:08.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:08 smithi102 ceph-mon[78322]: pgmap v200: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:08.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T19:02:08.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:08 smithi190 ceph-mon[65689]: pgmap v200: 65 pgs: 65 active+clean; 182 MiB data, 804 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:08.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T19:02:10.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:10 smithi102 ceph-mon[78322]: pgmap v201: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 0 op/s 2024-03-26T19:02:10.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:10 smithi190 ceph-mon[65689]: pgmap v201: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 0 op/s 2024-03-26T19:02:12.583 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:12 smithi190 ceph-mon[65689]: pgmap v202: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 9.6 MiB/s rd, 3 op/s 2024-03-26T19:02:12.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:12 smithi102 ceph-mon[78322]: pgmap v202: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 9.6 MiB/s rd, 3 op/s 2024-03-26T19:02:14.526 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:14 smithi102 ceph-mon[78322]: pgmap v203: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 12 MiB/s rd, 4 op/s 2024-03-26T19:02:14.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:14 smithi190 ceph-mon[65689]: pgmap v203: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 12 MiB/s rd, 4 op/s 2024-03-26T19:02:15.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:15 smithi102 ceph-mon[78322]: reconnect by client.14546 v1:172.21.15.190:0/1799840371 after 0 2024-03-26T19:02:15.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:15 smithi102 ceph-mon[78322]: reconnect by client.14540 v1:192.168.0.1:0/2355540585 after 0 2024-03-26T19:02:15.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:15 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] up:reconnect 2024-03-26T19:02:15.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:15 smithi102 ceph-mon[78322]: fsmap cephfs:1/1 {0=cephfs.smithi190.hwsvdg=up:reconnect} 2 up:standby 2024-03-26T19:02:15.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:15 smithi190 ceph-mon[65689]: reconnect by client.14546 v1:172.21.15.190:0/1799840371 after 0 2024-03-26T19:02:15.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:15 smithi190 ceph-mon[65689]: reconnect by client.14540 v1:192.168.0.1:0/2355540585 after 0 2024-03-26T19:02:15.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:15 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] up:reconnect 2024-03-26T19:02:15.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:15 smithi190 ceph-mon[65689]: fsmap cephfs:1/1 {0=cephfs.smithi190.hwsvdg=up:reconnect} 2 up:standby 2024-03-26T19:02:16.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:16 smithi102 ceph-mon[78322]: pgmap v204: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 5 op/s 2024-03-26T19:02:16.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:16 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] up:rejoin 2024-03-26T19:02:16.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:16 smithi102 ceph-mon[78322]: fsmap cephfs:1/1 {0=cephfs.smithi190.hwsvdg=up:rejoin} 2 up:standby 2024-03-26T19:02:16.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:16 smithi102 ceph-mon[78322]: daemon mds.cephfs.smithi190.hwsvdg is now active in filesystem cephfs as rank 0 2024-03-26T19:02:16.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:16 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:16.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:16 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:16.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:16 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:16.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:16 smithi190 ceph-mon[65689]: pgmap v204: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 5 op/s 2024-03-26T19:02:16.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:16 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] up:rejoin 2024-03-26T19:02:16.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:16 smithi190 ceph-mon[65689]: fsmap cephfs:1/1 {0=cephfs.smithi190.hwsvdg=up:rejoin} 2 up:standby 2024-03-26T19:02:16.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:16 smithi190 ceph-mon[65689]: daemon mds.cephfs.smithi190.hwsvdg is now active in filesystem cephfs as rank 0 2024-03-26T19:02:16.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:16 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:16.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:16 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:16.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:16 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:17.588 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:17 smithi102 ceph-mon[78322]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T19:02:17.588 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:17 smithi102 ceph-mon[78322]: Cluster is now healthy 2024-03-26T19:02:17.588 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:17 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] up:active 2024-03-26T19:02:17.588 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:17 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.102:6828/3007238267,v1:172.21.15.102:6829/3007238267] up:boot 2024-03-26T19:02:17.588 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:17 smithi102 ceph-mon[78322]: fsmap cephfs:1 {0=cephfs.smithi190.hwsvdg=up:active} 3 up:standby 2024-03-26T19:02:17.588 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:17 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T19:02:17.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:17 smithi190 ceph-mon[65689]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T19:02:17.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:17 smithi190 ceph-mon[65689]: Cluster is now healthy 2024-03-26T19:02:17.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:17 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] up:active 2024-03-26T19:02:17.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:17 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.102:6828/3007238267,v1:172.21.15.102:6829/3007238267] up:boot 2024-03-26T19:02:17.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:17 smithi190 ceph-mon[65689]: fsmap cephfs:1 {0=cephfs.smithi190.hwsvdg=up:active} 3 up:standby 2024-03-26T19:02:17.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:17 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T19:02:18.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:18 smithi102 ceph-mon[78322]: pgmap v205: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 95 B/s wr, 5 op/s 2024-03-26T19:02:18.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:18 smithi190 ceph-mon[65689]: pgmap v205: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 95 B/s wr, 5 op/s 2024-03-26T19:02:20.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:20 smithi102 ceph-mon[78322]: pgmap v206: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-26T19:02:20.781 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:20 smithi190 ceph-mon[65689]: pgmap v206: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-26T19:02:22.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:22.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:22.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:22 smithi190 ceph-mon[65689]: pgmap v207: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-26T19:02:22.309 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:22.309 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:22.309 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:22 smithi102 ceph-mon[78322]: pgmap v207: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-26T19:02:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:23.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:23.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:23.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:02:23.531 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:23.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:23.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:23.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:02:24.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:24 smithi102 ceph-mon[78322]: pgmap v208: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 6.8 MiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-26T19:02:24.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:24 smithi190 ceph-mon[65689]: pgmap v208: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 6.8 MiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-26T19:02:26.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: pgmap v209: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 4.8 MiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-26T19:02:26.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:26.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:26.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:26.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:02:26.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:26.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:26.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:26.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:26.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:26.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:26.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:26 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi190.vykizb"]}]: dispatch 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: pgmap v209: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 4.8 MiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:26.656 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:26 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi190.vykizb"]}]: dispatch 2024-03-26T19:02:27.608 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:27 smithi190 ceph-mon[65689]: Upgrade: It appears safe to stop mds.cephfs.smithi190.vykizb 2024-03-26T19:02:27.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:27 smithi102 ceph-mon[78322]: Upgrade: It appears safe to stop mds.cephfs.smithi190.vykizb 2024-03-26T19:02:28.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:28 smithi190 ceph-mon[65689]: pgmap v210: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 15 KiB/s wr, 7 op/s 2024-03-26T19:02:28.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:28 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:28.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:28 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.vykizb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T19:02:28.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:28 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:28.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:28 smithi102 ceph-mon[78322]: pgmap v210: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 15 KiB/s wr, 7 op/s 2024-03-26T19:02:28.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:28 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:28.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:28 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.vykizb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T19:02:28.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:28 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:29.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:29 smithi190 ceph-mon[65689]: Upgrade: Updating mds.cephfs.smithi190.vykizb 2024-03-26T19:02:29.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:29 smithi190 ceph-mon[65689]: Deploying daemon mds.cephfs.smithi190.vykizb on smithi190 2024-03-26T19:02:29.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:29 smithi102 ceph-mon[78322]: Upgrade: Updating mds.cephfs.smithi190.vykizb 2024-03-26T19:02:29.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:29 smithi102 ceph-mon[78322]: Deploying daemon mds.cephfs.smithi190.vykizb on smithi190 2024-03-26T19:02:30.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:30 smithi102 ceph-mon[78322]: pgmap v211: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-03-26T19:02:30.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:30 smithi190 ceph-mon[65689]: pgmap v211: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-03-26T19:02:31.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:31 smithi102 ceph-mon[78322]: osdmap e83: 6 total, 6 up, 6 in 2024-03-26T19:02:31.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:31 smithi102 ceph-mon[78322]: fsmap cephfs:1 {0=cephfs.smithi190.hwsvdg=up:active} 2 up:standby 2024-03-26T19:02:31.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:31 smithi190 ceph-mon[65689]: osdmap e83: 6 total, 6 up, 6 in 2024-03-26T19:02:31.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:31 smithi190 ceph-mon[65689]: fsmap cephfs:1 {0=cephfs.smithi190.hwsvdg=up:active} 2 up:standby 2024-03-26T19:02:32.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:32 smithi102 ceph-mon[78322]: pgmap v213: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:32.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:32 smithi190 ceph-mon[65689]: pgmap v213: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:33.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:33 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T19:02:33.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:33 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T19:02:34.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:34 smithi102 ceph-mon[78322]: pgmap v214: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:34.675 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:34 smithi190 ceph-mon[65689]: pgmap v214: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:36.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:36.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:36.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:36.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:35 smithi190 ceph-mon[65689]: pgmap v215: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:36.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:35 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.190:6824/782636706,v1:172.21.15.190:6825/782636706] up:boot 2024-03-26T19:02:36.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:35 smithi190 ceph-mon[65689]: fsmap cephfs:1 {0=cephfs.smithi190.hwsvdg=up:active} 3 up:standby 2024-03-26T19:02:36.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T19:02:36.182 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:36.182 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:36.182 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:36.182 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:35 smithi102 ceph-mon[78322]: pgmap v215: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:36.182 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:35 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.190:6824/782636706,v1:172.21.15.190:6825/782636706] up:boot 2024-03-26T19:02:36.183 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:35 smithi102 ceph-mon[78322]: fsmap cephfs:1 {0=cephfs.smithi190.hwsvdg=up:active} 3 up:standby 2024-03-26T19:02:36.183 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T19:02:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.187+0000 7f3d92c86700 1 -- 172.21.15.102:0/1601428326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d8c103510 msgr2=0x7f3d8c1058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.187+0000 7f3d92c86700 1 --2- 172.21.15.102:0/1601428326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d8c103510 0x7f3d8c1058f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009910 tx=0x7f3d78009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.187+0000 7f3d92c86700 1 -- 172.21.15.102:0/1601428326 shutdown_connections 2024-03-26T19:02:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.187+0000 7f3d92c86700 1 --2- 172.21.15.102:0/1601428326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d8c103510 0x7f3d8c1058f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.187+0000 7f3d92c86700 1 --2- 172.21.15.102:0/1601428326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d8c100bf0 0x7f3d8c102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.187+0000 7f3d92c86700 1 -- 172.21.15.102:0/1601428326 >> 172.21.15.102:0/1601428326 conn(0x7f3d8c0fa610 msgr2=0x7f3d8c0fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.187+0000 7f3d92c86700 1 -- 172.21.15.102:0/1601428326 shutdown_connections 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.187+0000 7f3d92c86700 1 -- 172.21.15.102:0/1601428326 wait complete. 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d92c86700 1 Processor -- start 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d92c86700 1 -- start start 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d92c86700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d8c100bf0 0x7f3d8c100550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d92c86700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d8c103510 0x7f3d8c0feba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d92c86700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0ff0e0 con 0x7f3d8c100bf0 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d92c86700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0ff220 con 0x7f3d8c103510 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d90a22700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d8c100bf0 0x7f3d8c100550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d90a22700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d8c100bf0 0x7f3d8c100550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56654/0 (socket says 172.21.15.102:56654) 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d90a22700 1 -- 172.21.15.102:0/4236269572 learned_addr learned my addr 172.21.15.102:0/4236269572 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d8bfff700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d8c103510 0x7f3d8c0feba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d90a22700 1 -- 172.21.15.102:0/4236269572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d8c103510 msgr2=0x7f3d8c0feba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d90a22700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d8c103510 0x7f3d8c0feba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d90a22700 1 -- 172.21.15.102:0/4236269572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d780095f0 con 0x7f3d8c100bf0 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d8bfff700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d8c103510 0x7f3d8c0feba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.188+0000 7f3d90a22700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d8c100bf0 0x7f3d8c100550 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000d900 tx=0x7f3d8000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.189+0000 7f3d89ffb700 1 -- 172.21.15.102:0/4236269572 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d800041f0 con 0x7f3d8c100bf0 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.189+0000 7f3d92c86700 1 -- 172.21.15.102:0/4236269572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8c0ff500 con 0x7f3d8c100bf0 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.189+0000 7f3d89ffb700 1 -- 172.21.15.102:0/4236269572 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3d80004ce0 con 0x7f3d8c100bf0 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.189+0000 7f3d89ffb700 1 -- 172.21.15.102:0/4236269572 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d8000b750 con 0x7f3d8c100bf0 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.189+0000 7f3d92c86700 1 -- 172.21.15.102:0/4236269572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8c19d9c0 con 0x7f3d8c100bf0 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.190+0000 7f3d92c86700 1 -- 172.21.15.102:0/4236269572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c18fb80 con 0x7f3d8c100bf0 2024-03-26T19:02:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.190+0000 7f3d89ffb700 1 -- 172.21.15.102:0/4236269572 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3d80004350 con 0x7f3d8c100bf0 2024-03-26T19:02:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.191+0000 7f3d89ffb700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3d7c06e460 0x7f3d7c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.191+0000 7f3d89ffb700 1 -- 172.21.15.102:0/4236269572 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3d8008e610 con 0x7f3d8c100bf0 2024-03-26T19:02:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.191+0000 7f3d8bfff700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3d7c06e460 0x7f3d7c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.191+0000 7f3d8bfff700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3d7c06e460 0x7f3d7c070910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800b5c0 tx=0x7f3d78005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.196+0000 7f3d89ffb700 1 -- 172.21.15.102:0/4236269572 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d8005b2f0 con 0x7f3d8c100bf0 2024-03-26T19:02:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.412+0000 7f3d92c86700 1 -- 172.21.15.102:0/4236269572 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d8c061190 con 0x7f3d7c06e460 2024-03-26T19:02:36.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.414+0000 7f3d89ffb700 1 -- 172.21.15.102:0/4236269572 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3d8c061190 con 0x7f3d7c06e460 2024-03-26T19:02:36.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.416+0000 7f3d92c86700 1 -- 172.21.15.102:0/4236269572 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3d7c06e460 msgr2=0x7f3d7c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:36.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.416+0000 7f3d92c86700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3d7c06e460 0x7f3d7c070910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800b5c0 tx=0x7f3d78005e20 comp rx=0 tx=0).stop 2024-03-26T19:02:36.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.416+0000 7f3d92c86700 1 -- 172.21.15.102:0/4236269572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d8c100bf0 msgr2=0x7f3d8c100550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:36.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.416+0000 7f3d92c86700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d8c100bf0 0x7f3d8c100550 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000d900 tx=0x7f3d8000dcc0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.416+0000 7f3d92c86700 1 -- 172.21.15.102:0/4236269572 shutdown_connections 2024-03-26T19:02:36.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.416+0000 7f3d92c86700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3d7c06e460 0x7f3d7c070910 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.416+0000 7f3d92c86700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d8c100bf0 0x7f3d8c100550 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.416+0000 7f3d92c86700 1 --2- 172.21.15.102:0/4236269572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d8c103510 0x7f3d8c0feba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.416+0000 7f3d92c86700 1 -- 172.21.15.102:0/4236269572 >> 172.21.15.102:0/4236269572 conn(0x7f3d8c0fa610 msgr2=0x7f3d8c0fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:36.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.416+0000 7f3d92c86700 1 -- 172.21.15.102:0/4236269572 shutdown_connections 2024-03-26T19:02:36.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.416+0000 7f3d92c86700 1 -- 172.21.15.102:0/4236269572 wait complete. 2024-03-26T19:02:36.422 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T19:02:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.521+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/233412729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f181017e0 msgr2=0x7f9f18101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.521+0000 7f9f1e8bf700 1 --2- 172.21.15.102:0/233412729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f181017e0 0x7f9f18101bf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009910 tx=0x7f9f08009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.521+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/233412729 shutdown_connections 2024-03-26T19:02:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.521+0000 7f9f1e8bf700 1 --2- 172.21.15.102:0/233412729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f18102130 0x7f9f180fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.521+0000 7f9f1e8bf700 1 --2- 172.21.15.102:0/233412729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f181017e0 0x7f9f18101bf0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.521+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/233412729 >> 172.21.15.102:0/233412729 conn(0x7f9f180f9e50 msgr2=0x7f9f180fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.521+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/233412729 shutdown_connections 2024-03-26T19:02:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.522+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/233412729 wait complete. 2024-03-26T19:02:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.522+0000 7f9f1e8bf700 1 Processor -- start 2024-03-26T19:02:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.522+0000 7f9f1e8bf700 1 -- start start 2024-03-26T19:02:36.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.522+0000 7f9f1e8bf700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f181017e0 0x7f9f18195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:36.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.522+0000 7f9f1e8bf700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f18102130 0x7f9f18195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:36.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.522+0000 7f9f1e8bf700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181964f0 con 0x7f9f18102130 2024-03-26T19:02:36.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.522+0000 7f9f1e8bf700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18196630 con 0x7f9f181017e0 2024-03-26T19:02:36.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f177fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f18102130 0x7f9f18195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:36.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f177fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f18102130 0x7f9f18195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56686/0 (socket says 172.21.15.102:56686) 2024-03-26T19:02:36.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f177fe700 1 -- 172.21.15.102:0/3317565140 learned_addr learned my addr 172.21.15.102:0/3317565140 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f17fff700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f181017e0 0x7f9f18195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f177fe700 1 -- 172.21.15.102:0/3317565140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f181017e0 msgr2=0x7f9f18195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f177fe700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f181017e0 0x7f9f18195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f177fe700 1 -- 172.21.15.102:0/3317565140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f080095f0 con 0x7f9f18102130 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f17fff700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f181017e0 0x7f9f18195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f177fe700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f18102130 0x7f9f18195ed0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00d900 tx=0x7f9f0c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f157fa700 1 -- 172.21.15.102:0/3317565140 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f0c009900 con 0x7f9f18102130 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f157fa700 1 -- 172.21.15.102:0/3317565140 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9f0c010460 con 0x7f9f18102130 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.523+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/3317565140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1819b0e0 con 0x7f9f18102130 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.524+0000 7f9f157fa700 1 -- 172.21.15.102:0/3317565140 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f0c00f5f0 con 0x7f9f18102130 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.524+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/3317565140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1819b630 con 0x7f9f18102130 2024-03-26T19:02:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.525+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/3317565140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f18108c70 con 0x7f9f18102130 2024-03-26T19:02:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.525+0000 7f9f157fa700 1 -- 172.21.15.102:0/3317565140 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9f0c0105d0 con 0x7f9f18102130 2024-03-26T19:02:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.525+0000 7f9f157fa700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9f0006e460 0x7f9f00070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.525+0000 7f9f157fa700 1 -- 172.21.15.102:0/3317565140 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9f0c08e190 con 0x7f9f18102130 2024-03-26T19:02:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.525+0000 7f9f17fff700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9f0006e460 0x7f9f00070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.526+0000 7f9f17fff700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9f0006e460 0x7f9f00070910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9f08005b40 tx=0x7f9f08005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.530+0000 7f9f157fa700 1 -- 172.21.15.102:0/3317565140 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9f0c05aef0 con 0x7f9f18102130 2024-03-26T19:02:36.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.746+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/3317565140 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f18105ed0 con 0x7f9f0006e460 2024-03-26T19:02:36.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.747+0000 7f9f157fa700 1 -- 172.21.15.102:0/3317565140 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9f18105ed0 con 0x7f9f0006e460 2024-03-26T19:02:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.749+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/3317565140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9f0006e460 msgr2=0x7f9f00070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.749+0000 7f9f1e8bf700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9f0006e460 0x7f9f00070910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9f08005b40 tx=0x7f9f08005a90 comp rx=0 tx=0).stop 2024-03-26T19:02:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.749+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/3317565140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f18102130 msgr2=0x7f9f18195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.749+0000 7f9f1e8bf700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f18102130 0x7f9f18195ed0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00d900 tx=0x7f9f0c00dcc0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.750+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/3317565140 shutdown_connections 2024-03-26T19:02:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.750+0000 7f9f1e8bf700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f181017e0 0x7f9f18195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.750+0000 7f9f1e8bf700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9f0006e460 0x7f9f00070910 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.750+0000 7f9f1e8bf700 1 --2- 172.21.15.102:0/3317565140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f18102130 0x7f9f18195ed0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.750+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/3317565140 >> 172.21.15.102:0/3317565140 conn(0x7f9f180f9e50 msgr2=0x7f9f181047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.750+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/3317565140 shutdown_connections 2024-03-26T19:02:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.750+0000 7f9f1e8bf700 1 -- 172.21.15.102:0/3317565140 wait complete. 2024-03-26T19:02:36.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.855+0000 7fe034617700 1 -- 172.21.15.102:0/1814062121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe02c0fe440 msgr2=0x7fe02c0fe850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:36.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.855+0000 7fe034617700 1 --2- 172.21.15.102:0/1814062121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe02c0fe440 0x7fe02c0fe850 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe028009910 tx=0x7fe028009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.855+0000 7fe034617700 1 -- 172.21.15.102:0/1814062121 shutdown_connections 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.855+0000 7fe034617700 1 --2- 172.21.15.102:0/1814062121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe02c0fed90 0x7fe02c1014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.855+0000 7fe034617700 1 --2- 172.21.15.102:0/1814062121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe02c0fe440 0x7fe02c0fe850 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.855+0000 7fe034617700 1 -- 172.21.15.102:0/1814062121 >> 172.21.15.102:0/1814062121 conn(0x7fe02c0f9e50 msgr2=0x7fe02c0fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.855+0000 7fe034617700 1 -- 172.21.15.102:0/1814062121 shutdown_connections 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.855+0000 7fe034617700 1 -- 172.21.15.102:0/1814062121 wait complete. 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.856+0000 7fe034617700 1 Processor -- start 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.856+0000 7fe034617700 1 -- start start 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.856+0000 7fe034617700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe02c0fe440 0x7fe02c100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.856+0000 7fe034617700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe02c0fed90 0x7fe02c0ff470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.856+0000 7fe034617700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0ff9b0 con 0x7fe02c0fed90 2024-03-26T19:02:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.856+0000 7fe034617700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0ffaf0 con 0x7fe02c0fe440 2024-03-26T19:02:36.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.856+0000 7fe031bb2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe02c0fed90 0x7fe02c0ff470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:36.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.856+0000 7fe0323b3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe02c0fe440 0x7fe02c100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:36.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.856+0000 7fe031bb2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe02c0fed90 0x7fe02c0ff470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56702/0 (socket says 172.21.15.102:56702) 2024-03-26T19:02:36.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.856+0000 7fe031bb2700 1 -- 172.21.15.102:0/1951632523 learned_addr learned my addr 172.21.15.102:0/1951632523 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:36.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.857+0000 7fe031bb2700 1 -- 172.21.15.102:0/1951632523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe02c0fe440 msgr2=0x7fe02c100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:36.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.857+0000 7fe031bb2700 1 --2- 172.21.15.102:0/1951632523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe02c0fe440 0x7fe02c100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:36.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.857+0000 7fe031bb2700 1 -- 172.21.15.102:0/1951632523 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0280095f0 con 0x7fe02c0fed90 2024-03-26T19:02:36.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.857+0000 7fe031bb2700 1 --2- 172.21.15.102:0/1951632523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe02c0fed90 0x7fe02c0ff470 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00d8d0 tx=0x7fe01c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:36.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.857+0000 7fe0237fe700 1 -- 172.21.15.102:0/1951632523 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe01c00f860 con 0x7fe02c0fed90 2024-03-26T19:02:36.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.857+0000 7fe0237fe700 1 -- 172.21.15.102:0/1951632523 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe01c00fe70 con 0x7fe02c0fed90 2024-03-26T19:02:36.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.857+0000 7fe034617700 1 -- 172.21.15.102:0/1951632523 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c0ffdd0 con 0x7fe02c0fed90 2024-03-26T19:02:36.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.857+0000 7fe0237fe700 1 -- 172.21.15.102:0/1951632523 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe01c00e5e0 con 0x7fe02c0fed90 2024-03-26T19:02:36.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.857+0000 7fe034617700 1 -- 172.21.15.102:0/1951632523 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c19da10 con 0x7fe02c0fed90 2024-03-26T19:02:36.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.858+0000 7fe034617700 1 -- 172.21.15.102:0/1951632523 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02c108c90 con 0x7fe02c0fed90 2024-03-26T19:02:36.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.859+0000 7fe0237fe700 1 -- 172.21.15.102:0/1951632523 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe01c010460 con 0x7fe02c0fed90 2024-03-26T19:02:36.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.859+0000 7fe0237fe700 1 --2- 172.21.15.102:0/1951632523 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe01806e460 0x7fe018070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:36.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.859+0000 7fe0237fe700 1 -- 172.21.15.102:0/1951632523 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe01c08e010 con 0x7fe02c0fed90 2024-03-26T19:02:36.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.859+0000 7fe0323b3700 1 --2- 172.21.15.102:0/1951632523 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe01806e460 0x7fe018070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:36.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.860+0000 7fe0323b3700 1 --2- 172.21.15.102:0/1951632523 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe01806e460 0x7fe018070910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe028006010 tx=0x7fe02800b560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:36.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:36.864+0000 7fe0237fe700 1 -- 172.21.15.102:0/1951632523 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe01c05acf0 con 0x7fe02c0fed90 2024-03-26T19:02:37.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.069+0000 7fe034617700 1 -- 172.21.15.102:0/1951632523 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe02c19dfc0 con 0x7fe01806e460 2024-03-26T19:02:37.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.080+0000 7fe0237fe700 1 -- 172.21.15.102:0/1951632523 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fe02c19dfc0 con 0x7fe01806e460 2024-03-26T19:02:37.075 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T19:02:37.075 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (14m) 16s ago 15m 25.7M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T19:02:37.075 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (15m) 16s ago 15m 8858k - 18.2.0 dc2bc1663786 4e80f31986bc 2024-03-26T19:02:37.075 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (14m) 98s ago 14m 8514k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T19:02:37.075 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (5m) 16s ago 15m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T19:02:37.075 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (4m) 98s ago 14m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T19:02:37.075 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (14m) 16s ago 15m 85.8M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (21s) 16s ago 10m 11.4M - 19.0.0-2311-g36051606 497bb87bc5da c530ed7dbd78 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (44s) 16s ago 10m 13.5M - 19.0.0-2311-g36051606 497bb87bc5da 63b8127a3849 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (10m) 98s ago 10m 16.8M - 18.2.0 dc2bc1663786 fd351a409d63 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 starting - - - - 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (7m) 16s ago 17m 508M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (6m) 98s ago 14m 451M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (6m) 16s ago 17m 50.2M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (5m) 98s ago 14m 36.6M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (15m) 16s ago 15m 16.6M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (14m) 98s ago 14m 15.1M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (4m) 16s ago 13m 105M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (3m) 16s ago 13m 114M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (3m) 16s ago 12m 97.0M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (2m) 98s ago 12m 98.2M 2240M 19.0.0-2311-g36051606 497bb87bc5da 03955331c962 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (2m) 98s ago 12m 102M 2240M 19.0.0-2311-g36051606 497bb87bc5da 23e140cd8961 2024-03-26T19:02:37.076 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (102s) 98s ago 11m 12.6M 2240M 19.0.0-2311-g36051606 497bb87bc5da 60afb0e4da18 2024-03-26T19:02:37.077 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (6m) 16s ago 15m 78.5M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T19:02:37.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.082+0000 7fe034617700 1 -- 172.21.15.102:0/1951632523 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe01806e460 msgr2=0x7fe018070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.082+0000 7fe034617700 1 --2- 172.21.15.102:0/1951632523 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe01806e460 0x7fe018070910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe028006010 tx=0x7fe02800b560 comp rx=0 tx=0).stop 2024-03-26T19:02:37.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.082+0000 7fe034617700 1 -- 172.21.15.102:0/1951632523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe02c0fed90 msgr2=0x7fe02c0ff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.082+0000 7fe034617700 1 --2- 172.21.15.102:0/1951632523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe02c0fed90 0x7fe02c0ff470 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00d8d0 tx=0x7fe01c00dbe0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.082+0000 7fe034617700 1 -- 172.21.15.102:0/1951632523 shutdown_connections 2024-03-26T19:02:37.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.082+0000 7fe034617700 1 --2- 172.21.15.102:0/1951632523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe02c0fe440 0x7fe02c100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.082+0000 7fe034617700 1 --2- 172.21.15.102:0/1951632523 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe01806e460 0x7fe018070910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.082+0000 7fe034617700 1 --2- 172.21.15.102:0/1951632523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe02c0fed90 0x7fe02c0ff470 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.082+0000 7fe034617700 1 -- 172.21.15.102:0/1951632523 >> 172.21.15.102:0/1951632523 conn(0x7fe02c0f9e50 msgr2=0x7fe02c1046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:37.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.082+0000 7fe034617700 1 -- 172.21.15.102:0/1951632523 shutdown_connections 2024-03-26T19:02:37.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.082+0000 7fe034617700 1 -- 172.21.15.102:0/1951632523 wait complete. 2024-03-26T19:02:37.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.185+0000 7f0f8b35a700 1 -- 172.21.15.102:0/743856331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f841013b0 msgr2=0x7f0f84101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.185+0000 7f0f8b35a700 1 --2- 172.21.15.102:0/743856331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f841013b0 0x7f0f84101800 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009910 tx=0x7f0f74009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:37.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.185+0000 7f0f8b35a700 1 -- 172.21.15.102:0/743856331 shutdown_connections 2024-03-26T19:02:37.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.185+0000 7f0f8b35a700 1 --2- 172.21.15.102:0/743856331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f841013b0 0x7f0f84101800 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.185+0000 7f0f8b35a700 1 --2- 172.21.15.102:0/743856331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f841001b0 0x7f0f841005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.185+0000 7f0f8b35a700 1 -- 172.21.15.102:0/743856331 >> 172.21.15.102:0/743856331 conn(0x7f0f840fb740 msgr2=0x7f0f840fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:37.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.185+0000 7f0f8b35a700 1 -- 172.21.15.102:0/743856331 shutdown_connections 2024-03-26T19:02:37.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.186+0000 7f0f8b35a700 1 -- 172.21.15.102:0/743856331 wait complete. 2024-03-26T19:02:37.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.186+0000 7f0f8b35a700 1 Processor -- start 2024-03-26T19:02:37.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.186+0000 7f0f8b35a700 1 -- start start 2024-03-26T19:02:37.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.186+0000 7f0f8b35a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f841001b0 0x7f0f841959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:37.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.186+0000 7f0f8b35a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f841013b0 0x7f0f84195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:37.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.186+0000 7f0f8b35a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f84196500 con 0x7f0f841001b0 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.186+0000 7f0f8b35a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f84196640 con 0x7f0f841013b0 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.186+0000 7f0f890f6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f841001b0 0x7f0f841959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.186+0000 7f0f890f6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f841001b0 0x7f0f841959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56732/0 (socket says 172.21.15.102:56732) 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.186+0000 7f0f890f6700 1 -- 172.21.15.102:0/1876994495 learned_addr learned my addr 172.21.15.102:0/1876994495 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.187+0000 7f0f888f5700 1 --2- 172.21.15.102:0/1876994495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f841013b0 0x7f0f84195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.187+0000 7f0f890f6700 1 -- 172.21.15.102:0/1876994495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f841013b0 msgr2=0x7f0f84195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.187+0000 7f0f890f6700 1 --2- 172.21.15.102:0/1876994495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f841013b0 0x7f0f84195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.187+0000 7f0f890f6700 1 -- 172.21.15.102:0/1876994495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f740095f0 con 0x7f0f841001b0 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.187+0000 7f0f890f6700 1 --2- 172.21.15.102:0/1876994495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f841001b0 0x7f0f841959a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000d900 tx=0x7f0f8000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.187+0000 7f0f7a7fc700 1 -- 172.21.15.102:0/1876994495 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f80009900 con 0x7f0f841001b0 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.187+0000 7f0f7a7fc700 1 -- 172.21.15.102:0/1876994495 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0f80010460 con 0x7f0f841001b0 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.187+0000 7f0f8b35a700 1 -- 172.21.15.102:0/1876994495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8419b0f0 con 0x7f0f841001b0 2024-03-26T19:02:37.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.188+0000 7f0f7a7fc700 1 -- 172.21.15.102:0/1876994495 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f8000f5b0 con 0x7f0f841001b0 2024-03-26T19:02:37.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.188+0000 7f0f8b35a700 1 -- 172.21.15.102:0/1876994495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8419b640 con 0x7f0f841001b0 2024-03-26T19:02:37.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.188+0000 7f0f8b35a700 1 -- 172.21.15.102:0/1876994495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f84108d20 con 0x7f0f841001b0 2024-03-26T19:02:37.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.189+0000 7f0f7a7fc700 1 -- 172.21.15.102:0/1876994495 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f0f800105d0 con 0x7f0f841001b0 2024-03-26T19:02:37.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.189+0000 7f0f7a7fc700 1 --2- 172.21.15.102:0/1876994495 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0f7006e460 0x7f0f70070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:37.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.189+0000 7f0f888f5700 1 --2- 172.21.15.102:0/1876994495 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0f7006e460 0x7f0f70070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:37.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.189+0000 7f0f7a7fc700 1 -- 172.21.15.102:0/1876994495 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0f8008dfa0 con 0x7f0f841001b0 2024-03-26T19:02:37.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.190+0000 7f0f888f5700 1 --2- 172.21.15.102:0/1876994495 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0f7006e460 0x7f0f70070910 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0f74006010 tx=0x7f0f7400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:37.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.194+0000 7f0f7a7fc700 1 -- 172.21.15.102:0/1876994495 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0f8005ac80 con 0x7f0f841001b0 2024-03-26T19:02:37.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.488+0000 7f0f8b35a700 1 -- 172.21.15.102:0/1876994495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0f8419bcd0 con 0x7f0f841001b0 2024-03-26T19:02:37.483 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:37 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:02:37.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.489+0000 7f0f7a7fc700 1 -- 172.21.15.102:0/1876994495 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f0f8005a810 con 0x7f0f841001b0 2024-03-26T19:02:37.484 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:02:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T19:02:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:02:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:02:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T19:02:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:02:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:02:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T19:02:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 6 2024-03-26T19:02:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:02:37.485 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T19:02:37.485 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-03-26T19:02:37.485 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 3 2024-03-26T19:02:37.485 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:02:37.485 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T19:02:37.485 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-03-26T19:02:37.485 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 13 2024-03-26T19:02:37.485 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T19:02:37.485 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:02:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.491+0000 7f0f8b35a700 1 -- 172.21.15.102:0/1876994495 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0f7006e460 msgr2=0x7f0f70070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.491+0000 7f0f8b35a700 1 --2- 172.21.15.102:0/1876994495 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0f7006e460 0x7f0f70070910 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0f74006010 tx=0x7f0f7400b540 comp rx=0 tx=0).stop 2024-03-26T19:02:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.491+0000 7f0f8b35a700 1 -- 172.21.15.102:0/1876994495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f841001b0 msgr2=0x7f0f841959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.491+0000 7f0f8b35a700 1 --2- 172.21.15.102:0/1876994495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f841001b0 0x7f0f841959a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000d900 tx=0x7f0f8000dc10 comp rx=0 tx=0).stop 2024-03-26T19:02:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.492+0000 7f0f8b35a700 1 -- 172.21.15.102:0/1876994495 shutdown_connections 2024-03-26T19:02:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.492+0000 7f0f8b35a700 1 --2- 172.21.15.102:0/1876994495 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f0f7006e460 0x7f0f70070910 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.492+0000 7f0f8b35a700 1 --2- 172.21.15.102:0/1876994495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f841001b0 0x7f0f841959a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.492+0000 7f0f8b35a700 1 --2- 172.21.15.102:0/1876994495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f841013b0 0x7f0f84195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.492+0000 7f0f8b35a700 1 -- 172.21.15.102:0/1876994495 >> 172.21.15.102:0/1876994495 conn(0x7f0f840fb740 msgr2=0x7f0f841045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.492+0000 7f0f8b35a700 1 -- 172.21.15.102:0/1876994495 shutdown_connections 2024-03-26T19:02:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.492+0000 7f0f8b35a700 1 -- 172.21.15.102:0/1876994495 wait complete. 2024-03-26T19:02:37.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.599+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4161496653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc100be0 msgr2=0x7f38cc102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.599+0000 7f38d2cc3700 1 --2- 172.21.15.102:0/4161496653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc100be0 0x7f38cc102fc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009910 tx=0x7f38c8009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:37.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.599+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4161496653 shutdown_connections 2024-03-26T19:02:37.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.599+0000 7f38d2cc3700 1 --2- 172.21.15.102:0/4161496653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38cc103500 0x7f38cc1058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.599+0000 7f38d2cc3700 1 --2- 172.21.15.102:0/4161496653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc100be0 0x7f38cc102fc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.599+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4161496653 >> 172.21.15.102:0/4161496653 conn(0x7f38cc0fa5c0 msgr2=0x7f38cc0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.599+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4161496653 shutdown_connections 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.599+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4161496653 wait complete. 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.599+0000 7f38d2cc3700 1 Processor -- start 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.599+0000 7f38d2cc3700 1 -- start start 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38d2cc3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38cc100be0 0x7f38cc195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38d2cc3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc103500 0x7f38cc195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38d2cc3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc1964b0 con 0x7f38cc103500 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38d2cc3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc1965f0 con 0x7f38cc100be0 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38c3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc103500 0x7f38cc195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38c3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc103500 0x7f38cc195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56760/0 (socket says 172.21.15.102:56760) 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38c3fff700 1 -- 172.21.15.102:0/4277699059 learned_addr learned my addr 172.21.15.102:0/4277699059 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38d0a5f700 1 --2- 172.21.15.102:0/4277699059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38cc100be0 0x7f38cc195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38c3fff700 1 -- 172.21.15.102:0/4277699059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38cc100be0 msgr2=0x7f38cc195950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38c3fff700 1 --2- 172.21.15.102:0/4277699059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38cc100be0 0x7f38cc195950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38c3fff700 1 -- 172.21.15.102:0/4277699059 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c80095f0 con 0x7f38cc103500 2024-03-26T19:02:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38c3fff700 1 --2- 172.21.15.102:0/4277699059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc103500 0x7f38cc195e90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00e920 tx=0x7f38bc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38c1ffb700 1 -- 172.21.15.102:0/4277699059 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38bc00c9d0 con 0x7f38cc103500 2024-03-26T19:02:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38c1ffb700 1 -- 172.21.15.102:0/4277699059 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f38bc010430 con 0x7f38cc103500 2024-03-26T19:02:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4277699059 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38cc19b0a0 con 0x7f38cc103500 2024-03-26T19:02:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.600+0000 7f38c1ffb700 1 -- 172.21.15.102:0/4277699059 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38bc018650 con 0x7f38cc103500 2024-03-26T19:02:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.601+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4277699059 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38cc19b5f0 con 0x7f38cc103500 2024-03-26T19:02:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.601+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4277699059 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38cc18fb70 con 0x7f38cc103500 2024-03-26T19:02:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.602+0000 7f38c1ffb700 1 -- 172.21.15.102:0/4277699059 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f38bc0105a0 con 0x7f38cc103500 2024-03-26T19:02:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.602+0000 7f38c1ffb700 1 --2- 172.21.15.102:0/4277699059 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f38b406e460 0x7f38b4070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.602+0000 7f38c1ffb700 1 -- 172.21.15.102:0/4277699059 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f38bc014070 con 0x7f38cc103500 2024-03-26T19:02:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.602+0000 7f38d0a5f700 1 --2- 172.21.15.102:0/4277699059 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f38b406e460 0x7f38b4070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.603+0000 7f38d0a5f700 1 --2- 172.21.15.102:0/4277699059 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f38b406e460 0x7f38b4070910 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009fd0 tx=0x7f38c8005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.607+0000 7f38c1ffb700 1 -- 172.21.15.102:0/4277699059 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f38bc053740 con 0x7f38cc103500 2024-03-26T19:02:37.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:37 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:02:37.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.854+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4277699059 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f38cc02cc30 con 0x7f38cc103500 2024-03-26T19:02:37.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.854+0000 7f38c1ffb700 1 -- 172.21.15.102:0/4277699059 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1897 (secure 0 0 0) 0x7f38bc05b3c0 con 0x7f38cc103500 2024-03-26T19:02:37.849 INFO:teuthology.orchestra.run.smithi102.stdout:e34 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 32 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T19:02:16.330672+0000 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T19:02:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 82 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 1 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:in 0 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=24293} 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 1 2024-03-26T19:02:37.851 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 24293 members: 24293 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{0:24293} state up:active seq 149 join_fscid=1 addr [v2:172.21.15.190:6826/624694186,v1:172.21.15.190:6827/624694186] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{-1:34436} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:02:37.852 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{-1:34468} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.102:6828/3007238267,v1:172.21.15.102:6829/3007238267] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:02:37.853 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.190:6824/782636706,v1:172.21.15.190:6825/782636706] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:02:37.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.856+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4277699059 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f38b406e460 msgr2=0x7f38b4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.856+0000 7f38d2cc3700 1 --2- 172.21.15.102:0/4277699059 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f38b406e460 0x7f38b4070910 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009fd0 tx=0x7f38c8005940 comp rx=0 tx=0).stop 2024-03-26T19:02:37.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.856+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4277699059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc103500 msgr2=0x7f38cc195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.856+0000 7f38d2cc3700 1 --2- 172.21.15.102:0/4277699059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc103500 0x7f38cc195e90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00e920 tx=0x7f38bc00ec30 comp rx=0 tx=0).stop 2024-03-26T19:02:37.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.857+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4277699059 shutdown_connections 2024-03-26T19:02:37.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.857+0000 7f38d2cc3700 1 --2- 172.21.15.102:0/4277699059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38cc100be0 0x7f38cc195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.857+0000 7f38d2cc3700 1 --2- 172.21.15.102:0/4277699059 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f38b406e460 0x7f38b4070910 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.857+0000 7f38d2cc3700 1 --2- 172.21.15.102:0/4277699059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc103500 0x7f38cc195e90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.857+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4277699059 >> 172.21.15.102:0/4277699059 conn(0x7f38cc0fa5c0 msgr2=0x7f38cc0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:37.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.857+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4277699059 shutdown_connections 2024-03-26T19:02:37.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.857+0000 7f38d2cc3700 1 -- 172.21.15.102:0/4277699059 wait complete. 2024-03-26T19:02:37.854 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 34 2024-03-26T19:02:37.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.961+0000 7f3856dfb700 1 -- 172.21.15.102:0/1894056336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850100170 msgr2=0x7f3850100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.961+0000 7f3856dfb700 1 --2- 172.21.15.102:0/1894056336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850100170 0x7f3850100580 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3844009910 tx=0x7f3844009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:37.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.961+0000 7f3856dfb700 1 -- 172.21.15.102:0/1894056336 shutdown_connections 2024-03-26T19:02:37.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.961+0000 7f3856dfb700 1 --2- 172.21.15.102:0/1894056336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3850101370 0x7f38501017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.961+0000 7f3856dfb700 1 --2- 172.21.15.102:0/1894056336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850100170 0x7f3850100580 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.961+0000 7f3856dfb700 1 -- 172.21.15.102:0/1894056336 >> 172.21.15.102:0/1894056336 conn(0x7f38500fb6e0 msgr2=0x7f38500fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.961+0000 7f3856dfb700 1 -- 172.21.15.102:0/1894056336 shutdown_connections 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.961+0000 7f3856dfb700 1 -- 172.21.15.102:0/1894056336 wait complete. 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.961+0000 7f3856dfb700 1 Processor -- start 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3856dfb700 1 -- start start 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3856dfb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850100170 0x7f3850195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3856dfb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3850101370 0x7f3850195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3856dfb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38501964f0 con 0x7f3850100170 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3856dfb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3850196630 con 0x7f3850101370 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3854b97700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850100170 0x7f3850195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3854b97700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850100170 0x7f3850195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56778/0 (socket says 172.21.15.102:56778) 2024-03-26T19:02:37.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3854b97700 1 -- 172.21.15.102:0/3093127230 learned_addr learned my addr 172.21.15.102:0/3093127230 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f384ffff700 1 --2- 172.21.15.102:0/3093127230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3850101370 0x7f3850195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3854b97700 1 -- 172.21.15.102:0/3093127230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3850101370 msgr2=0x7f3850195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3854b97700 1 --2- 172.21.15.102:0/3093127230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3850101370 0x7f3850195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3854b97700 1 -- 172.21.15.102:0/3093127230 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38440095f0 con 0x7f3850100170 2024-03-26T19:02:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3854b97700 1 --2- 172.21.15.102:0/3093127230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850100170 0x7f3850195990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f384400bda0 tx=0x7f384400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f384dffb700 1 -- 172.21.15.102:0/3093127230 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f384401d070 con 0x7f3850100170 2024-03-26T19:02:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f3856dfb700 1 -- 172.21.15.102:0/3093127230 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385019b080 con 0x7f3850100170 2024-03-26T19:02:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f384dffb700 1 -- 172.21.15.102:0/3093127230 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f384400f460 con 0x7f3850100170 2024-03-26T19:02:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.962+0000 7f384dffb700 1 -- 172.21.15.102:0/3093127230 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3844021620 con 0x7f3850100170 2024-03-26T19:02:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.963+0000 7f3856dfb700 1 -- 172.21.15.102:0/3093127230 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385019b570 con 0x7f3850100170 2024-03-26T19:02:37.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.964+0000 7f3856dfb700 1 -- 172.21.15.102:0/3093127230 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3850108cc0 con 0x7f3850100170 2024-03-26T19:02:37.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.964+0000 7f384dffb700 1 -- 172.21.15.102:0/3093127230 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3844003a60 con 0x7f3850100170 2024-03-26T19:02:37.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.965+0000 7f384dffb700 1 --2- 172.21.15.102:0/3093127230 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f384006e460 0x7f3840070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:37.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.965+0000 7f384dffb700 1 -- 172.21.15.102:0/3093127230 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f384408f770 con 0x7f3850100170 2024-03-26T19:02:37.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.965+0000 7f384ffff700 1 --2- 172.21.15.102:0/3093127230 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f384006e460 0x7f3840070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:37.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.965+0000 7f384ffff700 1 --2- 172.21.15.102:0/3093127230 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f384006e460 0x7f3840070910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f383c005950 tx=0x7f383c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:37.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:37.969+0000 7f384dffb700 1 -- 172.21.15.102:0/3093127230 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f384405c450 con 0x7f3850100170 2024-03-26T19:02:38.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.213+0000 7f3856dfb700 1 -- 172.21.15.102:0/3093127230 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f385019bc70 con 0x7f384006e460 2024-03-26T19:02:38.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.215+0000 7f384dffb700 1 -- 172.21.15.102:0/3093127230 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f385019bc70 con 0x7f384006e460 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: "mon", 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: "osd" 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "15/23 daemons upgraded", 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading mds daemons", 2024-03-26T19:02:38.210 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T19:02:38.211 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:02:38.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.217+0000 7f3856dfb700 1 -- 172.21.15.102:0/3093127230 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f384006e460 msgr2=0x7f3840070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:38.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.217+0000 7f3856dfb700 1 --2- 172.21.15.102:0/3093127230 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f384006e460 0x7f3840070910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f383c005950 tx=0x7f383c00b410 comp rx=0 tx=0).stop 2024-03-26T19:02:38.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.217+0000 7f3856dfb700 1 -- 172.21.15.102:0/3093127230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850100170 msgr2=0x7f3850195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:38.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.217+0000 7f3856dfb700 1 --2- 172.21.15.102:0/3093127230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850100170 0x7f3850195990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f384400bda0 tx=0x7f384400bdd0 comp rx=0 tx=0).stop 2024-03-26T19:02:38.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.217+0000 7f3856dfb700 1 -- 172.21.15.102:0/3093127230 shutdown_connections 2024-03-26T19:02:38.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.217+0000 7f3856dfb700 1 --2- 172.21.15.102:0/3093127230 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f384006e460 0x7f3840070910 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:38.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.217+0000 7f3856dfb700 1 --2- 172.21.15.102:0/3093127230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850100170 0x7f3850195990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:38.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.217+0000 7f3856dfb700 1 --2- 172.21.15.102:0/3093127230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3850101370 0x7f3850195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:38.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.217+0000 7f3856dfb700 1 -- 172.21.15.102:0/3093127230 >> 172.21.15.102:0/3093127230 conn(0x7f38500fb6e0 msgr2=0x7f38501045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:38.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.217+0000 7f3856dfb700 1 -- 172.21.15.102:0/3093127230 shutdown_connections 2024-03-26T19:02:38.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.217+0000 7f3856dfb700 1 -- 172.21.15.102:0/3093127230 wait complete. 2024-03-26T19:02:38.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.321+0000 7f9caf02a700 1 -- 172.21.15.102:0/727521422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8100be0 msgr2=0x7f9ca8102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:38.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.321+0000 7f9caf02a700 1 --2- 172.21.15.102:0/727521422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8100be0 0x7f9ca8102fc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009910 tx=0x7f9c9c009c20 comp rx=0 tx=0).stop 2024-03-26T19:02:38.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.321+0000 7f9caf02a700 1 -- 172.21.15.102:0/727521422 shutdown_connections 2024-03-26T19:02:38.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.321+0000 7f9caf02a700 1 --2- 172.21.15.102:0/727521422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca8103500 0x7f9ca81058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:38.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.321+0000 7f9caf02a700 1 --2- 172.21.15.102:0/727521422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8100be0 0x7f9ca8102fc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:38.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.321+0000 7f9caf02a700 1 -- 172.21.15.102:0/727521422 >> 172.21.15.102:0/727521422 conn(0x7f9ca80fa600 msgr2=0x7f9ca80fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:38.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.321+0000 7f9caf02a700 1 -- 172.21.15.102:0/727521422 shutdown_connections 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.321+0000 7f9caf02a700 1 -- 172.21.15.102:0/727521422 wait complete. 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9caf02a700 1 Processor -- start 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9caf02a700 1 -- start start 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9caf02a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8100be0 0x7f9ca81959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9caf02a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca8103500 0x7f9ca8195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9caf02a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca8196500 con 0x7f9ca8100be0 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9caf02a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca8196640 con 0x7f9ca8103500 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9cacdc6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8100be0 0x7f9ca81959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9cacdc6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8100be0 0x7f9ca81959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56808/0 (socket says 172.21.15.102:56808) 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9cacdc6700 1 -- 172.21.15.102:0/3594137813 learned_addr learned my addr 172.21.15.102:0/3594137813 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9ca7fff700 1 --2- 172.21.15.102:0/3594137813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca8103500 0x7f9ca8195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:38.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9cacdc6700 1 -- 172.21.15.102:0/3594137813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca8103500 msgr2=0x7f9ca8195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:38.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9cacdc6700 1 --2- 172.21.15.102:0/3594137813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca8103500 0x7f9ca8195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:38.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9cacdc6700 1 -- 172.21.15.102:0/3594137813 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c0095f0 con 0x7f9ca8100be0 2024-03-26T19:02:38.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.322+0000 7f9cacdc6700 1 --2- 172.21.15.102:0/3594137813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8100be0 0x7f9ca81959a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c000c00 tx=0x7f9c9c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:38.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.323+0000 7f9ca5ffb700 1 -- 172.21.15.102:0/3594137813 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c9c01d070 con 0x7f9ca8100be0 2024-03-26T19:02:38.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.323+0000 7f9ca5ffb700 1 -- 172.21.15.102:0/3594137813 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9c9c005dc0 con 0x7f9ca8100be0 2024-03-26T19:02:38.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.323+0000 7f9ca5ffb700 1 -- 172.21.15.102:0/3594137813 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c9c0226a0 con 0x7f9ca8100be0 2024-03-26T19:02:38.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.323+0000 7f9caf02a700 1 -- 172.21.15.102:0/3594137813 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca819b040 con 0x7f9ca8100be0 2024-03-26T19:02:38.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.323+0000 7f9caf02a700 1 -- 172.21.15.102:0/3594137813 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca819b530 con 0x7f9ca8100be0 2024-03-26T19:02:38.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.324+0000 7f9caf02a700 1 -- 172.21.15.102:0/3594137813 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca818fb90 con 0x7f9ca8100be0 2024-03-26T19:02:38.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.325+0000 7f9ca5ffb700 1 -- 172.21.15.102:0/3594137813 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9c9c0038f0 con 0x7f9ca8100be0 2024-03-26T19:02:38.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.325+0000 7f9ca5ffb700 1 --2- 172.21.15.102:0/3594137813 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c9806e460 0x7f9c98070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:02:38.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.325+0000 7f9ca5ffb700 1 -- 172.21.15.102:0/3594137813 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9c9c08f850 con 0x7f9ca8100be0 2024-03-26T19:02:38.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.325+0000 7f9ca7fff700 1 --2- 172.21.15.102:0/3594137813 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c9806e460 0x7f9c98070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:02:38.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.325+0000 7f9ca7fff700 1 --2- 172.21.15.102:0/3594137813 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c9806e460 0x7f9c98070910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009c60 tx=0x7f9c94009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:02:38.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.330+0000 7f9ca5ffb700 1 -- 172.21.15.102:0/3594137813 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9c9c05c5b0 con 0x7f9ca8100be0 2024-03-26T19:02:38.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.613+0000 7f9caf02a700 1 -- 172.21.15.102:0/3594137813 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9ca802cc30 con 0x7f9ca8100be0 2024-03-26T19:02:38.608 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:38 smithi102 ceph-mon[78322]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:38.608 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:38 smithi102 ceph-mon[78322]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:38.608 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:38 smithi102 ceph-mon[78322]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:38.608 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:38 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1876994495' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:38.609 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:38.609 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:38.609 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:38 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/4277699059' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:02:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.613+0000 7f9ca5ffb700 1 -- 172.21.15.102:0/3594137813 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f9c9c003680 con 0x7f9ca8100be0 2024-03-26T19:02:38.609 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T19:02:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.615+0000 7f9caf02a700 1 -- 172.21.15.102:0/3594137813 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c9806e460 msgr2=0x7f9c98070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.615+0000 7f9caf02a700 1 --2- 172.21.15.102:0/3594137813 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c9806e460 0x7f9c98070910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009c60 tx=0x7f9c94009260 comp rx=0 tx=0).stop 2024-03-26T19:02:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.615+0000 7f9caf02a700 1 -- 172.21.15.102:0/3594137813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8100be0 msgr2=0x7f9ca81959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:02:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.615+0000 7f9caf02a700 1 --2- 172.21.15.102:0/3594137813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8100be0 0x7f9ca81959a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c000c00 tx=0x7f9c9c00f690 comp rx=0 tx=0).stop 2024-03-26T19:02:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.615+0000 7f9caf02a700 1 -- 172.21.15.102:0/3594137813 shutdown_connections 2024-03-26T19:02:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.615+0000 7f9caf02a700 1 --2- 172.21.15.102:0/3594137813 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c9806e460 0x7f9c98070910 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.615+0000 7f9caf02a700 1 --2- 172.21.15.102:0/3594137813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8100be0 0x7f9ca81959a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.615+0000 7f9caf02a700 1 --2- 172.21.15.102:0/3594137813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca8103500 0x7f9ca8195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:02:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.615+0000 7f9caf02a700 1 -- 172.21.15.102:0/3594137813 >> 172.21.15.102:0/3594137813 conn(0x7f9ca80fa600 msgr2=0x7f9ca80fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:02:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.616+0000 7f9caf02a700 1 -- 172.21.15.102:0/3594137813 shutdown_connections 2024-03-26T19:02:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:02:38.616+0000 7f9caf02a700 1 -- 172.21.15.102:0/3594137813 wait complete. 2024-03-26T19:02:38.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:38 smithi190 ceph-mon[65689]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:38.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:38 smithi190 ceph-mon[65689]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:38.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:38 smithi190 ceph-mon[65689]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:38.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:38 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1876994495' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:38.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:38.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:38.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:38 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/4277699059' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:02:39.594 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:39 smithi190 ceph-mon[65689]: pgmap v216: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:39.594 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:39 smithi190 ceph-mon[65689]: from='client.34492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:39.594 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:39 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3594137813' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:02:39.594 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:39 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:39.594 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:39 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:39.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:39 smithi102 ceph-mon[78322]: pgmap v216: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:39.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:39 smithi102 ceph-mon[78322]: from='client.34492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:02:39.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:39 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3594137813' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:02:39.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:39 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:39.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:39 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:41.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:41 smithi190 ceph-mon[65689]: pgmap v217: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:41.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:41 smithi102 ceph-mon[78322]: pgmap v217: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:43.460 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: pgmap v218: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:43.460 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:43.460 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:43.460 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:43.460 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:02:43.460 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:43.460 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:43.460 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:43.460 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:43.461 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:43.461 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:43.461 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:43 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi190.hwsvdg"]}]: dispatch 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: pgmap v218: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:02:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:43 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi190.hwsvdg"]}]: dispatch 2024-03-26T19:02:44.466 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:44 smithi190 ceph-mon[65689]: Detected new or changed devices on smithi190 2024-03-26T19:02:44.467 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:44 smithi190 ceph-mon[65689]: Upgrade: It appears safe to stop mds.cephfs.smithi190.hwsvdg 2024-03-26T19:02:44.467 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:44 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:44.467 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:44 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.hwsvdg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T19:02:44.467 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:44 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:44.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:44 smithi102 ceph-mon[78322]: Detected new or changed devices on smithi190 2024-03-26T19:02:44.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:44 smithi102 ceph-mon[78322]: Upgrade: It appears safe to stop mds.cephfs.smithi190.hwsvdg 2024-03-26T19:02:44.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:44 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:44.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:44 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.hwsvdg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T19:02:44.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:44 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:02:45.485 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:45 smithi190 ceph-mon[65689]: pgmap v219: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:45.485 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:45 smithi190 ceph-mon[65689]: Upgrade: Updating mds.cephfs.smithi190.hwsvdg 2024-03-26T19:02:45.485 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:45 smithi190 ceph-mon[65689]: Deploying daemon mds.cephfs.smithi190.hwsvdg on smithi190 2024-03-26T19:02:45.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:45 smithi102 ceph-mon[78322]: pgmap v219: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:45.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:45 smithi102 ceph-mon[78322]: Upgrade: Updating mds.cephfs.smithi190.hwsvdg 2024-03-26T19:02:45.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:45 smithi102 ceph-mon[78322]: Deploying daemon mds.cephfs.smithi190.hwsvdg on smithi190 2024-03-26T19:02:46.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:46 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102[78318]: 2024-03-26T19:02:46.321+0000 7f9567616700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T19:02:46.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:46 smithi102 ceph-mon[78322]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T19:02:46.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:46 smithi102 ceph-mon[78322]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T19:02:46.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:46 smithi102 ceph-mon[78322]: osdmap e84: 6 total, 6 up, 6 in 2024-03-26T19:02:46.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:46 smithi102 ceph-mon[78322]: Standby daemon mds.cephfs.smithi102.tvsqaa assigned to filesystem cephfs as rank 0 2024-03-26T19:02:46.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:46 smithi102 ceph-mon[78322]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T19:02:46.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:46 smithi102 ceph-mon[78322]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-03-26T19:02:46.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:46 smithi102 ceph-mon[78322]: fsmap cephfs:1/1 {0=cephfs.smithi102.tvsqaa=up:replay} 2 up:standby 2024-03-26T19:02:46.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:46 smithi190 ceph-mon[65689]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T19:02:46.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:46 smithi190 ceph-mon[65689]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T19:02:46.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:46 smithi190 ceph-mon[65689]: osdmap e84: 6 total, 6 up, 6 in 2024-03-26T19:02:46.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:46 smithi190 ceph-mon[65689]: Standby daemon mds.cephfs.smithi102.tvsqaa assigned to filesystem cephfs as rank 0 2024-03-26T19:02:46.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:46 smithi190 ceph-mon[65689]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T19:02:46.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:46 smithi190 ceph-mon[65689]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-03-26T19:02:46.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:46 smithi190 ceph-mon[65689]: fsmap cephfs:1/1 {0=cephfs.smithi102.tvsqaa=up:replay} 2 up:standby 2024-03-26T19:02:47.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:47 smithi102 ceph-mon[78322]: pgmap v220: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:47.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:47 smithi190 ceph-mon[65689]: pgmap v220: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:02:49.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:49 smithi102 ceph-mon[78322]: pgmap v222: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 409 KiB/s rd, 0 op/s 2024-03-26T19:02:49.648 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:49 smithi190 ceph-mon[65689]: pgmap v222: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 409 KiB/s rd, 0 op/s 2024-03-26T19:02:51.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:51 smithi102 ceph-mon[78322]: pgmap v223: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 4.4 MiB/s rd, 1 op/s 2024-03-26T19:02:51.740 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:51 smithi190 ceph-mon[65689]: pgmap v223: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 4.4 MiB/s rd, 1 op/s 2024-03-26T19:02:53.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:53 smithi190 ceph-mon[65689]: pgmap v224: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 4 op/s 2024-03-26T19:02:53.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:53.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:02:53.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:53.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:53.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:53.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:53 smithi102 ceph-mon[78322]: pgmap v224: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 4 op/s 2024-03-26T19:02:53.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:53.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:02:53.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:53.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:53.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:02:54.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:54 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.190:6826/925215849,v1:172.21.15.190:6827/925215849] up:boot 2024-03-26T19:02:54.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:54 smithi190 ceph-mon[65689]: fsmap cephfs:1/1 {0=cephfs.smithi102.tvsqaa=up:replay} 3 up:standby 2024-03-26T19:02:54.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:54 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T19:02:54.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:54 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.190:6826/925215849,v1:172.21.15.190:6827/925215849] up:boot 2024-03-26T19:02:54.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:54 smithi102 ceph-mon[78322]: fsmap cephfs:1/1 {0=cephfs.smithi102.tvsqaa=up:replay} 3 up:standby 2024-03-26T19:02:54.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:54 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T19:02:55.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:55 smithi102 ceph-mon[78322]: pgmap v225: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4 op/s 2024-03-26T19:02:55.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:55 smithi102 ceph-mon[78322]: reconnect by client.14540 v1:192.168.0.1:0/2355540585 after 0 2024-03-26T19:02:55.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:55 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] up:reconnect 2024-03-26T19:02:55.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:55 smithi102 ceph-mon[78322]: fsmap cephfs:1/1 {0=cephfs.smithi102.tvsqaa=up:reconnect} 3 up:standby 2024-03-26T19:02:55.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:55 smithi102 ceph-mon[78322]: reconnect by client.14546 v1:172.21.15.190:0/1799840371 after 0.00899982 2024-03-26T19:02:55.639 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:55 smithi190 ceph-mon[65689]: pgmap v225: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4 op/s 2024-03-26T19:02:55.639 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:55 smithi190 ceph-mon[65689]: reconnect by client.14540 v1:192.168.0.1:0/2355540585 after 0 2024-03-26T19:02:55.639 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:55 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] up:reconnect 2024-03-26T19:02:55.639 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:55 smithi190 ceph-mon[65689]: fsmap cephfs:1/1 {0=cephfs.smithi102.tvsqaa=up:reconnect} 3 up:standby 2024-03-26T19:02:55.639 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:55 smithi190 ceph-mon[65689]: reconnect by client.14546 v1:172.21.15.190:0/1799840371 after 0.00899982 2024-03-26T19:02:56.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:56 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] up:rejoin 2024-03-26T19:02:56.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:56 smithi102 ceph-mon[78322]: fsmap cephfs:1/1 {0=cephfs.smithi102.tvsqaa=up:rejoin} 3 up:standby 2024-03-26T19:02:56.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:56 smithi102 ceph-mon[78322]: daemon mds.cephfs.smithi102.tvsqaa is now active in filesystem cephfs as rank 0 2024-03-26T19:02:56.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:56 smithi102 ceph-mon[78322]: pgmap v226: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 5 op/s 2024-03-26T19:02:56.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:56 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:56.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:56 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:56.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:56 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] up:rejoin 2024-03-26T19:02:56.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:56 smithi190 ceph-mon[65689]: fsmap cephfs:1/1 {0=cephfs.smithi102.tvsqaa=up:rejoin} 3 up:standby 2024-03-26T19:02:56.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:56 smithi190 ceph-mon[65689]: daemon mds.cephfs.smithi102.tvsqaa is now active in filesystem cephfs as rank 0 2024-03-26T19:02:56.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:56 smithi190 ceph-mon[65689]: pgmap v226: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 5 op/s 2024-03-26T19:02:56.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:56 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:56.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:56 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:57.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:57 smithi190 ceph-mon[65689]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T19:02:57.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:57 smithi190 ceph-mon[65689]: Cluster is now healthy 2024-03-26T19:02:57.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:57 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] up:active 2024-03-26T19:02:57.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:57 smithi190 ceph-mon[65689]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:active} 3 up:standby 2024-03-26T19:02:57.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:57.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:57 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:57.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:57 smithi102 ceph-mon[78322]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T19:02:57.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:57 smithi102 ceph-mon[78322]: Cluster is now healthy 2024-03-26T19:02:57.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:57 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] up:active 2024-03-26T19:02:57.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:57 smithi102 ceph-mon[78322]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:active} 3 up:standby 2024-03-26T19:02:57.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:57.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:57 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:02:58.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:02:58 smithi190 ceph-mon[65689]: pgmap v227: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 5 op/s 2024-03-26T19:02:58.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:02:58 smithi102 ceph-mon[78322]: pgmap v227: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 5 op/s 2024-03-26T19:03:01.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: pgmap v228: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-03-26T19:03:01.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:01.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:01.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:01.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:03:01.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:01.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:01.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:01.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:01.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:01.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:01.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:00 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:01.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: pgmap v228: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-03-26T19:03:01.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:01.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:01.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:01.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:03:01.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:01.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:01.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:01.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:01.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:01.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:01.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:00 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:02.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: Upgrade: Setting container_image for all mds 2024-03-26T19:03:02.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:02.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T19:03:02.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi102.hrbrsp"}]': finished 2024-03-26T19:03:02.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T19:03:02.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi102.tvsqaa"}]': finished 2024-03-26T19:03:02.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T19:03:02.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi190.hwsvdg"}]': finished 2024-03-26T19:03:02.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T19:03:02.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi190.vykizb"}]': finished 2024-03-26T19:03:02.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: Upgrade: Scaling up filesystem cephfs max_mds to 2 2024-03-26T19:03:02.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:01 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-26T19:03:02.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: Upgrade: Setting container_image for all mds 2024-03-26T19:03:02.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:02.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi102.hrbrsp"}]: dispatch 2024-03-26T19:03:02.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi102.hrbrsp"}]': finished 2024-03-26T19:03:02.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi102.tvsqaa"}]: dispatch 2024-03-26T19:03:02.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi102.tvsqaa"}]': finished 2024-03-26T19:03:02.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi190.hwsvdg"}]: dispatch 2024-03-26T19:03:02.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi190.hwsvdg"}]': finished 2024-03-26T19:03:02.308 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi190.vykizb"}]: dispatch 2024-03-26T19:03:02.309 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi190.vykizb"}]': finished 2024-03-26T19:03:02.309 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: Upgrade: Scaling up filesystem cephfs max_mds to 2 2024-03-26T19:03:02.309 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:01 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-26T19:03:03.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: pgmap v229: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-26T19:03:03.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-26T19:03:03.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: daemon mds.cephfs.smithi102.hrbrsp assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: Cluster is now healthy 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:active} 3 up:standby 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi102.hrbrsp=up:starting} 2 up:standby 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: daemon mds.cephfs.smithi102.hrbrsp is now active in filesystem cephfs as rank 1 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: Upgrade: Setting container_image for all rgw 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T19:03:03.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:03.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:02 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:03.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: pgmap v229: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-26T19:03:03.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: daemon mds.cephfs.smithi102.hrbrsp assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: Cluster is now healthy 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: fsmap cephfs:1 {0=cephfs.smithi102.tvsqaa=up:active} 3 up:standby 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi102.hrbrsp=up:starting} 2 up:standby 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: daemon mds.cephfs.smithi102.hrbrsp is now active in filesystem cephfs as rank 1 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: Upgrade: Setting container_image for all rgw 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:02 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:04.230 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:03 smithi102 ceph-mon[78322]: mds.? [v2:172.21.15.102:6828/3007238267,v1:172.21.15.102:6829/3007238267] up:active 2024-03-26T19:03:04.230 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:03 smithi102 ceph-mon[78322]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi102.hrbrsp=up:active} 2 up:standby 2024-03-26T19:03:04.230 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:04.230 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T19:03:04.230 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:03 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:04.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:03 smithi190 ceph-mon[65689]: mds.? [v2:172.21.15.102:6828/3007238267,v1:172.21.15.102:6829/3007238267] up:active 2024-03-26T19:03:04.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:03 smithi190 ceph-mon[65689]: fsmap cephfs:2 {0=cephfs.smithi102.tvsqaa=up:active,1=cephfs.smithi102.hrbrsp=up:active} 2 up:standby 2024-03-26T19:03:04.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:04.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi102", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T19:03:04.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:03 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:05.231 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:04 smithi102 ceph-mon[78322]: pgmap v230: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 3.8 MiB/s rd, 4.2 KiB/s wr, 6 op/s 2024-03-26T19:03:05.231 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:04 smithi102 ceph-mon[78322]: Upgrade: Updating ceph-exporter.smithi102 (1/2) 2024-03-26T19:03:05.231 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:04 smithi102 ceph-mon[78322]: Deploying daemon ceph-exporter.smithi102 on smithi102 2024-03-26T19:03:05.242 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:04 smithi190 ceph-mon[65689]: pgmap v230: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 3.8 MiB/s rd, 4.2 KiB/s wr, 6 op/s 2024-03-26T19:03:05.243 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:04 smithi190 ceph-mon[65689]: Upgrade: Updating ceph-exporter.smithi102 (1/2) 2024-03-26T19:03:05.243 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:04 smithi190 ceph-mon[65689]: Deploying daemon ceph-exporter.smithi102 on smithi102 2024-03-26T19:03:07.125 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:06 smithi102 ceph-mon[78322]: pgmap v231: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 2.8 MiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-26T19:03:07.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:06 smithi190 ceph-mon[65689]: pgmap v231: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 2.8 MiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-26T19:03:08.415 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:08.415 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:03:08.415 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:08.415 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:08.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:08.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:03:08.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:08.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.727+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/256060700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b4100250 msgr2=0x7fe5b41006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.727+0000 7fe5ba8d4700 1 --2- 172.21.15.102:0/256060700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b4100250 0x7fe5b41006a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8009910 tx=0x7fe5a8009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.727+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/256060700 shutdown_connections 2024-03-26T19:03:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.727+0000 7fe5ba8d4700 1 --2- 172.21.15.102:0/256060700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b4100250 0x7fe5b41006a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.727+0000 7fe5ba8d4700 1 --2- 172.21.15.102:0/256060700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5b40ff050 0x7fe5b40ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.727+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/256060700 >> 172.21.15.102:0/256060700 conn(0x7fe5b40fa5e0 msgr2=0x7fe5b40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.727+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/256060700 shutdown_connections 2024-03-26T19:03:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.727+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/256060700 wait complete. 2024-03-26T19:03:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.728+0000 7fe5ba8d4700 1 Processor -- start 2024-03-26T19:03:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.728+0000 7fe5ba8d4700 1 -- start start 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.728+0000 7fe5ba8d4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5b40ff050 0x7fe5b4195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.728+0000 7fe5ba8d4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b4100250 0x7fe5b4195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.728+0000 7fe5ba8d4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b41964f0 con 0x7fe5b4100250 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.728+0000 7fe5ba8d4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b4196630 con 0x7fe5b40ff050 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.728+0000 7fe5b37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b4100250 0x7fe5b4195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.728+0000 7fe5b37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b4100250 0x7fe5b4195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41706/0 (socket says 172.21.15.102:41706) 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.728+0000 7fe5b37fe700 1 -- 172.21.15.102:0/410929195 learned_addr learned my addr 172.21.15.102:0/410929195 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.728+0000 7fe5b3fff700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5b40ff050 0x7fe5b4195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.729+0000 7fe5b37fe700 1 -- 172.21.15.102:0/410929195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5b40ff050 msgr2=0x7fe5b4195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.729+0000 7fe5b37fe700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5b40ff050 0x7fe5b4195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.729+0000 7fe5b37fe700 1 -- 172.21.15.102:0/410929195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a80095f0 con 0x7fe5b4100250 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.729+0000 7fe5b3fff700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5b40ff050 0x7fe5b4195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T19:03:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.729+0000 7fe5b37fe700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b4100250 0x7fe5b4195ed0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8006010 tx=0x7fe5a800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:08.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.729+0000 7fe5b17fa700 1 -- 172.21.15.102:0/410929195 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5a801d070 con 0x7fe5b4100250 2024-03-26T19:03:08.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.729+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/410929195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b419b080 con 0x7fe5b4100250 2024-03-26T19:03:08.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.729+0000 7fe5b17fa700 1 -- 172.21.15.102:0/410929195 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe5a800f460 con 0x7fe5b4100250 2024-03-26T19:03:08.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.729+0000 7fe5b17fa700 1 -- 172.21.15.102:0/410929195 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5a8017630 con 0x7fe5b4100250 2024-03-26T19:03:08.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.729+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/410929195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b419b570 con 0x7fe5b4100250 2024-03-26T19:03:08.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.730+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/410929195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b4107ba0 con 0x7fe5b4100250 2024-03-26T19:03:08.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.731+0000 7fe5b17fa700 1 -- 172.21.15.102:0/410929195 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe5a8003a60 con 0x7fe5b4100250 2024-03-26T19:03:08.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.732+0000 7fe5b17fa700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe59c06e460 0x7fe59c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:08.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.732+0000 7fe5b17fa700 1 -- 172.21.15.102:0/410929195 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fe5a808ed20 con 0x7fe5b4100250 2024-03-26T19:03:08.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.732+0000 7fe5b3fff700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe59c06e460 0x7fe59c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:08.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.732+0000 7fe5b3fff700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe59c06e460 0x7fe59c070910 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe5b41000b0 tx=0x7fe5a400b670 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:08.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.737+0000 7fe5b17fa700 1 -- 172.21.15.102:0/410929195 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe5a805b920 con 0x7fe5b4100250 2024-03-26T19:03:08.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.994+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/410929195 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5b4068ae0 con 0x7fe59c06e460 2024-03-26T19:03:08.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.995+0000 7fe5b17fa700 1 -- 172.21.15.102:0/410929195 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7fe5b4068ae0 con 0x7fe59c06e460 2024-03-26T19:03:08.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.997+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/410929195 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe59c06e460 msgr2=0x7fe59c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:08.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.997+0000 7fe5ba8d4700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe59c06e460 0x7fe59c070910 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe5b41000b0 tx=0x7fe5a400b670 comp rx=0 tx=0).stop 2024-03-26T19:03:08.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.997+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/410929195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b4100250 msgr2=0x7fe5b4195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:08.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.997+0000 7fe5ba8d4700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b4100250 0x7fe5b4195ed0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8006010 tx=0x7fe5a800be20 comp rx=0 tx=0).stop 2024-03-26T19:03:08.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.998+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/410929195 shutdown_connections 2024-03-26T19:03:08.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.998+0000 7fe5ba8d4700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5b40ff050 0x7fe5b4195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:08.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.998+0000 7fe5ba8d4700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe59c06e460 0x7fe59c070910 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:08.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.998+0000 7fe5ba8d4700 1 --2- 172.21.15.102:0/410929195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b4100250 0x7fe5b4195ed0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:08.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.998+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/410929195 >> 172.21.15.102:0/410929195 conn(0x7fe5b40fa5e0 msgr2=0x7fe5b40683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:08.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.998+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/410929195 shutdown_connections 2024-03-26T19:03:08.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:08.998+0000 7fe5ba8d4700 1 -- 172.21.15.102:0/410929195 wait complete. 2024-03-26T19:03:09.004 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T19:03:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.115+0000 7f8991908700 1 -- 172.21.15.102:0/114958861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f898c100ca0 msgr2=0x7f898c103080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.115+0000 7f8991908700 1 --2- 172.21.15.102:0/114958861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f898c100ca0 0x7f898c103080 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f897c009910 tx=0x7f897c009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.116+0000 7f8991908700 1 -- 172.21.15.102:0/114958861 shutdown_connections 2024-03-26T19:03:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.116+0000 7f8991908700 1 --2- 172.21.15.102:0/114958861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f898c1035c0 0x7f898c1059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.116+0000 7f8991908700 1 --2- 172.21.15.102:0/114958861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f898c100ca0 0x7f898c103080 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.116+0000 7f8991908700 1 -- 172.21.15.102:0/114958861 >> 172.21.15.102:0/114958861 conn(0x7f898c0fa5e0 msgr2=0x7f898c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.116+0000 7f8991908700 1 -- 172.21.15.102:0/114958861 shutdown_connections 2024-03-26T19:03:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.116+0000 7f8991908700 1 -- 172.21.15.102:0/114958861 wait complete. 2024-03-26T19:03:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.116+0000 7f8991908700 1 Processor -- start 2024-03-26T19:03:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f8991908700 1 -- start start 2024-03-26T19:03:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f8991908700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f898c100ca0 0x7f898c100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f8991908700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f898c1035c0 0x7f898c0fec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f8991908700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c0ff190 con 0x7f898c100ca0 2024-03-26T19:03:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f8991908700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c0ff2d0 con 0x7f898c1035c0 2024-03-26T19:03:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f898affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f898c100ca0 0x7f898c100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f898affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f898c100ca0 0x7f898c100600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41726/0 (socket says 172.21.15.102:41726) 2024-03-26T19:03:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f898affd700 1 -- 172.21.15.102:0/2968273056 learned_addr learned my addr 172.21.15.102:0/2968273056 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f898a7fc700 1 --2- 172.21.15.102:0/2968273056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f898c1035c0 0x7f898c0fec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f898affd700 1 -- 172.21.15.102:0/2968273056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f898c1035c0 msgr2=0x7f898c0fec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f898affd700 1 --2- 172.21.15.102:0/2968273056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f898c1035c0 0x7f898c0fec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f898affd700 1 -- 172.21.15.102:0/2968273056 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c0095f0 con 0x7f898c100ca0 2024-03-26T19:03:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.117+0000 7f898affd700 1 --2- 172.21.15.102:0/2968273056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f898c100ca0 0x7f898c100600 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f897c006010 tx=0x7f897c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.118+0000 7f8990906700 1 -- 172.21.15.102:0/2968273056 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f897c01d070 con 0x7f898c100ca0 2024-03-26T19:03:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.118+0000 7f8990906700 1 -- 172.21.15.102:0/2968273056 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f897c00f460 con 0x7f898c100ca0 2024-03-26T19:03:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.118+0000 7f8991908700 1 -- 172.21.15.102:0/2968273056 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898c0ff550 con 0x7f898c100ca0 2024-03-26T19:03:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.118+0000 7f8991908700 1 -- 172.21.15.102:0/2968273056 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898c0ffa40 con 0x7f898c100ca0 2024-03-26T19:03:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.118+0000 7f8990906700 1 -- 172.21.15.102:0/2968273056 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f897c021640 con 0x7f898c100ca0 2024-03-26T19:03:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.119+0000 7f8991908700 1 -- 172.21.15.102:0/2968273056 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8978005320 con 0x7f898c100ca0 2024-03-26T19:03:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.119+0000 7f8990906700 1 -- 172.21.15.102:0/2968273056 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f897c003a60 con 0x7f898c100ca0 2024-03-26T19:03:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.120+0000 7f8990906700 1 --2- 172.21.15.102:0/2968273056 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f897406e460 0x7f8974070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.120+0000 7f8990906700 1 -- 172.21.15.102:0/2968273056 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f897c08f600 con 0x7f898c100ca0 2024-03-26T19:03:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.120+0000 7f898a7fc700 1 --2- 172.21.15.102:0/2968273056 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f897406e460 0x7f8974070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.120+0000 7f898a7fc700 1 --2- 172.21.15.102:0/2968273056 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f897406e460 0x7f8974070910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8980005950 tx=0x7f898000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.125+0000 7f8990906700 1 -- 172.21.15.102:0/2968273056 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f897c05c310 con 0x7f898c100ca0 2024-03-26T19:03:09.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.421+0000 7f8991908700 1 -- 172.21.15.102:0/2968273056 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8978000bf0 con 0x7f897406e460 2024-03-26T19:03:09.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.423+0000 7f8990906700 1 -- 172.21.15.102:0/2968273056 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f8978000bf0 con 0x7f897406e460 2024-03-26T19:03:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.426+0000 7f8991908700 1 -- 172.21.15.102:0/2968273056 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f897406e460 msgr2=0x7f8974070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.426+0000 7f8991908700 1 --2- 172.21.15.102:0/2968273056 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f897406e460 0x7f8974070910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8980005950 tx=0x7f898000b410 comp rx=0 tx=0).stop 2024-03-26T19:03:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.426+0000 7f8991908700 1 -- 172.21.15.102:0/2968273056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f898c100ca0 msgr2=0x7f898c100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.426+0000 7f8991908700 1 --2- 172.21.15.102:0/2968273056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f898c100ca0 0x7f898c100600 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f897c006010 tx=0x7f897c00bd70 comp rx=0 tx=0).stop 2024-03-26T19:03:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.426+0000 7f8991908700 1 -- 172.21.15.102:0/2968273056 shutdown_connections 2024-03-26T19:03:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.426+0000 7f8991908700 1 --2- 172.21.15.102:0/2968273056 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f897406e460 0x7f8974070910 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.426+0000 7f8991908700 1 --2- 172.21.15.102:0/2968273056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f898c100ca0 0x7f898c100600 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.426+0000 7f8991908700 1 --2- 172.21.15.102:0/2968273056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f898c1035c0 0x7f898c0fec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.426+0000 7f8991908700 1 -- 172.21.15.102:0/2968273056 >> 172.21.15.102:0/2968273056 conn(0x7f898c0fa5e0 msgr2=0x7f898c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.426+0000 7f8991908700 1 -- 172.21.15.102:0/2968273056 shutdown_connections 2024-03-26T19:03:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.426+0000 7f8991908700 1 -- 172.21.15.102:0/2968273056 wait complete. 2024-03-26T19:03:09.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:09 smithi190 ceph-mon[65689]: pgmap v232: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 9.6 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-03-26T19:03:09.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:09 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:09.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:09 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T19:03:09.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:09 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:09.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.550+0000 7f7600ccd700 1 -- 172.21.15.102:0/638412524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc101360 msgr2=0x7f75fc1017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:09.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.550+0000 7f7600ccd700 1 --2- 172.21.15.102:0/638412524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc101360 0x7f75fc1017b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009910 tx=0x7f75f0009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:09.544 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:09 smithi102 ceph-mon[78322]: pgmap v232: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 9.6 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-03-26T19:03:09.545 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:09 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:09.545 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:09 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T19:03:09.545 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:09 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.550+0000 7f7600ccd700 1 -- 172.21.15.102:0/638412524 shutdown_connections 2024-03-26T19:03:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.550+0000 7f7600ccd700 1 --2- 172.21.15.102:0/638412524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc101360 0x7f75fc1017b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.550+0000 7f7600ccd700 1 --2- 172.21.15.102:0/638412524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75fc100160 0x7f75fc100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.550+0000 7f7600ccd700 1 -- 172.21.15.102:0/638412524 >> 172.21.15.102:0/638412524 conn(0x7f75fc0fb6f0 msgr2=0x7f75fc0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.550+0000 7f7600ccd700 1 -- 172.21.15.102:0/638412524 shutdown_connections 2024-03-26T19:03:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.550+0000 7f7600ccd700 1 -- 172.21.15.102:0/638412524 wait complete. 2024-03-26T19:03:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.551+0000 7f7600ccd700 1 Processor -- start 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.551+0000 7f7600ccd700 1 -- start start 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.551+0000 7f7600ccd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc100160 0x7f75fc197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.551+0000 7f7600ccd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75fc101360 0x7f75fc198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.551+0000 7f7600ccd700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc1986a0 con 0x7f75fc100160 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.551+0000 7f7600ccd700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc1987e0 con 0x7f75fc101360 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.551+0000 7f75fa59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc100160 0x7f75fc197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.551+0000 7f75fa59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc100160 0x7f75fc197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41740/0 (socket says 172.21.15.102:41740) 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.551+0000 7f75fa59c700 1 -- 172.21.15.102:0/1609112984 learned_addr learned my addr 172.21.15.102:0/1609112984 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.551+0000 7f75f9d9b700 1 --2- 172.21.15.102:0/1609112984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75fc101360 0x7f75fc198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.552+0000 7f75fa59c700 1 -- 172.21.15.102:0/1609112984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75fc101360 msgr2=0x7f75fc198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.552+0000 7f75fa59c700 1 --2- 172.21.15.102:0/1609112984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75fc101360 0x7f75fc198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.552+0000 7f75fa59c700 1 -- 172.21.15.102:0/1609112984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f00095f0 con 0x7f75fc100160 2024-03-26T19:03:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.552+0000 7f75fa59c700 1 --2- 172.21.15.102:0/1609112984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc100160 0x7f75fc197b40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00d900 tx=0x7f75ec00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:09.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.552+0000 7f75eb7fe700 1 -- 172.21.15.102:0/1609112984 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75ec0041f0 con 0x7f75fc100160 2024-03-26T19:03:09.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.552+0000 7f75eb7fe700 1 -- 172.21.15.102:0/1609112984 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f75ec004350 con 0x7f75fc100160 2024-03-26T19:03:09.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.552+0000 7f75eb7fe700 1 -- 172.21.15.102:0/1609112984 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75ec003d20 con 0x7f75fc100160 2024-03-26T19:03:09.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.552+0000 7f7600ccd700 1 -- 172.21.15.102:0/1609112984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75fc19d290 con 0x7f75fc100160 2024-03-26T19:03:09.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.552+0000 7f7600ccd700 1 -- 172.21.15.102:0/1609112984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75fc19d7e0 con 0x7f75fc100160 2024-03-26T19:03:09.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.553+0000 7f7600ccd700 1 -- 172.21.15.102:0/1609112984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75fc105440 con 0x7f75fc100160 2024-03-26T19:03:09.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.554+0000 7f75eb7fe700 1 -- 172.21.15.102:0/1609112984 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f75ec009730 con 0x7f75fc100160 2024-03-26T19:03:09.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.555+0000 7f75eb7fe700 1 --2- 172.21.15.102:0/1609112984 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75e406e460 0x7f75e4070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:09.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.555+0000 7f75eb7fe700 1 -- 172.21.15.102:0/1609112984 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f75ec08e260 con 0x7f75fc100160 2024-03-26T19:03:09.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.555+0000 7f75f9d9b700 1 --2- 172.21.15.102:0/1609112984 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75e406e460 0x7f75e4070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:09.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.555+0000 7f75f9d9b700 1 --2- 172.21.15.102:0/1609112984 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75e406e460 0x7f75e4070910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f75f0006010 tx=0x7f75f000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:09.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.559+0000 7f75eb7fe700 1 -- 172.21.15.102:0/1609112984 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f75ec05af70 con 0x7f75fc100160 2024-03-26T19:03:09.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.812+0000 7f7600ccd700 1 -- 172.21.15.102:0/1609112984 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f75fc061190 con 0x7f75e406e460 2024-03-26T19:03:09.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.823+0000 7f75eb7fe700 1 -- 172.21.15.102:0/1609112984 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f75fc061190 con 0x7f75e406e460 2024-03-26T19:03:09.818 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T19:03:09.818 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (14m) 48s ago 16m 25.7M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T19:03:09.818 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 starting - - - - 2024-03-26T19:03:09.818 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (15m) 13s ago 15m 8845k - 18.2.0 dc2bc1663786 b1e5e9cccc1b 2024-03-26T19:03:09.818 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (5m) 48s ago 16m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T19:03:09.818 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (5m) 13s ago 15m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T19:03:09.818 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (14m) 48s ago 15m 85.8M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T19:03:09.818 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (54s) 48s ago 10m 11.4M - 19.0.0-2311-g36051606 497bb87bc5da c530ed7dbd78 2024-03-26T19:03:09.818 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (77s) 48s ago 10m 13.5M - 19.0.0-2311-g36051606 497bb87bc5da 63b8127a3849 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (17s) 13s ago 10m 12.5M - 19.0.0-2311-g36051606 497bb87bc5da 25df8033593e 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (35s) 13s ago 10m 14.9M - 19.0.0-2311-g36051606 497bb87bc5da 37260789b9ec 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (7m) 48s ago 17m 508M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (6m) 13s ago 15m 452M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (6m) 48s ago 17m 50.2M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (6m) 13s ago 15m 40.3M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (16m) 48s ago 16m 16.6M - 1.5.0 0da6a335fe13 05d8d7dba4a9 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (15m) 13s ago 15m 15.0M - 1.5.0 0da6a335fe13 d066107b901e 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (5m) 48s ago 14m 105M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (4m) 48s ago 13m 114M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (3m) 48s ago 13m 97.0M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (3m) 13s ago 13m 122M 2240M 19.0.0-2311-g36051606 497bb87bc5da 03955331c962 2024-03-26T19:03:09.819 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (2m) 13s ago 12m 113M 2240M 19.0.0-2311-g36051606 497bb87bc5da 23e140cd8961 2024-03-26T19:03:09.820 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (2m) 13s ago 12m 104M 2240M 19.0.0-2311-g36051606 497bb87bc5da 60afb0e4da18 2024-03-26T19:03:09.820 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (7m) 48s ago 15m 78.5M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T19:03:09.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.826+0000 7f7600ccd700 1 -- 172.21.15.102:0/1609112984 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75e406e460 msgr2=0x7f75e4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:09.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.826+0000 7f7600ccd700 1 --2- 172.21.15.102:0/1609112984 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75e406e460 0x7f75e4070910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f75f0006010 tx=0x7f75f000b540 comp rx=0 tx=0).stop 2024-03-26T19:03:09.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.826+0000 7f7600ccd700 1 -- 172.21.15.102:0/1609112984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc100160 msgr2=0x7f75fc197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:09.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.826+0000 7f7600ccd700 1 --2- 172.21.15.102:0/1609112984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc100160 0x7f75fc197b40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00d900 tx=0x7f75ec00dc10 comp rx=0 tx=0).stop 2024-03-26T19:03:09.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.826+0000 7f7600ccd700 1 -- 172.21.15.102:0/1609112984 shutdown_connections 2024-03-26T19:03:09.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.826+0000 7f7600ccd700 1 --2- 172.21.15.102:0/1609112984 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f75e406e460 0x7f75e4070910 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.826+0000 7f7600ccd700 1 --2- 172.21.15.102:0/1609112984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc100160 0x7f75fc197b40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.826+0000 7f7600ccd700 1 --2- 172.21.15.102:0/1609112984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75fc101360 0x7f75fc198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.826+0000 7f7600ccd700 1 -- 172.21.15.102:0/1609112984 >> 172.21.15.102:0/1609112984 conn(0x7f75fc0fb6f0 msgr2=0x7f75fc102580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:09.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.826+0000 7f7600ccd700 1 -- 172.21.15.102:0/1609112984 shutdown_connections 2024-03-26T19:03:09.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.826+0000 7f7600ccd700 1 -- 172.21.15.102:0/1609112984 wait complete. 2024-03-26T19:03:09.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.932+0000 7f4424d55700 1 -- 172.21.15.102:0/415517785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4420100c60 msgr2=0x7f4420103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:09.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.932+0000 7f4424d55700 1 --2- 172.21.15.102:0/415517785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4420100c60 0x7f4420103040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4410009910 tx=0x7f4410009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:09.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.932+0000 7f4424d55700 1 -- 172.21.15.102:0/415517785 shutdown_connections 2024-03-26T19:03:09.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.932+0000 7f4424d55700 1 --2- 172.21.15.102:0/415517785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4420103580 0x7f4420105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.932+0000 7f4424d55700 1 --2- 172.21.15.102:0/415517785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4420100c60 0x7f4420103040 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.932+0000 7f4424d55700 1 -- 172.21.15.102:0/415517785 >> 172.21.15.102:0/415517785 conn(0x7f44200fa5e0 msgr2=0x7f44200fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.932+0000 7f4424d55700 1 -- 172.21.15.102:0/415517785 shutdown_connections 2024-03-26T19:03:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.932+0000 7f4424d55700 1 -- 172.21.15.102:0/415517785 wait complete. 2024-03-26T19:03:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f4424d55700 1 Processor -- start 2024-03-26T19:03:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f4424d55700 1 -- start start 2024-03-26T19:03:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f4424d55700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4420100c60 0x7f44201005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f4424d55700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4420103580 0x7f44200fec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f4424d55700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44200ff150 con 0x7f4420100c60 2024-03-26T19:03:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f4424d55700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44200ff290 con 0x7f4420103580 2024-03-26T19:03:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f441e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4420100c60 0x7f44201005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f441e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4420100c60 0x7f44201005c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41772/0 (socket says 172.21.15.102:41772) 2024-03-26T19:03:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f441e59c700 1 -- 172.21.15.102:0/176955473 learned_addr learned my addr 172.21.15.102:0/176955473 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f441dd9b700 1 --2- 172.21.15.102:0/176955473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4420103580 0x7f44200fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f441e59c700 1 -- 172.21.15.102:0/176955473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4420103580 msgr2=0x7f44200fec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f441e59c700 1 --2- 172.21.15.102:0/176955473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4420103580 0x7f44200fec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f441e59c700 1 -- 172.21.15.102:0/176955473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44100095f0 con 0x7f4420100c60 2024-03-26T19:03:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.933+0000 7f441e59c700 1 --2- 172.21.15.102:0/176955473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4420100c60 0x7f44201005c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4410000c00 tx=0x7f441000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.934+0000 7f440f7fe700 1 -- 172.21.15.102:0/176955473 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f441001d070 con 0x7f4420100c60 2024-03-26T19:03:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.934+0000 7f440f7fe700 1 -- 172.21.15.102:0/176955473 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4410005dc0 con 0x7f4420100c60 2024-03-26T19:03:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.934+0000 7f4424d55700 1 -- 172.21.15.102:0/176955473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44200ff510 con 0x7f4420100c60 2024-03-26T19:03:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.934+0000 7f440f7fe700 1 -- 172.21.15.102:0/176955473 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44100226a0 con 0x7f4420100c60 2024-03-26T19:03:09.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.934+0000 7f4424d55700 1 -- 172.21.15.102:0/176955473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44200ffa00 con 0x7f4420100c60 2024-03-26T19:03:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.935+0000 7f4424d55700 1 -- 172.21.15.102:0/176955473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f442018fb70 con 0x7f4420100c60 2024-03-26T19:03:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.936+0000 7f440f7fe700 1 -- 172.21.15.102:0/176955473 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f44100038f0 con 0x7f4420100c60 2024-03-26T19:03:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.936+0000 7f440f7fe700 1 --2- 172.21.15.102:0/176955473 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f440806e460 0x7f4408070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.936+0000 7f440f7fe700 1 -- 172.21.15.102:0/176955473 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f441008f8c0 con 0x7f4420100c60 2024-03-26T19:03:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.936+0000 7f441dd9b700 1 --2- 172.21.15.102:0/176955473 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f440806e460 0x7f4408070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.937+0000 7f441dd9b700 1 --2- 172.21.15.102:0/176955473 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f440806e460 0x7f4408070910 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4414009c60 tx=0x7f4414009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:09.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:09.941+0000 7f440f7fe700 1 -- 172.21.15.102:0/176955473 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f441005c5d0 con 0x7f4420100c60 2024-03-26T19:03:10.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.326+0000 7f4424d55700 1 -- 172.21.15.102:0/176955473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f442002cc30 con 0x7f4420100c60 2024-03-26T19:03:10.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.327+0000 7f440f7fe700 1 -- 172.21.15.102:0/176955473 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f441005c160 con 0x7f4420100c60 2024-03-26T19:03:10.321 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:03:10.321 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T19:03:10.321 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:03:10.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:03:10.321 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T19:03:10.321 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:03:10.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:03:10.321 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T19:03:10.322 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 6 2024-03-26T19:03:10.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:03:10.322 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T19:03:10.322 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 4 2024-03-26T19:03:10.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:03:10.322 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T19:03:10.322 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 14 2024-03-26T19:03:10.322 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T19:03:10.322 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:03:10.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.329+0000 7f4424d55700 1 -- 172.21.15.102:0/176955473 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f440806e460 msgr2=0x7f4408070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:10.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.329+0000 7f4424d55700 1 --2- 172.21.15.102:0/176955473 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f440806e460 0x7f4408070910 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4414009c60 tx=0x7f4414009260 comp rx=0 tx=0).stop 2024-03-26T19:03:10.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.329+0000 7f4424d55700 1 -- 172.21.15.102:0/176955473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4420100c60 msgr2=0x7f44201005c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:10.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.329+0000 7f4424d55700 1 --2- 172.21.15.102:0/176955473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4420100c60 0x7f44201005c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4410000c00 tx=0x7f441000f690 comp rx=0 tx=0).stop 2024-03-26T19:03:10.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.329+0000 7f4424d55700 1 -- 172.21.15.102:0/176955473 shutdown_connections 2024-03-26T19:03:10.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.329+0000 7f4424d55700 1 --2- 172.21.15.102:0/176955473 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f440806e460 0x7f4408070910 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.329+0000 7f4424d55700 1 --2- 172.21.15.102:0/176955473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4420100c60 0x7f44201005c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.329+0000 7f4424d55700 1 --2- 172.21.15.102:0/176955473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4420103580 0x7f44200fec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.329+0000 7f4424d55700 1 -- 172.21.15.102:0/176955473 >> 172.21.15.102:0/176955473 conn(0x7f44200fa5e0 msgr2=0x7f44200fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:10.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.329+0000 7f4424d55700 1 -- 172.21.15.102:0/176955473 shutdown_connections 2024-03-26T19:03:10.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.329+0000 7f4424d55700 1 -- 172.21.15.102:0/176955473 wait complete. 2024-03-26T19:03:10.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.434+0000 7f909f7af700 1 -- 172.21.15.102:0/1511915499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9098100250 msgr2=0x7f90981006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:10.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.434+0000 7f909f7af700 1 --2- 172.21.15.102:0/1511915499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9098100250 0x7f90981006a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9088009910 tx=0x7f9088009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:10.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.434+0000 7f909f7af700 1 -- 172.21.15.102:0/1511915499 shutdown_connections 2024-03-26T19:03:10.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.434+0000 7f909f7af700 1 --2- 172.21.15.102:0/1511915499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9098100250 0x7f90981006a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.434+0000 7f909f7af700 1 --2- 172.21.15.102:0/1511915499 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90980ff050 0x7f90980ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.434+0000 7f909f7af700 1 -- 172.21.15.102:0/1511915499 >> 172.21.15.102:0/1511915499 conn(0x7f90980fa5e0 msgr2=0x7f90980fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:10.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.434+0000 7f909f7af700 1 -- 172.21.15.102:0/1511915499 shutdown_connections 2024-03-26T19:03:10.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.434+0000 7f909f7af700 1 -- 172.21.15.102:0/1511915499 wait complete. 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.435+0000 7f909f7af700 1 Processor -- start 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.435+0000 7f909f7af700 1 -- start start 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.435+0000 7f909f7af700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90980ff050 0x7f9098102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.435+0000 7f909f7af700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9098100250 0x7f9098101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.435+0000 7f909f7af700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9098101b70 con 0x7f90980ff050 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.435+0000 7f909f7af700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9098101cb0 con 0x7f9098100250 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.435+0000 7f909d54b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90980ff050 0x7f9098102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.435+0000 7f909d54b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90980ff050 0x7f9098102fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41786/0 (socket says 172.21.15.102:41786) 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.435+0000 7f909d54b700 1 -- 172.21.15.102:0/4194275103 learned_addr learned my addr 172.21.15.102:0/4194275103 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.435+0000 7f909cd4a700 1 --2- 172.21.15.102:0/4194275103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9098100250 0x7f9098101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.436+0000 7f909d54b700 1 -- 172.21.15.102:0/4194275103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9098100250 msgr2=0x7f9098101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.436+0000 7f909d54b700 1 --2- 172.21.15.102:0/4194275103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9098100250 0x7f9098101630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.436+0000 7f909d54b700 1 -- 172.21.15.102:0/4194275103 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90880095f0 con 0x7f90980ff050 2024-03-26T19:03:10.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.436+0000 7f909d54b700 1 --2- 172.21.15.102:0/4194275103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90980ff050 0x7f9098102fe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f909400c8f0 tx=0x7f909400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:10.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.436+0000 7f908e7fc700 1 -- 172.21.15.102:0/4194275103 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90940078e0 con 0x7f90980ff050 2024-03-26T19:03:10.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.436+0000 7f908e7fc700 1 -- 172.21.15.102:0/4194275103 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f909400f450 con 0x7f90980ff050 2024-03-26T19:03:10.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.436+0000 7f909f7af700 1 -- 172.21.15.102:0/4194275103 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9098101f90 con 0x7f90980ff050 2024-03-26T19:03:10.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.436+0000 7f908e7fc700 1 -- 172.21.15.102:0/4194275103 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9094018610 con 0x7f90980ff050 2024-03-26T19:03:10.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.436+0000 7f909f7af700 1 -- 172.21.15.102:0/4194275103 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909819d9f0 con 0x7f90980ff050 2024-03-26T19:03:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.437+0000 7f909f7af700 1 -- 172.21.15.102:0/4194275103 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9098107ba0 con 0x7f90980ff050 2024-03-26T19:03:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.438+0000 7f908e7fc700 1 -- 172.21.15.102:0/4194275103 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f909400f5c0 con 0x7f90980ff050 2024-03-26T19:03:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.438+0000 7f908e7fc700 1 --2- 172.21.15.102:0/4194275103 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f908406e460 0x7f9084070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:10.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.438+0000 7f909cd4a700 1 --2- 172.21.15.102:0/4194275103 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f908406e460 0x7f9084070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:10.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.438+0000 7f908e7fc700 1 -- 172.21.15.102:0/4194275103 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f909408e260 con 0x7f90980ff050 2024-03-26T19:03:10.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.439+0000 7f909cd4a700 1 --2- 172.21.15.102:0/4194275103 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f908406e460 0x7f9084070910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9088006010 tx=0x7f908800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:10.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.443+0000 7f908e7fc700 1 -- 172.21.15.102:0/4194275103 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f909405af70 con 0x7f90980ff050 2024-03-26T19:03:10.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:10 smithi190 ceph-mon[65689]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:10.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:10 smithi190 ceph-mon[65689]: Upgrade: Updating ceph-exporter.smithi190 (2/2) 2024-03-26T19:03:10.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:10 smithi190 ceph-mon[65689]: Deploying daemon ceph-exporter.smithi190 on smithi190 2024-03-26T19:03:10.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:10 smithi102 ceph-mon[78322]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:10.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:10 smithi102 ceph-mon[78322]: Upgrade: Updating ceph-exporter.smithi190 (2/2) 2024-03-26T19:03:10.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:10 smithi102 ceph-mon[78322]: Deploying daemon ceph-exporter.smithi190 on smithi190 2024-03-26T19:03:10.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.698+0000 7f909f7af700 1 -- 172.21.15.102:0/4194275103 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f909819dfa0 con 0x7f90980ff050 2024-03-26T19:03:10.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.698+0000 7f908e7fc700 1 -- 172.21.15.102:0/4194275103 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 43 v43) v1 ==== 76+0+1955 (secure 0 0 0) 0x7f909405ab00 con 0x7f90980ff050 2024-03-26T19:03:10.693 INFO:teuthology.orchestra.run.smithi102.stdout:e43 2024-03-26T19:03:10.693 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T19:03:10.693 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:03:10.693 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T19:03:10.693 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:03:10.693 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T19:03:10.693 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T19:03:10.693 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 43 2024-03-26T19:03:10.693 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T19:03:10.693 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T19:03:02.920923+0000 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 84 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T19:03:10.694 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=34436,1=34468} 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 34436 members: 34436,34468 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{0:34436} state up:active seq 19 join_fscid=1 addr [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{1:34468} state up:active seq 13 join_fscid=1 addr [v2:172.21.15.102:6828/3007238267,v1:172.21.15.102:6829/3007238267] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T19:03:10.695 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:03:10.696 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.190:6824/782636706,v1:172.21.15.190:6825/782636706] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:03:10.696 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:44373} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.190:6826/925215849,v1:172.21.15.190:6827/925215849] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:03:10.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.700+0000 7f909f7af700 1 -- 172.21.15.102:0/4194275103 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f908406e460 msgr2=0x7f9084070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:10.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.700+0000 7f909f7af700 1 --2- 172.21.15.102:0/4194275103 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f908406e460 0x7f9084070910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9088006010 tx=0x7f908800b540 comp rx=0 tx=0).stop 2024-03-26T19:03:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.700+0000 7f909f7af700 1 -- 172.21.15.102:0/4194275103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90980ff050 msgr2=0x7f9098102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.700+0000 7f909f7af700 1 --2- 172.21.15.102:0/4194275103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90980ff050 0x7f9098102fe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f909400c8f0 tx=0x7f909400cc00 comp rx=0 tx=0).stop 2024-03-26T19:03:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.701+0000 7f909f7af700 1 -- 172.21.15.102:0/4194275103 shutdown_connections 2024-03-26T19:03:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.701+0000 7f909f7af700 1 --2- 172.21.15.102:0/4194275103 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f908406e460 0x7f9084070910 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.701+0000 7f909f7af700 1 --2- 172.21.15.102:0/4194275103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90980ff050 0x7f9098102fe0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.701+0000 7f909f7af700 1 --2- 172.21.15.102:0/4194275103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9098100250 0x7f9098101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.701+0000 7f909f7af700 1 -- 172.21.15.102:0/4194275103 >> 172.21.15.102:0/4194275103 conn(0x7f90980fa5e0 msgr2=0x7f90980683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.701+0000 7f909f7af700 1 -- 172.21.15.102:0/4194275103 shutdown_connections 2024-03-26T19:03:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.701+0000 7f909f7af700 1 -- 172.21.15.102:0/4194275103 wait complete. 2024-03-26T19:03:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 43 2024-03-26T19:03:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.848+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2030279434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1dc100190 msgr2=0x7fc1dc1005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.848+0000 7fc1e15bb700 1 --2- 172.21.15.102:0/2030279434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1dc100190 0x7fc1dc1005a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0009910 tx=0x7fc1d0009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.848+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2030279434 shutdown_connections 2024-03-26T19:03:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.848+0000 7fc1e15bb700 1 --2- 172.21.15.102:0/2030279434 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1dc101390 0x7fc1dc1017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.848+0000 7fc1e15bb700 1 --2- 172.21.15.102:0/2030279434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1dc100190 0x7fc1dc1005a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.848+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2030279434 >> 172.21.15.102:0/2030279434 conn(0x7fc1dc0fb720 msgr2=0x7fc1dc0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.848+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2030279434 shutdown_connections 2024-03-26T19:03:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.848+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2030279434 wait complete. 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.849+0000 7fc1e15bb700 1 Processor -- start 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.849+0000 7fc1e15bb700 1 -- start start 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.849+0000 7fc1e15bb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1dc100190 0x7fc1dc195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.849+0000 7fc1e15bb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1dc101390 0x7fc1dc195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.849+0000 7fc1e15bb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc1964f0 con 0x7fc1dc100190 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.849+0000 7fc1e15bb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc196630 con 0x7fc1dc101390 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.850+0000 7fc1daffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1dc100190 0x7fc1dc195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.850+0000 7fc1daffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1dc100190 0x7fc1dc195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41798/0 (socket says 172.21.15.102:41798) 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.850+0000 7fc1daffd700 1 -- 172.21.15.102:0/2934499774 learned_addr learned my addr 172.21.15.102:0/2934499774 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.850+0000 7fc1da7fc700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1dc101390 0x7fc1dc195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.850+0000 7fc1da7fc700 1 -- 172.21.15.102:0/2934499774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1dc100190 msgr2=0x7fc1dc195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.850+0000 7fc1da7fc700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1dc100190 0x7fc1dc195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:10.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.850+0000 7fc1da7fc700 1 -- 172.21.15.102:0/2934499774 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d00095f0 con 0x7fc1dc101390 2024-03-26T19:03:10.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.850+0000 7fc1daffd700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1dc100190 0x7fc1dc195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T19:03:10.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.850+0000 7fc1da7fc700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1dc101390 0x7fc1dc195ed0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc00d6d0 tx=0x7fc1cc00d9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:10.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.850+0000 7fc1c3fff700 1 -- 172.21.15.102:0/2934499774 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1cc0041f0 con 0x7fc1dc101390 2024-03-26T19:03:10.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.851+0000 7fc1c3fff700 1 -- 172.21.15.102:0/2934499774 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc1cc004ce0 con 0x7fc1dc101390 2024-03-26T19:03:10.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.851+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2934499774 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1dc19b0e0 con 0x7fc1dc101390 2024-03-26T19:03:10.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.851+0000 7fc1c3fff700 1 -- 172.21.15.102:0/2934499774 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1cc00f690 con 0x7fc1dc101390 2024-03-26T19:03:10.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.851+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2934499774 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1dc19b630 con 0x7fc1dc101390 2024-03-26T19:03:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.852+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2934499774 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1dc108ce0 con 0x7fc1dc101390 2024-03-26T19:03:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.852+0000 7fc1c3fff700 1 -- 172.21.15.102:0/2934499774 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc1cc004350 con 0x7fc1dc101390 2024-03-26T19:03:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.853+0000 7fc1c3fff700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc1c406e460 0x7fc1c4070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.853+0000 7fc1c3fff700 1 -- 172.21.15.102:0/2934499774 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc1cc08dff0 con 0x7fc1dc101390 2024-03-26T19:03:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.853+0000 7fc1daffd700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc1c406e460 0x7fc1c4070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.853+0000 7fc1daffd700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc1c406e460 0x7fc1c4070910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0006010 tx=0x7fc1d000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:10.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:10.857+0000 7fc1c3fff700 1 -- 172.21.15.102:0/2934499774 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc1cc05ac80 con 0x7fc1dc101390 2024-03-26T19:03:11.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.139+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2934499774 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1dc105ce0 con 0x7fc1c406e460 2024-03-26T19:03:11.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.140+0000 7fc1c3fff700 1 -- 172.21.15.102:0/2934499774 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7fc1dc105ce0 con 0x7fc1c406e460 2024-03-26T19:03:11.136 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:03:11.136 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T19:03:11.136 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T19:03:11.136 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T19:03:11.136 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T19:03:11.136 INFO:teuthology.orchestra.run.smithi102.stdout: "mds", 2024-03-26T19:03:11.136 INFO:teuthology.orchestra.run.smithi102.stdout: "osd", 2024-03-26T19:03:11.136 INFO:teuthology.orchestra.run.smithi102.stdout: "mon", 2024-03-26T19:03:11.136 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T19:03:11.137 INFO:teuthology.orchestra.run.smithi102.stdout: "crash" 2024-03-26T19:03:11.137 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T19:03:11.137 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "16/23 daemons upgraded", 2024-03-26T19:03:11.137 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading ceph-exporter daemons", 2024-03-26T19:03:11.137 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T19:03:11.137 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:03:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.143+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2934499774 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc1c406e460 msgr2=0x7fc1c4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.143+0000 7fc1e15bb700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc1c406e460 0x7fc1c4070910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0006010 tx=0x7fc1d000b540 comp rx=0 tx=0).stop 2024-03-26T19:03:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.143+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2934499774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1dc101390 msgr2=0x7fc1dc195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.143+0000 7fc1e15bb700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1dc101390 0x7fc1dc195ed0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc00d6d0 tx=0x7fc1cc00d9e0 comp rx=0 tx=0).stop 2024-03-26T19:03:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.143+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2934499774 shutdown_connections 2024-03-26T19:03:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.143+0000 7fc1e15bb700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc1c406e460 0x7fc1c4070910 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.143+0000 7fc1e15bb700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1dc100190 0x7fc1dc195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.143+0000 7fc1e15bb700 1 --2- 172.21.15.102:0/2934499774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1dc101390 0x7fc1dc195ed0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.143+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2934499774 >> 172.21.15.102:0/2934499774 conn(0x7fc1dc0fb720 msgr2=0x7fc1dc1045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:11.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.144+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2934499774 shutdown_connections 2024-03-26T19:03:11.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.144+0000 7fc1e15bb700 1 -- 172.21.15.102:0/2934499774 wait complete. 2024-03-26T19:03:11.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.260+0000 7f3f1da24700 1 -- 172.21.15.102:0/2419285349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18103500 msgr2=0x7f3f181058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:11.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.260+0000 7f3f1da24700 1 --2- 172.21.15.102:0/2419285349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18103500 0x7f3f181058e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009910 tx=0x7f3f0c009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:11.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.260+0000 7f3f1da24700 1 -- 172.21.15.102:0/2419285349 shutdown_connections 2024-03-26T19:03:11.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.260+0000 7f3f1da24700 1 --2- 172.21.15.102:0/2419285349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18103500 0x7f3f181058e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:11.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.260+0000 7f3f1da24700 1 --2- 172.21.15.102:0/2419285349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f18100be0 0x7f3f18102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:11.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.260+0000 7f3f1da24700 1 -- 172.21.15.102:0/2419285349 >> 172.21.15.102:0/2419285349 conn(0x7f3f180fa5e0 msgr2=0x7f3f180fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:11.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.260+0000 7f3f1da24700 1 -- 172.21.15.102:0/2419285349 shutdown_connections 2024-03-26T19:03:11.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.260+0000 7f3f1da24700 1 -- 172.21.15.102:0/2419285349 wait complete. 2024-03-26T19:03:11.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.260+0000 7f3f1da24700 1 Processor -- start 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f1da24700 1 -- start start 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f1da24700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f18100be0 0x7f3f18195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f1da24700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18103500 0x7f3f18195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f1da24700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f18196490 con 0x7f3f18103500 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f1da24700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f181965d0 con 0x7f3f18100be0 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f167fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18103500 0x7f3f18195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f167fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18103500 0x7f3f18195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41814/0 (socket says 172.21.15.102:41814) 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f167fc700 1 -- 172.21.15.102:0/122009691 learned_addr learned my addr 172.21.15.102:0/122009691 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f16ffd700 1 --2- 172.21.15.102:0/122009691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f18100be0 0x7f3f18195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f167fc700 1 -- 172.21.15.102:0/122009691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f18100be0 msgr2=0x7f3f18195930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f167fc700 1 --2- 172.21.15.102:0/122009691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f18100be0 0x7f3f18195930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.261+0000 7f3f167fc700 1 -- 172.21.15.102:0/122009691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c0095f0 con 0x7f3f18103500 2024-03-26T19:03:11.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.262+0000 7f3f167fc700 1 --2- 172.21.15.102:0/122009691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18103500 0x7f3f18195e70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c000c00 tx=0x7f3f0c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:11.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.262+0000 7f3f1ca22700 1 -- 172.21.15.102:0/122009691 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f0c01d070 con 0x7f3f18103500 2024-03-26T19:03:11.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.262+0000 7f3f1ca22700 1 -- 172.21.15.102:0/122009691 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3f0c005dc0 con 0x7f3f18103500 2024-03-26T19:03:11.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.262+0000 7f3f1da24700 1 -- 172.21.15.102:0/122009691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1819b020 con 0x7f3f18103500 2024-03-26T19:03:11.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.262+0000 7f3f1da24700 1 -- 172.21.15.102:0/122009691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1819b510 con 0x7f3f18103500 2024-03-26T19:03:11.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.262+0000 7f3f1ca22700 1 -- 172.21.15.102:0/122009691 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f0c0226a0 con 0x7f3f18103500 2024-03-26T19:03:11.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.263+0000 7f3f1da24700 1 -- 172.21.15.102:0/122009691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1818fb70 con 0x7f3f18103500 2024-03-26T19:03:11.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.264+0000 7f3f1ca22700 1 -- 172.21.15.102:0/122009691 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3f0c0038f0 con 0x7f3f18103500 2024-03-26T19:03:11.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.264+0000 7f3f1ca22700 1 --2- 172.21.15.102:0/122009691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f0006e460 0x7f3f00070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:11.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.264+0000 7f3f1ca22700 1 -- 172.21.15.102:0/122009691 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f3f0c08f8a0 con 0x7f3f18103500 2024-03-26T19:03:11.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.264+0000 7f3f16ffd700 1 --2- 172.21.15.102:0/122009691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f0006e460 0x7f3f00070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:11.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.265+0000 7f3f16ffd700 1 --2- 172.21.15.102:0/122009691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f0006e460 0x7f3f00070910 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3f08009c60 tx=0x7f3f08009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:11.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.269+0000 7f3f1ca22700 1 -- 172.21.15.102:0/122009691 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f0c05c5b0 con 0x7f3f18103500 2024-03-26T19:03:11.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.563+0000 7f3f1da24700 1 -- 172.21.15.102:0/122009691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3f1802cc30 con 0x7f3f18103500 2024-03-26T19:03:11.558 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:11 smithi102 ceph-mon[78322]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:11.558 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:11 smithi102 ceph-mon[78322]: pgmap v233: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 9.6 KiB/s rd, 4.4 KiB/s wr, 11 op/s 2024-03-26T19:03:11.558 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:11 smithi102 ceph-mon[78322]: from='client.34514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:11.558 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:11 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/176955473' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:11.559 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:11 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/4194275103' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:03:11.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.563+0000 7f3f1ca22700 1 -- 172.21.15.102:0/122009691 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f3f0c003680 con 0x7f3f18103500 2024-03-26T19:03:11.559 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T19:03:11.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.566+0000 7f3f1da24700 1 -- 172.21.15.102:0/122009691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f0006e460 msgr2=0x7f3f00070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:11.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.566+0000 7f3f1da24700 1 --2- 172.21.15.102:0/122009691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f0006e460 0x7f3f00070910 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3f08009c60 tx=0x7f3f08009260 comp rx=0 tx=0).stop 2024-03-26T19:03:11.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.566+0000 7f3f1da24700 1 -- 172.21.15.102:0/122009691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18103500 msgr2=0x7f3f18195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:11.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.566+0000 7f3f1da24700 1 --2- 172.21.15.102:0/122009691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18103500 0x7f3f18195e70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c000c00 tx=0x7f3f0c00f690 comp rx=0 tx=0).stop 2024-03-26T19:03:11.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.566+0000 7f3f1da24700 1 -- 172.21.15.102:0/122009691 shutdown_connections 2024-03-26T19:03:11.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.566+0000 7f3f1da24700 1 --2- 172.21.15.102:0/122009691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f18100be0 0x7f3f18195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:11.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.566+0000 7f3f1da24700 1 --2- 172.21.15.102:0/122009691 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f0006e460 0x7f3f00070910 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:11.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.566+0000 7f3f1da24700 1 --2- 172.21.15.102:0/122009691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18103500 0x7f3f18195e70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:11.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.566+0000 7f3f1da24700 1 -- 172.21.15.102:0/122009691 >> 172.21.15.102:0/122009691 conn(0x7f3f180fa5e0 msgr2=0x7f3f180fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:11.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.566+0000 7f3f1da24700 1 -- 172.21.15.102:0/122009691 shutdown_connections 2024-03-26T19:03:11.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:11.566+0000 7f3f1da24700 1 -- 172.21.15.102:0/122009691 wait complete. 2024-03-26T19:03:11.646 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:11 smithi190 ceph-mon[65689]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:11.646 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:11 smithi190 ceph-mon[65689]: pgmap v233: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 9.6 KiB/s rd, 4.4 KiB/s wr, 11 op/s 2024-03-26T19:03:11.646 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:11 smithi190 ceph-mon[65689]: from='client.34514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:11.647 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:11 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/176955473' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:11.647 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:11 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/4194275103' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:03:12.550 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:12 smithi190 ceph-mon[65689]: from='client.44397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:12.550 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:12 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/122009691' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:03:12.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:12 smithi102 ceph-mon[78322]: from='client.44397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:12.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:12 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/122009691' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:03:13.548 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:13 smithi190 ceph-mon[65689]: pgmap v234: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 7.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T19:03:13.548 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:13.548 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:13.549 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:13 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:13.581 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:13 smithi102 ceph-mon[78322]: pgmap v234: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 7.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T19:03:13.581 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:13.581 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:13.581 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:13 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:15.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:15 smithi102 ceph-mon[78322]: pgmap v235: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 6.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-26T19:03:15.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:15.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:15.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:15 smithi190 ceph-mon[65689]: pgmap v235: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 6.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-26T19:03:15.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:15.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:17.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:17 smithi190 ceph-mon[65689]: pgmap v236: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-26T19:03:17.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:17 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:17.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:17 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:17.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:17 smithi102 ceph-mon[78322]: pgmap v236: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-26T19:03:17.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:17 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:17.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:17 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:18.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:18 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:18.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:18 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:18.730 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:18 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:18.730 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:18 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:19.514 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:19 smithi102 ceph-mon[78322]: pgmap v237: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 170 B/s wr, 1 op/s 2024-03-26T19:03:19.514 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:19 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:19.514 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:19 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:19.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:19 smithi190 ceph-mon[65689]: pgmap v237: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 170 B/s wr, 1 op/s 2024-03-26T19:03:19.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:19 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:19.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:19 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:20.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:20 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:20.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:20 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:20.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:20 smithi102 ceph-mon[78322]: pgmap v238: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-26T19:03:20.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:20 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:20.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:20 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:20.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:20 smithi190 ceph-mon[65689]: pgmap v238: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-26T19:03:23.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: pgmap v239: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:23.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:23.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:03:23.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi102"}]: dispatch 2024-03-26T19:03:23.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi102"}]': finished 2024-03-26T19:03:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi190"}]: dispatch 2024-03-26T19:03:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi190"}]': finished 2024-03-26T19:03:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: Upgrade: Setting container_image for all iscsi 2024-03-26T19:03:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: Upgrade: Setting container_image for all nfs 2024-03-26T19:03:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: Upgrade: Setting container_image for all nvmeof 2024-03-26T19:03:23.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: pgmap v239: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:23.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:23.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:03:23.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:03:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi102"}]: dispatch 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi102"}]': finished 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi190"}]: dispatch 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi190"}]': finished 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: Upgrade: Setting container_image for all iscsi 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: Upgrade: Setting container_image for all nfs 2024-03-26T19:03:23.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:23.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:23.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: Upgrade: Setting container_image for all nvmeof 2024-03-26T19:03:23.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:25.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:25 smithi102 ceph-mon[78322]: pgmap v240: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:25.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:25 smithi102 ceph-mon[78322]: Upgrade: Updating node-exporter.smithi102 (1/2) 2024-03-26T19:03:25.431 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:25 smithi102 ceph-mon[78322]: Deploying daemon node-exporter.smithi102 on smithi102 2024-03-26T19:03:25.496 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:25 smithi190 ceph-mon[65689]: pgmap v240: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:25.496 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:25 smithi190 ceph-mon[65689]: Upgrade: Updating node-exporter.smithi102 (1/2) 2024-03-26T19:03:25.496 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:25 smithi190 ceph-mon[65689]: Deploying daemon node-exporter.smithi102 on smithi102 2024-03-26T19:03:27.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:27 smithi190 ceph-mon[65689]: pgmap v241: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:27.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:27 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:27.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:27 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:27.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:27 smithi102 ceph-mon[78322]: pgmap v241: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:27.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:27 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:27.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:27 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:29.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:29 smithi190 ceph-mon[65689]: pgmap v242: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:29.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:29 smithi102 ceph-mon[78322]: pgmap v242: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:30.274 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:30 smithi190 ceph-mon[65689]: Upgrade: Updating node-exporter.smithi190 (2/2) 2024-03-26T19:03:30.274 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:30 smithi190 ceph-mon[65689]: Deploying daemon node-exporter.smithi190 on smithi190 2024-03-26T19:03:30.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:30 smithi102 ceph-mon[78322]: Upgrade: Updating node-exporter.smithi190 (2/2) 2024-03-26T19:03:30.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:30 smithi102 ceph-mon[78322]: Deploying daemon node-exporter.smithi190 on smithi190 2024-03-26T19:03:31.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:31 smithi190 ceph-mon[65689]: pgmap v243: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:31.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:31.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:31.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:31 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:31.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:31 smithi102 ceph-mon[78322]: pgmap v243: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:31.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:31.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:31.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:31 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:33.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:33 smithi102 ceph-mon[78322]: pgmap v244: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:33.451 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:33 smithi190 ceph-mon[65689]: pgmap v244: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:34.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:34 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:34.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:34 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:34.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:34 smithi102 ceph-mon[78322]: pgmap v245: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:34.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:34 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:34.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:34 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:34.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:34 smithi190 ceph-mon[65689]: pgmap v245: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:35.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:35.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:36.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:36.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:37.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:36 smithi102 ceph-mon[78322]: pgmap v246: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:37.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:36 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:37.128 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:36 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:37.218 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:36 smithi190 ceph-mon[65689]: pgmap v246: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:37.218 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:36 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:37.218 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:36 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:38.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:38.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:38.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:03:38.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:38.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:38.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:38.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:38.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:03:38.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:38.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:39.459 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:39 smithi102 ceph-mon[78322]: pgmap v247: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:39.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:39 smithi190 ceph-mon[65689]: pgmap v247: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:41.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: pgmap v248: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:41.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:41.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:41.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:41.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:03:41.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:41.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: pgmap v248: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:41.647 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:41.647 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:41.647 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:41.647 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:03:41.648 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:41.648 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:41.648 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.648 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.649 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.649 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.649 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.649 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.649 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.650 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.650 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.650 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.650 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.650 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.650 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:41.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.720+0000 7ff91b8e1700 1 -- 172.21.15.102:0/2054509306 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff914100250 msgr2=0x7ff9141006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:41.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.720+0000 7ff91b8e1700 1 --2- 172.21.15.102:0/2054509306 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff914100250 0x7ff9141006a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff904009910 tx=0x7ff904009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:41.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.720+0000 7ff91b8e1700 1 -- 172.21.15.102:0/2054509306 shutdown_connections 2024-03-26T19:03:41.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.720+0000 7ff91b8e1700 1 --2- 172.21.15.102:0/2054509306 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff914100250 0x7ff9141006a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:41.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.720+0000 7ff91b8e1700 1 --2- 172.21.15.102:0/2054509306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9140ff050 0x7ff9140ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:41.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.720+0000 7ff91b8e1700 1 -- 172.21.15.102:0/2054509306 >> 172.21.15.102:0/2054509306 conn(0x7ff9140fa5e0 msgr2=0x7ff9140fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:41.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.720+0000 7ff91b8e1700 1 -- 172.21.15.102:0/2054509306 shutdown_connections 2024-03-26T19:03:41.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.720+0000 7ff91b8e1700 1 -- 172.21.15.102:0/2054509306 wait complete. 2024-03-26T19:03:41.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.721+0000 7ff91b8e1700 1 Processor -- start 2024-03-26T19:03:41.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.721+0000 7ff91b8e1700 1 -- start start 2024-03-26T19:03:41.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.721+0000 7ff91b8e1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9140ff050 0x7ff914194860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:41.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.721+0000 7ff91b8e1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff914100250 0x7ff914194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:41.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.721+0000 7ff91b8e1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9141953c0 con 0x7ff9140ff050 2024-03-26T19:03:41.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.721+0000 7ff91b8e1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff914195500 con 0x7ff914100250 2024-03-26T19:03:41.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.721+0000 7ff91967d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9140ff050 0x7ff914194860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:41.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.721+0000 7ff91967d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9140ff050 0x7ff914194860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33894/0 (socket says 172.21.15.102:33894) 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.721+0000 7ff91967d700 1 -- 172.21.15.102:0/1243805740 learned_addr learned my addr 172.21.15.102:0/1243805740 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.722+0000 7ff918e7c700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff914100250 0x7ff914194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.722+0000 7ff91967d700 1 -- 172.21.15.102:0/1243805740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff914100250 msgr2=0x7ff914194da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.722+0000 7ff91967d700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff914100250 0x7ff914194da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.722+0000 7ff91967d700 1 -- 172.21.15.102:0/1243805740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9040095f0 con 0x7ff9140ff050 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.722+0000 7ff918e7c700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff914100250 0x7ff914194da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.722+0000 7ff91967d700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9140ff050 0x7ff914194860 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff91000d900 tx=0x7ff91000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.722+0000 7ff90a7fc700 1 -- 172.21.15.102:0/1243805740 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9100041f0 con 0x7ff9140ff050 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.722+0000 7ff90a7fc700 1 -- 172.21.15.102:0/1243805740 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff910004350 con 0x7ff9140ff050 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.722+0000 7ff90a7fc700 1 -- 172.21.15.102:0/1243805740 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff910003da0 con 0x7ff9140ff050 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.722+0000 7ff91b8e1700 1 -- 172.21.15.102:0/1243805740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff914068570 con 0x7ff9140ff050 2024-03-26T19:03:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.722+0000 7ff91b8e1700 1 -- 172.21.15.102:0/1243805740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff914068b40 con 0x7ff9140ff050 2024-03-26T19:03:41.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.723+0000 7ff91b8e1700 1 -- 172.21.15.102:0/1243805740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff914107ba0 con 0x7ff9140ff050 2024-03-26T19:03:41.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.724+0000 7ff90a7fc700 1 -- 172.21.15.102:0/1243805740 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff910009730 con 0x7ff9140ff050 2024-03-26T19:03:41.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.724+0000 7ff90a7fc700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff90006e460 0x7ff900070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:41.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.724+0000 7ff90a7fc700 1 -- 172.21.15.102:0/1243805740 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7ff91008e5a0 con 0x7ff9140ff050 2024-03-26T19:03:41.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.724+0000 7ff918e7c700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff90006e460 0x7ff900070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:41.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.725+0000 7ff918e7c700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff90006e460 0x7ff900070910 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff904005b40 tx=0x7ff904005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:41.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.733+0000 7ff90a7fc700 1 -- 172.21.15.102:0/1243805740 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff91005b230 con 0x7ff9140ff050 2024-03-26T19:03:41.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.966+0000 7ff91b8e1700 1 -- 172.21.15.102:0/1243805740 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff914104ba0 con 0x7ff90006e460 2024-03-26T19:03:41.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.968+0000 7ff90a7fc700 1 -- 172.21.15.102:0/1243805740 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+463 (secure 0 0 0) 0x7ff914104ba0 con 0x7ff90006e460 2024-03-26T19:03:41.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.970+0000 7ff91b8e1700 1 -- 172.21.15.102:0/1243805740 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff90006e460 msgr2=0x7ff900070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:41.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.970+0000 7ff91b8e1700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff90006e460 0x7ff900070910 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff904005b40 tx=0x7ff904005a90 comp rx=0 tx=0).stop 2024-03-26T19:03:41.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.970+0000 7ff91b8e1700 1 -- 172.21.15.102:0/1243805740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9140ff050 msgr2=0x7ff914194860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:41.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.970+0000 7ff91b8e1700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9140ff050 0x7ff914194860 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff91000d900 tx=0x7ff91000dcc0 comp rx=0 tx=0).stop 2024-03-26T19:03:41.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.970+0000 7ff91b8e1700 1 -- 172.21.15.102:0/1243805740 shutdown_connections 2024-03-26T19:03:41.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.970+0000 7ff91b8e1700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff90006e460 0x7ff900070910 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:41.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.970+0000 7ff91b8e1700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9140ff050 0x7ff914194860 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:41.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.970+0000 7ff91b8e1700 1 --2- 172.21.15.102:0/1243805740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff914100250 0x7ff914194da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:41.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.970+0000 7ff91b8e1700 1 -- 172.21.15.102:0/1243805740 >> 172.21.15.102:0/1243805740 conn(0x7ff9140fa5e0 msgr2=0x7ff914103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:41.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.970+0000 7ff91b8e1700 1 -- 172.21.15.102:0/1243805740 shutdown_connections 2024-03-26T19:03:41.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:41.970+0000 7ff91b8e1700 1 -- 172.21.15.102:0/1243805740 wait complete. 2024-03-26T19:03:41.976 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T19:03:42.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.080+0000 7f0569a81700 1 -- 172.21.15.102:0/3757344468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0564100250 msgr2=0x7f05641006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:42.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.080+0000 7f0569a81700 1 --2- 172.21.15.102:0/3757344468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0564100250 0x7f05641006a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0558009910 tx=0x7f0558009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:42.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.081+0000 7f0569a81700 1 -- 172.21.15.102:0/3757344468 shutdown_connections 2024-03-26T19:03:42.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.081+0000 7f0569a81700 1 --2- 172.21.15.102:0/3757344468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0564100250 0x7f05641006a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.081+0000 7f0569a81700 1 --2- 172.21.15.102:0/3757344468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05640ff050 0x7f05640ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.081+0000 7f0569a81700 1 -- 172.21.15.102:0/3757344468 >> 172.21.15.102:0/3757344468 conn(0x7f05640fa5e0 msgr2=0x7f05640fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:42.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.081+0000 7f0569a81700 1 -- 172.21.15.102:0/3757344468 shutdown_connections 2024-03-26T19:03:42.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.081+0000 7f0569a81700 1 -- 172.21.15.102:0/3757344468 wait complete. 2024-03-26T19:03:42.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.081+0000 7f0569a81700 1 Processor -- start 2024-03-26T19:03:42.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.081+0000 7f0569a81700 1 -- start start 2024-03-26T19:03:42.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f0569a81700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05640ff050 0x7f0564195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:42.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f0569a81700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0564100250 0x7f0564195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:42.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f0569a81700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05641964f0 con 0x7f05640ff050 2024-03-26T19:03:42.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f0569a81700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0564196630 con 0x7f0564100250 2024-03-26T19:03:42.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f05637fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05640ff050 0x7f0564195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:42.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f05637fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05640ff050 0x7f0564195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33924/0 (socket says 172.21.15.102:33924) 2024-03-26T19:03:42.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f05637fe700 1 -- 172.21.15.102:0/2686889803 learned_addr learned my addr 172.21.15.102:0/2686889803 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:42.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f0562ffd700 1 --2- 172.21.15.102:0/2686889803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0564100250 0x7f0564195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:42.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f05637fe700 1 -- 172.21.15.102:0/2686889803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0564100250 msgr2=0x7f0564195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:42.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f05637fe700 1 --2- 172.21.15.102:0/2686889803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0564100250 0x7f0564195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f05637fe700 1 -- 172.21.15.102:0/2686889803 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05580095f0 con 0x7f05640ff050 2024-03-26T19:03:42.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.082+0000 7f05637fe700 1 --2- 172.21.15.102:0/2686889803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05640ff050 0x7f0564195990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f055400e8f0 tx=0x7f055400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:42.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.083+0000 7f0560ff9700 1 -- 172.21.15.102:0/2686889803 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0554009790 con 0x7f05640ff050 2024-03-26T19:03:42.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.083+0000 7f0560ff9700 1 -- 172.21.15.102:0/2686889803 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f055400c810 con 0x7f05640ff050 2024-03-26T19:03:42.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.083+0000 7f0560ff9700 1 -- 172.21.15.102:0/2686889803 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0554003d20 con 0x7f05640ff050 2024-03-26T19:03:42.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.083+0000 7f0569a81700 1 -- 172.21.15.102:0/2686889803 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056419b0e0 con 0x7f05640ff050 2024-03-26T19:03:42.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.084+0000 7f0569a81700 1 -- 172.21.15.102:0/2686889803 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056419b600 con 0x7f05640ff050 2024-03-26T19:03:42.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.085+0000 7f0569a81700 1 -- 172.21.15.102:0/2686889803 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0564107ba0 con 0x7f05640ff050 2024-03-26T19:03:42.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.085+0000 7f0560ff9700 1 -- 172.21.15.102:0/2686889803 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f05540041f0 con 0x7f05640ff050 2024-03-26T19:03:42.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.086+0000 7f0560ff9700 1 --2- 172.21.15.102:0/2686889803 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f054c06e250 0x7f054c070700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:42.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.086+0000 7f0560ff9700 1 -- 172.21.15.102:0/2686889803 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f0554014070 con 0x7f05640ff050 2024-03-26T19:03:42.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.086+0000 7f0562ffd700 1 --2- 172.21.15.102:0/2686889803 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f054c06e250 0x7f054c070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:42.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.086+0000 7f0562ffd700 1 --2- 172.21.15.102:0/2686889803 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f054c06e250 0x7f054c070700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0558009fd0 tx=0x7f0558005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:42.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.095+0000 7f0560ff9700 1 -- 172.21.15.102:0/2686889803 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f055405b470 con 0x7f05640ff050 2024-03-26T19:03:42.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.347+0000 7f0569a81700 1 -- 172.21.15.102:0/2686889803 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0564068ae0 con 0x7f054c06e250 2024-03-26T19:03:42.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.349+0000 7f0560ff9700 1 -- 172.21.15.102:0/2686889803 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+463 (secure 0 0 0) 0x7f0564068ae0 con 0x7f054c06e250 2024-03-26T19:03:42.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.351+0000 7f0569a81700 1 -- 172.21.15.102:0/2686889803 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f054c06e250 msgr2=0x7f054c070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:42.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.351+0000 7f0569a81700 1 --2- 172.21.15.102:0/2686889803 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f054c06e250 0x7f054c070700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0558009fd0 tx=0x7f0558005940 comp rx=0 tx=0).stop 2024-03-26T19:03:42.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.351+0000 7f0569a81700 1 -- 172.21.15.102:0/2686889803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05640ff050 msgr2=0x7f0564195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:42.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.351+0000 7f0569a81700 1 --2- 172.21.15.102:0/2686889803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05640ff050 0x7f0564195990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f055400e8f0 tx=0x7f055400ec00 comp rx=0 tx=0).stop 2024-03-26T19:03:42.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.351+0000 7f0569a81700 1 -- 172.21.15.102:0/2686889803 shutdown_connections 2024-03-26T19:03:42.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.351+0000 7f0569a81700 1 --2- 172.21.15.102:0/2686889803 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f054c06e250 0x7f054c070700 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.351+0000 7f0569a81700 1 --2- 172.21.15.102:0/2686889803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05640ff050 0x7f0564195990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.351+0000 7f0569a81700 1 --2- 172.21.15.102:0/2686889803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0564100250 0x7f0564195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.351+0000 7f0569a81700 1 -- 172.21.15.102:0/2686889803 >> 172.21.15.102:0/2686889803 conn(0x7f05640fa5e0 msgr2=0x7f05640683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:42.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.351+0000 7f0569a81700 1 -- 172.21.15.102:0/2686889803 shutdown_connections 2024-03-26T19:03:42.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.351+0000 7f0569a81700 1 -- 172.21.15.102:0/2686889803 wait complete. 2024-03-26T19:03:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.468+0000 7f184753a700 1 -- 172.21.15.102:0/1904162317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18400ff050 msgr2=0x7f18400ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.468+0000 7f184753a700 1 --2- 172.21.15.102:0/1904162317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18400ff050 0x7f18400ff460 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f183c009910 tx=0x7f183c009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.468+0000 7f184753a700 1 -- 172.21.15.102:0/1904162317 shutdown_connections 2024-03-26T19:03:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.468+0000 7f184753a700 1 --2- 172.21.15.102:0/1904162317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1840100250 0x7f18401006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.468+0000 7f184753a700 1 --2- 172.21.15.102:0/1904162317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18400ff050 0x7f18400ff460 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.468+0000 7f184753a700 1 -- 172.21.15.102:0/1904162317 >> 172.21.15.102:0/1904162317 conn(0x7f18400fa5e0 msgr2=0x7f18400fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.468+0000 7f184753a700 1 -- 172.21.15.102:0/1904162317 shutdown_connections 2024-03-26T19:03:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.468+0000 7f184753a700 1 -- 172.21.15.102:0/1904162317 wait complete. 2024-03-26T19:03:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f184753a700 1 Processor -- start 2024-03-26T19:03:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f184753a700 1 -- start start 2024-03-26T19:03:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f184753a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18400ff050 0x7f18401959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f184753a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1840100250 0x7f1840195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f184753a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1840196520 con 0x7f18400ff050 2024-03-26T19:03:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f184753a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1840196660 con 0x7f1840100250 2024-03-26T19:03:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f18452d6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18400ff050 0x7f18401959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f18452d6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18400ff050 0x7f18401959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33936/0 (socket says 172.21.15.102:33936) 2024-03-26T19:03:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f18452d6700 1 -- 172.21.15.102:0/1630345420 learned_addr learned my addr 172.21.15.102:0/1630345420 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f1844ad5700 1 --2- 172.21.15.102:0/1630345420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1840100250 0x7f1840195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f18452d6700 1 -- 172.21.15.102:0/1630345420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1840100250 msgr2=0x7f1840195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f18452d6700 1 --2- 172.21.15.102:0/1630345420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1840100250 0x7f1840195f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.469+0000 7f18452d6700 1 -- 172.21.15.102:0/1630345420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c0095f0 con 0x7f18400ff050 2024-03-26T19:03:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.470+0000 7f18452d6700 1 --2- 172.21.15.102:0/1630345420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18400ff050 0x7f18401959c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f183c00bda0 tx=0x7f183c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.470+0000 7f18367fc700 1 -- 172.21.15.102:0/1630345420 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f183c01d070 con 0x7f18400ff050 2024-03-26T19:03:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.470+0000 7f18367fc700 1 -- 172.21.15.102:0/1630345420 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f183c004c50 con 0x7f18400ff050 2024-03-26T19:03:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.470+0000 7f184753a700 1 -- 172.21.15.102:0/1630345420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184019b0b0 con 0x7f18400ff050 2024-03-26T19:03:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.470+0000 7f18367fc700 1 -- 172.21.15.102:0/1630345420 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f183c00f690 con 0x7f18400ff050 2024-03-26T19:03:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.470+0000 7f184753a700 1 -- 172.21.15.102:0/1630345420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184019b5a0 con 0x7f18400ff050 2024-03-26T19:03:42.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.471+0000 7f184753a700 1 -- 172.21.15.102:0/1630345420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1840107bc0 con 0x7f18400ff050 2024-03-26T19:03:42.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.471+0000 7f18367fc700 1 -- 172.21.15.102:0/1630345420 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f183c00f900 con 0x7f18400ff050 2024-03-26T19:03:42.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.472+0000 7f18367fc700 1 --2- 172.21.15.102:0/1630345420 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f182c06e460 0x7f182c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:42.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.472+0000 7f18367fc700 1 -- 172.21.15.102:0/1630345420 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f183c08fc80 con 0x7f18400ff050 2024-03-26T19:03:42.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.472+0000 7f1844ad5700 1 --2- 172.21.15.102:0/1630345420 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f182c06e460 0x7f182c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:42.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.473+0000 7f1844ad5700 1 --2- 172.21.15.102:0/1630345420 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f182c06e460 0x7f182c070910 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1830005950 tx=0x7f1830005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:42.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.477+0000 7f18367fc700 1 -- 172.21.15.102:0/1630345420 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f183c05c910 con 0x7f18400ff050 2024-03-26T19:03:42.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.699+0000 7f184753a700 1 -- 172.21.15.102:0/1630345420 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f184019bca0 con 0x7f182c06e460 2024-03-26T19:03:42.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.713+0000 7f18367fc700 1 -- 172.21.15.102:0/1630345420 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f184019bca0 con 0x7f182c06e460 2024-03-26T19:03:42.708 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T19:03:42.708 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (15m) 6s ago 16m 25.7M - 0.25.0 c8568f914cd2 43a9c72483fa 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (35s) 6s ago 16m 8061k - 19.0.0-2311-g36051606 497bb87bc5da 2fb0a990cbf1 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (30s) 9s ago 15m 7537k - 19.0.0-2311-g36051606 497bb87bc5da 893c37bef7ca 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (6m) 6s ago 16m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (6m) 9s ago 15m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (15m) 6s ago 16m 83.0M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (87s) 6s ago 11m 15.9M - 19.0.0-2311-g36051606 497bb87bc5da c530ed7dbd78 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (110s) 6s ago 11m 79.3M - 19.0.0-2311-g36051606 497bb87bc5da 63b8127a3849 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (50s) 9s ago 11m 13.7M - 19.0.0-2311-g36051606 497bb87bc5da 25df8033593e 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (68s) 9s ago 11m 15.1M - 19.0.0-2311-g36051606 497bb87bc5da 37260789b9ec 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (8m) 6s ago 18m 510M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (7m) 9s ago 15m 452M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (7m) 6s ago 18m 54.5M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (6m) 9s ago 15m 42.3M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T19:03:42.709 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (16s) 6s ago 16m 7982k - 1.5.0 0da6a335fe13 9992a3ee4208 2024-03-26T19:03:42.710 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (12s) 9s ago 15m 3632k - 1.5.0 0da6a335fe13 6d6186a60a00 2024-03-26T19:03:42.710 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (5m) 6s ago 14m 107M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T19:03:42.710 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (5m) 6s ago 14m 116M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T19:03:42.710 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (4m) 6s ago 14m 98.2M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T19:03:42.710 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (3m) 9s ago 13m 122M 2240M 19.0.0-2311-g36051606 497bb87bc5da 03955331c962 2024-03-26T19:03:42.710 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (3m) 9s ago 13m 112M 2240M 19.0.0-2311-g36051606 497bb87bc5da 23e140cd8961 2024-03-26T19:03:42.710 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (2m) 9s ago 12m 103M 2240M 19.0.0-2311-g36051606 497bb87bc5da 60afb0e4da18 2024-03-26T19:03:42.710 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (7m) 6s ago 16m 83.4M - 2.43.0 a07b618ecd1d ab4805c8e089 2024-03-26T19:03:42.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.715+0000 7f184753a700 1 -- 172.21.15.102:0/1630345420 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f182c06e460 msgr2=0x7f182c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:42.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.715+0000 7f184753a700 1 --2- 172.21.15.102:0/1630345420 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f182c06e460 0x7f182c070910 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1830005950 tx=0x7f1830005cf0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.715+0000 7f184753a700 1 -- 172.21.15.102:0/1630345420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18400ff050 msgr2=0x7f18401959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:42.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.715+0000 7f184753a700 1 --2- 172.21.15.102:0/1630345420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18400ff050 0x7f18401959c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f183c00bda0 tx=0x7f183c00be80 comp rx=0 tx=0).stop 2024-03-26T19:03:42.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.715+0000 7f184753a700 1 -- 172.21.15.102:0/1630345420 shutdown_connections 2024-03-26T19:03:42.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.715+0000 7f184753a700 1 --2- 172.21.15.102:0/1630345420 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f182c06e460 0x7f182c070910 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.715+0000 7f184753a700 1 --2- 172.21.15.102:0/1630345420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18400ff050 0x7f18401959c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.715+0000 7f184753a700 1 --2- 172.21.15.102:0/1630345420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1840100250 0x7f1840195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.715+0000 7f184753a700 1 -- 172.21.15.102:0/1630345420 >> 172.21.15.102:0/1630345420 conn(0x7f18400fa5e0 msgr2=0x7f18400683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:42.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.715+0000 7f184753a700 1 -- 172.21.15.102:0/1630345420 shutdown_connections 2024-03-26T19:03:42.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.715+0000 7f184753a700 1 -- 172.21.15.102:0/1630345420 wait complete. 2024-03-26T19:03:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.848+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/3481280441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac0ff050 msgr2=0x7fd6ac0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.848+0000 7fd6b2c7c700 1 --2- 172.21.15.102:0/3481280441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac0ff050 0x7fd6ac0ff460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009910 tx=0x7fd6a0009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.849+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/3481280441 shutdown_connections 2024-03-26T19:03:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.849+0000 7fd6b2c7c700 1 --2- 172.21.15.102:0/3481280441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6ac100250 0x7fd6ac1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.849+0000 7fd6b2c7c700 1 --2- 172.21.15.102:0/3481280441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac0ff050 0x7fd6ac0ff460 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.849+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/3481280441 >> 172.21.15.102:0/3481280441 conn(0x7fd6ac0fa5e0 msgr2=0x7fd6ac0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:42.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.849+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/3481280441 shutdown_connections 2024-03-26T19:03:42.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.849+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/3481280441 wait complete. 2024-03-26T19:03:42.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.850+0000 7fd6b2c7c700 1 Processor -- start 2024-03-26T19:03:42.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.850+0000 7fd6b2c7c700 1 -- start start 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.850+0000 7fd6b2c7c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6ac0ff050 0x7fd6ac1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.850+0000 7fd6b2c7c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac100250 0x7fd6ac195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.850+0000 7fd6b2c7c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac196470 con 0x7fd6ac100250 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.850+0000 7fd6b2c7c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac1965b0 con 0x7fd6ac0ff050 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.850+0000 7fd6abfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac100250 0x7fd6ac195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.850+0000 7fd6abfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac100250 0x7fd6ac195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33968/0 (socket says 172.21.15.102:33968) 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.850+0000 7fd6abfff700 1 -- 172.21.15.102:0/1636085580 learned_addr learned my addr 172.21.15.102:0/1636085580 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.850+0000 7fd6b0a18700 1 --2- 172.21.15.102:0/1636085580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6ac0ff050 0x7fd6ac1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.851+0000 7fd6abfff700 1 -- 172.21.15.102:0/1636085580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6ac0ff050 msgr2=0x7fd6ac1959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.851+0000 7fd6abfff700 1 --2- 172.21.15.102:0/1636085580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6ac0ff050 0x7fd6ac1959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.851+0000 7fd6abfff700 1 -- 172.21.15.102:0/1636085580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a00095f0 con 0x7fd6ac100250 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.851+0000 7fd6abfff700 1 --2- 172.21.15.102:0/1636085580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac100250 0x7fd6ac195ee0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd69800c930 tx=0x7fd69800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.851+0000 7fd6a9ffb700 1 -- 172.21.15.102:0/1636085580 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd698007ad0 con 0x7fd6ac100250 2024-03-26T19:03:42.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.851+0000 7fd6a9ffb700 1 -- 172.21.15.102:0/1636085580 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd698007c30 con 0x7fd6ac100250 2024-03-26T19:03:42.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.851+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/1636085580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac19b070 con 0x7fd6ac100250 2024-03-26T19:03:42.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.851+0000 7fd6a9ffb700 1 -- 172.21.15.102:0/1636085580 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd698018630 con 0x7fd6ac100250 2024-03-26T19:03:42.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.851+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/1636085580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac19b590 con 0x7fd6ac100250 2024-03-26T19:03:42.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.852+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/1636085580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac107ba0 con 0x7fd6ac100250 2024-03-26T19:03:42.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.853+0000 7fd6a9ffb700 1 -- 172.21.15.102:0/1636085580 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd69800f450 con 0x7fd6ac100250 2024-03-26T19:03:42.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.854+0000 7fd6a9ffb700 1 --2- 172.21.15.102:0/1636085580 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd69c06e460 0x7fd69c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:42.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.854+0000 7fd6a9ffb700 1 -- 172.21.15.102:0/1636085580 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fd69808e690 con 0x7fd6ac100250 2024-03-26T19:03:42.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.854+0000 7fd6b0a18700 1 --2- 172.21.15.102:0/1636085580 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd69c06e460 0x7fd69c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:42.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.854+0000 7fd6b0a18700 1 --2- 172.21.15.102:0/1636085580 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd69c06e460 0x7fd69c070910 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0006010 tx=0x7fd6a000b580 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:42.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:42.859+0000 7fd6a9ffb700 1 -- 172.21.15.102:0/1636085580 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd69805b320 con 0x7fd6ac100250 2024-03-26T19:03:43.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.215+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/1636085580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd6ac19bc30 con 0x7fd6ac100250 2024-03-26T19:03:43.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.215+0000 7fd6a9ffb700 1 -- 172.21.15.102:0/1636085580 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fd69805aeb0 con 0x7fd6ac100250 2024-03-26T19:03:43.211 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:03:43.211 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T19:03:43.211 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:03:43.211 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 6 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 4 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 14 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T19:03:43.212 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:03:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.218+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/1636085580 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd69c06e460 msgr2=0x7fd69c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.218+0000 7fd6b2c7c700 1 --2- 172.21.15.102:0/1636085580 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd69c06e460 0x7fd69c070910 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0006010 tx=0x7fd6a000b580 comp rx=0 tx=0).stop 2024-03-26T19:03:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.218+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/1636085580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac100250 msgr2=0x7fd6ac195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.218+0000 7fd6b2c7c700 1 --2- 172.21.15.102:0/1636085580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac100250 0x7fd6ac195ee0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd69800c930 tx=0x7fd69800ccf0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.218+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/1636085580 shutdown_connections 2024-03-26T19:03:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.218+0000 7fd6b2c7c700 1 --2- 172.21.15.102:0/1636085580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6ac0ff050 0x7fd6ac1959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.218+0000 7fd6b2c7c700 1 --2- 172.21.15.102:0/1636085580 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd69c06e460 0x7fd69c070910 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.218+0000 7fd6b2c7c700 1 --2- 172.21.15.102:0/1636085580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac100250 0x7fd6ac195ee0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.218+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/1636085580 >> 172.21.15.102:0/1636085580 conn(0x7fd6ac0fa5e0 msgr2=0x7fd6ac0683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.218+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/1636085580 shutdown_connections 2024-03-26T19:03:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.218+0000 7fd6b2c7c700 1 -- 172.21.15.102:0/1636085580 wait complete. 2024-03-26T19:03:43.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:43 smithi190 ceph-mon[65689]: pgmap v249: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:43.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:43 smithi190 ceph-mon[65689]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:43.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:43 smithi190 ceph-mon[65689]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:43.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.341+0000 7f9ca81b3700 1 -- 172.21.15.102:0/3541475653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0100be0 msgr2=0x7f9ca0102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:43.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.341+0000 7f9ca81b3700 1 --2- 172.21.15.102:0/3541475653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0100be0 0x7f9ca0102fc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009910 tx=0x7f9c9c009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:43.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.341+0000 7f9ca81b3700 1 -- 172.21.15.102:0/3541475653 shutdown_connections 2024-03-26T19:03:43.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.341+0000 7f9ca81b3700 1 --2- 172.21.15.102:0/3541475653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca0103500 0x7f9ca01058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.341+0000 7f9ca81b3700 1 --2- 172.21.15.102:0/3541475653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0100be0 0x7f9ca0102fc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.341+0000 7f9ca81b3700 1 -- 172.21.15.102:0/3541475653 >> 172.21.15.102:0/3541475653 conn(0x7f9ca00fa5e0 msgr2=0x7f9ca00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:43.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.341+0000 7f9ca81b3700 1 -- 172.21.15.102:0/3541475653 shutdown_connections 2024-03-26T19:03:43.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.341+0000 7f9ca81b3700 1 -- 172.21.15.102:0/3541475653 wait complete. 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca81b3700 1 Processor -- start 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca81b3700 1 -- start start 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca81b3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0100be0 0x7f9ca0100540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca81b3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca0103500 0x7f9ca00feb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca81b3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00ff0d0 con 0x7f9ca0100be0 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca81b3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00ff210 con 0x7f9ca0103500 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca5f4f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0100be0 0x7f9ca0100540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca5f4f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0100be0 0x7f9ca0100540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33996/0 (socket says 172.21.15.102:33996) 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca5f4f700 1 -- 172.21.15.102:0/542145534 learned_addr learned my addr 172.21.15.102:0/542145534 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca574e700 1 --2- 172.21.15.102:0/542145534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca0103500 0x7f9ca00feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca5f4f700 1 -- 172.21.15.102:0/542145534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca0103500 msgr2=0x7f9ca00feb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca5f4f700 1 --2- 172.21.15.102:0/542145534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca0103500 0x7f9ca00feb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.342+0000 7f9ca5f4f700 1 -- 172.21.15.102:0/542145534 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c0095f0 con 0x7f9ca0100be0 2024-03-26T19:03:43.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.343+0000 7f9ca5f4f700 1 --2- 172.21.15.102:0/542145534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0100be0 0x7f9ca0100540 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c006010 tx=0x7f9c9c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:43.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.343+0000 7f9c96ffd700 1 -- 172.21.15.102:0/542145534 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c9c01d070 con 0x7f9ca0100be0 2024-03-26T19:03:43.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.343+0000 7f9c96ffd700 1 -- 172.21.15.102:0/542145534 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9c9c00f460 con 0x7f9ca0100be0 2024-03-26T19:03:43.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.343+0000 7f9ca81b3700 1 -- 172.21.15.102:0/542145534 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca00ff490 con 0x7f9ca0100be0 2024-03-26T19:03:43.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.343+0000 7f9c96ffd700 1 -- 172.21.15.102:0/542145534 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c9c021640 con 0x7f9ca0100be0 2024-03-26T19:03:43.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.343+0000 7f9ca81b3700 1 -- 172.21.15.102:0/542145534 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca00ff980 con 0x7f9ca0100be0 2024-03-26T19:03:43.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.344+0000 7f9ca81b3700 1 -- 172.21.15.102:0/542145534 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca018fb70 con 0x7f9ca0100be0 2024-03-26T19:03:43.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.345+0000 7f9c96ffd700 1 -- 172.21.15.102:0/542145534 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9c9c00fab0 con 0x7f9ca0100be0 2024-03-26T19:03:43.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.345+0000 7f9c96ffd700 1 --2- 172.21.15.102:0/542145534 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c8c06e460 0x7f9c8c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:43.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.346+0000 7f9c96ffd700 1 -- 172.21.15.102:0/542145534 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f9c9c08f6a0 con 0x7f9ca0100be0 2024-03-26T19:03:43.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.346+0000 7f9ca574e700 1 --2- 172.21.15.102:0/542145534 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c8c06e460 0x7f9c8c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:43.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.346+0000 7f9ca574e700 1 --2- 172.21.15.102:0/542145534 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c8c06e460 0x7f9c8c070910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9c90005950 tx=0x7f9c9000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:43.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.351+0000 7f9c96ffd700 1 -- 172.21.15.102:0/542145534 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9c9c05c3b0 con 0x7f9ca0100be0 2024-03-26T19:03:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:43 smithi102 ceph-mon[78322]: pgmap v249: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:43 smithi102 ceph-mon[78322]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:43.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:43 smithi102 ceph-mon[78322]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:43.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.774+0000 7f9ca81b3700 1 -- 172.21.15.102:0/542145534 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9ca002cc30 con 0x7f9ca0100be0 2024-03-26T19:03:43.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.775+0000 7f9c96ffd700 1 -- 172.21.15.102:0/542145534 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 43 v43) v1 ==== 76+0+1955 (secure 0 0 0) 0x7f9c9c05bf40 con 0x7f9ca0100be0 2024-03-26T19:03:43.771 INFO:teuthology.orchestra.run.smithi102.stdout:e43 2024-03-26T19:03:43.771 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T19:03:43.771 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:03:43.771 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T19:03:43.771 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:03:43.771 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T19:03:43.771 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T19:03:43.771 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 43 2024-03-26T19:03:43.771 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T19:03:02.920923+0000 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 84 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T19:03:43.772 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=34436,1=34468} 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 34436 members: 34436,34468 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{0:34436} state up:active seq 19 join_fscid=1 addr [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{1:34468} state up:active seq 13 join_fscid=1 addr [v2:172.21.15.102:6828/3007238267,v1:172.21.15.102:6829/3007238267] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:03:43.773 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:03:43.774 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:03:43.774 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T19:03:43.774 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:03:43.774 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.190:6824/782636706,v1:172.21.15.190:6825/782636706] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:03:43.774 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:44373} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.190:6826/925215849,v1:172.21.15.190:6827/925215849] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.777+0000 7f9ca81b3700 1 -- 172.21.15.102:0/542145534 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c8c06e460 msgr2=0x7f9c8c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.777+0000 7f9ca81b3700 1 --2- 172.21.15.102:0/542145534 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c8c06e460 0x7f9c8c070910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9c90005950 tx=0x7f9c9000b410 comp rx=0 tx=0).stop 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.777+0000 7f9ca81b3700 1 -- 172.21.15.102:0/542145534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0100be0 msgr2=0x7f9ca0100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.777+0000 7f9ca81b3700 1 --2- 172.21.15.102:0/542145534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0100be0 0x7f9ca0100540 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c006010 tx=0x7f9c9c00bd70 comp rx=0 tx=0).stop 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.777+0000 7f9ca81b3700 1 -- 172.21.15.102:0/542145534 shutdown_connections 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.777+0000 7f9ca81b3700 1 --2- 172.21.15.102:0/542145534 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f9c8c06e460 0x7f9c8c070910 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.777+0000 7f9ca81b3700 1 --2- 172.21.15.102:0/542145534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0100be0 0x7f9ca0100540 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.777+0000 7f9ca81b3700 1 --2- 172.21.15.102:0/542145534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca0103500 0x7f9ca00feb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.777+0000 7f9ca81b3700 1 -- 172.21.15.102:0/542145534 >> 172.21.15.102:0/542145534 conn(0x7f9ca00fa5e0 msgr2=0x7f9ca00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.777+0000 7f9ca81b3700 1 -- 172.21.15.102:0/542145534 shutdown_connections 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.777+0000 7f9ca81b3700 1 -- 172.21.15.102:0/542145534 wait complete. 2024-03-26T19:03:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 43 2024-03-26T19:03:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.891+0000 7f1d60008700 1 -- 172.21.15.102:0/1537726324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d58100150 msgr2=0x7f1d58100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.891+0000 7f1d60008700 1 --2- 172.21.15.102:0/1537726324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d58100150 0x7f1d58100560 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009910 tx=0x7f1d50009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.891+0000 7f1d60008700 1 -- 172.21.15.102:0/1537726324 shutdown_connections 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.891+0000 7f1d60008700 1 --2- 172.21.15.102:0/1537726324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d58101350 0x7f1d581017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.891+0000 7f1d60008700 1 --2- 172.21.15.102:0/1537726324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d58100150 0x7f1d58100560 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.891+0000 7f1d60008700 1 -- 172.21.15.102:0/1537726324 >> 172.21.15.102:0/1537726324 conn(0x7f1d580fb6e0 msgr2=0x7f1d580fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.891+0000 7f1d60008700 1 -- 172.21.15.102:0/1537726324 shutdown_connections 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.891+0000 7f1d60008700 1 -- 172.21.15.102:0/1537726324 wait complete. 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d60008700 1 Processor -- start 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d60008700 1 -- start start 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d60008700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d58100150 0x7f1d581959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d60008700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d58101350 0x7f1d58195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d60008700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d58196500 con 0x7f1d58100150 2024-03-26T19:03:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d60008700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d58196640 con 0x7f1d58101350 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d5dda4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d58100150 0x7f1d581959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d5dda4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d58100150 0x7f1d581959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34010/0 (socket says 172.21.15.102:34010) 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d5dda4700 1 -- 172.21.15.102:0/2759616110 learned_addr learned my addr 172.21.15.102:0/2759616110 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d5d5a3700 1 --2- 172.21.15.102:0/2759616110 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d58101350 0x7f1d58195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d5dda4700 1 -- 172.21.15.102:0/2759616110 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d58101350 msgr2=0x7f1d58195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d5dda4700 1 --2- 172.21.15.102:0/2759616110 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d58101350 0x7f1d58195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d5dda4700 1 -- 172.21.15.102:0/2759616110 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d500095f0 con 0x7f1d58100150 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.892+0000 7f1d5dda4700 1 --2- 172.21.15.102:0/2759616110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d58100150 0x7f1d581959a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1d50000c00 tx=0x7f1d5000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.893+0000 7f1d4effd700 1 -- 172.21.15.102:0/2759616110 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d5001d070 con 0x7f1d58100150 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.893+0000 7f1d4effd700 1 -- 172.21.15.102:0/2759616110 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1d50005dc0 con 0x7f1d58100150 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.893+0000 7f1d60008700 1 -- 172.21.15.102:0/2759616110 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5819b090 con 0x7f1d58100150 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.893+0000 7f1d60008700 1 -- 172.21.15.102:0/2759616110 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5819b580 con 0x7f1d58100150 2024-03-26T19:03:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.893+0000 7f1d4effd700 1 -- 172.21.15.102:0/2759616110 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d50022470 con 0x7f1d58100150 2024-03-26T19:03:43.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.894+0000 7f1d60008700 1 -- 172.21.15.102:0/2759616110 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d58108ca0 con 0x7f1d58100150 2024-03-26T19:03:43.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.896+0000 7f1d4effd700 1 -- 172.21.15.102:0/2759616110 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f1d50003810 con 0x7f1d58100150 2024-03-26T19:03:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.896+0000 7f1d4effd700 1 --2- 172.21.15.102:0/2759616110 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1d4406e460 0x7f1d44070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.896+0000 7f1d4effd700 1 -- 172.21.15.102:0/2759616110 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f1d5008fa70 con 0x7f1d58100150 2024-03-26T19:03:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.896+0000 7f1d5d5a3700 1 --2- 172.21.15.102:0/2759616110 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1d4406e460 0x7f1d44070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.897+0000 7f1d5d5a3700 1 --2- 172.21.15.102:0/2759616110 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1d4406e460 0x7f1d44070910 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009c60 tx=0x7f1d48009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:43.899+0000 7f1d4effd700 1 -- 172.21.15.102:0/2759616110 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d5005c780 con 0x7f1d58100150 2024-03-26T19:03:44.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.174+0000 7f1d60008700 1 -- 172.21.15.102:0/2759616110 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d58105ca0 con 0x7f1d4406e460 2024-03-26T19:03:44.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.176+0000 7f1d4effd700 1 -- 172.21.15.102:0/2759616110 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+463 (secure 0 0 0) 0x7f1d58105ca0 con 0x7f1d4406e460 2024-03-26T19:03:44.171 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "mds", 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "osd", 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "mon", 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph-exporter" 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "18/23 daemons upgraded", 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading prometheus daemons", 2024-03-26T19:03:44.172 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T19:03:44.173 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:03:44.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.179+0000 7f1d60008700 1 -- 172.21.15.102:0/2759616110 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1d4406e460 msgr2=0x7f1d44070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:44.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.179+0000 7f1d60008700 1 --2- 172.21.15.102:0/2759616110 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1d4406e460 0x7f1d44070910 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009c60 tx=0x7f1d48009260 comp rx=0 tx=0).stop 2024-03-26T19:03:44.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.179+0000 7f1d60008700 1 -- 172.21.15.102:0/2759616110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d58100150 msgr2=0x7f1d581959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:44.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.179+0000 7f1d60008700 1 --2- 172.21.15.102:0/2759616110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d58100150 0x7f1d581959a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1d50000c00 tx=0x7f1d5000f690 comp rx=0 tx=0).stop 2024-03-26T19:03:44.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.179+0000 7f1d60008700 1 -- 172.21.15.102:0/2759616110 shutdown_connections 2024-03-26T19:03:44.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.179+0000 7f1d60008700 1 --2- 172.21.15.102:0/2759616110 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f1d4406e460 0x7f1d44070910 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:44.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.179+0000 7f1d60008700 1 --2- 172.21.15.102:0/2759616110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d58100150 0x7f1d581959a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:44.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.179+0000 7f1d60008700 1 --2- 172.21.15.102:0/2759616110 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d58101350 0x7f1d58195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:44.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.179+0000 7f1d60008700 1 -- 172.21.15.102:0/2759616110 >> 172.21.15.102:0/2759616110 conn(0x7f1d580fb6e0 msgr2=0x7f1d58104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:44.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.179+0000 7f1d60008700 1 -- 172.21.15.102:0/2759616110 shutdown_connections 2024-03-26T19:03:44.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.179+0000 7f1d60008700 1 -- 172.21.15.102:0/2759616110 wait complete. 2024-03-26T19:03:44.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.286+0000 7f48d3d05700 1 -- 172.21.15.102:0/1749982701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48cc100250 msgr2=0x7f48cc1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.286+0000 7f48d3d05700 1 --2- 172.21.15.102:0/1749982701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48cc100250 0x7f48cc1006a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009910 tx=0x7f48bc009c20 comp rx=0 tx=0).stop 2024-03-26T19:03:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.286+0000 7f48d3d05700 1 -- 172.21.15.102:0/1749982701 shutdown_connections 2024-03-26T19:03:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.286+0000 7f48d3d05700 1 --2- 172.21.15.102:0/1749982701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48cc100250 0x7f48cc1006a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.286+0000 7f48d3d05700 1 --2- 172.21.15.102:0/1749982701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc0ff050 0x7f48cc0ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.286+0000 7f48d3d05700 1 -- 172.21.15.102:0/1749982701 >> 172.21.15.102:0/1749982701 conn(0x7f48cc0fa600 msgr2=0x7f48cc0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.286+0000 7f48d3d05700 1 -- 172.21.15.102:0/1749982701 shutdown_connections 2024-03-26T19:03:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.286+0000 7f48d3d05700 1 -- 172.21.15.102:0/1749982701 wait complete. 2024-03-26T19:03:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.287+0000 7f48d3d05700 1 Processor -- start 2024-03-26T19:03:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.287+0000 7f48d3d05700 1 -- start start 2024-03-26T19:03:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.287+0000 7f48d3d05700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc0ff050 0x7f48cc10cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.287+0000 7f48d3d05700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48cc100250 0x7f48cc10d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.287+0000 7f48d3d05700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc109860 con 0x7f48cc100250 2024-03-26T19:03:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.287+0000 7f48d3d05700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc1099d0 con 0x7f48cc0ff050 2024-03-26T19:03:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.287+0000 7f48d12a0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48cc100250 0x7f48cc10d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.287+0000 7f48d12a0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48cc100250 0x7f48cc10d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34024/0 (socket says 172.21.15.102:34024) 2024-03-26T19:03:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.287+0000 7f48d12a0700 1 -- 172.21.15.102:0/720700324 learned_addr learned my addr 172.21.15.102:0/720700324 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:03:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.287+0000 7f48d1aa1700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc0ff050 0x7f48cc10cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.288+0000 7f48d1aa1700 1 -- 172.21.15.102:0/720700324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48cc100250 msgr2=0x7f48cc10d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.288+0000 7f48d1aa1700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48cc100250 0x7f48cc10d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.288+0000 7f48d1aa1700 1 -- 172.21.15.102:0/720700324 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c8009520 con 0x7f48cc0ff050 2024-03-26T19:03:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.288+0000 7f48d12a0700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48cc100250 0x7f48cc10d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T19:03:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.288+0000 7f48d1aa1700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc0ff050 0x7f48cc10cd70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f48c800e940 tx=0x7f48c800ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.288+0000 7f48c2ffd700 1 -- 172.21.15.102:0/720700324 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48c800cb40 con 0x7f48cc0ff050 2024-03-26T19:03:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.288+0000 7f48c2ffd700 1 -- 172.21.15.102:0/720700324 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f48c800ee90 con 0x7f48cc0ff050 2024-03-26T19:03:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.288+0000 7f48d3d05700 1 -- 172.21.15.102:0/720700324 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc0095f0 con 0x7f48cc0ff050 2024-03-26T19:03:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.288+0000 7f48d3d05700 1 -- 172.21.15.102:0/720700324 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48cc10a070 con 0x7f48cc0ff050 2024-03-26T19:03:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.288+0000 7f48c2ffd700 1 -- 172.21.15.102:0/720700324 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48c8018650 con 0x7f48cc0ff050 2024-03-26T19:03:44.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.290+0000 7f48d3d05700 1 -- 172.21.15.102:0/720700324 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b0005320 con 0x7f48cc0ff050 2024-03-26T19:03:44.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.290+0000 7f48c2ffd700 1 -- 172.21.15.102:0/720700324 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f48c8018840 con 0x7f48cc0ff050 2024-03-26T19:03:44.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.290+0000 7f48c2ffd700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f48b806e4b0 0x7f48b8070960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:03:44.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.290+0000 7f48c2ffd700 1 -- 172.21.15.102:0/720700324 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f48c8014070 con 0x7f48cc0ff050 2024-03-26T19:03:44.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.290+0000 7f48d12a0700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f48b806e4b0 0x7f48b8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:03:44.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.291+0000 7f48d12a0700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f48b806e4b0 0x7f48b8070960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f48bc00b6f0 tx=0x7f48bc009f70 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:03:44.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.295+0000 7f48c2ffd700 1 -- 172.21.15.102:0/720700324 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f48c805c480 con 0x7f48cc0ff050 2024-03-26T19:03:44.389 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:44 smithi102 ceph-mon[78322]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:44.389 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:44 smithi102 ceph-mon[78322]: Upgrade: Updating prometheus.smithi102 2024-03-26T19:03:44.389 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:44 smithi102 ceph-mon[78322]: Deploying daemon prometheus.smithi102 on smithi102 2024-03-26T19:03:44.389 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:44 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1636085580' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:44.389 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:44 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/542145534' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:03:44.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:44 smithi190 ceph-mon[65689]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:44.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:44 smithi190 ceph-mon[65689]: Upgrade: Updating prometheus.smithi102 2024-03-26T19:03:44.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:44 smithi190 ceph-mon[65689]: Deploying daemon prometheus.smithi102 on smithi102 2024-03-26T19:03:44.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:44 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1636085580' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:44.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:44 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/542145534' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:03:44.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.604+0000 7f48d3d05700 1 -- 172.21.15.102:0/720700324 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f48b0005190 con 0x7f48cc0ff050 2024-03-26T19:03:44.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.604+0000 7f48c2ffd700 1 -- 172.21.15.102:0/720700324 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f48c805c010 con 0x7f48cc0ff050 2024-03-26T19:03:44.600 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T19:03:44.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.606+0000 7f48d3d05700 1 -- 172.21.15.102:0/720700324 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f48b806e4b0 msgr2=0x7f48b8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:44.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.606+0000 7f48d3d05700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f48b806e4b0 0x7f48b8070960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f48bc00b6f0 tx=0x7f48bc009f70 comp rx=0 tx=0).stop 2024-03-26T19:03:44.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.606+0000 7f48d3d05700 1 -- 172.21.15.102:0/720700324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc0ff050 msgr2=0x7f48cc10cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:03:44.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.606+0000 7f48d3d05700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc0ff050 0x7f48cc10cd70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f48c800e940 tx=0x7f48c800ec50 comp rx=0 tx=0).stop 2024-03-26T19:03:44.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.606+0000 7f48d3d05700 1 -- 172.21.15.102:0/720700324 shutdown_connections 2024-03-26T19:03:44.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.606+0000 7f48d3d05700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc0ff050 0x7f48cc10cd70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:44.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.606+0000 7f48d3d05700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f48b806e4b0 0x7f48b8070960 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:44.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.606+0000 7f48d3d05700 1 --2- 172.21.15.102:0/720700324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48cc100250 0x7f48cc10d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:03:44.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.606+0000 7f48d3d05700 1 -- 172.21.15.102:0/720700324 >> 172.21.15.102:0/720700324 conn(0x7f48cc0fa600 msgr2=0x7f48cc0fb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:03:44.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.606+0000 7f48d3d05700 1 -- 172.21.15.102:0/720700324 shutdown_connections 2024-03-26T19:03:44.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:03:44.606+0000 7f48d3d05700 1 -- 172.21.15.102:0/720700324 wait complete. 2024-03-26T19:03:45.249 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:45 smithi102 ceph-mon[78322]: pgmap v250: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:45.249 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:45 smithi102 ceph-mon[78322]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:45.249 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:45 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/720700324' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:03:45.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:45 smithi190 ceph-mon[65689]: pgmap v250: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:45.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:45 smithi190 ceph-mon[65689]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:03:45.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:45 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/720700324' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:03:47.430 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:47 smithi102 ceph-mon[78322]: pgmap v251: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:47.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:47 smithi190 ceph-mon[65689]: pgmap v251: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:49.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:49 smithi190 ceph-mon[65689]: pgmap v252: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:49.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:49 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:49.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:49 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:49.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:49 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:49.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:49 smithi102 ceph-mon[78322]: pgmap v252: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:49.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:49 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:49.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:49 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:49.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:49 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:51.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:51 smithi190 ceph-mon[65689]: pgmap v253: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:51.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:51 smithi102 ceph-mon[78322]: pgmap v253: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:53.282 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:53 smithi190 ceph-mon[65689]: pgmap v254: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:53.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:03:53.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:53 smithi102 ceph-mon[78322]: pgmap v254: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:53.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:03:54.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:54 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:54.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:54 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:54.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:54 smithi102 ceph-mon[78322]: pgmap v255: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:54.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:54 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:54.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:54 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:54.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:54 smithi190 ceph-mon[65689]: pgmap v255: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:55.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:55 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:55.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:55 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:56.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:55 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:56.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:55 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:56.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:56 smithi102 ceph-mon[78322]: pgmap v256: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:57.032 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:56 smithi190 ceph-mon[65689]: pgmap v256: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:59.233 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: pgmap v257: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:59.233 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:59.233 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:59.233 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:59.233 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:03:59.233 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:03:59 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: pgmap v257: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:03:59.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:59.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:59.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:03:59.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:03:59.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:03:59.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T19:03:59.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T19:03:59.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:03:59.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:03:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:03:59 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:01.467 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:01 smithi102 ceph-mon[78322]: pgmap v258: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:01.467 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:01 smithi102 ceph-mon[78322]: Upgrade: Updating alertmanager.smithi102 2024-03-26T19:04:01.467 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:01 smithi102 ceph-mon[78322]: Deploying daemon alertmanager.smithi102 on smithi102 2024-03-26T19:04:01.532 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:01 smithi190 ceph-mon[65689]: pgmap v258: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:01.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:01 smithi190 ceph-mon[65689]: Upgrade: Updating alertmanager.smithi102 2024-03-26T19:04:01.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:01 smithi190 ceph-mon[65689]: Deploying daemon alertmanager.smithi102 on smithi102 2024-03-26T19:04:03.430 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:03 smithi102 ceph-mon[78322]: pgmap v259: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:03.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:03 smithi190 ceph-mon[65689]: pgmap v259: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:05.261 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:05 smithi102 ceph-mon[78322]: pgmap v260: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:05.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:05 smithi190 ceph-mon[65689]: pgmap v260: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:06.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:06 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:06.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:06 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:06.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:06 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:06.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:06 smithi102 ceph-mon[78322]: pgmap v261: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:06.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:06 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:06.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:06 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:06.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:06 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:06.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:06 smithi190 ceph-mon[65689]: pgmap v261: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:07.782 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:07 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:04:07.878 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:07 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:04:08.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:08 smithi190 ceph-mon[65689]: pgmap v262: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:08.837 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:08 smithi102 ceph-mon[78322]: pgmap v262: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:11.378 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:11 smithi102 ceph-mon[78322]: pgmap v263: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:11.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:11 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:11.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:11 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:11.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:11 smithi190 ceph-mon[65689]: pgmap v263: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:11.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:11 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:11.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:11 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:12.309 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:12 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:12.310 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:12 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:12.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:12 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:12.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:12 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:13.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:13 smithi190 ceph-mon[65689]: pgmap v264: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:13.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:13 smithi102 ceph-mon[78322]: pgmap v264: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:14.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.720+0000 7f3f3757b700 1 -- 172.21.15.102:0/537380675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f30100160 msgr2=0x7f3f30100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.720+0000 7f3f3757b700 1 --2- 172.21.15.102:0/537380675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f30100160 0x7f3f30100570 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009910 tx=0x7f3f2c009c20 comp rx=0 tx=0).stop 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.720+0000 7f3f3757b700 1 -- 172.21.15.102:0/537380675 shutdown_connections 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.720+0000 7f3f3757b700 1 --2- 172.21.15.102:0/537380675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f30101360 0x7f3f301017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.720+0000 7f3f3757b700 1 --2- 172.21.15.102:0/537380675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f30100160 0x7f3f30100570 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.720+0000 7f3f3757b700 1 -- 172.21.15.102:0/537380675 >> 172.21.15.102:0/537380675 conn(0x7f3f300fb6f0 msgr2=0x7f3f300fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.720+0000 7f3f3757b700 1 -- 172.21.15.102:0/537380675 shutdown_connections 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.720+0000 7f3f3757b700 1 -- 172.21.15.102:0/537380675 wait complete. 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.721+0000 7f3f3757b700 1 Processor -- start 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.721+0000 7f3f3757b700 1 -- start start 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.721+0000 7f3f3757b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f30100160 0x7f3f30195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.721+0000 7f3f3757b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f30101360 0x7f3f30195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.721+0000 7f3f3757b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f301964a0 con 0x7f3f30101360 2024-03-26T19:04:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.721+0000 7f3f3757b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f301965e0 con 0x7f3f30100160 2024-03-26T19:04:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.721+0000 7f3f34b16700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f30101360 0x7f3f30195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.721+0000 7f3f34b16700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f30101360 0x7f3f30195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42172/0 (socket says 172.21.15.102:42172) 2024-03-26T19:04:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.721+0000 7f3f34b16700 1 -- 172.21.15.102:0/540930196 learned_addr learned my addr 172.21.15.102:0/540930196 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:04:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.721+0000 7f3f35317700 1 --2- 172.21.15.102:0/540930196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f30100160 0x7f3f30195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.722+0000 7f3f34b16700 1 -- 172.21.15.102:0/540930196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f30100160 msgr2=0x7f3f30195940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.722+0000 7f3f34b16700 1 --2- 172.21.15.102:0/540930196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f30100160 0x7f3f30195940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.722+0000 7f3f34b16700 1 -- 172.21.15.102:0/540930196 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c0095f0 con 0x7f3f30101360 2024-03-26T19:04:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.722+0000 7f3f34b16700 1 --2- 172.21.15.102:0/540930196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f30101360 0x7f3f30195e80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3f2000e920 tx=0x7f3f2000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.722+0000 7f3f267fc700 1 -- 172.21.15.102:0/540930196 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f2000c9d0 con 0x7f3f30101360 2024-03-26T19:04:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.722+0000 7f3f3757b700 1 -- 172.21.15.102:0/540930196 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3019b090 con 0x7f3f30101360 2024-03-26T19:04:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.722+0000 7f3f267fc700 1 -- 172.21.15.102:0/540930196 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3f20010430 con 0x7f3f30101360 2024-03-26T19:04:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.722+0000 7f3f267fc700 1 -- 172.21.15.102:0/540930196 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f20018650 con 0x7f3f30101360 2024-03-26T19:04:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.722+0000 7f3f3757b700 1 -- 172.21.15.102:0/540930196 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3019b5e0 con 0x7f3f30101360 2024-03-26T19:04:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.723+0000 7f3f3757b700 1 -- 172.21.15.102:0/540930196 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f30108cb0 con 0x7f3f30101360 2024-03-26T19:04:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.724+0000 7f3f267fc700 1 -- 172.21.15.102:0/540930196 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3f200105a0 con 0x7f3f30101360 2024-03-26T19:04:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.724+0000 7f3f267fc700 1 --2- 172.21.15.102:0/540930196 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f1c06e460 0x7f3f1c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.724+0000 7f3f267fc700 1 -- 172.21.15.102:0/540930196 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f3f20014070 con 0x7f3f30101360 2024-03-26T19:04:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.724+0000 7f3f35317700 1 --2- 172.21.15.102:0/540930196 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f1c06e460 0x7f3f1c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.725+0000 7f3f35317700 1 --2- 172.21.15.102:0/540930196 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f1c06e460 0x7f3f1c070910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009fd0 tx=0x7f3f2c005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:14.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.729+0000 7f3f267fc700 1 -- 172.21.15.102:0/540930196 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f20053740 con 0x7f3f30101360 2024-03-26T19:04:14.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:14.998+0000 7f3f3757b700 1 -- 172.21.15.102:0/540930196 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f30105cb0 con 0x7f3f1c06e460 2024-03-26T19:04:14.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.000+0000 7f3f267fc700 1 -- 172.21.15.102:0/540930196 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+465 (secure 0 0 0) 0x7f3f30105cb0 con 0x7f3f1c06e460 2024-03-26T19:04:14.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.003+0000 7f3f3757b700 1 -- 172.21.15.102:0/540930196 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f1c06e460 msgr2=0x7f3f1c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:14.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.003+0000 7f3f3757b700 1 --2- 172.21.15.102:0/540930196 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f1c06e460 0x7f3f1c070910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009fd0 tx=0x7f3f2c005a90 comp rx=0 tx=0).stop 2024-03-26T19:04:14.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.003+0000 7f3f3757b700 1 -- 172.21.15.102:0/540930196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f30101360 msgr2=0x7f3f30195e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:14.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.003+0000 7f3f3757b700 1 --2- 172.21.15.102:0/540930196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f30101360 0x7f3f30195e80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3f2000e920 tx=0x7f3f2000ec30 comp rx=0 tx=0).stop 2024-03-26T19:04:14.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.003+0000 7f3f3757b700 1 -- 172.21.15.102:0/540930196 shutdown_connections 2024-03-26T19:04:14.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.003+0000 7f3f3757b700 1 --2- 172.21.15.102:0/540930196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f30100160 0x7f3f30195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:14.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.003+0000 7f3f3757b700 1 --2- 172.21.15.102:0/540930196 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3f1c06e460 0x7f3f1c070910 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:14.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.003+0000 7f3f3757b700 1 --2- 172.21.15.102:0/540930196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f30101360 0x7f3f30195e80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.003+0000 7f3f3757b700 1 -- 172.21.15.102:0/540930196 >> 172.21.15.102:0/540930196 conn(0x7f3f300fb6f0 msgr2=0x7f3f30104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:15.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.003+0000 7f3f3757b700 1 -- 172.21.15.102:0/540930196 shutdown_connections 2024-03-26T19:04:15.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.003+0000 7f3f3757b700 1 -- 172.21.15.102:0/540930196 wait complete. 2024-03-26T19:04:15.014 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T19:04:15.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.154+0000 7fe32c2d3700 1 -- 172.21.15.102:0/318066154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe324103560 msgr2=0x7fe324105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:15.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.154+0000 7fe32c2d3700 1 --2- 172.21.15.102:0/318066154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe324103560 0x7fe324105940 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe314009910 tx=0x7fe314009c20 comp rx=0 tx=0).stop 2024-03-26T19:04:15.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.154+0000 7fe32c2d3700 1 -- 172.21.15.102:0/318066154 shutdown_connections 2024-03-26T19:04:15.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.154+0000 7fe32c2d3700 1 --2- 172.21.15.102:0/318066154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe324103560 0x7fe324105940 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.154+0000 7fe32c2d3700 1 --2- 172.21.15.102:0/318066154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe324100c40 0x7fe324103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.155+0000 7fe32c2d3700 1 -- 172.21.15.102:0/318066154 >> 172.21.15.102:0/318066154 conn(0x7fe3240fa600 msgr2=0x7fe3240fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:15.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.155+0000 7fe32c2d3700 1 -- 172.21.15.102:0/318066154 shutdown_connections 2024-03-26T19:04:15.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.155+0000 7fe32c2d3700 1 -- 172.21.15.102:0/318066154 wait complete. 2024-03-26T19:04:15.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.155+0000 7fe32c2d3700 1 Processor -- start 2024-03-26T19:04:15.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.155+0000 7fe32c2d3700 1 -- start start 2024-03-26T19:04:15.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32c2d3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe324100c40 0x7fe324195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:15.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32c2d3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe324103560 0x7fe324195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32c2d3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3241964b0 con 0x7fe324103560 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32c2d3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3241965f0 con 0x7fe324100c40 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32986e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe324103560 0x7fe324195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32a06f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe324100c40 0x7fe324195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32986e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe324103560 0x7fe324195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42190/0 (socket says 172.21.15.102:42190) 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32986e700 1 -- 172.21.15.102:0/2708609139 learned_addr learned my addr 172.21.15.102:0/2708609139 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32986e700 1 -- 172.21.15.102:0/2708609139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe324100c40 msgr2=0x7fe324195950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32986e700 1 --2- 172.21.15.102:0/2708609139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe324100c40 0x7fe324195950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32986e700 1 -- 172.21.15.102:0/2708609139 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3140095f0 con 0x7fe324103560 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32a06f700 1 --2- 172.21.15.102:0/2708609139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe324100c40 0x7fe324195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe32986e700 1 --2- 172.21.15.102:0/2708609139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe324103560 0x7fe324195e90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe314000c00 tx=0x7fe31400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe31b7fe700 1 -- 172.21.15.102:0/2708609139 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe31401d070 con 0x7fe324103560 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.156+0000 7fe31b7fe700 1 -- 172.21.15.102:0/2708609139 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe314005dc0 con 0x7fe324103560 2024-03-26T19:04:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.157+0000 7fe31b7fe700 1 -- 172.21.15.102:0/2708609139 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3140226a0 con 0x7fe324103560 2024-03-26T19:04:15.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.157+0000 7fe32c2d3700 1 -- 172.21.15.102:0/2708609139 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32419b040 con 0x7fe324103560 2024-03-26T19:04:15.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.157+0000 7fe32c2d3700 1 -- 172.21.15.102:0/2708609139 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32419b530 con 0x7fe324103560 2024-03-26T19:04:15.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.158+0000 7fe32c2d3700 1 -- 172.21.15.102:0/2708609139 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32418fb70 con 0x7fe324103560 2024-03-26T19:04:15.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.158+0000 7fe31b7fe700 1 -- 172.21.15.102:0/2708609139 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe3140038f0 con 0x7fe324103560 2024-03-26T19:04:15.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.159+0000 7fe31b7fe700 1 --2- 172.21.15.102:0/2708609139 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe31006e460 0x7fe310070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:15.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.159+0000 7fe31b7fe700 1 -- 172.21.15.102:0/2708609139 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fe31408f8a0 con 0x7fe324103560 2024-03-26T19:04:15.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.159+0000 7fe32a06f700 1 --2- 172.21.15.102:0/2708609139 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe31006e460 0x7fe310070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:15.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.160+0000 7fe32a06f700 1 --2- 172.21.15.102:0/2708609139 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe31006e460 0x7fe310070910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe320011490 tx=0x7fe320009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:15.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.164+0000 7fe31b7fe700 1 -- 172.21.15.102:0/2708609139 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe31405c5b0 con 0x7fe324103560 2024-03-26T19:04:15.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.432+0000 7fe32c2d3700 1 -- 172.21.15.102:0/2708609139 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe324061190 con 0x7fe31006e460 2024-03-26T19:04:15.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: pgmap v265: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:15.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:15.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:15.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:04:15.428 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.429 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.430 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.430 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.430 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.430 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:15 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.433+0000 7fe31b7fe700 1 -- 172.21.15.102:0/2708609139 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+460 (secure 0 0 0) 0x7fe324061190 con 0x7fe31006e460 2024-03-26T19:04:15.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.435+0000 7fe32c2d3700 1 -- 172.21.15.102:0/2708609139 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe31006e460 msgr2=0x7fe310070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:15.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.435+0000 7fe32c2d3700 1 --2- 172.21.15.102:0/2708609139 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe31006e460 0x7fe310070910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe320011490 tx=0x7fe320009260 comp rx=0 tx=0).stop 2024-03-26T19:04:15.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.435+0000 7fe32c2d3700 1 -- 172.21.15.102:0/2708609139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe324103560 msgr2=0x7fe324195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:15.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.435+0000 7fe32c2d3700 1 --2- 172.21.15.102:0/2708609139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe324103560 0x7fe324195e90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe314000c00 tx=0x7fe31400f690 comp rx=0 tx=0).stop 2024-03-26T19:04:15.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.435+0000 7fe32c2d3700 1 -- 172.21.15.102:0/2708609139 shutdown_connections 2024-03-26T19:04:15.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.435+0000 7fe32c2d3700 1 --2- 172.21.15.102:0/2708609139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe324100c40 0x7fe324195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.435+0000 7fe32c2d3700 1 --2- 172.21.15.102:0/2708609139 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe31006e460 0x7fe310070910 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.435+0000 7fe32c2d3700 1 --2- 172.21.15.102:0/2708609139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe324103560 0x7fe324195e90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.435+0000 7fe32c2d3700 1 -- 172.21.15.102:0/2708609139 >> 172.21.15.102:0/2708609139 conn(0x7fe3240fa600 msgr2=0x7fe3240fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:15.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.436+0000 7fe32c2d3700 1 -- 172.21.15.102:0/2708609139 shutdown_connections 2024-03-26T19:04:15.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.436+0000 7fe32c2d3700 1 -- 172.21.15.102:0/2708609139 wait complete. 2024-03-26T19:04:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.564+0000 7f2fba140700 1 -- 172.21.15.102:0/3879329707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fb4101350 msgr2=0x7f2fb41017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.564+0000 7f2fba140700 1 --2- 172.21.15.102:0/3879329707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fb4101350 0x7f2fb41017a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009910 tx=0x7f2fa8009c20 comp rx=0 tx=0).stop 2024-03-26T19:04:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.564+0000 7f2fba140700 1 -- 172.21.15.102:0/3879329707 shutdown_connections 2024-03-26T19:04:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.564+0000 7f2fba140700 1 --2- 172.21.15.102:0/3879329707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fb4101350 0x7f2fb41017a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.564+0000 7f2fba140700 1 --2- 172.21.15.102:0/3879329707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fb4100150 0x7f2fb4100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.564+0000 7f2fba140700 1 -- 172.21.15.102:0/3879329707 >> 172.21.15.102:0/3879329707 conn(0x7f2fb40fb6e0 msgr2=0x7f2fb40fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.564+0000 7f2fba140700 1 -- 172.21.15.102:0/3879329707 shutdown_connections 2024-03-26T19:04:15.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.564+0000 7f2fba140700 1 -- 172.21.15.102:0/3879329707 wait complete. 2024-03-26T19:04:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.565+0000 7f2fba140700 1 Processor -- start 2024-03-26T19:04:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.565+0000 7f2fba140700 1 -- start start 2024-03-26T19:04:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.565+0000 7f2fba140700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fb4100150 0x7f2fb4074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.565+0000 7f2fba140700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fb4101350 0x7f2fb4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.565+0000 7f2fba140700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb4073680 con 0x7f2fb4101350 2024-03-26T19:04:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.565+0000 7f2fba140700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40737c0 con 0x7f2fb4100150 2024-03-26T19:04:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.565+0000 7f2fb2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fb4101350 0x7f2fb4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.565+0000 7f2fb2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fb4101350 0x7f2fb4073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42204/0 (socket says 172.21.15.102:42204) 2024-03-26T19:04:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.565+0000 7f2fb2ffd700 1 -- 172.21.15.102:0/2439707982 learned_addr learned my addr 172.21.15.102:0/2439707982 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:04:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.565+0000 7f2fb37fe700 1 --2- 172.21.15.102:0/2439707982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fb4100150 0x7f2fb4074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.566+0000 7f2fb2ffd700 1 -- 172.21.15.102:0/2439707982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fb4100150 msgr2=0x7f2fb4074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.566+0000 7f2fb2ffd700 1 --2- 172.21.15.102:0/2439707982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fb4100150 0x7f2fb4074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.566+0000 7f2fb2ffd700 1 -- 172.21.15.102:0/2439707982 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa80095f0 con 0x7f2fb4101350 2024-03-26T19:04:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.566+0000 7f2fb2ffd700 1 --2- 172.21.15.102:0/2439707982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fb4101350 0x7f2fb4073140 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8006010 tx=0x7f2fa800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.566+0000 7f2fb0ff9700 1 -- 172.21.15.102:0/2439707982 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fa801d070 con 0x7f2fb4101350 2024-03-26T19:04:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.566+0000 7f2fba140700 1 -- 172.21.15.102:0/2439707982 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb4073a40 con 0x7f2fb4101350 2024-03-26T19:04:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.566+0000 7f2fb0ff9700 1 -- 172.21.15.102:0/2439707982 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2fa800f460 con 0x7f2fb4101350 2024-03-26T19:04:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.566+0000 7f2fb0ff9700 1 -- 172.21.15.102:0/2439707982 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fa8017630 con 0x7f2fb4101350 2024-03-26T19:04:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.567+0000 7f2fba140700 1 -- 172.21.15.102:0/2439707982 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb4073f30 con 0x7f2fb4101350 2024-03-26T19:04:15.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.568+0000 7f2fba140700 1 -- 172.21.15.102:0/2439707982 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb4108ca0 con 0x7f2fb4101350 2024-03-26T19:04:15.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.568+0000 7f2fb0ff9700 1 -- 172.21.15.102:0/2439707982 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2fa8003a60 con 0x7f2fb4101350 2024-03-26T19:04:15.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.569+0000 7f2fb0ff9700 1 --2- 172.21.15.102:0/2439707982 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2f9c06e460 0x7f2f9c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:15.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.569+0000 7f2fb0ff9700 1 -- 172.21.15.102:0/2439707982 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f2fa808ef40 con 0x7f2fb4101350 2024-03-26T19:04:15.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.569+0000 7f2fb37fe700 1 --2- 172.21.15.102:0/2439707982 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2f9c06e460 0x7f2f9c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:15.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.570+0000 7f2fb37fe700 1 --2- 172.21.15.102:0/2439707982 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2f9c06e460 0x7f2f9c070910 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4005950 tx=0x7f2fa40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:15.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.574+0000 7f2fb0ff9700 1 -- 172.21.15.102:0/2439707982 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2fa805bd00 con 0x7f2fb4101350 2024-03-26T19:04:15.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: pgmap v265: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:15.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:15.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:15.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:04:15.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:04:15.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:15.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.785 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:15 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.825+0000 7f2fba140700 1 -- 172.21.15.102:0/2439707982 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fb4105ca0 con 0x7f2f9c06e460 2024-03-26T19:04:15.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.841+0000 7f2fb0ff9700 1 -- 172.21.15.102:0/2439707982 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f2fb4105ca0 con 0x7f2f9c06e460 2024-03-26T19:04:15.837 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T19:04:15.837 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (10s) 5s ago 17m 15.4M - 0.25.0 c8568f914cd2 20e72e04e392 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (68s) 5s ago 17m 8108k - 19.0.0-2311-g36051606 497bb87bc5da 2fb0a990cbf1 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (63s) 42s ago 16m 7537k - 19.0.0-2311-g36051606 497bb87bc5da 893c37bef7ca 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (6m) 5s ago 17m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (6m) 42s ago 16m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (15m) 5s ago 17m 85.4M - 9.4.7 954c08fa6188 b9e2c16bf697 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (2m) 5s ago 11m 17.0M - 19.0.0-2311-g36051606 497bb87bc5da c530ed7dbd78 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (2m) 5s ago 11m 79.7M - 19.0.0-2311-g36051606 497bb87bc5da 63b8127a3849 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (83s) 42s ago 11m 13.7M - 19.0.0-2311-g36051606 497bb87bc5da 25df8033593e 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (101s) 42s ago 11m 15.1M - 19.0.0-2311-g36051606 497bb87bc5da 37260789b9ec 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (8m) 5s ago 18m 512M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (8m) 42s ago 16m 452M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T19:04:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (7m) 5s ago 18m 55.8M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T19:04:15.839 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (7m) 42s ago 16m 42.3M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T19:04:15.839 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (49s) 5s ago 17m 8304k - 1.5.0 0da6a335fe13 9992a3ee4208 2024-03-26T19:04:15.839 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (45s) 42s ago 16m 3632k - 1.5.0 0da6a335fe13 6d6186a60a00 2024-03-26T19:04:15.839 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (6m) 5s ago 15m 107M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T19:04:15.839 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (5m) 5s ago 15m 116M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T19:04:15.839 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (5m) 5s ago 14m 98.4M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T19:04:15.839 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (4m) 42s ago 14m 122M 2240M 19.0.0-2311-g36051606 497bb87bc5da 03955331c962 2024-03-26T19:04:15.839 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (3m) 42s ago 13m 112M 2240M 19.0.0-2311-g36051606 497bb87bc5da 23e140cd8961 2024-03-26T19:04:15.839 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (3m) 42s ago 13m 103M 2240M 19.0.0-2311-g36051606 497bb87bc5da 60afb0e4da18 2024-03-26T19:04:15.839 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (27s) 5s ago 16m 64.4M - 2.43.0 a07b618ecd1d 9b1b2a7dbd69 2024-03-26T19:04:15.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.844+0000 7f2fba140700 1 -- 172.21.15.102:0/2439707982 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2f9c06e460 msgr2=0x7f2f9c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:15.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.844+0000 7f2fba140700 1 --2- 172.21.15.102:0/2439707982 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2f9c06e460 0x7f2f9c070910 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4005950 tx=0x7f2fa40058e0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.844+0000 7f2fba140700 1 -- 172.21.15.102:0/2439707982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fb4101350 msgr2=0x7f2fb4073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:15.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.844+0000 7f2fba140700 1 --2- 172.21.15.102:0/2439707982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fb4101350 0x7f2fb4073140 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8006010 tx=0x7f2fa800be20 comp rx=0 tx=0).stop 2024-03-26T19:04:15.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.844+0000 7f2fba140700 1 -- 172.21.15.102:0/2439707982 shutdown_connections 2024-03-26T19:04:15.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.844+0000 7f2fba140700 1 --2- 172.21.15.102:0/2439707982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fb4100150 0x7f2fb4074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.844+0000 7f2fba140700 1 --2- 172.21.15.102:0/2439707982 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2f9c06e460 0x7f2f9c070910 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.844+0000 7f2fba140700 1 --2- 172.21.15.102:0/2439707982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fb4101350 0x7f2fb4073140 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.844+0000 7f2fba140700 1 -- 172.21.15.102:0/2439707982 >> 172.21.15.102:0/2439707982 conn(0x7f2fb40fb6e0 msgr2=0x7f2fb4104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:15.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.844+0000 7f2fba140700 1 -- 172.21.15.102:0/2439707982 shutdown_connections 2024-03-26T19:04:15.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.844+0000 7f2fba140700 1 -- 172.21.15.102:0/2439707982 wait complete. 2024-03-26T19:04:15.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.958+0000 7fde339f2700 1 -- 172.21.15.102:0/3951167512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c0fe7d0 msgr2=0x7fde2c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:15.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.958+0000 7fde339f2700 1 --2- 172.21.15.102:0/3951167512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c0fe7d0 0x7fde2c0febe0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fde24009910 tx=0x7fde24009c20 comp rx=0 tx=0).stop 2024-03-26T19:04:15.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.958+0000 7fde339f2700 1 -- 172.21.15.102:0/3951167512 shutdown_connections 2024-03-26T19:04:15.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.958+0000 7fde339f2700 1 --2- 172.21.15.102:0/3951167512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c0ff9d0 0x7fde2c0ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.958+0000 7fde339f2700 1 --2- 172.21.15.102:0/3951167512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c0fe7d0 0x7fde2c0febe0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.958+0000 7fde339f2700 1 -- 172.21.15.102:0/3951167512 >> 172.21.15.102:0/3951167512 conn(0x7fde2c0f9d80 msgr2=0x7fde2c0fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:15.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.958+0000 7fde339f2700 1 -- 172.21.15.102:0/3951167512 shutdown_connections 2024-03-26T19:04:15.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.958+0000 7fde339f2700 1 -- 172.21.15.102:0/3951167512 wait complete. 2024-03-26T19:04:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.958+0000 7fde339f2700 1 Processor -- start 2024-03-26T19:04:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.958+0000 7fde339f2700 1 -- start start 2024-03-26T19:04:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde339f2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c0fe7d0 0x7fde2c101750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde339f2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c0ff9d0 0x7fde2c19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde339f2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c19ac30 con 0x7fde2c0fe7d0 2024-03-26T19:04:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde339f2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c19ad70 con 0x7fde2c0ff9d0 2024-03-26T19:04:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde3178e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c0fe7d0 0x7fde2c101750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde3178e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c0fe7d0 0x7fde2c101750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42232/0 (socket says 172.21.15.102:42232) 2024-03-26T19:04:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde3178e700 1 -- 172.21.15.102:0/3521120003 learned_addr learned my addr 172.21.15.102:0/3521120003 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:04:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde30f8d700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c0ff9d0 0x7fde2c19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde3178e700 1 -- 172.21.15.102:0/3521120003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c0ff9d0 msgr2=0x7fde2c19a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde3178e700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c0ff9d0 0x7fde2c19a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde3178e700 1 -- 172.21.15.102:0/3521120003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde240095f0 con 0x7fde2c0fe7d0 2024-03-26T19:04:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde30f8d700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c0ff9d0 0x7fde2c19a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T19:04:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde3178e700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c0fe7d0 0x7fde2c101750 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fde2400bda0 tx=0x7fde2400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde2a7fc700 1 -- 172.21.15.102:0/3521120003 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde2401d070 con 0x7fde2c0fe7d0 2024-03-26T19:04:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.959+0000 7fde339f2700 1 -- 172.21.15.102:0/3521120003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2c19aff0 con 0x7fde2c0fe7d0 2024-03-26T19:04:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.960+0000 7fde2a7fc700 1 -- 172.21.15.102:0/3521120003 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fde24004c50 con 0x7fde2c0fe7d0 2024-03-26T19:04:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.960+0000 7fde2a7fc700 1 -- 172.21.15.102:0/3521120003 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde2400f690 con 0x7fde2c0fe7d0 2024-03-26T19:04:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.960+0000 7fde339f2700 1 -- 172.21.15.102:0/3521120003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2c19b4b0 con 0x7fde2c0fe7d0 2024-03-26T19:04:15.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.961+0000 7fde13fff700 1 -- 172.21.15.102:0/3521120003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0c0052f0 con 0x7fde2c0fe7d0 2024-03-26T19:04:15.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.961+0000 7fde2a7fc700 1 -- 172.21.15.102:0/3521120003 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fde24022a20 con 0x7fde2c0fe7d0 2024-03-26T19:04:15.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.962+0000 7fde2a7fc700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fde1406e460 0x7fde14070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:15.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.962+0000 7fde30f8d700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fde1406e460 0x7fde14070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:15.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.962+0000 7fde2a7fc700 1 -- 172.21.15.102:0/3521120003 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fde2408fb90 con 0x7fde2c0fe7d0 2024-03-26T19:04:15.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.962+0000 7fde30f8d700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fde1406e460 0x7fde14070910 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fde20005fd0 tx=0x7fde20005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:15.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:15.966+0000 7fde2a7fc700 1 -- 172.21.15.102:0/3521120003 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fde2405c8a0 con 0x7fde2c0fe7d0 2024-03-26T19:04:16.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.314+0000 7fde13fff700 1 -- 172.21.15.102:0/3521120003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fde0c0061d0 con 0x7fde2c0fe7d0 2024-03-26T19:04:16.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.315+0000 7fde2a7fc700 1 -- 172.21.15.102:0/3521120003 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fde2405c430 con 0x7fde2c0fe7d0 2024-03-26T19:04:16.311 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:04:16.311 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T19:04:16.311 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:04:16.311 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:04:16.311 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T19:04:16.311 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:04:16.312 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:04:16.312 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T19:04:16.312 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 6 2024-03-26T19:04:16.312 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:04:16.312 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T19:04:16.312 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 4 2024-03-26T19:04:16.312 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:04:16.312 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T19:04:16.312 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 14 2024-03-26T19:04:16.312 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T19:04:16.312 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:04:16.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.318+0000 7fde13fff700 1 -- 172.21.15.102:0/3521120003 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fde1406e460 msgr2=0x7fde14070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:16.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.318+0000 7fde13fff700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fde1406e460 0x7fde14070910 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fde20005fd0 tx=0x7fde20005e30 comp rx=0 tx=0).stop 2024-03-26T19:04:16.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.318+0000 7fde13fff700 1 -- 172.21.15.102:0/3521120003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c0fe7d0 msgr2=0x7fde2c101750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:16.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.318+0000 7fde13fff700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c0fe7d0 0x7fde2c101750 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fde2400bda0 tx=0x7fde2400be80 comp rx=0 tx=0).stop 2024-03-26T19:04:16.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.318+0000 7fde13fff700 1 -- 172.21.15.102:0/3521120003 shutdown_connections 2024-03-26T19:04:16.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.318+0000 7fde13fff700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fde1406e460 0x7fde14070910 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.318+0000 7fde13fff700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c0fe7d0 0x7fde2c101750 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.318+0000 7fde13fff700 1 --2- 172.21.15.102:0/3521120003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c0ff9d0 0x7fde2c19a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.318+0000 7fde13fff700 1 -- 172.21.15.102:0/3521120003 >> 172.21.15.102:0/3521120003 conn(0x7fde2c0f9d80 msgr2=0x7fde2c0fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:16.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.318+0000 7fde13fff700 1 -- 172.21.15.102:0/3521120003 shutdown_connections 2024-03-26T19:04:16.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.318+0000 7fde13fff700 1 -- 172.21.15.102:0/3521120003 wait complete. 2024-03-26T19:04:16.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:16 smithi102 ceph-mon[78322]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:16.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:16 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T19:04:16.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.426+0000 7f9117110700 1 -- 172.21.15.102:0/2152435860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9110100be0 msgr2=0x7f9110102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:16.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.426+0000 7f9117110700 1 --2- 172.21.15.102:0/2152435860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9110100be0 0x7f9110102fc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9104009910 tx=0x7f9104009c20 comp rx=0 tx=0).stop 2024-03-26T19:04:16.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.426+0000 7f9117110700 1 -- 172.21.15.102:0/2152435860 shutdown_connections 2024-03-26T19:04:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.426+0000 7f9117110700 1 --2- 172.21.15.102:0/2152435860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9110103500 0x7f91101058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.426+0000 7f9117110700 1 --2- 172.21.15.102:0/2152435860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9110100be0 0x7f9110102fc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.426+0000 7f9117110700 1 -- 172.21.15.102:0/2152435860 >> 172.21.15.102:0/2152435860 conn(0x7f91100fa600 msgr2=0x7f91100fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.426+0000 7f9117110700 1 -- 172.21.15.102:0/2152435860 shutdown_connections 2024-03-26T19:04:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.427+0000 7f9117110700 1 -- 172.21.15.102:0/2152435860 wait complete. 2024-03-26T19:04:16.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.427+0000 7f9117110700 1 Processor -- start 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.427+0000 7f9117110700 1 -- start start 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.427+0000 7f9117110700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9110100be0 0x7f9110195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f9117110700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9110103500 0x7f9110195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f9117110700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91101964e0 con 0x7f9110103500 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f9117110700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9110196620 con 0x7f9110100be0 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f910ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9110103500 0x7f9110195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f910ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9110103500 0x7f9110195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42244/0 (socket says 172.21.15.102:42244) 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f910ffff700 1 -- 172.21.15.102:0/2122213753 learned_addr learned my addr 172.21.15.102:0/2122213753 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f9114eac700 1 --2- 172.21.15.102:0/2122213753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9110100be0 0x7f9110195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f910ffff700 1 -- 172.21.15.102:0/2122213753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9110100be0 msgr2=0x7f9110195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f910ffff700 1 --2- 172.21.15.102:0/2122213753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9110100be0 0x7f9110195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f910ffff700 1 -- 172.21.15.102:0/2122213753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91040095f0 con 0x7f9110103500 2024-03-26T19:04:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f910ffff700 1 --2- 172.21.15.102:0/2122213753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9110103500 0x7f9110195ec0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00d8d0 tx=0x7f90fc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.428+0000 7f910dffb700 1 -- 172.21.15.102:0/2122213753 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90fc0098a0 con 0x7f9110103500 2024-03-26T19:04:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.429+0000 7f910dffb700 1 -- 172.21.15.102:0/2122213753 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f90fc010460 con 0x7f9110103500 2024-03-26T19:04:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.429+0000 7f910dffb700 1 -- 172.21.15.102:0/2122213753 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90fc00f5f0 con 0x7f9110103500 2024-03-26T19:04:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.429+0000 7f9117110700 1 -- 172.21.15.102:0/2122213753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911019b080 con 0x7f9110103500 2024-03-26T19:04:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.429+0000 7f9117110700 1 -- 172.21.15.102:0/2122213753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911019b5d0 con 0x7f9110103500 2024-03-26T19:04:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.430+0000 7f9117110700 1 -- 172.21.15.102:0/2122213753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911018fb70 con 0x7f9110103500 2024-03-26T19:04:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.430+0000 7f910dffb700 1 -- 172.21.15.102:0/2122213753 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f90fc009a00 con 0x7f9110103500 2024-03-26T19:04:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.431+0000 7f910dffb700 1 --2- 172.21.15.102:0/2122213753 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f910006e460 0x7f9100070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.431+0000 7f910dffb700 1 -- 172.21.15.102:0/2122213753 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f90fc08e250 con 0x7f9110103500 2024-03-26T19:04:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.431+0000 7f9114eac700 1 --2- 172.21.15.102:0/2122213753 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f910006e460 0x7f9100070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.432+0000 7f9114eac700 1 --2- 172.21.15.102:0/2122213753 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f910006e460 0x7f9100070910 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9104006010 tx=0x7f910400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.440+0000 7f910dffb700 1 -- 172.21.15.102:0/2122213753 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f90fc05af60 con 0x7f9110103500 2024-03-26T19:04:16.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:16 smithi190 ceph-mon[65689]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:16.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:16 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T19:04:16.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.802+0000 7f9117110700 1 -- 172.21.15.102:0/2122213753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f911002cc30 con 0x7f9110103500 2024-03-26T19:04:16.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.803+0000 7f910dffb700 1 -- 172.21.15.102:0/2122213753 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 43 v43) v1 ==== 76+0+1955 (secure 0 0 0) 0x7f90fc0167c0 con 0x7f9110103500 2024-03-26T19:04:16.799 INFO:teuthology.orchestra.run.smithi102.stdout:e43 2024-03-26T19:04:16.799 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T19:04:16.799 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:04:16.799 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T19:04:16.799 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:04:16.799 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T19:04:16.799 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T19:04:16.799 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 43 2024-03-26T19:04:16.799 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T19:04:16.799 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T19:03:02.920923+0000 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 84 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T19:04:16.800 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=34436,1=34468} 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 34436 members: 34436,34468 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{0:34436} state up:active seq 19 join_fscid=1 addr [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{1:34468} state up:active seq 13 join_fscid=1 addr [v2:172.21.15.102:6828/3007238267,v1:172.21.15.102:6829/3007238267] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:04:16.801 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T19:04:16.802 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:04:16.802 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.190:6824/782636706,v1:172.21.15.190:6825/782636706] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:04:16.802 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:44373} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.190:6826/925215849,v1:172.21.15.190:6827/925215849] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:04:16.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.806+0000 7f9117110700 1 -- 172.21.15.102:0/2122213753 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f910006e460 msgr2=0x7f9100070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:16.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.806+0000 7f9117110700 1 --2- 172.21.15.102:0/2122213753 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f910006e460 0x7f9100070910 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9104006010 tx=0x7f910400b540 comp rx=0 tx=0).stop 2024-03-26T19:04:16.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.806+0000 7f9117110700 1 -- 172.21.15.102:0/2122213753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9110103500 msgr2=0x7f9110195ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:16.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.806+0000 7f9117110700 1 --2- 172.21.15.102:0/2122213753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9110103500 0x7f9110195ec0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00d8d0 tx=0x7f90fc00dbe0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.806+0000 7f9117110700 1 -- 172.21.15.102:0/2122213753 shutdown_connections 2024-03-26T19:04:16.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.806+0000 7f9117110700 1 --2- 172.21.15.102:0/2122213753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9110100be0 0x7f9110195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.806+0000 7f9117110700 1 --2- 172.21.15.102:0/2122213753 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f910006e460 0x7f9100070910 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.806+0000 7f9117110700 1 --2- 172.21.15.102:0/2122213753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9110103500 0x7f9110195ec0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.806+0000 7f9117110700 1 -- 172.21.15.102:0/2122213753 >> 172.21.15.102:0/2122213753 conn(0x7f91100fa600 msgr2=0x7f91100fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:16.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.806+0000 7f9117110700 1 -- 172.21.15.102:0/2122213753 shutdown_connections 2024-03-26T19:04:16.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.806+0000 7f9117110700 1 -- 172.21.15.102:0/2122213753 wait complete. 2024-03-26T19:04:16.803 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 43 2024-03-26T19:04:16.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.936+0000 7f41a9906700 1 -- 172.21.15.102:0/985766989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41a40ff050 msgr2=0x7f41a40ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:16.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.936+0000 7f41a9906700 1 --2- 172.21.15.102:0/985766989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41a40ff050 0x7f41a40ff460 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4194009910 tx=0x7f4194009c20 comp rx=0 tx=0).stop 2024-03-26T19:04:16.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.936+0000 7f41a9906700 1 -- 172.21.15.102:0/985766989 shutdown_connections 2024-03-26T19:04:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.936+0000 7f41a9906700 1 --2- 172.21.15.102:0/985766989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41a4100250 0x7f41a41006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.936+0000 7f41a9906700 1 --2- 172.21.15.102:0/985766989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41a40ff050 0x7f41a40ff460 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.936+0000 7f41a9906700 1 -- 172.21.15.102:0/985766989 >> 172.21.15.102:0/985766989 conn(0x7f41a40fa5e0 msgr2=0x7f41a40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.936+0000 7f41a9906700 1 -- 172.21.15.102:0/985766989 shutdown_connections 2024-03-26T19:04:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.936+0000 7f41a9906700 1 -- 172.21.15.102:0/985766989 wait complete. 2024-03-26T19:04:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.937+0000 7f41a9906700 1 Processor -- start 2024-03-26T19:04:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.937+0000 7f41a9906700 1 -- start start 2024-03-26T19:04:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.937+0000 7f41a9906700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41a40ff050 0x7f41a41959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.937+0000 7f41a9906700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41a4100250 0x7f41a4195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.937+0000 7f41a9906700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a4196500 con 0x7f41a4100250 2024-03-26T19:04:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.937+0000 7f41a9906700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a4196640 con 0x7f41a40ff050 2024-03-26T19:04:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.938+0000 7f41a27fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41a4100250 0x7f41a4195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.938+0000 7f41a27fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41a4100250 0x7f41a4195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42262/0 (socket says 172.21.15.102:42262) 2024-03-26T19:04:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.938+0000 7f41a27fc700 1 -- 172.21.15.102:0/1713275383 learned_addr learned my addr 172.21.15.102:0/1713275383 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:04:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.938+0000 7f41a2ffd700 1 --2- 172.21.15.102:0/1713275383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41a40ff050 0x7f41a41959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.938+0000 7f41a27fc700 1 -- 172.21.15.102:0/1713275383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41a40ff050 msgr2=0x7f41a41959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.938+0000 7f41a27fc700 1 --2- 172.21.15.102:0/1713275383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41a40ff050 0x7f41a41959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.938+0000 7f41a27fc700 1 -- 172.21.15.102:0/1713275383 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940095f0 con 0x7f41a4100250 2024-03-26T19:04:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.938+0000 7f41a27fc700 1 --2- 172.21.15.102:0/1713275383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41a4100250 0x7f41a4195ee0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f419800d900 tx=0x7f419800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.938+0000 7f41a8904700 1 -- 172.21.15.102:0/1713275383 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41980041f0 con 0x7f41a4100250 2024-03-26T19:04:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.939+0000 7f41a8904700 1 -- 172.21.15.102:0/1713275383 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4198004ce0 con 0x7f41a4100250 2024-03-26T19:04:16.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.939+0000 7f41a8904700 1 -- 172.21.15.102:0/1713275383 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f419800b750 con 0x7f41a4100250 2024-03-26T19:04:16.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.939+0000 7f41a9906700 1 -- 172.21.15.102:0/1713275383 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a419b0f0 con 0x7f41a4100250 2024-03-26T19:04:16.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.939+0000 7f41a9906700 1 -- 172.21.15.102:0/1713275383 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a419b610 con 0x7f41a4100250 2024-03-26T19:04:16.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.941+0000 7f41a9906700 1 -- 172.21.15.102:0/1713275383 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a4107ba0 con 0x7f41a4100250 2024-03-26T19:04:16.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.942+0000 7f41a8904700 1 -- 172.21.15.102:0/1713275383 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4198004e50 con 0x7f41a4100250 2024-03-26T19:04:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.942+0000 7f41a8904700 1 --2- 172.21.15.102:0/1713275383 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f418c06e260 0x7f418c070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.942+0000 7f41a8904700 1 -- 172.21.15.102:0/1713275383 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f419808e1a0 con 0x7f41a4100250 2024-03-26T19:04:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.942+0000 7f41a2ffd700 1 --2- 172.21.15.102:0/1713275383 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f418c06e260 0x7f418c070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.943+0000 7f41a2ffd700 1 --2- 172.21.15.102:0/1713275383 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f418c06e260 0x7f418c070710 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4194006010 tx=0x7f419401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:16.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:16.951+0000 7f41a8904700 1 -- 172.21.15.102:0/1713275383 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f419805aeb0 con 0x7f41a4100250 2024-03-26T19:04:17.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.332+0000 7f41a9906700 1 -- 172.21.15.102:0/1713275383 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f41a4068ae0 con 0x7f418c06e260 2024-03-26T19:04:17.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.344+0000 7f41a8904700 1 -- 172.21.15.102:0/1713275383 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+460 (secure 0 0 0) 0x7f41a4068ae0 con 0x7f418c06e260 2024-03-26T19:04:17.340 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9", 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "mds", 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "osd", 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "mon", 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr", 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph-exporter" 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "18/23 daemons upgraded", 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading grafana daemons", 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T19:04:17.341 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:04:17.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.347+0000 7f41a9906700 1 -- 172.21.15.102:0/1713275383 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f418c06e260 msgr2=0x7f418c070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.347+0000 7f41a9906700 1 --2- 172.21.15.102:0/1713275383 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f418c06e260 0x7f418c070710 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4194006010 tx=0x7f419401a040 comp rx=0 tx=0).stop 2024-03-26T19:04:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.347+0000 7f41a9906700 1 -- 172.21.15.102:0/1713275383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41a4100250 msgr2=0x7f41a4195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.347+0000 7f41a9906700 1 --2- 172.21.15.102:0/1713275383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41a4100250 0x7f41a4195ee0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f419800d900 tx=0x7f419800dc10 comp rx=0 tx=0).stop 2024-03-26T19:04:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.347+0000 7f41a9906700 1 -- 172.21.15.102:0/1713275383 shutdown_connections 2024-03-26T19:04:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.347+0000 7f41a9906700 1 --2- 172.21.15.102:0/1713275383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41a40ff050 0x7f41a41959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.347+0000 7f41a9906700 1 --2- 172.21.15.102:0/1713275383 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f418c06e260 0x7f418c070710 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.347+0000 7f41a9906700 1 --2- 172.21.15.102:0/1713275383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41a4100250 0x7f41a4195ee0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.347+0000 7f41a9906700 1 -- 172.21.15.102:0/1713275383 >> 172.21.15.102:0/1713275383 conn(0x7f41a40fa5e0 msgr2=0x7f41a40683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.347+0000 7f41a9906700 1 -- 172.21.15.102:0/1713275383 shutdown_connections 2024-03-26T19:04:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.347+0000 7f41a9906700 1 -- 172.21.15.102:0/1713275383 wait complete. 2024-03-26T19:04:17.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.462+0000 7f96028e4700 1 -- 172.21.15.102:0/2381165038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95fc0ff050 msgr2=0x7f95fc0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:17.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.462+0000 7f96028e4700 1 --2- 172.21.15.102:0/2381165038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95fc0ff050 0x7f95fc0ff460 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009910 tx=0x7f95ec009c20 comp rx=0 tx=0).stop 2024-03-26T19:04:17.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.462+0000 7f96028e4700 1 -- 172.21.15.102:0/2381165038 shutdown_connections 2024-03-26T19:04:17.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.462+0000 7f96028e4700 1 --2- 172.21.15.102:0/2381165038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95fc100250 0x7f95fc1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:17.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.462+0000 7f96028e4700 1 --2- 172.21.15.102:0/2381165038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95fc0ff050 0x7f95fc0ff460 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:17.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.462+0000 7f96028e4700 1 -- 172.21.15.102:0/2381165038 >> 172.21.15.102:0/2381165038 conn(0x7f95fc0fa5e0 msgr2=0x7f95fc0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:17.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.462+0000 7f96028e4700 1 -- 172.21.15.102:0/2381165038 shutdown_connections 2024-03-26T19:04:17.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.462+0000 7f96028e4700 1 -- 172.21.15.102:0/2381165038 wait complete. 2024-03-26T19:04:17.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.463+0000 7f96028e4700 1 Processor -- start 2024-03-26T19:04:17.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.463+0000 7f96028e4700 1 -- start start 2024-03-26T19:04:17.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.463+0000 7f96028e4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95fc0ff050 0x7f95fc068f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.463+0000 7f96028e4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95fc100250 0x7f95fc19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.463+0000 7f96028e4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc19acb0 con 0x7f95fc0ff050 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.463+0000 7f96028e4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc19adf0 con 0x7f95fc100250 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.463+0000 7f95fbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95fc0ff050 0x7f95fc068f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.463+0000 7f95fbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95fc0ff050 0x7f95fc068f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42286/0 (socket says 172.21.15.102:42286) 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.463+0000 7f95fbfff700 1 -- 172.21.15.102:0/1010869035 learned_addr learned my addr 172.21.15.102:0/1010869035 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.463+0000 7f95fb7fe700 1 --2- 172.21.15.102:0/1010869035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95fc100250 0x7f95fc19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.464+0000 7f95fbfff700 1 -- 172.21.15.102:0/1010869035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95fc100250 msgr2=0x7f95fc19a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.464+0000 7f95fbfff700 1 --2- 172.21.15.102:0/1010869035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95fc100250 0x7f95fc19a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.464+0000 7f95fbfff700 1 -- 172.21.15.102:0/1010869035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0095f0 con 0x7f95fc0ff050 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.464+0000 7f95fbfff700 1 --2- 172.21.15.102:0/1010869035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95fc0ff050 0x7f95fc068f90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00bda0 tx=0x7f95ec00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.464+0000 7f95f97fa700 1 -- 172.21.15.102:0/1010869035 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95ec01d070 con 0x7f95fc0ff050 2024-03-26T19:04:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.464+0000 7f95f97fa700 1 -- 172.21.15.102:0/1010869035 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f95ec00f460 con 0x7f95fc0ff050 2024-03-26T19:04:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.464+0000 7f96028e4700 1 -- 172.21.15.102:0/1010869035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95fc19b070 con 0x7f95fc0ff050 2024-03-26T19:04:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.464+0000 7f95f97fa700 1 -- 172.21.15.102:0/1010869035 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95ec021620 con 0x7f95fc0ff050 2024-03-26T19:04:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.464+0000 7f96028e4700 1 -- 172.21.15.102:0/1010869035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95fc19b560 con 0x7f95fc0ff050 2024-03-26T19:04:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.466+0000 7f96028e4700 1 -- 172.21.15.102:0/1010869035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95fc107ba0 con 0x7f95fc0ff050 2024-03-26T19:04:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.467+0000 7f95f97fa700 1 -- 172.21.15.102:0/1010869035 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f95ec00f5d0 con 0x7f95fc0ff050 2024-03-26T19:04:17.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.467+0000 7f95f97fa700 1 --2- 172.21.15.102:0/1010869035 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f95e406e4b0 0x7f95e4070960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:17.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.467+0000 7f95f97fa700 1 -- 172.21.15.102:0/1010869035 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f95ec08f720 con 0x7f95fc0ff050 2024-03-26T19:04:17.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.467+0000 7f95fb7fe700 1 --2- 172.21.15.102:0/1010869035 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f95e406e4b0 0x7f95e4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:17.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.468+0000 7f95fb7fe700 1 --2- 172.21.15.102:0/1010869035 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f95e406e4b0 0x7f95e4070960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f95f0005950 tx=0x7f95f000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:17.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.474+0000 7f95f97fa700 1 -- 172.21.15.102:0/1010869035 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f95ec05c3b0 con 0x7f95fc0ff050 2024-03-26T19:04:17.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:17 smithi102 ceph-mon[78322]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:17.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:17 smithi102 ceph-mon[78322]: pgmap v266: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:17.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:17 smithi102 ceph-mon[78322]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:17.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:17 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3521120003' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:17.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:17 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2122213753' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:04:17.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:17 smithi190 ceph-mon[65689]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:17.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:17 smithi190 ceph-mon[65689]: pgmap v266: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:17.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:17 smithi190 ceph-mon[65689]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:17.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:17 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3521120003' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:17.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:17 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2122213753' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:04:17.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.803+0000 7f96028e4700 1 -- 172.21.15.102:0/1010869035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f95fc19bb90 con 0x7f95fc0ff050 2024-03-26T19:04:17.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.804+0000 7f95f97fa700 1 -- 172.21.15.102:0/1010869035 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f95ec05bf40 con 0x7f95fc0ff050 2024-03-26T19:04:17.800 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T19:04:17.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.806+0000 7f96028e4700 1 -- 172.21.15.102:0/1010869035 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f95e406e4b0 msgr2=0x7f95e4070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:17.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.806+0000 7f96028e4700 1 --2- 172.21.15.102:0/1010869035 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f95e406e4b0 0x7f95e4070960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f95f0005950 tx=0x7f95f000b410 comp rx=0 tx=0).stop 2024-03-26T19:04:17.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.806+0000 7f96028e4700 1 -- 172.21.15.102:0/1010869035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95fc0ff050 msgr2=0x7f95fc068f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:17.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.806+0000 7f96028e4700 1 --2- 172.21.15.102:0/1010869035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95fc0ff050 0x7f95fc068f90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00bda0 tx=0x7f95ec00bdd0 comp rx=0 tx=0).stop 2024-03-26T19:04:17.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.806+0000 7f96028e4700 1 -- 172.21.15.102:0/1010869035 shutdown_connections 2024-03-26T19:04:17.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.806+0000 7f96028e4700 1 --2- 172.21.15.102:0/1010869035 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f95e406e4b0 0x7f95e4070960 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:17.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.806+0000 7f96028e4700 1 --2- 172.21.15.102:0/1010869035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95fc0ff050 0x7f95fc068f90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:17.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.806+0000 7f96028e4700 1 --2- 172.21.15.102:0/1010869035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95fc100250 0x7f95fc19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:17.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.806+0000 7f96028e4700 1 -- 172.21.15.102:0/1010869035 >> 172.21.15.102:0/1010869035 conn(0x7f95fc0fa5e0 msgr2=0x7f95fc103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:17.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.806+0000 7f96028e4700 1 -- 172.21.15.102:0/1010869035 shutdown_connections 2024-03-26T19:04:17.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:17.806+0000 7f96028e4700 1 -- 172.21.15.102:0/1010869035 wait complete. 2024-03-26T19:04:18.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:18 smithi190 ceph-mon[65689]: Upgrade: Updating grafana.smithi102 2024-03-26T19:04:18.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:18 smithi190 ceph-mon[65689]: Deploying daemon grafana.smithi102 on smithi102 2024-03-26T19:04:18.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:18 smithi190 ceph-mon[65689]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:18.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:18 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1010869035' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:04:18.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:18 smithi102 ceph-mon[78322]: Upgrade: Updating grafana.smithi102 2024-03-26T19:04:18.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:18 smithi102 ceph-mon[78322]: Deploying daemon grafana.smithi102 on smithi102 2024-03-26T19:04:18.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:18 smithi102 ceph-mon[78322]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:18.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:18 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1010869035' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:04:19.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:19 smithi102 ceph-mon[78322]: pgmap v267: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:19.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:19 smithi190 ceph-mon[65689]: pgmap v267: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:21.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:21 smithi102 ceph-mon[78322]: pgmap v268: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:21.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:21 smithi190 ceph-mon[65689]: pgmap v268: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:22.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:22 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:04:22.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:22 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:04:23.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:23 smithi102 ceph-mon[78322]: pgmap v269: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:23.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:23 smithi190 ceph-mon[65689]: pgmap v269: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:25.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:25 smithi102 ceph-mon[78322]: pgmap v270: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:25.786 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:25 smithi190 ceph-mon[65689]: pgmap v270: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:27.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:27 smithi102 ceph-mon[78322]: pgmap v271: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:27.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:27 smithi190 ceph-mon[65689]: pgmap v271: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:29.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:29 smithi102 ceph-mon[78322]: pgmap v272: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:29.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:29 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:29.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:29 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:29.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:29 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:29.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:29 smithi190 ceph-mon[65689]: pgmap v272: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:29.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:29 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:29.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:29 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:29.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:29 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:31.629 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:31 smithi102 ceph-mon[78322]: pgmap v273: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:31.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:31 smithi190 ceph-mon[65689]: pgmap v273: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:33.597 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:33 smithi102 ceph-mon[78322]: pgmap v274: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:33.783 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:33 smithi190 ceph-mon[65689]: pgmap v274: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:35.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:34 smithi190 ceph-mon[65689]: pgmap v275: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:35.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:34 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:35.033 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:34 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:35.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:34 smithi102 ceph-mon[78322]: pgmap v275: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:35.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:34 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:35.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:34 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:36.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:36.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:35 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:36.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:36.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:35 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:37.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:36 smithi190 ceph-mon[65689]: pgmap v276: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:37.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:36 smithi102 ceph-mon[78322]: pgmap v276: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:38.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:37 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:04:38.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:37 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:04:39.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: pgmap v277: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:39.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:39.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:39.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:04:39.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:04:39.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.286 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T19:04:39.287 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-26T19:04:39.288 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T19:04:39.288 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T19:04:39.288 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T19:04:39.288 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T19:04:39.288 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.288 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.288 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.288 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.288 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.289 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.289 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T19:04:39.289 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T19:04:39.289 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:39.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: pgmap v277: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:04:39.381 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-26T19:04:39.382 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.383 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.384 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T19:04:39.384 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T19:04:39.384 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T19:04:39.384 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:40.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:39 smithi190 ceph-mon[65689]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T19:04:40.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:39 smithi190 ceph-mon[65689]: Upgrade: Finalizing container_image settings 2024-03-26T19:04:40.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:39 smithi190 ceph-mon[65689]: Upgrade: Complete! 2024-03-26T19:04:40.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:39 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:04:40.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:39 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:04:40.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:39 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:40.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:39 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:40.284 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:39 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:04:40.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:39 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:04:40.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:39 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:40.344 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:39 smithi102 ceph-mon[78322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T19:04:40.344 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:39 smithi102 ceph-mon[78322]: Upgrade: Finalizing container_image settings 2024-03-26T19:04:40.344 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:39 smithi102 ceph-mon[78322]: Upgrade: Complete! 2024-03-26T19:04:40.344 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:39 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:04:40.344 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:39 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:04:40.344 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:39 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:40.344 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:39 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:04:40.344 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:39 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:04:40.344 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:39 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:04:40.344 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:39 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:41.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:40 smithi190 ceph-mon[65689]: pgmap v278: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:41.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:40 smithi102 ceph-mon[78322]: pgmap v278: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:43.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:42 smithi190 ceph-mon[65689]: pgmap v279: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:43.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:42 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:43.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:42 smithi102 ceph-mon[78322]: pgmap v279: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:43.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:42 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:04:45.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:44 smithi190 ceph-mon[65689]: pgmap v280: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:45.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:44 smithi102 ceph-mon[78322]: pgmap v280: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:47.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:46 smithi102 ceph-mon[78322]: pgmap v281: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:47.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:46 smithi190 ceph-mon[65689]: pgmap v281: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:47.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.917+0000 7f741f2c4700 1 -- 172.21.15.102:0/2596800202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7418100c80 msgr2=0x7f7418103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:47.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.917+0000 7f741f2c4700 1 --2- 172.21.15.102:0/2596800202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7418100c80 0x7f7418103060 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7414009910 tx=0x7f7414009c20 comp rx=0 tx=0).stop 2024-03-26T19:04:47.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.917+0000 7f741f2c4700 1 -- 172.21.15.102:0/2596800202 shutdown_connections 2024-03-26T19:04:47.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.917+0000 7f741f2c4700 1 --2- 172.21.15.102:0/2596800202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74181035a0 0x7f7418105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:47.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.917+0000 7f741f2c4700 1 --2- 172.21.15.102:0/2596800202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7418100c80 0x7f7418103060 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:47.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.917+0000 7f741f2c4700 1 -- 172.21.15.102:0/2596800202 >> 172.21.15.102:0/2596800202 conn(0x7f74180fa5e0 msgr2=0x7f74180fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:47.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.917+0000 7f741f2c4700 1 -- 172.21.15.102:0/2596800202 shutdown_connections 2024-03-26T19:04:47.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.917+0000 7f741f2c4700 1 -- 172.21.15.102:0/2596800202 wait complete. 2024-03-26T19:04:47.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741f2c4700 1 Processor -- start 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741f2c4700 1 -- start start 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741f2c4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7418100c80 0x7f74181005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741f2c4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74181035a0 0x7f74180fec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741f2c4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74180ff170 con 0x7f74181035a0 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741f2c4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74180ff2b0 con 0x7f7418100c80 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741c85f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74181035a0 0x7f74180fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741c85f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74181035a0 0x7f74180fec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36568/0 (socket says 172.21.15.102:36568) 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741c85f700 1 -- 172.21.15.102:0/1663667254 learned_addr learned my addr 172.21.15.102:0/1663667254 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741d060700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7418100c80 0x7f74181005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741c85f700 1 -- 172.21.15.102:0/1663667254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7418100c80 msgr2=0x7f74181005e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741c85f700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7418100c80 0x7f74181005e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741c85f700 1 -- 172.21.15.102:0/1663667254 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74140095f0 con 0x7f74181035a0 2024-03-26T19:04:47.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.918+0000 7f741d060700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7418100c80 0x7f74181005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T19:04:47.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.919+0000 7f741c85f700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74181035a0 0x7f74180fec30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f740800b4b0 tx=0x7f740800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:47.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.919+0000 7f740e7fc700 1 -- 172.21.15.102:0/1663667254 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7408011800 con 0x7f74181035a0 2024-03-26T19:04:47.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.919+0000 7f740e7fc700 1 -- 172.21.15.102:0/1663667254 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7408011e20 con 0x7f74181035a0 2024-03-26T19:04:47.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.919+0000 7f741f2c4700 1 -- 172.21.15.102:0/1663667254 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74180ff590 con 0x7f74181035a0 2024-03-26T19:04:47.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.919+0000 7f740e7fc700 1 -- 172.21.15.102:0/1663667254 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f740800f2e0 con 0x7f74181035a0 2024-03-26T19:04:47.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.919+0000 7f741f2c4700 1 -- 172.21.15.102:0/1663667254 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741819da90 con 0x7f74181035a0 2024-03-26T19:04:47.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.920+0000 7f741f2c4700 1 -- 172.21.15.102:0/1663667254 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741818fc00 con 0x7f74181035a0 2024-03-26T19:04:47.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.921+0000 7f740e7fc700 1 -- 172.21.15.102:0/1663667254 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7408010380 con 0x7f74181035a0 2024-03-26T19:04:47.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.921+0000 7f740e7fc700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f740406e460 0x7f7404070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:47.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.921+0000 7f740e7fc700 1 -- 172.21.15.102:0/1663667254 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f740808df80 con 0x7f74181035a0 2024-03-26T19:04:47.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.921+0000 7f741d060700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f740406e460 0x7f7404070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:47.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.922+0000 7f741d060700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f740406e460 0x7f7404070910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f74140059d0 tx=0x7f7414005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:47.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:47.930+0000 7f740e7fc700 1 -- 172.21.15.102:0/1663667254 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f74080c2740 con 0x7f74181035a0 2024-03-26T19:04:48.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.257+0000 7f741f2c4700 1 -- 172.21.15.102:0/1663667254 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7418061190 con 0x7f740406e460 2024-03-26T19:04:48.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.258+0000 7f740e7fc700 1 -- 172.21.15.102:0/1663667254 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f7418061190 con 0x7f740406e460 2024-03-26T19:04:48.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.261+0000 7f741f2c4700 1 -- 172.21.15.102:0/1663667254 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f740406e460 msgr2=0x7f7404070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:48.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.261+0000 7f741f2c4700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f740406e460 0x7f7404070910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f74140059d0 tx=0x7f7414005960 comp rx=0 tx=0).stop 2024-03-26T19:04:48.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.261+0000 7f741f2c4700 1 -- 172.21.15.102:0/1663667254 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74181035a0 msgr2=0x7f74180fec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:48.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.261+0000 7f741f2c4700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74181035a0 0x7f74180fec30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f740800b4b0 tx=0x7f740800b870 comp rx=0 tx=0).stop 2024-03-26T19:04:48.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.261+0000 7f741f2c4700 1 -- 172.21.15.102:0/1663667254 shutdown_connections 2024-03-26T19:04:48.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.261+0000 7f741f2c4700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7418100c80 0x7f74181005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:48.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.261+0000 7f741f2c4700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f740406e460 0x7f7404070910 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:48.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.261+0000 7f741f2c4700 1 --2- 172.21.15.102:0/1663667254 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74181035a0 0x7f74180fec30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:48.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.261+0000 7f741f2c4700 1 -- 172.21.15.102:0/1663667254 >> 172.21.15.102:0/1663667254 conn(0x7f74180fa5e0 msgr2=0x7f74180fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:48.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.261+0000 7f741f2c4700 1 -- 172.21.15.102:0/1663667254 shutdown_connections 2024-03-26T19:04:48.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:48.261+0000 7f741f2c4700 1 -- 172.21.15.102:0/1663667254 wait complete. 2024-03-26T19:04:48.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph orch ps' 2024-03-26T19:04:49.204 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T19:04:49.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:48 smithi190 ceph-mon[65689]: pgmap v282: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:49.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:48 smithi190 ceph-mon[65689]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:49.319 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:48 smithi102 ceph-mon[78322]: pgmap v282: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:49.320 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:48 smithi102 ceph-mon[78322]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.854+0000 7f5260374700 1 -- 172.21.15.102:0/1883249125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580ffd30 msgr2=0x7f5258103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.854+0000 7f5260374700 1 --2- 172.21.15.102:0/1883249125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580ffd30 0x7f5258103ce0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5248009910 tx=0x7f5248009c20 comp rx=0 tx=0).stop 2024-03-26T19:04:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.854+0000 7f5260374700 1 -- 172.21.15.102:0/1883249125 shutdown_connections 2024-03-26T19:04:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.854+0000 7f5260374700 1 --2- 172.21.15.102:0/1883249125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580ffd30 0x7f5258103ce0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.854+0000 7f5260374700 1 --2- 172.21.15.102:0/1883249125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52580ff390 0x7f52580ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.854+0000 7f5260374700 1 -- 172.21.15.102:0/1883249125 >> 172.21.15.102:0/1883249125 conn(0x7f5258074f50 msgr2=0x7f5258075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.854+0000 7f5260374700 1 -- 172.21.15.102:0/1883249125 shutdown_connections 2024-03-26T19:04:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.854+0000 7f5260374700 1 -- 172.21.15.102:0/1883249125 wait complete. 2024-03-26T19:04:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.855+0000 7f5260374700 1 Processor -- start 2024-03-26T19:04:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.855+0000 7f5260374700 1 -- start start 2024-03-26T19:04:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.855+0000 7f5260374700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52580ff390 0x7f5258194970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.855+0000 7f5260374700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580ffd30 0x7f5258194eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.855+0000 7f5260374700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52581953f0 con 0x7f52580ffd30 2024-03-26T19:04:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.855+0000 7f5260374700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5258195560 con 0x7f52580ff390 2024-03-26T19:04:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.855+0000 7f525d90f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580ffd30 0x7f5258194eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.855+0000 7f525d90f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580ffd30 0x7f5258194eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48374/0 (socket says 172.21.15.102:48374) 2024-03-26T19:04:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.855+0000 7f525d90f700 1 -- 172.21.15.102:0/3196597118 learned_addr learned my addr 172.21.15.102:0/3196597118 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:04:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.855+0000 7f525e110700 1 --2- 172.21.15.102:0/3196597118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52580ff390 0x7f5258194970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.856+0000 7f525d90f700 1 -- 172.21.15.102:0/3196597118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52580ff390 msgr2=0x7f5258194970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.856+0000 7f525d90f700 1 --2- 172.21.15.102:0/3196597118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52580ff390 0x7f5258194970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.856+0000 7f525d90f700 1 -- 172.21.15.102:0/3196597118 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52480095f0 con 0x7f52580ffd30 2024-03-26T19:04:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.856+0000 7f525d90f700 1 --2- 172.21.15.102:0/3196597118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580ffd30 0x7f5258194eb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f52480092e0 tx=0x7f524800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.856+0000 7f524f7fe700 1 -- 172.21.15.102:0/3196597118 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f524801d070 con 0x7f52580ffd30 2024-03-26T19:04:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.856+0000 7f524f7fe700 1 -- 172.21.15.102:0/3196597118 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f524800f460 con 0x7f52580ffd30 2024-03-26T19:04:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.856+0000 7f524f7fe700 1 -- 172.21.15.102:0/3196597118 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52480175d0 con 0x7f52580ffd30 2024-03-26T19:04:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.857+0000 7f5260374700 1 -- 172.21.15.102:0/3196597118 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52581957e0 con 0x7f52580ffd30 2024-03-26T19:04:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.857+0000 7f5260374700 1 -- 172.21.15.102:0/3196597118 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52581a2260 con 0x7f52580ffd30 2024-03-26T19:04:50.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.858+0000 7f5260374700 1 -- 172.21.15.102:0/3196597118 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5258107600 con 0x7f52580ffd30 2024-03-26T19:04:50.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.858+0000 7f524f7fe700 1 -- 172.21.15.102:0/3196597118 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f524800f5d0 con 0x7f52580ffd30 2024-03-26T19:04:50.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.859+0000 7f524f7fe700 1 --2- 172.21.15.102:0/3196597118 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e460 0x7f5244070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:50.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.859+0000 7f525e110700 1 --2- 172.21.15.102:0/3196597118 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e460 0x7f5244070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:50.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.859+0000 7f524f7fe700 1 -- 172.21.15.102:0/3196597118 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f524808f6a0 con 0x7f52580ffd30 2024-03-26T19:04:50.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.859+0000 7f525e110700 1 --2- 172.21.15.102:0/3196597118 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e460 0x7f5244070910 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5254005950 tx=0x7f525400a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:50.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:50.867+0000 7f524f7fe700 1 -- 172.21.15.102:0/3196597118 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f524805c3b0 con 0x7f52580ffd30 2024-03-26T19:04:51.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.105+0000 7f5260374700 1 -- 172.21.15.102:0/3196597118 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f52581957e0 con 0x7f524406e460 2024-03-26T19:04:51.101 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:50 smithi102 ceph-mon[78322]: pgmap v283: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:51.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.119+0000 7f524f7fe700 1 -- 172.21.15.102:0/3196597118 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f52581957e0 con 0x7f524406e460 2024-03-26T19:04:51.116 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T19:04:51.116 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (46s) 17s ago 17m 15.5M - 0.25.0 c8568f914cd2 20e72e04e392 2024-03-26T19:04:51.116 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (104s) 17s ago 18m 8140k - 19.0.0-2311-g36051606 497bb87bc5da 2fb0a990cbf1 2024-03-26T19:04:51.116 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (98s) 77s ago 17m 7537k - 19.0.0-2311-g36051606 497bb87bc5da 893c37bef7ca 2024-03-26T19:04:51.116 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (7m) 17s ago 18m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T19:04:51.116 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (7m) 77s ago 17m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T19:04:51.116 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (22s) 17s ago 17m 51.0M - 9.4.12 2bacad6d85d8 d1a27623128a 2024-03-26T19:04:51.116 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (2m) 17s ago 12m 18.0M - 19.0.0-2311-g36051606 497bb87bc5da c530ed7dbd78 2024-03-26T19:04:51.116 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (2m) 17s ago 12m 79.8M - 19.0.0-2311-g36051606 497bb87bc5da 63b8127a3849 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (118s) 77s ago 12m 13.7M - 19.0.0-2311-g36051606 497bb87bc5da 25df8033593e 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (2m) 77s ago 12m 15.1M - 19.0.0-2311-g36051606 497bb87bc5da 37260789b9ec 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (9m) 17s ago 19m 520M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (8m) 77s ago 16m 452M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (8m) 17s ago 19m 55.3M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (8m) 77s ago 16m 42.3M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (84s) 17s ago 17m 8346k - 1.5.0 0da6a335fe13 9992a3ee4208 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (80s) 77s ago 16m 3632k - 1.5.0 0da6a335fe13 6d6186a60a00 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (6m) 17s ago 16m 107M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (6m) 17s ago 15m 116M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (5m) 17s ago 15m 98.6M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (5m) 77s ago 14m 122M 2240M 19.0.0-2311-g36051606 497bb87bc5da 03955331c962 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (4m) 77s ago 14m 112M 2240M 19.0.0-2311-g36051606 497bb87bc5da 23e140cd8961 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (3m) 77s ago 13m 103M 2240M 19.0.0-2311-g36051606 497bb87bc5da 60afb0e4da18 2024-03-26T19:04:51.117 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (62s) 17s ago 17m 64.5M - 2.43.0 a07b618ecd1d 9b1b2a7dbd69 2024-03-26T19:04:51.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.121+0000 7f5260374700 1 -- 172.21.15.102:0/3196597118 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e460 msgr2=0x7f5244070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:51.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.121+0000 7f5260374700 1 --2- 172.21.15.102:0/3196597118 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e460 0x7f5244070910 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5254005950 tx=0x7f525400a300 comp rx=0 tx=0).stop 2024-03-26T19:04:51.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.121+0000 7f5260374700 1 -- 172.21.15.102:0/3196597118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580ffd30 msgr2=0x7f5258194eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:51.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.121+0000 7f5260374700 1 --2- 172.21.15.102:0/3196597118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580ffd30 0x7f5258194eb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f52480092e0 tx=0x7f524800bd70 comp rx=0 tx=0).stop 2024-03-26T19:04:51.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.122+0000 7f5260374700 1 -- 172.21.15.102:0/3196597118 shutdown_connections 2024-03-26T19:04:51.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.122+0000 7f5260374700 1 --2- 172.21.15.102:0/3196597118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52580ff390 0x7f5258194970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:51.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.122+0000 7f5260374700 1 --2- 172.21.15.102:0/3196597118 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e460 0x7f5244070910 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:51.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.122+0000 7f5260374700 1 --2- 172.21.15.102:0/3196597118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580ffd30 0x7f5258194eb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:51.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.122+0000 7f5260374700 1 -- 172.21.15.102:0/3196597118 >> 172.21.15.102:0/3196597118 conn(0x7f5258074f50 msgr2=0x7f52580fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:51.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.122+0000 7f5260374700 1 -- 172.21.15.102:0/3196597118 shutdown_connections 2024-03-26T19:04:51.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:51.122+0000 7f5260374700 1 -- 172.21.15.102:0/3196597118 wait complete. 2024-03-26T19:04:51.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:50 smithi190 ceph-mon[65689]: pgmap v283: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:51.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph orch upgrade status' 2024-03-26T19:04:51.972 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T19:04:52.252 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:52 smithi102 ceph-mon[78322]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:52.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:52 smithi190 ceph-mon[65689]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:53.058 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:53 smithi102 ceph-mon[78322]: pgmap v284: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:53.058 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:04:53.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:53 smithi190 ceph-mon[65689]: pgmap v284: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:53.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:04:53.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.632+0000 7f8c46530700 1 -- 172.21.15.102:0/575449666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c400ffc60 msgr2=0x7f8c40103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:53.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.632+0000 7f8c46530700 1 --2- 172.21.15.102:0/575449666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c400ffc60 0x7f8c40103c30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009910 tx=0x7f8c34009c20 comp rx=0 tx=0).stop 2024-03-26T19:04:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.632+0000 7f8c46530700 1 -- 172.21.15.102:0/575449666 shutdown_connections 2024-03-26T19:04:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.632+0000 7f8c46530700 1 --2- 172.21.15.102:0/575449666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c400ffc60 0x7f8c40103c30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.632+0000 7f8c46530700 1 --2- 172.21.15.102:0/575449666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c400ff2c0 0x7f8c400ff690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.632+0000 7f8c46530700 1 -- 172.21.15.102:0/575449666 >> 172.21.15.102:0/575449666 conn(0x7f8c400fab80 msgr2=0x7f8c400fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.632+0000 7f8c46530700 1 -- 172.21.15.102:0/575449666 shutdown_connections 2024-03-26T19:04:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.632+0000 7f8c46530700 1 -- 172.21.15.102:0/575449666 wait complete. 2024-03-26T19:04:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.633+0000 7f8c46530700 1 Processor -- start 2024-03-26T19:04:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.633+0000 7f8c46530700 1 -- start start 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c46530700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c400ff2c0 0x7f8c40196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c46530700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c400ffc60 0x7f8c401970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c46530700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c40197700 con 0x7f8c400ff2c0 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c46530700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c40197870 con 0x7f8c400ffc60 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c3ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c400ff2c0 0x7f8c40196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c3ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c400ff2c0 0x7f8c40196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48398/0 (socket says 172.21.15.102:48398) 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c3ffff700 1 -- 172.21.15.102:0/3760936068 learned_addr learned my addr 172.21.15.102:0/3760936068 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c3f7fe700 1 --2- 172.21.15.102:0/3760936068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c400ffc60 0x7f8c401970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c3ffff700 1 -- 172.21.15.102:0/3760936068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c400ffc60 msgr2=0x7f8c401970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c3ffff700 1 --2- 172.21.15.102:0/3760936068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c400ffc60 0x7f8c401970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c3ffff700 1 -- 172.21.15.102:0/3760936068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c340095f0 con 0x7f8c400ff2c0 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c3ffff700 1 --2- 172.21.15.102:0/3760936068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c400ff2c0 0x7f8c40196ba0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8c3000d900 tx=0x7f8c3000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c3d7fa700 1 -- 172.21.15.102:0/3760936068 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c300041f0 con 0x7f8c400ff2c0 2024-03-26T19:04:53.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c3d7fa700 1 -- 172.21.15.102:0/3760936068 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c30004cf0 con 0x7f8c400ff2c0 2024-03-26T19:04:53.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.634+0000 7f8c3d7fa700 1 -- 172.21.15.102:0/3760936068 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3000b750 con 0x7f8c400ff2c0 2024-03-26T19:04:53.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.635+0000 7f8c46530700 1 -- 172.21.15.102:0/3760936068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4019fc10 con 0x7f8c400ff2c0 2024-03-26T19:04:53.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.635+0000 7f8c46530700 1 -- 172.21.15.102:0/3760936068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c401a00d0 con 0x7f8c400ff2c0 2024-03-26T19:04:53.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.636+0000 7f8c46530700 1 -- 172.21.15.102:0/3760936068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c40107630 con 0x7f8c400ff2c0 2024-03-26T19:04:53.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.637+0000 7f8c3d7fa700 1 -- 172.21.15.102:0/3760936068 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8c30004750 con 0x7f8c400ff2c0 2024-03-26T19:04:53.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.637+0000 7f8c3d7fa700 1 --2- 172.21.15.102:0/3760936068 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8c2806e460 0x7f8c28070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:53.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.637+0000 7f8c3d7fa700 1 -- 172.21.15.102:0/3760936068 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8c3008e230 con 0x7f8c400ff2c0 2024-03-26T19:04:53.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.637+0000 7f8c3f7fe700 1 --2- 172.21.15.102:0/3760936068 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8c2806e460 0x7f8c28070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:53.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.638+0000 7f8c3f7fe700 1 --2- 172.21.15.102:0/3760936068 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8c2806e460 0x7f8c28070910 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8c34006010 tx=0x7f8c3401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:53.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.642+0000 7f8c3d7fa700 1 -- 172.21.15.102:0/3760936068 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8c30057410 con 0x7f8c400ff2c0 2024-03-26T19:04:53.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.872+0000 7f8c46530700 1 -- 172.21.15.102:0/3760936068 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c4002cd10 con 0x7f8c2806e460 2024-03-26T19:04:53.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.873+0000 7f8c3d7fa700 1 -- 172.21.15.102:0/3760936068 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f8c4002cd10 con 0x7f8c2806e460 2024-03-26T19:04:53.869 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:04:53.869 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": null, 2024-03-26T19:04:53.869 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": false, 2024-03-26T19:04:53.869 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "", 2024-03-26T19:04:53.869 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-03-26T19:04:53.870 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": null, 2024-03-26T19:04:53.870 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "", 2024-03-26T19:04:53.870 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-03-26T19:04:53.870 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:04:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.875+0000 7f8c46530700 1 -- 172.21.15.102:0/3760936068 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8c2806e460 msgr2=0x7f8c28070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.875+0000 7f8c46530700 1 --2- 172.21.15.102:0/3760936068 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8c2806e460 0x7f8c28070910 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8c34006010 tx=0x7f8c3401a040 comp rx=0 tx=0).stop 2024-03-26T19:04:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.875+0000 7f8c46530700 1 -- 172.21.15.102:0/3760936068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c400ff2c0 msgr2=0x7f8c40196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.875+0000 7f8c46530700 1 --2- 172.21.15.102:0/3760936068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c400ff2c0 0x7f8c40196ba0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8c3000d900 tx=0x7f8c3000dc10 comp rx=0 tx=0).stop 2024-03-26T19:04:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.875+0000 7f8c46530700 1 -- 172.21.15.102:0/3760936068 shutdown_connections 2024-03-26T19:04:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.875+0000 7f8c46530700 1 --2- 172.21.15.102:0/3760936068 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8c2806e460 0x7f8c28070910 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.875+0000 7f8c46530700 1 --2- 172.21.15.102:0/3760936068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c400ff2c0 0x7f8c40196ba0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.875+0000 7f8c46530700 1 --2- 172.21.15.102:0/3760936068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c400ffc60 0x7f8c401970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.876+0000 7f8c46530700 1 -- 172.21.15.102:0/3760936068 >> 172.21.15.102:0/3760936068 conn(0x7f8c400fab80 msgr2=0x7f8c400fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.876+0000 7f8c46530700 1 -- 172.21.15.102:0/3760936068 shutdown_connections 2024-03-26T19:04:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:53.876+0000 7f8c46530700 1 -- 172.21.15.102:0/3760936068 wait complete. 2024-03-26T19:04:55.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:55 smithi190 ceph-mon[65689]: pgmap v285: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:55.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:55 smithi190 ceph-mon[65689]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:55.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:55 smithi102 ceph-mon[78322]: pgmap v285: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:55.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:55 smithi102 ceph-mon[78322]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:04:57.129 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:57 smithi102 ceph-mon[78322]: pgmap v286: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:57.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph health detail' 2024-03-26T19:04:57.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:57 smithi190 ceph-mon[65689]: pgmap v286: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:57.469 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T19:04:59.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.096+0000 7f794dd33700 1 -- 172.21.15.102:0/2320177645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79480ff390 msgr2=0x7f79480ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:59.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.096+0000 7f794dd33700 1 --2- 172.21.15.102:0/2320177645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79480ff390 0x7f79480ff760 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7938009910 tx=0x7f7938009c20 comp rx=0 tx=0).stop 2024-03-26T19:04:59.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.096+0000 7f794dd33700 1 -- 172.21.15.102:0/2320177645 shutdown_connections 2024-03-26T19:04:59.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.096+0000 7f794dd33700 1 --2- 172.21.15.102:0/2320177645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79480ffd30 0x7f7948103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:59.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.096+0000 7f794dd33700 1 --2- 172.21.15.102:0/2320177645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79480ff390 0x7f79480ff760 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:59.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.096+0000 7f794dd33700 1 -- 172.21.15.102:0/2320177645 >> 172.21.15.102:0/2320177645 conn(0x7f7948074f50 msgr2=0x7f7948075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.096+0000 7f794dd33700 1 -- 172.21.15.102:0/2320177645 shutdown_connections 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.096+0000 7f794dd33700 1 -- 172.21.15.102:0/2320177645 wait complete. 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.097+0000 7f794dd33700 1 Processor -- start 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.097+0000 7f794dd33700 1 -- start start 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.097+0000 7f794dd33700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79480ff390 0x7f7948195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.097+0000 7f794dd33700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79480ffd30 0x7f7948196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.097+0000 7f794dd33700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7948196940 con 0x7f79480ffd30 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.097+0000 7f794dd33700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794819a6d0 con 0x7f79480ff390 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.097+0000 7f7946ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79480ffd30 0x7f7948196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.097+0000 7f7946ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79480ffd30 0x7f7948196260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34802/0 (socket says 172.21.15.102:34802) 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.097+0000 7f7946ffd700 1 -- 172.21.15.102:0/1001385566 learned_addr learned my addr 172.21.15.102:0/1001385566 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.097+0000 7f79477fe700 1 --2- 172.21.15.102:0/1001385566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79480ff390 0x7f7948195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.098+0000 7f7946ffd700 1 -- 172.21.15.102:0/1001385566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79480ff390 msgr2=0x7f7948195d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.098+0000 7f7946ffd700 1 --2- 172.21.15.102:0/1001385566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79480ff390 0x7f7948195d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.098+0000 7f7946ffd700 1 -- 172.21.15.102:0/1001385566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79380095f0 con 0x7f79480ffd30 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.098+0000 7f7946ffd700 1 --2- 172.21.15.102:0/1001385566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79480ffd30 0x7f7948196260 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f793c00d900 tx=0x7f793c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.098+0000 7f7944ff9700 1 -- 172.21.15.102:0/1001385566 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f793c009900 con 0x7f79480ffd30 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.098+0000 7f794dd33700 1 -- 172.21.15.102:0/1001385566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794819a9b0 con 0x7f79480ffd30 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.098+0000 7f7944ff9700 1 -- 172.21.15.102:0/1001385566 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f793c010460 con 0x7f79480ffd30 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.098+0000 7f7944ff9700 1 -- 172.21.15.102:0/1001385566 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f793c00f5f0 con 0x7f79480ffd30 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.098+0000 7f794dd33700 1 -- 172.21.15.102:0/1001385566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794819af00 con 0x7f79480ffd30 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.099+0000 7f794dd33700 1 -- 172.21.15.102:0/1001385566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794804ea50 con 0x7f79480ffd30 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.100+0000 7f7944ff9700 1 -- 172.21.15.102:0/1001385566 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f793c0105d0 con 0x7f79480ffd30 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.100+0000 7f7944ff9700 1 --2- 172.21.15.102:0/1001385566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f793006e460 0x7f7930070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.100+0000 7f7944ff9700 1 -- 172.21.15.102:0/1001385566 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f793c08e010 con 0x7f79480ffd30 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.100+0000 7f79477fe700 1 --2- 172.21.15.102:0/1001385566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f793006e460 0x7f7930070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:04:59.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.101+0000 7f79477fe700 1 --2- 172.21.15.102:0/1001385566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f793006e460 0x7f7930070910 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7938006010 tx=0x7f793801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:04:59.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.105+0000 7f7944ff9700 1 -- 172.21.15.102:0/1001385566 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f793c057170 con 0x7f79480ffd30 2024-03-26T19:04:59.102 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:04:59 smithi102 ceph-mon[78322]: pgmap v287: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:59.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:04:59 smithi190 ceph-mon[65689]: pgmap v287: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:04:59.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.399+0000 7f794dd33700 1 -- 172.21.15.102:0/1001385566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7948197080 con 0x7f79480ffd30 2024-03-26T19:04:59.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.399+0000 7f7944ff9700 1 -- 172.21.15.102:0/1001385566 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f793c05a830 con 0x7f79480ffd30 2024-03-26T19:04:59.396 INFO:teuthology.orchestra.run.smithi102.stdout:HEALTH_OK 2024-03-26T19:04:59.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.402+0000 7f794dd33700 1 -- 172.21.15.102:0/1001385566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f793006e460 msgr2=0x7f7930070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.402+0000 7f794dd33700 1 --2- 172.21.15.102:0/1001385566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f793006e460 0x7f7930070910 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7938006010 tx=0x7f793801a040 comp rx=0 tx=0).stop 2024-03-26T19:04:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.402+0000 7f794dd33700 1 -- 172.21.15.102:0/1001385566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79480ffd30 msgr2=0x7f7948196260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:04:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.402+0000 7f794dd33700 1 --2- 172.21.15.102:0/1001385566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79480ffd30 0x7f7948196260 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f793c00d900 tx=0x7f793c00dc10 comp rx=0 tx=0).stop 2024-03-26T19:04:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.402+0000 7f794dd33700 1 -- 172.21.15.102:0/1001385566 shutdown_connections 2024-03-26T19:04:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.402+0000 7f794dd33700 1 --2- 172.21.15.102:0/1001385566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79480ff390 0x7f7948195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.402+0000 7f794dd33700 1 --2- 172.21.15.102:0/1001385566 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f793006e460 0x7f7930070910 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.402+0000 7f794dd33700 1 --2- 172.21.15.102:0/1001385566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79480ffd30 0x7f7948196260 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:04:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.402+0000 7f794dd33700 1 -- 172.21.15.102:0/1001385566 >> 172.21.15.102:0/1001385566 conn(0x7f7948074f50 msgr2=0x7f7948068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:04:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.402+0000 7f794dd33700 1 -- 172.21.15.102:0/1001385566 shutdown_connections 2024-03-26T19:04:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:04:59.402+0000 7f794dd33700 1 -- 172.21.15.102:0/1001385566 wait complete. 2024-03-26T19:05:00.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph versions' 2024-03-26T19:05:00.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:00 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1001385566' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:05:00.308 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T19:05:00.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:00 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1001385566' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T19:05:01.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:01 smithi190 ceph-mon[65689]: pgmap v288: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:01.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:01 smithi102 ceph-mon[78322]: pgmap v288: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:01.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.967+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/897895389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1e0103860 msgr2=0x7fe1e0103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:05:01.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.967+0000 7fe1e4e0c700 1 --2- 172.21.15.102:0/897895389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1e0103860 0x7fe1e0103c30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0009910 tx=0x7fe1d0009c20 comp rx=0 tx=0).stop 2024-03-26T19:05:01.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.967+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/897895389 shutdown_connections 2024-03-26T19:05:01.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.967+0000 7fe1e4e0c700 1 --2- 172.21.15.102:0/897895389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e00ff320 0x7fe1e00ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:05:01.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.967+0000 7fe1e4e0c700 1 --2- 172.21.15.102:0/897895389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1e0103860 0x7fe1e0103c30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:05:01.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.967+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/897895389 >> 172.21.15.102:0/897895389 conn(0x7fe1e00748b0 msgr2=0x7fe1e0074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:05:01.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.967+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/897895389 shutdown_connections 2024-03-26T19:05:01.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.967+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/897895389 wait complete. 2024-03-26T19:05:01.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.968+0000 7fe1e4e0c700 1 Processor -- start 2024-03-26T19:05:01.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.968+0000 7fe1e4e0c700 1 -- start start 2024-03-26T19:05:01.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.968+0000 7fe1e4e0c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1e00ff320 0x7fe1e019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:05:01.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.968+0000 7fe1e4e0c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e0103860 0x7fe1e019af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.968+0000 7fe1e4e0c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e019b5d0 con 0x7fe1e00ff320 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.968+0000 7fe1e4e0c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e0194a80 con 0x7fe1e0103860 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.968+0000 7fe1de59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1e00ff320 0x7fe1e019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.968+0000 7fe1de59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1e00ff320 0x7fe1e019aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34814/0 (socket says 172.21.15.102:34814) 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.968+0000 7fe1de59c700 1 -- 172.21.15.102:0/1985892133 learned_addr learned my addr 172.21.15.102:0/1985892133 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.969+0000 7fe1ddd9b700 1 --2- 172.21.15.102:0/1985892133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e0103860 0x7fe1e019af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.969+0000 7fe1de59c700 1 -- 172.21.15.102:0/1985892133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e0103860 msgr2=0x7fe1e019af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.969+0000 7fe1de59c700 1 --2- 172.21.15.102:0/1985892133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e0103860 0x7fe1e019af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.969+0000 7fe1de59c700 1 -- 172.21.15.102:0/1985892133 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d00095f0 con 0x7fe1e00ff320 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.969+0000 7fe1de59c700 1 --2- 172.21.15.102:0/1985892133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1e00ff320 0x7fe1e019aa00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0006010 tx=0x7fe1d000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.969+0000 7fe1cf7fe700 1 -- 172.21.15.102:0/1985892133 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1d001d070 con 0x7fe1e00ff320 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.969+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/1985892133 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e0194d00 con 0x7fe1e00ff320 2024-03-26T19:05:01.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.969+0000 7fe1cf7fe700 1 -- 172.21.15.102:0/1985892133 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1d000f460 con 0x7fe1e00ff320 2024-03-26T19:05:01.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.969+0000 7fe1cf7fe700 1 -- 172.21.15.102:0/1985892133 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1d0017630 con 0x7fe1e00ff320 2024-03-26T19:05:01.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.969+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/1985892133 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e01951f0 con 0x7fe1e00ff320 2024-03-26T19:05:01.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.970+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/1985892133 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e004ea50 con 0x7fe1e00ff320 2024-03-26T19:05:01.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.971+0000 7fe1cf7fe700 1 -- 172.21.15.102:0/1985892133 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe1d000f5d0 con 0x7fe1e00ff320 2024-03-26T19:05:01.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.971+0000 7fe1cf7fe700 1 --2- 172.21.15.102:0/1985892133 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe1c8070670 0x7fe1c8072b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:05:01.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.971+0000 7fe1cf7fe700 1 -- 172.21.15.102:0/1985892133 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fe1d008f900 con 0x7fe1e00ff320 2024-03-26T19:05:01.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.971+0000 7fe1ddd9b700 1 --2- 172.21.15.102:0/1985892133 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe1c8070670 0x7fe1c8072b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:05:01.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.972+0000 7fe1ddd9b700 1 --2- 172.21.15.102:0/1985892133 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe1c8070670 0x7fe1c8072b20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4005950 tx=0x7fe1d40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:05:01.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:01.976+0000 7fe1cf7fe700 1 -- 172.21.15.102:0/1985892133 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe1d005c590 con 0x7fe1e00ff320 2024-03-26T19:05:02.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.278+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/1985892133 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe1e01955f0 con 0x7fe1e00ff320 2024-03-26T19:05:02.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.278+0000 7fe1cf7fe700 1 -- 172.21.15.102:0/1985892133 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe1d005c120 con 0x7fe1e00ff320 2024-03-26T19:05:02.275 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:05:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T19:05:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:05:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:05:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T19:05:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:05:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:05:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T19:05:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 6 2024-03-26T19:05:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:05:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T19:05:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 4 2024-03-26T19:05:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:05:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T19:05:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 14 2024-03-26T19:05:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T19:05:02.276 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:05:02.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.281+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/1985892133 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe1c8070670 msgr2=0x7fe1c8072b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:05:02.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.281+0000 7fe1e4e0c700 1 --2- 172.21.15.102:0/1985892133 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe1c8070670 0x7fe1c8072b20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4005950 tx=0x7fe1d40058e0 comp rx=0 tx=0).stop 2024-03-26T19:05:02.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.281+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/1985892133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1e00ff320 msgr2=0x7fe1e019aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:05:02.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.281+0000 7fe1e4e0c700 1 --2- 172.21.15.102:0/1985892133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1e00ff320 0x7fe1e019aa00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0006010 tx=0x7fe1d000be20 comp rx=0 tx=0).stop 2024-03-26T19:05:02.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.281+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/1985892133 shutdown_connections 2024-03-26T19:05:02.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.281+0000 7fe1e4e0c700 1 --2- 172.21.15.102:0/1985892133 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe1c8070670 0x7fe1c8072b20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:05:02.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.281+0000 7fe1e4e0c700 1 --2- 172.21.15.102:0/1985892133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1e00ff320 0x7fe1e019aa00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:05:02.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.281+0000 7fe1e4e0c700 1 --2- 172.21.15.102:0/1985892133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e0103860 0x7fe1e019af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:05:02.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.281+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/1985892133 >> 172.21.15.102:0/1985892133 conn(0x7fe1e00748b0 msgr2=0x7fe1e00fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:05:02.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.281+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/1985892133 shutdown_connections 2024-03-26T19:05:02.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:05:02.281+0000 7fe1e4e0c700 1 -- 172.21.15.102:0/1985892133 wait complete. 2024-03-26T19:05:02.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-26T19:05:03.132 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T19:05:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:03 smithi190 ceph-mon[65689]: pgmap v289: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:03.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:03 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1985892133' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:05:03.296 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:03 smithi102 ceph-mon[78322]: pgmap v289: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:03.296 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:03 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1985892133' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:05:04.748 INFO:teuthology.orchestra.run.smithi102.stdout:wait for servicemap items w/ changing names to refresh 2024-03-26T19:05:05.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:05 smithi190 ceph-mon[65689]: pgmap v290: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:05.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:05 smithi102 ceph-mon[78322]: pgmap v290: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:05.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'sleep 60' 2024-03-26T19:05:05.655 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T19:05:07.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:07 smithi190 ceph-mon[65689]: pgmap v291: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:07.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:07 smithi102 ceph-mon[78322]: pgmap v291: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:08.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:05:08.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:05:09.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:09 smithi190 ceph-mon[65689]: pgmap v292: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:09.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:09 smithi102 ceph-mon[78322]: pgmap v292: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:11.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:11 smithi190 ceph-mon[65689]: pgmap v293: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:11.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:11 smithi102 ceph-mon[78322]: pgmap v293: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:13.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:13 smithi102 ceph-mon[78322]: pgmap v294: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:13.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:13 smithi190 ceph-mon[65689]: pgmap v294: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:15.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:15 smithi102 ceph-mon[78322]: pgmap v295: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:15.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:15 smithi190 ceph-mon[65689]: pgmap v295: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:17.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:17 smithi102 ceph-mon[78322]: pgmap v296: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:17.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:17 smithi190 ceph-mon[65689]: pgmap v296: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:19.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:19 smithi190 ceph-mon[65689]: pgmap v297: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:19.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:19 smithi102 ceph-mon[78322]: pgmap v297: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:21.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:21 smithi102 ceph-mon[78322]: pgmap v298: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:21.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:21 smithi190 ceph-mon[65689]: pgmap v298: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:23.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:23 smithi102 ceph-mon[78322]: pgmap v299: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:23.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:05:23.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:23 smithi190 ceph-mon[65689]: pgmap v299: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:23.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:05:25.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:25 smithi102 ceph-mon[78322]: pgmap v300: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:25.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:25 smithi190 ceph-mon[65689]: pgmap v300: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:27.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:27 smithi102 ceph-mon[78322]: pgmap v301: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:27.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:27 smithi190 ceph-mon[65689]: pgmap v301: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:29.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:29 smithi190 ceph-mon[65689]: pgmap v302: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:29.341 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:29 smithi102 ceph-mon[78322]: pgmap v302: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:31.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:31 smithi102 ceph-mon[78322]: pgmap v303: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:31.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:31 smithi190 ceph-mon[65689]: pgmap v303: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:33.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:33 smithi102 ceph-mon[78322]: pgmap v304: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:33.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:33 smithi190 ceph-mon[65689]: pgmap v304: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:35.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:35 smithi102 ceph-mon[78322]: pgmap v305: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:35.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:35 smithi190 ceph-mon[65689]: pgmap v305: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:37.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:37 smithi102 ceph-mon[78322]: pgmap v306: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:37.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:37 smithi190 ceph-mon[65689]: pgmap v306: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:38.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:05:38.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:05:39.283 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:39 smithi190 ceph-mon[65689]: pgmap v307: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:39.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:39 smithi102 ceph-mon[78322]: pgmap v307: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:40.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:40 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:05:40.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:40 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T19:05:41.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:41 smithi102 ceph-mon[78322]: pgmap v308: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:41.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:05:41.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:05:41.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:41 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:05:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:41 smithi190 ceph-mon[65689]: pgmap v308: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T19:05:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T19:05:41.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:41 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' 2024-03-26T19:05:43.336 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:43 smithi190 ceph-mon[65689]: pgmap v309: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:43.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:43 smithi102 ceph-mon[78322]: pgmap v309: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:45.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:45 smithi102 ceph-mon[78322]: pgmap v310: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:45.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:45 smithi190 ceph-mon[65689]: pgmap v310: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:47.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:47 smithi102 ceph-mon[78322]: pgmap v311: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:47.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:47 smithi190 ceph-mon[65689]: pgmap v311: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:49.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:49 smithi102 ceph-mon[78322]: pgmap v312: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:49.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:49 smithi190 ceph-mon[65689]: pgmap v312: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:51.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:51 smithi102 ceph-mon[78322]: pgmap v313: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:51.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:51 smithi190 ceph-mon[65689]: pgmap v313: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:53.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:53 smithi102 ceph-mon[78322]: pgmap v314: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:53.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:53 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:05:53.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:53 smithi190 ceph-mon[65689]: pgmap v314: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:53.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:53 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:05:55.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:55 smithi102 ceph-mon[78322]: pgmap v315: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:55.533 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:55 smithi190 ceph-mon[65689]: pgmap v315: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:57.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:57 smithi102 ceph-mon[78322]: pgmap v316: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:57.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:57 smithi190 ceph-mon[65689]: pgmap v316: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:59.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:05:59 smithi102 ceph-mon[78322]: pgmap v317: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:05:59.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:05:59 smithi190 ceph-mon[65689]: pgmap v317: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:01.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:01 smithi102 ceph-mon[78322]: pgmap v318: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:01.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:01 smithi190 ceph-mon[65689]: pgmap v318: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:03.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:03 smithi102 ceph-mon[78322]: pgmap v319: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:03.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:03 smithi190 ceph-mon[65689]: pgmap v319: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:05.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:05 smithi102 ceph-mon[78322]: pgmap v320: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:05.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:05 smithi190 ceph-mon[65689]: pgmap v320: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:07.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:07 smithi102 ceph-mon[78322]: pgmap v321: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:07.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:07 smithi190 ceph-mon[65689]: pgmap v321: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:07.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph orch ps' 2024-03-26T19:06:07.969 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T19:06:08.344 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:08 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:06:08.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:08 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:06:09.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:09 smithi102 ceph-mon[78322]: pgmap v322: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:09.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:09 smithi190 ceph-mon[65689]: pgmap v322: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.776+0000 7ff4db508700 1 -- 172.21.15.102:0/1551416863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4d40ffd30 msgr2=0x7ff4d4103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.776+0000 7ff4db508700 1 --2- 172.21.15.102:0/1551416863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4d40ffd30 0x7ff4d4103ce0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009910 tx=0x7ff4c4009c20 comp rx=0 tx=0).stop 2024-03-26T19:06:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.776+0000 7ff4db508700 1 -- 172.21.15.102:0/1551416863 shutdown_connections 2024-03-26T19:06:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.776+0000 7ff4db508700 1 --2- 172.21.15.102:0/1551416863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4d40ffd30 0x7ff4d4103ce0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.776+0000 7ff4db508700 1 --2- 172.21.15.102:0/1551416863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4d40ff390 0x7ff4d40ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.776+0000 7ff4db508700 1 -- 172.21.15.102:0/1551416863 >> 172.21.15.102:0/1551416863 conn(0x7ff4d4074f50 msgr2=0x7ff4d4075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.776+0000 7ff4db508700 1 -- 172.21.15.102:0/1551416863 shutdown_connections 2024-03-26T19:06:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.776+0000 7ff4db508700 1 -- 172.21.15.102:0/1551416863 wait complete. 2024-03-26T19:06:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.777+0000 7ff4db508700 1 Processor -- start 2024-03-26T19:06:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.777+0000 7ff4db508700 1 -- start start 2024-03-26T19:06:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.777+0000 7ff4db508700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4d40ff390 0x7ff4d406fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.777+0000 7ff4db508700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4d40ffd30 0x7ff4d406ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.777+0000 7ff4db508700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d4072b70 con 0x7ff4d40ff390 2024-03-26T19:06:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.777+0000 7ff4db508700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d4072cb0 con 0x7ff4d40ffd30 2024-03-26T19:06:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.777+0000 7ff4d92a4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4d40ff390 0x7ff4d406fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4d92a4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4d40ff390 0x7ff4d406fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46572/0 (socket says 172.21.15.102:46572) 2024-03-26T19:06:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4d92a4700 1 -- 172.21.15.102:0/2393825964 learned_addr learned my addr 172.21.15.102:0/2393825964 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4d8aa3700 1 --2- 172.21.15.102:0/2393825964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4d40ffd30 0x7ff4d406ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4d92a4700 1 -- 172.21.15.102:0/2393825964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4d40ffd30 msgr2=0x7ff4d406ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4d92a4700 1 --2- 172.21.15.102:0/2393825964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4d40ffd30 0x7ff4d406ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4d92a4700 1 -- 172.21.15.102:0/2393825964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c40095f0 con 0x7ff4d40ff390 2024-03-26T19:06:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4d92a4700 1 --2- 172.21.15.102:0/2393825964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4d40ff390 0x7ff4d406fa60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000d8d0 tx=0x7ff4d000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4ca7fc700 1 -- 172.21.15.102:0/2393825964 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4d00098a0 con 0x7ff4d40ff390 2024-03-26T19:06:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4ca7fc700 1 -- 172.21.15.102:0/2393825964 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4d0010460 con 0x7ff4d40ff390 2024-03-26T19:06:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4ca7fc700 1 -- 172.21.15.102:0/2393825964 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4d000f5f0 con 0x7ff4d40ff390 2024-03-26T19:06:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4db508700 1 -- 172.21.15.102:0/2393825964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d40704e0 con 0x7ff4d40ff390 2024-03-26T19:06:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.778+0000 7ff4db508700 1 -- 172.21.15.102:0/2393825964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d406d850 con 0x7ff4d40ff390 2024-03-26T19:06:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.779+0000 7ff4db508700 1 -- 172.21.15.102:0/2393825964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d4107600 con 0x7ff4d40ff390 2024-03-26T19:06:09.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.780+0000 7ff4ca7fc700 1 -- 172.21.15.102:0/2393825964 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff4d0009a00 con 0x7ff4d40ff390 2024-03-26T19:06:09.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.781+0000 7ff4ca7fc700 1 --2- 172.21.15.102:0/2393825964 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff4c006e460 0x7ff4c0070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:09.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.781+0000 7ff4ca7fc700 1 -- 172.21.15.102:0/2393825964 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7ff4d008e300 con 0x7ff4d40ff390 2024-03-26T19:06:09.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.781+0000 7ff4d8aa3700 1 --2- 172.21.15.102:0/2393825964 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff4c006e460 0x7ff4c0070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:09.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.781+0000 7ff4d8aa3700 1 --2- 172.21.15.102:0/2393825964 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff4c006e460 0x7ff4c0070910 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4006010 tx=0x7ff4c401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:09.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:09.786+0000 7ff4ca7fc700 1 -- 172.21.15.102:0/2393825964 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff4d0057460 con 0x7ff4d40ff390 2024-03-26T19:06:10.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.018+0000 7ff4db508700 1 -- 172.21.15.102:0/2393825964 --> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4d402cca0 con 0x7ff4c006e460 2024-03-26T19:06:10.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.029+0000 7ff4ca7fc700 1 -- 172.21.15.102:0/2393825964 <== mgr.34102 v2:172.21.15.102:6800/4269744347 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7ff4d402cca0 con 0x7ff4c006e460 2024-03-26T19:06:10.026 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.smithi102 smithi102 *:9093,9094 running (2m) 96s ago 19m 15.5M - 0.25.0 c8568f914cd2 20e72e04e392 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi102 smithi102 running (3m) 96s ago 19m 8140k - 19.0.0-2311-g36051606 497bb87bc5da 2fb0a990cbf1 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-exporter.smithi190 smithi190 running (2m) 2m ago 18m 7537k - 19.0.0-2311-g36051606 497bb87bc5da 893c37bef7ca 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi102 smithi102 running (8m) 96s ago 19m 6665k - 19.0.0-2311-g36051606 497bb87bc5da c12387a5d9f6 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:crash.smithi190 smithi190 running (8m) 2m ago 18m 6656k - 19.0.0-2311-g36051606 497bb87bc5da 511462a6de49 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.smithi102 smithi102 *:3000 running (101s) 96s ago 18m 51.0M - 9.4.12 2bacad6d85d8 d1a27623128a 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.hrbrsp smithi102 running (3m) 96s ago 13m 18.0M - 19.0.0-2311-g36051606 497bb87bc5da c530ed7dbd78 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi102.tvsqaa smithi102 running (4m) 96s ago 13m 79.8M - 19.0.0-2311-g36051606 497bb87bc5da 63b8127a3849 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.hwsvdg smithi190 running (3m) 2m ago 13m 13.7M - 19.0.0-2311-g36051606 497bb87bc5da 25df8033593e 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:mds.cephfs.smithi190.vykizb smithi190 running (3m) 2m ago 13m 15.1M - 19.0.0-2311-g36051606 497bb87bc5da 37260789b9ec 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi102.rsfpnv smithi102 *:8443,9283,8765 running (10m) 96s ago 20m 520M - 19.0.0-2311-g36051606 497bb87bc5da a6c5da68ca06 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.smithi190.zrixos smithi190 *:8443,9283,8765 running (9m) 2m ago 18m 452M - 19.0.0-2311-g36051606 497bb87bc5da 956ab3e89a5e 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi102 smithi102 running (9m) 96s ago 20m 55.3M 2048M 19.0.0-2311-g36051606 497bb87bc5da 9aaa70093760 2024-03-26T19:06:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:mon.smithi190 smithi190 running (9m) 2m ago 18m 42.3M 2048M 19.0.0-2311-g36051606 497bb87bc5da e80478bd14bd 2024-03-26T19:06:10.028 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi102 smithi102 *:9100 running (2m) 96s ago 19m 8346k - 1.5.0 0da6a335fe13 9992a3ee4208 2024-03-26T19:06:10.028 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.smithi190 smithi190 *:9100 running (2m) 2m ago 18m 3632k - 1.5.0 0da6a335fe13 6d6186a60a00 2024-03-26T19:06:10.028 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (8m) 96s ago 17m 107M 1216M 19.0.0-2311-g36051606 497bb87bc5da aec2fe1c7a8b 2024-03-26T19:06:10.028 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (7m) 96s ago 16m 116M 1216M 19.0.0-2311-g36051606 497bb87bc5da 3149459e0c02 2024-03-26T19:06:10.028 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (6m) 96s ago 16m 98.6M 1216M 19.0.0-2311-g36051606 497bb87bc5da 60206f27cb55 2024-03-26T19:06:10.028 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi190 running (6m) 2m ago 16m 122M 2240M 19.0.0-2311-g36051606 497bb87bc5da 03955331c962 2024-03-26T19:06:10.028 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi190 running (5m) 2m ago 15m 112M 2240M 19.0.0-2311-g36051606 497bb87bc5da 23e140cd8961 2024-03-26T19:06:10.028 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi190 running (5m) 2m ago 15m 103M 2240M 19.0.0-2311-g36051606 497bb87bc5da 60afb0e4da18 2024-03-26T19:06:10.028 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.smithi102 smithi102 *:9095 running (2m) 96s ago 18m 64.5M - 2.43.0 a07b618ecd1d 9b1b2a7dbd69 2024-03-26T19:06:10.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.032+0000 7ff4db508700 1 -- 172.21.15.102:0/2393825964 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff4c006e460 msgr2=0x7ff4c0070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:10.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.032+0000 7ff4db508700 1 --2- 172.21.15.102:0/2393825964 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff4c006e460 0x7ff4c0070910 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4006010 tx=0x7ff4c401a040 comp rx=0 tx=0).stop 2024-03-26T19:06:10.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.032+0000 7ff4db508700 1 -- 172.21.15.102:0/2393825964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4d40ff390 msgr2=0x7ff4d406fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:10.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.032+0000 7ff4db508700 1 --2- 172.21.15.102:0/2393825964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4d40ff390 0x7ff4d406fa60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000d8d0 tx=0x7ff4d000dbe0 comp rx=0 tx=0).stop 2024-03-26T19:06:10.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.032+0000 7ff4db508700 1 -- 172.21.15.102:0/2393825964 shutdown_connections 2024-03-26T19:06:10.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.032+0000 7ff4db508700 1 --2- 172.21.15.102:0/2393825964 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7ff4c006e460 0x7ff4c0070910 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:10.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.032+0000 7ff4db508700 1 --2- 172.21.15.102:0/2393825964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4d40ff390 0x7ff4d406fa60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:10.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.032+0000 7ff4db508700 1 --2- 172.21.15.102:0/2393825964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4d40ffd30 0x7ff4d406ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:10.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.032+0000 7ff4db508700 1 -- 172.21.15.102:0/2393825964 >> 172.21.15.102:0/2393825964 conn(0x7ff4d4074f50 msgr2=0x7ff4d40fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:10.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.032+0000 7ff4db508700 1 -- 172.21.15.102:0/2393825964 shutdown_connections 2024-03-26T19:06:10.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:10.032+0000 7ff4db508700 1 -- 172.21.15.102:0/2393825964 wait complete. 2024-03-26T19:06:10.653 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph versions' 2024-03-26T19:06:10.892 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T19:06:11.266 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:11 smithi102 ceph-mon[78322]: pgmap v323: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:11.266 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:11 smithi102 ceph-mon[78322]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:06:11.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:11 smithi190 ceph-mon[65689]: pgmap v323: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:11.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:11 smithi190 ceph-mon[65689]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T19:06:12.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.580+0000 7fcaeae29700 1 -- 172.21.15.102:0/2692015240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcae40feef0 msgr2=0x7fcae40ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.580+0000 7fcaeae29700 1 --2- 172.21.15.102:0/2692015240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcae40feef0 0x7fcae40ff2c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcad8009910 tx=0x7fcad8009c20 comp rx=0 tx=0).stop 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.580+0000 7fcaeae29700 1 -- 172.21.15.102:0/2692015240 shutdown_connections 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.580+0000 7fcaeae29700 1 --2- 172.21.15.102:0/2692015240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcae40ff800 0x7fcae410c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.580+0000 7fcaeae29700 1 --2- 172.21.15.102:0/2692015240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcae40feef0 0x7fcae40ff2c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.580+0000 7fcaeae29700 1 -- 172.21.15.102:0/2692015240 >> 172.21.15.102:0/2692015240 conn(0x7fcae40fab80 msgr2=0x7fcae40fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.581+0000 7fcaeae29700 1 -- 172.21.15.102:0/2692015240 shutdown_connections 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.581+0000 7fcaeae29700 1 -- 172.21.15.102:0/2692015240 wait complete. 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.581+0000 7fcaeae29700 1 Processor -- start 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.581+0000 7fcaeae29700 1 -- start start 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcaeae29700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcae40feef0 0x7fcae4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcaeae29700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcae40ff800 0x7fcae41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcaeae29700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae4198bc0 con 0x7fcae40feef0 2024-03-26T19:06:12.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcaeae29700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae419c950 con 0x7fcae40ff800 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcae8bc5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcae40feef0 0x7fcae4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcae8bc5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcae40feef0 0x7fcae4197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46586/0 (socket says 172.21.15.102:46586) 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcae8bc5700 1 -- 172.21.15.102:0/3092505018 learned_addr learned my addr 172.21.15.102:0/3092505018 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcae3fff700 1 --2- 172.21.15.102:0/3092505018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcae40ff800 0x7fcae41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcae8bc5700 1 -- 172.21.15.102:0/3092505018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcae40ff800 msgr2=0x7fcae41984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcae8bc5700 1 --2- 172.21.15.102:0/3092505018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcae40ff800 0x7fcae41984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcae8bc5700 1 -- 172.21.15.102:0/3092505018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad80095f0 con 0x7fcae40feef0 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcae8bc5700 1 --2- 172.21.15.102:0/3092505018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcae40feef0 0x7fcae4197fa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcad8000c00 tx=0x7fcad80055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcae1ffb700 1 -- 172.21.15.102:0/3092505018 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcad801d070 con 0x7fcae40feef0 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.582+0000 7fcae1ffb700 1 -- 172.21.15.102:0/3092505018 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcad800f460 con 0x7fcae40feef0 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.583+0000 7fcae1ffb700 1 -- 172.21.15.102:0/3092505018 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcad8021640 con 0x7fcae40feef0 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.583+0000 7fcaeae29700 1 -- 172.21.15.102:0/3092505018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae419cbd0 con 0x7fcae40feef0 2024-03-26T19:06:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.583+0000 7fcaeae29700 1 -- 172.21.15.102:0/3092505018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae419d120 con 0x7fcae40feef0 2024-03-26T19:06:12.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.584+0000 7fcaeae29700 1 -- 172.21.15.102:0/3092505018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae4109cd0 con 0x7fcae40feef0 2024-03-26T19:06:12.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.584+0000 7fcae1ffb700 1 -- 172.21.15.102:0/3092505018 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fcad800fa60 con 0x7fcae40feef0 2024-03-26T19:06:12.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.585+0000 7fcae1ffb700 1 --2- 172.21.15.102:0/3092505018 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcad406e460 0x7fcad4070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:12.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.585+0000 7fcae1ffb700 1 -- 172.21.15.102:0/3092505018 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fcad808f6a0 con 0x7fcae40feef0 2024-03-26T19:06:12.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.585+0000 7fcae3fff700 1 --2- 172.21.15.102:0/3092505018 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcad406e460 0x7fcad4070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:12.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.585+0000 7fcae3fff700 1 --2- 172.21.15.102:0/3092505018 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcad406e460 0x7fcad4070910 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcae41995c0 tx=0x7fcad000a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:12.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.590+0000 7fcae1ffb700 1 -- 172.21.15.102:0/3092505018 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcad8058800 con 0x7fcae40feef0 2024-03-26T19:06:12.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.939+0000 7fcaeae29700 1 -- 172.21.15.102:0/3092505018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcae419d400 con 0x7fcae40feef0 2024-03-26T19:06:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.940+0000 7fcae1ffb700 1 -- 172.21.15.102:0/3092505018 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fcad805bec0 con 0x7fcae40feef0 2024-03-26T19:06:12.937 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-26T19:06:12.937 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-03-26T19:06:12.937 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:06:12.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:06:12.937 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-03-26T19:06:12.937 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 2 2024-03-26T19:06:12.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:06:12.937 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-03-26T19:06:12.937 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 6 2024-03-26T19:06:12.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:06:12.938 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-26T19:06:12.938 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 4 2024-03-26T19:06:12.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-26T19:06:12.938 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-03-26T19:06:12.938 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)": 14 2024-03-26T19:06:12.938 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-26T19:06:12.938 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-26T19:06:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.943+0000 7fcaeae29700 1 -- 172.21.15.102:0/3092505018 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcad406e460 msgr2=0x7fcad4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.943+0000 7fcaeae29700 1 --2- 172.21.15.102:0/3092505018 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcad406e460 0x7fcad4070910 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcae41995c0 tx=0x7fcad000a300 comp rx=0 tx=0).stop 2024-03-26T19:06:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.943+0000 7fcaeae29700 1 -- 172.21.15.102:0/3092505018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcae40feef0 msgr2=0x7fcae4197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.943+0000 7fcaeae29700 1 --2- 172.21.15.102:0/3092505018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcae40feef0 0x7fcae4197fa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcad8000c00 tx=0x7fcad80055f0 comp rx=0 tx=0).stop 2024-03-26T19:06:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.943+0000 7fcaeae29700 1 -- 172.21.15.102:0/3092505018 shutdown_connections 2024-03-26T19:06:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.943+0000 7fcaeae29700 1 --2- 172.21.15.102:0/3092505018 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcad406e460 0x7fcad4070910 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.943+0000 7fcaeae29700 1 --2- 172.21.15.102:0/3092505018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcae40feef0 0x7fcae4197fa0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.943+0000 7fcaeae29700 1 --2- 172.21.15.102:0/3092505018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcae40ff800 0x7fcae41984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.943+0000 7fcaeae29700 1 -- 172.21.15.102:0/3092505018 >> 172.21.15.102:0/3092505018 conn(0x7fcae40fab80 msgr2=0x7fcae40fc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.943+0000 7fcaeae29700 1 -- 172.21.15.102:0/3092505018 shutdown_connections 2024-03-26T19:06:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:12.943+0000 7fcaeae29700 1 -- 172.21.15.102:0/3092505018 wait complete. 2024-03-26T19:06:13.363 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:13 smithi102 ceph-mon[78322]: pgmap v324: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:13.363 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:13 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3092505018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:06:13.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:13 smithi190 ceph-mon[65689]: pgmap v324: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:13.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:13 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3092505018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:06:13.536 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-26T19:06:13.768 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T19:06:15.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:15 smithi102 ceph-mon[78322]: pgmap v325: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:15.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.449+0000 7fae94d96700 1 -- 172.21.15.102:0/2619632419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900ff390 msgr2=0x7fae900ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:15.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.449+0000 7fae94d96700 1 --2- 172.21.15.102:0/2619632419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900ff390 0x7fae900ff760 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fae80009910 tx=0x7fae80009c20 comp rx=0 tx=0).stop 2024-03-26T19:06:15.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.449+0000 7fae94d96700 1 -- 172.21.15.102:0/2619632419 shutdown_connections 2024-03-26T19:06:15.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.449+0000 7fae94d96700 1 --2- 172.21.15.102:0/2619632419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae900ffd30 0x7fae90103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:15.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.449+0000 7fae94d96700 1 --2- 172.21.15.102:0/2619632419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900ff390 0x7fae900ff760 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:15.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.449+0000 7fae94d96700 1 -- 172.21.15.102:0/2619632419 >> 172.21.15.102:0/2619632419 conn(0x7fae90074f50 msgr2=0x7fae90075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:15.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.449+0000 7fae94d96700 1 -- 172.21.15.102:0/2619632419 shutdown_connections 2024-03-26T19:06:15.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.449+0000 7fae94d96700 1 -- 172.21.15.102:0/2619632419 wait complete. 2024-03-26T19:06:15.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.450+0000 7fae94d96700 1 Processor -- start 2024-03-26T19:06:15.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.450+0000 7fae94d96700 1 -- start start 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.450+0000 7fae94d96700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae900ff390 0x7fae90195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.450+0000 7fae94d96700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900ffd30 0x7fae90196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.450+0000 7fae94d96700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae90196a50 con 0x7fae900ffd30 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.450+0000 7fae94d96700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9019a7e0 con 0x7fae900ff390 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.450+0000 7fae8dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900ffd30 0x7fae90196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.450+0000 7fae8dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900ffd30 0x7fae90196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46604/0 (socket says 172.21.15.102:46604) 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.450+0000 7fae8dd9b700 1 -- 172.21.15.102:0/1804860548 learned_addr learned my addr 172.21.15.102:0/1804860548 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.450+0000 7fae8e59c700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae900ff390 0x7fae90195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.451+0000 7fae8dd9b700 1 -- 172.21.15.102:0/1804860548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae900ff390 msgr2=0x7fae90195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.451+0000 7fae8dd9b700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae900ff390 0x7fae90195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.451+0000 7fae8dd9b700 1 -- 172.21.15.102:0/1804860548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae800095f0 con 0x7fae900ffd30 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.451+0000 7fae8e59c700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae900ff390 0x7fae90195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.451+0000 7fae8dd9b700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900ffd30 0x7fae90196370 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fae8400b480 tx=0x7fae8400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.451+0000 7fae7f7fe700 1 -- 172.21.15.102:0/1804860548 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae840118c0 con 0x7fae900ffd30 2024-03-26T19:06:15.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.451+0000 7fae7f7fe700 1 -- 172.21.15.102:0/1804860548 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae84010380 con 0x7fae900ffd30 2024-03-26T19:06:15.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.451+0000 7fae94d96700 1 -- 172.21.15.102:0/1804860548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9019aac0 con 0x7fae900ffd30 2024-03-26T19:06:15.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.451+0000 7fae94d96700 1 -- 172.21.15.102:0/1804860548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9019b010 con 0x7fae900ffd30 2024-03-26T19:06:15.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.452+0000 7fae7f7fe700 1 -- 172.21.15.102:0/1804860548 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae8400f4f0 con 0x7fae900ffd30 2024-03-26T19:06:15.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.452+0000 7fae94d96700 1 -- 172.21.15.102:0/1804860548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae70005320 con 0x7fae900ffd30 2024-03-26T19:06:15.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.453+0000 7fae7f7fe700 1 -- 172.21.15.102:0/1804860548 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fae840104f0 con 0x7fae900ffd30 2024-03-26T19:06:15.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.453+0000 7fae7f7fe700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae7806e250 0x7fae78070700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:15.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.454+0000 7fae7f7fe700 1 -- 172.21.15.102:0/1804860548 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fae8408e030 con 0x7fae900ffd30 2024-03-26T19:06:15.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.454+0000 7fae8e59c700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae7806e250 0x7fae78070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:15.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.454+0000 7fae8e59c700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae7806e250 0x7fae78070700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fae800059d0 tx=0x7fae8001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:15.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.462+0000 7fae7f7fe700 1 -- 172.21.15.102:0/1804860548 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fae84057210 con 0x7fae900ffd30 2024-03-26T19:06:15.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:15 smithi190 ceph-mon[65689]: pgmap v325: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:15.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.809+0000 7fae94d96700 1 -- 172.21.15.102:0/1804860548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fae70006200 con 0x7fae900ffd30 2024-03-26T19:06:15.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.809+0000 7fae7f7fe700 1 -- 172.21.15.102:0/1804860548 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fae8405a8d0 con 0x7fae900ffd30 2024-03-26T19:06:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.812+0000 7fae94d96700 1 -- 172.21.15.102:0/1804860548 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae7806e250 msgr2=0x7fae78070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.812+0000 7fae94d96700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae7806e250 0x7fae78070700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fae800059d0 tx=0x7fae8001a040 comp rx=0 tx=0).stop 2024-03-26T19:06:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.812+0000 7fae94d96700 1 -- 172.21.15.102:0/1804860548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900ffd30 msgr2=0x7fae90196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.812+0000 7fae94d96700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900ffd30 0x7fae90196370 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fae8400b480 tx=0x7fae8400b840 comp rx=0 tx=0).stop 2024-03-26T19:06:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.812+0000 7fae94d96700 1 -- 172.21.15.102:0/1804860548 shutdown_connections 2024-03-26T19:06:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.812+0000 7fae94d96700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae900ff390 0x7fae90195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.812+0000 7fae94d96700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae7806e250 0x7fae78070700 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.812+0000 7fae94d96700 1 --2- 172.21.15.102:0/1804860548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900ffd30 0x7fae90196370 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.812+0000 7fae94d96700 1 -- 172.21.15.102:0/1804860548 >> 172.21.15.102:0/1804860548 conn(0x7fae90074f50 msgr2=0x7fae90103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.812+0000 7fae94d96700 1 -- 172.21.15.102:0/1804860548 shutdown_connections 2024-03-26T19:06:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:15.812+0000 7fae94d96700 1 -- 172.21.15.102:0/1804860548 wait complete. 2024-03-26T19:06:15.822 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-03-26T19:06:16.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:16 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1804860548' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:06:16.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -e sha1=360516069d9393362c4cc6eb9371680fe16d66ab -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-26T19:06:16.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:16 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1804860548' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:06:16.710 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T19:06:17.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:17 smithi102 ceph-mon[78322]: pgmap v326: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:17.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:17 smithi190 ceph-mon[65689]: pgmap v326: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:18.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.432+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1682281491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde4100e40 msgr2=0x7fcde4104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:18.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.432+0000 7fcdea9ba700 1 --2- 172.21.15.102:0/1682281491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde4100e40 0x7fcde4104e10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8009910 tx=0x7fcdd8009c20 comp rx=0 tx=0).stop 2024-03-26T19:06:18.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.432+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1682281491 shutdown_connections 2024-03-26T19:06:18.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.432+0000 7fcdea9ba700 1 --2- 172.21.15.102:0/1682281491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde4100e40 0x7fcde4104e10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:18.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.432+0000 7fcdea9ba700 1 --2- 172.21.15.102:0/1682281491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcde41004a0 0x7fcde4100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:18.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.432+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1682281491 >> 172.21.15.102:0/1682281491 conn(0x7fcde40fbd60 msgr2=0x7fcde40fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:18.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.432+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1682281491 shutdown_connections 2024-03-26T19:06:18.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.432+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1682281491 wait complete. 2024-03-26T19:06:18.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.432+0000 7fcdea9ba700 1 Processor -- start 2024-03-26T19:06:18.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.432+0000 7fcdea9ba700 1 -- start start 2024-03-26T19:06:18.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcdea9ba700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde41004a0 0x7fcde4195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcdea9ba700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcde4100e40 0x7fcde4196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcdea9ba700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde4196960 con 0x7fcde41004a0 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcdea9ba700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde419a6f0 con 0x7fcde4100e40 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcde3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde41004a0 0x7fcde4195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcde3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde41004a0 0x7fcde4195d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46622/0 (socket says 172.21.15.102:46622) 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcde3fff700 1 -- 172.21.15.102:0/1494038861 learned_addr learned my addr 172.21.15.102:0/1494038861 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcde37fe700 1 --2- 172.21.15.102:0/1494038861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcde4100e40 0x7fcde4196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcde3fff700 1 -- 172.21.15.102:0/1494038861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcde4100e40 msgr2=0x7fcde4196280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcde3fff700 1 --2- 172.21.15.102:0/1494038861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcde4100e40 0x7fcde4196280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcde3fff700 1 -- 172.21.15.102:0/1494038861 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd80095f0 con 0x7fcde41004a0 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.433+0000 7fcde3fff700 1 --2- 172.21.15.102:0/1494038861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde41004a0 0x7fcde4195d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400d900 tx=0x7fcdd400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.434+0000 7fcde17fa700 1 -- 172.21.15.102:0/1494038861 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdd40041f0 con 0x7fcde41004a0 2024-03-26T19:06:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.434+0000 7fcde17fa700 1 -- 172.21.15.102:0/1494038861 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdd4004cf0 con 0x7fcde41004a0 2024-03-26T19:06:18.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.434+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1494038861 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde419a9d0 con 0x7fcde41004a0 2024-03-26T19:06:18.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.434+0000 7fcde17fa700 1 -- 172.21.15.102:0/1494038861 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdd400b750 con 0x7fcde41004a0 2024-03-26T19:06:18.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.434+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1494038861 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde419af20 con 0x7fcde41004a0 2024-03-26T19:06:18.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.435+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1494038861 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde4108810 con 0x7fcde41004a0 2024-03-26T19:06:18.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.436+0000 7fcde17fa700 1 -- 172.21.15.102:0/1494038861 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fcdd4004350 con 0x7fcde41004a0 2024-03-26T19:06:18.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.436+0000 7fcde17fa700 1 --2- 172.21.15.102:0/1494038861 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcdcc06e460 0x7fcdcc070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:18.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.436+0000 7fcde17fa700 1 -- 172.21.15.102:0/1494038861 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fcdd408e4f0 con 0x7fcde41004a0 2024-03-26T19:06:18.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.436+0000 7fcde37fe700 1 --2- 172.21.15.102:0/1494038861 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcdcc06e460 0x7fcdcc070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:18.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.437+0000 7fcde37fe700 1 --2- 172.21.15.102:0/1494038861 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcdcc06e460 0x7fcdcc070910 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcde4197360 tx=0x7fcdd801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:18.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.443+0000 7fcde17fa700 1 -- 172.21.15.102:0/1494038861 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcdd40576d0 con 0x7fcde41004a0 2024-03-26T19:06:18.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.732+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1494038861 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcde419b230 con 0x7fcde41004a0 2024-03-26T19:06:18.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.733+0000 7fcde17fa700 1 -- 172.21.15.102:0/1494038861 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fcdd405ad90 con 0x7fcde41004a0 2024-03-26T19:06:18.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.735+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1494038861 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcdcc06e460 msgr2=0x7fcdcc070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:18.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.735+0000 7fcdea9ba700 1 --2- 172.21.15.102:0/1494038861 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcdcc06e460 0x7fcdcc070910 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcde4197360 tx=0x7fcdd801a040 comp rx=0 tx=0).stop 2024-03-26T19:06:18.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.735+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1494038861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde41004a0 msgr2=0x7fcde4195d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:18.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.735+0000 7fcdea9ba700 1 --2- 172.21.15.102:0/1494038861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde41004a0 0x7fcde4195d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400d900 tx=0x7fcdd400dcc0 comp rx=0 tx=0).stop 2024-03-26T19:06:18.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.735+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1494038861 shutdown_connections 2024-03-26T19:06:18.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.735+0000 7fcdea9ba700 1 --2- 172.21.15.102:0/1494038861 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcdcc06e460 0x7fcdcc070910 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.735+0000 7fcdea9ba700 1 --2- 172.21.15.102:0/1494038861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcde41004a0 0x7fcde4195d40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.735+0000 7fcdea9ba700 1 --2- 172.21.15.102:0/1494038861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcde4100e40 0x7fcde4196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.735+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1494038861 >> 172.21.15.102:0/1494038861 conn(0x7fcde40fbd60 msgr2=0x7fcde40fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.735+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1494038861 shutdown_connections 2024-03-26T19:06:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:18.735+0000 7fcdea9ba700 1 -- 172.21.15.102:0/1494038861 wait complete. 2024-03-26T19:06:18.746 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.0.0-2311-g36051606 (360516069d9393362c4cc6eb9371680fe16d66ab) squid (dev)" 2024-03-26T19:06:19.304 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:19 smithi102 ceph-mon[78322]: pgmap v327: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:19.304 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:19 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1494038861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:06:19.324 DEBUG:teuthology.parallel:result is None 2024-03-26T19:06:19.324 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T19:06:19.336 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T19:06:19.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-26T19:06:19.424 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:19 smithi190 ceph-mon[65689]: pgmap v327: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:19.424 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:19 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1494038861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T19:06:19.554 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/mon.smithi102/config 2024-03-26T19:06:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.195+0000 7efbf78ff700 1 -- 172.21.15.102:0/335772952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbf0068fd0 msgr2=0x7efbf0104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.195+0000 7efbf78ff700 1 --2- 172.21.15.102:0/335772952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbf0068fd0 0x7efbf0104df0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efbe0009910 tx=0x7efbe0009c20 comp rx=0 tx=0).stop 2024-03-26T19:06:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.195+0000 7efbf78ff700 1 -- 172.21.15.102:0/335772952 shutdown_connections 2024-03-26T19:06:21.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.195+0000 7efbf78ff700 1 --2- 172.21.15.102:0/335772952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbf0068fd0 0x7efbf0104df0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:21.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.195+0000 7efbf78ff700 1 --2- 172.21.15.102:0/335772952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efbf0068630 0x7efbf0068a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:21.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.195+0000 7efbf78ff700 1 -- 172.21.15.102:0/335772952 >> 172.21.15.102:0/335772952 conn(0x7efbf0074f50 msgr2=0x7efbf0075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:21.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.195+0000 7efbf78ff700 1 -- 172.21.15.102:0/335772952 shutdown_connections 2024-03-26T19:06:21.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.195+0000 7efbf78ff700 1 -- 172.21.15.102:0/335772952 wait complete. 2024-03-26T19:06:21.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf78ff700 1 Processor -- start 2024-03-26T19:06:21.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf78ff700 1 -- start start 2024-03-26T19:06:21.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf78ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbf0068630 0x7efbf00ff200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:21.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf78ff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efbf0068fd0 0x7efbf00ff740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf78ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf00ffc80 con 0x7efbf0068630 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf78ff700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf00ffdc0 con 0x7efbf0068fd0 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf569b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbf0068630 0x7efbf00ff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf569b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbf0068630 0x7efbf00ff200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59612/0 (socket says 172.21.15.102:59612) 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf569b700 1 -- 172.21.15.102:0/1722949362 learned_addr learned my addr 172.21.15.102:0/1722949362 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf4e9a700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efbf0068fd0 0x7efbf00ff740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf569b700 1 -- 172.21.15.102:0/1722949362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efbf0068fd0 msgr2=0x7efbf00ff740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf569b700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efbf0068fd0 0x7efbf00ff740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf569b700 1 -- 172.21.15.102:0/1722949362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe00095f0 con 0x7efbf0068630 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.196+0000 7efbf4e9a700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efbf0068fd0 0x7efbf00ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.197+0000 7efbf569b700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbf0068630 0x7efbf00ff200 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efbec00c8a0 tx=0x7efbec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.197+0000 7efbe67fc700 1 -- 172.21.15.102:0/1722949362 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbec00cea0 con 0x7efbf0068630 2024-03-26T19:06:21.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.197+0000 7efbf78ff700 1 -- 172.21.15.102:0/1722949362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf01a1ff0 con 0x7efbf0068630 2024-03-26T19:06:21.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.197+0000 7efbe67fc700 1 -- 172.21.15.102:0/1722949362 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efbec004830 con 0x7efbf0068630 2024-03-26T19:06:21.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.197+0000 7efbe67fc700 1 -- 172.21.15.102:0/1722949362 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbec0055b0 con 0x7efbf0068630 2024-03-26T19:06:21.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.197+0000 7efbf78ff700 1 -- 172.21.15.102:0/1722949362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf01a2400 con 0x7efbf0068630 2024-03-26T19:06:21.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.198+0000 7efbf78ff700 1 -- 172.21.15.102:0/1722949362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf01086d0 con 0x7efbf0068630 2024-03-26T19:06:21.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.199+0000 7efbe67fc700 1 -- 172.21.15.102:0/1722949362 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7efbec0049a0 con 0x7efbf0068630 2024-03-26T19:06:21.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.200+0000 7efbe67fc700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efbdc06e460 0x7efbdc070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:21.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.200+0000 7efbe67fc700 1 -- 172.21.15.102:0/1722949362 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7efbec08e010 con 0x7efbf0068630 2024-03-26T19:06:21.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.200+0000 7efbf4e9a700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efbdc06e460 0x7efbdc070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:21.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.200+0000 7efbf4e9a700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efbdc06e460 0x7efbdc070910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efbe000b5c0 tx=0x7efbe0005480 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:21.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.204+0000 7efbe67fc700 1 -- 172.21.15.102:0/1722949362 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efbec0571f0 con 0x7efbf0068630 2024-03-26T19:06:21.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:21 smithi102 ceph-mon[78322]: pgmap v328: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:21.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.512+0000 7efbf78ff700 1 -- 172.21.15.102:0/1722949362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efbf004ea50 con 0x7efbf0068630 2024-03-26T19:06:21.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.513+0000 7efbe67fc700 1 -- 172.21.15.102:0/1722949362 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 43 v43) v1 ==== 76+0+1955 (secure 0 0 0) 0x7efbec01d070 con 0x7efbf0068630 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:e43 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 43 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-26T18:52:19.251412+0000 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-03-26T19:03:02.920923+0000 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-03-26T19:06:21.511 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 60 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 84 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 2 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=34436,1=34468} 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-03-26T19:06:21.512 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 1 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 34436 members: 34436,34468 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.tvsqaa{0:34436} state up:active seq 19 join_fscid=1 addr [v2:172.21.15.102:6826/2063637342,v1:172.21.15.102:6827/2063637342] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi102.hrbrsp{1:34468} state up:active seq 13 join_fscid=1 addr [v2:172.21.15.102:6828/3007238267,v1:172.21.15.102:6829/3007238267] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.vykizb{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.190:6824/782636706,v1:172.21.15.190:6825/782636706] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:06:21.513 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.cephfs.smithi190.hwsvdg{-1:44373} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.190:6826/925215849,v1:172.21.15.190:6827/925215849] compat {c=[1],r=[1],i=[1fff]}] 2024-03-26T19:06:21.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.516+0000 7efbf78ff700 1 -- 172.21.15.102:0/1722949362 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efbdc06e460 msgr2=0x7efbdc070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:21.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.516+0000 7efbf78ff700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efbdc06e460 0x7efbdc070910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efbe000b5c0 tx=0x7efbe0005480 comp rx=0 tx=0).stop 2024-03-26T19:06:21.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.516+0000 7efbf78ff700 1 -- 172.21.15.102:0/1722949362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbf0068630 msgr2=0x7efbf00ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:21.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.516+0000 7efbf78ff700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbf0068630 0x7efbf00ff200 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efbec00c8a0 tx=0x7efbec00cc60 comp rx=0 tx=0).stop 2024-03-26T19:06:21.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.516+0000 7efbf78ff700 1 -- 172.21.15.102:0/1722949362 shutdown_connections 2024-03-26T19:06:21.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.516+0000 7efbf78ff700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7efbdc06e460 0x7efbdc070910 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:21.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.516+0000 7efbf78ff700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbf0068630 0x7efbf00ff200 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:21.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.516+0000 7efbf78ff700 1 --2- 172.21.15.102:0/1722949362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efbf0068fd0 0x7efbf00ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:21.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.516+0000 7efbf78ff700 1 -- 172.21.15.102:0/1722949362 >> 172.21.15.102:0/1722949362 conn(0x7efbf0074f50 msgr2=0x7efbf00fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:21.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.516+0000 7efbf78ff700 1 -- 172.21.15.102:0/1722949362 shutdown_connections 2024-03-26T19:06:21.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:21.516+0000 7efbf78ff700 1 -- 172.21.15.102:0/1722949362 wait complete. 2024-03-26T19:06:21.515 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 43 2024-03-26T19:06:21.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:21 smithi190 ceph-mon[65689]: pgmap v328: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:22.227 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:22 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1722949362' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:06:22.248 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-03-26T19:06:22.260 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T19:06:22.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.384+0000 7f8425c0b640 1 -- 172.21.15.102:0/3408608866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8420101160 msgr2=0x7f8420109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:22.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.384+0000 7f8425c0b640 1 --2- 172.21.15.102:0/3408608866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8420101160 0x7f8420109650 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8408009c50 tx=0x7f840802c230 comp rx=0 tx=0).stop 2024-03-26T19:06:22.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.384+0000 7f8425c0b640 1 -- 172.21.15.102:0/3408608866 shutdown_connections 2024-03-26T19:06:22.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.384+0000 7f8425c0b640 1 --2- 172.21.15.102:0/3408608866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8420101160 0x7f8420109650 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:22.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.384+0000 7f8425c0b640 1 --2- 172.21.15.102:0/3408608866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8420100850 0x7f8420100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:22.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.384+0000 7f8425c0b640 1 -- 172.21.15.102:0/3408608866 >> 172.21.15.102:0/3408608866 conn(0x7f8420075270 msgr2=0x7f8420075670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:22.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.384+0000 7f8425c0b640 1 -- 172.21.15.102:0/3408608866 shutdown_connections 2024-03-26T19:06:22.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.384+0000 7f8425c0b640 1 -- 172.21.15.102:0/3408608866 wait complete. 2024-03-26T19:06:22.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.385+0000 7f8425c0b640 1 Processor -- start 2024-03-26T19:06:22.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.385+0000 7f8425c0b640 1 -- start start 2024-03-26T19:06:22.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.385+0000 7f8425c0b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8420100850 0x7f84201952f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:22.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.385+0000 7f8425c0b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8420101160 0x7f8420195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:22.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.385+0000 7f8425c0b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8420195ec0 con 0x7f8420100850 2024-03-26T19:06:22.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.385+0000 7f8425c0b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8420199c40 con 0x7f8420101160 2024-03-26T19:06:22.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.385+0000 7f841f7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8420100850 0x7f84201952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:22.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f841f7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8420100850 0x7f84201952f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59636/0 (socket says 172.21.15.102:59636) 2024-03-26T19:06:22.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f841f7fe640 1 -- 172.21.15.102:0/4221820453 learned_addr learned my addr 172.21.15.102:0/4221820453 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f841effd640 1 --2- 172.21.15.102:0/4221820453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8420101160 0x7f8420195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f841f7fe640 1 -- 172.21.15.102:0/4221820453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8420101160 msgr2=0x7f8420195830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f841f7fe640 1 --2- 172.21.15.102:0/4221820453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8420101160 0x7f8420195830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f841f7fe640 1 -- 172.21.15.102:0/4221820453 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8408009900 con 0x7f8420100850 2024-03-26T19:06:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f841f7fe640 1 --2- 172.21.15.102:0/4221820453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8420100850 0x7f84201952f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f841000c8f0 tx=0x7f841000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f841cff9640 1 -- 172.21.15.102:0/4221820453 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8410007b80 con 0x7f8420100850 2024-03-26T19:06:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f841cff9640 1 -- 172.21.15.102:0/4221820453 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84100054e0 con 0x7f8420100850 2024-03-26T19:06:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f841cff9640 1 -- 172.21.15.102:0/4221820453 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f841000f450 con 0x7f8420100850 2024-03-26T19:06:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f8425c0b640 1 -- 172.21.15.102:0/4221820453 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8420199f20 con 0x7f8420100850 2024-03-26T19:06:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.386+0000 7f8425c0b640 1 -- 172.21.15.102:0/4221820453 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842019a450 con 0x7f8420100850 2024-03-26T19:06:22.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.387+0000 7f8425c0b640 1 -- 172.21.15.102:0/4221820453 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842004e000 con 0x7f8420100850 2024-03-26T19:06:22.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.388+0000 7f841cff9640 1 -- 172.21.15.102:0/4221820453 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8410004f60 con 0x7f8420100850 2024-03-26T19:06:22.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.389+0000 7f841cff9640 1 --2- 172.21.15.102:0/4221820453 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f840c06e490 0x7f840c070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:22.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.389+0000 7f841cff9640 1 -- 172.21.15.102:0/4221820453 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f841008e1e0 con 0x7f8420100850 2024-03-26T19:06:22.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.389+0000 7f841effd640 1 --2- 172.21.15.102:0/4221820453 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f840c06e490 0x7f840c070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:22.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.389+0000 7f841effd640 1 --2- 172.21.15.102:0/4221820453 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f840c06e490 0x7f840c070940 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8408000f80 tx=0x7f8408009800 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:22.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.398+0000 7f841cff9640 1 -- 172.21.15.102:0/4221820453 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f84100573c0 con 0x7f8420100850 2024-03-26T19:06:22.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:22 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1722949362' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T19:06:22.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.579+0000 7f8425c0b640 1 -- 172.21.15.102:0/4221820453 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8420106cc0 con 0x7f8420100850 2024-03-26T19:06:22.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.580+0000 7f841cff9640 1 -- 172.21.15.102:0/4221820453 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16965 (secure 0 0 0) 0x7f841005aa80 con 0x7f8420100850 2024-03-26T19:06:22.578 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:22.578 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":84,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","created":"2024-03-26T18:45:25.446329+0000","modified":"2024-03-26T19:02:46.321700+0000","last_up_change":"2024-03-26T19:01:03.417320+0000","last_in_change":"2024-03-26T18:50:45.452094+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T18:49:50.172501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-26T18:52:17.727534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-26T18:52:18.482241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5458d389-eb88-4798-92bc-b442168b37f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6803","nonce":3021596332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6805","nonce":3021596332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6809","nonce":3021596332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6807","nonce":3021596332}]},"public_addr":"172.21.15.102:6803/3021596332","cluster_addr":"172.21.15.102:6805/3021596332","heartbeat_back_addr":"172.21.15.102:6809/3021596332","heartbeat_front_addr":"172.21.15.102:6807/3021596332","state":["exists","up"]},{"osd":1,"uuid":"87508f24-8b3c-4f6d-add2-00ad59300da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6811","nonce":3336954503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6813","nonce":3336954503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6817","nonce":3336954503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6815","nonce":3336954503}]},"public_addr":"172.21.15.102:6811/3336954503","cluster_addr":"172.21.15.102:6813/3336954503","heartbeat_back_addr":"172.21.15.102:6817/3336954503","heartbeat_front_addr":"172.21.15.102:6815/3336954503","state":["exists","up"]},{"osd":2,"uuid":"c0179546-10c5-41f2-af0c-be93c2bbf5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6819","nonce":618947256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6821","nonce":618947256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6825","nonce":618947256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6823","nonce":618947256}]},"public_addr":"172.21.15.102:6819/618947256","cluster_addr":"172.21.15.102:6821/618947256","heartbeat_back_addr":"172.21.15.102:6825/618947256","heartbeat_front_addr":"172.21.15.102:6823/618947256","state":["exists","up"]},{"osd":3,"uuid":"4d524f7b-8b31-469b-b0c2-b94079febaeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6801","nonce":2294582584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6803","nonce":2294582584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6807","nonce":2294582584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6805","nonce":2294582584}]},"public_addr":"172.21.15.190:6801/2294582584","cluster_addr":"172.21.15.190:6803/2294582584","heartbeat_back_addr":"172.21.15.190:6807/2294582584","heartbeat_front_addr":"172.21.15.190:6805/2294582584","state":["exists","up"]},{"osd":4,"uuid":"865baef8-dd72-4449-b3a4-893801bc2958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6809","nonce":2940868243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6811","nonce":2940868243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6815","nonce":2940868243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6813","nonce":2940868243}]},"public_addr":"172.21.15.190:6809/2940868243","cluster_addr":"172.21.15.190:6811/2940868243","heartbeat_back_addr":"172.21.15.190:6815/2940868243","heartbeat_front_addr":"172.21.15.190:6813/2940868243","state":["exists","up"]},{"osd":5,"uuid":"5fd67007-d7ae-43b6-8002-258fbfc5a797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6817","nonce":3222089930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6819","nonce":3222089930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6823","nonce":3222089930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6821","nonce":3222089930}]},"public_addr":"172.21.15.190:6817/3222089930","cluster_addr":"172.21.15.190:6819/3222089930","heartbeat_back_addr":"172.21.15.190:6823/3222089930","heartbeat_front_addr":"172.21.15.190:6821/3222089930","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:48:51.274164+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:18.020320+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:46.066766+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:10.365857+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:36.582495+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:51:02.797707+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6827/624694186":"2024-03-27T19:02:46.321671+0000","172.21.15.190:6826/624694186":"2024-03-27T19:02:46.321671+0000","172.21.15.102:6829/989894576":"2024-03-27T19:02:06.598591+0000","172.21.15.102:6828/989894576":"2024-03-27T19:02:06.598591+0000","172.21.15.102:0/2151777534":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1215990907":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/2396893787":"2024-03-27T18:56:37.181768+0000","172.21.15.102:6801/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/3353740154":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1994330698":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6827/2550866692":"2024-03-27T18:52:35.022250+0000","172.21.15.102:0/1842664791":"2024-03-27T18:47:33.839189+0000","172.21.15.102:6801/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.190:0/2308588851":"2024-03-27T18:55:42.356278+0000","172.21.15.102:0/1295801893":"2024-03-27T18:45:55.925005+0000","172.21.15.102:6800/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6826/2550866692":"2024-03-27T18:52:35.022250+0000","172.21.15.102:0/2094779473":"2024-03-27T18:54:47.731502+0000","172.21.15.190:0/300222988":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6801/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/2686253586":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/1212290737":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/2752725029":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/2027932959":"2024-03-27T18:56:37.181768+0000","172.21.15.190:0/3113541506":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6800/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/216723823":"2024-03-27T18:54:47.731502+0000","172.21.15.190:6828/2858265260":"2024-03-27T18:55:42.356278+0000","172.21.15.190:6829/2858265260":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6800/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1380712530":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/990121767":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1480032448":"2024-03-27T18:54:47.731502+0000","172.21.15.190:0/1923628519":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6801/2831156761":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1818262291":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1615769300":"2024-03-27T18:54:47.731502+0000","172.21.15.190:0/833763291":"2024-03-27T18:55:42.356278+0000","172.21.15.190:0/3941911909":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6800/748923615":"2024-03-27T18:54:47.731502+0000","172.21.15.102:0/4276276941":"2024-03-27T18:47:33.839189+0000","172.21.15.102:6801/748923615":"2024-03-27T18:54:47.731502+0000","172.21.15.102:6800/2831156761":"2024-03-27T18:56:37.181768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T19:06:22.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.582+0000 7f8425c0b640 1 -- 172.21.15.102:0/4221820453 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f840c06e490 msgr2=0x7f840c070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:22.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.582+0000 7f8425c0b640 1 --2- 172.21.15.102:0/4221820453 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f840c06e490 0x7f840c070940 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8408000f80 tx=0x7f8408009800 comp rx=0 tx=0).stop 2024-03-26T19:06:22.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.582+0000 7f8425c0b640 1 -- 172.21.15.102:0/4221820453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8420100850 msgr2=0x7f84201952f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:22.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.582+0000 7f8425c0b640 1 --2- 172.21.15.102:0/4221820453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8420100850 0x7f84201952f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f841000c8f0 tx=0x7f841000cdb0 comp rx=0 tx=0).stop 2024-03-26T19:06:22.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.582+0000 7f8425c0b640 1 -- 172.21.15.102:0/4221820453 shutdown_connections 2024-03-26T19:06:22.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.582+0000 7f8425c0b640 1 --2- 172.21.15.102:0/4221820453 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f840c06e490 0x7f840c070940 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:22.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.582+0000 7f8425c0b640 1 --2- 172.21.15.102:0/4221820453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8420101160 0x7f8420195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:22.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.582+0000 7f8425c0b640 1 --2- 172.21.15.102:0/4221820453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8420100850 0x7f84201952f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:22.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.582+0000 7f8425c0b640 1 -- 172.21.15.102:0/4221820453 >> 172.21.15.102:0/4221820453 conn(0x7f8420075270 msgr2=0x7f8420103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:22.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.582+0000 7f8425c0b640 1 -- 172.21.15.102:0/4221820453 shutdown_connections 2024-03-26T19:06:22.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.583+0000 7f8425c0b640 1 -- 172.21.15.102:0/4221820453 wait complete. 2024-03-26T19:06:22.591 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T18:49:50.172501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-26T18:52:17.727534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-26T18:52:18.482241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T19:06:22.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T19:06:22.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.722+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/1591697694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a48104590 msgr2=0x7f8a48104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:22.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.722+0000 7f8a4cbc0640 1 --2- 172.21.15.102:0/1591697694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a48104590 0x7f8a48104960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009880 tx=0x7f8a3802bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:22.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.722+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/1591697694 shutdown_connections 2024-03-26T19:06:22.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.722+0000 7f8a4cbc0640 1 --2- 172.21.15.102:0/1591697694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a480fe5b0 0x7f8a480fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:22.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.722+0000 7f8a4cbc0640 1 --2- 172.21.15.102:0/1591697694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a48104590 0x7f8a48104960 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:22.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.722+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/1591697694 >> 172.21.15.102:0/1591697694 conn(0x7f8a480743d0 msgr2=0x7f8a480747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:22.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.723+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/1591697694 shutdown_connections 2024-03-26T19:06:22.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.723+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/1591697694 wait complete. 2024-03-26T19:06:22.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.723+0000 7f8a4cbc0640 1 Processor -- start 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.723+0000 7f8a4cbc0640 1 -- start start 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.723+0000 7f8a4cbc0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a480fe5b0 0x7f8a48196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.723+0000 7f8a4cbc0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a48104590 0x7f8a48196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.723+0000 7f8a4cbc0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a48196d20 con 0x7f8a48104590 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.723+0000 7f8a4cbc0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a48196e60 con 0x7f8a480fe5b0 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a3ffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a48104590 0x7f8a48196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a3ffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a48104590 0x7f8a48196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59658/0 (socket says 172.21.15.102:59658) 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a3ffff640 1 -- 172.21.15.102:0/3532922935 learned_addr learned my addr 172.21.15.102:0/3532922935 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a4659c640 1 --2- 172.21.15.102:0/3532922935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a480fe5b0 0x7f8a48196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a3ffff640 1 -- 172.21.15.102:0/3532922935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a480fe5b0 msgr2=0x7f8a48196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a3ffff640 1 --2- 172.21.15.102:0/3532922935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a480fe5b0 0x7f8a48196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a3ffff640 1 -- 172.21.15.102:0/3532922935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a38009560 con 0x7f8a48104590 2024-03-26T19:06:22.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a3ffff640 1 --2- 172.21.15.102:0/3532922935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a48104590 0x7f8a48196640 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000d8d0 tx=0x7f8a3000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:22.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a3f7fe640 1 -- 172.21.15.102:0/3532922935 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a30009c00 con 0x7f8a48104590 2024-03-26T19:06:22.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a3f7fe640 1 -- 172.21.15.102:0/3532922935 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a30009d60 con 0x7f8a48104590 2024-03-26T19:06:22.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/3532922935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4819f170 con 0x7f8a48104590 2024-03-26T19:06:22.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a3f7fe640 1 -- 172.21.15.102:0/3532922935 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a300049d0 con 0x7f8a48104590 2024-03-26T19:06:22.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.724+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/3532922935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4819f6a0 con 0x7f8a48104590 2024-03-26T19:06:22.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.725+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/3532922935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4804e000 con 0x7f8a48104590 2024-03-26T19:06:22.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.726+0000 7f8a3f7fe640 1 -- 172.21.15.102:0/3532922935 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8a30004c90 con 0x7f8a48104590 2024-03-26T19:06:22.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.726+0000 7f8a3f7fe640 1 --2- 172.21.15.102:0/3532922935 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8a3406e3d0 0x7f8a34070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:22.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.726+0000 7f8a3f7fe640 1 -- 172.21.15.102:0/3532922935 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8a3008e0f0 con 0x7f8a48104590 2024-03-26T19:06:22.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.726+0000 7f8a4659c640 1 --2- 172.21.15.102:0/3532922935 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8a3406e3d0 0x7f8a34070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:22.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.727+0000 7f8a4659c640 1 --2- 172.21.15.102:0/3532922935 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8a3406e3d0 0x7f8a34070880 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8a38006010 tx=0x7f8a38005dd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:22.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:22.735+0000 7f8a3f7fe640 1 -- 172.21.15.102:0/3532922935 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8a30057250 con 0x7f8a48104590 2024-03-26T19:06:23.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.023+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/3532922935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a4810b940 con 0x7f8a48104590 2024-03-26T19:06:23.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.024+0000 7f8a3f7fe640 1 -- 172.21.15.102:0/3532922935 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a3005a910 con 0x7f8a48104590 2024-03-26T19:06:23.021 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-26T19:06:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.026+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/3532922935 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8a3406e3d0 msgr2=0x7f8a34070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.026+0000 7f8a4cbc0640 1 --2- 172.21.15.102:0/3532922935 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8a3406e3d0 0x7f8a34070880 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8a38006010 tx=0x7f8a38005dd0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.026+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/3532922935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a48104590 msgr2=0x7f8a48196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.026+0000 7f8a4cbc0640 1 --2- 172.21.15.102:0/3532922935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a48104590 0x7f8a48196640 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000d8d0 tx=0x7f8a3000dd90 comp rx=0 tx=0).stop 2024-03-26T19:06:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.026+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/3532922935 shutdown_connections 2024-03-26T19:06:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.026+0000 7f8a4cbc0640 1 --2- 172.21.15.102:0/3532922935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a48104590 0x7f8a48196640 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.026+0000 7f8a4cbc0640 1 --2- 172.21.15.102:0/3532922935 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8a3406e3d0 0x7f8a34070880 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.026+0000 7f8a4cbc0640 1 --2- 172.21.15.102:0/3532922935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a480fe5b0 0x7f8a48196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.026+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/3532922935 >> 172.21.15.102:0/3532922935 conn(0x7f8a480743d0 msgr2=0x7f8a480fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:23.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.026+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/3532922935 shutdown_connections 2024-03-26T19:06:23.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.026+0000 7f8a4cbc0640 1 -- 172.21.15.102:0/3532922935 wait complete. 2024-03-26T19:06:23.034 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-26T19:06:23.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.149+0000 7f5c205a3640 1 -- 172.21.15.102:0/2568504689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c180fe5b0 msgr2=0x7f5c180fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.149+0000 7f5c205a3640 1 --2- 172.21.15.102:0/2568504689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c180fe5b0 0x7f5c180fea20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009880 tx=0x7f5c0802bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.149+0000 7f5c205a3640 1 -- 172.21.15.102:0/2568504689 shutdown_connections 2024-03-26T19:06:23.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.149+0000 7f5c205a3640 1 --2- 172.21.15.102:0/2568504689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c180fe5b0 0x7f5c180fea20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.149+0000 7f5c205a3640 1 --2- 172.21.15.102:0/2568504689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c18104590 0x7f5c18104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.149+0000 7f5c205a3640 1 -- 172.21.15.102:0/2568504689 >> 172.21.15.102:0/2568504689 conn(0x7f5c180743d0 msgr2=0x7f5c180747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:23.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.149+0000 7f5c205a3640 1 -- 172.21.15.102:0/2568504689 shutdown_connections 2024-03-26T19:06:23.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.149+0000 7f5c205a3640 1 -- 172.21.15.102:0/2568504689 wait complete. 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.150+0000 7f5c205a3640 1 Processor -- start 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.150+0000 7f5c205a3640 1 -- start start 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.150+0000 7f5c205a3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c180fe5b0 0x7f5c181941b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.150+0000 7f5c205a3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c18104590 0x7f5c181946f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.150+0000 7f5c205a3640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c18194d80 con 0x7f5c180fe5b0 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.150+0000 7f5c205a3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c18198b50 con 0x7f5c18104590 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.150+0000 7f5c1e33f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c180fe5b0 0x7f5c181941b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.150+0000 7f5c1e33f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c180fe5b0 0x7f5c181941b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59668/0 (socket says 172.21.15.102:59668) 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.150+0000 7f5c1e33f640 1 -- 172.21.15.102:0/3393213085 learned_addr learned my addr 172.21.15.102:0/3393213085 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.150+0000 7f5c1db3e640 1 --2- 172.21.15.102:0/3393213085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c18104590 0x7f5c181946f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.151+0000 7f5c1e33f640 1 -- 172.21.15.102:0/3393213085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c18104590 msgr2=0x7f5c181946f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.151+0000 7f5c1e33f640 1 --2- 172.21.15.102:0/3393213085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c18104590 0x7f5c181946f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.151+0000 7f5c1e33f640 1 -- 172.21.15.102:0/3393213085 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c08009560 con 0x7f5c180fe5b0 2024-03-26T19:06:23.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.151+0000 7f5c1e33f640 1 --2- 172.21.15.102:0/3393213085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c180fe5b0 0x7f5c181941b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5c1400c8f0 tx=0x7f5c1400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:23.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.151+0000 7f5c0f7fe640 1 -- 172.21.15.102:0/3393213085 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c14007b80 con 0x7f5c180fe5b0 2024-03-26T19:06:23.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.151+0000 7f5c0f7fe640 1 -- 172.21.15.102:0/3393213085 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c140054e0 con 0x7f5c180fe5b0 2024-03-26T19:06:23.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.151+0000 7f5c205a3640 1 -- 172.21.15.102:0/3393213085 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c18198e30 con 0x7f5c180fe5b0 2024-03-26T19:06:23.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.151+0000 7f5c0f7fe640 1 -- 172.21.15.102:0/3393213085 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c1400f450 con 0x7f5c180fe5b0 2024-03-26T19:06:23.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.151+0000 7f5c205a3640 1 -- 172.21.15.102:0/3393213085 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c181993e0 con 0x7f5c180fe5b0 2024-03-26T19:06:23.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.152+0000 7f5c205a3640 1 -- 172.21.15.102:0/3393213085 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1804e000 con 0x7f5c180fe5b0 2024-03-26T19:06:23.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.152+0000 7f5c0f7fe640 1 -- 172.21.15.102:0/3393213085 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5c14004f60 con 0x7f5c180fe5b0 2024-03-26T19:06:23.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.153+0000 7f5c0f7fe640 1 --2- 172.21.15.102:0/3393213085 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5c0406e3d0 0x7f5c04070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:23.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.153+0000 7f5c0f7fe640 1 -- 172.21.15.102:0/3393213085 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f5c1408e2e0 con 0x7f5c180fe5b0 2024-03-26T19:06:23.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.153+0000 7f5c1db3e640 1 --2- 172.21.15.102:0/3393213085 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5c0406e3d0 0x7f5c04070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:23.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.154+0000 7f5c1db3e640 1 --2- 172.21.15.102:0/3393213085 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5c0406e3d0 0x7f5c04070880 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5c08006010 tx=0x7f5c08005b50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:23.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.159+0000 7f5c0f7fe640 1 -- 172.21.15.102:0/3393213085 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5c14057470 con 0x7f5c180fe5b0 2024-03-26T19:06:23.379 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:23 smithi102 ceph-mon[78322]: pgmap v329: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:23.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:23 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:06:23.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:23 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/4221820453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T19:06:23.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:23 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3532922935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T19:06:23.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.430+0000 7f5c205a3640 1 -- 172.21.15.102:0/3393213085 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f5c1810b990 con 0x7f5c180fe5b0 2024-03-26T19:06:23.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.430+0000 7f5c0f7fe640 1 -- 172.21.15.102:0/3393213085 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f5c1405ab30 con 0x7f5c180fe5b0 2024-03-26T19:06:23.427 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-26T19:06:23.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.432+0000 7f5c205a3640 1 -- 172.21.15.102:0/3393213085 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5c0406e3d0 msgr2=0x7f5c04070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.432+0000 7f5c205a3640 1 --2- 172.21.15.102:0/3393213085 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5c0406e3d0 0x7f5c04070880 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5c08006010 tx=0x7f5c08005b50 comp rx=0 tx=0).stop 2024-03-26T19:06:23.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.432+0000 7f5c205a3640 1 -- 172.21.15.102:0/3393213085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c180fe5b0 msgr2=0x7f5c181941b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.432+0000 7f5c205a3640 1 --2- 172.21.15.102:0/3393213085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c180fe5b0 0x7f5c181941b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5c1400c8f0 tx=0x7f5c1400cdb0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.432+0000 7f5c205a3640 1 -- 172.21.15.102:0/3393213085 shutdown_connections 2024-03-26T19:06:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.432+0000 7f5c205a3640 1 --2- 172.21.15.102:0/3393213085 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f5c0406e3d0 0x7f5c04070880 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.432+0000 7f5c205a3640 1 --2- 172.21.15.102:0/3393213085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c18104590 0x7f5c181946f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.432+0000 7f5c205a3640 1 --2- 172.21.15.102:0/3393213085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c180fe5b0 0x7f5c181941b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.432+0000 7f5c205a3640 1 -- 172.21.15.102:0/3393213085 >> 172.21.15.102:0/3393213085 conn(0x7f5c180743d0 msgr2=0x7f5c180fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.432+0000 7f5c205a3640 1 -- 172.21.15.102:0/3393213085 shutdown_connections 2024-03-26T19:06:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.432+0000 7f5c205a3640 1 -- 172.21.15.102:0/3393213085 wait complete. 2024-03-26T19:06:23.443 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-26T19:06:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:23 smithi190 ceph-mon[65689]: pgmap v329: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:23 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:06:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:23 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/4221820453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T19:06:23.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:23 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3532922935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T19:06:23.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.554+0000 7f662810a640 1 -- 172.21.15.102:0/4150189509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6620067890 msgr2=0x7f6620067ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.554+0000 7f662810a640 1 --2- 172.21.15.102:0/4150189509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6620067890 0x7f6620067ce0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6610009880 tx=0x7f661002bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.554+0000 7f662810a640 1 -- 172.21.15.102:0/4150189509 shutdown_connections 2024-03-26T19:06:23.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.554+0000 7f662810a640 1 --2- 172.21.15.102:0/4150189509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6620067890 0x7f6620067ce0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.554+0000 7f662810a640 1 --2- 172.21.15.102:0/4150189509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6620104500 0x7f6620067350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.554+0000 7f662810a640 1 -- 172.21.15.102:0/4150189509 >> 172.21.15.102:0/4150189509 conn(0x7f66200743d0 msgr2=0x7f66200747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:23.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.554+0000 7f662810a640 1 -- 172.21.15.102:0/4150189509 shutdown_connections 2024-03-26T19:06:23.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.554+0000 7f662810a640 1 -- 172.21.15.102:0/4150189509 wait complete. 2024-03-26T19:06:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.555+0000 7f662810a640 1 Processor -- start 2024-03-26T19:06:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.555+0000 7f662810a640 1 -- start start 2024-03-26T19:06:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.555+0000 7f662810a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6620067890 0x7f66201a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.555+0000 7f662810a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6620104500 0x7f66201a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.555+0000 7f662810a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66201a2270 con 0x7f6620104500 2024-03-26T19:06:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.555+0000 7f662810a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662019b750 con 0x7f6620067890 2024-03-26T19:06:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.555+0000 7f66256a5640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6620104500 0x7f66201a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.555+0000 7f66256a5640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6620104500 0x7f66201a1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59690/0 (socket says 172.21.15.102:59690) 2024-03-26T19:06:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.555+0000 7f66256a5640 1 -- 172.21.15.102:0/1828745074 learned_addr learned my addr 172.21.15.102:0/1828745074 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.555+0000 7f6625ea6640 1 --2- 172.21.15.102:0/1828745074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6620067890 0x7f66201a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.556+0000 7f66256a5640 1 -- 172.21.15.102:0/1828745074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6620067890 msgr2=0x7f66201a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.556+0000 7f66256a5640 1 --2- 172.21.15.102:0/1828745074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6620067890 0x7f66201a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.556+0000 7f66256a5640 1 -- 172.21.15.102:0/1828745074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6610009560 con 0x7f6620104500 2024-03-26T19:06:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.556+0000 7f66256a5640 1 --2- 172.21.15.102:0/1828745074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6620104500 0x7f66201a1c20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6610009230 tx=0x7f6610004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.556+0000 7f6616ffd640 1 -- 172.21.15.102:0/1828745074 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66100035f0 con 0x7f6620104500 2024-03-26T19:06:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.556+0000 7f6616ffd640 1 -- 172.21.15.102:0/1828745074 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6610003c10 con 0x7f6620104500 2024-03-26T19:06:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.556+0000 7f662810a640 1 -- 172.21.15.102:0/1828745074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662019b9a0 con 0x7f6620104500 2024-03-26T19:06:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.556+0000 7f6616ffd640 1 -- 172.21.15.102:0/1828745074 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6610034440 con 0x7f6620104500 2024-03-26T19:06:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.556+0000 7f662810a640 1 -- 172.21.15.102:0/1828745074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662019bea0 con 0x7f6620104500 2024-03-26T19:06:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.557+0000 7f662810a640 1 -- 172.21.15.102:0/1828745074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f662004e000 con 0x7f6620104500 2024-03-26T19:06:23.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.558+0000 7f6616ffd640 1 -- 172.21.15.102:0/1828745074 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6610004f60 con 0x7f6620104500 2024-03-26T19:06:23.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.558+0000 7f6616ffd640 1 --2- 172.21.15.102:0/1828745074 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f660c06e3d0 0x7f660c070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:23.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.558+0000 7f6616ffd640 1 -- 172.21.15.102:0/1828745074 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f66100b0820 con 0x7f6620104500 2024-03-26T19:06:23.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.558+0000 7f6625ea6640 1 --2- 172.21.15.102:0/1828745074 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f660c06e3d0 0x7f660c070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:23.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.559+0000 7f6625ea6640 1 --2- 172.21.15.102:0/1828745074 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f660c06e3d0 0x7f660c070880 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f661c00abe0 tx=0x7f661c00a2d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:23.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.563+0000 7f6616ffd640 1 -- 172.21.15.102:0/1828745074 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6610079980 con 0x7f6620104500 2024-03-26T19:06:23.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.761+0000 7f662810a640 1 -- 172.21.15.102:0/1828745074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f6620113010 con 0x7f6620104500 2024-03-26T19:06:23.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.761+0000 7f6616ffd640 1 -- 172.21.15.102:0/1828745074 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f661007d040 con 0x7f6620104500 2024-03-26T19:06:23.759 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-26T19:06:23.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.763+0000 7f662810a640 1 -- 172.21.15.102:0/1828745074 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f660c06e3d0 msgr2=0x7f660c070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.763+0000 7f662810a640 1 --2- 172.21.15.102:0/1828745074 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f660c06e3d0 0x7f660c070880 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f661c00abe0 tx=0x7f661c00a2d0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.763+0000 7f662810a640 1 -- 172.21.15.102:0/1828745074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6620104500 msgr2=0x7f66201a1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.763+0000 7f662810a640 1 --2- 172.21.15.102:0/1828745074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6620104500 0x7f66201a1c20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6610009230 tx=0x7f6610004040 comp rx=0 tx=0).stop 2024-03-26T19:06:23.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.763+0000 7f662810a640 1 -- 172.21.15.102:0/1828745074 shutdown_connections 2024-03-26T19:06:23.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.763+0000 7f662810a640 1 --2- 172.21.15.102:0/1828745074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6620104500 0x7f66201a1c20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.763+0000 7f662810a640 1 --2- 172.21.15.102:0/1828745074 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f660c06e3d0 0x7f660c070880 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.763+0000 7f662810a640 1 --2- 172.21.15.102:0/1828745074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6620067890 0x7f66201a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.763+0000 7f662810a640 1 -- 172.21.15.102:0/1828745074 >> 172.21.15.102:0/1828745074 conn(0x7f66200743d0 msgr2=0x7f66200fc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:23.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.763+0000 7f662810a640 1 -- 172.21.15.102:0/1828745074 shutdown_connections 2024-03-26T19:06:23.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.763+0000 7f662810a640 1 -- 172.21.15.102:0/1828745074 wait complete. 2024-03-26T19:06:23.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T19:06:23.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.896+0000 7fab8c50a640 1 -- 172.21.15.102:0/924121250 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab840fe5a0 msgr2=0x7fab840fea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.896+0000 7fab8c50a640 1 --2- 172.21.15.102:0/924121250 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab840fe5a0 0x7fab840fea10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fab74009880 tx=0x7fab7402bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.896+0000 7fab8c50a640 1 -- 172.21.15.102:0/924121250 shutdown_connections 2024-03-26T19:06:23.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.896+0000 7fab8c50a640 1 --2- 172.21.15.102:0/924121250 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab840fe5a0 0x7fab840fea10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.896+0000 7fab8c50a640 1 --2- 172.21.15.102:0/924121250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab841045d0 0x7fab841049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.896+0000 7fab8c50a640 1 -- 172.21.15.102:0/924121250 >> 172.21.15.102:0/924121250 conn(0x7fab84075270 msgr2=0x7fab84075670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:23.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.896+0000 7fab8c50a640 1 -- 172.21.15.102:0/924121250 shutdown_connections 2024-03-26T19:06:23.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.896+0000 7fab8c50a640 1 -- 172.21.15.102:0/924121250 wait complete. 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab8c50a640 1 Processor -- start 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab8c50a640 1 -- start start 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab8c50a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab840fe5a0 0x7fab84196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab8c50a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab841045d0 0x7fab84196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab8c50a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab84196b30 con 0x7fab841045d0 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab8c50a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab84196ca0 con 0x7fab840fe5a0 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab89aa5640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab841045d0 0x7fab84196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab89aa5640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab841045d0 0x7fab84196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59720/0 (socket says 172.21.15.102:59720) 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab89aa5640 1 -- 172.21.15.102:0/210284885 learned_addr learned my addr 172.21.15.102:0/210284885 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab8a2a6640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab840fe5a0 0x7fab84196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab89aa5640 1 -- 172.21.15.102:0/210284885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab840fe5a0 msgr2=0x7fab84196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab89aa5640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab840fe5a0 0x7fab84196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab89aa5640 1 -- 172.21.15.102:0/210284885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab74009560 con 0x7fab841045d0 2024-03-26T19:06:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab8a2a6640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab840fe5a0 0x7fab84196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:06:23.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab89aa5640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab841045d0 0x7fab84196560 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fab74009230 tx=0x7fab74004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:23.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.897+0000 7fab7b7fe640 1 -- 172.21.15.102:0/210284885 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab740035f0 con 0x7fab841045d0 2024-03-26T19:06:23.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.898+0000 7fab7b7fe640 1 -- 172.21.15.102:0/210284885 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab74003c10 con 0x7fab841045d0 2024-03-26T19:06:23.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.898+0000 7fab8c50a640 1 -- 172.21.15.102:0/210284885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8419f090 con 0x7fab841045d0 2024-03-26T19:06:23.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.898+0000 7fab7b7fe640 1 -- 172.21.15.102:0/210284885 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab74005670 con 0x7fab841045d0 2024-03-26T19:06:23.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.898+0000 7fab8c50a640 1 -- 172.21.15.102:0/210284885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8419f4d0 con 0x7fab841045d0 2024-03-26T19:06:23.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.899+0000 7fab8c50a640 1 -- 172.21.15.102:0/210284885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8404e000 con 0x7fab841045d0 2024-03-26T19:06:23.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.899+0000 7fab7b7fe640 1 -- 172.21.15.102:0/210284885 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fab74004dd0 con 0x7fab841045d0 2024-03-26T19:06:23.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.899+0000 7fab7b7fe640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab7006e430 0x7fab700708e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:23.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.900+0000 7fab7b7fe640 1 -- 172.21.15.102:0/210284885 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fab740b0850 con 0x7fab841045d0 2024-03-26T19:06:23.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.900+0000 7fab8a2a6640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab7006e430 0x7fab700708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:23.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.900+0000 7fab8a2a6640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab7006e430 0x7fab700708e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fab80009b40 tx=0x7fab80005e50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:23.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:23.904+0000 7fab7b7fe640 1 -- 172.21.15.102:0/210284885 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab740799b0 con 0x7fab841045d0 2024-03-26T19:06:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.122+0000 7fab8c50a640 1 -- 172.21.15.102:0/210284885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab84067900 con 0x7fab841045d0 2024-03-26T19:06:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.122+0000 7fab7b7fe640 1 -- 172.21.15.102:0/210284885 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5236 (secure 0 0 0) 0x7fab7407d070 con 0x7fab841045d0 2024-03-26T19:06:24.119 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:24.120 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44343,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/782636706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":782636706},{"type":"v1","addr":"172.21.15.190:6825","nonce":782636706}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44373,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6827/925215849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":925215849},{"type":"v1","addr":"172.21.15.190:6827","nonce":925215849}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:03:02.920923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34436,"mds_1":34468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34436":{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":0,"incarnation":36,"state":"up:active","state_seq":19,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34468":{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":1,"incarnation":42,"state":"up:active","state_seq":13,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34436,"qdb_cluster":[34436,34468]},"id":1}]} 2024-03-26T19:06:24.120 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 43 2024-03-26T19:06:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.124+0000 7fab8c50a640 1 -- 172.21.15.102:0/210284885 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab7006e430 msgr2=0x7fab700708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.124+0000 7fab8c50a640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab7006e430 0x7fab700708e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fab80009b40 tx=0x7fab80005e50 comp rx=0 tx=0).stop 2024-03-26T19:06:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.124+0000 7fab8c50a640 1 -- 172.21.15.102:0/210284885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab841045d0 msgr2=0x7fab84196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.124+0000 7fab8c50a640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab841045d0 0x7fab84196560 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fab74009230 tx=0x7fab74004040 comp rx=0 tx=0).stop 2024-03-26T19:06:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.124+0000 7fab8c50a640 1 -- 172.21.15.102:0/210284885 shutdown_connections 2024-03-26T19:06:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.124+0000 7fab8c50a640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab841045d0 0x7fab84196560 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.124+0000 7fab8c50a640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fab7006e430 0x7fab700708e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.124+0000 7fab8c50a640 1 --2- 172.21.15.102:0/210284885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab840fe5a0 0x7fab84196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.124+0000 7fab8c50a640 1 -- 172.21.15.102:0/210284885 >> 172.21.15.102:0/210284885 conn(0x7fab84075270 msgr2=0x7fab840fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.124+0000 7fab8c50a640 1 -- 172.21.15.102:0/210284885 shutdown_connections 2024-03-26T19:06:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.124+0000 7fab8c50a640 1 -- 172.21.15.102:0/210284885 wait complete. 2024-03-26T19:06:24.132 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 17, 'max_mds': 2, 'flags': 18} 2024-03-26T19:06:24.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 18 2024-03-26T19:06:24.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.242+0000 7fc83359e640 1 -- 172.21.15.102:0/3315347101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8340fedc0 msgr2=0x7fc8340ff230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:24.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.242+0000 7fc83359e640 1 --2- 172.21.15.102:0/3315347101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8340fedc0 0x7fc8340ff230 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc828009880 tx=0x7fc82802bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.242+0000 7fc83359e640 1 -- 172.21.15.102:0/3315347101 shutdown_connections 2024-03-26T19:06:24.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.242+0000 7fc83359e640 1 --2- 172.21.15.102:0/3315347101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8340fedc0 0x7fc8340ff230 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.242+0000 7fc83359e640 1 --2- 172.21.15.102:0/3315347101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8341034f0 0x7fc8341038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.242+0000 7fc83359e640 1 -- 172.21.15.102:0/3315347101 >> 172.21.15.102:0/3315347101 conn(0x7fc8340743d0 msgr2=0x7fc8340747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.242+0000 7fc83359e640 1 -- 172.21.15.102:0/3315347101 shutdown_connections 2024-03-26T19:06:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.242+0000 7fc83359e640 1 -- 172.21.15.102:0/3315347101 wait complete. 2024-03-26T19:06:24.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.243+0000 7fc83359e640 1 Processor -- start 2024-03-26T19:06:24.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.243+0000 7fc83359e640 1 -- start start 2024-03-26T19:06:24.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.243+0000 7fc83359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8340fedc0 0x7fc834194080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:24.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.243+0000 7fc83359e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8341034f0 0x7fc8341945c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:24.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.243+0000 7fc83359e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc834194c50 con 0x7fc8341034f0 2024-03-26T19:06:24.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.243+0000 7fc83359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8341989d0 con 0x7fc8340fedc0 2024-03-26T19:06:24.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.243+0000 7fc831d9b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8341034f0 0x7fc8341945c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:24.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.243+0000 7fc831d9b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8341034f0 0x7fc8341945c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59738/0 (socket says 172.21.15.102:59738) 2024-03-26T19:06:24.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.243+0000 7fc831d9b640 1 -- 172.21.15.102:0/4140914826 learned_addr learned my addr 172.21.15.102:0/4140914826 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:24.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.244+0000 7fc83259c640 1 --2- 172.21.15.102:0/4140914826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8340fedc0 0x7fc834194080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:24.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.244+0000 7fc831d9b640 1 -- 172.21.15.102:0/4140914826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8340fedc0 msgr2=0x7fc834194080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:24.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.244+0000 7fc831d9b640 1 --2- 172.21.15.102:0/4140914826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8340fedc0 0x7fc834194080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.244+0000 7fc831d9b640 1 -- 172.21.15.102:0/4140914826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc828009560 con 0x7fc8341034f0 2024-03-26T19:06:24.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.244+0000 7fc831d9b640 1 --2- 172.21.15.102:0/4140914826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8341034f0 0x7fc8341945c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc828009230 tx=0x7fc828005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:24.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.244+0000 7fc8237fe640 1 -- 172.21.15.102:0/4140914826 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8280044a0 con 0x7fc8341034f0 2024-03-26T19:06:24.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.244+0000 7fc83359e640 1 -- 172.21.15.102:0/4140914826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc834198c50 con 0x7fc8341034f0 2024-03-26T19:06:24.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.244+0000 7fc8237fe640 1 -- 172.21.15.102:0/4140914826 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc828003ab0 con 0x7fc8341034f0 2024-03-26T19:06:24.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.244+0000 7fc8237fe640 1 -- 172.21.15.102:0/4140914826 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8280055f0 con 0x7fc8341034f0 2024-03-26T19:06:24.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.244+0000 7fc83359e640 1 -- 172.21.15.102:0/4140914826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc834199150 con 0x7fc8341034f0 2024-03-26T19:06:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.245+0000 7fc83359e640 1 -- 172.21.15.102:0/4140914826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83404e000 con 0x7fc8341034f0 2024-03-26T19:06:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.246+0000 7fc8237fe640 1 -- 172.21.15.102:0/4140914826 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc828004e60 con 0x7fc8341034f0 2024-03-26T19:06:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.246+0000 7fc8237fe640 1 --2- 172.21.15.102:0/4140914826 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc81c06e3d0 0x7fc81c070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.246+0000 7fc8237fe640 1 -- 172.21.15.102:0/4140914826 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc8280b07f0 con 0x7fc8341034f0 2024-03-26T19:06:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.246+0000 7fc83259c640 1 --2- 172.21.15.102:0/4140914826 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc81c06e3d0 0x7fc81c070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.246+0000 7fc83259c640 1 --2- 172.21.15.102:0/4140914826 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc81c06e3d0 0x7fc81c070880 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc824009bd0 tx=0x7fc824005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:24.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.252+0000 7fc8237fe640 1 -- 172.21.15.102:0/4140914826 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc828079a00 con 0x7fc8341034f0 2024-03-26T19:06:24.473 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:24 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3393213085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-26T19:06:24.473 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:24 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1828745074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-26T19:06:24.473 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:24 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/210284885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T19:06:24.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.554+0000 7fc83359e640 1 -- 172.21.15.102:0/4140914826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 18, "format": "json"} v 0) v1 -- 0x7fc834106720 con 0x7fc8341034f0 2024-03-26T19:06:24.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.555+0000 7fc8237fe640 1 -- 172.21.15.102:0/4140914826 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 18, "format": "json"}]=0 dumped fsmap epoch 18 v43) v1 ==== 107+0+4879 (secure 0 0 0) 0x7fc82807d0c0 con 0x7fc8341034f0 2024-03-26T19:06:24.552 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:24.552 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14502,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2674848848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2674848848},{"type":"v1","addr":"172.21.15.102:6827","nonce":2674848848}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T18:53:56.228104+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14490,"mds_1":24283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14490":{"gid":14490,"name":"cephfs.smithi102.hrbrsp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.102:6829/989894576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":989894576},{"type":"v1","addr":"172.21.15.102:6829","nonce":989894576}]},"join_fscid":1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24283":{"gid":24283,"name":"cephfs.smithi190.vykizb","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.190:6825/2769333078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2769333078},{"type":"v1","addr":"172.21.15.190:6825","nonce":2769333078}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-26T19:06:24.552 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 18 2024-03-26T19:06:24.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.558+0000 7fc83359e640 1 -- 172.21.15.102:0/4140914826 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc81c06e3d0 msgr2=0x7fc81c070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:24.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.558+0000 7fc83359e640 1 --2- 172.21.15.102:0/4140914826 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc81c06e3d0 0x7fc81c070880 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc824009bd0 tx=0x7fc824005f90 comp rx=0 tx=0).stop 2024-03-26T19:06:24.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.558+0000 7fc83359e640 1 -- 172.21.15.102:0/4140914826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8341034f0 msgr2=0x7fc8341945c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:24.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.558+0000 7fc83359e640 1 --2- 172.21.15.102:0/4140914826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8341034f0 0x7fc8341945c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc828009230 tx=0x7fc828005e50 comp rx=0 tx=0).stop 2024-03-26T19:06:24.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.558+0000 7fc83359e640 1 -- 172.21.15.102:0/4140914826 shutdown_connections 2024-03-26T19:06:24.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.558+0000 7fc83359e640 1 --2- 172.21.15.102:0/4140914826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8341034f0 0x7fc8341945c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.558+0000 7fc83359e640 1 --2- 172.21.15.102:0/4140914826 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc81c06e3d0 0x7fc81c070880 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.558+0000 7fc83359e640 1 --2- 172.21.15.102:0/4140914826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8340fedc0 0x7fc834194080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.558+0000 7fc83359e640 1 -- 172.21.15.102:0/4140914826 >> 172.21.15.102:0/4140914826 conn(0x7fc8340743d0 msgr2=0x7fc8340fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:24.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.558+0000 7fc83359e640 1 -- 172.21.15.102:0/4140914826 shutdown_connections 2024-03-26T19:06:24.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.558+0000 7fc83359e640 1 -- 172.21.15.102:0/4140914826 wait complete. 2024-03-26T19:06:24.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:24 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3393213085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-26T19:06:24.568 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:24 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1828745074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-26T19:06:24.568 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:24 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/210284885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T19:06:24.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-03-26T19:06:24.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.680+0000 7f2fdc950640 1 -- 172.21.15.102:0/2807192538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd8067350 msgr2=0x7f2fd80677c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:24.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.680+0000 7f2fdc950640 1 --2- 172.21.15.102:0/2807192538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd8067350 0x7f2fd80677c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009880 tx=0x7f2fc002bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.680+0000 7f2fdc950640 1 -- 172.21.15.102:0/2807192538 shutdown_connections 2024-03-26T19:06:24.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.680+0000 7f2fdc950640 1 --2- 172.21.15.102:0/2807192538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd8067350 0x7f2fd80677c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.680+0000 7f2fdc950640 1 --2- 172.21.15.102:0/2807192538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fd8105690 0x7f2fd8105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.680+0000 7f2fdc950640 1 -- 172.21.15.102:0/2807192538 >> 172.21.15.102:0/2807192538 conn(0x7f2fd80743d0 msgr2=0x7f2fd80747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:24.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.680+0000 7f2fdc950640 1 -- 172.21.15.102:0/2807192538 shutdown_connections 2024-03-26T19:06:24.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.680+0000 7f2fdc950640 1 -- 172.21.15.102:0/2807192538 wait complete. 2024-03-26T19:06:24.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fdc950640 1 Processor -- start 2024-03-26T19:06:24.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fdc950640 1 -- start start 2024-03-26T19:06:24.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fdc950640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fd8067350 0x7f2fd8195340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:24.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fdc950640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd8105690 0x7f2fd8195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:24.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fdc950640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd8195f10 con 0x7f2fd8105690 2024-03-26T19:06:24.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fdc950640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd8199ce0 con 0x7f2fd8067350 2024-03-26T19:06:24.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fcffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd8105690 0x7f2fd8195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fcffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd8105690 0x7f2fd8195880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59764/0 (socket says 172.21.15.102:59764) 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fcffff640 1 -- 172.21.15.102:0/1824815218 learned_addr learned my addr 172.21.15.102:0/1824815218 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fd659c640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fd8067350 0x7f2fd8195340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fcffff640 1 -- 172.21.15.102:0/1824815218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fd8067350 msgr2=0x7f2fd8195340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.681+0000 7f2fcffff640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fd8067350 0x7f2fd8195340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.682+0000 7f2fcffff640 1 -- 172.21.15.102:0/1824815218 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc0009560 con 0x7f2fd8105690 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.682+0000 7f2fd659c640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fd8067350 0x7f2fd8195340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.682+0000 7f2fcffff640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd8105690 0x7f2fd8195880 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009230 tx=0x7f2fc0005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.682+0000 7f2fcf7fe640 1 -- 172.21.15.102:0/1824815218 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fc00044a0 con 0x7f2fd8105690 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.682+0000 7f2fcf7fe640 1 -- 172.21.15.102:0/1824815218 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fc0003ab0 con 0x7f2fd8105690 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.682+0000 7f2fdc950640 1 -- 172.21.15.102:0/1824815218 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd8199f60 con 0x7f2fd8105690 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.682+0000 7f2fcf7fe640 1 -- 172.21.15.102:0/1824815218 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fc00055f0 con 0x7f2fd8105690 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.682+0000 7f2fdc950640 1 -- 172.21.15.102:0/1824815218 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd819a460 con 0x7f2fd8105690 2024-03-26T19:06:24.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.683+0000 7f2fdc950640 1 -- 172.21.15.102:0/1824815218 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd804e000 con 0x7f2fd8105690 2024-03-26T19:06:24.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.684+0000 7f2fcf7fe640 1 -- 172.21.15.102:0/1824815218 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2fc0004e60 con 0x7f2fd8105690 2024-03-26T19:06:24.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.684+0000 7f2fcf7fe640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2fc406e3d0 0x7f2fc4070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:24.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.684+0000 7f2fcf7fe640 1 -- 172.21.15.102:0/1824815218 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f2fc00b0860 con 0x7f2fd8105690 2024-03-26T19:06:24.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.684+0000 7f2fd659c640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2fc406e3d0 0x7f2fc4070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:24.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.685+0000 7f2fd659c640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2fc406e3d0 0x7f2fc4070880 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8009b50 tx=0x7f2fc8005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:24.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.689+0000 7f2fcf7fe640 1 -- 172.21.15.102:0/1824815218 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2fc00799c0 con 0x7f2fd8105690 2024-03-26T19:06:24.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.910+0000 7f2fdc950640 1 -- 172.21.15.102:0/1824815218 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7f2fd8107aa0 con 0x7f2fd8105690 2024-03-26T19:06:24.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.910+0000 7f2fcf7fe640 1 -- 172.21.15.102:0/1824815218 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v43) v1 ==== 107+0+4880 (secure 0 0 0) 0x7f2fc007d080 con 0x7f2fd8105690 2024-03-26T19:06:24.907 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:24.908 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14502,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2674848848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2674848848},{"type":"v1","addr":"172.21.15.102:6827","nonce":2674848848}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T18:55:27.824831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14490,"mds_1":24283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14490":{"gid":14490,"name":"cephfs.smithi102.hrbrsp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.102:6829/989894576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":989894576},{"type":"v1","addr":"172.21.15.102:6829","nonce":989894576}]},"join_fscid":1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24283":{"gid":24283,"name":"cephfs.smithi190.vykizb","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.190:6825/2769333078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2769333078},{"type":"v1","addr":"172.21.15.190:6825","nonce":2769333078}]},"join_fscid":1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-26T19:06:24.908 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 19 2024-03-26T19:06:24.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.912+0000 7f2fdc950640 1 -- 172.21.15.102:0/1824815218 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2fc406e3d0 msgr2=0x7f2fc4070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:24.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.912+0000 7f2fdc950640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2fc406e3d0 0x7f2fc4070880 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8009b50 tx=0x7f2fc8005f90 comp rx=0 tx=0).stop 2024-03-26T19:06:24.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.912+0000 7f2fdc950640 1 -- 172.21.15.102:0/1824815218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd8105690 msgr2=0x7f2fd8195880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:24.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.912+0000 7f2fdc950640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd8105690 0x7f2fd8195880 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009230 tx=0x7f2fc0005e50 comp rx=0 tx=0).stop 2024-03-26T19:06:24.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.912+0000 7f2fdc950640 1 -- 172.21.15.102:0/1824815218 shutdown_connections 2024-03-26T19:06:24.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.912+0000 7f2fdc950640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd8105690 0x7f2fd8195880 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.912+0000 7f2fdc950640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2fc406e3d0 0x7f2fc4070880 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.912+0000 7f2fdc950640 1 --2- 172.21.15.102:0/1824815218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fd8067350 0x7f2fd8195340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:24.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.912+0000 7f2fdc950640 1 -- 172.21.15.102:0/1824815218 >> 172.21.15.102:0/1824815218 conn(0x7f2fd80743d0 msgr2=0x7f2fd80fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:24.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.912+0000 7f2fdc950640 1 -- 172.21.15.102:0/1824815218 shutdown_connections 2024-03-26T19:06:24.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:24.912+0000 7f2fdc950640 1 -- 172.21.15.102:0/1824815218 wait complete. 2024-03-26T19:06:24.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-03-26T19:06:25.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.029+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3119101995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef0104590 msgr2=0x7f6ef0104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:25.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.029+0000 7f6ef5a7d640 1 --2- 172.21.15.102:0/3119101995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef0104590 0x7f6ef0104960 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009880 tx=0x7f6ee002bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.029+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3119101995 shutdown_connections 2024-03-26T19:06:25.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.029+0000 7f6ef5a7d640 1 --2- 172.21.15.102:0/3119101995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef00fe5b0 0x7f6ef00fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.029+0000 7f6ef5a7d640 1 --2- 172.21.15.102:0/3119101995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef0104590 0x7f6ef0104960 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.029+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3119101995 >> 172.21.15.102:0/3119101995 conn(0x7f6ef00743d0 msgr2=0x7f6ef00747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:25.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.029+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3119101995 shutdown_connections 2024-03-26T19:06:25.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.029+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3119101995 wait complete. 2024-03-26T19:06:25.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6ef5a7d640 1 Processor -- start 2024-03-26T19:06:25.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6ef5a7d640 1 -- start start 2024-03-26T19:06:25.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6ef5a7d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef00fe5b0 0x7f6ef0195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:25.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6ef5a7d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef01959d0 0x7f6ef0199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:25.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6ef5a7d640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef0195fc0 con 0x7f6ef01959d0 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6ef5a7d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef0196130 con 0x7f6ef00fe5b0 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6eeeffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef01959d0 0x7f6ef0199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6eeeffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef01959d0 0x7f6ef0199e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59772/0 (socket says 172.21.15.102:59772) 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6eeeffd640 1 -- 172.21.15.102:0/3343773971 learned_addr learned my addr 172.21.15.102:0/3343773971 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6eef7fe640 1 --2- 172.21.15.102:0/3343773971 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef00fe5b0 0x7f6ef0195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6eeeffd640 1 -- 172.21.15.102:0/3343773971 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef00fe5b0 msgr2=0x7f6ef0195490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6eeeffd640 1 --2- 172.21.15.102:0/3343773971 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef00fe5b0 0x7f6ef0195490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.030+0000 7f6eeeffd640 1 -- 172.21.15.102:0/3343773971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee0009560 con 0x7f6ef01959d0 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.031+0000 7f6eeeffd640 1 --2- 172.21.15.102:0/3343773971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef01959d0 0x7f6ef0199e30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400d6f0 tx=0x7f6ee400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.031+0000 7f6eecff9640 1 -- 172.21.15.102:0/3343773971 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ee4009a30 con 0x7f6ef01959d0 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.031+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3343773971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef019a430 con 0x7f6ef01959d0 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.031+0000 7f6eecff9640 1 -- 172.21.15.102:0/3343773971 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ee40054e0 con 0x7f6ef01959d0 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.031+0000 7f6eecff9640 1 -- 172.21.15.102:0/3343773971 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ee4010460 con 0x7f6ef01959d0 2024-03-26T19:06:25.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.031+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3343773971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef019aa10 con 0x7f6ef01959d0 2024-03-26T19:06:25.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.032+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3343773971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef019ac90 con 0x7f6ef01959d0 2024-03-26T19:06:25.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.032+0000 7f6eecff9640 1 -- 172.21.15.102:0/3343773971 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6ee4009b90 con 0x7f6ef01959d0 2024-03-26T19:06:25.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.033+0000 7f6eecff9640 1 --2- 172.21.15.102:0/3343773971 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6ed806e3d0 0x7f6ed8070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:25.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.033+0000 7f6eef7fe640 1 --2- 172.21.15.102:0/3343773971 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6ed806e3d0 0x7f6ed8070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:25.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.033+0000 7f6eecff9640 1 -- 172.21.15.102:0/3343773971 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f6ee408e000 con 0x7f6ef01959d0 2024-03-26T19:06:25.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.033+0000 7f6eef7fe640 1 --2- 172.21.15.102:0/3343773971 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6ed806e3d0 0x7f6ed8070880 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009230 tx=0x7f6ee0005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:25.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.037+0000 7f6eecff9640 1 -- 172.21.15.102:0/3343773971 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6ee4057160 con 0x7f6ef01959d0 2024-03-26T19:06:25.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.322+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3343773971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7f6ef010b7e0 con 0x7f6ef01959d0 2024-03-26T19:06:25.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.323+0000 7f6eecff9640 1 -- 172.21.15.102:0/3343773971 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v43) v1 ==== 107+0+4894 (secure 0 0 0) 0x7f6ee405a820 con 0x7f6ef01959d0 2024-03-26T19:06:25.320 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:25.320 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14502,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2674848848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2674848848},{"type":"v1","addr":"172.21.15.102:6827","nonce":2674848848}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T18:58:16.067523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14490,"mds_1":24283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14490":{"gid":14490,"name":"cephfs.smithi102.hrbrsp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.102:6829/989894576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":989894576},{"type":"v1","addr":"172.21.15.102:6829","nonce":989894576}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24283":{"gid":24283,"name":"cephfs.smithi190.vykizb","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.190:6825/2769333078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2769333078},{"type":"v1","addr":"172.21.15.190:6825","nonce":2769333078}]},"join_fscid":1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14490,"qdb_cluster":[14490,24283]},"id":1}]} 2024-03-26T19:06:25.320 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 20 2024-03-26T19:06:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.326+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3343773971 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6ed806e3d0 msgr2=0x7f6ed8070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.326+0000 7f6ef5a7d640 1 --2- 172.21.15.102:0/3343773971 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6ed806e3d0 0x7f6ed8070880 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009230 tx=0x7f6ee0005fd0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.326+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3343773971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef01959d0 msgr2=0x7f6ef0199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.326+0000 7f6ef5a7d640 1 --2- 172.21.15.102:0/3343773971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef01959d0 0x7f6ef0199e30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400d6f0 tx=0x7f6ee400dbb0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.326+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3343773971 shutdown_connections 2024-03-26T19:06:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.326+0000 7f6ef5a7d640 1 --2- 172.21.15.102:0/3343773971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ef01959d0 0x7f6ef0199e30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.326+0000 7f6ef5a7d640 1 --2- 172.21.15.102:0/3343773971 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f6ed806e3d0 0x7f6ed8070880 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.326+0000 7f6ef5a7d640 1 --2- 172.21.15.102:0/3343773971 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef00fe5b0 0x7f6ef0195490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.326+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3343773971 >> 172.21.15.102:0/3343773971 conn(0x7f6ef00743d0 msgr2=0x7f6ef0102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.326+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3343773971 shutdown_connections 2024-03-26T19:06:25.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.326+0000 7f6ef5a7d640 1 -- 172.21.15.102:0/3343773971 wait complete. 2024-03-26T19:06:25.332 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:25 smithi102 ceph-mon[78322]: pgmap v330: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:25.333 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:25 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/4140914826' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-03-26T19:06:25.333 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:25 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1824815218' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-03-26T19:06:25.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-03-26T19:06:25.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.453+0000 7f79e1ae7640 1 -- 172.21.15.102:0/3140387849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc104590 msgr2=0x7f79dc104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.453+0000 7f79e1ae7640 1 --2- 172.21.15.102:0/3140387849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc104590 0x7f79dc104960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f79cc009880 tx=0x7f79cc02bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.453+0000 7f79e1ae7640 1 -- 172.21.15.102:0/3140387849 shutdown_connections 2024-03-26T19:06:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.453+0000 7f79e1ae7640 1 --2- 172.21.15.102:0/3140387849 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79dc0fe5b0 0x7f79dc0fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.453+0000 7f79e1ae7640 1 --2- 172.21.15.102:0/3140387849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc104590 0x7f79dc104960 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.453+0000 7f79e1ae7640 1 -- 172.21.15.102:0/3140387849 >> 172.21.15.102:0/3140387849 conn(0x7f79dc0743d0 msgr2=0x7f79dc0747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.453+0000 7f79e1ae7640 1 -- 172.21.15.102:0/3140387849 shutdown_connections 2024-03-26T19:06:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.453+0000 7f79e1ae7640 1 -- 172.21.15.102:0/3140387849 wait complete. 2024-03-26T19:06:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.453+0000 7f79e1ae7640 1 Processor -- start 2024-03-26T19:06:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.453+0000 7f79e1ae7640 1 -- start start 2024-03-26T19:06:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79e1ae7640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0fe5b0 0x7f79dc1941c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79e1ae7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79dc104590 0x7f79dc194700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79e1ae7640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc194d90 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79e1ae7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc198b60 con 0x7f79dc104590 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79db7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0fe5b0 0x7f79dc1941c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79db7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0fe5b0 0x7f79dc1941c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59796/0 (socket says 172.21.15.102:59796) 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79db7fe640 1 -- 172.21.15.102:0/2468218525 learned_addr learned my addr 172.21.15.102:0/2468218525 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79daffd640 1 --2- 172.21.15.102:0/2468218525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79dc104590 0x7f79dc194700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79db7fe640 1 -- 172.21.15.102:0/2468218525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79dc104590 msgr2=0x7f79dc194700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79db7fe640 1 --2- 172.21.15.102:0/2468218525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79dc104590 0x7f79dc194700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79db7fe640 1 -- 172.21.15.102:0/2468218525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79cc009560 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79db7fe640 1 --2- 172.21.15.102:0/2468218525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0fe5b0 0x7f79dc1941c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f79cc009230 tx=0x7f79cc005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79d8ff9640 1 -- 172.21.15.102:0/2468218525 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79cc0044a0 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79e1ae7640 1 -- 172.21.15.102:0/2468218525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc198de0 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79d8ff9640 1 -- 172.21.15.102:0/2468218525 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79cc003ab0 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79e1ae7640 1 -- 172.21.15.102:0/2468218525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc199360 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.454+0000 7f79d8ff9640 1 -- 172.21.15.102:0/2468218525 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79cc0055f0 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.456+0000 7f79e1ae7640 1 -- 172.21.15.102:0/2468218525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc04e000 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.456+0000 7f79d8ff9640 1 -- 172.21.15.102:0/2468218525 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f79cc004e60 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.456+0000 7f79d8ff9640 1 --2- 172.21.15.102:0/2468218525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f79c406e3d0 0x7f79c4070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:25.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.456+0000 7f79d8ff9640 1 -- 172.21.15.102:0/2468218525 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f79cc0b0880 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.457+0000 7f79daffd640 1 --2- 172.21.15.102:0/2468218525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f79c406e3d0 0x7f79c4070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:25.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.457+0000 7f79daffd640 1 --2- 172.21.15.102:0/2468218525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f79c406e3d0 0x7f79c4070880 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f79dc1957d0 tx=0x7f79d0005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:25.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.465+0000 7f79d8ff9640 1 -- 172.21.15.102:0/2468218525 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f79cc0799e0 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:25 smithi190 ceph-mon[65689]: pgmap v330: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:25.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:25 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/4140914826' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-03-26T19:06:25.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:25 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1824815218' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-03-26T19:06:25.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.685+0000 7f79e1ae7640 1 -- 172.21.15.102:0/2468218525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7f79dc10b750 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.685+0000 7f79d8ff9640 1 -- 172.21.15.102:0/2468218525 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v43) v1 ==== 107+0+4893 (secure 0 0 0) 0x7f79cc07d0a0 con 0x7f79dc0fe5b0 2024-03-26T19:06:25.683 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:25.683 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14502,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2674848848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2674848848},{"type":"v1","addr":"172.21.15.102:6827","nonce":2674848848}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T18:58:42.198340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14490,"mds_1":24283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14490":{"gid":14490,"name":"cephfs.smithi102.hrbrsp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.102:6829/989894576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":989894576},{"type":"v1","addr":"172.21.15.102:6829","nonce":989894576}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24283":{"gid":24283,"name":"cephfs.smithi190.vykizb","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.190:6825/2769333078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2769333078},{"type":"v1","addr":"172.21.15.190:6825","nonce":2769333078}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14490,"qdb_cluster":[24283,14490]},"id":1}]} 2024-03-26T19:06:25.683 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 21 2024-03-26T19:06:25.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.687+0000 7f79e1ae7640 1 -- 172.21.15.102:0/2468218525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f79c406e3d0 msgr2=0x7f79c4070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:25.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.688+0000 7f79e1ae7640 1 --2- 172.21.15.102:0/2468218525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f79c406e3d0 0x7f79c4070880 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f79dc1957d0 tx=0x7f79d0005f90 comp rx=0 tx=0).stop 2024-03-26T19:06:25.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.688+0000 7f79e1ae7640 1 -- 172.21.15.102:0/2468218525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0fe5b0 msgr2=0x7f79dc1941c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:25.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.688+0000 7f79e1ae7640 1 --2- 172.21.15.102:0/2468218525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0fe5b0 0x7f79dc1941c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f79cc009230 tx=0x7f79cc005e50 comp rx=0 tx=0).stop 2024-03-26T19:06:25.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.688+0000 7f79e1ae7640 1 -- 172.21.15.102:0/2468218525 shutdown_connections 2024-03-26T19:06:25.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.688+0000 7f79e1ae7640 1 --2- 172.21.15.102:0/2468218525 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f79c406e3d0 0x7f79c4070880 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.688+0000 7f79e1ae7640 1 --2- 172.21.15.102:0/2468218525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79dc104590 0x7f79dc194700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.688+0000 7f79e1ae7640 1 --2- 172.21.15.102:0/2468218525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79dc0fe5b0 0x7f79dc1941c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.688+0000 7f79e1ae7640 1 -- 172.21.15.102:0/2468218525 >> 172.21.15.102:0/2468218525 conn(0x7f79dc0743d0 msgr2=0x7f79dc0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:25.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.688+0000 7f79e1ae7640 1 -- 172.21.15.102:0/2468218525 shutdown_connections 2024-03-26T19:06:25.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.688+0000 7f79e1ae7640 1 -- 172.21.15.102:0/2468218525 wait complete. 2024-03-26T19:06:25.696 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-03-26T19:06:25.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.804+0000 7f26e9718640 1 -- 172.21.15.102:0/1062299187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26e40fe5b0 msgr2=0x7f26e40fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:25.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.804+0000 7f26e9718640 1 --2- 172.21.15.102:0/1062299187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26e40fe5b0 0x7f26e40fea20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f26d8009880 tx=0x7f26d802bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.804+0000 7f26e9718640 1 -- 172.21.15.102:0/1062299187 shutdown_connections 2024-03-26T19:06:25.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.804+0000 7f26e9718640 1 --2- 172.21.15.102:0/1062299187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26e40fe5b0 0x7f26e40fea20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.804+0000 7f26e9718640 1 --2- 172.21.15.102:0/1062299187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26e4104590 0x7f26e4104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.804+0000 7f26e9718640 1 -- 172.21.15.102:0/1062299187 >> 172.21.15.102:0/1062299187 conn(0x7f26e40743d0 msgr2=0x7f26e40747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.804+0000 7f26e9718640 1 -- 172.21.15.102:0/1062299187 shutdown_connections 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.804+0000 7f26e9718640 1 -- 172.21.15.102:0/1062299187 wait complete. 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.804+0000 7f26e9718640 1 Processor -- start 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e9718640 1 -- start start 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e9718640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26e40fe5b0 0x7f26e4196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e9718640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26e4104590 0x7f26e4196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e9718640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4196c20 con 0x7f26e40fe5b0 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e9718640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4196d90 con 0x7f26e4104590 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e2ffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26e40fe5b0 0x7f26e4196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e2ffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26e40fe5b0 0x7f26e4196110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59820/0 (socket says 172.21.15.102:59820) 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e2ffd640 1 -- 172.21.15.102:0/994899975 learned_addr learned my addr 172.21.15.102:0/994899975 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e27fc640 1 --2- 172.21.15.102:0/994899975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26e4104590 0x7f26e4196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e2ffd640 1 -- 172.21.15.102:0/994899975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26e4104590 msgr2=0x7f26e4196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e2ffd640 1 --2- 172.21.15.102:0/994899975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26e4104590 0x7f26e4196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e2ffd640 1 -- 172.21.15.102:0/994899975 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d8009560 con 0x7f26e40fe5b0 2024-03-26T19:06:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.805+0000 7f26e2ffd640 1 --2- 172.21.15.102:0/994899975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26e40fe5b0 0x7f26e4196110 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f26d400d6f0 tx=0x7f26d400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.806+0000 7f26cbfff640 1 -- 172.21.15.102:0/994899975 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26d4009a30 con 0x7f26e40fe5b0 2024-03-26T19:06:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.806+0000 7f26e9718640 1 -- 172.21.15.102:0/994899975 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e419f180 con 0x7f26e40fe5b0 2024-03-26T19:06:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.806+0000 7f26cbfff640 1 -- 172.21.15.102:0/994899975 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26d40054e0 con 0x7f26e40fe5b0 2024-03-26T19:06:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.806+0000 7f26cbfff640 1 -- 172.21.15.102:0/994899975 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26d4010460 con 0x7f26e40fe5b0 2024-03-26T19:06:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.806+0000 7f26e9718640 1 -- 172.21.15.102:0/994899975 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e419f6b0 con 0x7f26e40fe5b0 2024-03-26T19:06:25.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.807+0000 7f26e9718640 1 -- 172.21.15.102:0/994899975 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e404e000 con 0x7f26e40fe5b0 2024-03-26T19:06:25.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.807+0000 7f26cbfff640 1 -- 172.21.15.102:0/994899975 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f26d4009b90 con 0x7f26e40fe5b0 2024-03-26T19:06:25.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.808+0000 7f26cbfff640 1 --2- 172.21.15.102:0/994899975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f26cc06e3d0 0x7f26cc070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:25.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.808+0000 7f26cbfff640 1 -- 172.21.15.102:0/994899975 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f26d408e000 con 0x7f26e40fe5b0 2024-03-26T19:06:25.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.808+0000 7f26e27fc640 1 --2- 172.21.15.102:0/994899975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f26cc06e3d0 0x7f26cc070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:25.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.808+0000 7f26e27fc640 1 --2- 172.21.15.102:0/994899975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f26cc06e3d0 0x7f26cc070880 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f26e4067bc0 tx=0x7f26d8000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:25.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:25.816+0000 7f26cbfff640 1 -- 172.21.15.102:0/994899975 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f26d4057160 con 0x7f26e40fe5b0 2024-03-26T19:06:26.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.037+0000 7f26e9718640 1 -- 172.21.15.102:0/994899975 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f26e4067900 con 0x7f26e40fe5b0 2024-03-26T19:06:26.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.037+0000 7f26cbfff640 1 -- 172.21.15.102:0/994899975 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v43) v1 ==== 107+0+4889 (secure 0 0 0) 0x7f26d405a820 con 0x7f26e40fe5b0 2024-03-26T19:06:26.035 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:26.035 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14502,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2674848848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2674848848},{"type":"v1","addr":"172.21.15.102:6827","nonce":2674848848}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:01:03.863994+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":14490,"mds_1":24283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14490":{"gid":14490,"name":"cephfs.smithi102.hrbrsp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.102:6829/989894576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":989894576},{"type":"v1","addr":"172.21.15.102:6829","nonce":989894576}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24283":{"gid":24283,"name":"cephfs.smithi190.vykizb","rank":1,"incarnation":8,"state":"up:stopping","state_seq":4,"addr":"172.21.15.190:6825/2769333078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2769333078},{"type":"v1","addr":"172.21.15.190:6825","nonce":2769333078}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14490,"qdb_cluster":[14490]},"id":1}]} 2024-03-26T19:06:26.035 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 22 2024-03-26T19:06:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.039+0000 7f26e9718640 1 -- 172.21.15.102:0/994899975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f26cc06e3d0 msgr2=0x7f26cc070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.039+0000 7f26e9718640 1 --2- 172.21.15.102:0/994899975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f26cc06e3d0 0x7f26cc070880 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f26e4067bc0 tx=0x7f26d8000bc0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.039+0000 7f26e9718640 1 -- 172.21.15.102:0/994899975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26e40fe5b0 msgr2=0x7f26e4196110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.039+0000 7f26e9718640 1 --2- 172.21.15.102:0/994899975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26e40fe5b0 0x7f26e4196110 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f26d400d6f0 tx=0x7f26d400dbb0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.039+0000 7f26e9718640 1 -- 172.21.15.102:0/994899975 shutdown_connections 2024-03-26T19:06:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.039+0000 7f26e9718640 1 --2- 172.21.15.102:0/994899975 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f26cc06e3d0 0x7f26cc070880 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.039+0000 7f26e9718640 1 --2- 172.21.15.102:0/994899975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26e4104590 0x7f26e4196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.039+0000 7f26e9718640 1 --2- 172.21.15.102:0/994899975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26e40fe5b0 0x7f26e4196110 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.039+0000 7f26e9718640 1 -- 172.21.15.102:0/994899975 >> 172.21.15.102:0/994899975 conn(0x7f26e40743d0 msgr2=0x7f26e40fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.040+0000 7f26e9718640 1 -- 172.21.15.102:0/994899975 shutdown_connections 2024-03-26T19:06:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.040+0000 7f26e9718640 1 -- 172.21.15.102:0/994899975 wait complete. 2024-03-26T19:06:26.047 DEBUG:tasks.fs:max_mds reduced in epoch 22 2024-03-26T19:06:26.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-03-26T19:06:26.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.155+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2791455875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4a4067c60 msgr2=0x7fd4a410bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.155+0000 7fd4ab60b640 1 --2- 172.21.15.102:0/2791455875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4a4067c60 0x7fd4a410bbe0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd494009880 tx=0x7fd49402bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.155+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2791455875 shutdown_connections 2024-03-26T19:06:26.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.155+0000 7fd4ab60b640 1 --2- 172.21.15.102:0/2791455875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4a4067c60 0x7fd4a410bbe0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.155+0000 7fd4ab60b640 1 --2- 172.21.15.102:0/2791455875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4a4067350 0x7fd4a4067720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.155+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2791455875 >> 172.21.15.102:0/2791455875 conn(0x7fd4a40743d0 msgr2=0x7fd4a40747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:26.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.155+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2791455875 shutdown_connections 2024-03-26T19:06:26.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.155+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2791455875 wait complete. 2024-03-26T19:06:26.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4ab60b640 1 Processor -- start 2024-03-26T19:06:26.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4ab60b640 1 -- start start 2024-03-26T19:06:26.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4ab60b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4a4067350 0x7fd4a4197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:26.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4ab60b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4a4067c60 0x7fd4a4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:26.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4ab60b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a41980f0 con 0x7fd4a4067c60 2024-03-26T19:06:26.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4ab60b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a419bec0 con 0x7fd4a4067350 2024-03-26T19:06:26.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4a8ba6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4a4067c60 0x7fd4a4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4a8ba6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4a4067c60 0x7fd4a4197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59840/0 (socket says 172.21.15.102:59840) 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4a8ba6640 1 -- 172.21.15.102:0/2671724979 learned_addr learned my addr 172.21.15.102:0/2671724979 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4a93a7640 1 --2- 172.21.15.102:0/2671724979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4a4067350 0x7fd4a4197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4a8ba6640 1 -- 172.21.15.102:0/2671724979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4a4067350 msgr2=0x7fd4a4197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4a8ba6640 1 --2- 172.21.15.102:0/2671724979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4a4067350 0x7fd4a4197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.156+0000 7fd4a8ba6640 1 -- 172.21.15.102:0/2671724979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd494009560 con 0x7fd4a4067c60 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.157+0000 7fd4a8ba6640 1 --2- 172.21.15.102:0/2671724979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4a4067c60 0x7fd4a4197a60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd494009230 tx=0x7fd494005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.157+0000 7fd49a7fc640 1 -- 172.21.15.102:0/2671724979 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4940044a0 con 0x7fd4a4067c60 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.157+0000 7fd49a7fc640 1 -- 172.21.15.102:0/2671724979 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd494003ab0 con 0x7fd4a4067c60 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.157+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2671724979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a419c140 con 0x7fd4a4067c60 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.157+0000 7fd49a7fc640 1 -- 172.21.15.102:0/2671724979 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4940055f0 con 0x7fd4a4067c60 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.157+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2671724979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a419c640 con 0x7fd4a4067c60 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.158+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2671724979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a404e000 con 0x7fd4a4067c60 2024-03-26T19:06:26.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.158+0000 7fd49a7fc640 1 -- 172.21.15.102:0/2671724979 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd494004e60 con 0x7fd4a4067c60 2024-03-26T19:06:26.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.159+0000 7fd49a7fc640 1 --2- 172.21.15.102:0/2671724979 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd49006e3d0 0x7fd490070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:26.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.159+0000 7fd4a93a7640 1 --2- 172.21.15.102:0/2671724979 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd49006e3d0 0x7fd490070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:26.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.159+0000 7fd49a7fc640 1 -- 172.21.15.102:0/2671724979 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fd4940b07b0 con 0x7fd4a4067c60 2024-03-26T19:06:26.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.159+0000 7fd4a93a7640 1 --2- 172.21.15.102:0/2671724979 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd49006e3d0 0x7fd490070880 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0009bd0 tx=0x7fd4a0005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:26.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.164+0000 7fd49a7fc640 1 -- 172.21.15.102:0/2671724979 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd4940799c0 con 0x7fd4a4067c60 2024-03-26T19:06:26.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:26 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3343773971' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-03-26T19:06:26.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:26 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2468218525' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-26T19:06:26.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:26 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/994899975' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-26T19:06:26.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.383+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2671724979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7fd4a41092c0 con 0x7fd4a4067c60 2024-03-26T19:06:26.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.384+0000 7fd49a7fc640 1 -- 172.21.15.102:0/2671724979 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v43) v1 ==== 107+0+4890 (secure 0 0 0) 0x7fd49407d080 con 0x7fd4a4067c60 2024-03-26T19:06:26.381 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:26.381 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14502,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2674848848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2674848848},{"type":"v1","addr":"172.21.15.102:6827","nonce":2674848848}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:01:12.201799+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":14490,"mds_1":24283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14490":{"gid":14490,"name":"cephfs.smithi102.hrbrsp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.102:6829/989894576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":989894576},{"type":"v1","addr":"172.21.15.102:6829","nonce":989894576}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24283":{"gid":24283,"name":"cephfs.smithi190.vykizb","rank":1,"incarnation":8,"state":"up:stopping","state_seq":4,"addr":"172.21.15.190:6825/2769333078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2769333078},{"type":"v1","addr":"172.21.15.190:6825","nonce":2769333078}]},"join_fscid":1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14490,"qdb_cluster":[14490]},"id":1}]} 2024-03-26T19:06:26.381 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 23 2024-03-26T19:06:26.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.386+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2671724979 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd49006e3d0 msgr2=0x7fd490070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.386+0000 7fd4ab60b640 1 --2- 172.21.15.102:0/2671724979 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd49006e3d0 0x7fd490070880 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0009bd0 tx=0x7fd4a0005f90 comp rx=0 tx=0).stop 2024-03-26T19:06:26.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.386+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2671724979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4a4067c60 msgr2=0x7fd4a4197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.386+0000 7fd4ab60b640 1 --2- 172.21.15.102:0/2671724979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4a4067c60 0x7fd4a4197a60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd494009230 tx=0x7fd494005e50 comp rx=0 tx=0).stop 2024-03-26T19:06:26.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.386+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2671724979 shutdown_connections 2024-03-26T19:06:26.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.386+0000 7fd4ab60b640 1 --2- 172.21.15.102:0/2671724979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4a4067c60 0x7fd4a4197a60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.386+0000 7fd4ab60b640 1 --2- 172.21.15.102:0/2671724979 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd49006e3d0 0x7fd490070880 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.386+0000 7fd4ab60b640 1 --2- 172.21.15.102:0/2671724979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4a4067350 0x7fd4a4197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.386+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2671724979 >> 172.21.15.102:0/2671724979 conn(0x7fd4a40743d0 msgr2=0x7fd4a40fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:26.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.386+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2671724979 shutdown_connections 2024-03-26T19:06:26.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.386+0000 7fd4ab60b640 1 -- 172.21.15.102:0/2671724979 wait complete. 2024-03-26T19:06:26.395 DEBUG:tasks.fs:max_mds reduced in epoch 23 2024-03-26T19:06:26.396 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-03-26T19:06:26.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.525+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4209276436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c58104610 msgr2=0x7f3c581049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.525+0000 7f3c5c8f2640 1 --2- 172.21.15.102:0/4209276436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c58104610 0x7f3c581049e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009880 tx=0x7f3c4802bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.525+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4209276436 shutdown_connections 2024-03-26T19:06:26.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.525+0000 7f3c5c8f2640 1 --2- 172.21.15.102:0/4209276436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c58067310 0x7f3c58067780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.525+0000 7f3c5c8f2640 1 --2- 172.21.15.102:0/4209276436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c58104610 0x7f3c581049e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.525+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4209276436 >> 172.21.15.102:0/4209276436 conn(0x7f3c58074580 msgr2=0x7f3c58074980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:26.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.525+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4209276436 shutdown_connections 2024-03-26T19:06:26.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.525+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4209276436 wait complete. 2024-03-26T19:06:26.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.526+0000 7f3c5c8f2640 1 Processor -- start 2024-03-26T19:06:26.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.526+0000 7f3c5c8f2640 1 -- start start 2024-03-26T19:06:26.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.526+0000 7f3c5c8f2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c58067310 0x7f3c58195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:26.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c5c8f2640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c58104610 0x7f3c581957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:26.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c5c8f2640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58195e50 con 0x7f3c58104610 2024-03-26T19:06:26.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c5c8f2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58199c20 con 0x7f3c58067310 2024-03-26T19:06:26.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c4dd9b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c58104610 0x7f3c581957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:26.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c4dd9b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c58104610 0x7f3c581957c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59852/0 (socket says 172.21.15.102:59852) 2024-03-26T19:06:26.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c4dd9b640 1 -- 172.21.15.102:0/4120279533 learned_addr learned my addr 172.21.15.102:0/4120279533 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:26.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c5659c640 1 --2- 172.21.15.102:0/4120279533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c58067310 0x7f3c58195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:26.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c4dd9b640 1 -- 172.21.15.102:0/4120279533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c58067310 msgr2=0x7f3c58195280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c4dd9b640 1 --2- 172.21.15.102:0/4120279533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c58067310 0x7f3c58195280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c4dd9b640 1 -- 172.21.15.102:0/4120279533 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c48009560 con 0x7f3c58104610 2024-03-26T19:06:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c4dd9b640 1 --2- 172.21.15.102:0/4120279533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c58104610 0x7f3c581957c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000d6f0 tx=0x7f3c4000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c4ffff640 1 -- 172.21.15.102:0/4120279533 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c40009a30 con 0x7f3c58104610 2024-03-26T19:06:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4120279533 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c58199f00 con 0x7f3c58104610 2024-03-26T19:06:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c4ffff640 1 -- 172.21.15.102:0/4120279533 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c400054e0 con 0x7f3c58104610 2024-03-26T19:06:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c4ffff640 1 -- 172.21.15.102:0/4120279533 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c40010460 con 0x7f3c58104610 2024-03-26T19:06:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.527+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4120279533 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5819a4e0 con 0x7f3c58104610 2024-03-26T19:06:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.528+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4120279533 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5804e000 con 0x7f3c58104610 2024-03-26T19:06:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.529+0000 7f3c4ffff640 1 -- 172.21.15.102:0/4120279533 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3c40009b90 con 0x7f3c58104610 2024-03-26T19:06:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.529+0000 7f3c4ffff640 1 --2- 172.21.15.102:0/4120279533 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3c4406e3d0 0x7f3c44070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.530+0000 7f3c4ffff640 1 -- 172.21.15.102:0/4120279533 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f3c4008df50 con 0x7f3c58104610 2024-03-26T19:06:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.530+0000 7f3c5659c640 1 --2- 172.21.15.102:0/4120279533 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3c4406e3d0 0x7f3c44070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.530+0000 7f3c5659c640 1 --2- 172.21.15.102:0/4120279533 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3c4406e3d0 0x7f3c44070880 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3c48006010 tx=0x7f3c48005dd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:26.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:26 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3343773971' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-03-26T19:06:26.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:26 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2468218525' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-26T19:06:26.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:26 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/994899975' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-26T19:06:26.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.538+0000 7f3c4ffff640 1 -- 172.21.15.102:0/4120279533 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3c40057160 con 0x7f3c58104610 2024-03-26T19:06:26.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.764+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4120279533 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7f3c581965d0 con 0x7f3c58104610 2024-03-26T19:06:26.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.765+0000 7f3c4ffff640 1 -- 172.21.15.102:0/4120279533 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v43) v1 ==== 107+0+4091 (secure 0 0 0) 0x7f3c4005a820 con 0x7f3c58104610 2024-03-26T19:06:26.762 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:26.762 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14502,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2674848848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2674848848},{"type":"v1","addr":"172.21.15.102:6827","nonce":2674848848}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:01:37.203002+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14490},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_14490":{"gid":14490,"name":"cephfs.smithi102.hrbrsp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.102:6829/989894576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":989894576},{"type":"v1","addr":"172.21.15.102:6829","nonce":989894576}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14490,"qdb_cluster":[14490]},"id":1}]} 2024-03-26T19:06:26.763 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 24 2024-03-26T19:06:26.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.767+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4120279533 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3c4406e3d0 msgr2=0x7f3c44070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.767+0000 7f3c5c8f2640 1 --2- 172.21.15.102:0/4120279533 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3c4406e3d0 0x7f3c44070880 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3c48006010 tx=0x7f3c48005dd0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.767+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4120279533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c58104610 msgr2=0x7f3c581957c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.767+0000 7f3c5c8f2640 1 --2- 172.21.15.102:0/4120279533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c58104610 0x7f3c581957c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000d6f0 tx=0x7f3c4000dbb0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.767+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4120279533 shutdown_connections 2024-03-26T19:06:26.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.767+0000 7f3c5c8f2640 1 --2- 172.21.15.102:0/4120279533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c58104610 0x7f3c581957c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.767+0000 7f3c5c8f2640 1 --2- 172.21.15.102:0/4120279533 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3c4406e3d0 0x7f3c44070880 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.767+0000 7f3c5c8f2640 1 --2- 172.21.15.102:0/4120279533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c58067310 0x7f3c58195280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.767+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4120279533 >> 172.21.15.102:0/4120279533 conn(0x7f3c58074580 msgr2=0x7f3c580fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:26.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.767+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4120279533 shutdown_connections 2024-03-26T19:06:26.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.767+0000 7f3c5c8f2640 1 -- 172.21.15.102:0/4120279533 wait complete. 2024-03-26T19:06:26.775 DEBUG:tasks.fs:max_mds reduced in epoch 24 2024-03-26T19:06:26.775 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-03-26T19:06:26.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.888+0000 7fb093318640 1 -- 172.21.15.102:0/1648876997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08c0fe5b0 msgr2=0x7fb08c0fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.888+0000 7fb093318640 1 --2- 172.21.15.102:0/1648876997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08c0fe5b0 0x7fb08c0fea20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009880 tx=0x7fb07c02bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.888+0000 7fb093318640 1 -- 172.21.15.102:0/1648876997 shutdown_connections 2024-03-26T19:06:26.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.888+0000 7fb093318640 1 --2- 172.21.15.102:0/1648876997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08c0fe5b0 0x7fb08c0fea20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.888+0000 7fb093318640 1 --2- 172.21.15.102:0/1648876997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb08c104590 0x7fb08c104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.888+0000 7fb093318640 1 -- 172.21.15.102:0/1648876997 >> 172.21.15.102:0/1648876997 conn(0x7fb08c0743d0 msgr2=0x7fb08c0747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.888+0000 7fb093318640 1 -- 172.21.15.102:0/1648876997 shutdown_connections 2024-03-26T19:06:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.888+0000 7fb093318640 1 -- 172.21.15.102:0/1648876997 wait complete. 2024-03-26T19:06:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.889+0000 7fb093318640 1 Processor -- start 2024-03-26T19:06:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.889+0000 7fb093318640 1 -- start start 2024-03-26T19:06:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.889+0000 7fb093318640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb08c0fe5b0 0x7fb08c1941b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.889+0000 7fb093318640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08c104590 0x7fb08c1946f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.889+0000 7fb093318640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c194d80 con 0x7fb08c104590 2024-03-26T19:06:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.889+0000 7fb093318640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c198b50 con 0x7fb08c0fe5b0 2024-03-26T19:06:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.889+0000 7fb0908b3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08c104590 0x7fb08c1946f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.889+0000 7fb0908b3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08c104590 0x7fb08c1946f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59864/0 (socket says 172.21.15.102:59864) 2024-03-26T19:06:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.889+0000 7fb0908b3640 1 -- 172.21.15.102:0/2767948769 learned_addr learned my addr 172.21.15.102:0/2767948769 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.890+0000 7fb0910b4640 1 --2- 172.21.15.102:0/2767948769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb08c0fe5b0 0x7fb08c1941b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.890+0000 7fb0908b3640 1 -- 172.21.15.102:0/2767948769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb08c0fe5b0 msgr2=0x7fb08c1941b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.890+0000 7fb0908b3640 1 --2- 172.21.15.102:0/2767948769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb08c0fe5b0 0x7fb08c1941b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.890+0000 7fb0908b3640 1 -- 172.21.15.102:0/2767948769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c009560 con 0x7fb08c104590 2024-03-26T19:06:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.890+0000 7fb0908b3640 1 --2- 172.21.15.102:0/2767948769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08c104590 0x7fb08c1946f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009230 tx=0x7fb07c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.890+0000 7fb0827fc640 1 -- 172.21.15.102:0/2767948769 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb07c0035f0 con 0x7fb08c104590 2024-03-26T19:06:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.890+0000 7fb0827fc640 1 -- 172.21.15.102:0/2767948769 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb07c003c10 con 0x7fb08c104590 2024-03-26T19:06:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.890+0000 7fb093318640 1 -- 172.21.15.102:0/2767948769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08c198dd0 con 0x7fb08c104590 2024-03-26T19:06:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.890+0000 7fb0827fc640 1 -- 172.21.15.102:0/2767948769 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb07c005670 con 0x7fb08c104590 2024-03-26T19:06:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.890+0000 7fb093318640 1 -- 172.21.15.102:0/2767948769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08c199350 con 0x7fb08c104590 2024-03-26T19:06:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.891+0000 7fb093318640 1 -- 172.21.15.102:0/2767948769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08c04e000 con 0x7fb08c104590 2024-03-26T19:06:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.892+0000 7fb0827fc640 1 -- 172.21.15.102:0/2767948769 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb07c004dd0 con 0x7fb08c104590 2024-03-26T19:06:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.892+0000 7fb0827fc640 1 --2- 172.21.15.102:0/2767948769 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb07806e380 0x7fb078070830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.892+0000 7fb0827fc640 1 -- 172.21.15.102:0/2767948769 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fb07c0b0850 con 0x7fb08c104590 2024-03-26T19:06:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.892+0000 7fb0910b4640 1 --2- 172.21.15.102:0/2767948769 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb07806e380 0x7fb078070830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.893+0000 7fb0910b4640 1 --2- 172.21.15.102:0/2767948769 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb07806e380 0x7fb078070830 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb088005fd0 tx=0x7fb088005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:26.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:26.901+0000 7fb0827fc640 1 -- 172.21.15.102:0/2767948769 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb07c0799b0 con 0x7fb08c104590 2024-03-26T19:06:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.122+0000 7fb093318640 1 -- 172.21.15.102:0/2767948769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7fb08c195500 con 0x7fb08c104590 2024-03-26T19:06:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.123+0000 7fb0827fc640 1 -- 172.21.15.102:0/2767948769 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v43) v1 ==== 107+0+4870 (secure 0 0 0) 0x7fb07c07d070 con 0x7fb08c104590 2024-03-26T19:06:27.121 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:27.121 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14502,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2674848848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2674848848},{"type":"v1","addr":"172.21.15.102:6827","nonce":2674848848}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":44315,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/925070513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":925070513},{"type":"v1","addr":"172.21.15.190:6825","nonce":925070513}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:01:37.203002+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14490},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_14490":{"gid":14490,"name":"cephfs.smithi102.hrbrsp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.102:6829/989894576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":989894576},{"type":"v1","addr":"172.21.15.102:6829","nonce":989894576}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14490,"qdb_cluster":[14490]},"id":1}]} 2024-03-26T19:06:27.122 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-26T19:06:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.126+0000 7fb093318640 1 -- 172.21.15.102:0/2767948769 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb07806e380 msgr2=0x7fb078070830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.126+0000 7fb093318640 1 --2- 172.21.15.102:0/2767948769 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb07806e380 0x7fb078070830 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb088005fd0 tx=0x7fb088005850 comp rx=0 tx=0).stop 2024-03-26T19:06:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.126+0000 7fb093318640 1 -- 172.21.15.102:0/2767948769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08c104590 msgr2=0x7fb08c1946f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.126+0000 7fb093318640 1 --2- 172.21.15.102:0/2767948769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08c104590 0x7fb08c1946f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009230 tx=0x7fb07c004040 comp rx=0 tx=0).stop 2024-03-26T19:06:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.126+0000 7fb093318640 1 -- 172.21.15.102:0/2767948769 shutdown_connections 2024-03-26T19:06:27.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.126+0000 7fb093318640 1 --2- 172.21.15.102:0/2767948769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08c104590 0x7fb08c1946f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.126+0000 7fb093318640 1 --2- 172.21.15.102:0/2767948769 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb07806e380 0x7fb078070830 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.126+0000 7fb093318640 1 --2- 172.21.15.102:0/2767948769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb08c0fe5b0 0x7fb08c1941b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.126+0000 7fb093318640 1 -- 172.21.15.102:0/2767948769 >> 172.21.15.102:0/2767948769 conn(0x7fb08c0743d0 msgr2=0x7fb08c0fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:27.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.126+0000 7fb093318640 1 -- 172.21.15.102:0/2767948769 shutdown_connections 2024-03-26T19:06:27.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.126+0000 7fb093318640 1 -- 172.21.15.102:0/2767948769 wait complete. 2024-03-26T19:06:27.135 DEBUG:tasks.fs:max_mds reduced in epoch 25 2024-03-26T19:06:27.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-03-26T19:06:27.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.245+0000 7fac1893c640 1 -- 172.21.15.102:0/2866125493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac14105660 msgr2=0x7fac14105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:27.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.245+0000 7fac1893c640 1 --2- 172.21.15.102:0/2866125493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac14105660 0x7fac14105a30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fac04009880 tx=0x7fac0402bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.245+0000 7fac1893c640 1 -- 172.21.15.102:0/2866125493 shutdown_connections 2024-03-26T19:06:27.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.245+0000 7fac1893c640 1 --2- 172.21.15.102:0/2866125493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac140ff650 0x7fac140ffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.245+0000 7fac1893c640 1 --2- 172.21.15.102:0/2866125493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac14105660 0x7fac14105a30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.245+0000 7fac1893c640 1 -- 172.21.15.102:0/2866125493 >> 172.21.15.102:0/2866125493 conn(0x7fac140fb170 msgr2=0x7fac140fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:27.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.245+0000 7fac1893c640 1 -- 172.21.15.102:0/2866125493 shutdown_connections 2024-03-26T19:06:27.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.245+0000 7fac1893c640 1 -- 172.21.15.102:0/2866125493 wait complete. 2024-03-26T19:06:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.246+0000 7fac1893c640 1 Processor -- start 2024-03-26T19:06:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.246+0000 7fac1893c640 1 -- start start 2024-03-26T19:06:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.246+0000 7fac1893c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac140ff650 0x7fac141952f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.246+0000 7fac1893c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac14105660 0x7fac14195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.246+0000 7fac1893c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac14195ec0 con 0x7fac14105660 2024-03-26T19:06:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.246+0000 7fac1893c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac14199c90 con 0x7fac140ff650 2024-03-26T19:06:27.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.246+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac14105660 0x7fac14195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:27.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.246+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac14105660 0x7fac14195830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59880/0 (socket says 172.21.15.102:59880) 2024-03-26T19:06:27.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.246+0000 7fac0bfff640 1 -- 172.21.15.102:0/2841655730 learned_addr learned my addr 172.21.15.102:0/2841655730 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:27.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.247+0000 7fac1259c640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac140ff650 0x7fac141952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:27.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.247+0000 7fac0bfff640 1 -- 172.21.15.102:0/2841655730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac140ff650 msgr2=0x7fac141952f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:27.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.247+0000 7fac0bfff640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac140ff650 0x7fac141952f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.247+0000 7fac0bfff640 1 -- 172.21.15.102:0/2841655730 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac04009560 con 0x7fac14105660 2024-03-26T19:06:27.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.247+0000 7fac1259c640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac140ff650 0x7fac141952f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:06:27.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.247+0000 7fac0bfff640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac14105660 0x7fac14195830 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00d8d0 tx=0x7fabfc00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:27.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.247+0000 7fac0b7fe640 1 -- 172.21.15.102:0/2841655730 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabfc00bea0 con 0x7fac14105660 2024-03-26T19:06:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.247+0000 7fac0b7fe640 1 -- 172.21.15.102:0/2841655730 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabfc009c00 con 0x7fac14105660 2024-03-26T19:06:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.247+0000 7fac1893c640 1 -- 172.21.15.102:0/2841655730 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac14199f70 con 0x7fac14105660 2024-03-26T19:06:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.247+0000 7fac0b7fe640 1 -- 172.21.15.102:0/2841655730 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabfc00f710 con 0x7fac14105660 2024-03-26T19:06:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.247+0000 7fac1893c640 1 -- 172.21.15.102:0/2841655730 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1419a550 con 0x7fac14105660 2024-03-26T19:06:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.248+0000 7fac1893c640 1 -- 172.21.15.102:0/2841655730 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1404e000 con 0x7fac14105660 2024-03-26T19:06:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.249+0000 7fac0b7fe640 1 -- 172.21.15.102:0/2841655730 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fabfc00f990 con 0x7fac14105660 2024-03-26T19:06:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.249+0000 7fac0b7fe640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fac0006e3d0 0x7fac00070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.250+0000 7fac0b7fe640 1 -- 172.21.15.102:0/2841655730 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fabfc08e1b0 con 0x7fac14105660 2024-03-26T19:06:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.250+0000 7fac1259c640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fac0006e3d0 0x7fac00070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.250+0000 7fac1259c640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fac0006e3d0 0x7fac00070880 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fac04005ec0 tx=0x7fac04005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:27.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.254+0000 7fac0b7fe640 1 -- 172.21.15.102:0/2841655730 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fabfc057310 con 0x7fac14105660 2024-03-26T19:06:27.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.508+0000 7fac1893c640 1 -- 172.21.15.102:0/2841655730 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7fac14196640 con 0x7fac14105660 2024-03-26T19:06:27.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.509+0000 7fac0b7fe640 1 -- 172.21.15.102:0/2841655730 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v43) v1 ==== 107+0+4088 (secure 0 0 0) 0x7fabfc05a9d0 con 0x7fac14105660 2024-03-26T19:06:27.506 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:27.506 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":44315,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/925070513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":925070513},{"type":"v1","addr":"172.21.15.190:6825","nonce":925070513}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:01:37.203002+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14490},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_14490":{"gid":14490,"name":"cephfs.smithi102.hrbrsp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.102:6829/989894576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":989894576},{"type":"v1","addr":"172.21.15.102:6829","nonce":989894576}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14490,"qdb_cluster":[14490]},"id":1}]} 2024-03-26T19:06:27.506 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 26 2024-03-26T19:06:27.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.511+0000 7fac1893c640 1 -- 172.21.15.102:0/2841655730 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fac0006e3d0 msgr2=0x7fac00070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:27.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.511+0000 7fac1893c640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fac0006e3d0 0x7fac00070880 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fac04005ec0 tx=0x7fac04005fd0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.511+0000 7fac1893c640 1 -- 172.21.15.102:0/2841655730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac14105660 msgr2=0x7fac14195830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:27.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.511+0000 7fac1893c640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac14105660 0x7fac14195830 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00d8d0 tx=0x7fabfc00dd90 comp rx=0 tx=0).stop 2024-03-26T19:06:27.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.511+0000 7fac1893c640 1 -- 172.21.15.102:0/2841655730 shutdown_connections 2024-03-26T19:06:27.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.511+0000 7fac1893c640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac14105660 0x7fac14195830 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.511+0000 7fac1893c640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fac0006e3d0 0x7fac00070880 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.511+0000 7fac1893c640 1 --2- 172.21.15.102:0/2841655730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac140ff650 0x7fac141952f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.511+0000 7fac1893c640 1 -- 172.21.15.102:0/2841655730 >> 172.21.15.102:0/2841655730 conn(0x7fac140fb170 msgr2=0x7fac140fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:27.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.511+0000 7fac1893c640 1 -- 172.21.15.102:0/2841655730 shutdown_connections 2024-03-26T19:06:27.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.511+0000 7fac1893c640 1 -- 172.21.15.102:0/2841655730 wait complete. 2024-03-26T19:06:27.521 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:27 smithi102 ceph-mon[78322]: pgmap v331: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:27.521 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:27 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2671724979' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-26T19:06:27.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:27 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/4120279533' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-26T19:06:27.522 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:27 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2767948769' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-26T19:06:27.524 DEBUG:tasks.fs:max_mds reduced in epoch 26 2024-03-26T19:06:27.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-03-26T19:06:27.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:27 smithi190 ceph-mon[65689]: pgmap v331: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:27.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:27 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2671724979' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-26T19:06:27.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:27 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/4120279533' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-26T19:06:27.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:27 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2767948769' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-26T19:06:27.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.636+0000 7fd82ec54640 1 -- 172.21.15.102:0/2337025008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828067350 msgr2=0x7fd8280677c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:27.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.636+0000 7fd82ec54640 1 --2- 172.21.15.102:0/2337025008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828067350 0x7fd8280677c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd818009880 tx=0x7fd81802bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.636+0000 7fd82ec54640 1 -- 172.21.15.102:0/2337025008 shutdown_connections 2024-03-26T19:06:27.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.636+0000 7fd82ec54640 1 --2- 172.21.15.102:0/2337025008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828067350 0x7fd8280677c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.636+0000 7fd82ec54640 1 --2- 172.21.15.102:0/2337025008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd828105690 0x7fd828105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.636+0000 7fd82ec54640 1 -- 172.21.15.102:0/2337025008 >> 172.21.15.102:0/2337025008 conn(0x7fd8280743d0 msgr2=0x7fd8280747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:27.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.636+0000 7fd82ec54640 1 -- 172.21.15.102:0/2337025008 shutdown_connections 2024-03-26T19:06:27.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.636+0000 7fd82ec54640 1 -- 172.21.15.102:0/2337025008 wait complete. 2024-03-26T19:06:27.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd82ec54640 1 Processor -- start 2024-03-26T19:06:27.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd82ec54640 1 -- start start 2024-03-26T19:06:27.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd82ec54640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd828067350 0x7fd828195340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:27.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd82ec54640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828105690 0x7fd828195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:27.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd82ec54640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd828195e80 con 0x7fd828105690 2024-03-26T19:06:27.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd82ec54640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd828195ff0 con 0x7fd828067350 2024-03-26T19:06:27.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd81ffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828105690 0x7fd828195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:27.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd81ffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828105690 0x7fd828195880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59900/0 (socket says 172.21.15.102:59900) 2024-03-26T19:06:27.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd81ffff640 1 -- 172.21.15.102:0/1270530539 learned_addr learned my addr 172.21.15.102:0/1270530539 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:27.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd82c9f0640 1 --2- 172.21.15.102:0/1270530539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd828067350 0x7fd828195340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:27.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd81ffff640 1 -- 172.21.15.102:0/1270530539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd828067350 msgr2=0x7fd828195340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:27.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd81ffff640 1 --2- 172.21.15.102:0/1270530539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd828067350 0x7fd828195340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.637+0000 7fd81ffff640 1 -- 172.21.15.102:0/1270530539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd818009560 con 0x7fd828105690 2024-03-26T19:06:27.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.638+0000 7fd81ffff640 1 --2- 172.21.15.102:0/1270530539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828105690 0x7fd828195880 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd818009230 tx=0x7fd818004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:27.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.638+0000 7fd81dffb640 1 -- 172.21.15.102:0/1270530539 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8180035f0 con 0x7fd828105690 2024-03-26T19:06:27.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.638+0000 7fd81dffb640 1 -- 172.21.15.102:0/1270530539 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd818003c10 con 0x7fd828105690 2024-03-26T19:06:27.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.638+0000 7fd82ec54640 1 -- 172.21.15.102:0/1270530539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd828199e20 con 0x7fd828105690 2024-03-26T19:06:27.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.638+0000 7fd81dffb640 1 -- 172.21.15.102:0/1270530539 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd818034440 con 0x7fd828105690 2024-03-26T19:06:27.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.638+0000 7fd82ec54640 1 -- 172.21.15.102:0/1270530539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82819a320 con 0x7fd828105690 2024-03-26T19:06:27.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.639+0000 7fd82ec54640 1 -- 172.21.15.102:0/1270530539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82804e000 con 0x7fd828105690 2024-03-26T19:06:27.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.640+0000 7fd81dffb640 1 -- 172.21.15.102:0/1270530539 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd818004f60 con 0x7fd828105690 2024-03-26T19:06:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.640+0000 7fd81dffb640 1 --2- 172.21.15.102:0/1270530539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81006e3d0 0x7fd810070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.640+0000 7fd81dffb640 1 -- 172.21.15.102:0/1270530539 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fd8180b0820 con 0x7fd828105690 2024-03-26T19:06:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.640+0000 7fd82c9f0640 1 --2- 172.21.15.102:0/1270530539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81006e3d0 0x7fd810070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.641+0000 7fd82c9f0640 1 --2- 172.21.15.102:0/1270530539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81006e3d0 0x7fd810070880 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd82400abe0 tx=0x7fd82400a2d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:27.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.645+0000 7fd81dffb640 1 -- 172.21.15.102:0/1270530539 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd818079980 con 0x7fd828105690 2024-03-26T19:06:27.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.863+0000 7fd82ec54640 1 -- 172.21.15.102:0/1270530539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7fd828107aa0 con 0x7fd828105690 2024-03-26T19:06:27.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.864+0000 7fd81dffb640 1 -- 172.21.15.102:0/1270530539 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v43) v1 ==== 107+0+4938 (secure 0 0 0) 0x7fd81807d040 con 0x7fd828105690 2024-03-26T19:06:27.860 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:27.860 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44315,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/925070513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":925070513},{"type":"v1","addr":"172.21.15.190:6825","nonce":925070513}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:01:37.203002+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14490},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_14490":{"gid":14490,"name":"cephfs.smithi102.hrbrsp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.102:6829/989894576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":989894576},{"type":"v1","addr":"172.21.15.102:6829","nonce":989894576}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14490,"qdb_cluster":[14490]},"id":1}]} 2024-03-26T19:06:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-03-26T19:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.866+0000 7fd82ec54640 1 -- 172.21.15.102:0/1270530539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81006e3d0 msgr2=0x7fd810070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.866+0000 7fd82ec54640 1 --2- 172.21.15.102:0/1270530539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81006e3d0 0x7fd810070880 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd82400abe0 tx=0x7fd82400a2d0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.866+0000 7fd82ec54640 1 -- 172.21.15.102:0/1270530539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828105690 msgr2=0x7fd828195880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.866+0000 7fd82ec54640 1 --2- 172.21.15.102:0/1270530539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828105690 0x7fd828195880 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd818009230 tx=0x7fd818004040 comp rx=0 tx=0).stop 2024-03-26T19:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.866+0000 7fd82ec54640 1 -- 172.21.15.102:0/1270530539 shutdown_connections 2024-03-26T19:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.866+0000 7fd82ec54640 1 --2- 172.21.15.102:0/1270530539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd828105690 0x7fd828195880 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.866+0000 7fd82ec54640 1 --2- 172.21.15.102:0/1270530539 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd81006e3d0 0x7fd810070880 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.866+0000 7fd82ec54640 1 --2- 172.21.15.102:0/1270530539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd828067350 0x7fd828195340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.866+0000 7fd82ec54640 1 -- 172.21.15.102:0/1270530539 >> 172.21.15.102:0/1270530539 conn(0x7fd8280743d0 msgr2=0x7fd8280fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.866+0000 7fd82ec54640 1 -- 172.21.15.102:0/1270530539 shutdown_connections 2024-03-26T19:06:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:27.866+0000 7fd82ec54640 1 -- 172.21.15.102:0/1270530539 wait complete. 2024-03-26T19:06:27.876 DEBUG:tasks.fs:max_mds reduced in epoch 27 2024-03-26T19:06:27.876 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-03-26T19:06:28.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.015+0000 7fc0cbf18640 1 -- 172.21.15.102:0/1390272480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c4104590 msgr2=0x7fc0c4104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:28.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.015+0000 7fc0cbf18640 1 --2- 172.21.15.102:0/1390272480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c4104590 0x7fc0c4104960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0009880 tx=0x7fc0c002bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.015+0000 7fc0cbf18640 1 -- 172.21.15.102:0/1390272480 shutdown_connections 2024-03-26T19:06:28.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.015+0000 7fc0cbf18640 1 --2- 172.21.15.102:0/1390272480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0c40fe5b0 0x7fc0c40fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.015+0000 7fc0cbf18640 1 --2- 172.21.15.102:0/1390272480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c4104590 0x7fc0c4104960 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.015+0000 7fc0cbf18640 1 -- 172.21.15.102:0/1390272480 >> 172.21.15.102:0/1390272480 conn(0x7fc0c40743d0 msgr2=0x7fc0c40747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:28.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.015+0000 7fc0cbf18640 1 -- 172.21.15.102:0/1390272480 shutdown_connections 2024-03-26T19:06:28.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.015+0000 7fc0cbf18640 1 -- 172.21.15.102:0/1390272480 wait complete. 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0cbf18640 1 Processor -- start 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0cbf18640 1 -- start start 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0cbf18640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0c40fe5b0 0x7fc0c41952b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0cbf18640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c4104590 0x7fc0c41957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0cbf18640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c4195e80 con 0x7fc0c4104590 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0cbf18640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c4199c50 con 0x7fc0c40fe5b0 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0c94b3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c4104590 0x7fc0c41957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0c94b3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c4104590 0x7fc0c41957f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59910/0 (socket says 172.21.15.102:59910) 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0c94b3640 1 -- 172.21.15.102:0/197768029 learned_addr learned my addr 172.21.15.102:0/197768029 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0c9cb4640 1 --2- 172.21.15.102:0/197768029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0c40fe5b0 0x7fc0c41952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0c94b3640 1 -- 172.21.15.102:0/197768029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0c40fe5b0 msgr2=0x7fc0c41952b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0c94b3640 1 --2- 172.21.15.102:0/197768029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0c40fe5b0 0x7fc0c41952b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0c94b3640 1 -- 172.21.15.102:0/197768029 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c0009560 con 0x7fc0c4104590 2024-03-26T19:06:28.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.016+0000 7fc0c94b3640 1 --2- 172.21.15.102:0/197768029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c4104590 0x7fc0c41957f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400d6f0 tx=0x7fc0b400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:28.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.017+0000 7fc0baffd640 1 -- 172.21.15.102:0/197768029 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0b4009a30 con 0x7fc0c4104590 2024-03-26T19:06:28.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.017+0000 7fc0cbf18640 1 -- 172.21.15.102:0/197768029 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c4199f30 con 0x7fc0c4104590 2024-03-26T19:06:28.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.017+0000 7fc0baffd640 1 -- 172.21.15.102:0/197768029 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0b40054e0 con 0x7fc0c4104590 2024-03-26T19:06:28.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.017+0000 7fc0baffd640 1 -- 172.21.15.102:0/197768029 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0b4010460 con 0x7fc0c4104590 2024-03-26T19:06:28.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.017+0000 7fc0cbf18640 1 -- 172.21.15.102:0/197768029 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c419a510 con 0x7fc0c4104590 2024-03-26T19:06:28.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.018+0000 7fc0cbf18640 1 -- 172.21.15.102:0/197768029 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c404e000 con 0x7fc0c4104590 2024-03-26T19:06:28.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.018+0000 7fc0baffd640 1 -- 172.21.15.102:0/197768029 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc0b4009b90 con 0x7fc0c4104590 2024-03-26T19:06:28.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.018+0000 7fc0baffd640 1 --2- 172.21.15.102:0/197768029 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc0b006e3d0 0x7fc0b0070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:28.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.019+0000 7fc0baffd640 1 -- 172.21.15.102:0/197768029 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc0b408df50 con 0x7fc0c4104590 2024-03-26T19:06:28.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.019+0000 7fc0c9cb4640 1 --2- 172.21.15.102:0/197768029 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc0b006e3d0 0x7fc0b0070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:28.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.019+0000 7fc0c9cb4640 1 --2- 172.21.15.102:0/197768029 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc0b006e3d0 0x7fc0b0070880 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0006010 tx=0x7fc0c0000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:28.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.023+0000 7fc0baffd640 1 -- 172.21.15.102:0/197768029 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc0b4057160 con 0x7fc0c4104590 2024-03-26T19:06:28.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.265+0000 7fc0cbf18640 1 -- 172.21.15.102:0/197768029 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7fc0c4196600 con 0x7fc0c4104590 2024-03-26T19:06:28.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.266+0000 7fc0baffd640 1 -- 172.21.15.102:0/197768029 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v43) v1 ==== 107+0+4139 (secure 0 0 0) 0x7fc0b405a820 con 0x7fc0c4104590 2024-03-26T19:06:28.262 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:28.262 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44315,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/925070513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":925070513},{"type":"v1","addr":"172.21.15.190:6825","nonce":925070513}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:06.598806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[1],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-26T19:06:28.262 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 28 2024-03-26T19:06:28.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.268+0000 7fc0cbf18640 1 -- 172.21.15.102:0/197768029 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc0b006e3d0 msgr2=0x7fc0b0070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:28.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.268+0000 7fc0cbf18640 1 --2- 172.21.15.102:0/197768029 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc0b006e3d0 0x7fc0b0070880 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0006010 tx=0x7fc0c0000bc0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.268+0000 7fc0cbf18640 1 -- 172.21.15.102:0/197768029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c4104590 msgr2=0x7fc0c41957f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:28.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.268+0000 7fc0cbf18640 1 --2- 172.21.15.102:0/197768029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c4104590 0x7fc0c41957f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400d6f0 tx=0x7fc0b400dbb0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.268+0000 7fc0cbf18640 1 -- 172.21.15.102:0/197768029 shutdown_connections 2024-03-26T19:06:28.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.268+0000 7fc0cbf18640 1 --2- 172.21.15.102:0/197768029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c4104590 0x7fc0c41957f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.268+0000 7fc0cbf18640 1 --2- 172.21.15.102:0/197768029 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc0b006e3d0 0x7fc0b0070880 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.268+0000 7fc0cbf18640 1 --2- 172.21.15.102:0/197768029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0c40fe5b0 0x7fc0c41952b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.268+0000 7fc0cbf18640 1 -- 172.21.15.102:0/197768029 >> 172.21.15.102:0/197768029 conn(0x7fc0c40743d0 msgr2=0x7fc0c40fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:28.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.268+0000 7fc0cbf18640 1 -- 172.21.15.102:0/197768029 shutdown_connections 2024-03-26T19:06:28.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.268+0000 7fc0cbf18640 1 -- 172.21.15.102:0/197768029 wait complete. 2024-03-26T19:06:28.275 DEBUG:tasks.fs:max_mds reduced in epoch 28 2024-03-26T19:06:28.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-03-26T19:06:28.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.384+0000 7f9705a0b640 1 -- 172.21.15.102:0/834919180 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97000fe5b0 msgr2=0x7f97000fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.384+0000 7f9705a0b640 1 --2- 172.21.15.102:0/834919180 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97000fe5b0 0x7f97000fea20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009880 tx=0x7f96e802bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.384+0000 7f9705a0b640 1 -- 172.21.15.102:0/834919180 shutdown_connections 2024-03-26T19:06:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.384+0000 7f9705a0b640 1 --2- 172.21.15.102:0/834919180 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97000fe5b0 0x7f97000fea20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.384+0000 7f9705a0b640 1 --2- 172.21.15.102:0/834919180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9700104590 0x7f9700104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.384+0000 7f9705a0b640 1 -- 172.21.15.102:0/834919180 >> 172.21.15.102:0/834919180 conn(0x7f97000743d0 msgr2=0x7f97000747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.384+0000 7f9705a0b640 1 -- 172.21.15.102:0/834919180 shutdown_connections 2024-03-26T19:06:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.384+0000 7f9705a0b640 1 -- 172.21.15.102:0/834919180 wait complete. 2024-03-26T19:06:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.384+0000 7f9705a0b640 1 Processor -- start 2024-03-26T19:06:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f9705a0b640 1 -- start start 2024-03-26T19:06:28.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f9705a0b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97000fe5b0 0x7f97001952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:28.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f9705a0b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9700104590 0x7f9700195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:28.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f9705a0b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9700195eb0 con 0x7f9700104590 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f9705a0b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9700199c80 con 0x7f97000fe5b0 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f96f67fc640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9700104590 0x7f9700195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f96f67fc640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9700104590 0x7f9700195820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59924/0 (socket says 172.21.15.102:59924) 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f96f67fc640 1 -- 172.21.15.102:0/1024523672 learned_addr learned my addr 172.21.15.102:0/1024523672 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f96feffd640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97000fe5b0 0x7f97001952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f96f67fc640 1 -- 172.21.15.102:0/1024523672 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97000fe5b0 msgr2=0x7f97001952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f96f67fc640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97000fe5b0 0x7f97001952e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f96f67fc640 1 -- 172.21.15.102:0/1024523672 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e8009560 con 0x7f9700104590 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f96feffd640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97000fe5b0 0x7f97001952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.385+0000 7f96f67fc640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9700104590 0x7f9700195820 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009230 tx=0x7f96e8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.386+0000 7f9704a09640 1 -- 172.21.15.102:0/1024523672 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96e80035f0 con 0x7f9700104590 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.386+0000 7f9704a09640 1 -- 172.21.15.102:0/1024523672 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96e8003c10 con 0x7f9700104590 2024-03-26T19:06:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.386+0000 7f9705a0b640 1 -- 172.21.15.102:0/1024523672 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9700199f00 con 0x7f9700104590 2024-03-26T19:06:28.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.386+0000 7f9704a09640 1 -- 172.21.15.102:0/1024523672 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96e8005670 con 0x7f9700104590 2024-03-26T19:06:28.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.386+0000 7f9705a0b640 1 -- 172.21.15.102:0/1024523672 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970019a480 con 0x7f9700104590 2024-03-26T19:06:28.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.387+0000 7f9705a0b640 1 -- 172.21.15.102:0/1024523672 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970004e000 con 0x7f9700104590 2024-03-26T19:06:28.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.387+0000 7f9704a09640 1 -- 172.21.15.102:0/1024523672 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f96e8004dd0 con 0x7f9700104590 2024-03-26T19:06:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.388+0000 7f9704a09640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96ec06e380 0x7f96ec070830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.388+0000 7f9704a09640 1 -- 172.21.15.102:0/1024523672 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f96e80b0850 con 0x7f9700104590 2024-03-26T19:06:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.388+0000 7f96feffd640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96ec06e380 0x7f96ec070830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.388+0000 7f96feffd640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96ec06e380 0x7f96ec070830 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f96f0009b40 tx=0x7f96f0005e50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.392+0000 7f9704a09640 1 -- 172.21.15.102:0/1024523672 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f96e80799b0 con 0x7f9700104590 2024-03-26T19:06:28.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:28 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2841655730' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-26T19:06:28.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:28 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1270530539' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-26T19:06:28.630 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:28 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2841655730' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-26T19:06:28.630 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:28 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1270530539' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-26T19:06:28.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.656+0000 7f9705a0b640 1 -- 172.21.15.102:0/1024523672 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7f9700196630 con 0x7f9700104590 2024-03-26T19:06:28.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.656+0000 7f9704a09640 1 -- 172.21.15.102:0/1024523672 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v43) v1 ==== 107+0+4150 (secure 0 0 0) 0x7f96e807d070 con 0x7f9700104590 2024-03-26T19:06:28.652 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44315,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/925070513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":925070513},{"type":"v1","addr":"172.21.15.190:6825","nonce":925070513}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:06.605064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24293},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_24293":{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":0,"incarnation":29,"state":"up:replay","state_seq":2,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-26T19:06:28.653 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 29 2024-03-26T19:06:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.659+0000 7f9705a0b640 1 -- 172.21.15.102:0/1024523672 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96ec06e380 msgr2=0x7f96ec070830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.659+0000 7f9705a0b640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96ec06e380 0x7f96ec070830 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f96f0009b40 tx=0x7f96f0005e50 comp rx=0 tx=0).stop 2024-03-26T19:06:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.659+0000 7f9705a0b640 1 -- 172.21.15.102:0/1024523672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9700104590 msgr2=0x7f9700195820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.659+0000 7f9705a0b640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9700104590 0x7f9700195820 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009230 tx=0x7f96e8004040 comp rx=0 tx=0).stop 2024-03-26T19:06:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.659+0000 7f9705a0b640 1 -- 172.21.15.102:0/1024523672 shutdown_connections 2024-03-26T19:06:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.659+0000 7f9705a0b640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9700104590 0x7f9700195820 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.659+0000 7f9705a0b640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f96ec06e380 0x7f96ec070830 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.659+0000 7f9705a0b640 1 --2- 172.21.15.102:0/1024523672 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97000fe5b0 0x7f97001952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.659+0000 7f9705a0b640 1 -- 172.21.15.102:0/1024523672 >> 172.21.15.102:0/1024523672 conn(0x7f97000743d0 msgr2=0x7f9700067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:28.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.659+0000 7f9705a0b640 1 -- 172.21.15.102:0/1024523672 shutdown_connections 2024-03-26T19:06:28.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.659+0000 7f9705a0b640 1 -- 172.21.15.102:0/1024523672 wait complete. 2024-03-26T19:06:28.666 DEBUG:tasks.fs:max_mds reduced in epoch 29 2024-03-26T19:06:28.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-03-26T19:06:28.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.785+0000 7f577780b640 1 -- 172.21.15.102:0/3710437850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57701056a0 msgr2=0x7f5770105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:28.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.785+0000 7f577780b640 1 --2- 172.21.15.102:0/3710437850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57701056a0 0x7f5770105a70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f576c009880 tx=0x7f576c02bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.785+0000 7f577780b640 1 -- 172.21.15.102:0/3710437850 shutdown_connections 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.785+0000 7f577780b640 1 --2- 172.21.15.102:0/3710437850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f57700ff690 0x7f57700ffb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.785+0000 7f577780b640 1 --2- 172.21.15.102:0/3710437850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57701056a0 0x7f5770105a70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.785+0000 7f577780b640 1 -- 172.21.15.102:0/3710437850 >> 172.21.15.102:0/3710437850 conn(0x7f57700fb1d0 msgr2=0x7f57700fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.785+0000 7f577780b640 1 -- 172.21.15.102:0/3710437850 shutdown_connections 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.785+0000 7f577780b640 1 -- 172.21.15.102:0/3710437850 wait complete. 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.785+0000 7f577780b640 1 Processor -- start 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.785+0000 7f577780b640 1 -- start start 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f577780b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f57700ff690 0x7f5770195310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f577780b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57701056a0 0x7f5770195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f577780b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5770195ee0 con 0x7f57701056a0 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f577780b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5770199cb0 con 0x7f57700ff690 2024-03-26T19:06:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f5774da6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57701056a0 0x7f5770195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f5774da6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57701056a0 0x7f5770195850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56906/0 (socket says 172.21.15.102:56906) 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f5774da6640 1 -- 172.21.15.102:0/1717592944 learned_addr learned my addr 172.21.15.102:0/1717592944 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f57755a7640 1 --2- 172.21.15.102:0/1717592944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f57700ff690 0x7f5770195310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f5774da6640 1 -- 172.21.15.102:0/1717592944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f57700ff690 msgr2=0x7f5770195310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f5774da6640 1 --2- 172.21.15.102:0/1717592944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f57700ff690 0x7f5770195310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f5774da6640 1 -- 172.21.15.102:0/1717592944 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c009560 con 0x7f57701056a0 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f5774da6640 1 --2- 172.21.15.102:0/1717592944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57701056a0 0x7f5770195850 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f576000d6f0 tx=0x7f576000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f57667fc640 1 -- 172.21.15.102:0/1717592944 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5760009a30 con 0x7f57701056a0 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f57667fc640 1 -- 172.21.15.102:0/1717592944 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57600054e0 con 0x7f57701056a0 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f57667fc640 1 -- 172.21.15.102:0/1717592944 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5760010460 con 0x7f57701056a0 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f577780b640 1 -- 172.21.15.102:0/1717592944 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5770199f90 con 0x7f57701056a0 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.786+0000 7f577780b640 1 -- 172.21.15.102:0/1717592944 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577019a570 con 0x7f57701056a0 2024-03-26T19:06:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.787+0000 7f577780b640 1 -- 172.21.15.102:0/1717592944 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577004e000 con 0x7f57701056a0 2024-03-26T19:06:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.788+0000 7f57667fc640 1 -- 172.21.15.102:0/1717592944 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5760009b90 con 0x7f57701056a0 2024-03-26T19:06:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.788+0000 7f57667fc640 1 --2- 172.21.15.102:0/1717592944 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f575c06e3d0 0x7f575c070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.788+0000 7f57667fc640 1 -- 172.21.15.102:0/1717592944 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f576008de60 con 0x7f57701056a0 2024-03-26T19:06:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.788+0000 7f57755a7640 1 --2- 172.21.15.102:0/1717592944 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f575c06e3d0 0x7f575c070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:28.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.789+0000 7f57755a7640 1 --2- 172.21.15.102:0/1717592944 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f575c06e3d0 0x7f575c070880 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f576c02efd0 tx=0x7f576c02cf90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:28.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:28.796+0000 7f57667fc640 1 -- 172.21.15.102:0/1717592944 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f57600570f0 con 0x7f57701056a0 2024-03-26T19:06:29.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.016+0000 7f577780b640 1 -- 172.21.15.102:0/1717592944 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7f5770196660 con 0x7f57701056a0 2024-03-26T19:06:29.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.016+0000 7f57667fc640 1 -- 172.21.15.102:0/1717592944 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v43) v1 ==== 107+0+4155 (secure 0 0 0) 0x7f5760017090 con 0x7f57701056a0 2024-03-26T19:06:29.013 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:29.013 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44315,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/925070513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":925070513},{"type":"v1","addr":"172.21.15.190:6825","nonce":925070513}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:14.064518+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24293},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_24293":{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":0,"incarnation":29,"state":"up:reconnect","state_seq":147,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-26T19:06:29.013 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 30 2024-03-26T19:06:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.018+0000 7f577780b640 1 -- 172.21.15.102:0/1717592944 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f575c06e3d0 msgr2=0x7f575c070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.018+0000 7f577780b640 1 --2- 172.21.15.102:0/1717592944 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f575c06e3d0 0x7f575c070880 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f576c02efd0 tx=0x7f576c02cf90 comp rx=0 tx=0).stop 2024-03-26T19:06:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.018+0000 7f577780b640 1 -- 172.21.15.102:0/1717592944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57701056a0 msgr2=0x7f5770195850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.018+0000 7f577780b640 1 --2- 172.21.15.102:0/1717592944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57701056a0 0x7f5770195850 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f576000d6f0 tx=0x7f576000dbb0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.018+0000 7f577780b640 1 -- 172.21.15.102:0/1717592944 shutdown_connections 2024-03-26T19:06:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.018+0000 7f577780b640 1 --2- 172.21.15.102:0/1717592944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57701056a0 0x7f5770195850 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.018+0000 7f577780b640 1 --2- 172.21.15.102:0/1717592944 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f575c06e3d0 0x7f575c070880 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.018+0000 7f577780b640 1 --2- 172.21.15.102:0/1717592944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f57700ff690 0x7f5770195310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.018+0000 7f577780b640 1 -- 172.21.15.102:0/1717592944 >> 172.21.15.102:0/1717592944 conn(0x7f57700fb1d0 msgr2=0x7f57700fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.019+0000 7f577780b640 1 -- 172.21.15.102:0/1717592944 shutdown_connections 2024-03-26T19:06:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.019+0000 7f577780b640 1 -- 172.21.15.102:0/1717592944 wait complete. 2024-03-26T19:06:29.026 DEBUG:tasks.fs:max_mds reduced in epoch 30 2024-03-26T19:06:29.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-03-26T19:06:29.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.135+0000 7f69a2e71640 1 -- 172.21.15.102:0/3892229288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f699c104590 msgr2=0x7f699c104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.135+0000 7f69a2e71640 1 --2- 172.21.15.102:0/3892229288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f699c104590 0x7f699c104960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6990009880 tx=0x7f699002bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.135+0000 7f69a2e71640 1 -- 172.21.15.102:0/3892229288 shutdown_connections 2024-03-26T19:06:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.135+0000 7f69a2e71640 1 --2- 172.21.15.102:0/3892229288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c0fe5b0 0x7f699c0fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.135+0000 7f69a2e71640 1 --2- 172.21.15.102:0/3892229288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f699c104590 0x7f699c104960 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.135+0000 7f69a2e71640 1 -- 172.21.15.102:0/3892229288 >> 172.21.15.102:0/3892229288 conn(0x7f699c0743d0 msgr2=0x7f699c0747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.135+0000 7f69a2e71640 1 -- 172.21.15.102:0/3892229288 shutdown_connections 2024-03-26T19:06:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.135+0000 7f69a2e71640 1 -- 172.21.15.102:0/3892229288 wait complete. 2024-03-26T19:06:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.135+0000 7f69a2e71640 1 Processor -- start 2024-03-26T19:06:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a2e71640 1 -- start start 2024-03-26T19:06:29.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a2e71640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f699c0fe5b0 0x7f699c1941c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:29.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a2e71640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c104590 0x7f699c194700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:29.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a2e71640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c194d90 con 0x7f699c0fe5b0 2024-03-26T19:06:29.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a2e71640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c198b60 con 0x7f699c104590 2024-03-26T19:06:29.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a0c0d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f699c0fe5b0 0x7f699c1941c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a0c0d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f699c0fe5b0 0x7f699c1941c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56924/0 (socket says 172.21.15.102:56924) 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a0c0d640 1 -- 172.21.15.102:0/3897300569 learned_addr learned my addr 172.21.15.102:0/3897300569 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f699bfff640 1 --2- 172.21.15.102:0/3897300569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c104590 0x7f699c194700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a0c0d640 1 -- 172.21.15.102:0/3897300569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c104590 msgr2=0x7f699c194700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a0c0d640 1 --2- 172.21.15.102:0/3897300569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c104590 0x7f699c194700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a0c0d640 1 -- 172.21.15.102:0/3897300569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6990009560 con 0x7f699c0fe5b0 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.136+0000 7f69a0c0d640 1 --2- 172.21.15.102:0/3897300569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f699c0fe5b0 0x7f699c1941c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6990009230 tx=0x7f6990004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.137+0000 7f6999ffb640 1 -- 172.21.15.102:0/3897300569 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69900035f0 con 0x7f699c0fe5b0 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.137+0000 7f69a2e71640 1 -- 172.21.15.102:0/3897300569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699c198de0 con 0x7f699c0fe5b0 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.137+0000 7f6999ffb640 1 -- 172.21.15.102:0/3897300569 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6990003c10 con 0x7f699c0fe5b0 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.137+0000 7f6999ffb640 1 -- 172.21.15.102:0/3897300569 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6990034440 con 0x7f699c0fe5b0 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.137+0000 7f69a2e71640 1 -- 172.21.15.102:0/3897300569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c199360 con 0x7f699c0fe5b0 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.138+0000 7f69a2e71640 1 -- 172.21.15.102:0/3897300569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699c04e000 con 0x7f699c0fe5b0 2024-03-26T19:06:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.138+0000 7f6999ffb640 1 -- 172.21.15.102:0/3897300569 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6990004fd0 con 0x7f699c0fe5b0 2024-03-26T19:06:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.139+0000 7f6999ffb640 1 --2- 172.21.15.102:0/3897300569 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f698c06e3d0 0x7f698c070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.139+0000 7f699bfff640 1 --2- 172.21.15.102:0/3897300569 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f698c06e3d0 0x7f698c070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.139+0000 7f6999ffb640 1 -- 172.21.15.102:0/3897300569 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f69900b0910 con 0x7f699c0fe5b0 2024-03-26T19:06:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.139+0000 7f699bfff640 1 --2- 172.21.15.102:0/3897300569 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f698c06e3d0 0x7f698c070880 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6988005fd0 tx=0x7f6988005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:29.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.145+0000 7f6999ffb640 1 -- 172.21.15.102:0/3897300569 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6990079a70 con 0x7f699c0fe5b0 2024-03-26T19:06:29.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.362+0000 7f69a2e71640 1 -- 172.21.15.102:0/3897300569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7f699c195510 con 0x7f699c0fe5b0 2024-03-26T19:06:29.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.363+0000 7f6999ffb640 1 -- 172.21.15.102:0/3897300569 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v43) v1 ==== 107+0+4152 (secure 0 0 0) 0x7f699007d130 con 0x7f699c0fe5b0 2024-03-26T19:06:29.360 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:29.360 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44315,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/925070513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":925070513},{"type":"v1","addr":"172.21.15.190:6825","nonce":925070513}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:14.327795+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24293},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_24293":{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":0,"incarnation":29,"state":"up:rejoin","state_seq":148,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-26T19:06:29.361 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 31 2024-03-26T19:06:29.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.365+0000 7f69a2e71640 1 -- 172.21.15.102:0/3897300569 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f698c06e3d0 msgr2=0x7f698c070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.365+0000 7f69a2e71640 1 --2- 172.21.15.102:0/3897300569 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f698c06e3d0 0x7f698c070880 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6988005fd0 tx=0x7f6988005e20 comp rx=0 tx=0).stop 2024-03-26T19:06:29.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.366+0000 7f69a2e71640 1 -- 172.21.15.102:0/3897300569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f699c0fe5b0 msgr2=0x7f699c1941c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.366+0000 7f69a2e71640 1 --2- 172.21.15.102:0/3897300569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f699c0fe5b0 0x7f699c1941c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6990009230 tx=0x7f6990004040 comp rx=0 tx=0).stop 2024-03-26T19:06:29.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.366+0000 7f69a2e71640 1 -- 172.21.15.102:0/3897300569 shutdown_connections 2024-03-26T19:06:29.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.366+0000 7f69a2e71640 1 --2- 172.21.15.102:0/3897300569 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f698c06e3d0 0x7f698c070880 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.366+0000 7f69a2e71640 1 --2- 172.21.15.102:0/3897300569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c104590 0x7f699c194700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.366+0000 7f69a2e71640 1 --2- 172.21.15.102:0/3897300569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f699c0fe5b0 0x7f699c1941c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.366+0000 7f69a2e71640 1 -- 172.21.15.102:0/3897300569 >> 172.21.15.102:0/3897300569 conn(0x7f699c0743d0 msgr2=0x7f699c0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:29.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.366+0000 7f69a2e71640 1 -- 172.21.15.102:0/3897300569 shutdown_connections 2024-03-26T19:06:29.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.366+0000 7f69a2e71640 1 -- 172.21.15.102:0/3897300569 wait complete. 2024-03-26T19:06:29.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:29 smithi102 ceph-mon[78322]: pgmap v332: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:29.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:29 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/197768029' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-26T19:06:29.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:29 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1024523672' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-03-26T19:06:29.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:29 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1717592944' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-03-26T19:06:29.378 DEBUG:tasks.fs:max_mds reduced in epoch 31 2024-03-26T19:06:29.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-03-26T19:06:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.486+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/3106202853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a04104590 msgr2=0x7f0a04104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.486+0000 7f0a0c0d5640 1 --2- 172.21.15.102:0/3106202853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a04104590 0x7f0a04104960 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009880 tx=0x7f09f402bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.486+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/3106202853 shutdown_connections 2024-03-26T19:06:29.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.486+0000 7f0a0c0d5640 1 --2- 172.21.15.102:0/3106202853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a040fe5b0 0x7f0a040fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.486+0000 7f0a0c0d5640 1 --2- 172.21.15.102:0/3106202853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a04104590 0x7f0a04104960 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.486+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/3106202853 >> 172.21.15.102:0/3106202853 conn(0x7f0a040743d0 msgr2=0x7f0a040747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:29.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.486+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/3106202853 shutdown_connections 2024-03-26T19:06:29.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.486+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/3106202853 wait complete. 2024-03-26T19:06:29.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a0c0d5640 1 Processor -- start 2024-03-26T19:06:29.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a0c0d5640 1 -- start start 2024-03-26T19:06:29.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a0c0d5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a040fe5b0 0x7f0a041941a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:29.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a0c0d5640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a04104590 0x7f0a041946e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:29.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a0c0d5640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a04194d70 con 0x7f0a04104590 2024-03-26T19:06:29.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a0c0d5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a04198b40 con 0x7f0a040fe5b0 2024-03-26T19:06:29.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a09670640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a04104590 0x7f0a041946e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:29.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a09670640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a04104590 0x7f0a041946e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56956/0 (socket says 172.21.15.102:56956) 2024-03-26T19:06:29.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a09670640 1 -- 172.21.15.102:0/499371327 learned_addr learned my addr 172.21.15.102:0/499371327 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:29.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a09670640 1 -- 172.21.15.102:0/499371327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a040fe5b0 msgr2=0x7f0a041941a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a09670640 1 --2- 172.21.15.102:0/499371327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a040fe5b0 0x7f0a041941a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.487+0000 7f0a09670640 1 -- 172.21.15.102:0/499371327 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f4009560 con 0x7f0a04104590 2024-03-26T19:06:29.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.488+0000 7f0a09670640 1 --2- 172.21.15.102:0/499371327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a04104590 0x7f0a041946e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009900 tx=0x7f0a00009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:29.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.488+0000 7f09faffd640 1 -- 172.21.15.102:0/499371327 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a00012070 con 0x7f0a04104590 2024-03-26T19:06:29.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.488+0000 7f09faffd640 1 -- 172.21.15.102:0/499371327 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a0000d200 con 0x7f0a04104590 2024-03-26T19:06:29.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.488+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/499371327 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a04198dc0 con 0x7f0a04104590 2024-03-26T19:06:29.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.488+0000 7f09faffd640 1 -- 172.21.15.102:0/499371327 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a00007aa0 con 0x7f0a04104590 2024-03-26T19:06:29.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.488+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/499371327 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a04199320 con 0x7f0a04104590 2024-03-26T19:06:29.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.489+0000 7f09faffd640 1 -- 172.21.15.102:0/499371327 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f0a00016030 con 0x7f0a04104590 2024-03-26T19:06:29.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.489+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/499371327 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0404e000 con 0x7f0a04104590 2024-03-26T19:06:29.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.490+0000 7f09faffd640 1 --2- 172.21.15.102:0/499371327 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f09f006e3d0 0x7f09f0070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:29.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.490+0000 7f09faffd640 1 -- 172.21.15.102:0/499371327 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f0a0008f6e0 con 0x7f0a04104590 2024-03-26T19:06:29.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.490+0000 7f0a09e71640 1 --2- 172.21.15.102:0/499371327 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f09f006e3d0 0x7f09f0070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:29.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.490+0000 7f0a09e71640 1 --2- 172.21.15.102:0/499371327 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f09f006e3d0 0x7f09f0070880 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f09f4036040 tx=0x7f09f4000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:29.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.495+0000 7f09faffd640 1 -- 172.21.15.102:0/499371327 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a000588c0 con 0x7f0a04104590 2024-03-26T19:06:29.521 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:29 smithi190 ceph-mon[65689]: pgmap v332: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:29.521 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:29 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/197768029' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-26T19:06:29.522 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:29 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1024523672' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-03-26T19:06:29.522 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:29 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1717592944' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-03-26T19:06:29.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.722+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/499371327 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f0a0410b970 con 0x7f0a04104590 2024-03-26T19:06:29.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.722+0000 7f09faffd640 1 -- 172.21.15.102:0/499371327 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v43) v1 ==== 107+0+5011 (secure 0 0 0) 0x7f0a0005bf80 con 0x7f0a04104590 2024-03-26T19:06:29.720 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:29.720 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44315,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/925070513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":925070513},{"type":"v1","addr":"172.21.15.190:6825","nonce":925070513}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:16.330672+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24293},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_24293":{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":0,"incarnation":29,"state":"up:active","state_seq":149,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24293,"qdb_cluster":[24293]},"id":1}]} 2024-03-26T19:06:29.720 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 32 2024-03-26T19:06:29.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.724+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/499371327 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f09f006e3d0 msgr2=0x7f09f0070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.724+0000 7f0a0c0d5640 1 --2- 172.21.15.102:0/499371327 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f09f006e3d0 0x7f09f0070880 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f09f4036040 tx=0x7f09f4000bc0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.724+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/499371327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a04104590 msgr2=0x7f0a041946e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.724+0000 7f0a0c0d5640 1 --2- 172.21.15.102:0/499371327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a04104590 0x7f0a041946e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009900 tx=0x7f0a00009dc0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.724+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/499371327 shutdown_connections 2024-03-26T19:06:29.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.724+0000 7f0a0c0d5640 1 --2- 172.21.15.102:0/499371327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a04104590 0x7f0a041946e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.724+0000 7f0a0c0d5640 1 --2- 172.21.15.102:0/499371327 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f09f006e3d0 0x7f09f0070880 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.724+0000 7f0a0c0d5640 1 --2- 172.21.15.102:0/499371327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a040fe5b0 0x7f0a041941a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.724+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/499371327 >> 172.21.15.102:0/499371327 conn(0x7f0a040743d0 msgr2=0x7f0a040fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:29.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.724+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/499371327 shutdown_connections 2024-03-26T19:06:29.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.725+0000 7f0a0c0d5640 1 -- 172.21.15.102:0/499371327 wait complete. 2024-03-26T19:06:29.736 DEBUG:tasks.fs:max_mds reduced in epoch 32 2024-03-26T19:06:29.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-03-26T19:06:29.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.867+0000 7f58cb24a640 1 -- 172.21.15.102:0/3153513651 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c40fdf70 msgr2=0x7f58c40fe340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.867+0000 7f58cb24a640 1 --2- 172.21.15.102:0/3153513651 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c40fdf70 0x7f58c40fe340 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f58b8009880 tx=0x7f58b802bd10 comp rx=0 tx=0).stop 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.867+0000 7f58cb24a640 1 -- 172.21.15.102:0/3153513651 shutdown_connections 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.867+0000 7f58cb24a640 1 --2- 172.21.15.102:0/3153513651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58c40fe910 0x7f58c41028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.867+0000 7f58cb24a640 1 --2- 172.21.15.102:0/3153513651 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c40fdf70 0x7f58c40fe340 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.867+0000 7f58cb24a640 1 -- 172.21.15.102:0/3153513651 >> 172.21.15.102:0/3153513651 conn(0x7f58c4074970 msgr2=0x7f58c4074d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.867+0000 7f58cb24a640 1 -- 172.21.15.102:0/3153513651 shutdown_connections 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.867+0000 7f58cb24a640 1 -- 172.21.15.102:0/3153513651 wait complete. 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58cb24a640 1 Processor -- start 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58cb24a640 1 -- start start 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58cb24a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58c40fdf70 0x7f58c41976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58cb24a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c40fe910 0x7f58c4197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58cb24a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c41982e0 con 0x7f58c40fe910 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58cb24a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c419afb0 con 0x7f58c40fdf70 2024-03-26T19:06:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58c3fff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c40fe910 0x7f58c4197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:29.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58c3fff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c40fe910 0x7f58c4197c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56978/0 (socket says 172.21.15.102:56978) 2024-03-26T19:06:29.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58c3fff640 1 -- 172.21.15.102:0/2096396601 learned_addr learned my addr 172.21.15.102:0/2096396601 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:29.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58c3fff640 1 -- 172.21.15.102:0/2096396601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58c40fdf70 msgr2=0x7f58c41976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:29.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58c3fff640 1 --2- 172.21.15.102:0/2096396601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58c40fdf70 0x7f58c41976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:29.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.868+0000 7f58c3fff640 1 -- 172.21.15.102:0/2096396601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b8009560 con 0x7f58c40fe910 2024-03-26T19:06:29.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.869+0000 7f58c3fff640 1 --2- 172.21.15.102:0/2096396601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c40fe910 0x7f58c4197c00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f58b0006fd0 tx=0x7f58b000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:29.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.869+0000 7f58c1ffb640 1 -- 172.21.15.102:0/2096396601 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58b0011800 con 0x7f58c40fe910 2024-03-26T19:06:29.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.869+0000 7f58c1ffb640 1 -- 172.21.15.102:0/2096396601 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58b0011e20 con 0x7f58c40fe910 2024-03-26T19:06:29.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.869+0000 7f58cb24a640 1 -- 172.21.15.102:0/2096396601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c419b290 con 0x7f58c40fe910 2024-03-26T19:06:29.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.869+0000 7f58c1ffb640 1 -- 172.21.15.102:0/2096396601 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58b000f2e0 con 0x7f58c40fe910 2024-03-26T19:06:29.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.869+0000 7f58cb24a640 1 -- 172.21.15.102:0/2096396601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c419b7f0 con 0x7f58c40fe910 2024-03-26T19:06:29.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.870+0000 7f58cb24a640 1 -- 172.21.15.102:0/2096396601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c404e000 con 0x7f58c40fe910 2024-03-26T19:06:29.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.870+0000 7f58c1ffb640 1 -- 172.21.15.102:0/2096396601 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f58b0010380 con 0x7f58c40fe910 2024-03-26T19:06:29.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.871+0000 7f58c1ffb640 1 --2- 172.21.15.102:0/2096396601 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58b406e3d0 0x7f58b4070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:29.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.871+0000 7f58c1ffb640 1 -- 172.21.15.102:0/2096396601 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f58b008e160 con 0x7f58c40fe910 2024-03-26T19:06:29.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.871+0000 7f58c8fe6640 1 --2- 172.21.15.102:0/2096396601 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58b406e3d0 0x7f58b4070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:29.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.871+0000 7f58c8fe6640 1 --2- 172.21.15.102:0/2096396601 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58b406e3d0 0x7f58b4070880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f58b8036040 tx=0x7f58b8000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:29.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:29.875+0000 7f58c1ffb640 1 -- 172.21.15.102:0/2096396601 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f58b00572c0 con 0x7f58c40fe910 2024-03-26T19:06:30.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.094+0000 7f58cb24a640 1 -- 172.21.15.102:0/2096396601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f58c41061e0 con 0x7f58c40fe910 2024-03-26T19:06:30.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.094+0000 7f58c1ffb640 1 -- 172.21.15.102:0/2096396601 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v43) v1 ==== 107+0+4232 (secure 0 0 0) 0x7f58b005a980 con 0x7f58c40fe910 2024-03-26T19:06:30.092 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:30.092 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:16.330672+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24293},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_24293":{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":0,"incarnation":29,"state":"up:active","state_seq":149,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24293,"qdb_cluster":[24293]},"id":1}]} 2024-03-26T19:06:30.092 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 33 2024-03-26T19:06:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.097+0000 7f58cb24a640 1 -- 172.21.15.102:0/2096396601 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58b406e3d0 msgr2=0x7f58b4070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.097+0000 7f58cb24a640 1 --2- 172.21.15.102:0/2096396601 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58b406e3d0 0x7f58b4070880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f58b8036040 tx=0x7f58b8000bc0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.097+0000 7f58cb24a640 1 -- 172.21.15.102:0/2096396601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c40fe910 msgr2=0x7f58c4197c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.097+0000 7f58cb24a640 1 --2- 172.21.15.102:0/2096396601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c40fe910 0x7f58c4197c00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f58b0006fd0 tx=0x7f58b000b860 comp rx=0 tx=0).stop 2024-03-26T19:06:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.097+0000 7f58cb24a640 1 -- 172.21.15.102:0/2096396601 shutdown_connections 2024-03-26T19:06:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.097+0000 7f58cb24a640 1 --2- 172.21.15.102:0/2096396601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c40fe910 0x7f58c4197c00 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.097+0000 7f58cb24a640 1 --2- 172.21.15.102:0/2096396601 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f58b406e3d0 0x7f58b4070880 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.097+0000 7f58cb24a640 1 --2- 172.21.15.102:0/2096396601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58c40fdf70 0x7f58c41976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.097+0000 7f58cb24a640 1 -- 172.21.15.102:0/2096396601 >> 172.21.15.102:0/2096396601 conn(0x7f58c4074970 msgr2=0x7f58c40fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:30.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.097+0000 7f58cb24a640 1 -- 172.21.15.102:0/2096396601 shutdown_connections 2024-03-26T19:06:30.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.097+0000 7f58cb24a640 1 -- 172.21.15.102:0/2096396601 wait complete. 2024-03-26T19:06:30.106 DEBUG:tasks.fs:max_mds reduced in epoch 33 2024-03-26T19:06:30.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-03-26T19:06:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.235+0000 7f8982eb9640 1 -- 172.21.15.102:0/3574014496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f897c104590 msgr2=0x7f897c104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.235+0000 7f8982eb9640 1 --2- 172.21.15.102:0/3574014496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f897c104590 0x7f897c104960 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8970009880 tx=0x7f897002bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.235+0000 7f8982eb9640 1 -- 172.21.15.102:0/3574014496 shutdown_connections 2024-03-26T19:06:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.235+0000 7f8982eb9640 1 --2- 172.21.15.102:0/3574014496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c0fe5b0 0x7f897c0fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.235+0000 7f8982eb9640 1 --2- 172.21.15.102:0/3574014496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f897c104590 0x7f897c104960 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.235+0000 7f8982eb9640 1 -- 172.21.15.102:0/3574014496 >> 172.21.15.102:0/3574014496 conn(0x7f897c0743d0 msgr2=0x7f897c0747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.235+0000 7f8982eb9640 1 -- 172.21.15.102:0/3574014496 shutdown_connections 2024-03-26T19:06:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.235+0000 7f8982eb9640 1 -- 172.21.15.102:0/3574014496 wait complete. 2024-03-26T19:06:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f8982eb9640 1 Processor -- start 2024-03-26T19:06:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f8982eb9640 1 -- start start 2024-03-26T19:06:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f8982eb9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c0fe5b0 0x7f897c196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f8982eb9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f897c104590 0x7f897c196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f8982eb9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897c196c90 con 0x7f897c104590 2024-03-26T19:06:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f8982eb9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897c196dd0 con 0x7f897c0fe5b0 2024-03-26T19:06:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f897bfff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f897c104590 0x7f897c196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f897bfff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f897c104590 0x7f897c196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56992/0 (socket says 172.21.15.102:56992) 2024-03-26T19:06:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f897bfff640 1 -- 172.21.15.102:0/2676124034 learned_addr learned my addr 172.21.15.102:0/2676124034 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f8980c55640 1 --2- 172.21.15.102:0/2676124034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c0fe5b0 0x7f897c196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f897bfff640 1 -- 172.21.15.102:0/2676124034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c0fe5b0 msgr2=0x7f897c196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f897bfff640 1 --2- 172.21.15.102:0/2676124034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c0fe5b0 0x7f897c196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f897bfff640 1 -- 172.21.15.102:0/2676124034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8970009560 con 0x7f897c104590 2024-03-26T19:06:30.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.236+0000 7f897bfff640 1 --2- 172.21.15.102:0/2676124034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f897c104590 0x7f897c196640 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f896800c8a0 tx=0x7f896800cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:30.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.237+0000 7f8979ffb640 1 -- 172.21.15.102:0/2676124034 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8968007b80 con 0x7f897c104590 2024-03-26T19:06:30.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.237+0000 7f8979ffb640 1 -- 172.21.15.102:0/2676124034 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8968007ce0 con 0x7f897c104590 2024-03-26T19:06:30.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.237+0000 7f8982eb9640 1 -- 172.21.15.102:0/2676124034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f897c19f170 con 0x7f897c104590 2024-03-26T19:06:30.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.237+0000 7f8979ffb640 1 -- 172.21.15.102:0/2676124034 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8968018810 con 0x7f897c104590 2024-03-26T19:06:30.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.237+0000 7f8982eb9640 1 -- 172.21.15.102:0/2676124034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f897c19f620 con 0x7f897c104590 2024-03-26T19:06:30.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.238+0000 7f8982eb9640 1 -- 172.21.15.102:0/2676124034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f897c04e000 con 0x7f897c104590 2024-03-26T19:06:30.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.238+0000 7f8979ffb640 1 -- 172.21.15.102:0/2676124034 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8968018a80 con 0x7f897c104590 2024-03-26T19:06:30.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.239+0000 7f8979ffb640 1 --2- 172.21.15.102:0/2676124034 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f896c06e3d0 0x7f896c070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:30.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.239+0000 7f8979ffb640 1 -- 172.21.15.102:0/2676124034 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f896808e5e0 con 0x7f897c104590 2024-03-26T19:06:30.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.239+0000 7f8980c55640 1 --2- 172.21.15.102:0/2676124034 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f896c06e3d0 0x7f896c070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:30.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.240+0000 7f8980c55640 1 --2- 172.21.15.102:0/2676124034 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f896c06e3d0 0x7f896c070880 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8970006010 tx=0x7f8970005b50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:30.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.244+0000 7f8979ffb640 1 -- 172.21.15.102:0/2676124034 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8968057740 con 0x7f897c104590 2024-03-26T19:06:30.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:30 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3897300569' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-03-26T19:06:30.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:30 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/499371327' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-03-26T19:06:30.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:30 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2096396601' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-03-26T19:06:30.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.523+0000 7f8982eb9640 1 -- 172.21.15.102:0/2676124034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7f897c067900 con 0x7f897c104590 2024-03-26T19:06:30.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.523+0000 7f8979ffb640 1 -- 172.21.15.102:0/2676124034 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v43) v1 ==== 107+0+5079 (secure 0 0 0) 0x7f896805ae00 con 0x7f897c104590 2024-03-26T19:06:30.521 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:30.521 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44343,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/782636706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":782636706},{"type":"v1","addr":"172.21.15.190:6825","nonce":782636706}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:16.330672+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24293},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_24293":{"gid":24293,"name":"cephfs.smithi190.hwsvdg","rank":0,"incarnation":29,"state":"up:active","state_seq":149,"addr":"172.21.15.190:6827/624694186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":624694186},{"type":"v1","addr":"172.21.15.190:6827","nonce":624694186}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24293,"qdb_cluster":[24293]},"id":1}]} 2024-03-26T19:06:30.521 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 34 2024-03-26T19:06:30.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.526+0000 7f8982eb9640 1 -- 172.21.15.102:0/2676124034 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f896c06e3d0 msgr2=0x7f896c070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:30.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.526+0000 7f8982eb9640 1 --2- 172.21.15.102:0/2676124034 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f896c06e3d0 0x7f896c070880 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8970006010 tx=0x7f8970005b50 comp rx=0 tx=0).stop 2024-03-26T19:06:30.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.526+0000 7f8982eb9640 1 -- 172.21.15.102:0/2676124034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f897c104590 msgr2=0x7f897c196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:30.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.526+0000 7f8982eb9640 1 --2- 172.21.15.102:0/2676124034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f897c104590 0x7f897c196640 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f896800c8a0 tx=0x7f896800cd60 comp rx=0 tx=0).stop 2024-03-26T19:06:30.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.526+0000 7f8982eb9640 1 -- 172.21.15.102:0/2676124034 shutdown_connections 2024-03-26T19:06:30.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.526+0000 7f8982eb9640 1 --2- 172.21.15.102:0/2676124034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f897c104590 0x7f897c196640 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.526+0000 7f8982eb9640 1 --2- 172.21.15.102:0/2676124034 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f896c06e3d0 0x7f896c070880 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.526+0000 7f8982eb9640 1 --2- 172.21.15.102:0/2676124034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c0fe5b0 0x7f897c196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.526+0000 7f8982eb9640 1 -- 172.21.15.102:0/2676124034 >> 172.21.15.102:0/2676124034 conn(0x7f897c0743d0 msgr2=0x7f897c0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:30.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.526+0000 7f8982eb9640 1 -- 172.21.15.102:0/2676124034 shutdown_connections 2024-03-26T19:06:30.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.526+0000 7f8982eb9640 1 -- 172.21.15.102:0/2676124034 wait complete. 2024-03-26T19:06:30.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:30 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3897300569' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-03-26T19:06:30.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:30 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/499371327' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-03-26T19:06:30.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:30 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2096396601' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-03-26T19:06:30.535 DEBUG:tasks.fs:max_mds reduced in epoch 34 2024-03-26T19:06:30.535 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-03-26T19:06:30.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.645+0000 7f525cd3c640 1 -- 172.21.15.102:0/1857186269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5258104590 msgr2=0x7f5258104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:30.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.645+0000 7f525cd3c640 1 --2- 172.21.15.102:0/1857186269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5258104590 0x7f5258104960 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5248009880 tx=0x7f524802bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.645+0000 7f525cd3c640 1 -- 172.21.15.102:0/1857186269 shutdown_connections 2024-03-26T19:06:30.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.645+0000 7f525cd3c640 1 --2- 172.21.15.102:0/1857186269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52580fe5b0 0x7f52580fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.645+0000 7f525cd3c640 1 --2- 172.21.15.102:0/1857186269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5258104590 0x7f5258104960 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.645+0000 7f525cd3c640 1 -- 172.21.15.102:0/1857186269 >> 172.21.15.102:0/1857186269 conn(0x7f52580743d0 msgr2=0x7f52580747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:30.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.646+0000 7f525cd3c640 1 -- 172.21.15.102:0/1857186269 shutdown_connections 2024-03-26T19:06:30.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.646+0000 7f525cd3c640 1 -- 172.21.15.102:0/1857186269 wait complete. 2024-03-26T19:06:30.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.646+0000 7f525cd3c640 1 Processor -- start 2024-03-26T19:06:30.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f525cd3c640 1 -- start start 2024-03-26T19:06:30.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f525cd3c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580fe5b0 0x7f52581941a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:30.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f525cd3c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5258104590 0x7f52581946e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:30.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f525cd3c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5258194d70 con 0x7f52580fe5b0 2024-03-26T19:06:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f525cd3c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5258198b40 con 0x7f5258104590 2024-03-26T19:06:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f525659c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580fe5b0 0x7f52581941a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f525659c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580fe5b0 0x7f52581941a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57018/0 (socket says 172.21.15.102:57018) 2024-03-26T19:06:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f525659c640 1 -- 172.21.15.102:0/1305419726 learned_addr learned my addr 172.21.15.102:0/1305419726 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f524dd9b640 1 --2- 172.21.15.102:0/1305419726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5258104590 0x7f52581946e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f525659c640 1 -- 172.21.15.102:0/1305419726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5258104590 msgr2=0x7f52581946e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f525659c640 1 --2- 172.21.15.102:0/1305419726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5258104590 0x7f52581946e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.647+0000 7f525659c640 1 -- 172.21.15.102:0/1305419726 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5248009560 con 0x7f52580fe5b0 2024-03-26T19:06:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.648+0000 7f525659c640 1 --2- 172.21.15.102:0/1305419726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580fe5b0 0x7f52581941a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5248009230 tx=0x7f5248004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.648+0000 7f524ffff640 1 -- 172.21.15.102:0/1305419726 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52480035f0 con 0x7f52580fe5b0 2024-03-26T19:06:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.648+0000 7f525cd3c640 1 -- 172.21.15.102:0/1305419726 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5258198dc0 con 0x7f52580fe5b0 2024-03-26T19:06:30.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.648+0000 7f524ffff640 1 -- 172.21.15.102:0/1305419726 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5248003c10 con 0x7f52580fe5b0 2024-03-26T19:06:30.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.648+0000 7f524ffff640 1 -- 172.21.15.102:0/1305419726 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5248005670 con 0x7f52580fe5b0 2024-03-26T19:06:30.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.648+0000 7f525cd3c640 1 -- 172.21.15.102:0/1305419726 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52581992c0 con 0x7f52580fe5b0 2024-03-26T19:06:30.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.649+0000 7f525cd3c640 1 -- 172.21.15.102:0/1305419726 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525804e000 con 0x7f52580fe5b0 2024-03-26T19:06:30.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.650+0000 7f524ffff640 1 -- 172.21.15.102:0/1305419726 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5248004dd0 con 0x7f52580fe5b0 2024-03-26T19:06:30.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.650+0000 7f524ffff640 1 --2- 172.21.15.102:0/1305419726 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e380 0x7f5244070830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:30.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.650+0000 7f524ffff640 1 -- 172.21.15.102:0/1305419726 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f52480b07a0 con 0x7f52580fe5b0 2024-03-26T19:06:30.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.650+0000 7f524dd9b640 1 --2- 172.21.15.102:0/1305419726 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e380 0x7f5244070830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:30.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.651+0000 7f524dd9b640 1 --2- 172.21.15.102:0/1305419726 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e380 0x7f5244070830 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f52581957b0 tx=0x7f524000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:30.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.659+0000 7f524ffff640 1 -- 172.21.15.102:0/1305419726 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f52480799b0 con 0x7f52580fe5b0 2024-03-26T19:06:30.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.931+0000 7f525cd3c640 1 -- 172.21.15.102:0/1305419726 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f525810b7a0 con 0x7f52580fe5b0 2024-03-26T19:06:30.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.932+0000 7f524ffff640 1 -- 172.21.15.102:0/1305419726 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v43) v1 ==== 107+0+4278 (secure 0 0 0) 0x7f524807d070 con 0x7f52580fe5b0 2024-03-26T19:06:30.929 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:30.929 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44343,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/782636706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":782636706},{"type":"v1","addr":"172.21.15.190:6825","nonce":782636706}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:46.321937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[1],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-26T19:06:30.930 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 35 2024-03-26T19:06:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.935+0000 7f525cd3c640 1 -- 172.21.15.102:0/1305419726 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e380 msgr2=0x7f5244070830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.935+0000 7f525cd3c640 1 --2- 172.21.15.102:0/1305419726 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e380 0x7f5244070830 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f52581957b0 tx=0x7f524000b410 comp rx=0 tx=0).stop 2024-03-26T19:06:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.935+0000 7f525cd3c640 1 -- 172.21.15.102:0/1305419726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580fe5b0 msgr2=0x7f52581941a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.935+0000 7f525cd3c640 1 --2- 172.21.15.102:0/1305419726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580fe5b0 0x7f52581941a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5248009230 tx=0x7f5248004040 comp rx=0 tx=0).stop 2024-03-26T19:06:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.935+0000 7f525cd3c640 1 -- 172.21.15.102:0/1305419726 shutdown_connections 2024-03-26T19:06:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.935+0000 7f525cd3c640 1 --2- 172.21.15.102:0/1305419726 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f524406e380 0x7f5244070830 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.935+0000 7f525cd3c640 1 --2- 172.21.15.102:0/1305419726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5258104590 0x7f52581946e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.935+0000 7f525cd3c640 1 --2- 172.21.15.102:0/1305419726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52580fe5b0 0x7f52581941a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.935+0000 7f525cd3c640 1 -- 172.21.15.102:0/1305419726 >> 172.21.15.102:0/1305419726 conn(0x7f52580743d0 msgr2=0x7f52580fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.935+0000 7f525cd3c640 1 -- 172.21.15.102:0/1305419726 shutdown_connections 2024-03-26T19:06:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:30.935+0000 7f525cd3c640 1 -- 172.21.15.102:0/1305419726 wait complete. 2024-03-26T19:06:30.943 DEBUG:tasks.fs:max_mds reduced in epoch 35 2024-03-26T19:06:30.943 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-03-26T19:06:31.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.052+0000 7fb9375eb640 1 -- 172.21.15.102:0/1849854279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb930104590 msgr2=0x7fb930104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:31.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.052+0000 7fb9375eb640 1 --2- 172.21.15.102:0/1849854279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb930104590 0x7fb930104960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb92c009900 tx=0x7fb92c02df20 comp rx=0 tx=0).stop 2024-03-26T19:06:31.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.052+0000 7fb9375eb640 1 -- 172.21.15.102:0/1849854279 shutdown_connections 2024-03-26T19:06:31.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.052+0000 7fb9375eb640 1 --2- 172.21.15.102:0/1849854279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9300fe5b0 0x7fb9300fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.052+0000 7fb9375eb640 1 --2- 172.21.15.102:0/1849854279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb930104590 0x7fb930104960 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.052+0000 7fb9375eb640 1 -- 172.21.15.102:0/1849854279 >> 172.21.15.102:0/1849854279 conn(0x7fb9300743d0 msgr2=0x7fb9300747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:31.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.053+0000 7fb9375eb640 1 -- 172.21.15.102:0/1849854279 shutdown_connections 2024-03-26T19:06:31.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.053+0000 7fb9375eb640 1 -- 172.21.15.102:0/1849854279 wait complete. 2024-03-26T19:06:31.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.053+0000 7fb9375eb640 1 Processor -- start 2024-03-26T19:06:31.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.053+0000 7fb9375eb640 1 -- start start 2024-03-26T19:06:31.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb9375eb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9300fe5b0 0x7fb9301931d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:31.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb9375eb640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb930104590 0x7fb930193710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:31.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb9375eb640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb930193da0 con 0x7fb930104590 2024-03-26T19:06:31.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb9375eb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb930197b70 con 0x7fb9300fe5b0 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb934b86640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb930104590 0x7fb930193710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb934b86640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb930104590 0x7fb930193710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57028/0 (socket says 172.21.15.102:57028) 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb934b86640 1 -- 172.21.15.102:0/4217709329 learned_addr learned my addr 172.21.15.102:0/4217709329 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb935387640 1 --2- 172.21.15.102:0/4217709329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9300fe5b0 0x7fb9301931d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb934b86640 1 -- 172.21.15.102:0/4217709329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9300fe5b0 msgr2=0x7fb9301931d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb934b86640 1 --2- 172.21.15.102:0/4217709329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9300fe5b0 0x7fb9301931d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb934b86640 1 -- 172.21.15.102:0/4217709329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c009560 con 0x7fb930104590 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb934b86640 1 --2- 172.21.15.102:0/4217709329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb930104590 0x7fb930193710 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb92000d6f0 tx=0x7fb92000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb9267fc640 1 -- 172.21.15.102:0/4217709329 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb920009a30 con 0x7fb930104590 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb9267fc640 1 -- 172.21.15.102:0/4217709329 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9200054e0 con 0x7fb930104590 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb9375eb640 1 -- 172.21.15.102:0/4217709329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb930197e50 con 0x7fb930104590 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb9267fc640 1 -- 172.21.15.102:0/4217709329 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb920010460 con 0x7fb930104590 2024-03-26T19:06:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.054+0000 7fb9375eb640 1 -- 172.21.15.102:0/4217709329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9301983b0 con 0x7fb930104590 2024-03-26T19:06:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.056+0000 7fb9375eb640 1 -- 172.21.15.102:0/4217709329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93004e000 con 0x7fb930104590 2024-03-26T19:06:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.056+0000 7fb9267fc640 1 -- 172.21.15.102:0/4217709329 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb920009b90 con 0x7fb930104590 2024-03-26T19:06:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.056+0000 7fb9267fc640 1 --2- 172.21.15.102:0/4217709329 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb91c06e3d0 0x7fb91c070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:31.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.056+0000 7fb935387640 1 --2- 172.21.15.102:0/4217709329 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb91c06e3d0 0x7fb91c070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:31.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.056+0000 7fb9267fc640 1 -- 172.21.15.102:0/4217709329 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fb92008df10 con 0x7fb930104590 2024-03-26T19:06:31.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.057+0000 7fb935387640 1 --2- 172.21.15.102:0/4217709329 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb91c06e3d0 0x7fb91c070880 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb92c038040 tx=0x7fb92c005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:31.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.061+0000 7fb9267fc640 1 -- 172.21.15.102:0/4217709329 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9200570f0 con 0x7fb930104590 2024-03-26T19:06:31.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.315+0000 7fb9375eb640 1 -- 172.21.15.102:0/4217709329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7fb9301945c0 con 0x7fb930104590 2024-03-26T19:06:31.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.316+0000 7fb9267fc640 1 -- 172.21.15.102:0/4217709329 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v43) v1 ==== 107+0+4357 (secure 0 0 0) 0x7fb920017090 con 0x7fb930104590 2024-03-26T19:06:31.313 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:31.313 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44343,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/782636706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":782636706},{"type":"v1","addr":"172.21.15.190:6825","nonce":782636706}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:46.327380+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34436},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_34436":{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-26T19:06:31.314 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 36 2024-03-26T19:06:31.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.319+0000 7fb9375eb640 1 -- 172.21.15.102:0/4217709329 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb91c06e3d0 msgr2=0x7fb91c070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:31.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.319+0000 7fb9375eb640 1 --2- 172.21.15.102:0/4217709329 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb91c06e3d0 0x7fb91c070880 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb92c038040 tx=0x7fb92c005fd0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.319+0000 7fb9375eb640 1 -- 172.21.15.102:0/4217709329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb930104590 msgr2=0x7fb930193710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:31.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.319+0000 7fb9375eb640 1 --2- 172.21.15.102:0/4217709329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb930104590 0x7fb930193710 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb92000d6f0 tx=0x7fb92000dbb0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.319+0000 7fb9375eb640 1 -- 172.21.15.102:0/4217709329 shutdown_connections 2024-03-26T19:06:31.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.319+0000 7fb9375eb640 1 --2- 172.21.15.102:0/4217709329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb930104590 0x7fb930193710 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.319+0000 7fb9375eb640 1 --2- 172.21.15.102:0/4217709329 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb91c06e3d0 0x7fb91c070880 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.319+0000 7fb9375eb640 1 --2- 172.21.15.102:0/4217709329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9300fe5b0 0x7fb9301931d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.319+0000 7fb9375eb640 1 -- 172.21.15.102:0/4217709329 >> 172.21.15.102:0/4217709329 conn(0x7fb9300743d0 msgr2=0x7fb930103880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.319+0000 7fb9375eb640 1 -- 172.21.15.102:0/4217709329 shutdown_connections 2024-03-26T19:06:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.319+0000 7fb9375eb640 1 -- 172.21.15.102:0/4217709329 wait complete. 2024-03-26T19:06:31.326 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:31 smithi102 ceph-mon[78322]: pgmap v333: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:31.326 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:31 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2676124034' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-03-26T19:06:31.326 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:31 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1305419726' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-03-26T19:06:31.327 DEBUG:tasks.fs:max_mds reduced in epoch 36 2024-03-26T19:06:31.327 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-03-26T19:06:31.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.433+0000 7f4a643fa640 1 -- 172.21.15.102:0/157321404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a5c104590 msgr2=0x7f4a5c104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:31.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.433+0000 7f4a643fa640 1 --2- 172.21.15.102:0/157321404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a5c104590 0x7f4a5c104960 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009880 tx=0x7f4a5802bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.433+0000 7f4a643fa640 1 -- 172.21.15.102:0/157321404 shutdown_connections 2024-03-26T19:06:31.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.433+0000 7f4a643fa640 1 --2- 172.21.15.102:0/157321404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a5c0fe5b0 0x7f4a5c0fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.433+0000 7f4a643fa640 1 --2- 172.21.15.102:0/157321404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a5c104590 0x7f4a5c104960 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.433+0000 7f4a643fa640 1 -- 172.21.15.102:0/157321404 >> 172.21.15.102:0/157321404 conn(0x7f4a5c0743d0 msgr2=0x7f4a5c0747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:31.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.433+0000 7f4a643fa640 1 -- 172.21.15.102:0/157321404 shutdown_connections 2024-03-26T19:06:31.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.433+0000 7f4a643fa640 1 -- 172.21.15.102:0/157321404 wait complete. 2024-03-26T19:06:31.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a643fa640 1 Processor -- start 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a643fa640 1 -- start start 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a643fa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a5c0fe5b0 0x7f4a5c1941c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a643fa640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a5c104590 0x7f4a5c194700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a643fa640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c194d90 con 0x7f4a5c104590 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a643fa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c198b60 con 0x7f4a5c0fe5b0 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a61995640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a5c104590 0x7f4a5c194700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a61995640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a5c104590 0x7f4a5c194700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57050/0 (socket says 172.21.15.102:57050) 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a61995640 1 -- 172.21.15.102:0/3010749898 learned_addr learned my addr 172.21.15.102:0/3010749898 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a62196640 1 --2- 172.21.15.102:0/3010749898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a5c0fe5b0 0x7f4a5c1941c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a61995640 1 -- 172.21.15.102:0/3010749898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a5c0fe5b0 msgr2=0x7f4a5c1941c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a61995640 1 --2- 172.21.15.102:0/3010749898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a5c0fe5b0 0x7f4a5c1941c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.434+0000 7f4a61995640 1 -- 172.21.15.102:0/3010749898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a58009560 con 0x7f4a5c104590 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.435+0000 7f4a61995640 1 --2- 172.21.15.102:0/3010749898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a5c104590 0x7f4a5c194700 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000e970 tx=0x7f4a5000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:31.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.435+0000 7f4a4f7fe640 1 -- 172.21.15.102:0/3010749898 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a5000cc90 con 0x7f4a5c104590 2024-03-26T19:06:31.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.435+0000 7f4a643fa640 1 -- 172.21.15.102:0/3010749898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5c198e40 con 0x7f4a5c104590 2024-03-26T19:06:31.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.435+0000 7f4a4f7fe640 1 -- 172.21.15.102:0/3010749898 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a5000cdf0 con 0x7f4a5c104590 2024-03-26T19:06:31.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.435+0000 7f4a4f7fe640 1 -- 172.21.15.102:0/3010749898 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a50010430 con 0x7f4a5c104590 2024-03-26T19:06:31.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.435+0000 7f4a643fa640 1 -- 172.21.15.102:0/3010749898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5c199420 con 0x7f4a5c104590 2024-03-26T19:06:31.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.436+0000 7f4a643fa640 1 -- 172.21.15.102:0/3010749898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c04e000 con 0x7f4a5c104590 2024-03-26T19:06:31.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.436+0000 7f4a4f7fe640 1 -- 172.21.15.102:0/3010749898 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4a500046c0 con 0x7f4a5c104590 2024-03-26T19:06:31.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.437+0000 7f4a4f7fe640 1 --2- 172.21.15.102:0/3010749898 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f4a4806e3d0 0x7f4a48070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:31.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.437+0000 7f4a4f7fe640 1 -- 172.21.15.102:0/3010749898 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f4a50014070 con 0x7f4a5c104590 2024-03-26T19:06:31.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.437+0000 7f4a62196640 1 --2- 172.21.15.102:0/3010749898 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f4a4806e3d0 0x7f4a48070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:31.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.438+0000 7f4a62196640 1 --2- 172.21.15.102:0/3010749898 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f4a4806e3d0 0x7f4a48070880 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4a58006010 tx=0x7f4a58005dd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:31.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.442+0000 7f4a4f7fe640 1 -- 172.21.15.102:0/3010749898 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4a50057af0 con 0x7f4a5c104590 2024-03-26T19:06:31.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:31 smithi190 ceph-mon[65689]: pgmap v333: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:31.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:31 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2676124034' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-03-26T19:06:31.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:31 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1305419726' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-03-26T19:06:31.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.683+0000 7f4a643fa640 1 -- 172.21.15.102:0/3010749898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7f4a5c195510 con 0x7f4a5c104590 2024-03-26T19:06:31.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.684+0000 7f4a4f7fe640 1 -- 172.21.15.102:0/3010749898 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v43) v1 ==== 107+0+5204 (secure 0 0 0) 0x7f4a5005b1b0 con 0x7f4a5c104590 2024-03-26T19:06:31.681 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:31.681 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44343,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/782636706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":782636706},{"type":"v1","addr":"172.21.15.190:6825","nonce":782636706}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44373,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6827/925215849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":925215849},{"type":"v1","addr":"172.21.15.190:6827","nonce":925215849}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:46.327380+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34436},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_34436":{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-26T19:06:31.682 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 37 2024-03-26T19:06:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.686+0000 7f4a643fa640 1 -- 172.21.15.102:0/3010749898 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f4a4806e3d0 msgr2=0x7f4a48070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.686+0000 7f4a643fa640 1 --2- 172.21.15.102:0/3010749898 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f4a4806e3d0 0x7f4a48070880 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4a58006010 tx=0x7f4a58005dd0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.686+0000 7f4a643fa640 1 -- 172.21.15.102:0/3010749898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a5c104590 msgr2=0x7f4a5c194700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.686+0000 7f4a643fa640 1 --2- 172.21.15.102:0/3010749898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a5c104590 0x7f4a5c194700 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000e970 tx=0x7f4a5000ed70 comp rx=0 tx=0).stop 2024-03-26T19:06:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.686+0000 7f4a643fa640 1 -- 172.21.15.102:0/3010749898 shutdown_connections 2024-03-26T19:06:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.686+0000 7f4a643fa640 1 --2- 172.21.15.102:0/3010749898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a5c104590 0x7f4a5c194700 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.686+0000 7f4a643fa640 1 --2- 172.21.15.102:0/3010749898 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f4a4806e3d0 0x7f4a48070880 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.686+0000 7f4a643fa640 1 --2- 172.21.15.102:0/3010749898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a5c0fe5b0 0x7f4a5c1941c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.686+0000 7f4a643fa640 1 -- 172.21.15.102:0/3010749898 >> 172.21.15.102:0/3010749898 conn(0x7f4a5c0743d0 msgr2=0x7f4a5c0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.686+0000 7f4a643fa640 1 -- 172.21.15.102:0/3010749898 shutdown_connections 2024-03-26T19:06:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.686+0000 7f4a643fa640 1 -- 172.21.15.102:0/3010749898 wait complete. 2024-03-26T19:06:31.694 DEBUG:tasks.fs:max_mds reduced in epoch 37 2024-03-26T19:06:31.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-03-26T19:06:31.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.812+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3794242904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8067350 msgr2=0x7fd0f80677c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:31.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.812+0000 7fd0fff5c640 1 --2- 172.21.15.102:0/3794242904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8067350 0x7fd0f80677c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009880 tx=0x7fd0f402bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.812+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3794242904 shutdown_connections 2024-03-26T19:06:31.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.812+0000 7fd0fff5c640 1 --2- 172.21.15.102:0/3794242904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8067350 0x7fd0f80677c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.812+0000 7fd0fff5c640 1 --2- 172.21.15.102:0/3794242904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f8105690 0x7fd0f8105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.812+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3794242904 >> 172.21.15.102:0/3794242904 conn(0x7fd0f80743d0 msgr2=0x7fd0f80747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.812+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3794242904 shutdown_connections 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.813+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3794242904 wait complete. 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.813+0000 7fd0fff5c640 1 Processor -- start 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.813+0000 7fd0fff5c640 1 -- start start 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.813+0000 7fd0fff5c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8067350 0x7fd0f81952b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.813+0000 7fd0fff5c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f8105690 0x7fd0f81957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.813+0000 7fd0fff5c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f8195e80 con 0x7fd0f8067350 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.813+0000 7fd0fff5c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f8199c50 con 0x7fd0f8105690 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.813+0000 7fd0fdcf8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8067350 0x7fd0f81952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.813+0000 7fd0fdcf8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8067350 0x7fd0f81952b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57080/0 (socket says 172.21.15.102:57080) 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.813+0000 7fd0fdcf8640 1 -- 172.21.15.102:0/3498376915 learned_addr learned my addr 172.21.15.102:0/3498376915 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.814+0000 7fd0fd4f7640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f8105690 0x7fd0f81957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:31.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.814+0000 7fd0fdcf8640 1 -- 172.21.15.102:0/3498376915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f8105690 msgr2=0x7fd0f81957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:31.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.814+0000 7fd0fdcf8640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f8105690 0x7fd0f81957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:31.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.814+0000 7fd0fdcf8640 1 -- 172.21.15.102:0/3498376915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f4009560 con 0x7fd0f8067350 2024-03-26T19:06:31.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.814+0000 7fd0fd4f7640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f8105690 0x7fd0f81957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T19:06:31.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.814+0000 7fd0fdcf8640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8067350 0x7fd0f81952b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800d6f0 tx=0x7fd0e800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:31.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.814+0000 7fd0eeffd640 1 -- 172.21.15.102:0/3498376915 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0e8009a30 con 0x7fd0f8067350 2024-03-26T19:06:31.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.814+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3498376915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f8199f30 con 0x7fd0f8067350 2024-03-26T19:06:31.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.814+0000 7fd0eeffd640 1 -- 172.21.15.102:0/3498376915 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0e80054e0 con 0x7fd0f8067350 2024-03-26T19:06:31.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.814+0000 7fd0eeffd640 1 -- 172.21.15.102:0/3498376915 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0e8010460 con 0x7fd0f8067350 2024-03-26T19:06:31.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.814+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3498376915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f819a510 con 0x7fd0f8067350 2024-03-26T19:06:31.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.815+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3498376915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f804e000 con 0x7fd0f8067350 2024-03-26T19:06:31.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.816+0000 7fd0eeffd640 1 -- 172.21.15.102:0/3498376915 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd0e8009b90 con 0x7fd0f8067350 2024-03-26T19:06:31.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.816+0000 7fd0eeffd640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd0e406e3d0 0x7fd0e4070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:31.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.816+0000 7fd0eeffd640 1 -- 172.21.15.102:0/3498376915 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fd0e808e000 con 0x7fd0f8067350 2024-03-26T19:06:31.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.816+0000 7fd0fd4f7640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd0e406e3d0 0x7fd0e4070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:31.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.817+0000 7fd0fd4f7640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd0e406e3d0 0x7fd0e4070880 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd0f81968c0 tx=0x7fd0f402cb40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:31.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:31.821+0000 7fd0eeffd640 1 -- 172.21.15.102:0/3498376915 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd0e8057160 con 0x7fd0f8067350 2024-03-26T19:06:32.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.046+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3498376915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7fd0f8196600 con 0x7fd0f8067350 2024-03-26T19:06:32.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.046+0000 7fd0eeffd640 1 -- 172.21.15.102:0/3498376915 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v43) v1 ==== 107+0+5208 (secure 0 0 0) 0x7fd0e805a820 con 0x7fd0f8067350 2024-03-26T19:06:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:32.044 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44343,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/782636706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":782636706},{"type":"v1","addr":"172.21.15.190:6825","nonce":782636706}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44373,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6827/925215849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":925215849},{"type":"v1","addr":"172.21.15.190:6827","nonce":925215849}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:53.776128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34436},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_34436":{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":0,"incarnation":36,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-26T19:06:32.044 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 38 2024-03-26T19:06:32.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.049+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3498376915 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd0e406e3d0 msgr2=0x7fd0e4070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:32.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.049+0000 7fd0fff5c640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd0e406e3d0 0x7fd0e4070880 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd0f81968c0 tx=0x7fd0f402cb40 comp rx=0 tx=0).stop 2024-03-26T19:06:32.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.049+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3498376915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8067350 msgr2=0x7fd0f81952b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:32.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.049+0000 7fd0fff5c640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8067350 0x7fd0f81952b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800d6f0 tx=0x7fd0e800dbb0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.049+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3498376915 shutdown_connections 2024-03-26T19:06:32.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.049+0000 7fd0fff5c640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fd0e406e3d0 0x7fd0e4070880 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.049+0000 7fd0fff5c640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f8105690 0x7fd0f81957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.049+0000 7fd0fff5c640 1 --2- 172.21.15.102:0/3498376915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8067350 0x7fd0f81952b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.049+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3498376915 >> 172.21.15.102:0/3498376915 conn(0x7fd0f80743d0 msgr2=0x7fd0f80fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:32.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.049+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3498376915 shutdown_connections 2024-03-26T19:06:32.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.049+0000 7fd0fff5c640 1 -- 172.21.15.102:0/3498376915 wait complete. 2024-03-26T19:06:32.058 DEBUG:tasks.fs:max_mds reduced in epoch 38 2024-03-26T19:06:32.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 39 2024-03-26T19:06:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.168+0000 7f96792c4640 1 -- 172.21.15.102:0/3693958132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96740ff000 msgr2=0x7f9674109a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.168+0000 7f96792c4640 1 --2- 172.21.15.102:0/3693958132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96740ff000 0x7f9674109a70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f96680098b0 tx=0x7f966802bcd0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.168+0000 7f96792c4640 1 -- 172.21.15.102:0/3693958132 shutdown_connections 2024-03-26T19:06:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.168+0000 7f96792c4640 1 --2- 172.21.15.102:0/3693958132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96740ff000 0x7f9674109a70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.168+0000 7f96792c4640 1 --2- 172.21.15.102:0/3693958132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96740fe6f0 0x7f96740feac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.168+0000 7f96792c4640 1 -- 172.21.15.102:0/3693958132 >> 172.21.15.102:0/3693958132 conn(0x7f9674073d20 msgr2=0x7f9674074120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.168+0000 7f96792c4640 1 -- 172.21.15.102:0/3693958132 shutdown_connections 2024-03-26T19:06:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.168+0000 7f96792c4640 1 -- 172.21.15.102:0/3693958132 wait complete. 2024-03-26T19:06:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.169+0000 7f96792c4640 1 Processor -- start 2024-03-26T19:06:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.169+0000 7f96792c4640 1 -- start start 2024-03-26T19:06:32.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.169+0000 7f96792c4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96740fe6f0 0x7f96741952f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:32.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.169+0000 7f96792c4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96740ff000 0x7f9674195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.169+0000 7f96792c4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9674195ec0 con 0x7f96740fe6f0 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.169+0000 7f96792c4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9674199c90 con 0x7f96740ff000 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.169+0000 7f9672ffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96740fe6f0 0x7f96741952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.169+0000 7f9672ffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96740fe6f0 0x7f96741952f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57092/0 (socket says 172.21.15.102:57092) 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.169+0000 7f9672ffd640 1 -- 172.21.15.102:0/2225485957 learned_addr learned my addr 172.21.15.102:0/2225485957 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.169+0000 7f96727fc640 1 --2- 172.21.15.102:0/2225485957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96740ff000 0x7f9674195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.170+0000 7f9672ffd640 1 -- 172.21.15.102:0/2225485957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96740ff000 msgr2=0x7f9674195830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.170+0000 7f9672ffd640 1 --2- 172.21.15.102:0/2225485957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96740ff000 0x7f9674195830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.170+0000 7f9672ffd640 1 -- 172.21.15.102:0/2225485957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9668009560 con 0x7f96740fe6f0 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.170+0000 7f9672ffd640 1 --2- 172.21.15.102:0/2225485957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96740fe6f0 0x7f96741952f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f966400c810 tx=0x7f966400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.170+0000 7f965bfff640 1 -- 172.21.15.102:0/2225485957 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96640041f0 con 0x7f96740fe6f0 2024-03-26T19:06:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.170+0000 7f965bfff640 1 -- 172.21.15.102:0/2225485957 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9664007e70 con 0x7f96740fe6f0 2024-03-26T19:06:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.170+0000 7f965bfff640 1 -- 172.21.15.102:0/2225485957 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96640049d0 con 0x7f96740fe6f0 2024-03-26T19:06:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.170+0000 7f96792c4640 1 -- 172.21.15.102:0/2225485957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9674199f70 con 0x7f96740fe6f0 2024-03-26T19:06:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.170+0000 7f96792c4640 1 -- 172.21.15.102:0/2225485957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967419a4a0 con 0x7f96740fe6f0 2024-03-26T19:06:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.171+0000 7f96792c4640 1 -- 172.21.15.102:0/2225485957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967404e000 con 0x7f96740fe6f0 2024-03-26T19:06:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.172+0000 7f965bfff640 1 -- 172.21.15.102:0/2225485957 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f96640079b0 con 0x7f96740fe6f0 2024-03-26T19:06:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.172+0000 7f965bfff640 1 --2- 172.21.15.102:0/2225485957 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f965c06e3d0 0x7f965c070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.172+0000 7f965bfff640 1 -- 172.21.15.102:0/2225485957 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f966408e190 con 0x7f96740fe6f0 2024-03-26T19:06:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.172+0000 7f96727fc640 1 --2- 172.21.15.102:0/2225485957 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f965c06e3d0 0x7f965c070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.173+0000 7f96727fc640 1 --2- 172.21.15.102:0/2225485957 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f965c06e3d0 0x7f965c070880 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9668006010 tx=0x7f966802df90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:32.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.179+0000 7f965bfff640 1 -- 172.21.15.102:0/2225485957 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9664057370 con 0x7f96740fe6f0 2024-03-26T19:06:32.310 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:32 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/4217709329' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-03-26T19:06:32.310 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:32 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3010749898' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-03-26T19:06:32.310 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:32 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3498376915' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-03-26T19:06:32.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.451+0000 7f96792c4640 1 -- 172.21.15.102:0/2225485957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 39, "format": "json"} v 0) v1 -- 0x7f96741070e0 con 0x7f96740fe6f0 2024-03-26T19:06:32.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.451+0000 7f965bfff640 1 -- 172.21.15.102:0/2225485957 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 39, "format": "json"}]=0 dumped fsmap epoch 39 v43) v1 ==== 107+0+5205 (secure 0 0 0) 0x7f966405aa30 con 0x7f96740fe6f0 2024-03-26T19:06:32.449 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:32.449 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44343,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/782636706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":782636706},{"type":"v1","addr":"172.21.15.190:6825","nonce":782636706}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44373,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6827/925215849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":925215849},{"type":"v1","addr":"172.21.15.190:6827","nonce":925215849}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:54.380126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34436},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_34436":{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":0,"incarnation":36,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-26T19:06:32.449 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 39 2024-03-26T19:06:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.453+0000 7f96792c4640 1 -- 172.21.15.102:0/2225485957 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f965c06e3d0 msgr2=0x7f965c070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.453+0000 7f96792c4640 1 --2- 172.21.15.102:0/2225485957 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f965c06e3d0 0x7f965c070880 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9668006010 tx=0x7f966802df90 comp rx=0 tx=0).stop 2024-03-26T19:06:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.453+0000 7f96792c4640 1 -- 172.21.15.102:0/2225485957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96740fe6f0 msgr2=0x7f96741952f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.453+0000 7f96792c4640 1 --2- 172.21.15.102:0/2225485957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96740fe6f0 0x7f96741952f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f966400c810 tx=0x7f966400ccd0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.453+0000 7f96792c4640 1 -- 172.21.15.102:0/2225485957 shutdown_connections 2024-03-26T19:06:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.453+0000 7f96792c4640 1 --2- 172.21.15.102:0/2225485957 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f965c06e3d0 0x7f965c070880 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.453+0000 7f96792c4640 1 --2- 172.21.15.102:0/2225485957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96740ff000 0x7f9674195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.453+0000 7f96792c4640 1 --2- 172.21.15.102:0/2225485957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96740fe6f0 0x7f96741952f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.453+0000 7f96792c4640 1 -- 172.21.15.102:0/2225485957 >> 172.21.15.102:0/2225485957 conn(0x7f9674073d20 msgr2=0x7f96741042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.454+0000 7f96792c4640 1 -- 172.21.15.102:0/2225485957 shutdown_connections 2024-03-26T19:06:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.454+0000 7f96792c4640 1 -- 172.21.15.102:0/2225485957 wait complete. 2024-03-26T19:06:32.463 DEBUG:tasks.fs:max_mds reduced in epoch 39 2024-03-26T19:06:32.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 40 2024-03-26T19:06:32.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:32 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/4217709329' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-03-26T19:06:32.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:32 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3010749898' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-03-26T19:06:32.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:32 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3498376915' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-03-26T19:06:32.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.574+0000 7fc642111640 1 -- 172.21.15.102:0/3317355106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c105660 msgr2=0x7fc63c105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:32.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.574+0000 7fc642111640 1 --2- 172.21.15.102:0/3317355106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c105660 0x7fc63c105a30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009c20 tx=0x7fc62c02c1b0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.574+0000 7fc642111640 1 -- 172.21.15.102:0/3317355106 shutdown_connections 2024-03-26T19:06:32.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.574+0000 7fc642111640 1 --2- 172.21.15.102:0/3317355106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63c0ff650 0x7fc63c0ffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.574+0000 7fc642111640 1 --2- 172.21.15.102:0/3317355106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c105660 0x7fc63c105a30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.574+0000 7fc642111640 1 -- 172.21.15.102:0/3317355106 >> 172.21.15.102:0/3317355106 conn(0x7fc63c0fb170 msgr2=0x7fc63c0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:32.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.574+0000 7fc642111640 1 -- 172.21.15.102:0/3317355106 shutdown_connections 2024-03-26T19:06:32.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.574+0000 7fc642111640 1 -- 172.21.15.102:0/3317355106 wait complete. 2024-03-26T19:06:32.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.574+0000 7fc642111640 1 Processor -- start 2024-03-26T19:06:32.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc642111640 1 -- start start 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc642111640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ff650 0x7fc63c1953b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc642111640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63c105660 0x7fc63c1958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc642111640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c195f80 con 0x7fc63c0ff650 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc642111640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c199d50 con 0x7fc63c105660 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc63b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ff650 0x7fc63c1953b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc63b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ff650 0x7fc63c1953b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57098/0 (socket says 172.21.15.102:57098) 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc63b7fe640 1 -- 172.21.15.102:0/3896604777 learned_addr learned my addr 172.21.15.102:0/3896604777 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc63affd640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63c105660 0x7fc63c1958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc63b7fe640 1 -- 172.21.15.102:0/3896604777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63c105660 msgr2=0x7fc63c1958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc63b7fe640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63c105660 0x7fc63c1958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc63b7fe640 1 -- 172.21.15.102:0/3896604777 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62c009900 con 0x7fc63c0ff650 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.575+0000 7fc63affd640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63c105660 0x7fc63c1958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:06:32.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.576+0000 7fc63b7fe640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ff650 0x7fc63c1953b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc62c032b60 tx=0x7fc62c032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:32.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.576+0000 7fc638ff9640 1 -- 172.21.15.102:0/3896604777 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc62c034820 con 0x7fc63c0ff650 2024-03-26T19:06:32.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.576+0000 7fc638ff9640 1 -- 172.21.15.102:0/3896604777 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc62c003e30 con 0x7fc63c0ff650 2024-03-26T19:06:32.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.576+0000 7fc642111640 1 -- 172.21.15.102:0/3896604777 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63c199fd0 con 0x7fc63c0ff650 2024-03-26T19:06:32.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.576+0000 7fc638ff9640 1 -- 172.21.15.102:0/3896604777 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc62c03f440 con 0x7fc63c0ff650 2024-03-26T19:06:32.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.576+0000 7fc642111640 1 -- 172.21.15.102:0/3896604777 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63c19a4d0 con 0x7fc63c0ff650 2024-03-26T19:06:32.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.577+0000 7fc642111640 1 -- 172.21.15.102:0/3896604777 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63c04e000 con 0x7fc63c0ff650 2024-03-26T19:06:32.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.577+0000 7fc638ff9640 1 -- 172.21.15.102:0/3896604777 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc62c03b040 con 0x7fc63c0ff650 2024-03-26T19:06:32.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.578+0000 7fc638ff9640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc62406e3d0 0x7fc624070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:32.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.578+0000 7fc638ff9640 1 -- 172.21.15.102:0/3896604777 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc62c0b11c0 con 0x7fc63c0ff650 2024-03-26T19:06:32.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.578+0000 7fc63affd640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc62406e3d0 0x7fc624070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:32.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.578+0000 7fc63affd640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc62406e3d0 0x7fc624070880 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc63c1969c0 tx=0x7fc630005d30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.587+0000 7fc638ff9640 1 -- 172.21.15.102:0/3896604777 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc62c07a320 con 0x7fc63c0ff650 2024-03-26T19:06:32.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.989+0000 7fc642111640 1 -- 172.21.15.102:0/3896604777 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 40, "format": "json"} v 0) v1 -- 0x7fc63c196700 con 0x7fc63c0ff650 2024-03-26T19:06:32.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.990+0000 7fc638ff9640 1 -- 172.21.15.102:0/3896604777 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 40, "format": "json"}]=0 dumped fsmap epoch 40 v43) v1 ==== 107+0+5214 (secure 0 0 0) 0x7fc62c07d9e0 con 0x7fc63c0ff650 2024-03-26T19:06:32.987 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44343,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/782636706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":782636706},{"type":"v1","addr":"172.21.15.190:6825","nonce":782636706}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44373,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6827/925215849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":925215849},{"type":"v1","addr":"172.21.15.190:6827","nonce":925215849}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:02:56.378341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34436},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_34436":{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":0,"incarnation":36,"state":"up:active","state_seq":19,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34436,"qdb_cluster":[34436]},"id":1}]} 2024-03-26T19:06:32.988 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 40 2024-03-26T19:06:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.993+0000 7fc642111640 1 -- 172.21.15.102:0/3896604777 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc62406e3d0 msgr2=0x7fc624070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.993+0000 7fc642111640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc62406e3d0 0x7fc624070880 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc63c1969c0 tx=0x7fc630005d30 comp rx=0 tx=0).stop 2024-03-26T19:06:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.993+0000 7fc642111640 1 -- 172.21.15.102:0/3896604777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ff650 msgr2=0x7fc63c1953b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.993+0000 7fc642111640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ff650 0x7fc63c1953b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc62c032b60 tx=0x7fc62c032b90 comp rx=0 tx=0).stop 2024-03-26T19:06:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.993+0000 7fc642111640 1 -- 172.21.15.102:0/3896604777 shutdown_connections 2024-03-26T19:06:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.993+0000 7fc642111640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fc62406e3d0 0x7fc624070880 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.993+0000 7fc642111640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63c105660 0x7fc63c1958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.993+0000 7fc642111640 1 --2- 172.21.15.102:0/3896604777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ff650 0x7fc63c1953b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:32.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.993+0000 7fc642111640 1 -- 172.21.15.102:0/3896604777 >> 172.21.15.102:0/3896604777 conn(0x7fc63c0fb170 msgr2=0x7fc63c0fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:32.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.993+0000 7fc642111640 1 -- 172.21.15.102:0/3896604777 shutdown_connections 2024-03-26T19:06:32.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:32.993+0000 7fc642111640 1 -- 172.21.15.102:0/3896604777 wait complete. 2024-03-26T19:06:33.003 DEBUG:tasks.fs:max_mds reduced in epoch 40 2024-03-26T19:06:33.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 41 2024-03-26T19:06:33.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.115+0000 7f3a928b9640 1 -- 172.21.15.102:0/2832130941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a8c1056a0 msgr2=0x7f3a8c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:33.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.115+0000 7f3a928b9640 1 --2- 172.21.15.102:0/2832130941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a8c1056a0 0x7f3a8c105a70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009880 tx=0x7f3a7c02bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.115+0000 7f3a928b9640 1 -- 172.21.15.102:0/2832130941 shutdown_connections 2024-03-26T19:06:33.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.115+0000 7f3a928b9640 1 --2- 172.21.15.102:0/2832130941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a8c0ff690 0x7f3a8c0ffb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.115+0000 7f3a928b9640 1 --2- 172.21.15.102:0/2832130941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a8c1056a0 0x7f3a8c105a70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.115+0000 7f3a928b9640 1 -- 172.21.15.102:0/2832130941 >> 172.21.15.102:0/2832130941 conn(0x7f3a8c0fb1d0 msgr2=0x7f3a8c0fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:33.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.115+0000 7f3a928b9640 1 -- 172.21.15.102:0/2832130941 shutdown_connections 2024-03-26T19:06:33.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.115+0000 7f3a928b9640 1 -- 172.21.15.102:0/2832130941 wait complete. 2024-03-26T19:06:33.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.115+0000 7f3a928b9640 1 Processor -- start 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.115+0000 7f3a928b9640 1 -- start start 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a928b9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a8c0ff690 0x7f3a8c195370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a928b9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a8c1056a0 0x7f3a8c1958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a928b9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c195f40 con 0x7f3a8c1056a0 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a928b9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c199d10 con 0x7f3a8c0ff690 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a8b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a8c1056a0 0x7f3a8c1958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a8b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a8c1056a0 0x7f3a8c1958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57108/0 (socket says 172.21.15.102:57108) 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a8b7fe640 1 -- 172.21.15.102:0/3471996638 learned_addr learned my addr 172.21.15.102:0/3471996638 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a8bfff640 1 --2- 172.21.15.102:0/3471996638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a8c0ff690 0x7f3a8c195370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a8b7fe640 1 -- 172.21.15.102:0/3471996638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a8c0ff690 msgr2=0x7f3a8c195370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a8b7fe640 1 --2- 172.21.15.102:0/3471996638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a8c0ff690 0x7f3a8c195370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a8b7fe640 1 -- 172.21.15.102:0/3471996638 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c009560 con 0x7f3a8c1056a0 2024-03-26T19:06:33.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.116+0000 7f3a8b7fe640 1 --2- 172.21.15.102:0/3471996638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a8c1056a0 0x7f3a8c1958b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000d6f0 tx=0x7f3a8000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:33.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.117+0000 7f3a897fa640 1 -- 172.21.15.102:0/3471996638 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a80009a30 con 0x7f3a8c1056a0 2024-03-26T19:06:33.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.117+0000 7f3a928b9640 1 -- 172.21.15.102:0/3471996638 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8c199ff0 con 0x7f3a8c1056a0 2024-03-26T19:06:33.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.117+0000 7f3a897fa640 1 -- 172.21.15.102:0/3471996638 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a800054e0 con 0x7f3a8c1056a0 2024-03-26T19:06:33.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.117+0000 7f3a897fa640 1 -- 172.21.15.102:0/3471996638 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a80010460 con 0x7f3a8c1056a0 2024-03-26T19:06:33.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.117+0000 7f3a928b9640 1 -- 172.21.15.102:0/3471996638 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8c19a550 con 0x7f3a8c1056a0 2024-03-26T19:06:33.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.118+0000 7f3a928b9640 1 -- 172.21.15.102:0/3471996638 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8c04e000 con 0x7f3a8c1056a0 2024-03-26T19:06:33.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.118+0000 7f3a897fa640 1 -- 172.21.15.102:0/3471996638 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3a80009b90 con 0x7f3a8c1056a0 2024-03-26T19:06:33.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.119+0000 7f3a897fa640 1 --2- 172.21.15.102:0/3471996638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3a7406e3d0 0x7f3a74070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:33.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.119+0000 7f3a897fa640 1 -- 172.21.15.102:0/3471996638 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f3a8008df50 con 0x7f3a8c1056a0 2024-03-26T19:06:33.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.119+0000 7f3a8bfff640 1 --2- 172.21.15.102:0/3471996638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3a7406e3d0 0x7f3a74070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:33.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.119+0000 7f3a8bfff640 1 --2- 172.21.15.102:0/3471996638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3a7406e3d0 0x7f3a74070880 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c000c00 tx=0x7f3a7c005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:33.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.124+0000 7f3a897fa640 1 -- 172.21.15.102:0/3471996638 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3a80057160 con 0x7f3a8c1056a0 2024-03-26T19:06:33.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.344+0000 7f3a928b9640 1 -- 172.21.15.102:0/3471996638 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 41, "format": "json"} v 0) v1 -- 0x7f3a8c1966c0 con 0x7f3a8c1056a0 2024-03-26T19:06:33.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.345+0000 7f3a897fa640 1 -- 172.21.15.102:0/3471996638 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 41, "format": "json"}]=0 dumped fsmap epoch 41 v43) v1 ==== 107+0+5214 (secure 0 0 0) 0x7f3a8005a820 con 0x7f3a8c1056a0 2024-03-26T19:06:33.342 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:33.342 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44343,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/782636706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":782636706},{"type":"v1","addr":"172.21.15.190:6825","nonce":782636706}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44373,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6827/925215849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":925215849},{"type":"v1","addr":"172.21.15.190:6827","nonce":925215849}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:03:00.911316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0],"up":{"mds_0":34436},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_34436":{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":0,"incarnation":36,"state":"up:active","state_seq":19,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34436,"qdb_cluster":[34436]},"id":1}]} 2024-03-26T19:06:33.342 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 41 2024-03-26T19:06:33.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.347+0000 7f3a928b9640 1 -- 172.21.15.102:0/3471996638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3a7406e3d0 msgr2=0x7f3a74070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:33.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.347+0000 7f3a928b9640 1 --2- 172.21.15.102:0/3471996638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3a7406e3d0 0x7f3a74070880 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c000c00 tx=0x7f3a7c005fd0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.347+0000 7f3a928b9640 1 -- 172.21.15.102:0/3471996638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a8c1056a0 msgr2=0x7f3a8c1958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:33.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.347+0000 7f3a928b9640 1 --2- 172.21.15.102:0/3471996638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a8c1056a0 0x7f3a8c1958b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000d6f0 tx=0x7f3a8000dbb0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.347+0000 7f3a928b9640 1 -- 172.21.15.102:0/3471996638 shutdown_connections 2024-03-26T19:06:33.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.347+0000 7f3a928b9640 1 --2- 172.21.15.102:0/3471996638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a8c1056a0 0x7f3a8c1958b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.347+0000 7f3a928b9640 1 --2- 172.21.15.102:0/3471996638 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f3a7406e3d0 0x7f3a74070880 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.347+0000 7f3a928b9640 1 --2- 172.21.15.102:0/3471996638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a8c0ff690 0x7f3a8c195370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.347+0000 7f3a928b9640 1 -- 172.21.15.102:0/3471996638 >> 172.21.15.102:0/3471996638 conn(0x7f3a8c0fb1d0 msgr2=0x7f3a8c0fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:33.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.347+0000 7f3a928b9640 1 -- 172.21.15.102:0/3471996638 shutdown_connections 2024-03-26T19:06:33.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.347+0000 7f3a928b9640 1 -- 172.21.15.102:0/3471996638 wait complete. 2024-03-26T19:06:33.356 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:33 smithi102 ceph-mon[78322]: pgmap v334: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:33.356 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:33 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2225485957' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-03-26T19:06:33.356 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:33 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3896604777' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-03-26T19:06:33.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 42 2024-03-26T19:06:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.478+0000 7f160455c640 1 -- 172.21.15.102:0/1627597514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15fc1045b0 msgr2=0x7f15fc104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.478+0000 7f160455c640 1 --2- 172.21.15.102:0/1627597514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15fc1045b0 0x7f15fc104980 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009c20 tx=0x7f15f802c1b0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.478+0000 7f160455c640 1 -- 172.21.15.102:0/1627597514 shutdown_connections 2024-03-26T19:06:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.478+0000 7f160455c640 1 --2- 172.21.15.102:0/1627597514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15fc0fe580 0x7f15fc0fe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.478+0000 7f160455c640 1 --2- 172.21.15.102:0/1627597514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15fc1045b0 0x7f15fc104980 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.478+0000 7f160455c640 1 -- 172.21.15.102:0/1627597514 >> 172.21.15.102:0/1627597514 conn(0x7f15fc0fa040 msgr2=0x7f15fc0fc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.479+0000 7f160455c640 1 -- 172.21.15.102:0/1627597514 shutdown_connections 2024-03-26T19:06:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.479+0000 7f160455c640 1 -- 172.21.15.102:0/1627597514 wait complete. 2024-03-26T19:06:33.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.479+0000 7f160455c640 1 Processor -- start 2024-03-26T19:06:33.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.479+0000 7f160455c640 1 -- start start 2024-03-26T19:06:33.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f160455c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15fc0fe580 0x7f15fc1941a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f160455c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15fc1045b0 0x7f15fc1946e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f160455c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc194d70 con 0x7f15fc0fe580 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f160455c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc198b40 con 0x7f15fc1045b0 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f16022f8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15fc0fe580 0x7f15fc1941a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f16022f8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15fc0fe580 0x7f15fc1941a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57122/0 (socket says 172.21.15.102:57122) 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f16022f8640 1 -- 172.21.15.102:0/1461508339 learned_addr learned my addr 172.21.15.102:0/1461508339 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f1601af7640 1 --2- 172.21.15.102:0/1461508339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15fc1045b0 0x7f15fc1946e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f16022f8640 1 -- 172.21.15.102:0/1461508339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15fc1045b0 msgr2=0x7f15fc1946e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f16022f8640 1 --2- 172.21.15.102:0/1461508339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15fc1045b0 0x7f15fc1946e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f16022f8640 1 -- 172.21.15.102:0/1461508339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f8009900 con 0x7f15fc0fe580 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f16022f8640 1 --2- 172.21.15.102:0/1461508339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15fc0fe580 0x7f15fc1941a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f15f8037040 tx=0x7f15f8032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f15f37fe640 1 -- 172.21.15.102:0/1461508339 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15f8034820 con 0x7f15fc0fe580 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f160455c640 1 -- 172.21.15.102:0/1461508339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15fc198dc0 con 0x7f15fc0fe580 2024-03-26T19:06:33.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f15f37fe640 1 -- 172.21.15.102:0/1461508339 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15f8003e30 con 0x7f15fc0fe580 2024-03-26T19:06:33.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f15f37fe640 1 -- 172.21.15.102:0/1461508339 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15f803f440 con 0x7f15fc0fe580 2024-03-26T19:06:33.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.480+0000 7f160455c640 1 -- 172.21.15.102:0/1461508339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15fc199340 con 0x7f15fc0fe580 2024-03-26T19:06:33.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.481+0000 7f160455c640 1 -- 172.21.15.102:0/1461508339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15fc04e000 con 0x7f15fc0fe580 2024-03-26T19:06:33.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.482+0000 7f15f37fe640 1 -- 172.21.15.102:0/1461508339 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f15f803b040 con 0x7f15fc0fe580 2024-03-26T19:06:33.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.482+0000 7f15f37fe640 1 --2- 172.21.15.102:0/1461508339 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f15e806e3d0 0x7f15e8070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:33.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.482+0000 7f1601af7640 1 --2- 172.21.15.102:0/1461508339 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f15e806e3d0 0x7f15e8070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:33.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.483+0000 7f15f37fe640 1 -- 172.21.15.102:0/1461508339 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f15f80b1540 con 0x7f15fc0fe580 2024-03-26T19:06:33.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.483+0000 7f1601af7640 1 --2- 172.21.15.102:0/1461508339 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f15e806e3d0 0x7f15e8070880 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f15fc1957b0 tx=0x7f15ec00a4c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:33.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.487+0000 7f15f37fe640 1 -- 172.21.15.102:0/1461508339 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f15f807a760 con 0x7f15fc0fe580 2024-03-26T19:06:33.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:33 smithi190 ceph-mon[65689]: pgmap v334: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:33.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:33 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2225485957' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-03-26T19:06:33.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:33 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3896604777' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-03-26T19:06:33.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.746+0000 7f160455c640 1 -- 172.21.15.102:0/1461508339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 42, "format": "json"} v 0) v1 -- 0x7f15fc1069c0 con 0x7f15fc0fe580 2024-03-26T19:06:33.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.746+0000 7f15f37fe640 1 -- 172.21.15.102:0/1461508339 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 42, "format": "json"}]=0 dumped fsmap epoch 42 v43) v1 ==== 107+0+5231 (secure 0 0 0) 0x7f15f807de20 con 0x7f15fc0fe580 2024-03-26T19:06:33.744 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:33.744 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44343,"name":"cephfs.smithi190.vykizb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6825/782636706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":782636706},{"type":"v1","addr":"172.21.15.190:6825","nonce":782636706}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44373,"name":"cephfs.smithi190.hwsvdg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6827/925215849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":925215849},{"type":"v1","addr":"172.21.15.190:6827","nonce":925215849}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-26T18:52:19.251412+0000","modified":"2024-03-26T19:03:01.913604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34436,"mds_1":34468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34436":{"gid":34436,"name":"cephfs.smithi102.tvsqaa","rank":0,"incarnation":36,"state":"up:active","state_seq":19,"addr":"172.21.15.102:6827/2063637342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2063637342},{"type":"v1","addr":"172.21.15.102:6827","nonce":2063637342}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34468":{"gid":34468,"name":"cephfs.smithi102.hrbrsp","rank":1,"incarnation":42,"state":"up:starting","state_seq":1,"addr":"172.21.15.102:6829/3007238267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3007238267},{"type":"v1","addr":"172.21.15.102:6829","nonce":3007238267}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34436,"qdb_cluster":[34436]},"id":1}]} 2024-03-26T19:06:33.745 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 42 2024-03-26T19:06:33.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.748+0000 7f160455c640 1 -- 172.21.15.102:0/1461508339 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f15e806e3d0 msgr2=0x7f15e8070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:33.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.748+0000 7f160455c640 1 --2- 172.21.15.102:0/1461508339 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f15e806e3d0 0x7f15e8070880 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f15fc1957b0 tx=0x7f15ec00a4c0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.748+0000 7f160455c640 1 -- 172.21.15.102:0/1461508339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15fc0fe580 msgr2=0x7f15fc1941a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:33.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.748+0000 7f160455c640 1 --2- 172.21.15.102:0/1461508339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15fc0fe580 0x7f15fc1941a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f15f8037040 tx=0x7f15f8032b60 comp rx=0 tx=0).stop 2024-03-26T19:06:33.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.748+0000 7f160455c640 1 -- 172.21.15.102:0/1461508339 shutdown_connections 2024-03-26T19:06:33.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.748+0000 7f160455c640 1 --2- 172.21.15.102:0/1461508339 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f15e806e3d0 0x7f15e8070880 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.748+0000 7f160455c640 1 --2- 172.21.15.102:0/1461508339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15fc1045b0 0x7f15fc1946e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.748+0000 7f160455c640 1 --2- 172.21.15.102:0/1461508339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15fc0fe580 0x7f15fc1941a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:33.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.748+0000 7f160455c640 1 -- 172.21.15.102:0/1461508339 >> 172.21.15.102:0/1461508339 conn(0x7f15fc0fa040 msgr2=0x7f15fc0fbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:33.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.749+0000 7f160455c640 1 -- 172.21.15.102:0/1461508339 shutdown_connections 2024-03-26T19:06:33.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:33.749+0000 7f160455c640 1 -- 172.21.15.102:0/1461508339 wait complete. 2024-03-26T19:06:33.757 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-26T19:06:33.772 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-26T19:06:33.773 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T19:06:33.773 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-26T19:06:33.790 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-26T19:06:33.790 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-26T19:06:33.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T19:06:34.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.013+0000 7fae42881640 1 -- 172.21.15.102:0/1482615551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3c072eb0 msgr2=0x7fae3c10b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.013+0000 7fae42881640 1 --2- 172.21.15.102:0/1482615551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3c072eb0 0x7fae3c10b9c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fae30009880 tx=0x7fae3002bd70 comp rx=0 tx=0).stop 2024-03-26T19:06:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.013+0000 7fae42881640 1 -- 172.21.15.102:0/1482615551 shutdown_connections 2024-03-26T19:06:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.013+0000 7fae42881640 1 --2- 172.21.15.102:0/1482615551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3c072eb0 0x7fae3c10b9c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.013+0000 7fae42881640 1 --2- 172.21.15.102:0/1482615551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae3c0725a0 0x7fae3c072970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.013+0000 7fae42881640 1 -- 172.21.15.102:0/1482615551 >> 172.21.15.102:0/1482615551 conn(0x7fae3c0fb140 msgr2=0x7fae3c0fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.013+0000 7fae42881640 1 -- 172.21.15.102:0/1482615551 shutdown_connections 2024-03-26T19:06:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.013+0000 7fae42881640 1 -- 172.21.15.102:0/1482615551 wait complete. 2024-03-26T19:06:34.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.014+0000 7fae42881640 1 Processor -- start 2024-03-26T19:06:34.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.014+0000 7fae42881640 1 -- start start 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.014+0000 7fae42881640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae3c0725a0 0x7fae3c197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.014+0000 7fae42881640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3c072eb0 0x7fae3c197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.014+0000 7fae42881640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c198000 con 0x7fae3c072eb0 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.014+0000 7fae42881640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c19bdd0 con 0x7fae3c0725a0 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.014+0000 7fae3b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3c072eb0 0x7fae3c197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.014+0000 7fae3b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3c072eb0 0x7fae3c197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57132/0 (socket says 172.21.15.102:57132) 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.014+0000 7fae3b7fe640 1 -- 172.21.15.102:0/1130107278 learned_addr learned my addr 172.21.15.102:0/1130107278 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.015+0000 7fae3bfff640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae3c0725a0 0x7fae3c197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.015+0000 7fae3b7fe640 1 -- 172.21.15.102:0/1130107278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae3c0725a0 msgr2=0x7fae3c197430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.015+0000 7fae3b7fe640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae3c0725a0 0x7fae3c197430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.015+0000 7fae3b7fe640 1 -- 172.21.15.102:0/1130107278 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae30009560 con 0x7fae3c072eb0 2024-03-26T19:06:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.015+0000 7fae3bfff640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae3c0725a0 0x7fae3c197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T19:06:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.015+0000 7fae3b7fe640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3c072eb0 0x7fae3c197970 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fae30009230 tx=0x7fae30004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.015+0000 7fae397fa640 1 -- 172.21.15.102:0/1130107278 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae300035f0 con 0x7fae3c072eb0 2024-03-26T19:06:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.015+0000 7fae397fa640 1 -- 172.21.15.102:0/1130107278 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae30003c10 con 0x7fae3c072eb0 2024-03-26T19:06:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.015+0000 7fae397fa640 1 -- 172.21.15.102:0/1130107278 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae3003e450 con 0x7fae3c072eb0 2024-03-26T19:06:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.015+0000 7fae42881640 1 -- 172.21.15.102:0/1130107278 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c19c050 con 0x7fae3c072eb0 2024-03-26T19:06:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.015+0000 7fae42881640 1 -- 172.21.15.102:0/1130107278 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c19c5b0 con 0x7fae3c072eb0 2024-03-26T19:06:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.016+0000 7fae42881640 1 -- 172.21.15.102:0/1130107278 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c04e000 con 0x7fae3c072eb0 2024-03-26T19:06:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.017+0000 7fae397fa640 1 -- 172.21.15.102:0/1130107278 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fae30004f60 con 0x7fae3c072eb0 2024-03-26T19:06:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.017+0000 7fae397fa640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae2406e490 0x7fae24070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.017+0000 7fae397fa640 1 -- 172.21.15.102:0/1130107278 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fae300b1070 con 0x7fae3c072eb0 2024-03-26T19:06:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.017+0000 7fae3bfff640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae2406e490 0x7fae24070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.018+0000 7fae3bfff640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae2406e490 0x7fae24070940 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0058c0 tx=0x7fae2c005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.022+0000 7fae397fa640 1 -- 172.21.15.102:0/1130107278 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fae3007a190 con 0x7fae3c072eb0 2024-03-26T19:06:34.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.207+0000 7fae42881640 1 -- 172.21.15.102:0/1130107278 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae3c10ea30 con 0x7fae3c072eb0 2024-03-26T19:06:34.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.208+0000 7fae397fa640 1 -- 172.21.15.102:0/1130107278 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16965 (secure 0 0 0) 0x7fae3007d850 con 0x7fae3c072eb0 2024-03-26T19:06:34.206 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:34.206 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":84,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","created":"2024-03-26T18:45:25.446329+0000","modified":"2024-03-26T19:02:46.321700+0000","last_up_change":"2024-03-26T19:01:03.417320+0000","last_in_change":"2024-03-26T18:50:45.452094+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T18:49:50.172501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-26T18:52:17.727534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-26T18:52:18.482241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5458d389-eb88-4798-92bc-b442168b37f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6803","nonce":3021596332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6805","nonce":3021596332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6809","nonce":3021596332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6807","nonce":3021596332}]},"public_addr":"172.21.15.102:6803/3021596332","cluster_addr":"172.21.15.102:6805/3021596332","heartbeat_back_addr":"172.21.15.102:6809/3021596332","heartbeat_front_addr":"172.21.15.102:6807/3021596332","state":["exists","up"]},{"osd":1,"uuid":"87508f24-8b3c-4f6d-add2-00ad59300da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6811","nonce":3336954503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6813","nonce":3336954503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6817","nonce":3336954503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6815","nonce":3336954503}]},"public_addr":"172.21.15.102:6811/3336954503","cluster_addr":"172.21.15.102:6813/3336954503","heartbeat_back_addr":"172.21.15.102:6817/3336954503","heartbeat_front_addr":"172.21.15.102:6815/3336954503","state":["exists","up"]},{"osd":2,"uuid":"c0179546-10c5-41f2-af0c-be93c2bbf5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6819","nonce":618947256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6821","nonce":618947256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6825","nonce":618947256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6823","nonce":618947256}]},"public_addr":"172.21.15.102:6819/618947256","cluster_addr":"172.21.15.102:6821/618947256","heartbeat_back_addr":"172.21.15.102:6825/618947256","heartbeat_front_addr":"172.21.15.102:6823/618947256","state":["exists","up"]},{"osd":3,"uuid":"4d524f7b-8b31-469b-b0c2-b94079febaeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6801","nonce":2294582584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6803","nonce":2294582584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6807","nonce":2294582584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6805","nonce":2294582584}]},"public_addr":"172.21.15.190:6801/2294582584","cluster_addr":"172.21.15.190:6803/2294582584","heartbeat_back_addr":"172.21.15.190:6807/2294582584","heartbeat_front_addr":"172.21.15.190:6805/2294582584","state":["exists","up"]},{"osd":4,"uuid":"865baef8-dd72-4449-b3a4-893801bc2958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6809","nonce":2940868243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6811","nonce":2940868243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6815","nonce":2940868243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6813","nonce":2940868243}]},"public_addr":"172.21.15.190:6809/2940868243","cluster_addr":"172.21.15.190:6811/2940868243","heartbeat_back_addr":"172.21.15.190:6815/2940868243","heartbeat_front_addr":"172.21.15.190:6813/2940868243","state":["exists","up"]},{"osd":5,"uuid":"5fd67007-d7ae-43b6-8002-258fbfc5a797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6817","nonce":3222089930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6819","nonce":3222089930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6823","nonce":3222089930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6821","nonce":3222089930}]},"public_addr":"172.21.15.190:6817/3222089930","cluster_addr":"172.21.15.190:6819/3222089930","heartbeat_back_addr":"172.21.15.190:6823/3222089930","heartbeat_front_addr":"172.21.15.190:6821/3222089930","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:48:51.274164+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:18.020320+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:46.066766+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:10.365857+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:36.582495+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:51:02.797707+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6827/624694186":"2024-03-27T19:02:46.321671+0000","172.21.15.190:6826/624694186":"2024-03-27T19:02:46.321671+0000","172.21.15.102:6829/989894576":"2024-03-27T19:02:06.598591+0000","172.21.15.102:6828/989894576":"2024-03-27T19:02:06.598591+0000","172.21.15.102:0/2151777534":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1215990907":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/2396893787":"2024-03-27T18:56:37.181768+0000","172.21.15.102:6801/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/3353740154":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1994330698":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6827/2550866692":"2024-03-27T18:52:35.022250+0000","172.21.15.102:0/1842664791":"2024-03-27T18:47:33.839189+0000","172.21.15.102:6801/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.190:0/2308588851":"2024-03-27T18:55:42.356278+0000","172.21.15.102:0/1295801893":"2024-03-27T18:45:55.925005+0000","172.21.15.102:6800/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6826/2550866692":"2024-03-27T18:52:35.022250+0000","172.21.15.102:0/2094779473":"2024-03-27T18:54:47.731502+0000","172.21.15.190:0/300222988":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6801/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/2686253586":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/1212290737":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/2752725029":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/2027932959":"2024-03-27T18:56:37.181768+0000","172.21.15.190:0/3113541506":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6800/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/216723823":"2024-03-27T18:54:47.731502+0000","172.21.15.190:6828/2858265260":"2024-03-27T18:55:42.356278+0000","172.21.15.190:6829/2858265260":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6800/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1380712530":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/990121767":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1480032448":"2024-03-27T18:54:47.731502+0000","172.21.15.190:0/1923628519":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6801/2831156761":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1818262291":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1615769300":"2024-03-27T18:54:47.731502+0000","172.21.15.190:0/833763291":"2024-03-27T18:55:42.356278+0000","172.21.15.190:0/3941911909":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6800/748923615":"2024-03-27T18:54:47.731502+0000","172.21.15.102:0/4276276941":"2024-03-27T18:47:33.839189+0000","172.21.15.102:6801/748923615":"2024-03-27T18:54:47.731502+0000","172.21.15.102:6800/2831156761":"2024-03-27T18:56:37.181768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T19:06:34.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.210+0000 7fae42881640 1 -- 172.21.15.102:0/1130107278 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae2406e490 msgr2=0x7fae24070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.210+0000 7fae42881640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae2406e490 0x7fae24070940 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0058c0 tx=0x7fae2c005850 comp rx=0 tx=0).stop 2024-03-26T19:06:34.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.210+0000 7fae42881640 1 -- 172.21.15.102:0/1130107278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3c072eb0 msgr2=0x7fae3c197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.210+0000 7fae42881640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3c072eb0 0x7fae3c197970 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fae30009230 tx=0x7fae30004040 comp rx=0 tx=0).stop 2024-03-26T19:06:34.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.211+0000 7fae42881640 1 -- 172.21.15.102:0/1130107278 shutdown_connections 2024-03-26T19:06:34.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.211+0000 7fae42881640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3c072eb0 0x7fae3c197970 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.211+0000 7fae42881640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fae2406e490 0x7fae24070940 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.211+0000 7fae42881640 1 --2- 172.21.15.102:0/1130107278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae3c0725a0 0x7fae3c197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.211+0000 7fae42881640 1 -- 172.21.15.102:0/1130107278 >> 172.21.15.102:0/1130107278 conn(0x7fae3c0fb140 msgr2=0x7fae3c106200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:34.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.211+0000 7fae42881640 1 -- 172.21.15.102:0/1130107278 shutdown_connections 2024-03-26T19:06:34.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.211+0000 7fae42881640 1 -- 172.21.15.102:0/1130107278 wait complete. 2024-03-26T19:06:34.219 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T18:49:50.172501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-26T18:52:17.727534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-26T18:52:18.482241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T19:06:34.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T19:06:34.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.327+0000 7fcb48167640 1 -- 172.21.15.102:0/29546263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb40067350 msgr2=0x7fcb400677c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.327+0000 7fcb48167640 1 --2- 172.21.15.102:0/29546263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb40067350 0x7fcb400677c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009880 tx=0x7fcb3002bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.327+0000 7fcb48167640 1 -- 172.21.15.102:0/29546263 shutdown_connections 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.327+0000 7fcb48167640 1 --2- 172.21.15.102:0/29546263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb40067350 0x7fcb400677c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.327+0000 7fcb48167640 1 --2- 172.21.15.102:0/29546263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb40105690 0x7fcb40105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.327+0000 7fcb48167640 1 -- 172.21.15.102:0/29546263 >> 172.21.15.102:0/29546263 conn(0x7fcb400743d0 msgr2=0x7fcb400747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.327+0000 7fcb48167640 1 -- 172.21.15.102:0/29546263 shutdown_connections 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.327+0000 7fcb48167640 1 -- 172.21.15.102:0/29546263 wait complete. 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb48167640 1 Processor -- start 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb48167640 1 -- start start 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb48167640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb40067350 0x7fcb401952b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb48167640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb40105690 0x7fcb401957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb48167640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb40195e80 con 0x7fcb40067350 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb48167640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb40199c50 con 0x7fcb40105690 2024-03-26T19:06:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb45f03640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb40067350 0x7fcb401952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb45f03640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb40067350 0x7fcb401952b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57148/0 (socket says 172.21.15.102:57148) 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb45f03640 1 -- 172.21.15.102:0/3904487917 learned_addr learned my addr 172.21.15.102:0/3904487917 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb45702640 1 --2- 172.21.15.102:0/3904487917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb40105690 0x7fcb401957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb45f03640 1 -- 172.21.15.102:0/3904487917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb40105690 msgr2=0x7fcb401957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb45f03640 1 --2- 172.21.15.102:0/3904487917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb40105690 0x7fcb401957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.328+0000 7fcb45f03640 1 -- 172.21.15.102:0/3904487917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30009560 con 0x7fcb40067350 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.329+0000 7fcb45f03640 1 --2- 172.21.15.102:0/3904487917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb40067350 0x7fcb401952b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00e970 tx=0x7fcb3c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.329+0000 7fcb36ffd640 1 -- 172.21.15.102:0/3904487917 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb3c00cc90 con 0x7fcb40067350 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.329+0000 7fcb36ffd640 1 -- 172.21.15.102:0/3904487917 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb3c00cdf0 con 0x7fcb40067350 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.329+0000 7fcb48167640 1 -- 172.21.15.102:0/3904487917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb40199f30 con 0x7fcb40067350 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.329+0000 7fcb36ffd640 1 -- 172.21.15.102:0/3904487917 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb3c010430 con 0x7fcb40067350 2024-03-26T19:06:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.329+0000 7fcb48167640 1 -- 172.21.15.102:0/3904487917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4019a4e0 con 0x7fcb40067350 2024-03-26T19:06:34.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.330+0000 7fcb48167640 1 -- 172.21.15.102:0/3904487917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4004e000 con 0x7fcb40067350 2024-03-26T19:06:34.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.330+0000 7fcb36ffd640 1 -- 172.21.15.102:0/3904487917 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fcb3c0046c0 con 0x7fcb40067350 2024-03-26T19:06:34.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.331+0000 7fcb36ffd640 1 --2- 172.21.15.102:0/3904487917 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcb2c06e3d0 0x7fcb2c070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:34.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.331+0000 7fcb45702640 1 --2- 172.21.15.102:0/3904487917 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcb2c06e3d0 0x7fcb2c070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:34.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.331+0000 7fcb36ffd640 1 -- 172.21.15.102:0/3904487917 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fcb3c014070 con 0x7fcb40067350 2024-03-26T19:06:34.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.332+0000 7fcb45702640 1 --2- 172.21.15.102:0/3904487917 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcb2c06e3d0 0x7fcb2c070880 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcb30006010 tx=0x7fcb30005dd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:34.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.336+0000 7fcb36ffd640 1 -- 172.21.15.102:0/3904487917 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcb3c057b00 con 0x7fcb40067350 2024-03-26T19:06:34.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.533+0000 7fcb48167640 1 -- 172.21.15.102:0/3904487917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcb4010ca90 con 0x7fcb40067350 2024-03-26T19:06:34.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.533+0000 7fcb36ffd640 1 -- 172.21.15.102:0/3904487917 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcb3c05b1c0 con 0x7fcb40067350 2024-03-26T19:06:34.531 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-26T19:06:34.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.535+0000 7fcb48167640 1 -- 172.21.15.102:0/3904487917 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcb2c06e3d0 msgr2=0x7fcb2c070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.535+0000 7fcb48167640 1 --2- 172.21.15.102:0/3904487917 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcb2c06e3d0 0x7fcb2c070880 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcb30006010 tx=0x7fcb30005dd0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.535+0000 7fcb48167640 1 -- 172.21.15.102:0/3904487917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb40067350 msgr2=0x7fcb401952b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.535+0000 7fcb48167640 1 --2- 172.21.15.102:0/3904487917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb40067350 0x7fcb401952b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00e970 tx=0x7fcb3c00ed70 comp rx=0 tx=0).stop 2024-03-26T19:06:34.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.535+0000 7fcb48167640 1 -- 172.21.15.102:0/3904487917 shutdown_connections 2024-03-26T19:06:34.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.535+0000 7fcb48167640 1 --2- 172.21.15.102:0/3904487917 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fcb2c06e3d0 0x7fcb2c070880 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.535+0000 7fcb48167640 1 --2- 172.21.15.102:0/3904487917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb40105690 0x7fcb401957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.535+0000 7fcb48167640 1 --2- 172.21.15.102:0/3904487917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb40067350 0x7fcb401952b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.535+0000 7fcb48167640 1 -- 172.21.15.102:0/3904487917 >> 172.21.15.102:0/3904487917 conn(0x7fcb400743d0 msgr2=0x7fcb400fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:34.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.536+0000 7fcb48167640 1 -- 172.21.15.102:0/3904487917 shutdown_connections 2024-03-26T19:06:34.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.536+0000 7fcb48167640 1 -- 172.21.15.102:0/3904487917 wait complete. 2024-03-26T19:06:34.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:34 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3471996638' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-03-26T19:06:34.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:34 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1461508339' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 42, "format": "json"}]: dispatch 2024-03-26T19:06:34.542 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:34 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3471996638' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-03-26T19:06:34.542 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:34 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1461508339' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 42, "format": "json"}]: dispatch 2024-03-26T19:06:34.544 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-26T19:06:34.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.654+0000 7fe551677640 1 -- 172.21.15.102:0/2560102810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54c0fe5b0 msgr2=0x7fe54c0fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.654+0000 7fe551677640 1 --2- 172.21.15.102:0/2560102810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54c0fe5b0 0x7fe54c0fea20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe534009880 tx=0x7fe53402bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.654+0000 7fe551677640 1 -- 172.21.15.102:0/2560102810 shutdown_connections 2024-03-26T19:06:34.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.654+0000 7fe551677640 1 --2- 172.21.15.102:0/2560102810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54c0fe5b0 0x7fe54c0fea20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.654+0000 7fe551677640 1 --2- 172.21.15.102:0/2560102810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54c104590 0x7fe54c104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.654+0000 7fe551677640 1 -- 172.21.15.102:0/2560102810 >> 172.21.15.102:0/2560102810 conn(0x7fe54c0743d0 msgr2=0x7fe54c0747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:34.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.654+0000 7fe551677640 1 -- 172.21.15.102:0/2560102810 shutdown_connections 2024-03-26T19:06:34.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.654+0000 7fe551677640 1 -- 172.21.15.102:0/2560102810 wait complete. 2024-03-26T19:06:34.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.654+0000 7fe551677640 1 Processor -- start 2024-03-26T19:06:34.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe551677640 1 -- start start 2024-03-26T19:06:34.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe551677640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54c0fe5b0 0x7fe54c196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:34.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe551677640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54c104590 0x7fe54c196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:34.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe551677640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c196c20 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe551677640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c196d90 con 0x7fe54c104590 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe54affd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54c0fe5b0 0x7fe54c196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe54affd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54c0fe5b0 0x7fe54c196110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57178/0 (socket says 172.21.15.102:57178) 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe54affd640 1 -- 172.21.15.102:0/2959722365 learned_addr learned my addr 172.21.15.102:0/2959722365 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe5427fc640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54c104590 0x7fe54c196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe54affd640 1 -- 172.21.15.102:0/2959722365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54c104590 msgr2=0x7fe54c196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe54affd640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54c104590 0x7fe54c196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe54affd640 1 -- 172.21.15.102:0/2959722365 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe534009560 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.655+0000 7fe5427fc640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54c104590 0x7fe54c196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.656+0000 7fe54affd640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54c0fe5b0 0x7fe54c196110 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00d6f0 tx=0x7fe53c00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.656+0000 7fe548ff9640 1 -- 172.21.15.102:0/2959722365 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe53c009a30 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.656+0000 7fe551677640 1 -- 172.21.15.102:0/2959722365 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c19f180 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.656+0000 7fe548ff9640 1 -- 172.21.15.102:0/2959722365 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe53c0054e0 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.656+0000 7fe548ff9640 1 -- 172.21.15.102:0/2959722365 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe53c010460 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.656+0000 7fe551677640 1 -- 172.21.15.102:0/2959722365 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c19f6b0 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.657+0000 7fe551677640 1 -- 172.21.15.102:0/2959722365 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c04e000 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.658+0000 7fe548ff9640 1 -- 172.21.15.102:0/2959722365 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe53c009b90 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.658+0000 7fe548ff9640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe53806e3d0 0x7fe538070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:34.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.658+0000 7fe5427fc640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe53806e3d0 0x7fe538070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:34.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.658+0000 7fe548ff9640 1 -- 172.21.15.102:0/2959722365 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fe53c08e000 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.659+0000 7fe5427fc640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe53806e3d0 0x7fe538070880 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe534005ec0 tx=0x7fe534005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:34.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.667+0000 7fe548ff9640 1 -- 172.21.15.102:0/2959722365 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe53c057160 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.938+0000 7fe551677640 1 -- 172.21.15.102:0/2959722365 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fe54c067900 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.938+0000 7fe548ff9640 1 -- 172.21.15.102:0/2959722365 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe54c067900 con 0x7fe54c0fe5b0 2024-03-26T19:06:34.936 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-26T19:06:34.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.941+0000 7fe551677640 1 -- 172.21.15.102:0/2959722365 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe53806e3d0 msgr2=0x7fe538070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.941+0000 7fe551677640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe53806e3d0 0x7fe538070880 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe534005ec0 tx=0x7fe534005fd0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.941+0000 7fe551677640 1 -- 172.21.15.102:0/2959722365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54c0fe5b0 msgr2=0x7fe54c196110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:34.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.941+0000 7fe551677640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54c0fe5b0 0x7fe54c196110 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00d6f0 tx=0x7fe53c00dbb0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.941+0000 7fe551677640 1 -- 172.21.15.102:0/2959722365 shutdown_connections 2024-03-26T19:06:34.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.941+0000 7fe551677640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fe53806e3d0 0x7fe538070880 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.941+0000 7fe551677640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54c104590 0x7fe54c196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.941+0000 7fe551677640 1 --2- 172.21.15.102:0/2959722365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54c0fe5b0 0x7fe54c196110 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:34.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.941+0000 7fe551677640 1 -- 172.21.15.102:0/2959722365 >> 172.21.15.102:0/2959722365 conn(0x7fe54c0743d0 msgr2=0x7fe54c0fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:34.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.941+0000 7fe551677640 1 -- 172.21.15.102:0/2959722365 shutdown_connections 2024-03-26T19:06:34.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:34.941+0000 7fe551677640 1 -- 172.21.15.102:0/2959722365 wait complete. 2024-03-26T19:06:34.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-26T19:06:35.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.066+0000 7f13bc24a640 1 -- 172.21.15.102:0/2588524538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b4104590 msgr2=0x7f13b4104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:35.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.066+0000 7f13bc24a640 1 --2- 172.21.15.102:0/2588524538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b4104590 0x7f13b4104960 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f13b0009880 tx=0x7f13b002bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.066+0000 7f13bc24a640 1 -- 172.21.15.102:0/2588524538 shutdown_connections 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.066+0000 7f13bc24a640 1 --2- 172.21.15.102:0/2588524538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b40fe5b0 0x7f13b40fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.066+0000 7f13bc24a640 1 --2- 172.21.15.102:0/2588524538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b4104590 0x7f13b4104960 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.066+0000 7f13bc24a640 1 -- 172.21.15.102:0/2588524538 >> 172.21.15.102:0/2588524538 conn(0x7f13b40743d0 msgr2=0x7f13b40747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.066+0000 7f13bc24a640 1 -- 172.21.15.102:0/2588524538 shutdown_connections 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.066+0000 7f13bc24a640 1 -- 172.21.15.102:0/2588524538 wait complete. 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13bc24a640 1 Processor -- start 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13bc24a640 1 -- start start 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13bc24a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b40fe5b0 0x7f13b41974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13bc24a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b4104590 0x7f13b41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13bc24a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4197fe0 con 0x7f13b4104590 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13bc24a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4198150 con 0x7f13b40fe5b0 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13b97e5640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b4104590 0x7f13b41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:35.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13b97e5640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b4104590 0x7f13b41979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57192/0 (socket says 172.21.15.102:57192) 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13b97e5640 1 -- 172.21.15.102:0/479816653 learned_addr learned my addr 172.21.15.102:0/479816653 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13b9fe6640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b40fe5b0 0x7f13b41974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13b97e5640 1 -- 172.21.15.102:0/479816653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b40fe5b0 msgr2=0x7f13b41974a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13b97e5640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b40fe5b0 0x7f13b41974a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13b97e5640 1 -- 172.21.15.102:0/479816653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b0009560 con 0x7f13b4104590 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.067+0000 7f13b9fe6640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b40fe5b0 0x7f13b41974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.068+0000 7f13b97e5640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b4104590 0x7f13b41979e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f13a400c8a0 tx=0x7f13a400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.068+0000 7f13aaffd640 1 -- 172.21.15.102:0/479816653 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13a4007b80 con 0x7f13b4104590 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.068+0000 7f13bc24a640 1 -- 172.21.15.102:0/479816653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b419bfe0 con 0x7f13b4104590 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.068+0000 7f13aaffd640 1 -- 172.21.15.102:0/479816653 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13a4007ce0 con 0x7f13b4104590 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.068+0000 7f13aaffd640 1 -- 172.21.15.102:0/479816653 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13a4018810 con 0x7f13b4104590 2024-03-26T19:06:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.068+0000 7f13bc24a640 1 -- 172.21.15.102:0/479816653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b419c5c0 con 0x7f13b4104590 2024-03-26T19:06:35.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.069+0000 7f13bc24a640 1 -- 172.21.15.102:0/479816653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b404e000 con 0x7f13b4104590 2024-03-26T19:06:35.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.069+0000 7f13aaffd640 1 -- 172.21.15.102:0/479816653 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f13a4018a80 con 0x7f13b4104590 2024-03-26T19:06:35.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.070+0000 7f13aaffd640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f13a006e3d0 0x7f13a0070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:35.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.070+0000 7f13aaffd640 1 -- 172.21.15.102:0/479816653 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f13a408e6a0 con 0x7f13b4104590 2024-03-26T19:06:35.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.070+0000 7f13b9fe6640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f13a006e3d0 0x7f13a0070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:35.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.070+0000 7f13b9fe6640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f13a006e3d0 0x7f13a0070880 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f13b0000c00 tx=0x7f13b002cc20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:35.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.074+0000 7f13aaffd640 1 -- 172.21.15.102:0/479816653 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f13a4057800 con 0x7f13b4104590 2024-03-26T19:06:35.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.281+0000 7f13bc24a640 1 -- 172.21.15.102:0/479816653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f13b41987f0 con 0x7f13b4104590 2024-03-26T19:06:35.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.281+0000 7f13aaffd640 1 -- 172.21.15.102:0/479816653 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f13a405aec0 con 0x7f13b4104590 2024-03-26T19:06:35.279 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-26T19:06:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.283+0000 7f13bc24a640 1 -- 172.21.15.102:0/479816653 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f13a006e3d0 msgr2=0x7f13a0070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.283+0000 7f13bc24a640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f13a006e3d0 0x7f13a0070880 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f13b0000c00 tx=0x7f13b002cc20 comp rx=0 tx=0).stop 2024-03-26T19:06:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.283+0000 7f13bc24a640 1 -- 172.21.15.102:0/479816653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b4104590 msgr2=0x7f13b41979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.283+0000 7f13bc24a640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b4104590 0x7f13b41979e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f13a400c8a0 tx=0x7f13a400cd60 comp rx=0 tx=0).stop 2024-03-26T19:06:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.283+0000 7f13bc24a640 1 -- 172.21.15.102:0/479816653 shutdown_connections 2024-03-26T19:06:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.283+0000 7f13bc24a640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b4104590 0x7f13b41979e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.283+0000 7f13bc24a640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f13a006e3d0 0x7f13a0070880 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.283+0000 7f13bc24a640 1 --2- 172.21.15.102:0/479816653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b40fe5b0 0x7f13b41974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.283+0000 7f13bc24a640 1 -- 172.21.15.102:0/479816653 >> 172.21.15.102:0/479816653 conn(0x7f13b40743d0 msgr2=0x7f13b40fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:35.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.284+0000 7f13bc24a640 1 -- 172.21.15.102:0/479816653 shutdown_connections 2024-03-26T19:06:35.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.284+0000 7f13bc24a640 1 -- 172.21.15.102:0/479816653 wait complete. 2024-03-26T19:06:35.290 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:35 smithi102 ceph-mon[78322]: pgmap v335: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:35.291 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:35 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/1130107278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T19:06:35.291 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:35 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3904487917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T19:06:35.291 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:35 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2959722365' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-26T19:06:35.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-26T19:06:35.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.399+0000 7fa186559640 1 -- 172.21.15.102:0/2736724807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1800fe450 msgr2=0x7fa1800fe820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:35.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.399+0000 7fa186559640 1 --2- 172.21.15.102:0/2736724807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1800fe450 0x7fa1800fe820 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa170009880 tx=0x7fa17002bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.399+0000 7fa186559640 1 -- 172.21.15.102:0/2736724807 shutdown_connections 2024-03-26T19:06:35.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.399+0000 7fa186559640 1 --2- 172.21.15.102:0/2736724807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1800fed60 0x7fa18010bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.399+0000 7fa186559640 1 --2- 172.21.15.102:0/2736724807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1800fe450 0x7fa1800fe820 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.399+0000 7fa186559640 1 -- 172.21.15.102:0/2736724807 >> 172.21.15.102:0/2736724807 conn(0x7fa180075270 msgr2=0x7fa180075670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:35.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.399+0000 7fa186559640 1 -- 172.21.15.102:0/2736724807 shutdown_connections 2024-03-26T19:06:35.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.399+0000 7fa186559640 1 -- 172.21.15.102:0/2736724807 wait complete. 2024-03-26T19:06:35.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.399+0000 7fa186559640 1 Processor -- start 2024-03-26T19:06:35.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.399+0000 7fa186559640 1 -- start start 2024-03-26T19:06:35.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa186559640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1800fe450 0x7fa180197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:35.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa186559640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1800fed60 0x7fa1801978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:35.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa186559640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa180197f30 con 0x7fa1800fed60 2024-03-26T19:06:35.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa186559640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18019bd00 con 0x7fa1800fe450 2024-03-26T19:06:35.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa17f7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1800fed60 0x7fa1801978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa17f7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1800fed60 0x7fa1801978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57212/0 (socket says 172.21.15.102:57212) 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa17f7fe640 1 -- 172.21.15.102:0/2235883700 learned_addr learned my addr 172.21.15.102:0/2235883700 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa17ffff640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1800fe450 0x7fa180197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa17f7fe640 1 -- 172.21.15.102:0/2235883700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1800fe450 msgr2=0x7fa180197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa17f7fe640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1800fe450 0x7fa180197360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa17f7fe640 1 -- 172.21.15.102:0/2235883700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa170009560 con 0x7fa1800fed60 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa17ffff640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1800fe450 0x7fa180197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa17f7fe640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1800fed60 0x7fa1801978a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa16800e970 tx=0x7fa16800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.400+0000 7fa17d7fa640 1 -- 172.21.15.102:0/2235883700 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa16800cc90 con 0x7fa1800fed60 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.401+0000 7fa17d7fa640 1 -- 172.21.15.102:0/2235883700 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa16800cdf0 con 0x7fa1800fed60 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.401+0000 7fa186559640 1 -- 172.21.15.102:0/2235883700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa18019bfe0 con 0x7fa1800fed60 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.401+0000 7fa17d7fa640 1 -- 172.21.15.102:0/2235883700 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa168010430 con 0x7fa1800fed60 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.401+0000 7fa186559640 1 -- 172.21.15.102:0/2235883700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18019c540 con 0x7fa1800fed60 2024-03-26T19:06:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.402+0000 7fa186559640 1 -- 172.21.15.102:0/2235883700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18004e000 con 0x7fa1800fed60 2024-03-26T19:06:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.402+0000 7fa17d7fa640 1 -- 172.21.15.102:0/2235883700 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa1680046c0 con 0x7fa1800fed60 2024-03-26T19:06:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.403+0000 7fa17d7fa640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa16c06e490 0x7fa16c070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.403+0000 7fa17ffff640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa16c06e490 0x7fa16c070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.403+0000 7fa17d7fa640 1 -- 172.21.15.102:0/2235883700 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fa168014070 con 0x7fa1800fed60 2024-03-26T19:06:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.403+0000 7fa17ffff640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa16c06e490 0x7fa16c070940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa170036040 tx=0x7fa170000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.407+0000 7fa17d7fa640 1 -- 172.21.15.102:0/2235883700 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa168057b80 con 0x7fa1800fed60 2024-03-26T19:06:35.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:35 smithi190 ceph-mon[65689]: pgmap v335: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:35.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:35 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/1130107278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T19:06:35.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:35 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3904487917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T19:06:35.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:35 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2959722365' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-26T19:06:35.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.655+0000 7fa186559640 1 -- 172.21.15.102:0/2235883700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa18010e9c0 con 0x7fa1800fed60 2024-03-26T19:06:35.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.656+0000 7fa17d7fa640 1 -- 172.21.15.102:0/2235883700 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 43 entries v84) v1 ==== 81+0+2579 (secure 0 0 0) 0x7fa16805b240 con 0x7fa1800fed60 2024-03-26T19:06:35.653 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:6827/624694186 2024-03-27T19:02:46.321671+0000 2024-03-26T19:06:35.653 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:6826/624694186 2024-03-27T19:02:46.321671+0000 2024-03-26T19:06:35.653 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6829/989894576 2024-03-27T19:02:06.598591+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6828/989894576 2024-03-27T19:02:06.598591+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2151777534 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1215990907 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2396893787 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/1874485118 2024-03-27T18:46:28.035077+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/3353740154 2024-03-27T18:47:33.839189+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1994330698 2024-03-27T18:46:28.035077+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6827/2550866692 2024-03-27T18:52:35.022250+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1842664791 2024-03-27T18:47:33.839189+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/308643739 2024-03-27T18:45:55.925005+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/2308588851 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:35.654 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1295801893 2024-03-27T18:45:55.925005+0000 2024-03-26T19:06:35.655 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/1874485118 2024-03-27T18:46:28.035077+0000 2024-03-26T19:06:35.655 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6826/2550866692 2024-03-27T18:52:35.022250+0000 2024-03-26T19:06:35.655 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2094779473 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:35.655 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/300222988 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:35.655 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/3517043895 2024-03-27T18:47:33.839189+0000 2024-03-26T19:06:35.655 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2686253586 2024-03-27T18:45:55.925005+0000 2024-03-26T19:06:35.655 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1212290737 2024-03-27T18:46:28.035077+0000 2024-03-26T19:06:35.655 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2752725029 2024-03-27T18:46:28.035077+0000 2024-03-26T19:06:35.655 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2027932959 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:35.656 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/3113541506 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:35.656 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/308643739 2024-03-27T18:45:55.925005+0000 2024-03-26T19:06:35.656 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/216723823 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:35.656 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:6828/2858265260 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:35.656 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:6829/2858265260 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:35.656 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/3517043895 2024-03-27T18:47:33.839189+0000 2024-03-26T19:06:35.656 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1380712530 2024-03-27T18:45:55.925005+0000 2024-03-26T19:06:35.656 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/990121767 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:35.656 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1480032448 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:35.657 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/1923628519 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:35.657 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/2831156761 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:35.657 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1818262291 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:35.657 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1615769300 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:35.657 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/833763291 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:35.657 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/3941911909 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:35.657 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/748923615 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:35.657 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/4276276941 2024-03-27T18:47:33.839189+0000 2024-03-26T19:06:35.657 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/748923615 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:35.657 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/2831156761 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:listed 43 entries 2024-03-26T19:06:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.658+0000 7fa186559640 1 -- 172.21.15.102:0/2235883700 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa16c06e490 msgr2=0x7fa16c070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.658+0000 7fa186559640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa16c06e490 0x7fa16c070940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa170036040 tx=0x7fa170000bc0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.658+0000 7fa186559640 1 -- 172.21.15.102:0/2235883700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1800fed60 msgr2=0x7fa1801978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.658+0000 7fa186559640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1800fed60 0x7fa1801978a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa16800e970 tx=0x7fa16800ed70 comp rx=0 tx=0).stop 2024-03-26T19:06:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.658+0000 7fa186559640 1 -- 172.21.15.102:0/2235883700 shutdown_connections 2024-03-26T19:06:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.658+0000 7fa186559640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1800fed60 0x7fa1801978a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.658+0000 7fa186559640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa16c06e490 0x7fa16c070940 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.658+0000 7fa186559640 1 --2- 172.21.15.102:0/2235883700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1800fe450 0x7fa180197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.658+0000 7fa186559640 1 -- 172.21.15.102:0/2235883700 >> 172.21.15.102:0/2235883700 conn(0x7fa180075270 msgr2=0x7fa1800fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.658+0000 7fa186559640 1 -- 172.21.15.102:0/2235883700 shutdown_connections 2024-03-26T19:06:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.658+0000 7fa186559640 1 -- 172.21.15.102:0/2235883700 wait complete. 2024-03-26T19:06:35.669 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-26T19:06:35.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T19:06:35.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-26T19:06:35.699 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T19:06:35.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T19:06:35.699 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-26T19:06:35.763 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T19:06:35.763 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-26T19:06:35.782 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-26T19:06:35.782 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-26T19:06:35.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T19:06:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.964+0000 7f8f8086a640 1 -- 172.21.15.102:0/1945185567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c1045d0 msgr2=0x7f8f7c1049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.964+0000 7f8f8086a640 1 --2- 172.21.15.102:0/1945185567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c1045d0 0x7f8f7c1049a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009880 tx=0x7f8f6c02bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.964+0000 7f8f8086a640 1 -- 172.21.15.102:0/1945185567 shutdown_connections 2024-03-26T19:06:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.964+0000 7f8f8086a640 1 --2- 172.21.15.102:0/1945185567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f7c0fe5a0 0x7f8f7c0fea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.964+0000 7f8f8086a640 1 --2- 172.21.15.102:0/1945185567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c1045d0 0x7f8f7c1049a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.964+0000 7f8f8086a640 1 -- 172.21.15.102:0/1945185567 >> 172.21.15.102:0/1945185567 conn(0x7f8f7c075270 msgr2=0x7f8f7c075670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.964+0000 7f8f8086a640 1 -- 172.21.15.102:0/1945185567 shutdown_connections 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.964+0000 7f8f8086a640 1 -- 172.21.15.102:0/1945185567 wait complete. 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f8086a640 1 Processor -- start 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f8086a640 1 -- start start 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f8086a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f7c0fe5a0 0x7f8f7c1940d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f8086a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c1045d0 0x7f8f7c194610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f8086a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c194ca0 con 0x7f8f7c1045d0 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f8086a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c198a70 con 0x7f8f7c0fe5a0 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f79d9b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c1045d0 0x7f8f7c194610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f79d9b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c1045d0 0x7f8f7c194610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57226/0 (socket says 172.21.15.102:57226) 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f79d9b640 1 -- 172.21.15.102:0/3753980865 learned_addr learned my addr 172.21.15.102:0/3753980865 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f7a59c640 1 --2- 172.21.15.102:0/3753980865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f7c0fe5a0 0x7f8f7c1940d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f79d9b640 1 -- 172.21.15.102:0/3753980865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f7c0fe5a0 msgr2=0x7f8f7c1940d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f79d9b640 1 --2- 172.21.15.102:0/3753980865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f7c0fe5a0 0x7f8f7c1940d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:35.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.965+0000 7f8f79d9b640 1 -- 172.21.15.102:0/3753980865 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c009560 con 0x7f8f7c1045d0 2024-03-26T19:06:35.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.966+0000 7f8f79d9b640 1 --2- 172.21.15.102:0/3753980865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c1045d0 0x7f8f7c194610 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000d6f0 tx=0x7f8f7000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:35.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.966+0000 7f8f6b7fe640 1 -- 172.21.15.102:0/3753980865 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f70009a30 con 0x7f8f7c1045d0 2024-03-26T19:06:35.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.966+0000 7f8f8086a640 1 -- 172.21.15.102:0/3753980865 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7c198d50 con 0x7f8f7c1045d0 2024-03-26T19:06:35.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.966+0000 7f8f6b7fe640 1 -- 172.21.15.102:0/3753980865 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f700054e0 con 0x7f8f7c1045d0 2024-03-26T19:06:35.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.966+0000 7f8f6b7fe640 1 -- 172.21.15.102:0/3753980865 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f70010460 con 0x7f8f7c1045d0 2024-03-26T19:06:35.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.966+0000 7f8f8086a640 1 -- 172.21.15.102:0/3753980865 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7c199330 con 0x7f8f7c1045d0 2024-03-26T19:06:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.967+0000 7f8f8086a640 1 -- 172.21.15.102:0/3753980865 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7c04e000 con 0x7f8f7c1045d0 2024-03-26T19:06:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.967+0000 7f8f6b7fe640 1 -- 172.21.15.102:0/3753980865 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8f70009b90 con 0x7f8f7c1045d0 2024-03-26T19:06:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.968+0000 7f8f6b7fe640 1 --2- 172.21.15.102:0/3753980865 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f6406e490 0x7f8f64070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.968+0000 7f8f7a59c640 1 --2- 172.21.15.102:0/3753980865 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f6406e490 0x7f8f64070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.968+0000 7f8f6b7fe640 1 -- 172.21.15.102:0/3753980865 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8f7008e000 con 0x7f8f7c1045d0 2024-03-26T19:06:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.968+0000 7f8f7a59c640 1 --2- 172.21.15.102:0/3753980865 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f6406e490 0x7f8f64070940 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c006010 tx=0x7f8f6c000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:35.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:35.973+0000 7f8f6b7fe640 1 -- 172.21.15.102:0/3753980865 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8f70057160 con 0x7f8f7c1045d0 2024-03-26T19:06:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.234+0000 7f8f8086a640 1 -- 172.21.15.102:0/3753980865 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f7c0008d0 con 0x7f8f7c1045d0 2024-03-26T19:06:36.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.235+0000 7f8f6b7fe640 1 -- 172.21.15.102:0/3753980865 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16965 (secure 0 0 0) 0x7f8f7005a820 con 0x7f8f7c1045d0 2024-03-26T19:06:36.233 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-26T19:06:36.233 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":84,"fsid":"ed4e027c-eba0-11ee-95d0-87774f69a715","created":"2024-03-26T18:45:25.446329+0000","modified":"2024-03-26T19:02:46.321700+0000","last_up_change":"2024-03-26T19:01:03.417320+0000","last_in_change":"2024-03-26T18:50:45.452094+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T18:49:50.172501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-26T18:52:17.727534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-26T18:52:18.482241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5458d389-eb88-4798-92bc-b442168b37f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6803","nonce":3021596332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6805","nonce":3021596332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6809","nonce":3021596332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":3021596332},{"type":"v1","addr":"172.21.15.102:6807","nonce":3021596332}]},"public_addr":"172.21.15.102:6803/3021596332","cluster_addr":"172.21.15.102:6805/3021596332","heartbeat_back_addr":"172.21.15.102:6809/3021596332","heartbeat_front_addr":"172.21.15.102:6807/3021596332","state":["exists","up"]},{"osd":1,"uuid":"87508f24-8b3c-4f6d-add2-00ad59300da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6811","nonce":3336954503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6813","nonce":3336954503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6817","nonce":3336954503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3336954503},{"type":"v1","addr":"172.21.15.102:6815","nonce":3336954503}]},"public_addr":"172.21.15.102:6811/3336954503","cluster_addr":"172.21.15.102:6813/3336954503","heartbeat_back_addr":"172.21.15.102:6817/3336954503","heartbeat_front_addr":"172.21.15.102:6815/3336954503","state":["exists","up"]},{"osd":2,"uuid":"c0179546-10c5-41f2-af0c-be93c2bbf5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6819","nonce":618947256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6821","nonce":618947256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6825","nonce":618947256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":618947256},{"type":"v1","addr":"172.21.15.102:6823","nonce":618947256}]},"public_addr":"172.21.15.102:6819/618947256","cluster_addr":"172.21.15.102:6821/618947256","heartbeat_back_addr":"172.21.15.102:6825/618947256","heartbeat_front_addr":"172.21.15.102:6823/618947256","state":["exists","up"]},{"osd":3,"uuid":"4d524f7b-8b31-469b-b0c2-b94079febaeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6801","nonce":2294582584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6803","nonce":2294582584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6807","nonce":2294582584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2294582584},{"type":"v1","addr":"172.21.15.190:6805","nonce":2294582584}]},"public_addr":"172.21.15.190:6801/2294582584","cluster_addr":"172.21.15.190:6803/2294582584","heartbeat_back_addr":"172.21.15.190:6807/2294582584","heartbeat_front_addr":"172.21.15.190:6805/2294582584","state":["exists","up"]},{"osd":4,"uuid":"865baef8-dd72-4449-b3a4-893801bc2958","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6809","nonce":2940868243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6811","nonce":2940868243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6815","nonce":2940868243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":2940868243},{"type":"v1","addr":"172.21.15.190:6813","nonce":2940868243}]},"public_addr":"172.21.15.190:6809/2940868243","cluster_addr":"172.21.15.190:6811/2940868243","heartbeat_back_addr":"172.21.15.190:6815/2940868243","heartbeat_front_addr":"172.21.15.190:6813/2940868243","state":["exists","up"]},{"osd":5,"uuid":"5fd67007-d7ae-43b6-8002-258fbfc5a797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6817","nonce":3222089930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6819","nonce":3222089930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6823","nonce":3222089930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":3222089930},{"type":"v1","addr":"172.21.15.190:6821","nonce":3222089930}]},"public_addr":"172.21.15.190:6817/3222089930","cluster_addr":"172.21.15.190:6819/3222089930","heartbeat_back_addr":"172.21.15.190:6823/3222089930","heartbeat_front_addr":"172.21.15.190:6821/3222089930","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:48:51.274164+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:18.020320+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:49:46.066766+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:10.365857+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:50:36.582495+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T18:51:02.797707+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6827/624694186":"2024-03-27T19:02:46.321671+0000","172.21.15.190:6826/624694186":"2024-03-27T19:02:46.321671+0000","172.21.15.102:6829/989894576":"2024-03-27T19:02:06.598591+0000","172.21.15.102:6828/989894576":"2024-03-27T19:02:06.598591+0000","172.21.15.102:0/2151777534":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1215990907":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/2396893787":"2024-03-27T18:56:37.181768+0000","172.21.15.102:6801/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/3353740154":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1994330698":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6827/2550866692":"2024-03-27T18:52:35.022250+0000","172.21.15.102:0/1842664791":"2024-03-27T18:47:33.839189+0000","172.21.15.102:6801/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.190:0/2308588851":"2024-03-27T18:55:42.356278+0000","172.21.15.102:0/1295801893":"2024-03-27T18:45:55.925005+0000","172.21.15.102:6800/1874485118":"2024-03-27T18:46:28.035077+0000","172.21.15.102:6826/2550866692":"2024-03-27T18:52:35.022250+0000","172.21.15.102:0/2094779473":"2024-03-27T18:54:47.731502+0000","172.21.15.190:0/300222988":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6801/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/2686253586":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/1212290737":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/2752725029":"2024-03-27T18:46:28.035077+0000","172.21.15.102:0/2027932959":"2024-03-27T18:56:37.181768+0000","172.21.15.190:0/3113541506":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6800/308643739":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/216723823":"2024-03-27T18:54:47.731502+0000","172.21.15.190:6828/2858265260":"2024-03-27T18:55:42.356278+0000","172.21.15.190:6829/2858265260":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6800/3517043895":"2024-03-27T18:47:33.839189+0000","172.21.15.102:0/1380712530":"2024-03-27T18:45:55.925005+0000","172.21.15.102:0/990121767":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1480032448":"2024-03-27T18:54:47.731502+0000","172.21.15.190:0/1923628519":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6801/2831156761":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1818262291":"2024-03-27T18:56:37.181768+0000","172.21.15.102:0/1615769300":"2024-03-27T18:54:47.731502+0000","172.21.15.190:0/833763291":"2024-03-27T18:55:42.356278+0000","172.21.15.190:0/3941911909":"2024-03-27T18:55:42.356278+0000","172.21.15.102:6800/748923615":"2024-03-27T18:54:47.731502+0000","172.21.15.102:0/4276276941":"2024-03-27T18:47:33.839189+0000","172.21.15.102:6801/748923615":"2024-03-27T18:54:47.731502+0000","172.21.15.102:6800/2831156761":"2024-03-27T18:56:37.181768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T19:06:36.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.238+0000 7f8f8086a640 1 -- 172.21.15.102:0/3753980865 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f6406e490 msgr2=0x7f8f64070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:36.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.238+0000 7f8f8086a640 1 --2- 172.21.15.102:0/3753980865 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f6406e490 0x7f8f64070940 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c006010 tx=0x7f8f6c000bc0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.238+0000 7f8f8086a640 1 -- 172.21.15.102:0/3753980865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c1045d0 msgr2=0x7f8f7c194610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.238+0000 7f8f8086a640 1 --2- 172.21.15.102:0/3753980865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c1045d0 0x7f8f7c194610 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000d6f0 tx=0x7f8f7000dbb0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.238+0000 7f8f8086a640 1 -- 172.21.15.102:0/3753980865 shutdown_connections 2024-03-26T19:06:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.238+0000 7f8f8086a640 1 --2- 172.21.15.102:0/3753980865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c1045d0 0x7f8f7c194610 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.238+0000 7f8f8086a640 1 --2- 172.21.15.102:0/3753980865 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f8f6406e490 0x7f8f64070940 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.238+0000 7f8f8086a640 1 --2- 172.21.15.102:0/3753980865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f7c0fe5a0 0x7f8f7c1940d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.238+0000 7f8f8086a640 1 -- 172.21.15.102:0/3753980865 >> 172.21.15.102:0/3753980865 conn(0x7f8f7c075270 msgr2=0x7f8f7c0fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.238+0000 7f8f8086a640 1 -- 172.21.15.102:0/3753980865 shutdown_connections 2024-03-26T19:06:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.238+0000 7f8f8086a640 1 -- 172.21.15.102:0/3753980865 wait complete. 2024-03-26T19:06:36.246 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T18:49:50.172501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-26T18:52:17.727534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-26T18:52:18.482241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T19:06:36.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T19:06:36.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.358+0000 7f2d839c0640 1 -- 172.21.15.102:0/4196431539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7c067350 msgr2=0x7f2d7c0677c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:36.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.358+0000 7f2d839c0640 1 --2- 172.21.15.102:0/4196431539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7c067350 0x7f2d7c0677c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009880 tx=0x7f2d6c02bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.358+0000 7f2d839c0640 1 -- 172.21.15.102:0/4196431539 shutdown_connections 2024-03-26T19:06:36.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.358+0000 7f2d839c0640 1 --2- 172.21.15.102:0/4196431539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7c067350 0x7f2d7c0677c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.358+0000 7f2d839c0640 1 --2- 172.21.15.102:0/4196431539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d7c105690 0x7f2d7c105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.358+0000 7f2d839c0640 1 -- 172.21.15.102:0/4196431539 >> 172.21.15.102:0/4196431539 conn(0x7f2d7c0743d0 msgr2=0x7f2d7c0747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:36.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.358+0000 7f2d839c0640 1 -- 172.21.15.102:0/4196431539 shutdown_connections 2024-03-26T19:06:36.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.358+0000 7f2d839c0640 1 -- 172.21.15.102:0/4196431539 wait complete. 2024-03-26T19:06:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.359+0000 7f2d839c0640 1 Processor -- start 2024-03-26T19:06:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.359+0000 7f2d839c0640 1 -- start start 2024-03-26T19:06:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.359+0000 7f2d839c0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7c067350 0x7f2d7c195380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.359+0000 7f2d839c0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d7c105690 0x7f2d7c1958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.359+0000 7f2d839c0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c195f50 con 0x7f2d7c067350 2024-03-26T19:06:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.359+0000 7f2d839c0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c199d20 con 0x7f2d7c105690 2024-03-26T19:06:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d8175c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7c067350 0x7f2d7c195380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d8175c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7c067350 0x7f2d7c195380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57236/0 (socket says 172.21.15.102:57236) 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d8175c640 1 -- 172.21.15.102:0/2304406411 learned_addr learned my addr 172.21.15.102:0/2304406411 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d80f5b640 1 --2- 172.21.15.102:0/2304406411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d7c105690 0x7f2d7c1958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d8175c640 1 -- 172.21.15.102:0/2304406411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d7c105690 msgr2=0x7f2d7c1958c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d8175c640 1 --2- 172.21.15.102:0/2304406411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d7c105690 0x7f2d7c1958c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d8175c640 1 -- 172.21.15.102:0/2304406411 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c009560 con 0x7f2d7c067350 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d8175c640 1 --2- 172.21.15.102:0/2304406411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7c067350 0x7f2d7c195380 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800b420 tx=0x7f2d7800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d727fc640 1 -- 172.21.15.102:0/2304406411 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d78011800 con 0x7f2d7c067350 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d839c0640 1 -- 172.21.15.102:0/2304406411 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c19a000 con 0x7f2d7c067350 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d727fc640 1 -- 172.21.15.102:0/2304406411 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d78011e20 con 0x7f2d7c067350 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d727fc640 1 -- 172.21.15.102:0/2304406411 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d7800f2e0 con 0x7f2d7c067350 2024-03-26T19:06:36.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.360+0000 7f2d839c0640 1 -- 172.21.15.102:0/2304406411 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c19a5b0 con 0x7f2d7c067350 2024-03-26T19:06:36.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.362+0000 7f2d839c0640 1 -- 172.21.15.102:0/2304406411 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c04e000 con 0x7f2d7c067350 2024-03-26T19:06:36.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.362+0000 7f2d727fc640 1 -- 172.21.15.102:0/2304406411 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2d78010380 con 0x7f2d7c067350 2024-03-26T19:06:36.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.363+0000 7f2d727fc640 1 --2- 172.21.15.102:0/2304406411 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2d6806e420 0x7f2d680708d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:36.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.363+0000 7f2d727fc640 1 -- 172.21.15.102:0/2304406411 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f2d7808dd90 con 0x7f2d7c067350 2024-03-26T19:06:36.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.363+0000 7f2d80f5b640 1 --2- 172.21.15.102:0/2304406411 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2d6806e420 0x7f2d680708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:36.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.367+0000 7f2d80f5b640 1 --2- 172.21.15.102:0/2304406411 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2d6806e420 0x7f2d680708d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c006010 tx=0x7f2d6c005b50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:36.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.372+0000 7f2d727fc640 1 -- 172.21.15.102:0/2304406411 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2d78056e70 con 0x7f2d7c067350 2024-03-26T19:06:36.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:36 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/479816653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-26T19:06:36.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:36 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2235883700' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T19:06:36.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.629+0000 7f2d839c0640 1 -- 172.21.15.102:0/2304406411 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d7c10c880 con 0x7f2d7c067350 2024-03-26T19:06:36.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.629+0000 7f2d727fc640 1 -- 172.21.15.102:0/2304406411 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d7805a530 con 0x7f2d7c067350 2024-03-26T19:06:36.627 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-26T19:06:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.631+0000 7f2d839c0640 1 -- 172.21.15.102:0/2304406411 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2d6806e420 msgr2=0x7f2d680708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.631+0000 7f2d839c0640 1 --2- 172.21.15.102:0/2304406411 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2d6806e420 0x7f2d680708d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c006010 tx=0x7f2d6c005b50 comp rx=0 tx=0).stop 2024-03-26T19:06:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.631+0000 7f2d839c0640 1 -- 172.21.15.102:0/2304406411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7c067350 msgr2=0x7f2d7c195380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.631+0000 7f2d839c0640 1 --2- 172.21.15.102:0/2304406411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7c067350 0x7f2d7c195380 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800b420 tx=0x7f2d7800b8e0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.631+0000 7f2d839c0640 1 -- 172.21.15.102:0/2304406411 shutdown_connections 2024-03-26T19:06:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.631+0000 7f2d839c0640 1 --2- 172.21.15.102:0/2304406411 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f2d6806e420 0x7f2d680708d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.631+0000 7f2d839c0640 1 --2- 172.21.15.102:0/2304406411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d7c105690 0x7f2d7c1958c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.631+0000 7f2d839c0640 1 --2- 172.21.15.102:0/2304406411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7c067350 0x7f2d7c195380 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.631+0000 7f2d839c0640 1 -- 172.21.15.102:0/2304406411 >> 172.21.15.102:0/2304406411 conn(0x7f2d7c0743d0 msgr2=0x7f2d7c0fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.631+0000 7f2d839c0640 1 -- 172.21.15.102:0/2304406411 shutdown_connections 2024-03-26T19:06:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.632+0000 7f2d839c0640 1 -- 172.21.15.102:0/2304406411 wait complete. 2024-03-26T19:06:36.630 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:36 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/479816653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-26T19:06:36.630 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:36 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2235883700' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T19:06:36.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-26T19:06:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.747+0000 7f47b02e2640 1 -- 172.21.15.102:0/2731702817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a8104590 msgr2=0x7f47a8104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.747+0000 7f47b02e2640 1 --2- 172.21.15.102:0/2731702817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a8104590 0x7f47a8104960 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f47a4009880 tx=0x7f47a402bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.747+0000 7f47b02e2640 1 -- 172.21.15.102:0/2731702817 shutdown_connections 2024-03-26T19:06:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.747+0000 7f47b02e2640 1 --2- 172.21.15.102:0/2731702817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47a80fe5b0 0x7f47a80fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.747+0000 7f47b02e2640 1 --2- 172.21.15.102:0/2731702817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a8104590 0x7f47a8104960 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.747+0000 7f47b02e2640 1 -- 172.21.15.102:0/2731702817 >> 172.21.15.102:0/2731702817 conn(0x7f47a80743d0 msgr2=0x7f47a80747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.747+0000 7f47b02e2640 1 -- 172.21.15.102:0/2731702817 shutdown_connections 2024-03-26T19:06:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.747+0000 7f47b02e2640 1 -- 172.21.15.102:0/2731702817 wait complete. 2024-03-26T19:06:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.747+0000 7f47b02e2640 1 Processor -- start 2024-03-26T19:06:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.747+0000 7f47b02e2640 1 -- start start 2024-03-26T19:06:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47b02e2640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a80fe5b0 0x7f47a8197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47b02e2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47a8104590 0x7f47a8197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47b02e2640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a8198050 con 0x7f47a80fe5b0 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47b02e2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a81981c0 con 0x7f47a8104590 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47ae07e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a80fe5b0 0x7f47a8197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47ae07e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a80fe5b0 0x7f47a8197510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57266/0 (socket says 172.21.15.102:57266) 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47ae07e640 1 -- 172.21.15.102:0/3488182756 learned_addr learned my addr 172.21.15.102:0/3488182756 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47ad87d640 1 --2- 172.21.15.102:0/3488182756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47a8104590 0x7f47a8197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47ae07e640 1 -- 172.21.15.102:0/3488182756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47a8104590 msgr2=0x7f47a8197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47ae07e640 1 --2- 172.21.15.102:0/3488182756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47a8104590 0x7f47a8197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47ae07e640 1 -- 172.21.15.102:0/3488182756 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a4009560 con 0x7f47a80fe5b0 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47ae07e640 1 --2- 172.21.15.102:0/3488182756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a80fe5b0 0x7f47a8197510 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f47a4009230 tx=0x7f47a4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f479f7fe640 1 -- 172.21.15.102:0/3488182756 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a40035f0 con 0x7f47a80fe5b0 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f479f7fe640 1 -- 172.21.15.102:0/3488182756 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47a4003c10 con 0x7f47a80fe5b0 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f479f7fe640 1 -- 172.21.15.102:0/3488182756 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a4034440 con 0x7f47a80fe5b0 2024-03-26T19:06:36.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47b02e2640 1 -- 172.21.15.102:0/3488182756 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a819bff0 con 0x7f47a80fe5b0 2024-03-26T19:06:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.748+0000 7f47b02e2640 1 -- 172.21.15.102:0/3488182756 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a819c4f0 con 0x7f47a80fe5b0 2024-03-26T19:06:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.749+0000 7f47b02e2640 1 -- 172.21.15.102:0/3488182756 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a804e000 con 0x7f47a80fe5b0 2024-03-26T19:06:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.750+0000 7f479f7fe640 1 -- 172.21.15.102:0/3488182756 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f47a4004fd0 con 0x7f47a80fe5b0 2024-03-26T19:06:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.750+0000 7f479f7fe640 1 --2- 172.21.15.102:0/3488182756 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f479406e3d0 0x7f4794070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.751+0000 7f479f7fe640 1 -- 172.21.15.102:0/3488182756 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f47a40b0910 con 0x7f47a80fe5b0 2024-03-26T19:06:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.751+0000 7f47ad87d640 1 --2- 172.21.15.102:0/3488182756 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f479406e3d0 0x7f4794070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.751+0000 7f47ad87d640 1 --2- 172.21.15.102:0/3488182756 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f479406e3d0 0x7f4794070880 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4798005e10 tx=0x7f4798005da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:36.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.755+0000 7f479f7fe640 1 -- 172.21.15.102:0/3488182756 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f47a4079a70 con 0x7f47a80fe5b0 2024-03-26T19:06:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.936+0000 7f47b02e2640 1 -- 172.21.15.102:0/3488182756 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f47a8198860 con 0x7f47a80fe5b0 2024-03-26T19:06:36.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.936+0000 7f479f7fe640 1 -- 172.21.15.102:0/3488182756 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f47a407d130 con 0x7f47a80fe5b0 2024-03-26T19:06:36.934 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-26T19:06:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.938+0000 7f47b02e2640 1 -- 172.21.15.102:0/3488182756 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f479406e3d0 msgr2=0x7f4794070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.938+0000 7f47b02e2640 1 --2- 172.21.15.102:0/3488182756 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f479406e3d0 0x7f4794070880 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4798005e10 tx=0x7f4798005da0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.938+0000 7f47b02e2640 1 -- 172.21.15.102:0/3488182756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a80fe5b0 msgr2=0x7f47a8197510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.938+0000 7f47b02e2640 1 --2- 172.21.15.102:0/3488182756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a80fe5b0 0x7f47a8197510 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f47a4009230 tx=0x7f47a4004040 comp rx=0 tx=0).stop 2024-03-26T19:06:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.938+0000 7f47b02e2640 1 -- 172.21.15.102:0/3488182756 shutdown_connections 2024-03-26T19:06:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.938+0000 7f47b02e2640 1 --2- 172.21.15.102:0/3488182756 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7f479406e3d0 0x7f4794070880 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.938+0000 7f47b02e2640 1 --2- 172.21.15.102:0/3488182756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47a8104590 0x7f47a8197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.938+0000 7f47b02e2640 1 --2- 172.21.15.102:0/3488182756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a80fe5b0 0x7f47a8197510 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.938+0000 7f47b02e2640 1 -- 172.21.15.102:0/3488182756 >> 172.21.15.102:0/3488182756 conn(0x7f47a80743d0 msgr2=0x7f47a80673b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:36.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.939+0000 7f47b02e2640 1 -- 172.21.15.102:0/3488182756 shutdown_connections 2024-03-26T19:06:36.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:36.939+0000 7f47b02e2640 1 -- 172.21.15.102:0/3488182756 wait complete. 2024-03-26T19:06:36.947 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-26T19:06:37.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.054+0000 7fb5f7941640 1 -- 172.21.15.102:0/3433724003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f0101340 msgr2=0x7fb5f0109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.054+0000 7fb5f7941640 1 --2- 172.21.15.102:0/3433724003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f0101340 0x7fb5f0109830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0098b0 tx=0x7fb5ec02bd60 comp rx=0 tx=0).stop 2024-03-26T19:06:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.054+0000 7fb5f7941640 1 -- 172.21.15.102:0/3433724003 shutdown_connections 2024-03-26T19:06:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.054+0000 7fb5f7941640 1 --2- 172.21.15.102:0/3433724003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f0101340 0x7fb5f0109830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.054+0000 7fb5f7941640 1 --2- 172.21.15.102:0/3433724003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0100a30 0x7fb5f0100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.054+0000 7fb5f7941640 1 -- 172.21.15.102:0/3433724003 >> 172.21.15.102:0/3433724003 conn(0x7fb5f00fa260 msgr2=0x7fb5f00fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.054+0000 7fb5f7941640 1 -- 172.21.15.102:0/3433724003 shutdown_connections 2024-03-26T19:06:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.054+0000 7fb5f7941640 1 -- 172.21.15.102:0/3433724003 wait complete. 2024-03-26T19:06:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.055+0000 7fb5f7941640 1 Processor -- start 2024-03-26T19:06:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.055+0000 7fb5f7941640 1 -- start start 2024-03-26T19:06:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.055+0000 7fb5f7941640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f0100a30 0x7fb5f0195670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.055+0000 7fb5f7941640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0101340 0x7fb5f0195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.055+0000 7fb5f7941640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f0196240 con 0x7fb5f0100a30 2024-03-26T19:06:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.055+0000 7fb5f7941640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f019a010 con 0x7fb5f0101340 2024-03-26T19:06:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.055+0000 7fb5f56dd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f0100a30 0x7fb5f0195670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.055+0000 7fb5f56dd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f0100a30 0x7fb5f0195670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57286/0 (socket says 172.21.15.102:57286) 2024-03-26T19:06:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.055+0000 7fb5f56dd640 1 -- 172.21.15.102:0/700932193 learned_addr learned my addr 172.21.15.102:0/700932193 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.056+0000 7fb5f56dd640 1 -- 172.21.15.102:0/700932193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0101340 msgr2=0x7fb5f0195bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.056+0000 7fb5f4edc640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0101340 0x7fb5f0195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.056+0000 7fb5f56dd640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0101340 0x7fb5f0195bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.056+0000 7fb5f56dd640 1 -- 172.21.15.102:0/700932193 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec009560 con 0x7fb5f0100a30 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.056+0000 7fb5f4edc640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0101340 0x7fb5f0195bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.056+0000 7fb5f56dd640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f0100a30 0x7fb5f0195670 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000b420 tx=0x7fb5e000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.056+0000 7fb5e67fc640 1 -- 172.21.15.102:0/700932193 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5e0011800 con 0x7fb5f0100a30 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.056+0000 7fb5e67fc640 1 -- 172.21.15.102:0/700932193 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5e0011e20 con 0x7fb5f0100a30 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.056+0000 7fb5f7941640 1 -- 172.21.15.102:0/700932193 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f019a2f0 con 0x7fb5f0100a30 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.056+0000 7fb5e67fc640 1 -- 172.21.15.102:0/700932193 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5e000f2e0 con 0x7fb5f0100a30 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.056+0000 7fb5f7941640 1 -- 172.21.15.102:0/700932193 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f019a850 con 0x7fb5f0100a30 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.057+0000 7fb5f7941640 1 -- 172.21.15.102:0/700932193 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f004e000 con 0x7fb5f0100a30 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.057+0000 7fb5e67fc640 1 -- 172.21.15.102:0/700932193 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb5e0010380 con 0x7fb5f0100a30 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.058+0000 7fb5e67fc640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb5dc06e3d0 0x7fb5dc070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.058+0000 7fb5e67fc640 1 -- 172.21.15.102:0/700932193 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fb5e008ddc0 con 0x7fb5f0100a30 2024-03-26T19:06:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.058+0000 7fb5f4edc640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb5dc06e3d0 0x7fb5dc070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:37.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.058+0000 7fb5f4edc640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb5dc06e3d0 0x7fb5dc070880 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec005ec0 tx=0x7fb5ec005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:37.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.062+0000 7fb5e67fc640 1 -- 172.21.15.102:0/700932193 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb5e0056f20 con 0x7fb5f0100a30 2024-03-26T19:06:37.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.243+0000 7fb5f7941640 1 -- 172.21.15.102:0/700932193 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fb5f0066860 con 0x7fb5f0100a30 2024-03-26T19:06:37.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.243+0000 7fb5e67fc640 1 -- 172.21.15.102:0/700932193 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb5e005a5e0 con 0x7fb5f0100a30 2024-03-26T19:06:37.241 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-26T19:06:37.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.245+0000 7fb5f7941640 1 -- 172.21.15.102:0/700932193 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb5dc06e3d0 msgr2=0x7fb5dc070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.245+0000 7fb5f7941640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb5dc06e3d0 0x7fb5dc070880 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec005ec0 tx=0x7fb5ec005fd0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.245+0000 7fb5f7941640 1 -- 172.21.15.102:0/700932193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f0100a30 msgr2=0x7fb5f0195670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.245+0000 7fb5f7941640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f0100a30 0x7fb5f0195670 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000b420 tx=0x7fb5e000b8e0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.245+0000 7fb5f7941640 1 -- 172.21.15.102:0/700932193 shutdown_connections 2024-03-26T19:06:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.245+0000 7fb5f7941640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fb5dc06e3d0 0x7fb5dc070880 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.245+0000 7fb5f7941640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0101340 0x7fb5f0195bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.245+0000 7fb5f7941640 1 --2- 172.21.15.102:0/700932193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f0100a30 0x7fb5f0195670 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.245+0000 7fb5f7941640 1 -- 172.21.15.102:0/700932193 >> 172.21.15.102:0/700932193 conn(0x7fb5f00fa260 msgr2=0x7fb5f0104070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.245+0000 7fb5f7941640 1 -- 172.21.15.102:0/700932193 shutdown_connections 2024-03-26T19:06:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.245+0000 7fb5f7941640 1 -- 172.21.15.102:0/700932193 wait complete. 2024-03-26T19:06:37.253 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:37 smithi102 ceph-mon[78322]: pgmap v336: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:37.253 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:37 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3753980865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T19:06:37.253 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:37 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2304406411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T19:06:37.253 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:37 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/3488182756' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-26T19:06:37.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-26T19:06:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.362+0000 7fa3ad578640 1 -- 172.21.15.102:0/570463536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3a81045d0 msgr2=0x7fa3a81049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.362+0000 7fa3ad578640 1 --2- 172.21.15.102:0/570463536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3a81045d0 0x7fa3a81049a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa398009880 tx=0x7fa39802bce0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.362+0000 7fa3ad578640 1 -- 172.21.15.102:0/570463536 shutdown_connections 2024-03-26T19:06:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.362+0000 7fa3ad578640 1 --2- 172.21.15.102:0/570463536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3a80fe5a0 0x7fa3a80fea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.362+0000 7fa3ad578640 1 --2- 172.21.15.102:0/570463536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3a81045d0 0x7fa3a81049a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.362+0000 7fa3ad578640 1 -- 172.21.15.102:0/570463536 >> 172.21.15.102:0/570463536 conn(0x7fa3a8075270 msgr2=0x7fa3a8075670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.362+0000 7fa3ad578640 1 -- 172.21.15.102:0/570463536 shutdown_connections 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.362+0000 7fa3ad578640 1 -- 172.21.15.102:0/570463536 wait complete. 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.362+0000 7fa3ad578640 1 Processor -- start 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa3ad578640 1 -- start start 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa3ad578640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3a80fe5a0 0x7fa3a8194070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa3ad578640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3a81045d0 0x7fa3a81945b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa3ad578640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a8194bb0 con 0x7fa3a81045d0 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa3ad578640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a8194d20 con 0x7fa3a80fe5a0 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa39ffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3a81045d0 0x7fa3a81945b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa39ffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3a81045d0 0x7fa3a81945b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57306/0 (socket says 172.21.15.102:57306) 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa39ffff640 1 -- 172.21.15.102:0/2421478909 learned_addr learned my addr 172.21.15.102:0/2421478909 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-26T19:06:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa3a6ffd640 1 --2- 172.21.15.102:0/2421478909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3a80fe5a0 0x7fa3a8194070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:37.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa39ffff640 1 -- 172.21.15.102:0/2421478909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3a80fe5a0 msgr2=0x7fa3a8194070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:37.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa39ffff640 1 --2- 172.21.15.102:0/2421478909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3a80fe5a0 0x7fa3a8194070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.363+0000 7fa39ffff640 1 -- 172.21.15.102:0/2421478909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa398009560 con 0x7fa3a81045d0 2024-03-26T19:06:37.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.364+0000 7fa39ffff640 1 --2- 172.21.15.102:0/2421478909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3a81045d0 0x7fa3a81945b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa39000c8a0 tx=0x7fa39000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:37.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.364+0000 7fa3a4ff9640 1 -- 172.21.15.102:0/2421478909 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa390007b80 con 0x7fa3a81045d0 2024-03-26T19:06:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.364+0000 7fa3ad578640 1 -- 172.21.15.102:0/2421478909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a8198bb0 con 0x7fa3a81045d0 2024-03-26T19:06:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.364+0000 7fa3ad578640 1 -- 172.21.15.102:0/2421478909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a8199190 con 0x7fa3a81045d0 2024-03-26T19:06:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.364+0000 7fa3a4ff9640 1 -- 172.21.15.102:0/2421478909 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa390007ce0 con 0x7fa3a81045d0 2024-03-26T19:06:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.364+0000 7fa3a4ff9640 1 -- 172.21.15.102:0/2421478909 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa390018850 con 0x7fa3a81045d0 2024-03-26T19:06:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.365+0000 7fa3ad578640 1 -- 172.21.15.102:0/2421478909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa388005280 con 0x7fa3a81045d0 2024-03-26T19:06:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.366+0000 7fa3a4ff9640 1 -- 172.21.15.102:0/2421478909 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 37) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa3900189b0 con 0x7fa3a81045d0 2024-03-26T19:06:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.366+0000 7fa3a4ff9640 1 --2- 172.21.15.102:0/2421478909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa39406e490 0x7fa394070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T19:06:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.366+0000 7fa3a4ff9640 1 -- 172.21.15.102:0/2421478909 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fa39008e530 con 0x7fa3a81045d0 2024-03-26T19:06:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.366+0000 7fa3a6ffd640 1 --2- 172.21.15.102:0/2421478909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa39406e490 0x7fa394070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T19:06:37.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.367+0000 7fa3a6ffd640 1 --2- 172.21.15.102:0/2421478909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa39406e490 0x7fa394070940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa398006010 tx=0x7fa398005b50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T19:06:37.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.371+0000 7fa3a4ff9640 1 -- 172.21.15.102:0/2421478909 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa390057690 con 0x7fa3a81045d0 2024-03-26T19:06:37.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:37 smithi190 ceph-mon[65689]: pgmap v336: 65 pgs: 65 active+clean; 182 MiB data, 805 MiB used, 536 GiB / 536 GiB avail 2024-03-26T19:06:37.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:37 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3753980865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T19:06:37.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:37 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2304406411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T19:06:37.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:37 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/3488182756' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-26T19:06:37.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.567+0000 7fa3ad578640 1 -- 172.21.15.102:0/2421478909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa388005ab0 con 0x7fa3a81045d0 2024-03-26T19:06:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.568+0000 7fa3a4ff9640 1 -- 172.21.15.102:0/2421478909 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 43 entries v84) v1 ==== 81+0+2579 (secure 0 0 0) 0x7fa39005ad50 con 0x7fa3a81045d0 2024-03-26T19:06:37.565 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:6827/624694186 2024-03-27T19:02:46.321671+0000 2024-03-26T19:06:37.565 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:6826/624694186 2024-03-27T19:02:46.321671+0000 2024-03-26T19:06:37.565 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6829/989894576 2024-03-27T19:02:06.598591+0000 2024-03-26T19:06:37.565 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6828/989894576 2024-03-27T19:02:06.598591+0000 2024-03-26T19:06:37.565 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2151777534 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1215990907 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2396893787 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/1874485118 2024-03-27T18:46:28.035077+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/3353740154 2024-03-27T18:47:33.839189+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1994330698 2024-03-27T18:46:28.035077+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6827/2550866692 2024-03-27T18:52:35.022250+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1842664791 2024-03-27T18:47:33.839189+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/308643739 2024-03-27T18:45:55.925005+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/2308588851 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1295801893 2024-03-27T18:45:55.925005+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/1874485118 2024-03-27T18:46:28.035077+0000 2024-03-26T19:06:37.566 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6826/2550866692 2024-03-27T18:52:35.022250+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2094779473 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/300222988 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/3517043895 2024-03-27T18:47:33.839189+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2686253586 2024-03-27T18:45:55.925005+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1212290737 2024-03-27T18:46:28.035077+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2752725029 2024-03-27T18:46:28.035077+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2027932959 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/3113541506 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/308643739 2024-03-27T18:45:55.925005+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/216723823 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:6828/2858265260 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:6829/2858265260 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:37.567 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/3517043895 2024-03-27T18:47:33.839189+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1380712530 2024-03-27T18:45:55.925005+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/990121767 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1480032448 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/1923628519 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/2831156761 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1818262291 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1615769300 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/833763291 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.190:0/3941911909 2024-03-27T18:55:42.356278+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/748923615 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/4276276941 2024-03-27T18:47:33.839189+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/748923615 2024-03-27T18:54:47.731502+0000 2024-03-26T19:06:37.568 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/2831156761 2024-03-27T18:56:37.181768+0000 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:listed 43 entries 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.570+0000 7fa3ad578640 1 -- 172.21.15.102:0/2421478909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa39406e490 msgr2=0x7fa394070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.570+0000 7fa3ad578640 1 --2- 172.21.15.102:0/2421478909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa39406e490 0x7fa394070940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa398006010 tx=0x7fa398005b50 comp rx=0 tx=0).stop 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.570+0000 7fa3ad578640 1 -- 172.21.15.102:0/2421478909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3a81045d0 msgr2=0x7fa3a81945b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.570+0000 7fa3ad578640 1 --2- 172.21.15.102:0/2421478909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3a81045d0 0x7fa3a81945b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa39000c8a0 tx=0x7fa39000cd60 comp rx=0 tx=0).stop 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.570+0000 7fa3ad578640 1 -- 172.21.15.102:0/2421478909 shutdown_connections 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.570+0000 7fa3ad578640 1 --2- 172.21.15.102:0/2421478909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3a81045d0 0x7fa3a81945b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.570+0000 7fa3ad578640 1 --2- 172.21.15.102:0/2421478909 >> [v2:172.21.15.102:6800/4269744347,v1:172.21.15.102:6801/4269744347] conn(0x7fa39406e490 0x7fa394070940 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.570+0000 7fa3ad578640 1 --2- 172.21.15.102:0/2421478909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3a80fe5a0 0x7fa3a8194070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.570+0000 7fa3ad578640 1 -- 172.21.15.102:0/2421478909 >> 172.21.15.102:0/2421478909 conn(0x7fa3a8075270 msgr2=0x7fa3a80fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.571+0000 7fa3ad578640 1 -- 172.21.15.102:0/2421478909 shutdown_connections 2024-03-26T19:06:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-26T19:06:37.571+0000 7fa3ad578640 1 -- 172.21.15.102:0/2421478909 wait complete. 2024-03-26T19:06:37.579 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-26T19:06:37.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T19:06:37.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-03-26T19:06:37.615 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi190.front.sepia.ceph.com 2024-03-26T19:06:37.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T19:06:37.616 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-26T19:06:37.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T19:06:37.769 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-26T19:06:37.784 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-26T19:06:37.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T19:06:37.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-26T19:06:37.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T19:06:37.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-03-26T19:06:37.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T19:06:37.953 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2024-03-26T19:06:37.971 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-26T19:06:37.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T19:06:37.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-26T19:06:38.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T19:06:38.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link delete ceph-brx 2024-03-26T19:06:38.143 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T19:06:38.154 INFO:tasks.cephadm:Teardown begin 2024-03-26T19:06:38.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T19:06:38.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T19:06:38.212 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T19:06:38.213 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T19:06:38.237 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T19:06:38.268 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T19:06:38.268 INFO:tasks.cephadm.mon.smithi102:Stopping mon.smithi102... 2024-03-26T19:06:38.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi102 2024-03-26T19:06:38.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:38 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/700932193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-26T19:06:38.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:38 smithi190 ceph-mon[65689]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:06:38.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:38 smithi190 ceph-mon[65689]: from='client.? 172.21.15.102:0/2421478909' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T19:06:38.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:38 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/700932193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-26T19:06:38.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:38 smithi102 ceph-mon[78322]: from='mgr.34102 172.21.15.102:0/3435350339' entity='mgr.smithi102.rsfpnv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T19:06:38.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:38 smithi102 ceph-mon[78322]: from='client.? 172.21.15.102:0/2421478909' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T19:06:38.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:38 smithi102 systemd[1]: Stopping Ceph mon.smithi102 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:06:38.880 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:38 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102[78318]: 2024-03-26T19:06:38.629+0000 7f956ce21700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi102 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T19:06:38.880 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:38 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102[78318]: 2024-03-26T19:06:38.629+0000 7f956ce21700 -1 mon.smithi102@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T19:06:38.880 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:38 smithi102 podman[129886]: 2024-03-26 19:06:38.822704951 +0000 UTC m=+0.321069327 container died 9aaa700937607b9246599dc3dc8c91e72cc0eb42f6ebd347b4fc354fe774a4fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-26T19:06:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:39 smithi102 podman[129886]: 2024-03-26 19:06:39.18731037 +0000 UTC m=+0.685674745 container cleanup 9aaa700937607b9246599dc3dc8c91e72cc0eb42f6ebd347b4fc354fe774a4fd (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T19:06:39.380 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Mar 26 19:06:39 smithi102 bash[129886]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi102 2024-03-26T19:06:39.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi102.service' 2024-03-26T19:06:39.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T19:06:39.687 INFO:tasks.cephadm.mon.smithi102:Stopped mon.smithi102 2024-03-26T19:06:39.688 INFO:tasks.cephadm.mon.smithi190:Stopping mon.smithi190... 2024-03-26T19:06:39.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi190 2024-03-26T19:06:40.016 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:39 smithi190 systemd[1]: Stopping Ceph mon.smithi190 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:06:40.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:40 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190[65685]: 2024-03-26T19:06:40.018+0000 7f767631d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi190 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T19:06:40.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:40 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190[65685]: 2024-03-26T19:06:40.018+0000 7f767631d700 -1 mon.smithi190@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T19:06:40.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:40 smithi190 podman[95266]: 2024-03-26 19:06:40.121042198 +0000 UTC m=+0.236259635 container died e80478bd14bd74e2ab2c34ae28dac57fbf5ab80749d3fb0d1108f2aefc8d69cf (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T19:06:40.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:40 smithi190 podman[95266]: 2024-03-26 19:06:40.56754227 +0000 UTC m=+0.682759703 container cleanup e80478bd14bd74e2ab2c34ae28dac57fbf5ab80749d3fb0d1108f2aefc8d69cf (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5) 2024-03-26T19:06:40.784 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:Mar 26 19:06:40 smithi190 bash[95266]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-mon-smithi190 2024-03-26T19:06:40.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@mon.smithi190.service' 2024-03-26T19:06:41.037 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T19:06:41.038 INFO:tasks.cephadm.mon.smithi190:Stopped mon.smithi190 2024-03-26T19:06:41.038 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T19:06:41.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.0 2024-03-26T19:06:41.380 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:41 smithi102 systemd[1]: Stopping Ceph osd.0 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:06:41.880 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:41 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[84832]: 2024-03-26T19:06:41.383+0000 7f79a902a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T19:06:41.880 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:41 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[84832]: 2024-03-26T19:06:41.383+0000 7f79a902a700 -1 osd.0 84 *** Got signal Terminated *** 2024-03-26T19:06:41.880 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:41 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0[84832]: 2024-03-26T19:06:41.383+0000 7f79a902a700 -1 osd.0 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T19:06:47.048 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:46 smithi102 podman[130102]: 2024-03-26 19:06:46.751238065 +0000 UTC m=+5.495356799 container died aec2fe1c7a8b97a70c36828c7bac441a8667be85d75b900c607ae7e87e99939f (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T19:06:47.380 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:47 smithi102 podman[130102]: 2024-03-26 19:06:47.2344823 +0000 UTC m=+5.978601022 container cleanup aec2fe1c7a8b97a70c36828c7bac441a8667be85d75b900c607ae7e87e99939f (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T19:06:47.380 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:47 smithi102 bash[130102]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0 2024-03-26T19:06:47.816 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:47 smithi102 podman[130121]: 2024-03-26 19:06:47.503140885 +0000 UTC m=+0.746195828 container remove aec2fe1c7a8b97a70c36828c7bac441a8667be85d75b900c607ae7e87e99939f (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T19:06:48.104 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:47 smithi102 podman[130212]: 2024-03-26 19:06:47.72510468 +0000 UTC m=+0.026893022 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:06:48.104 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:47 smithi102 podman[130212]: 2024-03-26 19:06:47.854215139 +0000 UTC m=+0.156003467 container create c1806e5e675a9d7a9eb37af1d8ab65e9e521aff6cfead6397aeaae0d33aa931c (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T19:06:48.380 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:48 smithi102 podman[130212]: 2024-03-26 19:06:48.10754459 +0000 UTC m=+0.409332927 container init c1806e5e675a9d7a9eb37af1d8ab65e9e521aff6cfead6397aeaae0d33aa931c (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.schema-version=1.0) 2024-03-26T19:06:48.380 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:48 smithi102 podman[130212]: 2024-03-26 19:06:48.111442594 +0000 UTC m=+0.413230923 container start c1806e5e675a9d7a9eb37af1d8ab65e9e521aff6cfead6397aeaae0d33aa931c (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T19:06:48.380 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:48 smithi102 podman[130212]: 2024-03-26 19:06:48.198984015 +0000 UTC m=+0.500772353 container attach c1806e5e675a9d7a9eb37af1d8ab65e9e521aff6cfead6397aeaae0d33aa931c (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T19:06:48.737 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 26 19:06:48 smithi102 podman[130212]: 2024-03-26 19:06:48.45122176 +0000 UTC m=+0.753010090 container died c1806e5e675a9d7a9eb37af1d8ab65e9e521aff6cfead6397aeaae0d33aa931c (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T19:06:49.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T19:06:49.095 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T19:06:49.095 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T19:06:49.095 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T19:06:49.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.1 2024-03-26T19:06:49.380 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:49 smithi102 systemd[1]: Stopping Ceph osd.1 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:06:49.840 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:49 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[88121]: 2024-03-26T19:06:49.439+0000 7faa7258e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T19:06:49.840 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:49 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[88121]: 2024-03-26T19:06:49.439+0000 7faa7258e700 -1 osd.1 84 *** Got signal Terminated *** 2024-03-26T19:06:49.840 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:49 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1[88121]: 2024-03-26T19:06:49.439+0000 7faa7258e700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T19:06:55.311 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:54 smithi102 podman[130326]: 2024-03-26 19:06:54.892463395 +0000 UTC m=+5.571923826 container died 3149459e0c02ee53695e81d55c84d24dd5345b9bed3f7f5ef54c1244ecb829c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T19:06:55.630 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:55 smithi102 podman[130326]: 2024-03-26 19:06:55.314628693 +0000 UTC m=+5.994089118 container cleanup 3149459e0c02ee53695e81d55c84d24dd5345b9bed3f7f5ef54c1244ecb829c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T19:06:55.630 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:55 smithi102 bash[130326]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1 2024-03-26T19:06:55.630 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:55 smithi102 podman[130343]: 2024-03-26 19:06:55.524618643 +0000 UTC m=+0.632292831 container remove 3149459e0c02ee53695e81d55c84d24dd5345b9bed3f7f5ef54c1244ecb829c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T19:06:56.266 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:55 smithi102 podman[130430]: 2024-03-26 19:06:55.879408498 +0000 UTC m=+0.018116436 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:06:56.267 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:55 smithi102 podman[130430]: 2024-03-26 19:06:55.983952515 +0000 UTC m=+0.122660439 container create d65b5b242d6f1119c8044b12efcdcb4764ba6ee73a92a71d5fa481aba4e0765d (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T19:06:56.267 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:56 smithi102 podman[130430]: 2024-03-26 19:06:56.195368883 +0000 UTC m=+0.334076818 container init d65b5b242d6f1119c8044b12efcdcb4764ba6ee73a92a71d5fa481aba4e0765d (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-26T19:06:56.267 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:56 smithi102 podman[130430]: 2024-03-26 19:06:56.198975309 +0000 UTC m=+0.337683234 container start d65b5b242d6f1119c8044b12efcdcb4764ba6ee73a92a71d5fa481aba4e0765d (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-03-26T19:06:56.267 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:56 smithi102 podman[130430]: 2024-03-26 19:06:56.270046864 +0000 UTC m=+0.408754797 container attach d65b5b242d6f1119c8044b12efcdcb4764ba6ee73a92a71d5fa481aba4e0765d (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T19:06:56.630 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 26 19:06:56 smithi102 podman[130430]: 2024-03-26 19:06:56.514947305 +0000 UTC m=+0.653655217 container died d65b5b242d6f1119c8044b12efcdcb4764ba6ee73a92a71d5fa481aba4e0765d (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T19:06:57.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T19:06:57.117 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T19:06:57.117 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T19:06:57.117 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T19:06:57.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.2 2024-03-26T19:06:57.380 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:06:57 smithi102 systemd[1]: Stopping Ceph osd.2 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:06:57.880 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:06:57 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[91448]: 2024-03-26T19:06:57.509+0000 7fbfc35d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T19:06:57.880 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:06:57 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[91448]: 2024-03-26T19:06:57.509+0000 7fbfc35d6700 -1 osd.2 84 *** Got signal Terminated *** 2024-03-26T19:06:57.880 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:06:57 smithi102 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2[91448]: 2024-03-26T19:06:57.509+0000 7fbfc35d6700 -1 osd.2 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T19:07:03.116 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:07:02 smithi102 podman[130536]: 2024-03-26 19:07:02.835765409 +0000 UTC m=+5.451011674 container died 60206f27cb55553818b5f86f0d61a42786e62eeb3ce3e59bff3bb021cdc4fa2b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T19:07:03.380 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:07:03 smithi102 podman[130536]: 2024-03-26 19:07:03.318818891 +0000 UTC m=+5.934065158 container cleanup 60206f27cb55553818b5f86f0d61a42786e62eeb3ce3e59bff3bb021cdc4fa2b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T19:07:03.380 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:07:03 smithi102 bash[130536]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2 2024-03-26T19:07:03.880 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:07:03 smithi102 podman[130554]: 2024-03-26 19:07:03.586796709 +0000 UTC m=+0.751224216 container remove 60206f27cb55553818b5f86f0d61a42786e62eeb3ce3e59bff3bb021cdc4fa2b (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T19:07:04.179 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:07:03 smithi102 podman[130648]: 2024-03-26 19:07:03.82580517 +0000 UTC m=+0.022021511 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:07:04.179 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:07:03 smithi102 podman[130648]: 2024-03-26 19:07:03.94679323 +0000 UTC m=+0.143009556 container create 4ba77047e52d69cb87b320ef499b3fc511a46b63216809b027a4bf149905da88 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T19:07:04.179 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:07:04 smithi102 podman[130648]: 2024-03-26 19:07:04.124297184 +0000 UTC m=+0.320513512 container init 4ba77047e52d69cb87b320ef499b3fc511a46b63216809b027a4bf149905da88 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T19:07:04.179 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:07:04 smithi102 podman[130648]: 2024-03-26 19:07:04.127621163 +0000 UTC m=+0.323837497 container start 4ba77047e52d69cb87b320ef499b3fc511a46b63216809b027a4bf149905da88 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T19:07:04.180 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:07:04 smithi102 podman[130648]: 2024-03-26 19:07:04.182577644 +0000 UTC m=+0.378793979 container attach 4ba77047e52d69cb87b320ef499b3fc511a46b63216809b027a4bf149905da88 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T19:07:04.630 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 26 19:07:04 smithi102 podman[130648]: 2024-03-26 19:07:04.428114969 +0000 UTC m=+0.624331295 container died 4ba77047e52d69cb87b320ef499b3fc511a46b63216809b027a4bf149905da88 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-26T19:07:04.967 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T19:07:05.008 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T19:07:05.008 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T19:07:05.008 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T19:07:05.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.3 2024-03-26T19:07:05.284 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:05 smithi190 systemd[1]: Stopping Ceph osd.3 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:07:05.784 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:05 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[70972]: 2024-03-26T19:07:05.312+0000 7fd814856700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T19:07:05.784 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:05 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[70972]: 2024-03-26T19:07:05.312+0000 7fd814856700 -1 osd.3 84 *** Got signal Terminated *** 2024-03-26T19:07:05.784 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:05 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3[70972]: 2024-03-26T19:07:05.312+0000 7fd814856700 -1 osd.3 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T19:07:08.284 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:07 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:07.880+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:09.284 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:08 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:08.856+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:10.284 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:09 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:09.793+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:46.238855+0000 front 2024-03-26T19:06:46.238748+0000 (oldest deadline 2024-03-26T19:07:09.738752+0000) 2024-03-26T19:07:10.285 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:09 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:09.890+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:10.993 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:10 smithi190 podman[95402]: 2024-03-26 19:07:10.700322348 +0000 UTC m=+5.491270497 container died 03955331c9621e6f8ad50bf62790749950242db6f6a2affac1561e4fe89d0f57 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T19:07:10.994 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:10 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:10.816+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:46.238855+0000 front 2024-03-26T19:06:46.238748+0000 (oldest deadline 2024-03-26T19:07:09.738752+0000) 2024-03-26T19:07:10.994 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:10 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:10.882+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:11.284 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:11 smithi190 podman[95402]: 2024-03-26 19:07:11.095984621 +0000 UTC m=+5.886932759 container cleanup 03955331c9621e6f8ad50bf62790749950242db6f6a2affac1561e4fe89d0f57 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240320.145742-3605160, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T19:07:11.285 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:11 smithi190 bash[95402]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3 2024-03-26T19:07:11.663 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:11 smithi190 podman[95422]: 2024-03-26 19:07:11.347191051 +0000 UTC m=+0.636808801 container remove 03955331c9621e6f8ad50bf62790749950242db6f6a2affac1561e4fe89d0f57 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240320.145742-3605160, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T19:07:11.946 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:11 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:11.784+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:46.238855+0000 front 2024-03-26T19:06:46.238748+0000 (oldest deadline 2024-03-26T19:07:09.738752+0000) 2024-03-26T19:07:11.947 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:11 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:11.886+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:11.947 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:11 smithi190 podman[95513]: 2024-03-26 19:07:11.570006555 +0000 UTC m=+0.018480234 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:07:11.947 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:11 smithi190 podman[95513]: 2024-03-26 19:07:11.689560777 +0000 UTC m=+0.138034423 container create af89c32be4c980d4d6b8b683dfc2e3f7ae8b0a308465486f3f3065d83e693286 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T19:07:11.947 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:11 smithi190 podman[95513]: 2024-03-26 19:07:11.950480147 +0000 UTC m=+0.398953793 container init af89c32be4c980d4d6b8b683dfc2e3f7ae8b0a308465486f3f3065d83e693286 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T19:07:12.239 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:11 smithi190 podman[95513]: 2024-03-26 19:07:11.954002303 +0000 UTC m=+0.402475954 container start af89c32be4c980d4d6b8b683dfc2e3f7ae8b0a308465486f3f3065d83e693286 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T19:07:12.239 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:12 smithi190 podman[95513]: 2024-03-26 19:07:12.033642361 +0000 UTC m=+0.482116025 container attach af89c32be4c980d4d6b8b683dfc2e3f7ae8b0a308465486f3f3065d83e693286 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-26T19:07:12.514 INFO:journalctl@ceph.osd.3.smithi190.stdout:Mar 26 19:07:12 smithi190 podman[95513]: 2024-03-26 19:07:12.243131943 +0000 UTC m=+0.691605608 container died af89c32be4c980d4d6b8b683dfc2e3f7ae8b0a308465486f3f3065d83e693286 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T19:07:12.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T19:07:12.960 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T19:07:12.961 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T19:07:12.961 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T19:07:12.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.4 2024-03-26T19:07:13.270 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:12 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:12.870+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:13.271 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:12 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:12.799+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:46.238855+0000 front 2024-03-26T19:06:46.238748+0000 (oldest deadline 2024-03-26T19:07:09.738752+0000) 2024-03-26T19:07:13.271 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:13 smithi190 systemd[1]: Stopping Ceph osd.4 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:07:13.271 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:13 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:13.273+0000 7efe14574700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T19:07:13.534 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:13 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:13.273+0000 7efe14574700 -1 osd.4 84 *** Got signal Terminated *** 2024-03-26T19:07:13.535 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:13 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:13.273+0000 7efe14574700 -1 osd.4 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T19:07:14.034 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:13 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:13.761+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:46.238855+0000 front 2024-03-26T19:06:46.238748+0000 (oldest deadline 2024-03-26T19:07:09.738752+0000) 2024-03-26T19:07:14.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:13 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:13.888+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:15.035 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:14 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:14.754+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:46.238855+0000 front 2024-03-26T19:06:46.238748+0000 (oldest deadline 2024-03-26T19:07:09.738752+0000) 2024-03-26T19:07:15.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:14 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:14.844+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:16.284 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:15 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:15.803+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:46.238855+0000 front 2024-03-26T19:06:46.238748+0000 (oldest deadline 2024-03-26T19:07:09.738752+0000) 2024-03-26T19:07:16.284 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:15 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:15.803+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:52.039646+0000 front 2024-03-26T19:06:52.039588+0000 (oldest deadline 2024-03-26T19:07:15.539603+0000) 2024-03-26T19:07:16.284 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:15 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:15.881+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:16.285 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:15 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:15.881+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:49.824934+0000 front 2024-03-26T19:06:49.824966+0000 (oldest deadline 2024-03-26T19:07:15.124896+0000) 2024-03-26T19:07:17.034 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:16 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:16.770+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:46.238855+0000 front 2024-03-26T19:06:46.238748+0000 (oldest deadline 2024-03-26T19:07:09.738752+0000) 2024-03-26T19:07:17.035 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:16 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:16.770+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:52.039646+0000 front 2024-03-26T19:06:52.039588+0000 (oldest deadline 2024-03-26T19:07:15.539603+0000) 2024-03-26T19:07:17.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:16 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:16.880+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:17.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:16 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:16.880+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:49.824934+0000 front 2024-03-26T19:06:49.824966+0000 (oldest deadline 2024-03-26T19:07:15.124896+0000) 2024-03-26T19:07:18.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:17 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:17.897+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:18.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:17 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:17.897+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:49.824934+0000 front 2024-03-26T19:06:49.824966+0000 (oldest deadline 2024-03-26T19:07:15.124896+0000) 2024-03-26T19:07:18.035 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:17 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:17.779+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:46.238855+0000 front 2024-03-26T19:06:46.238748+0000 (oldest deadline 2024-03-26T19:07:09.738752+0000) 2024-03-26T19:07:18.035 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:17 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4[73840]: 2024-03-26T19:07:17.779+0000 7efe0cb60700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:52.039646+0000 front 2024-03-26T19:06:52.039588+0000 (oldest deadline 2024-03-26T19:07:15.539603+0000) 2024-03-26T19:07:19.034 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:18 smithi190 podman[95631]: 2024-03-26 19:07:18.740314155 +0000 UTC m=+5.547535178 container died 23e140cd8961fed85f06b4a21ea6e977aaff572cc0a952bac22f0e5fc8b56235 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T19:07:19.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:18 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:18.850+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:19.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:18 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:18.850+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:49.824934+0000 front 2024-03-26T19:06:49.824966+0000 (oldest deadline 2024-03-26T19:07:15.124896+0000) 2024-03-26T19:07:19.363 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:19 smithi190 podman[95631]: 2024-03-26 19:07:19.148915682 +0000 UTC m=+5.956136707 container cleanup 23e140cd8961fed85f06b4a21ea6e977aaff572cc0a952bac22f0e5fc8b56235 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T19:07:19.363 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:19 smithi190 bash[95631]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4 2024-03-26T19:07:19.689 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:19 smithi190 podman[95650]: 2024-03-26 19:07:19.366729413 +0000 UTC m=+0.616693453 container remove 23e140cd8961fed85f06b4a21ea6e977aaff572cc0a952bac22f0e5fc8b56235 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T19:07:19.941 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:19 smithi190 podman[95743]: 2024-03-26 19:07:19.59565141 +0000 UTC m=+0.017422678 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:07:19.942 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:19 smithi190 podman[95743]: 2024-03-26 19:07:19.700846251 +0000 UTC m=+0.122617518 container create 159e3c4d0f02fa75bdcfaf7fb68bd018aa9adbd28f3df5c020021b49e50beb2a (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240320.145742-3605160, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T19:07:19.942 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:19 smithi190 podman[95743]: 2024-03-26 19:07:19.945075844 +0000 UTC m=+0.366847112 container init 159e3c4d0f02fa75bdcfaf7fb68bd018aa9adbd28f3df5c020021b49e50beb2a (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_BRANCH=HEAD) 2024-03-26T19:07:19.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:19 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:19.842+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:19.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:19 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:19.842+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:49.824934+0000 front 2024-03-26T19:06:49.824966+0000 (oldest deadline 2024-03-26T19:07:15.124896+0000) 2024-03-26T19:07:20.229 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:19 smithi190 podman[95743]: 2024-03-26 19:07:19.94817721 +0000 UTC m=+0.369948481 container start 159e3c4d0f02fa75bdcfaf7fb68bd018aa9adbd28f3df5c020021b49e50beb2a (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T19:07:20.229 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:20 smithi190 podman[95743]: 2024-03-26 19:07:20.053332508 +0000 UTC m=+0.475103776 container attach 159e3c4d0f02fa75bdcfaf7fb68bd018aa9adbd28f3df5c020021b49e50beb2a (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True) 2024-03-26T19:07:20.534 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 26 19:07:20 smithi190 podman[95743]: 2024-03-26 19:07:20.233069723 +0000 UTC m=+0.654840999 container died 159e3c4d0f02fa75bdcfaf7fb68bd018aa9adbd28f3df5c020021b49e50beb2a (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T19:07:20.853 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:20 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:20.796+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:20.853 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:20 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:20.796+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:49.824934+0000 front 2024-03-26T19:06:49.824966+0000 (oldest deadline 2024-03-26T19:07:15.124896+0000) 2024-03-26T19:07:20.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T19:07:20.927 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T19:07:20.927 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T19:07:20.927 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T19:07:20.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.5 2024-03-26T19:07:21.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:21 smithi190 systemd[1]: Stopping Ceph osd.5 for ed4e027c-eba0-11ee-95d0-87774f69a715... 2024-03-26T19:07:21.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:21 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:21.249+0000 7f9ea954a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T19:07:21.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:21 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:21.249+0000 7f9ea954a700 -1 osd.5 84 *** Got signal Terminated *** 2024-03-26T19:07:21.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:21 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:21.249+0000 7f9ea954a700 -1 osd.5 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T19:07:22.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:21 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:21.766+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:22.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:21 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:21.766+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:49.824934+0000 front 2024-03-26T19:06:49.824966+0000 (oldest deadline 2024-03-26T19:07:15.124896+0000) 2024-03-26T19:07:23.284 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:22 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:22.801+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:23.285 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:22 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:22.801+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:49.824934+0000 front 2024-03-26T19:06:49.824966+0000 (oldest deadline 2024-03-26T19:07:15.124896+0000) 2024-03-26T19:07:24.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:23 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:23.767+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:24.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:23 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:23.767+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:49.824934+0000 front 2024-03-26T19:06:49.824966+0000 (oldest deadline 2024-03-26T19:07:15.124896+0000) 2024-03-26T19:07:24.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:23 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:23.767+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-26T19:07:01.327054+0000 front 2024-03-26T19:07:01.327074+0000 (oldest deadline 2024-03-26T19:07:23.627096+0000) 2024-03-26T19:07:25.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:24 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:24.796+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:25.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:24 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:24.796+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:49.824934+0000 front 2024-03-26T19:06:49.824966+0000 (oldest deadline 2024-03-26T19:07:15.124896+0000) 2024-03-26T19:07:25.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:24 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:24.796+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-26T19:07:01.327054+0000 front 2024-03-26T19:07:01.327074+0000 (oldest deadline 2024-03-26T19:07:23.627096+0000) 2024-03-26T19:07:26.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:25 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:25.756+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-03-26T19:06:43.523921+0000 front 2024-03-26T19:06:43.523988+0000 (oldest deadline 2024-03-26T19:07:07.023809+0000) 2024-03-26T19:07:26.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:25 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:25.756+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-03-26T19:06:49.824934+0000 front 2024-03-26T19:06:49.824966+0000 (oldest deadline 2024-03-26T19:07:15.124896+0000) 2024-03-26T19:07:26.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:25 smithi190 ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5[76739]: 2024-03-26T19:07:25.756+0000 7f9ea1b36700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-03-26T19:07:01.327054+0000 front 2024-03-26T19:07:01.327074+0000 (oldest deadline 2024-03-26T19:07:23.627096+0000) 2024-03-26T19:07:26.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:26 smithi190 podman[95849]: 2024-03-26 19:07:26.71447308 +0000 UTC m=+5.566464355 container died 60afb0e4da18d26a13aca2d153eea7b2cce51f55cf9c741be2670074139f57cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240320.145742-3605160, ceph=True) 2024-03-26T19:07:27.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:27 smithi190 podman[95849]: 2024-03-26 19:07:27.284624097 +0000 UTC m=+6.136615414 container cleanup 60afb0e4da18d26a13aca2d153eea7b2cce51f55cf9c741be2670074139f57cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-26T19:07:27.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:27 smithi190 bash[95849]: ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5 2024-03-26T19:07:27.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:27 smithi190 podman[95866]: 2024-03-26 19:07:27.519066802 +0000 UTC m=+0.798908034 container remove 60afb0e4da18d26a13aca2d153eea7b2cce51f55cf9c741be2670074139f57cc (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-26T19:07:28.169 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:27 smithi190 podman[95961]: 2024-03-26 19:07:27.753148641 +0000 UTC m=+0.017769676 image pull 497bb87bc5da137bba1d1c5a539351c8d0563edd13b9adcc04bcd9377cdbb016 quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9 2024-03-26T19:07:28.170 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:27 smithi190 podman[95961]: 2024-03-26 19:07:27.861599469 +0000 UTC m=+0.126220536 container create b6c87bfa996fd14c108b1e04cec55fb2f971d2619f44b493e08b79cfe4507799 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240320.145742-3605160, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T19:07:28.170 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:28 smithi190 podman[95961]: 2024-03-26 19:07:28.080459571 +0000 UTC m=+0.345080596 container init b6c87bfa996fd14c108b1e04cec55fb2f971d2619f44b493e08b79cfe4507799 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240320.145742-3605160) 2024-03-26T19:07:28.170 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:28 smithi190 podman[95961]: 2024-03-26 19:07:28.083442193 +0000 UTC m=+0.348063224 container start b6c87bfa996fd14c108b1e04cec55fb2f971d2619f44b493e08b79cfe4507799 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T19:07:28.170 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:28 smithi190 podman[95961]: 2024-03-26 19:07:28.172069931 +0000 UTC m=+0.436690971 container attach b6c87bfa996fd14c108b1e04cec55fb2f971d2619f44b493e08b79cfe4507799 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240320.145742-3605160, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-26T19:07:28.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 26 19:07:28 smithi190 podman[95961]: 2024-03-26 19:07:28.382975147 +0000 UTC m=+0.647596173 container died b6c87bfa996fd14c108b1e04cec55fb2f971d2619f44b493e08b79cfe4507799 (image=quay.ceph.io/ceph-ci/ceph@sha256:d8dd1e9aa1093beda52502a2a16674060866f86acfc8d3eb725f25f5e1398ac9, name=ceph-ed4e027c-eba0-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240320.145742-3605160, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T19:07:29.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed4e027c-eba0-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T19:07:29.071 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T19:07:29.072 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T19:07:29.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T19:07:29.222 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T19:07:52.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T19:07:52.647 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T19:08:06.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T19:08:06.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T19:08:06.428 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T19:08:06.430 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819/remote/smithi102/crash 2024-03-26T19:08:06.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/crash -- . 2024-03-26T19:08:06.469 INFO:teuthology.orchestra.run.smithi102.stderr:tar: /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T19:08:06.469 INFO:teuthology.orchestra.run.smithi102.stderr:tar: Error is not recoverable: exiting now 2024-03-26T19:08:06.471 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819/remote/smithi190/crash 2024-03-26T19:08:06.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/crash -- . 2024-03-26T19:08:06.498 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T19:08:06.498 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now 2024-03-26T19:08:06.500 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T19:08:06.500 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v '\(PG_DEGRADED' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-26T19:08:06.553 INFO:teuthology.orchestra.run.smithi102.stdout:2024-03-26T18:57:59.283743+0000 mon.smithi102 (mon.0) 263 : cluster [WRN] Health check failed: Reduced data availability: 27 pgs peering (PG_AVAILABILITY) 2024-03-26T19:08:06.554 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T19:08:06.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[SEC\]' /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v '\(PG_DEGRADED' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-26T19:08:06.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]' /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v '\(PG_DEGRADED' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-26T19:08:06.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[WRN\]' /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v '\(PG_DEGRADED' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-26T19:08:06.777 INFO:teuthology.orchestra.run.smithi102.stdout:2024-03-26T18:57:59.283743+0000 mon.smithi102 (mon.0) 263 : cluster [WRN] Health check failed: Reduced data availability: 27 pgs peering (PG_AVAILABILITY) 2024-03-26T19:08:06.777 INFO:tasks.cephadm:Compressing logs... 2024-03-26T19:08:06.777 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T19:08:06.820 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T19:08:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory 2024-03-26T19:08:06.846 INFO:teuthology.orchestra.run.smithi102.stderr: /var/log/ceph/cephadm.log 2024-03-26T19:08:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi102.tvsqaa.log 2024-03-26T19:08:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mon.smithi102.log 2024-03-26T19:08:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi102.tvsqaa.log: gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-client.ceph-exporter.smithi102.log 2024-03-26T19:08:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mon.smithi102.log: gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi102.hrbrsp.log 2024-03-26T19:08:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-client.ceph-exporter.smithi102.log: gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T19:08:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi102.hrbrsp.log: gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T19:08:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mgr.smithi102.rsfpnv.log 2024-03-26T19:08:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T19:08:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mgr.smithi102.rsfpnv.log: 94.0% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-client.ceph-exporter.smithi102.log.gz 2024-03-26T19:08:06.851 INFO:teuthology.orchestra.run.smithi102.stderr: 91.6% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T19:08:06.853 INFO:teuthology.orchestra.run.smithi190.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:: No such file or directory 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi190.vykizb.log 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi190.vykizb.log: gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-client.ceph-exporter.smithi190.log 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi190.hwsvdg.log 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.4.log: /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-client.ceph-exporter.smithi190.log: gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi190.hwsvdg.log: gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.5.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mgr.smithi190.zrixos.log 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mon.smithi190.log 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.3.log: /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mgr.smithi190.zrixos.log: 94.0% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-client.ceph-exporter.smithi190.log.gz 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T19:08:06.854 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mon.smithi190.log: 91.7% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T19:08:06.862 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T19:08:06.862 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.log 2024-03-26T19:08:06.863 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.cephadm.log: 85.9% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T19:08:06.863 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T19:08:06.864 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.log 2024-03-26T19:08:06.864 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.cephadm.log: 85.8% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T19:08:06.867 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.log: 87.9% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T19:08:06.867 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T19:08:06.869 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.2.log: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T19:08:06.876 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T19:08:06.879 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.log: 87.9% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T19:08:06.885 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-volume.log: 91.7% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T19:08:06.898 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-volume.log: 89.1% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mgr.smithi190.zrixos.log.gz 2024-03-26T19:08:06.902 INFO:teuthology.orchestra.run.smithi190.stderr: 91.9% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T19:08:07.236 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mgr.smithi102.rsfpnv.log.gz 2024-03-26T19:08:07.239 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mon.smithi190.log.gz 2024-03-26T19:08:07.688 INFO:teuthology.orchestra.run.smithi102.stderr: 90.6% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mon.smithi102.log.gz 2024-03-26T19:08:11.081 INFO:teuthology.orchestra.run.smithi102.stderr: 93.5% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T19:08:11.645 INFO:teuthology.orchestra.run.smithi190.stderr: 95.0% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi190.hwsvdg.log.gz 2024-03-26T19:08:11.732 INFO:teuthology.orchestra.run.smithi102.stderr: 93.4% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T19:08:11.882 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T19:08:12.575 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T19:08:12.810 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T19:08:13.064 INFO:teuthology.orchestra.run.smithi102.stderr: 94.9% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi102.tvsqaa.log.gz 2024-03-26T19:08:13.342 INFO:teuthology.orchestra.run.smithi102.stderr: 93.5% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T19:08:45.354 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi190.vykizb.log.gz 2024-03-26T19:08:45.355 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-26T19:08:45.355 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m38.521s 2024-03-26T19:08:45.356 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m58.431s 2024-03-26T19:08:45.356 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m1.966s 2024-03-26T19:08:57.520 INFO:teuthology.orchestra.run.smithi102.stderr: 92.8% -- replaced with /var/log/ceph/ed4e027c-eba0-11ee-95d0-87774f69a715/ceph-mds.cephfs.smithi102.hrbrsp.log.gz 2024-03-26T19:08:57.536 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-26T19:08:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m50.690s 2024-03-26T19:08:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:user 1m11.389s 2024-03-26T19:08:57.537 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m2.257s 2024-03-26T19:08:57.537 INFO:tasks.cephadm:Archiving logs... 2024-03-26T19:08:57.537 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819/remote/smithi102/log 2024-03-26T19:08:57.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T19:09:08.514 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819/remote/smithi190/log 2024-03-26T19:09:08.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T19:09:17.185 INFO:tasks.cephadm:Removing cluster... 2024-03-26T19:09:17.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 --force 2024-03-26T19:09:17.349 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T19:09:17.704 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ed4e027c-eba0-11ee-95d0-87774f69a715 --force 2024-03-26T19:09:17.850 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: ed4e027c-eba0-11ee-95d0-87774f69a715 2024-03-26T19:09:18.202 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T19:09:18.202 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T19:09:18.219 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T19:09:18.235 INFO:tasks.cephadm:Teardown complete 2024-03-26T19:09:18.236 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T19:09:18.278 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T19:09:18.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T19:09:18.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T19:09:18.365 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T19:09:18.379 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T19:09:18.380 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T19:09:18.383 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T19:09:18.396 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-26T19:09:18.396 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-03-26T19:09:18.400 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T19:09:18.400 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-03-26T19:09:18.400 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +596us[ +596us] +/- 74ms 2024-03-26T19:09:18.400 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -759us[ -734us] +/- 68ms 2024-03-26T19:09:18.400 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +253us[ +278us] +/- 73ms 2024-03-26T19:09:18.400 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T19:09:18.402 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T19:09:18.402 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-26T19:09:18.402 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +599us[ +594us] +/- 74ms 2024-03-26T19:09:18.403 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -780us[ -785us] +/- 68ms 2024-03-26T19:09:18.403 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +237us[ +232us] +/- 73ms 2024-03-26T19:09:18.403 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T19:09:18.403 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T19:09:18.417 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T19:09:18.419 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T19:09:18.438 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T19:09:18.533 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T19:09:18.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T19:09:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T19:09:18.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T19:09:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T19:09:18.675 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T19:09:18.687 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T19:09:18.698 INFO:teuthology.task.internal:Duration was 1886.032051 seconds 2024-03-26T19:09:18.698 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T19:09:18.710 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T19:09:18.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T19:09:18.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T19:09:18.754 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T19:09:18.759 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T19:09:19.172 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T19:09:19.173 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-03-26T19:09:19.173 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T19:09:19.234 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-03-26T19:09:19.234 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T19:09:19.257 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T19:09:19.258 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T19:09:19.277 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T19:09:19.327 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T19:09:19.327 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T19:09:19.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T19:09:19.963 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T19:09:19.982 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T19:09:19.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T19:09:20.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T19:09:20.038 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T19:09:20.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T19:09:20.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T19:09:20.077 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-03-26T19:09:20.103 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-03-26T19:09:20.120 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T19:09:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T19:09:20.148 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T19:09:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T19:09:20.174 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T19:09:20.189 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T19:09:20.191 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819/remote/smithi102 2024-03-26T19:09:20.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T19:09:20.250 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819/remote/smithi190 2024-03-26T19:09:20.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T19:09:20.295 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T19:09:20.296 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T19:09:20.298 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T19:09:20.341 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T19:09:20.355 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T19:09:20.356 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T19:09:20.374 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T19:09:20.375 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T19:09:20.377 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T19:09:20.389 INFO:teuthology.orchestra.run.smithi102.stdout: 265945 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 19:09 /home/ubuntu/cephtest 2024-03-26T19:09:20.396 INFO:teuthology.orchestra.run.smithi190.stdout: 265317 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 19:09 /home/ubuntu/cephtest 2024-03-26T19:09:20.397 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T19:09:20.408 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T19:09:20.432 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T19:09:20.458 DEBUG:teuthology.nuke:shortname: smithi102 2024-03-26T19:09:20.458 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T19:09:20.480 DEBUG:teuthology.nuke:shortname: smithi190 2024-03-26T19:09:20.481 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T19:09:20.484 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 18:28:57.764628', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZpbodomQLvthcoICNQcf6ibB9ilx0/GNeph2m0RDf7GpIp5VcACBcLxaaQvoespXI1c/rn5zInyCJ3nJZ5s2k='} 2024-03-26T19:09:20.501 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-03-26_17:10:07-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7623819', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 18:28:57.766374', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLf6khNxzXZdL3tBloMh3uw9/F6YAfmhLce1rGzKsbtkP/pBLC3kxF6TIpePJZGs1ZVsUdXgTIeg8vFfa9brxd8='} 2024-03-26T19:09:20.534 INFO:teuthology.orchestra.console.smithi102:Power off 2024-03-26T19:09:20.534 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T19:09:20.558 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off 2024-03-26T19:09:20.559 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T19:09:20.576 INFO:teuthology.orchestra.console.smithi190:Power off 2024-03-26T19:09:20.577 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T19:09:20.587 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-26T19:09:20.601 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2024-03-26T19:09:20.601 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T19:09:20.625 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-03-26T19:09:24.589 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T19:09:24.626 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T19:09:24.699 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-26T19:09:24.737 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-03-26T19:09:28.702 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T19:09:28.739 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T19:09:28.813 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-26T19:09:28.850 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2024-03-26T19:09:28.950 INFO:teuthology.orchestra.console.smithi190:Power off completed 2024-03-26T19:09:32.814 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T19:09:32.924 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off 2024-03-26T19:09:33.025 INFO:teuthology.orchestra.console.smithi102:Power off completed 2024-03-26T19:09:33.129 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1886.0320506095886 failure_reason: '"2024-03-26T18:57:59.283743+0000 mon.smithi102 (mon.0) 263 : cluster [WRN] Health check failed: Reduced data availability: 27 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_lflores@teuthology success: false 2024-03-26T19:09:33.129 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T19:09:33.207 INFO:teuthology.run:FAIL